Re: [cfe-users] Strange problem with C preprocessor include file searching (Mac, llvm 10)
I have some more clues here. I can reproduce this strange behavior with just
bash, no python needed.
$ oslc -IincA -IincB blah.osl
adding 'incA'<-- my debug, shows that the '-IincA' was parsed on the
command line
adding 'incB'<-- my debug, shows that the '-IincB' was parsed on the
command line
#include "..." search starts here:
#include <...> search starts here:
incA<-- LLVM diagnostics, shows I passed incA in includedirs
list
incB<-- LLVM diagnostics, shows I passed incB in includedirs
list
End of search list.
Compiled blah.osl -> blah.oso
Works fine. Now what if I pass the same commands to 'bash -c':
$ /bin/bash -c "oslc -IincA -IincB blah.osl"
adding 'incA'
adding 'incB'
#include "..." search starts here:
#include <...> search starts here:
incA
incB
End of search list.
error: blah.osl:3:10: fatal error: cannot open file 'incA/b.h': No such file or
directory
#include "b.h"
^
FAILED blah.osl
But this ONLY fails if I'm using llvm 10. If I rebuild my app with llvm@9 from
homebrew instead of llvm (which is llvm 10):
$ /bin/bash -c "oslc -IincA -IincB blah.osl"
adding 'incA'
adding 'incB'
#include "..." search starts here:
#include <...> search starts here:
incA
incB
End of search list.
Compiled blah.osl -> blah.oso
Works fine again. I don't have a working theory for what's going on here.
So I know the -I commands are making it to my program, and I know I'm passing
those paths to libclang, because its own diagnostics list those directories.
But it nonetheless fails to find headers that aren't in the very first included
searchpath -- ONLY for llvm 10, ONLY on Mac, ONLY if I'm doing it through a
second spawned shell (works fine when I directly type the command).
Any guesses?
> On Apr 27, 2020, at 12:00 PM, Larry Gritz via cfe-users
> wrote:
>
> Excuse if this is a tricky explanation; I'm not sure I understand what's
> going on.
>
> I have a C-like language and compiler for which I use clang libraries to do
> the preprocessing. My compiler lets users specify `-I` directories for
> searchpaths for includes, per usual convention. I'm doing something like this:
>
>clang::HeaderSearchOptions &headerOpts =
> compilerinst.getHeaderSearchOpts();
>headerOpts.UseBuiltinIncludes = 0;
>headerOpts.UseStandardSystemIncludes = 0;
>headerOpts.UseStandardCXXIncludes = 0;
>for (auto&& inc : includepaths) {
>headerOpts.AddPath (inc, clang::frontend::Quoted,
>false /* not a framework */,
>true /* ignore sys root */);
>}
>
>
> For the sake of a simple failure case, I have header a.h in directory incA/,
> and header b.h in incB/, and my test program just consists of
>
>#include "a.h"
>#include "b.h"
>
> Also, I have set this to turn on some debugging:
>headerOpts.Verbose = 1; // DEBUGGING
>
> Now, when I invoke my compiler from the command line,
>
>oslc -IincA -IincB test.osl
>
> I get this output:
>
>#include "..." search starts here:
>#include <...> search starts here:
> incA
> incB
>End of search list.
>
> and my compile succeeds. As expected, and as it has for many many years.
>
> But, as part of my compiler's test suite, there is a python script involved
> that boils down to:
>
>#!/usr/bin/env python
>import subprocess
>subprocess.call ('oslc -IincA -IincB test.osl', shell=True)
>
> When I run the python program,
>
>python mytest.py
>
> then I get this output:
>
>#include "..." search starts here:
>#include <...> search starts here:
> incA
> incB
>End of search list.
>error: test.osl:3:10: fatal error: cannot open file 'incA/b.h': No such
> file or directory
>#include "b.h"
> ^
>FAILED test.osl
>
> Wha? So I've poked around a bit with the behavior, and near as I can tell,
> even though the diagnostics say that both incA and incB are in the search
> list, it's only actually searching the first directory listed.
>
> Now, this only happens on OSX, and only when I'm using clang 10 libraries
> (installed via Homebrew, though also when I build clang from scratch). Works
> fine on Linux. Works fine on all platforms for clang 9, 8, 7, 6, and I've
> been using this since back to 3.3 or so. Only had this problem after
> upgrading to clang/llvm 10, and only on OSX. Fails the same way for python
> 2.7 and 3.7.
>
> If I change the subprocess.call to:
>
>subprocess.call (['oslc', '-IincA', '-IincB', 'blah.osl'], shell=False)
>
> it succeeds. (But in real life, this isn't an adequate workaround, because I
> want to use shell=True and keep the whole command line together, because it's
> really an arbitrary shell command that has output redirect.)
>
> Does any of this ring a bell for anybody? Or does anyone have suggestions for
> what to try next?
>
>
--
Larry Gritz
l...@larrygritz.com
Re: [cfe-users] Strange problem with C preprocessor include file searching (Mac, llvm 10)
So I was able to also reproduce simply as this:
works: oslc -IincA -IincB blah.osl
broken: DYLD_LIBRARY_PATH= oslc -IincA -IincB blah.osl
The only thing in my DYLD_LIBRARY_PATH was /usr/local/opt/llvm
It seems that what was happening was that in cases where I didn't have
/usr/local/opt/llvm/lib in my DYLD_LIBRARY_PATH (as was the case when I ran my
compiler from a subshell), it was ending up linking TWO copies of libc++.1.so,
one that was build with LLVM, and one system version in /usr/lib, and somehow
that led to this weird behavior.
There is apparently some known linkage problems related to this, with the
homebrew build of llvm when on Mojave. From what I understand, Catalina's
support for a newer xcode apparently also fixes, but neither specific xcode
version nor having DYLD_LIBRARY_PATH set a particular way is an assumption I
can make with respect to my users.
I also noticed that this isn't a problem when I statically link llvm and clang
libraries to my app, so my solution for now is to just force my app's build
system to use the static llvm+clang libs rather than dynamic when on OSX and
the llvm version is >= 10.
-- lg
> On Apr 28, 2020, at 2:23 PM, Larry Gritz wrote:
>
> I have some more clues here. I can reproduce this strange behavior with just
> bash, no python needed.
>
>
> $ oslc -IincA -IincB blah.osl
> adding 'incA'<-- my debug, shows that the '-IincA' was parsed on the
> command line
> adding 'incB'<-- my debug, shows that the '-IincB' was parsed on the
> command line
> #include "..." search starts here:
> #include <...> search starts here:
> incA<-- LLVM diagnostics, shows I passed incA in includedirs
> list
> incB<-- LLVM diagnostics, shows I passed incB in includedirs
> list
> End of search list.
> Compiled blah.osl -> blah.oso
>
>
> Works fine. Now what if I pass the same commands to 'bash -c':
>
>
> $ /bin/bash -c "oslc -IincA -IincB blah.osl"
> adding 'incA'
> adding 'incB'
> #include "..." search starts here:
> #include <...> search starts here:
> incA
> incB
> End of search list.
> error: blah.osl:3:10: fatal error: cannot open file 'incA/b.h': No such file
> or directory
> #include "b.h"
> ^
> FAILED blah.osl
>
>
> But this ONLY fails if I'm using llvm 10. If I rebuild my app with llvm@9
> from homebrew instead of llvm (which is llvm 10):
>
> $ /bin/bash -c "oslc -IincA -IincB blah.osl"
> adding 'incA'
> adding 'incB'
> #include "..." search starts here:
> #include <...> search starts here:
> incA
> incB
> End of search list.
> Compiled blah.osl -> blah.oso
>
>
> Works fine again. I don't have a working theory for what's going on here.
>
> So I know the -I commands are making it to my program, and I know I'm passing
> those paths to libclang, because its own diagnostics list those directories.
> But it nonetheless fails to find headers that aren't in the very first
> included searchpath -- ONLY for llvm 10, ONLY on Mac, ONLY if I'm doing it
> through a second spawned shell (works fine when I directly type the command).
>
> Any guesses?
>
>
>> On Apr 27, 2020, at 12:00 PM, Larry Gritz via cfe-users
>> wrote:
>>
>> Excuse if this is a tricky explanation; I'm not sure I understand what's
>> going on.
>>
>> I have a C-like language and compiler for which I use clang libraries to do
>> the preprocessing. My compiler lets users specify `-I` directories for
>> searchpaths for includes, per usual convention. I'm doing something like
>> this:
>>
>> clang::HeaderSearchOptions &headerOpts =
>> compilerinst.getHeaderSearchOpts();
>> headerOpts.UseBuiltinIncludes = 0;
>> headerOpts.UseStandardSystemIncludes = 0;
>> headerOpts.UseStandardCXXIncludes = 0;
>> for (auto&& inc : includepaths) {
>> headerOpts.AddPath (inc, clang::frontend::Quoted,
>> false /* not a framework */,
>> true /* ignore sys root */);
>> }
>>
>>
>> For the sake of a simple failure case, I have header a.h in directory incA/,
>> and header b.h in incB/, and my test program just consists of
>>
>> #include "a.h"
>> #include "b.h"
>>
>> Also, I have set this to turn on some debugging:
>> headerOpts.Verbose = 1; // DEBUGGING
>>
>> Now, when I invoke my compiler from the command line,
>>
>> oslc -IincA -IincB test.osl
>>
>> I get this output:
>>
>> #include "..." search starts here:
>> #include <...> search starts here:
>>incA
>>incB
>> End of search list.
>>
>> and my compile succeeds. As expected, and as it has for many many years.
>>
>> But, as part of my compiler's test suite, there is a python script involved
>> that boils down to:
>>
>> #!/usr/bin/env python
>> import subprocess
>> subprocess.call ('oslc -IincA -IincB test.osl', shell=True)
>>
>> When I run the python program,
>>
>> python mytest.py
>>
>> then I get this output:
>>
>> #include "..." search starts here:
>>
