[lldb-dev] Linux: Failing lldb unit tests and TestLinuxCore.py
Hi By using "ninja check-lldb-unit" some of the unit tests fail with the tip of the master. (I am using Ubuntu Linux.) The error is always related to the CommandLine options: FAIL: lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences (120 of 363) TEST 'lldb-Unit :: ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences' FAILED Note: Google Test filter = PythonDataObjectsTest.TestBorrowedReferences [==] Running 1 test from 1 test case. [--] Global test environment set-up. [--] 1 test from PythonDataObjectsTest [ RUN ] PythonDataObjectsTest.TestBorrowedReferences : CommandLine Error: Option 'help-list' registered more than once! LLVM ERROR: inconsistency in registered CommandLine options The other lit tests pass, except TestLinuxCore.py. I already set up LD_LIBRARY_PATH to point to $build/bin. Am I missing something else? Any help would be appreciated. Thanks, Gábor ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] Linux: Failing lldb unit tests and TestLinuxCore.py
Hello Gabor, The error message (Option 'help-list' registered more than once) leads me to believe that this is caused by an uncommon (for lldb, at least) build configuration. Can you share your cmake configuration? I suspect TestLinuxCore is also failing due to unexpected build configuration, specifically missing targets in llvm (mips?). I'll try to fix that this week. On Tue, 26 Jun 2018 at 14:44, Gábor Márton via lldb-dev wrote: > > Hi > > By using "ninja check-lldb-unit" some of the unit tests fail with the tip of > the master. (I am using Ubuntu Linux.) The error is always related to the > CommandLine options: > > FAIL: lldb-Unit :: > ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences > (120 of 363) > TEST 'lldb-Unit :: > ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences' > FAILED > Note: Google Test filter = PythonDataObjectsTest.TestBorrowedReferences > [==] Running 1 test from 1 test case. > [--] Global test environment set-up. > [--] 1 test from PythonDataObjectsTest > [ RUN ] PythonDataObjectsTest.TestBorrowedReferences > : CommandLine Error: Option 'help-list' registered more than once! > LLVM ERROR: inconsistency in registered CommandLine options > > The other lit tests pass, except TestLinuxCore.py. > > I already set up LD_LIBRARY_PATH to point to $build/bin. > Am I missing something else? > > Any help would be appreciated. > > Thanks, > Gábor > ___ > lldb-dev mailing list > lldb-dev@lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] Linux: Failing lldb unit tests and TestLinuxCore.py
Hi Pavel, I do an in-tree build, so lldb is next to clang. And I use ccache, could that be a problem? If yes then why just with the unit tests? Here is how I configured cmake: cmake ../../git/llvm -G Ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_C_COMPILER=clang -DBUILD_SHARED_LIBS=1 -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_ENABLE_ASSERTIONS=1 Thanks, Gabor On Tue, Jun 26, 2018 at 5:05 PM Pavel Labath wrote: > Hello Gabor, > > The error message (Option 'help-list' registered more than once) leads > me to believe that this is caused by an uncommon (for lldb, at least) > build configuration. Can you share your cmake configuration? > > I suspect TestLinuxCore is also failing due to unexpected build > configuration, specifically missing targets in llvm (mips?). I'll try > to fix that this week. > On Tue, 26 Jun 2018 at 14:44, Gábor Márton via lldb-dev > wrote: > > > > Hi > > > > By using "ninja check-lldb-unit" some of the unit tests fail with the > tip of the master. (I am using Ubuntu Linux.) The error is always related > to the CommandLine options: > > > > FAIL: lldb-Unit :: > ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences > (120 of 363) > > TEST 'lldb-Unit :: > ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences' > FAILED > > Note: Google Test filter = PythonDataObjectsTest.TestBorrowedReferences > > [==] Running 1 test from 1 test case. > > [--] Global test environment set-up. > > [--] 1 test from PythonDataObjectsTest > > [ RUN ] PythonDataObjectsTest.TestBorrowedReferences > > : CommandLine Error: Option 'help-list' registered more than once! > > LLVM ERROR: inconsistency in registered CommandLine options > > > > The other lit tests pass, except TestLinuxCore.py. > > > > I already set up LD_LIBRARY_PATH to point to $build/bin. > > Am I missing something else? > > > > Any help would be appreciated. > > > > Thanks, > > Gábor > > ___ > > lldb-dev mailing list > > lldb-dev@lists.llvm.org > > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev > ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] Linux: Failing lldb unit tests and TestLinuxCore.py
The BUILD_SHARED_LIBS part is likely to be the issue. LLDB libraries have cyclic dependencies, which don't play well with shared libraries. I think somebody "fixed" it some time ago (i.e., just made the flag be ignored for lldb), but I'm not sure if he also checked the unit tests. I'll see if I can reproduce that on my end. On Tue, 26 Jun 2018 at 16:28, Gábor Márton wrote: > > Hi Pavel, > > I do an in-tree build, so lldb is next to clang. > And I use ccache, could that be a problem? If yes then why just with the unit > tests? > Here is how I configured cmake: > cmake ../../git/llvm -G Ninja -DCMAKE_EXPORT_COMPILE_COMMANDS=1 > -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=clang++ > -DCMAKE_C_COMPILER=clang -DBUILD_SHARED_LIBS=1 -DLLVM_TARGETS_TO_BUILD=X86 > -DLLVM_ENABLE_ASSERTIONS=1 > > Thanks, > Gabor > > On Tue, Jun 26, 2018 at 5:05 PM Pavel Labath wrote: >> >> Hello Gabor, >> >> The error message (Option 'help-list' registered more than once) leads >> me to believe that this is caused by an uncommon (for lldb, at least) >> build configuration. Can you share your cmake configuration? >> >> I suspect TestLinuxCore is also failing due to unexpected build >> configuration, specifically missing targets in llvm (mips?). I'll try >> to fix that this week. >> On Tue, 26 Jun 2018 at 14:44, Gábor Márton via lldb-dev >> wrote: >> > >> > Hi >> > >> > By using "ninja check-lldb-unit" some of the unit tests fail with the tip >> > of the master. (I am using Ubuntu Linux.) The error is always related to >> > the CommandLine options: >> > >> > FAIL: lldb-Unit :: >> > ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences >> > (120 of 363) >> > TEST 'lldb-Unit :: >> > ScriptInterpreter/Python/./ScriptInterpreterPythonTests/PythonDataObjectsTest.TestBorrowedReferences' >> > FAILED >> > Note: Google Test filter = PythonDataObjectsTest.TestBorrowedReferences >> > [==] Running 1 test from 1 test case. >> > [--] Global test environment set-up. >> > [--] 1 test from PythonDataObjectsTest >> > [ RUN ] PythonDataObjectsTest.TestBorrowedReferences >> > : CommandLine Error: Option 'help-list' registered more than once! >> > LLVM ERROR: inconsistency in registered CommandLine options >> > >> > The other lit tests pass, except TestLinuxCore.py. >> > >> > I already set up LD_LIBRARY_PATH to point to $build/bin. >> > Am I missing something else? >> > >> > Any help would be appreciated. >> > >> > Thanks, >> > Gábor >> > ___ >> > lldb-dev mailing list >> > lldb-dev@lists.llvm.org >> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
[lldb-dev] RFC: libtrace
Hi all, We have been thinking internally about a lightweight llvm-based ptracer. To address one question up front: the primary way in which this differs from LLDB is that it targets a more narrow use case -- there is no scripting support, no clang integration, no dynamic extensibility, no support for running jitted code in the target, and no user interface. We have several use cases internally that call for varying levels of functionality from such a utility, and being able to use as little as possible of the library as is necessary for the given task is important for the scale in which we wish to use it. We are still in early discussions and planning, but I think this would be a good addition to the LLVM upstream. Since we’re approaching this as a set of small isolated components, my thinking is to work on this completely upstream, directly under the llvm project (as opposed to making a separate subproject), but I’m open to discussion if anyone feels differently. LLDB has solved a lot of the difficult problems needed for such a tool. So in the spirit of code reuse, we think it’s worth trying componentize LLDB by sinking pieces into LLVM and rebasing LLDB as well as these smaller tools on top of these components, so that smaller tools can reduce code duplication and contribute to the overall health of the code base. At the same time we think that in doing so we can break things up into more granular pieces, ultimately exposing a larger testing surface and enabling us to create exhaustive tests, giving LLDB more fine grained testing of important subsystems. A good example of this would be LLDB’s DWARF parsing code, which is more featureful than LLVM’s but has kind of evolved in parallel. Sinking this into LLVM would be one early target of such an effort, although over time there would likely be more. Anyone have any thoughts / strong opinions on this proposal, or where the code should live? Also, does anyone have any suggestions on things they’d like to see come out of this? Whether it’s a specific new tool, new functionality to an existing tool, an architectural or design change to some existing tool or library, or something else entirely, all feedback and ideas are welcome. Thanks, Zach ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] RFC: libtrace
Just to be clear, by "no clang integration" do you mean "no expression parser" or do you mean something more radical? For instance, adding a TypeSystem and its DWARF parser for C family languages that uses a different underlying representation than Clang AST's to store the results would be a lot of work that wouldn't be terribly interesting to lldb. I don't think that's what you meant, but wanted to be sure. Jim > On Jun 26, 2018, at 11:58 AM, Zachary Turner via lldb-dev > wrote: > > Hi all, > > We have been thinking internally about a lightweight llvm-based ptracer. To > address one question up front: the primary way in which this differs from > LLDB is that it targets a more narrow use case -- there is no scripting > support, no clang integration, no dynamic extensibility, no support for > running jitted code in the target, and no user interface. We have several > use cases internally that call for varying levels of functionality from such > a utility, and being able to use as little as possible of the library as is > necessary for the given task is important for the scale in which we wish to > use it. > > We are still in early discussions and planning, but I think this would be a > good addition to the LLVM upstream. Since we’re approaching this as a set of > small isolated components, my thinking is to work on this completely > upstream, directly under the llvm project (as opposed to making a separate > subproject), but I’m open to discussion if anyone feels differently. > > LLDB has solved a lot of the difficult problems needed for such a tool. So > in the spirit of code reuse, we think it’s worth trying componentize LLDB by > sinking pieces into LLVM and rebasing LLDB as well as these smaller tools on > top of these components, so that smaller tools can reduce code duplication > and contribute to the overall health of the code base. At the same time we > think that in doing so we can break things up into more granular pieces, > ultimately exposing a larger testing surface and enabling us to create > exhaustive tests, giving LLDB more fine grained testing of important > subsystems. > > A good example of this would be LLDB’s DWARF parsing code, which is more > featureful than LLVM’s but has kind of evolved in parallel. Sinking this > into LLVM would be one early target of such an effort, although over time > there would likely be more. > > Anyone have any thoughts / strong opinions on this proposal, or where the > code should live? Also, does anyone have any suggestions on things they’d > like to see come out of this? Whether it’s a specific new tool, new > functionality to an existing tool, an architectural or design change to some > existing tool or library, or something else entirely, all feedback and ideas > are welcome. > > Thanks, > Zach > > ___ > lldb-dev mailing list > lldb-dev@lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] RFC: libtrace
no expression parser or knowledge of any specific programming language. Basically I just mean that the parsing of the native DWARF format itself is in scope, but anything beyond that is out of scope. For symbolication we have things like llvm-symbolizer that already just work and are built on top of LLVM's dwarf parsing code. Similarly, LLDB's type system could be built on top of it as well. Given that I think everyone mostly agrees that unifying on one DWARF parser is a good idea in principle, this would mean no functional change from LLDB's point of view, it would just continue to do exactly what it does regarding parsing C++ expressions and converting these into types that clang understands. It will probably be useful someday to have an expression parser and language specific type system, but when that comes I don't think we'd want anything radically different than what LLDB already has. On Tue, Jun 26, 2018 at 12:26 PM Jim Ingham wrote: > Just to be clear, by "no clang integration" do you mean "no expression > parser" or do you mean something more radical? For instance, adding a > TypeSystem and its DWARF parser for C family languages that uses a > different underlying representation than Clang AST's to store the results > would be a lot of work that wouldn't be terribly interesting to lldb. I > don't think that's what you meant, but wanted to be sure. > > Jim > > > On Jun 26, 2018, at 11:58 AM, Zachary Turner via lldb-dev < > lldb-dev@lists.llvm.org> wrote: > > > > Hi all, > > > > We have been thinking internally about a lightweight llvm-based > ptracer. To address one question up front: the primary way in which this > differs from LLDB is that it targets a more narrow use case -- there is no > scripting support, no clang integration, no dynamic extensibility, no > support for running jitted code in the target, and no user interface. We > have several use cases internally that call for varying levels of > functionality from such a utility, and being able to use as little as > possible of the library as is necessary for the given task is important for > the scale in which we wish to use it. > > > > We are still in early discussions and planning, but I think this would > be a good addition to the LLVM upstream. Since we’re approaching this as a > set of small isolated components, my thinking is to work on this completely > upstream, directly under the llvm project (as opposed to making a separate > subproject), but I’m open to discussion if anyone feels differently. > > > > LLDB has solved a lot of the difficult problems needed for such a tool. > So in the spirit of code reuse, we think it’s worth trying componentize > LLDB by sinking pieces into LLVM and rebasing LLDB as well as these smaller > tools on top of these components, so that smaller tools can reduce code > duplication and contribute to the overall health of the code base. At the > same time we think that in doing so we can break things up into more > granular pieces, ultimately exposing a larger testing surface and enabling > us to create exhaustive tests, giving LLDB more fine grained testing of > important subsystems. > > > > A good example of this would be LLDB’s DWARF parsing code, which is more > featureful than LLVM’s but has kind of evolved in parallel. Sinking this > into LLVM would be one early target of such an effort, although over time > there would likely be more. > > > > Anyone have any thoughts / strong opinions on this proposal, or where > the code should live? Also, does anyone have any suggestions on things > they’d like to see come out of this? Whether it’s a specific new tool, new > functionality to an existing tool, an architectural or design change to > some existing tool or library, or something else entirely, all feedback and > ideas are welcome. > > > > Thanks, > > Zach > > > > ___ > > lldb-dev mailing list > > lldb-dev@lists.llvm.org > > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev > > ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] RFC: libtrace
So you aren't planning to print values at all, just stop points (i.e. you are only interested in the line table and function symbols part of DWARF)? Given what you've described so far, I'm wondering if what you really want is the NativeProcess classes with some symbol-file reading pulled in? Is there anything that you couldn't do from there? Jim > On Jun 26, 2018, at 12:48 PM, Zachary Turner wrote: > > no expression parser or knowledge of any specific programming language. > > Basically I just mean that the parsing of the native DWARF format itself is > in scope, but anything beyond that is out of scope. For symbolication we > have things like llvm-symbolizer that already just work and are built on top > of LLVM's dwarf parsing code. Similarly, LLDB's type system could be built > on top of it as well. Given that I think everyone mostly agrees that > unifying on one DWARF parser is a good idea in principle, this would mean no > functional change from LLDB's point of view, it would just continue to do > exactly what it does regarding parsing C++ expressions and converting these > into types that clang understands. > > It will probably be useful someday to have an expression parser and language > specific type system, but when that comes I don't think we'd want anything > radically different than what LLDB already has. > > On Tue, Jun 26, 2018 at 12:26 PM Jim Ingham wrote: > Just to be clear, by "no clang integration" do you mean "no expression > parser" or do you mean something more radical? For instance, adding a > TypeSystem and its DWARF parser for C family languages that uses a different > underlying representation than Clang AST's to store the results would be a > lot of work that wouldn't be terribly interesting to lldb. I don't think > that's what you meant, but wanted to be sure. > > Jim > > > On Jun 26, 2018, at 11:58 AM, Zachary Turner via lldb-dev > > wrote: > > > > Hi all, > > > > We have been thinking internally about a lightweight llvm-based ptracer. > > To address one question up front: the primary way in which this differs > > from LLDB is that it targets a more narrow use case -- there is no > > scripting support, no clang integration, no dynamic extensibility, no > > support for running jitted code in the target, and no user interface. We > > have several use cases internally that call for varying levels of > > functionality from such a utility, and being able to use as little as > > possible of the library as is necessary for the given task is important for > > the scale in which we wish to use it. > > > > We are still in early discussions and planning, but I think this would be a > > good addition to the LLVM upstream. Since we’re approaching this as a set > > of small isolated components, my thinking is to work on this completely > > upstream, directly under the llvm project (as opposed to making a separate > > subproject), but I’m open to discussion if anyone feels differently. > > > > LLDB has solved a lot of the difficult problems needed for such a tool. So > > in the spirit of code reuse, we think it’s worth trying componentize LLDB > > by sinking pieces into LLVM and rebasing LLDB as well as these smaller > > tools on top of these components, so that smaller tools can reduce code > > duplication and contribute to the overall health of the code base. At the > > same time we think that in doing so we can break things up into more > > granular pieces, ultimately exposing a larger testing surface and enabling > > us to create exhaustive tests, giving LLDB more fine grained testing of > > important subsystems. > > > > A good example of this would be LLDB’s DWARF parsing code, which is more > > featureful than LLVM’s but has kind of evolved in parallel. Sinking this > > into LLVM would be one early target of such an effort, although over time > > there would likely be more. > > > > Anyone have any thoughts / strong opinions on this proposal, or where the > > code should live? Also, does anyone have any suggestions on things they’d > > like to see come out of this? Whether it’s a specific new tool, new > > functionality to an existing tool, an architectural or design change to > > some existing tool or library, or something else entirely, all feedback and > > ideas are welcome. > > > > Thanks, > > Zach > > > > ___ > > lldb-dev mailing list > > lldb-dev@lists.llvm.org > > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev > ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] RFC: libtrace
You'd probably need to pull the Unwinder in if you want backtraces, but that part shouldn't be that hard to disentangle. I don't think you'd need much else? Basing your work on NativeProcess rather than lldb proper would also cut the number of observer processes in half and avoid the context switches between the server and the debugger. That seems more appropriate for a lightweight tool. Jim > On Jun 26, 2018, at 12:59 PM, Jim Ingham via lldb-dev > wrote: > > So you aren't planning to print values at all, just stop points (i.e. you are > only interested in the line table and function symbols part of DWARF)? > > Given what you've described so far, I'm wondering if what you really want is > the NativeProcess classes with some symbol-file reading pulled in? Is there > anything that you couldn't do from there? > > Jim > > >> On Jun 26, 2018, at 12:48 PM, Zachary Turner wrote: >> >> no expression parser or knowledge of any specific programming language. >> >> Basically I just mean that the parsing of the native DWARF format itself is >> in scope, but anything beyond that is out of scope. For symbolication we >> have things like llvm-symbolizer that already just work and are built on top >> of LLVM's dwarf parsing code. Similarly, LLDB's type system could be built >> on top of it as well. Given that I think everyone mostly agrees that >> unifying on one DWARF parser is a good idea in principle, this would mean no >> functional change from LLDB's point of view, it would just continue to do >> exactly what it does regarding parsing C++ expressions and converting these >> into types that clang understands. >> >> It will probably be useful someday to have an expression parser and language >> specific type system, but when that comes I don't think we'd want anything >> radically different than what LLDB already has. >> >> On Tue, Jun 26, 2018 at 12:26 PM Jim Ingham wrote: >> Just to be clear, by "no clang integration" do you mean "no expression >> parser" or do you mean something more radical? For instance, adding a >> TypeSystem and its DWARF parser for C family languages that uses a different >> underlying representation than Clang AST's to store the results would be a >> lot of work that wouldn't be terribly interesting to lldb. I don't think >> that's what you meant, but wanted to be sure. >> >> Jim >> >>> On Jun 26, 2018, at 11:58 AM, Zachary Turner via lldb-dev >>> wrote: >>> >>> Hi all, >>> >>> We have been thinking internally about a lightweight llvm-based ptracer. >>> To address one question up front: the primary way in which this differs >>> from LLDB is that it targets a more narrow use case -- there is no >>> scripting support, no clang integration, no dynamic extensibility, no >>> support for running jitted code in the target, and no user interface. We >>> have several use cases internally that call for varying levels of >>> functionality from such a utility, and being able to use as little as >>> possible of the library as is necessary for the given task is important for >>> the scale in which we wish to use it. >>> >>> We are still in early discussions and planning, but I think this would be a >>> good addition to the LLVM upstream. Since we’re approaching this as a set >>> of small isolated components, my thinking is to work on this completely >>> upstream, directly under the llvm project (as opposed to making a separate >>> subproject), but I’m open to discussion if anyone feels differently. >>> >>> LLDB has solved a lot of the difficult problems needed for such a tool. So >>> in the spirit of code reuse, we think it’s worth trying componentize LLDB >>> by sinking pieces into LLVM and rebasing LLDB as well as these smaller >>> tools on top of these components, so that smaller tools can reduce code >>> duplication and contribute to the overall health of the code base. At the >>> same time we think that in doing so we can break things up into more >>> granular pieces, ultimately exposing a larger testing surface and enabling >>> us to create exhaustive tests, giving LLDB more fine grained testing of >>> important subsystems. >>> >>> A good example of this would be LLDB’s DWARF parsing code, which is more >>> featureful than LLVM’s but has kind of evolved in parallel. Sinking this >>> into LLVM would be one early target of such an effort, although over time >>> there would likely be more. >>> >>> Anyone have any thoughts / strong opinions on this proposal, or where the >>> code should live? Also, does anyone have any suggestions on things they’d >>> like to see come out of this? Whether it’s a specific new tool, new >>> functionality to an existing tool, an architectural or design change to >>> some existing tool or library, or something else entirely, all feedback and >>> ideas are welcome. >>> >>> Thanks, >>> Zach >>> >>> ___ >>> lldb-dev mailing list >>> lldb-dev@l
Re: [lldb-dev] [llvm-dev] RFC: libtrace
> On Jun 26, 2018, at 11:58 AM, Zachary Turner via llvm-dev > wrote: > > Hi all, > > We have been thinking internally about a lightweight llvm-based ptracer. To > address one question up front: the primary way in which this differs from > LLDB is that it targets a more narrow use case -- there is no scripting > support, no clang integration, no dynamic extensibility, no support for > running jitted code in the target, and no user interface. We have several > use cases internally that call for varying levels of functionality from such > a utility, and being able to use as little as possible of the library as is > necessary for the given task is important for the scale in which we wish to > use it. > > We are still in early discussions and planning, but I think this would be a > good addition to the LLVM upstream. Since we’re approaching this as a set of > small isolated components, my thinking is to work on this completely > upstream, directly under the llvm project (as opposed to making a separate > subproject), but I’m open to discussion if anyone feels differently. > > LLDB has solved a lot of the difficult problems needed for such a tool. So > in the spirit of code reuse, we think it’s worth trying componentize LLDB by > sinking pieces into LLVM and rebasing LLDB as well as these smaller tools on > top of these components, so that smaller tools can reduce code duplication > and contribute to the overall health of the code base. Do you have a rough idea of what components specifically the new tool would need to function? > At the same time we think that in doing so we can break things up into more > granular pieces, ultimately exposing a larger testing surface and enabling us > to create exhaustive tests, giving LLDB more fine grained testing of > important subsystems. Are you thinking of the new utility as something that would naturally live in llvm/tools or as something that would live in the LLDB repository? > > A good example of this would be LLDB’s DWARF parsing code, which is more > featureful than LLVM’s but has kind of evolved in parallel. Sinking this > into LLVM would be one early target of such an effort, although over time > there would likely be more. As you are undoubtedly aware we've been carefully rearchitecting LLVM's DWARF parser over the last few years to eventually become featureful enough so that LLDB could use it, so any help on that front would be most welcome. As long as we are careful to not regress in performance/lazyness, features and fault-tolerance, deduplicating the implementations can only be good for LLVM and LLDB. -- adrian > > Anyone have any thoughts / strong opinions on this proposal, or where the > code should live? Also, does anyone have any suggestions on things they’d > like to see come out of this? Whether it’s a specific new tool, new > functionality to an existing tool, an architectural or design change to some > existing tool or library, or something else entirely, all feedback and ideas > are welcome. > > Thanks, > Zach > > ___ > LLVM Developers mailing list > llvm-...@lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] RFC: libtrace
The various NativeProcess implementations are definitely a good starting point and I'll probably be looking at them to understand all the ins and outs of each platform. I'm not sure if the API / interface we want will be the same, so I don't think we can just copy it all down. But a lot of the core logic we probably can. Depending on how much of it we end up implementing and how close we get to the current functionality of the NativeProcess classes, this could be another area for code reuse similar to what I mentioned with the DWARF reading. i.e. we could write lots of low-level tests of the tracing functionality specifically, then update the NativeProcess implementations to use this. On Tue, Jun 26, 2018 at 1:09 PM Jim Ingham wrote: > You'd probably need to pull the Unwinder in if you want backtraces, but > that part shouldn't be that hard to disentangle. I don't think you'd need > much else? > > Basing your work on NativeProcess rather than lldb proper would also cut > the number of observer processes in half and avoid the context switches > between the server and the debugger. That seems more appropriate for a > lightweight tool. > > Jim > > > > On Jun 26, 2018, at 12:59 PM, Jim Ingham via lldb-dev < > lldb-dev@lists.llvm.org> wrote: > > > > So you aren't planning to print values at all, just stop points (i.e. > you are only interested in the line table and function symbols part of > DWARF)? > > > > Given what you've described so far, I'm wondering if what you really > want is the NativeProcess classes with some symbol-file reading pulled in? > Is there anything that you couldn't do from there? > > > > Jim > > > > > >> On Jun 26, 2018, at 12:48 PM, Zachary Turner > wrote: > >> > >> no expression parser or knowledge of any specific programming language. > >> > >> Basically I just mean that the parsing of the native DWARF format > itself is in scope, but anything beyond that is out of scope. For > symbolication we have things like llvm-symbolizer that already just work > and are built on top of LLVM's dwarf parsing code. Similarly, LLDB's type > system could be built on top of it as well. Given that I think everyone > mostly agrees that unifying on one DWARF parser is a good idea in > principle, this would mean no functional change from LLDB's point of view, > it would just continue to do exactly what it does regarding parsing C++ > expressions and converting these into types that clang understands. > >> > >> It will probably be useful someday to have an expression parser and > language specific type system, but when that comes I don't think we'd want > anything radically different than what LLDB already has. > >> > >> On Tue, Jun 26, 2018 at 12:26 PM Jim Ingham wrote: > >> Just to be clear, by "no clang integration" do you mean "no expression > parser" or do you mean something more radical? For instance, adding a > TypeSystem and its DWARF parser for C family languages that uses a > different underlying representation than Clang AST's to store the results > would be a lot of work that wouldn't be terribly interesting to lldb. I > don't think that's what you meant, but wanted to be sure. > >> > >> Jim > >> > >>> On Jun 26, 2018, at 11:58 AM, Zachary Turner via lldb-dev < > lldb-dev@lists.llvm.org> wrote: > >>> > >>> Hi all, > >>> > >>> We have been thinking internally about a lightweight llvm-based > ptracer. To address one question up front: the primary way in which this > differs from LLDB is that it targets a more narrow use case -- there is no > scripting support, no clang integration, no dynamic extensibility, no > support for running jitted code in the target, and no user interface. We > have several use cases internally that call for varying levels of > functionality from such a utility, and being able to use as little as > possible of the library as is necessary for the given task is important for > the scale in which we wish to use it. > >>> > >>> We are still in early discussions and planning, but I think this would > be a good addition to the LLVM upstream. Since we’re approaching this as a > set of small isolated components, my thinking is to work on this completely > upstream, directly under the llvm project (as opposed to making a separate > subproject), but I’m open to discussion if anyone feels differently. > >>> > >>> LLDB has solved a lot of the difficult problems needed for such a > tool. So in the spirit of code reuse, we think it’s worth trying > componentize LLDB by sinking pieces into LLVM and rebasing LLDB as well as > these smaller tools on top of these components, so that smaller tools can > reduce code duplication and contribute to the overall health of the code > base. At the same time we think that in doing so we can break things up > into more granular pieces, ultimately exposing a larger testing surface and > enabling us to create exhaustive tests, giving LLDB more fine grained > testing of important subsystems. > >>> > >>> A good example of this would b
Re: [lldb-dev] [llvm-dev] RFC: libtrace
On Tue, Jun 26, 2018 at 1:28 PM Adrian Prantl wrote: > > > > On Jun 26, 2018, at 11:58 AM, Zachary Turner via llvm-dev < > llvm-...@lists.llvm.org> wrote: > > > > Hi all, > > > > We have been thinking internally about a lightweight llvm-based > ptracer. To address one question up front: the primary way in which this > differs from LLDB is that it targets a more narrow use case -- there is no > scripting support, no clang integration, no dynamic extensibility, no > support for running jitted code in the target, and no user interface. We > have several use cases internally that call for varying levels of > functionality from such a utility, and being able to use as little as > possible of the library as is necessary for the given task is important for > the scale in which we wish to use it. > > > > We are still in early discussions and planning, but I think this would > be a good addition to the LLVM upstream. Since we’re approaching this as a > set of small isolated components, my thinking is to work on this completely > upstream, directly under the llvm project (as opposed to making a separate > subproject), but I’m open to discussion if anyone feels differently. > > > > LLDB has solved a lot of the difficult problems needed for such a tool. > So in the spirit of code reuse, we think it’s worth trying componentize > LLDB by sinking pieces into LLVM and rebasing LLDB as well as these smaller > tools on top of these components, so that smaller tools can reduce code > duplication and contribute to the overall health of the code base. > > Do you have a rough idea of what components specifically the new tool > would need to function? > * process & thread control * platform agnostic ptrace wrapper (not all platforms even have ptrace, and those that do the usage and capabilities vary quite a bit) * install various kinds of traps * monitor cpu performance counters * symbol file parsing * symbol resolution (name <-> addr and line <-> addr) * unwinding and backtrace generation > > > At the same time we think that in doing so we can break things up into > more granular pieces, ultimately exposing a larger testing surface and > enabling us to create exhaustive tests, giving LLDB more fine grained > testing of important subsystems. > > Are you thinking of the new utility as something that would naturally live > in llvm/tools or as something that would live in the LLDB repository? > I would rather put it under LLDB and then link LLDB against certain pieces in cases where that makes sense. > > > > > A good example of this would be LLDB’s DWARF parsing code, which is more > featureful than LLVM’s but has kind of evolved in parallel. Sinking this > into LLVM would be one early target of such an effort, although over time > there would likely be more. > > As you are undoubtedly aware we've been carefully rearchitecting LLVM's > DWARF parser over the last few years to eventually become featureful enough > so that LLDB could use it, so any help on that front would be most welcome. > As long as we are careful to not regress in performance/lazyness, features > and fault-tolerance, deduplicating the implementations can only be good for > LLVM and LLDB. > > Yea, this is the general idea. Has anyone actively been working on this specific effort recently? To my knowledge someone started and then never finished, but the efforts also never made it upstream, so my understanding is that it's a goal, but one that nobody has made significant headway on. ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [llvm-dev] RFC: libtrace
> On Jun 26, 2018, at 1:38 PM, Zachary Turner wrote: > >> On Tue, Jun 26, 2018 at 1:28 PM Adrian Prantl wrote: >> >>> > On Jun 26, 2018, at 11:58 AM, Zachary Turner via llvm-dev >>> > wrote: >>> > A good example of this would be LLDB’s DWARF parsing code, which is more >>> > featureful than LLVM’s but has kind of evolved in parallel. Sinking this >>> > into LLVM would be one early target of such an effort, although over time >>> > there would likely be more. >>> >>> As you are undoubtedly aware we've been carefully rearchitecting LLVM's >>> DWARF parser over the last few years to eventually become featureful enough >>> so that LLDB could use it, so any help on that front would be most welcome. >>> As long as we are careful to not regress in performance/lazyness, features >>> and fault-tolerance, deduplicating the implementations can only be good for >>> LLVM and LLDB. >>> >> Yea, this is the general idea. Has anyone actively been working on this >> specific effort recently? To my knowledge someone started and then never >> finished, but the efforts also never made it upstream, so my understanding >> is that it's a goal, but one that nobody has made significant headway on. > That's not true. Greg Clayton started the effort in 2016 and landed many of the ground-breaking changes. The design ideas fleshed out during that initial effort (thanks to David Blaikie who spent a lot of time reviewing the new interfaces!) such as improved error handling where then picked up the entire team of contributors who worked on DWARF 5 support in LLVM and we've continued down that path ever since. The greatly improved llvm-dwarfdump was also born out of this effort, for example. We also payed attention that every refactoring of LLDB DWARF parser code would bring it closer to the new LLVM parser interface to narrow the gaps between the implementations. -- adrian ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [llvm-dev] RFC: libtrace
Ahh, thanks. I thought those changes never landed, but it's good to hear that they did. On Tue, Jun 26, 2018 at 1:49 PM Adrian Prantl wrote: > > > On Jun 26, 2018, at 1:38 PM, Zachary Turner wrote: > > > >> On Tue, Jun 26, 2018 at 1:28 PM Adrian Prantl > wrote: > >> > >>> > On Jun 26, 2018, at 11:58 AM, Zachary Turner via llvm-dev < > llvm-...@lists.llvm.org> wrote: > >>> > A good example of this would be LLDB’s DWARF parsing code, which is > more featureful than LLVM’s but has kind of evolved in parallel. Sinking > this into LLVM would be one early target of such an effort, although over > time there would likely be more. > >>> > >>> As you are undoubtedly aware we've been carefully rearchitecting > LLVM's DWARF parser over the last few years to eventually become featureful > enough so that LLDB could use it, so any help on that front would be most > welcome. As long as we are careful to not regress in performance/lazyness, > features and fault-tolerance, deduplicating the implementations can only be > good for LLVM and LLDB. > >>> > >> Yea, this is the general idea. Has anyone actively been working on > this specific effort recently? To my knowledge someone started and then > never finished, but the efforts also never made it upstream, so my > understanding is that it's a goal, but one that nobody has made significant > headway on. > > > That's not true. Greg Clayton started the effort in 2016 and landed many > of the ground-breaking changes. The design ideas fleshed out during that > initial effort (thanks to David Blaikie who spent a lot of time reviewing > the new interfaces!) such as improved error handling where then picked up > the entire team of contributors who worked on DWARF 5 support in LLVM and > we've continued down that path ever since. The greatly improved > llvm-dwarfdump was also born out of this effort, for example. We also payed > attention that every refactoring of LLDB DWARF parser code would bring it > closer to the new LLVM parser interface to narrow the gaps between the > implementations. > > -- adrian > > ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [llvm-dev] RFC: libtrace
One important question, does this tool need to work remotely? I'm guessing the answer to this is no, since if you are working remotely you won't have a performant enough solution to really be an effective tracer. And if the guts of the debugger are remote, you care a lot less about the complexity of the remote part. If you can always debug with the Host platform - in lldb terms - then it really does seem like you want to start with the NativeProcess classes. That won't get you macOS hosting, but OTOH this would be good reason to get macOS onto the NativeProcess classes/lldb-server and off of debugserver... > On Jun 26, 2018, at 1:38 PM, Zachary Turner via lldb-dev > wrote: > > > > On Tue, Jun 26, 2018 at 1:28 PM Adrian Prantl wrote: > > > > On Jun 26, 2018, at 11:58 AM, Zachary Turner via llvm-dev > > wrote: > > > > Hi all, > > > > We have been thinking internally about a lightweight llvm-based ptracer. > > To address one question up front: the primary way in which this differs > > from LLDB is that it targets a more narrow use case -- there is no > > scripting support, no clang integration, no dynamic extensibility, no > > support for running jitted code in the target, and no user interface. We > > have several use cases internally that call for varying levels of > > functionality from such a utility, and being able to use as little as > > possible of the library as is necessary for the given task is important for > > the scale in which we wish to use it. > > > > We are still in early discussions and planning, but I think this would be a > > good addition to the LLVM upstream. Since we’re approaching this as a set > > of small isolated components, my thinking is to work on this completely > > upstream, directly under the llvm project (as opposed to making a separate > > subproject), but I’m open to discussion if anyone feels differently. > > > > LLDB has solved a lot of the difficult problems needed for such a tool. So > > in the spirit of code reuse, we think it’s worth trying componentize LLDB > > by sinking pieces into LLVM and rebasing LLDB as well as these smaller > > tools on top of these components, so that smaller tools can reduce code > > duplication and contribute to the overall health of the code base. > > Do you have a rough idea of what components specifically the new tool would > need to function? > > * process & thread control > * platform agnostic ptrace wrapper (not all platforms even have ptrace, and > those that do the usage and capabilities vary quite a bit) > * install various kinds of traps > * monitor cpu performance counters This part is all the job of the NativeProcess classes. That's not terribly surprising, since their whole reason for being was as a low-level abstraction for process control without any of the higher-level work that lldb does. > * symbol file parsing > * symbol resolution (name <-> addr and line <-> addr) This will involve getting object-file readers and a symbol file reader into your trace tool. These should be pretty easy to extract from lldb, though you probably don't need the plugin architecture. > * unwinding and backtrace generation Jason says this will be somewhat tricky to pull out of lldb. OTOH much of the complexity of unwind is reconstructing all the non-volatile registers, and if you don't care about values, you don't really need that. So some kind of lightweight pc/sp only backtrace would be more appropriate, and probably faster for your needs. Jim > > > > > At the same time we think that in doing so we can break things up into > > more granular pieces, ultimately exposing a larger testing surface and > > enabling us to create exhaustive tests, giving LLDB more fine grained > > testing of important subsystems. > > Are you thinking of the new utility as something that would naturally live in > llvm/tools or as something that would live in the LLDB repository? > I would rather put it under LLDB and then link LLDB against certain pieces in > cases where that makes sense. > > > > > > A good example of this would be LLDB’s DWARF parsing code, which is more > > featureful than LLVM’s but has kind of evolved in parallel. Sinking this > > into LLVM would be one early target of such an effort, although over time > > there would likely be more. > > As you are undoubtedly aware we've been carefully rearchitecting LLVM's DWARF > parser over the last few years to eventually become featureful enough so that > LLDB could use it, so any help on that front would be most welcome. As long > as we are careful to not regress in performance/lazyness, features and > fault-tolerance, deduplicating the implementations can only be good for LLVM > and LLDB. > > Yea, this is the general idea. Has anyone actively been working on this > specific effort recently? To my knowledge someone started and then never > finished, but the efforts also never made it upstream, so my unders
Re: [lldb-dev] [llvm-dev] RFC: libtrace
> On Jun 26, 2018, at 2:00 PM, Jim Ingham via lldb-dev > wrote: > > >> * unwinding and backtrace generation > > Jason says this will be somewhat tricky to pull out of lldb. OTOH much of > the complexity of unwind is reconstructing all the non-volatile registers, > and if you don't care about values, you don't really need that. So some kind > of lightweight pc/sp only backtrace would be more appropriate, and probably > faster for your needs. If it were me & performance were the utmost concern, and I had a restricted platform set that I needed to support where I can assume the presence of eh_frame and that it is trustworthy in prologue/epilogues, then I'd probably just write a simple Unwind/RegisterContext plugin pair that exclusively live off of that. If it's just stack walking, and we can assume no omit-frame-pointer code and we can assume the 0th function is always stopped in a non-prologue/epilogue location, then even simpler would be the old RegisterContextMacOSXFrameBackchain plugin would get you there. That's what we used before we had the modern unwind/registercontext plugin that we use today. It doesn't track spilled registers at all, it just looks for saved pc/framepointer values on the stack. A general problem with stopping the inferior process and examining things is that it is slow. Even if you use a NativeHost approach and get debugserver/lldb-server out of the equation, if you stop in a hot location it's very difficult to make this performant. We've prototyped things like this in the past and it was always far too slow. I don't know what your use case looks like, but I do worry about having one process controlling an inferior process in general for fast-turnaround data collection/experiments, it doesn't seem like the best way to go about it. > > Jim > >> >> >> >>> At the same time we think that in doing so we can break things up into more >>> granular pieces, ultimately exposing a larger testing surface and enabling >>> us to create exhaustive tests, giving LLDB more fine grained testing of >>> important subsystems. >> >> Are you thinking of the new utility as something that would naturally live >> in llvm/tools or as something that would live in the LLDB repository? >> I would rather put it under LLDB and then link LLDB against certain pieces >> in cases where that makes sense. >> >> >>> >>> A good example of this would be LLDB’s DWARF parsing code, which is more >>> featureful than LLVM’s but has kind of evolved in parallel. Sinking this >>> into LLVM would be one early target of such an effort, although over time >>> there would likely be more. >> >> As you are undoubtedly aware we've been carefully rearchitecting LLVM's >> DWARF parser over the last few years to eventually become featureful enough >> so that LLDB could use it, so any help on that front would be most welcome. >> As long as we are careful to not regress in performance/lazyness, features >> and fault-tolerance, deduplicating the implementations can only be good for >> LLVM and LLDB. >> >> Yea, this is the general idea. Has anyone actively been working on this >> specific effort recently? To my knowledge someone started and then never >> finished, but the efforts also never made it upstream, so my understanding >> is that it's a goal, but one that nobody has made significant headway on. >> ___ >> lldb-dev mailing list >> lldb-dev@lists.llvm.org >> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev > > ___ > lldb-dev mailing list > lldb-dev@lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev ___ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] RFC: libtrace
Yes that’s what I’ve been thinking about as well. One thing I’ve been giving a lot of thought to is whether to serialize the handling of trace events. I want to balance the “this is a library and you should be able to get it to work for you no matter what your use case is” aspect with the “you really just don’t want to go there, we know what’s best for you” aspect. Then there’s the fact that not all platforms behave the same, but we’d like a consistent set of expectations that makes it easy to use for everyone. So I’m leaning towards having the library serialize all tace events, because it’s a nice common denominator that every platform can implement. To be clear though, I don’t mean that if 2 processes are being traced simultaneously and A stops followed by B stopping, then the tool will necessarily block before handling B’s stop. I just mean that A and B’s stop handlers will be invoked on a single thread (not the threads which are tracing A or B). So A stops, posts its stop event on the blessed thread and waits. Then B stops and does the same thing. A’s handler runs, for whatever reason decides it will continue later, saves off the event somewhere, then processes B’s. Later something happens, it decides to continue A, signals A’s thread which wakes up. I think this kind of design eliminates a large class of race conditions without sacrificing any performance. LLDB doesn’t currently work like this, but it would be nice not to end up with another split similar to the dwarf split, so I’m curious if you can think of any fundamental assumptions of LLDB’s architecture that this would violate. This way we’d at least know that it’s possible to use the api in lldb (assuming it does everything lldb needs obviously) Thoughts? On Tue, Jun 26, 2018 at 1:09 PM Jim Ingham wrote: > You'd probably need to pull the Unwinder in if you want backtraces, but > that part shouldn't be that hard to disentangle. I don't think you'd need > much else? > > Basing your work on NativeProcess rather than lldb proper would also cut > the number of observer processes in half and avoid the context switches > between the server and the debugger. That seems more appropriate for a > lightweight tool. > > Jim > > > > On Jun 26, 2018, at 12:59 PM, Jim Ingham via lldb-dev < > lldb-dev@lists.llvm.org> wrote: > > > > So you aren't planning to print values at all, just stop points (i.e. > you are only interested in the line table and function symbols part of > DWARF)? > > > > Given what you've described so far, I'm wondering if what you really > want is the NativeProcess classes with some symbol-file reading pulled in? > Is there anything that you couldn't do from there? > > > > Jim > > > > > >> On Jun 26, 2018, at 12:48 PM, Zachary Turner > wrote: > >> > >> no expression parser or knowledge of any specific programming language. > >> > >> Basically I just mean that the parsing of the native DWARF format > itself is in scope, but anything beyond that is out of scope. For > symbolication we have things like llvm-symbolizer that already just work > and are built on top of LLVM's dwarf parsing code. Similarly, LLDB's type > system could be built on top of it as well. Given that I think everyone > mostly agrees that unifying on one DWARF parser is a good idea in > principle, this would mean no functional change from LLDB's point of view, > it would just continue to do exactly what it does regarding parsing C++ > expressions and converting these into types that clang understands. > >> > >> It will probably be useful someday to have an expression parser and > language specific type system, but when that comes I don't think we'd want > anything radically different than what LLDB already has. > >> > >> On Tue, Jun 26, 2018 at 12:26 PM Jim Ingham wrote: > >> Just to be clear, by "no clang integration" do you mean "no expression > parser" or do you mean something more radical? For instance, adding a > TypeSystem and its DWARF parser for C family languages that uses a > different underlying representation than Clang AST's to store the results > would be a lot of work that wouldn't be terribly interesting to lldb. I > don't think that's what you meant, but wanted to be sure. > >> > >> Jim > >> > >>> On Jun 26, 2018, at 11:58 AM, Zachary Turner via lldb-dev < > lldb-dev@lists.llvm.org> wrote: > >>> > >>> Hi all, > >>> > >>> We have been thinking internally about a lightweight llvm-based > ptracer. To address one question up front: the primary way in which this > differs from LLDB is that it targets a more narrow use case -- there is no > scripting support, no clang integration, no dynamic extensibility, no > support for running jitted code in the target, and no user interface. We > have several use cases internally that call for varying levels of > functionality from such a utility, and being able to use as little as > possible of the library as is necessary for the given task is important for > the scale in which we wish to use i
