[lldb-dev] [RFC] Upstreaming Reproducer Capture/Replay for the API Test Suite
Hi everyone, Reproducers in LLDB are currently tested through (1) unit tests, (2) dedicated end-to-end shell tests and (3) the `lldb-check-repro` suite which runs all the shell tests against a replayed reproducer. While this already provides great coverage, we're still missing out on about 800 API tests. These tests are particularly interesting to the reproducers, because as opposed to the shell tests, which only exercises a subset of SB API calls used to implement the driver, they cover the majority of the API surface. To further qualify reproducer and to improve test coverage, I want to capture and replay the API test suite as well. Conceptually, this can be split up in two stages: 1. Capture a reproducer and replay it with the driver. This exercises the reproducer instrumentation (serialization and deserialization) for all the APIs used in our test suite. While a bunch of issues with the reproducer instrumentation can be detected at compile time, a large subset only triggers through assertions at runtime. However, this approach by itself only verifies that we can (de)serialize API calls and their arguments. It has no knowledge of the expected results and therefore cannot verify the results of the API calls. 2. Capture a reproducer and replay it with dotest.py. Rather than having the command line driver execute every API call one after another, we can have dotest.py call the Python API as it normally would, intercept the call, replay it from the reproducer, and return the replayed result. The interception can be hidden behind the existing LLDB_RECORD_* macros, which contains sufficient type info to drive replay. It then simply re-invokes itself with the arguments deserialized from the reproducer and returns that result. Just as with the shell tests, this approach allows us to reuse the existing API tests, completely transparently, to check the reproducer output. I have worked on this over the past month and have shown that it is possible to achieve both stages. I have a downstream fork that contains the necessary changes. All the runtime issues found in stage 1 have been fixed upstream. With the exception of about 30 tests that fail because the GDB packets diverge during replay, all the tests can be replayed with the driver. About 120 tests, which include the 30 mentioned earlier, fail to replay for stage 2. This isn't entirely unexpected, just like the shell tests, there are tests that simply are not expected to work. The reproducers don't currently capture the output of the inferior and synchronization through external files won't work either, as those paths will get remapped by the VFS. This requires manually triage. I would like to start upstreaming this work so we can start running this in CI. The majority of the changes are limited to the reproducer instrumentation, but some changes are needed in the test suite as well, and there would be a new decorator to skip the unsupported tests. I'm splitting up the changes in self-contained patches, but wanted to send out this RFC with the bigger picture first. Please let me know what you think! Cheers, Jonas ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
Re: [lldb-dev] [RFC] Upstreaming Reproducer Capture/Replay for the API Test Suite
> On Apr 6, 2020, at 2:24 PM, Jonas Devlieghere via lldb-dev > wrote: > > Hi everyone, > > Reproducers in LLDB are currently tested through (1) unit tests, (2) > dedicated end-to-end shell tests and (3) the `lldb-check-repro` suite which > runs all the shell tests against a replayed reproducer. While this already > provides great coverage, we're still missing out on about 800 API tests. > These tests are particularly interesting to the reproducers, because as > opposed to the shell tests, which only exercises a subset of SB API calls > used to implement the driver, they cover the majority of the API surface. > > To further qualify reproducer and to improve test coverage, I want to capture > and replay the API test suite as well. Conceptually, this can be split up in > two stages: > > 1. Capture a reproducer and replay it with the driver. This exercises the > reproducer instrumentation (serialization and deserialization) for all the > APIs used in our test suite. While a bunch of issues with the reproducer > instrumentation can be detected at compile time, a large subset only triggers > through assertions at runtime. However, this approach by itself only verifies > that we can (de)serialize API calls and their arguments. It has no knowledge > of the expected results and therefore cannot verify the results of the API > calls. > > 2. Capture a reproducer and replay it with dotest.py. Rather than having the > command line driver execute every API call one after another, we can have > dotest.py call the Python API as it normally would, intercept the call, > replay it from the reproducer, and return the replayed result. The > interception can be hidden behind the existing LLDB_RECORD_* macros, which > contains sufficient type info to drive replay. It then simply re-invokes > itself with the arguments deserialized from the reproducer and returns that > result. Just as with the shell tests, this approach allows us to reuse the > existing API tests, completely transparently, to check the reproducer output. > > I have worked on this over the past month and have shown that it is possible > to achieve both stages. I have a downstream fork that contains the necessary > changes. > > All the runtime issues found in stage 1 have been fixed upstream. With the > exception of about 30 tests that fail because the GDB packets diverge during > replay, all the tests can be replayed with the driver. > > About 120 tests, which include the 30 mentioned earlier, fail to replay for > stage 2. This isn't entirely unexpected, just like the shell tests, there are > tests that simply are not expected to work. The reproducers don't currently > capture the output of the inferior and synchronization through external files > won't work either, as those paths will get remapped by the VFS. This requires > manually triage. > > I would like to start upstreaming this work so we can start running this in > CI. The majority of the changes are limited to the reproducer > instrumentation, but some changes are needed in the test suite as well, and > there would be a new decorator to skip the unsupported tests. I'm splitting > up the changes in self-contained patches, but wanted to send out this RFC > with the bigger picture first. I personally believe this is a required step to make sure: a) Reproducers can jump from being a prototype idea to something that can actually run in production b) Whenever we add a new test [or presumably a new API] we get coverage for-free. c) We have a verification mechanism to make sure we don’t regress across the large surface API and not only what the unittests & shell tests cover. I personally would be really glad to see this being upstreamed. I also would like to thank you for doing the work in a downstream branch until you proved this was achievable. — D ___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
[lldb-dev] [Bug 35484] Lack of signing for lldb-debugserver should be detected at configure time.
https://bugs.llvm.org/show_bug.cgi?id=35484 Davide Italiano changed: What|Removed |Added Resolution|--- |FIXED Status|REOPENED|RESOLVED --- Comment #7 from Davide Italiano --- We fixed this a while ago. -- You are receiving this mail because: You are the assignee for the bug.___ lldb-dev mailing list lldb-dev@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
[lldb-dev] Saving and restoring STDIN in the ScriptInterpreter
Hi Pavel, Jonas,
I was trying to reduce a bug through c-reduce, so I decided to write a SBAPI
script to make it easier.
I did find out, that after the first iteration, the reduction gets stuck
forever.
I sampled the process and I saw the following (trimmed for readability).
Call graph:
[…]
8455
lldb_private::CommandInterpreter::GetScriptInterpreter(bool) (in _lldb.so) +
84 [0x111aff826]
8455
lldb_private::PluginManager::GetScriptInterpreterForLanguage(lldb::ScriptLanguage,
lldb_private::CommandInterpreter&) (in _lldb.so) + 99 [0x111a1efcf]
8455
lldb_private::ScriptInterpreterPython::CreateInstance(lldb_private::CommandInterpreter&)
(in _lldb.so) + 26 [0x111d128f4]
8455
std::__1::shared_ptr
std::__1::shared_ptr::make_shared(lldb_private::CommandInterpreter&&&)
(in _lldb.so) + 72 [0x111d1b976]
8455
lldb_private::ScriptInterpreterPython::ScriptInterpreterPython(lldb_private::CommandInterpreter&)
(in _lldb.so) + 353 [0x111d11ff3]
8455
lldb_private::ScriptInterpreterPython::InitializePrivate() (in _lldb.so) + 494
[0x111d12594]
8455
(anonymous namespace)::InitializePythonRAII::~InitializePythonRAII() (in
_lldb.so) + 146 [0x111d1b446]
8455
lldb_private::TerminalState::Restore() const (in _lldb.so) + 74 [0x111ac8268]
8455 tcsetattr (in libsystem_c.dylib) + 110 [0x7fff7b95b585]
8455 ioctl (in libsystem_kernel.dylib) + 151 [0x7fff7ba19b44]
8455 __ioctl (in libsystem_kernel.dylib) + 10 [0x7fff7ba19b5a]
It looks like lldb gets stuck forever in `tcsetattr()`, and there are no other
threads waiting so it’s not entirely obvious to me why it’s waiting there.
I was never able to reproduce this with an interactive session, I suspect this
is somehow related to the fact that c-reduce spawns a thread in the background,
hence it doesn’t have a TTY associated.
I looked at the code that does this, and I wasn’t really able to find a reason
why we need to do this work. Jim thinks it might have been needed historically.
`git blame` doesn’t really help that much either. If I remove the code,
everything still passes and it’s functional, but before moving forward with
this I would like to collect your opinions.
$ git diff
diff --git
a/lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp
b/lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp
index ee94a183e0d..c53b3bd0fb6 100644
--- a/lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp
+++ b/lldb/source/Plugins/ScriptInterpreter/Python/ScriptInterpreterPython.cpp
@@ -224,10 +224,6 @@ struct InitializePythonRAII {
public:
InitializePythonRAII()
: m_gil_state(PyGILState_UNLOCKED), m_was_already_initialized(false) {
-// Python will muck with STDIN terminal state, so save off any current TTY
-// settings so we can restore them.
-m_stdin_tty_state.Save(STDIN_FILENO, false);
-
InitializePythonHome();
#ifdef LLDB_USE_LIBEDIT_READLINE_COMPAT_MODULE
@@ -271,8 +267,6 @@ public:
// We initialized the threads in this function, just unlock the GIL.
PyEval_SaveThread();
}
-
-m_stdin_tty_state.Restore();
}
private:___
lldb-dev mailing list
lldb-dev@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
