Re: [lldb-dev] Compiling LLDB on Centos 5 (dont judge me)

[Todd Fiala via lldb-dev]

I fixed several RHEL-derived distribution-related issues in top of tree
LLDB a few weeks back.  It was for RHEL 7-derived builds, so quite a bit
newer.  If you are building your lldb off of top-of-tree, hopefully this
will mostly become an exercise of:

* Do you have a new-enough python.  (I believe at this point you really
want a 2.7, but with CentOS 5 you might be far back enough to be 2.4 or
2.6.  We probably limp along with 2.6 since that was our baseline for a
while).  Worst case here you can download a newer python and build from
source, then specify it to cmake via the right python cmake variables.

* Do you have a new-enough c++ compiler.  Since you're able to build a
clang, that should take care of that issue.

* Other libs you'll need.  I needed to install these:

yum install libedit-devel
yum install python-devel
yum install ncurses-devel
yum install swig
yum install libxml2-devel

* cmake - you'll need cmake 2.8.12.2 or later.  This one I'm pretty
sure you're going to need to build this from scratch.

* ninja - you're going to want to build the ninja build tool to
optimize your build speed and use -GNinja with your cmake command
line.

It'll be interesting to hear what else you find.


On Tue, Oct 27, 2015 at 8:37 PM, Mark Chandler via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> Using gcc 4.8 from devtoolset2. Clang builds fine so might try building
> with that tomorrow.
>
> Sent from my iPhone
>
> On Oct 27, 2015, at 7:46 PM, Oleksiy Vyalov  wrote:
>
> Hi Mark,
>
> what compiler do you use? Could you try to build LLDB with clang 3.5
> specifying it via DCMAKE_C_COMPILER and DCMAKE_CXX_COMPILER flags?
> It might be a problem with sysroot setup since some headers are not found
> - you can try to pass a custom sysroot with
> -DCMAKE_CXX_FLAGS="--sysroot=..."
>
> On Tue, Oct 27, 2015 at 2:05 PM, Mark Chandler via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> Hi All,
>>
>>
>>
>> Trying to get LLDB to compile on centos 5 to help reduce the size of
>> cores of programs crashing on some servers however im running into some
>> compile issues with it. Looks like some features are used from newer kernel
>> versions and was wondering what the procedure is to get this fixed and
>> updated into lldb.
>>
>>
>>
>> These are the errors so far:
>>
>>
>> https://gist.githubusercontent.com/lodle/47493c8ea2a51eff5322/raw/ce158e1d10d1df363bdd3a77b11ff3d8661e8144/lldb_make.txt
>>
>> Some of them are using O_CLOEXEC flag and headers that don’t exist and
>> was easy enough to add works around. The ones at the end im not sure about.
>>
>>
>>
>>
>>
>> Build Script:
>>
>> https://gist.github.com/lodle/e24a80907bbf7a7b72f6
>>
>>
>>
>>
>>
>> Thanks for the help.
>>
>>
>>
>>
>>
>> *Mark Chandler*
>>
>> Battle.Net  Infrastructure | Blizzard Entertainment
>>
>> *(P) *949-955-1380 x15353
>>
>>
>>
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>
>
>
> --
> Oleksiy Vyalov | Software Engineer | ovya...@google.com
>
>
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Refactoring dotest as a Python package

[Todd Fiala via lldb-dev]

On Tue, Oct 27, 2015 at 8:12 PM, Zachary Turner  wrote:

> Todd, I have one question.  If I understand correctly, we currently run
> dotest.py as a script, which imports dosep and calls some method in dosep,
> and dosep then again exec's dotest.
>
> Can you think of a pythonic way to make this work under the new layout?
> To be clear, I have it working, just not in a pythonic way.  The problem is
> that if we put code in lldbsuite's __init__.py, then this code won't be
> run when we exec dotest, because we'll be running it as a script instead of
> importing it as a package.  __init__.py is a very handy way to run some
> per-package initialization, so I'd like to be able to take advantage of it.
>
>
Hey Zachary,

Sorry I missed this.  I *think* you have this all worked out, right?

I think the way to fix this is:

* top-level dotest.py just imports whatever is needed and calls it
(something from the package, that is basically the contents of the old
dotest.py).

* The new dotest.py (inside the package) does what it used to do, calling
dosep.py when it is a concurrent run.

* Dosep.py would call the top level dotest.py when execing, with the right
flags as before to indicate that the new run is a dotest.py inferior.

I think that would do it and still get your initialization.

-Todd


> The way I currently have it working is just write `import lldbsuite` at
> the top of dotest.py, but that seems a little arbitrary that a file can't
> be exec'ed unless it imports the package that it's supposed to be contained
> in.
>
> So to re-iterate: the problem is that under the new layout the user will
> run lldb/test/dotest.py, but dosep will try to exec
> lldb/packages/Python/lldbsuite/test/dotest.py, which is intended to be
> imported as a package.
>
> What if we have dosep instead do this:
>
> # Execute the same script that the user originall ran from the command
> line,
> # which under this new system will be lldb/test/dotest.py, even though
> dosep
> # and the *real* dotest now reside in lldb/packages/Python/lldbsuite/test
> import __main__ as main
> exec main.__file__
>
> Can you think of any problem with this?  Another option is to use
> sys.argv[0].  Not sure if there's any practical difference between the two
> methods.
>
> On Tue, Oct 27, 2015 at 7:29 PM Zachary Turner  wrote:
>
>> Ok, I'll do it tomorrow.  Since it's a big code move I was a little
>> worried it would break someone's bot or the Xcode build, but I guess we can
>> deal with issues that pop up afterwards.
>>
>> On Tue, Oct 27, 2015 at 5:14 PM Jim Ingham  wrote:
>>
>>> It seems like everybody is okay with the idea of this, so I don't see
>>> the need for a review of the details of this stage.  If you think there's
>>> anything tricky call it out in words, otherwise I say just commit it.
>>>
>>> Jim
>>>
>>>
>>> > On Oct 27, 2015, at 4:30 PM, Zachary Turner via lldb-dev <
>>> lldb-dev@lists.llvm.org> wrote:
>>> >
>>> > I have the first part of the patch in, and the second part of the
>>> patch (which is essentially just a whole-folder rename with a couple of
>>> fixups) ready to go.  What's the best way to have this reviewed?  Uploading
>>> a 7MB patch to Phabricator probably isn't going to work out very well.
>>> >
>>> > On Tue, Oct 27, 2015 at 1:40 PM Zachary Turner 
>>> wrote:
>>> > I think I have a way to split this into two smaller CLs.  I'm testing
>>> this at the moment, if so it will allow the first CL to be most of the
>>> preparation for the rename without the rename, and then the second CL
>>> should literally just be a straight move with only 1-2 line code change.
>>> So I'll try to put this first CL up for review shortly.
>>> >
>>> > On Tue, Oct 27, 2015 at 12:49 PM Zachary Turner 
>>> wrote:
>>> > I've got a patch locally to make all of our Python stuff part of an
>>> lldb package called `lldbsuite`.  Currently we've got a bunch of standalone
>>> scripts that live in various directories such as `lldb/test`, or
>>> `lldb/scripts`, and possibly some  other locations, and this organization
>>> makes it hard to share code because it is incompatible with Python's
>>> built-in code reuse mechanism, which is its package system.
>>> >
>>> > The problem is, this patch is *big*.  Functionally there weren't many
>>> major changes, but it renames the entire test directory.  To be clear, it
>>> still leaves `test/dotest.py` in place, so nobody has to change their
>>> workflow or do anything differently.  If you used to write "cd test &&
>>> dotest.py" you can still do that.  dotest.py is now just a 2 line wrapper
>>> around the package, so it looks like:
>>> >
>>> > import lldbsuite.test
>>> > lldbsuite.test.run_suite()
>>> >
>>> > the advantage of this method is that lldbsuite can contain
>>> subpackages.  It already contains one subpackage, which is lldbsuite.test,
>>> and I plan to move some of the Python code in `lldb/scripts` there as well,
>>> so that we have lldbsuite.scripts.  Then we can add a third submodule,
>>> lldbsuite.shared, and now

Re: [lldb-dev] Compiling LLDB on Centos 5 (dont judge me)

[Todd Fiala via lldb-dev]

Great, glad to hear, Mark!

On Thu, Oct 29, 2015 at 4:57 PM, Mark Chandler 
wrote:

>
> For My Centos 5 build:
>
> * Using python 2.7 (From devtoolset 2)
>
> * Using gcc 4.8 (From devtoolset 2)
>
> * disabled libedit and python bindings
>
> * Built cmake from source
>
> * Make is fine so didnt bother with ninja
>
>
> Applying the patch then gives me a working liblldb.so.
>
>
> Also want to say that its refreshing seeing such a good api. The interface
> is easy to understand and is standalone without pulling in a bunch of
> complex types from other libraries. Well done lldb team!
>
>
> Mark
>
>
>
> --
> *From:* Todd Fiala 
> *Sent:* Thursday, October 29, 2015 4:17 PM
> *To:* Mark Chandler
> *Cc:* Oleksiy Vyalov; lldb-dev@lists.llvm.org
> *Subject:* Re: [lldb-dev] Compiling LLDB on Centos 5 (dont judge me)
>
> I fixed several RHEL-derived distribution-related issues in top of tree
> LLDB a few weeks back.  It was for RHEL 7-derived builds, so quite a bit
> newer.  If you are building your lldb off of top-of-tree, hopefully this
> will mostly become an exercise of:
>
> * Do you have a new-enough python.  (I believe at this point you really
> want a 2.7, but with CentOS 5 you might be far back enough to be 2.4 or
> 2.6.  We probably limp along with 2.6 since that was our baseline for a
> while).  Worst case here you can download a newer python and build from
> source, then specify it to cmake via the right python cmake variables.
>
> * Do you have a new-enough c++ compiler.  Since you're able to build a
> clang, that should take care of that issue.
>
> * Other libs you'll need.  I needed to install these:
>
> yum install libedit-devel
> yum install python-devel
> yum install ncurses-devel
> yum install swig
> yum install libxml2-devel
>
> * cmake - you'll need cmake 2.8.12.2 or later.  This one I'm pretty sure 
> you're going to need to build this from scratch.
>
> * ninja - you're going to want to build the ninja build tool to optimize your 
> build speed and use -GNinja with your cmake command line.
>
> It'll be interesting to hear what else you find.
>
>
> On Tue, Oct 27, 2015 at 8:37 PM, Mark Chandler via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> Using gcc 4.8 from devtoolset2. Clang builds fine so might try building
>> with that tomorrow.
>>
>> Sent from my iPhone
>>
>> On Oct 27, 2015, at 7:46 PM, Oleksiy Vyalov  wrote:
>>
>> Hi Mark,
>>
>> what compiler do you use? Could you try to build LLDB with clang 3.5
>> specifying it via DCMAKE_C_COMPILER and DCMAKE_CXX_COMPILER flags?
>> It might be a problem with sysroot setup since some headers are not found
>> - you can try to pass a custom sysroot with
>> -DCMAKE_CXX_FLAGS="--sysroot=..."
>>
>> On Tue, Oct 27, 2015 at 2:05 PM, Mark Chandler via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> Hi All,
>>>
>>>
>>>
>>> Trying to get LLDB to compile on centos 5 to help reduce the size of
>>> cores of programs crashing on some servers however im running into some
>>> compile issues with it. Looks like some features are used from newer kernel
>>> versions and was wondering what the procedure is to get this fixed and
>>> updated into lldb.
>>>
>>>
>>>
>>> These are the errors so far:
>>>
>>>
>>> https://gist.githubusercontent.com/lodle/47493c8ea2a51eff5322/raw/ce158e1d10d1df363bdd3a77b11ff3d8661e8144/lldb_make.txt
>>>
>>> Some of them are using O_CLOEXEC flag and headers that don’t exist and
>>> was easy enough to add works around. The ones at the end im not sure about.
>>>
>>>
>>>
>>>
>>>
>>> Build Script:
>>>
>>> https://gist.github.com/lodle/e24a80907bbf7a7b72f6
>>>
>>>
>>>
>>>
>>>
>>> Thanks for the help.
>>>
>>>
>>>
>>>
>>>
>>> *Mark Chandler*
>>>
>>> Battle.Net  Infrastructure | Blizzard Entertainment
>>>
>>> *(P) *949-955-1380 x15353
>>>
>>>
>>>
>>> ___
>>> lldb-dev mailing list
>>> lldb-dev@lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>
>>>
>>
>>
>> --
>> Oleksiy Vyalov | Software Engineer | ovya...@google.com
>>
>>
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Apple LLDB OS X build bot

[Todd Fiala via lldb-dev]

Hi Tamas!

It looks like I forgot to mention, but at this time the buildbot is not
running the tests.  Step one was to get the build building and report
breakages.

The next frontier will be running the tests.  I'll send out another email
once the tests are run and included in the build.  No ETA yet but I'll post
once I have one.

-Todd

On Mon, Nov 2, 2015 at 11:30 AM, Tamas Berghammer 
wrote:

> Hi Todd,
>
> Thank you for setting up the new buildbot. I have a few questions about it:
> * Is it running the test suit or only do a build?
> * If the test suit is run then where can we see the result of the tests?
>
> Thanks,
> Tamas
>
> On Wed, Oct 28, 2015 at 2:03 PM Todd Fiala via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> Hi all,
>>
>> I've made a few changes to the Apple OS X buildbot today.  These are
>> mostly minor, but the key is to make sure we all know when it's broken.
>>
>> First off, it now builds the lldb-tool scheme using the Debug
>> configuration.  (Previously it was building a BuildAndIntegration
>> configuration, which nobody outside Apple is ever going to be able to build
>> right).
>>
>> Second, it no longer tries to build a signed debugserver and instead uses
>> the system debugserver.
>>
>> At this point, if you get an email on a broken build, please make sure to
>> do the typical courteous thing and (1) fix it if you know how, (2) reach
>> out and ask us if we know how if it is a platform-specific issue, or (3)
>> revert until we figure out a way to get it working for everyone.
>>
>> You can get to the builder here:
>> http://lab.llvm.org:8080/green/job/LLDB/
>>
>> It's part of the newer Jenkins-style builders that llvm.org has been
>> trying out.
>>
>> It is configured to send emails on a transition from green to red.
>>
>> Here's the current green build:
>> http://lab.llvm.org:8080/green/job/LLDB/13827/
>>
>> Thanks!
>> --
>> -Todd
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Is this a test, or a build? Help with link you supplied / Tony

[Todd Fiala via lldb-dev]

I'm not sure I fully understand the question, but the GreenDragon CI LLVM
Build Cluster has an LLDB builder for OS X which looks like you're
referencing with the links above.

It gets built by a few stages: one to grab sources, another to do the
building.  I just poked over there and the build looks like it has been
green for quite a while (the Build and Check stage, available from here
).

That builder is not yet running tests.  So if we're talking about the same
thing, it shouldn't be saying anything about test failures.  I'm not sure
what the blocking means.

On Mon, Nov 2, 2015 at 2:36 PM, Anthony Orlando via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> I went to the link you supplied (not the green), and not sure which of the
> below you would like me to select. Please let me know what I can do to help
> with this issue.
>
>
>
> Static
>
>- [image: Success]LLDB, using Stage 1 RA  (Acquire sources)
>(blocking)
>
> Other executed recently
>
>- [image: In progress]LLDB, using Stage 1 RA  (Build and Check)
>(blocking)
>
>
>
> Static
>
>- [image: In progress]LLDB, using Stage 1 RA  (Build and Check)
>(blocking)
>
> Other executed recently
>
>- [image: Success]LLDB, using Stage 1 RA  (Acquire sources)
>(blocking)
>-
>
>
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Attaching to a stopped (cored) process hangs lldb-server

[Todd Fiala via lldb-dev]

Hey Pavel,

I think Mark is also on RHEL 5-era, so this going *way* back in the kernel
space.  It is entirely possible he is seeing different behavior based on
that.  We only recently started working on RHEL 7 and (I've heard reports
of) 6.  So this could just be legitimate behavioral difference that we
won't see on much newer Ubuntu kernels and/or configuration differences
between RHEL and Debian-based kernels.

On Tue, Nov 3, 2015 at 5:47 PM, Mark Chandler via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> The ptrace options per thread id also fail so I removed that as well. Atm
> lldb-server is seg-faulting in ThreadAttach that im trying to work out why.
>
>
>
> Mark Chandler
> Battle.Net Engineering Systems | Blizzard Entertainment
> (P) 949-955-1380 x15353
>
> -Original Message-
> From: Pavel Labath [mailto:lab...@google.com]
> Sent: Tuesday, November 03, 2015 5:41 PM
> To: Greg Clayton 
> Cc: Mark Chandler ; lldb-dev@lists.llvm.org
> Subject: Re: [lldb-dev] Attaching to a stopped (cored) process hangs
> lldb-server
>
> I'm following this discussion, but I don't yet understand what is going on
> here completely. What I am sure is that the problem here is not the S+
> state, as that just means "interruptible sleep, foreground process", and a
> lot of processes have that state and we attach to them just fine. I would
> need to investigate what are the exact properties or this cored state. I'll
> try to take a look when I get some spare cycles, but that might not happen
> very soon.
>
> Mark, have you investigated what is the next thing to fail after you
> remove the waitpid call?
>
> pl
>
> On 3 November 2015 at 16:48, Greg Clayton via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
> > Can someone with linux experience chime in here? It shouldn't be too
> hard to figure out which flag 'S' is in. On MacOS we can get a process info
> structure from a pid and that will have bits set that indicate 'S'...
> >
> > If you want to checkin this tool into the LLDB source tree at
> trunk/tools/core_tool then we can get more people to work on it and improve
> it. It would be nice to have this available for all linux users. I would
> love to see an JSON output mode that is parseable by automated tools
> instead of people saving text formats that must be text scraped.
> >
> > If you can get this into a tool, others can help get this working. Any
> interest in this?
> >
> > Greg
> >
> >> On Nov 3, 2015, at 4:41 PM, Mark Chandler 
> wrote:
> >>
> >> The biggest tell is that the process state is already 'S' or stopped. I
> don’t know lldb at all to make a change to fix this though.
> >>
> >>
> >> Mark Chandler
> >> Battle.Net Engineering Systems | Blizzard Entertainment
> >> (P) 949-955-1380 x15353
> >>
> >> -Original Message-
> >> From: Greg Clayton [mailto:gclay...@apple.com]
> >> Sent: Tuesday, November 03, 2015 4:39 PM
> >> To: Mark Chandler 
> >> Cc: lldb-dev@lists.llvm.org
> >> Subject: Re: [lldb-dev] Attaching to a stopped (cored) process hangs
> >> lldb-server
> >>
> >> Makes sense about not writing the core file to disk.
> >>
> >> Is there a way you can detect this "core" mode where we don't have to
> waitpid? Seems like that www.sourceware.org message had ideas on how to
> detect this case?
> >>
> >> Greg
> >>
> >>> On Nov 3, 2015, at 4:36 PM, Mark Chandler 
> wrote:
> >>>
> >>> Not able to do that as the servers have no hard drives (use ram disk
> >>> and net boot) and the tool is trying to avoid a core storm that
> >>> takes down the network file share. I found out what is causing it to
> >>> hang, there is a call to waitpid in NativeLinuxProcess.cpp that
> >>> waits forever. As the process is already stopped, I disabled that
> >>> and it looks to be working
> >>>
> >>> Mark Chandler
> >>> Battle.Net Engineering Systems | Blizzard Entertainment
> >>> (P) 949-955-1380 x15353
> >>>
> >>> -Original Message-
> >>> From: Greg Clayton [mailto:gclay...@apple.com]
> >>> Sent: Tuesday, November 03, 2015 4:34 PM
> >>> To: Mark Chandler 
> >>> Cc: lldb-dev@lists.llvm.org
> >>> Subject: Re: [lldb-dev] Attaching to a stopped (cored) process hangs
> >>> lldb-server
> >>>
> >>> One different approach is to have your tool write all STDIN to a file
> (the core file comes into the tool as STDIN bytes) and then hand LLDB the
> core file and do any needed backtracing and data gathering from the core
> file instead of actually attaching to the process for real. All executable
> and shared library object files (ELF files) from the core file are still on
> disk so you can get symbols and use the debug info, so LLDB should be able
> to load all frames up and symbolicate up the crash location. It should be
> just as good as having the process around without any bad side affects.
> Core files are less useful if they must be archived and symbolicated later
> because the executable files might not be around anymore since things like
> test suites might produce binaries for testing and remove them after the
> test i

Re: [lldb-dev] Attaching to a stopped (cored) process hangs lldb-server

[Todd Fiala via lldb-dev]

Although doing any kind of waitpid() in the case of a core file doesn't
make sense.

On Wed, Nov 4, 2015 at 9:44 AM, Todd Fiala  wrote:

> Hey Pavel,
>
> I think Mark is also on RHEL 5-era, so this going *way* back in the kernel
> space.  It is entirely possible he is seeing different behavior based on
> that.  We only recently started working on RHEL 7 and (I've heard reports
> of) 6.  So this could just be legitimate behavioral difference that we
> won't see on much newer Ubuntu kernels and/or configuration differences
> between RHEL and Debian-based kernels.
>
> On Tue, Nov 3, 2015 at 5:47 PM, Mark Chandler via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> The ptrace options per thread id also fail so I removed that as well. Atm
>> lldb-server is seg-faulting in ThreadAttach that im trying to work out why.
>>
>>
>>
>> Mark Chandler
>> Battle.Net Engineering Systems | Blizzard Entertainment
>> (P) 949-955-1380 x15353
>>
>> -Original Message-
>> From: Pavel Labath [mailto:lab...@google.com]
>> Sent: Tuesday, November 03, 2015 5:41 PM
>> To: Greg Clayton 
>> Cc: Mark Chandler ; lldb-dev@lists.llvm.org
>> Subject: Re: [lldb-dev] Attaching to a stopped (cored) process hangs
>> lldb-server
>>
>> I'm following this discussion, but I don't yet understand what is going
>> on here completely. What I am sure is that the problem here is not the S+
>> state, as that just means "interruptible sleep, foreground process", and a
>> lot of processes have that state and we attach to them just fine. I would
>> need to investigate what are the exact properties or this cored state. I'll
>> try to take a look when I get some spare cycles, but that might not happen
>> very soon.
>>
>> Mark, have you investigated what is the next thing to fail after you
>> remove the waitpid call?
>>
>> pl
>>
>> On 3 November 2015 at 16:48, Greg Clayton via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>> > Can someone with linux experience chime in here? It shouldn't be too
>> hard to figure out which flag 'S' is in. On MacOS we can get a process info
>> structure from a pid and that will have bits set that indicate 'S'...
>> >
>> > If you want to checkin this tool into the LLDB source tree at
>> trunk/tools/core_tool then we can get more people to work on it and improve
>> it. It would be nice to have this available for all linux users. I would
>> love to see an JSON output mode that is parseable by automated tools
>> instead of people saving text formats that must be text scraped.
>> >
>> > If you can get this into a tool, others can help get this working. Any
>> interest in this?
>> >
>> > Greg
>> >
>> >> On Nov 3, 2015, at 4:41 PM, Mark Chandler 
>> wrote:
>> >>
>> >> The biggest tell is that the process state is already 'S' or stopped.
>> I don’t know lldb at all to make a change to fix this though.
>> >>
>> >>
>> >> Mark Chandler
>> >> Battle.Net Engineering Systems | Blizzard Entertainment
>> >> (P) 949-955-1380 x15353
>> >>
>> >> -Original Message-
>> >> From: Greg Clayton [mailto:gclay...@apple.com]
>> >> Sent: Tuesday, November 03, 2015 4:39 PM
>> >> To: Mark Chandler 
>> >> Cc: lldb-dev@lists.llvm.org
>> >> Subject: Re: [lldb-dev] Attaching to a stopped (cored) process hangs
>> >> lldb-server
>> >>
>> >> Makes sense about not writing the core file to disk.
>> >>
>> >> Is there a way you can detect this "core" mode where we don't have to
>> waitpid? Seems like that www.sourceware.org message had ideas on how to
>> detect this case?
>> >>
>> >> Greg
>> >>
>> >>> On Nov 3, 2015, at 4:36 PM, Mark Chandler 
>> wrote:
>> >>>
>> >>> Not able to do that as the servers have no hard drives (use ram disk
>> >>> and net boot) and the tool is trying to avoid a core storm that
>> >>> takes down the network file share. I found out what is causing it to
>> >>> hang, there is a call to waitpid in NativeLinuxProcess.cpp that
>> >>> waits forever. As the process is already stopped, I disabled that
>> >>> and it looks to be working
>> >>>
>> >>> Mark Chandler
>> >>> Battle.Net Engineering Systems | Blizzard Entertainment
>> >>> (P) 949-955-1380 x15353
>> >>>
>> >>> -Original Message-
>> >>> From: Greg Clayton [mailto:gclay...@apple.com]
>> >>> Sent: Tuesday, November 03, 2015 4:34 PM
>> >>> To: Mark Chandler 
>> >>> Cc: lldb-dev@lists.llvm.org
>> >>> Subject: Re: [lldb-dev] Attaching to a stopped (cored) process hangs
>> >>> lldb-server
>> >>>
>> >>> One different approach is to have your tool write all STDIN to a file
>> (the core file comes into the tool as STDIN bytes) and then hand LLDB the
>> core file and do any needed backtracing and data gathering from the core
>> file instead of actually attaching to the process for real. All executable
>> and shared library object files (ELF files) from the core file are still on
>> disk so you can get symbols and use the debug info, so LLDB should be able
>> to load all frames up and symbolicate up the crash location. It should be
>> just as good as having the process around witho

[lldb-dev] swig generation

[Todd Fiala via lldb-dev]

Hi all,

I'd like to do a few things with our swig generation and handling:

* Create a maintainer-mode style setup where the swig Python bindings are
generated and checked into the repo (I'll call it the static python
binding).

This will be used by default, removing the need for most people and all
builders/bots from having swig on their system just for lldb.  In the event
that Windows needs a different version (e.g. for, say, Python 3.5 support
that is incompatible with the swig-1.3.40-generated bindings), we can
support multiple source bindings, or we can post process the swig-1.3.x
generated bindings.

We'll keep them by swig-{swig-major}-{swig-minor}.  Internally over at
Apple, we will stick with the swig-1.x bindings.  I don't think we will
care about the dot release (1.3.40 vs. 1.3.39, for example).  We'll just
make sure to use the latest for a {swig-major}.{swig-minor}.  As always, SB
API changes generated by swig need to remain swig-1.3 compatible (i.e.
swig-1.3 must be capable of generating usable Python wrappers).

Ideally we don't need more than one set of bindings so we can avoid needing
to generate multiple ones when we do it.


* Add an explicit Python binding generation step.

This would only be used by those who are touching the bindings (adding SB
API or adjusting the documentation of existing SB API).  In the event that
we need two bindings, we may just need a handshake that says "okay, we'll
take care of the swig 1.3 bindings, and {somebody who needs the other
binding} generates it for the other."  As SBAPI is additive only, this
should generally be fine as the worst case I can think of that would happen
would be failure to see new SBAPI in Python for a very brief time.  Since
maintainers will be writing Python tests to check their new SBAPIs, failure
to do the explicit generation step will be immediately obvious as a test
failure.  (The challenge here will likely be the swig 1.3.x requirement).

If generating and testing new binding content with the right swig (i.e.
swig 1.3.x) becomes a real issue, we may be able to support a "please use
my local swig, whatever it is, but don't check in the static binding), with
a handshake that says "somebody with swig 1.3, please generate the modified
binding and check in).  We'll see if this becomes an issue.  I don't see
this as insurmountable.


* Clean up the swig generation scripts

These look to have been implemented as direct ports of the older OS X style
bash scripts.  This Python script is very much the essence of using the
paradigms of one language in another, and being the worse for it.  It
implements features that are already present in the standard Python lib,
and is more verbose than it needs to be.

Also, it is likely the script needs to be broken out into a few parts.  The
scripts don't just generate the python binding using swig, they also setup
(for OS X) some packaging and other bits.  Right now none of that is
clearly broken out.  When we move to an explicit binding generation mode,
this does need to be broken out better.


* Get OS X Xcode build using the same bindings machinery as others.

I tried this a while back (having Xcode adopt the Python-based swig wrapper
handling code), but gave up after hitting a few bugs where the translated
behavior was incorrect for Xcode.  I will move over to adopting this on
Xcode if possible while going through these changes.

The primary goal here is to remove the requirement of having swig on the
system (e.g. for builders and most developers), shifting that burden a bit
more to those who actually modify the Python binding.

As a potential added benefit, this opens us up to easier modification of
that binding generation step, which may prove to be useful later.

Let me know if you have any feedback before I dive into this.

-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] swig generation

[Todd Fiala via lldb-dev]

On Fri, Nov 13, 2015 at 9:43 AM, Zachary Turner  wrote:

> On Fri, Nov 13, 2015 at 9:02 AM Todd Fiala via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> Hi all,
>>
>> I'd like to do a few things with our swig generation and handling:
>>
>> * Create a maintainer-mode style setup where the swig Python bindings are
>> generated and checked into the repo (I'll call it the static python
>> binding).
>>
>> This will be used by default, removing the need for most people and all
>> builders/bots from having swig on their system just for lldb.  In the event
>> that Windows needs a different version (e.g. for, say, Python 3.5 support
>> that is incompatible with the swig-1.3.40-generated bindings), we can
>> support multiple source bindings, or we can post process the swig-1.3.x
>> generated bindings.  We'll keep them by swig-{swig-major}-{swig-minor}.
>> Internally over at Apple, we will stick with the swig-1.x bindings.  I
>> don't think we will care about the dot release (1.3.40 vs. 1.3.39, for
>> example).  We'll just make sure to use the latest for a
>> {swig-major}.{swig-minor}.  As always, SB API changes generated by swig
>> need to remain swig-1.3 compatible (i.e. swig-1.3 must be capable of
>> generating usable Python wrappers).
>>
> I know you said you plan to stay on 1.x, but in this world of having SWIG
> bindings checked in, does that open the door to potentially moving beyond
> SWIG 1.x for the upstream?  I feel like the SWIG 1.x requirement holds the
> upstream back, because there are a lot of hacks to workaround bugs and
> limitations in SWIG, and it only understands the absolute most basic C /
> C++ language constructs.  So we end up being limited in how we can design
> SB APIs, all for reasons that having nothing to do with the upstream
> project requirements, which is kind of a bummer.  It seems like there could
> be a path here for the upstream to move forward, and anyone who is forced
> to stay on an earlier version of SWIG could maintain whatever changes are
> necessary to make this possible in a local repository.
>
>
I may leave Greg to discuss that aspect (w/r/t using newer features).  The
SB API itself is intended to be a very bare-bones linkage environment that
does not break linkage requirements in ways that typical C++ binary
libraries do (i.e. it does not use virtuals and follows a number of rules
about data members so that it remains binary compatible across
compiler/linker changes).


>
>
>>
>> * Clean up the swig generation scripts
>>
>> These look to have been implemented as direct ports of the older OS X
>> style bash scripts.  This Python script is very much the essence of using
>> the paradigms of one language in another, and being the worse for it.  It
>> implements features that are already present in the standard Python lib,
>> and is more verbose than it needs to be.
>>
> Ugh, +1000.  Every time I have to crack these files open my head
> explodes.  More than happy to help with whatever porting is necessary.
>
>

Haha that's exactly how I feel :-)


>
>> Also, it is likely the script needs to be broken out into a few parts.
>> The scripts don't just generate the python binding using swig, they also
>> setup (for OS X) some packaging and other bits.  Right now none of that is
>> clearly broken out.  When we move to an explicit binding generation mode,
>> this does need to be broken out better.
>>
>>
>> * Get OS X Xcode build using the same bindings machinery as others.
>>
> Yay.
>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] swig generation

[Todd Fiala via lldb-dev]

On Fri, Nov 13, 2015 at 10:24 AM, Todd Fiala  wrote:

>
>
> On Fri, Nov 13, 2015 at 9:43 AM, Zachary Turner 
> wrote:
>
>> On Fri, Nov 13, 2015 at 9:02 AM Todd Fiala via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> Hi all,
>>>
>>> I'd like to do a few things with our swig generation and handling:
>>>
>>> * Create a maintainer-mode style setup where the swig Python bindings
>>> are generated and checked into the repo (I'll call it the static python
>>> binding).
>>>
>>> This will be used by default, removing the need for most people and all
>>> builders/bots from having swig on their system just for lldb.  In the event
>>> that Windows needs a different version (e.g. for, say, Python 3.5 support
>>> that is incompatible with the swig-1.3.40-generated bindings), we can
>>> support multiple source bindings, or we can post process the swig-1.3.x
>>> generated bindings.  We'll keep them by swig-{swig-major}-{swig-minor}.
>>> Internally over at Apple, we will stick with the swig-1.x bindings.  I
>>> don't think we will care about the dot release (1.3.40 vs. 1.3.39, for
>>> example).  We'll just make sure to use the latest for a
>>> {swig-major}.{swig-minor}.  As always, SB API changes generated by swig
>>> need to remain swig-1.3 compatible (i.e. swig-1.3 must be capable of
>>> generating usable Python wrappers).
>>>
>> I know you said you plan to stay on 1.x, but in this world of having SWIG
>> bindings checked in, does that open the door to potentially moving beyond
>> SWIG 1.x for the upstream?  I feel like the SWIG 1.x requirement holds the
>> upstream back, because there are a lot of hacks to workaround bugs and
>> limitations in SWIG,
>>
>
We do have the ability to post-process this any way we want after the swig
generation.  I see no reason why we couldn't get move advanced with what we
post-process, especially since that can be limited to the
maintainer-mode-style "update the static checked-in product after
generating the bindings" step).


> and it only understands the absolute most basic C / C++ language
>> constructs.  So we end up being limited in how we can design SB APIs, all
>> for reasons that having nothing to do with the upstream project
>> requirements, which is kind of a bummer.  It seems like there could be a
>> path here for the upstream to move forward, and anyone who is forced to
>> stay on an earlier version of SWIG could maintain whatever changes are
>> necessary to make this possible in a local repository.
>>
>>
> I may leave Greg to discuss that aspect (w/r/t using newer features).  The
> SB API itself is intended to be a very bare-bones linkage environment that
> does not break linkage requirements in ways that typical C++ binary
> libraries do (i.e. it does not use virtuals and follows a number of rules
> about data members so that it remains binary compatible across
> compiler/linker changes).
>
>
>>
>>
>>>
>>> * Clean up the swig generation scripts
>>>
>>> These look to have been implemented as direct ports of the older OS X
>>> style bash scripts.  This Python script is very much the essence of using
>>> the paradigms of one language in another, and being the worse for it.  It
>>> implements features that are already present in the standard Python lib,
>>> and is more verbose than it needs to be.
>>>
>> Ugh, +1000.  Every time I have to crack these files open my head
>> explodes.  More than happy to help with whatever porting is necessary.
>>
>>
>
> Haha that's exactly how I feel :-)
>
>
>>
>>> Also, it is likely the script needs to be broken out into a few parts.
>>> The scripts don't just generate the python binding using swig, they also
>>> setup (for OS X) some packaging and other bits.  Right now none of that is
>>> clearly broken out.  When we move to an explicit binding generation mode,
>>> this does need to be broken out better.
>>>
>>>
>>> * Get OS X Xcode build using the same bindings machinery as others.
>>>
>> Yay.
>>
>>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] swig generation

[Todd Fiala via lldb-dev]

I will probably tackle this as two phases:

Phase 1:
* Python script modernization (the python swig wrapper generation)
* Move Xcode onto it.

Phase 2:
* The maintainer-mode, static Python binding generation changes for cmake
and Xcode.

I want to make sure we have proven, still-functional Python from the new
Python script before we move forward.  I will also probably turn on the
Green Dragon OS X public builder's test running at this stage so we can
also see that.

And I want to work out any issues with the current script logic and what OS
X was doing before shifting any of this around.

On Fri, Nov 13, 2015 at 9:02 AM, Todd Fiala  wrote:

> Hi all,
>
> I'd like to do a few things with our swig generation and handling:
>
> * Create a maintainer-mode style setup where the swig Python bindings are
> generated and checked into the repo (I'll call it the static python
> binding).
>
> This will be used by default, removing the need for most people and all
> builders/bots from having swig on their system just for lldb.  In the event
> that Windows needs a different version (e.g. for, say, Python 3.5 support
> that is incompatible with the swig-1.3.40-generated bindings), we can
> support multiple source bindings, or we can post process the swig-1.3.x
> generated bindings.
>
> We'll keep them by swig-{swig-major}-{swig-minor}.  Internally over at
> Apple, we will stick with the swig-1.x bindings.  I don't think we will
> care about the dot release (1.3.40 vs. 1.3.39, for example).  We'll just
> make sure to use the latest for a {swig-major}.{swig-minor}.  As always, SB
> API changes generated by swig need to remain swig-1.3 compatible (i.e.
> swig-1.3 must be capable of generating usable Python wrappers).
>
> Ideally we don't need more than one set of bindings so we can avoid
> needing to generate multiple ones when we do it.
>
>
> * Add an explicit Python binding generation step.
>
> This would only be used by those who are touching the bindings (adding SB
> API or adjusting the documentation of existing SB API).  In the event that
> we need two bindings, we may just need a handshake that says "okay, we'll
> take care of the swig 1.3 bindings, and {somebody who needs the other
> binding} generates it for the other."  As SBAPI is additive only, this
> should generally be fine as the worst case I can think of that would happen
> would be failure to see new SBAPI in Python for a very brief time.  Since
> maintainers will be writing Python tests to check their new SBAPIs, failure
> to do the explicit generation step will be immediately obvious as a test
> failure.  (The challenge here will likely be the swig 1.3.x requirement).
>
> If generating and testing new binding content with the right swig (i.e.
> swig 1.3.x) becomes a real issue, we may be able to support a "please use
> my local swig, whatever it is, but don't check in the static binding), with
> a handshake that says "somebody with swig 1.3, please generate the modified
> binding and check in).  We'll see if this becomes an issue.  I don't see
> this as insurmountable.
>
>
> * Clean up the swig generation scripts
>
> These look to have been implemented as direct ports of the older OS X
> style bash scripts.  This Python script is very much the essence of using
> the paradigms of one language in another, and being the worse for it.  It
> implements features that are already present in the standard Python lib,
> and is more verbose than it needs to be.
>
> Also, it is likely the script needs to be broken out into a few parts.
> The scripts don't just generate the python binding using swig, they also
> setup (for OS X) some packaging and other bits.  Right now none of that is
> clearly broken out.  When we move to an explicit binding generation mode,
> this does need to be broken out better.
>
>
> * Get OS X Xcode build using the same bindings machinery as others.
>
> I tried this a while back (having Xcode adopt the Python-based swig
> wrapper handling code), but gave up after hitting a few bugs where the
> translated behavior was incorrect for Xcode.  I will move over to adopting
> this on Xcode if possible while going through these changes.
>
> The primary goal here is to remove the requirement of having swig on the
> system (e.g. for builders and most developers), shifting that burden a bit
> more to those who actually modify the Python binding.
>
> As a potential added benefit, this opens us up to easier modification of
> that binding generation step, which may prove to be useful later.
>
> Let me know if you have any feedback before I dive into this.
>
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Two CLs requiring changes to the Xcode project

[Todd Fiala via lldb-dev]

Looks like everything was covered.  Thanks, all!

On Thu, Nov 12, 2015 at 7:07 PM, Zachary Turner  wrote:

> Thanks!  I actually forgot, there's one more file for the lldb-gtest
> target.  It's PythonExceptionStateTests.cpp.
>
> Thanks for the help
>
> On Thu, Nov 12, 2015 at 6:05 PM Jason Molenda  wrote:
>
>> Ah, my bad.  It's the lldb-gtest target.
>>
>>
>> > On Nov 12, 2015, at 5:49 PM, Zachary Turner  wrote:
>> >
>> > Hmm, can you ask Todd about it?  He said he added one, but I'm not sure
>> how it works.
>> >
>> > On Thu, Nov 12, 2015 at 5:46 PM Jason Molenda 
>> wrote:
>> > Done in r252998.
>> >
>> > I didn't see anything in the xcode project file about a gtest target.
>> >
>> > J
>> >
>> > > On Nov 12, 2015, at 5:39 PM, Zachary Turner via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>> > >
>> > > Hi all,
>> > >
>> > > I submitted r252993 and 252994.  These changes will require a
>> corresponding change in the Xcode workspace.  Would anyone mind making
>> those changes for me?  It should be pretty simple, just need to add a .cpp
>> and .h file to the gtest target for ScriptInterpreterPythonTests, and add
>> PythonExceptionState.cpp to Plugins/ScriptInterpreter/Python
>> > > ___
>> > > lldb-dev mailing list
>> > > lldb-dev@lists.llvm.org
>> > > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>> >
>>
>>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] swig generation

[Todd Fiala via lldb-dev]

On Fri, Nov 13, 2015 at 9:02 AM, Todd Fiala  wrote:

> Hi all,
>
> I'd like to do a few things with our swig generation and handling:
>
> * Create a maintainer-mode style setup where the swig Python bindings are
> generated and checked into the repo (I'll call it the static python
> binding).
>
> This will be used by default, removing the need for most people and all
> builders/bots from having swig on their system just for lldb.  In the event
> that Windows needs a different version (e.g. for, say, Python 3.5 support
> that is incompatible with the swig-1.3.40-generated bindings), we can
> support multiple source bindings, or we can post process the swig-1.3.x
> generated bindings.
>
> We'll keep them by swig-{swig-major}-{swig-minor}.  Internally over at
> Apple, we will stick with the swig-1.x bindings.  I don't think we will
> care about the dot release (1.3.40 vs. 1.3.39, for example).  We'll just
> make sure to use the latest for a {swig-major}.{swig-minor}.  As always, SB
> API changes generated by swig need to remain swig-1.3 compatible (i.e.
> swig-1.3 must be capable of generating usable Python wrappers).
>
> Ideally we don't need more than one set of bindings so we can avoid
> needing to generate multiple ones when we do it.
>
>
> * Add an explicit Python binding generation step.
>
> This would only be used by those who are touching the bindings (adding SB
> API or adjusting the documentation of existing SB API).  In the event that
> we need two bindings, we may just need a handshake that says "okay, we'll
> take care of the swig 1.3 bindings, and {somebody who needs the other
> binding} generates it for the other."  As SBAPI is additive only, this
> should generally be fine as the worst case I can think of that would happen
> would be failure to see new SBAPI in Python for a very brief time.  Since
> maintainers will be writing Python tests to check their new SBAPIs, failure
> to do the explicit generation step will be immediately obvious as a test
> failure.  (The challenge here will likely be the swig 1.3.x requirement).
>
> If generating and testing new binding content with the right swig (i.e.
> swig 1.3.x) becomes a real issue, we may be able to support a "please use
> my local swig, whatever it is, but don't check in the static binding), with
> a handshake that says "somebody with swig 1.3, please generate the modified
> binding and check in).  We'll see if this becomes an issue.  I don't see
> this as insurmountable.
>
>
> * Clean up the swig generation scripts
>
> These look to have been implemented as direct ports of the older OS X
> style bash scripts.  This Python script is very much the essence of using
> the paradigms of one language in another, and being the worse for it.  It
> implements features that are already present in the standard Python lib,
> and is more verbose than it needs to be.
>
> Also, it is likely the script needs to be broken out into a few parts.
> The scripts don't just generate the python binding using swig, they also
> setup (for OS X) some packaging and other bits.  Right now none of that is
> clearly broken out.  When we move to an explicit binding generation mode,
> this does need to be broken out better.
>
>
> * Get OS X Xcode build using the same bindings machinery as others.
>
> I tried this a while back (having Xcode adopt the Python-based swig
> wrapper handling code), but gave up after hitting a few bugs where the
> translated behavior was incorrect for Xcode.  I will move over to adopting
> this on Xcode if possible while going through these changes.
>

I've begun this process.  I've got Xcode build (and the Xcode build only)
wired up to scripts/prepare_bindings.py.  This, and the Python-specific
binding preparation script, are considerably shorter than the older
implementation.  It also addresses the bugs that prevented the former from
working with the Xcode build.  The new scripts are pylint clean  (with
stock configuration) except for a small handful of places that I intend to
revisit when I have a few cycles to really tidy things up.  (Right now I'm
just going for the 90% wins).

Tomorrow I will fix up the tail end (the finalization) in a similar manner
and plug it into the Xcode build.  I'll also try it locally on Linux and
work out any issues there.

I'll be tackling the static binding area (and will put up for review) only
after I know the cleaned up binding preparation scripts are working
properly everywhere.  I just didn't want to tackle that until I really
understood what the scripts were already doing and it was in a state I felt
I could maintain.


>
> The primary goal here is to remove the requirement of having swig on the
> system (e.g. for builders and most developers), shifting that burden a bit
> more to those who actually modify the Python binding.
>
> As a potential added benefit, this opens us up to easier modification of
> that binding generation step, which may prove to be useful later.
>
> Let me know if you have any feedback 

Re: [lldb-dev] swig generation

[Todd Fiala via lldb-dev]

On Mon, Nov 16, 2015 at 11:28 PM, Todd Fiala  wrote:

>
>
> On Fri, Nov 13, 2015 at 9:02 AM, Todd Fiala  wrote:
>
>> Hi all,
>>
>> I'd like to do a few things with our swig generation and handling:
>>
>> * Create a maintainer-mode style setup where the swig Python bindings are
>> generated and checked into the repo (I'll call it the static python
>> binding).
>>
>> This will be used by default, removing the need for most people and all
>> builders/bots from having swig on their system just for lldb.  In the event
>> that Windows needs a different version (e.g. for, say, Python 3.5 support
>> that is incompatible with the swig-1.3.40-generated bindings), we can
>> support multiple source bindings, or we can post process the swig-1.3.x
>> generated bindings.
>>
>> We'll keep them by swig-{swig-major}-{swig-minor}.  Internally over at
>> Apple, we will stick with the swig-1.x bindings.  I don't think we will
>> care about the dot release (1.3.40 vs. 1.3.39, for example).  We'll just
>> make sure to use the latest for a {swig-major}.{swig-minor}.  As always, SB
>> API changes generated by swig need to remain swig-1.3 compatible (i.e.
>> swig-1.3 must be capable of generating usable Python wrappers).
>>
>> Ideally we don't need more than one set of bindings so we can avoid
>> needing to generate multiple ones when we do it.
>>
>>
>> * Add an explicit Python binding generation step.
>>
>> This would only be used by those who are touching the bindings (adding SB
>> API or adjusting the documentation of existing SB API).  In the event that
>> we need two bindings, we may just need a handshake that says "okay, we'll
>> take care of the swig 1.3 bindings, and {somebody who needs the other
>> binding} generates it for the other."  As SBAPI is additive only, this
>> should generally be fine as the worst case I can think of that would happen
>> would be failure to see new SBAPI in Python for a very brief time.  Since
>> maintainers will be writing Python tests to check their new SBAPIs, failure
>> to do the explicit generation step will be immediately obvious as a test
>> failure.  (The challenge here will likely be the swig 1.3.x requirement).
>>
>> If generating and testing new binding content with the right swig (i.e.
>> swig 1.3.x) becomes a real issue, we may be able to support a "please use
>> my local swig, whatever it is, but don't check in the static binding), with
>> a handshake that says "somebody with swig 1.3, please generate the modified
>> binding and check in).  We'll see if this becomes an issue.  I don't see
>> this as insurmountable.
>>
>>
>> * Clean up the swig generation scripts
>>
>> These look to have been implemented as direct ports of the older OS X
>> style bash scripts.  This Python script is very much the essence of using
>> the paradigms of one language in another, and being the worse for it.  It
>> implements features that are already present in the standard Python lib,
>> and is more verbose than it needs to be.
>>
>> Also, it is likely the script needs to be broken out into a few parts.
>> The scripts don't just generate the python binding using swig, they also
>> setup (for OS X) some packaging and other bits.  Right now none of that is
>> clearly broken out.  When we move to an explicit binding generation mode,
>> this does need to be broken out better.
>>
>>
>> * Get OS X Xcode build using the same bindings machinery as others.
>>
>> I tried this a while back (having Xcode adopt the Python-based swig
>> wrapper handling code), but gave up after hitting a few bugs where the
>> translated behavior was incorrect for Xcode.  I will move over to adopting
>> this on Xcode if possible while going through these changes.
>>
>
> I've begun this process.
>

With r253317, that is.


> I've got Xcode build (and the Xcode build only) wired up to
> scripts/prepare_bindings.py.  This, and the Python-specific binding
> preparation script, are considerably shorter than the older
> implementation.  It also addresses the bugs that prevented the former from
> working with the Xcode build.  The new scripts are pylint clean  (with
> stock configuration) except for a small handful of places that I intend to
> revisit when I have a few cycles to really tidy things up.  (Right now I'm
> just going for the 90% wins).
>
> Tomorrow I will fix up the tail end (the finalization) in a similar manner
> and plug it into the Xcode build.  I'll also try it locally on Linux and
> work out any issues there.
>
> I'll be tackling the static binding area (and will put up for review) only
> after I know the cleaned up binding preparation scripts are working
> properly everywhere.  I just didn't want to tackle that until I really
> understood what the scripts were already doing and it was in a state I felt
> I could maintain.
>
>
>> The primary goal here is to remove the requirement of having swig on the
>> system (e.g. for builders and most developers), shifting that burden a bit
>> more to those who actually modify the Pytho

Re: [lldb-dev] [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.

[Todd Fiala via lldb-dev]

Nothing concrete at the moment; however, it could be interesting to look at
the clang community and see what could be done for llvm-based language
implementations.  The angle that I think would be interesting would be if
we can generate bindings more effectively based on the in-depth
understanding of the language that is afforded by languages built on top of
LLVM.  This is probably less interesting for Python (particularly since we
have a functioning solution) and more interesting for languages built on
LLVM or clang.

Honestly, though, I haven't spent much time on that.

For the time being, I am going to not change the path for everyone on swig,
and only use a static binding if swig cannot be found.  This will be
minimal impact for everyone and doesn't interfere with anyone using a
specific version of swig.  We can revisit larger questions about
who/what/when on static bindings after we gain some experience with
enabling them for those who don't have swig.  We can review and adjust
based on our collective experience.  The two files this seems like it will
be are the LLDBWrapPython.cpp and the lldb.py file that comes out of
python.  I hope to have this working in the next day or so.


On Tue, Nov 17, 2015 at 7:26 PM, Bruce Mitchener 
wrote:

> Stepping one step back further in the thread ...
>
> On Wed, Nov 18, 2015 at 8:35 AM, Zachary Turner via lldb-commits <
> lldb-comm...@lists.llvm.org> wrote:
>
>> Moving this back over to the list since I'm sure others have some input
>> here.  Also +lldb-dev since it has more visibility than lldb-commits.
>>
>>
>> On Tue, Nov 17, 2015 at 11:25 AM Zachary Turner 
>> wrote:
>>
>>> On Tue, Nov 17, 2015 at 8:18 AM Todd Fiala  wrote:
>>>
 Breaking out the binding generation into a separate step will also be
 important for a couple reasons:

 * (from before) I want to eliminate the requirement for the vast
 majority of the builds to have a swig on their system, and

 * (not stated before) we'd like to move away from swig for binding
 generation at some point.

>>>
> Is there any discussion or thoughts about what the options would be for
> moving away from swig?
>
>  - Bruce
>
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.

[Todd Fiala via lldb-dev]

> that comes out of python

that comes out of swig, rather (i.e. the binding generation output).

On Tue, Nov 17, 2015 at 8:03 PM, Todd Fiala  wrote:

> Nothing concrete at the moment; however, it could be interesting to look
> at the clang community and see what could be done for llvm-based language
> implementations.  The angle that I think would be interesting would be if
> we can generate bindings more effectively based on the in-depth
> understanding of the language that is afforded by languages built on top of
> LLVM.  This is probably less interesting for Python (particularly since we
> have a functioning solution) and more interesting for languages built on
> LLVM or clang.
>
> Honestly, though, I haven't spent much time on that.
>
> For the time being, I am going to not change the path for everyone on
> swig, and only use a static binding if swig cannot be found.  This will be
> minimal impact for everyone and doesn't interfere with anyone using a
> specific version of swig.  We can revisit larger questions about
> who/what/when on static bindings after we gain some experience with
> enabling them for those who don't have swig.  We can review and adjust
> based on our collective experience.  The two files this seems like it will
> be are the LLDBWrapPython.cpp and the lldb.py file that comes out of
> python.  I hope to have this working in the next day or so.
>
>
> On Tue, Nov 17, 2015 at 7:26 PM, Bruce Mitchener <
> bruce.mitche...@gmail.com> wrote:
>
>> Stepping one step back further in the thread ...
>>
>> On Wed, Nov 18, 2015 at 8:35 AM, Zachary Turner via lldb-commits <
>> lldb-comm...@lists.llvm.org> wrote:
>>
>>> Moving this back over to the list since I'm sure others have some input
>>> here.  Also +lldb-dev since it has more visibility than lldb-commits.
>>>
>>>
>>> On Tue, Nov 17, 2015 at 11:25 AM Zachary Turner 
>>> wrote:
>>>
 On Tue, Nov 17, 2015 at 8:18 AM Todd Fiala 
 wrote:

> Breaking out the binding generation into a separate step will also be
> important for a couple reasons:
>
> * (from before) I want to eliminate the requirement for the vast
> majority of the builds to have a swig on their system, and
>
> * (not stated before) we'd like to move away from swig for binding
> generation at some point.
>

>> Is there any discussion or thoughts about what the options would be for
>> moving away from swig?
>>
>>  - Bruce
>>
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.

[Todd Fiala via lldb-dev]

Usage of static bindings for the prepare_bindings.py script went in here:
r253448.

Only Xcode provides the flag to use it.  The commit description indicates
how it works in this incarnation.  Essentially it only uses the static
Python binding if and only if a swig isn't specified or cannot be found.

I'll be testing this on Ubuntu 14.04 and 15.10 in the morning.  Once I have
that working, I'll provide a cmake flag to vector over to it, defaulting to
not do so.

-Todd


On Tue, Nov 17, 2015 at 8:12 PM, Todd Fiala  wrote:

> > that comes out of python
>
> that comes out of swig, rather (i.e. the binding generation output).
>
> On Tue, Nov 17, 2015 at 8:03 PM, Todd Fiala  wrote:
>
>> Nothing concrete at the moment; however, it could be interesting to look
>> at the clang community and see what could be done for llvm-based language
>> implementations.  The angle that I think would be interesting would be if
>> we can generate bindings more effectively based on the in-depth
>> understanding of the language that is afforded by languages built on top of
>> LLVM.  This is probably less interesting for Python (particularly since we
>> have a functioning solution) and more interesting for languages built on
>> LLVM or clang.
>>
>> Honestly, though, I haven't spent much time on that.
>>
>> For the time being, I am going to not change the path for everyone on
>> swig, and only use a static binding if swig cannot be found.  This will be
>> minimal impact for everyone and doesn't interfere with anyone using a
>> specific version of swig.  We can revisit larger questions about
>> who/what/when on static bindings after we gain some experience with
>> enabling them for those who don't have swig.  We can review and adjust
>> based on our collective experience.  The two files this seems like it will
>> be are the LLDBWrapPython.cpp and the lldb.py file that comes out of
>> python.  I hope to have this working in the next day or so.
>>
>>
>> On Tue, Nov 17, 2015 at 7:26 PM, Bruce Mitchener <
>> bruce.mitche...@gmail.com> wrote:
>>
>>> Stepping one step back further in the thread ...
>>>
>>> On Wed, Nov 18, 2015 at 8:35 AM, Zachary Turner via lldb-commits <
>>> lldb-comm...@lists.llvm.org> wrote:
>>>
 Moving this back over to the list since I'm sure others have some input
 here.  Also +lldb-dev since it has more visibility than lldb-commits.


 On Tue, Nov 17, 2015 at 11:25 AM Zachary Turner 
 wrote:

> On Tue, Nov 17, 2015 at 8:18 AM Todd Fiala 
> wrote:
>
>> Breaking out the binding generation into a separate step will also be
>> important for a couple reasons:
>>
>> * (from before) I want to eliminate the requirement for the vast
>> majority of the builds to have a swig on their system, and
>>
>> * (not stated before) we'd like to move away from swig for binding
>> generation at some point.
>>
>
>>> Is there any discussion or thoughts about what the options would be for
>>> moving away from swig?
>>>
>>>  - Bruce
>>>
>>>
>>
>>
>>
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.

[Todd Fiala via lldb-dev]

Checking in the static bindings is no different than projects checking in
autoconf config baked scripts so that the vast majority of people don't
need to run autoconf just to get a setup that rarely changes.  There is
precedent for this going back a long way on open source projects.

I'm backing off having anyone else use them if they don't want, and we
(Apple) will keep those up to date.  Nobody else will use them.  Totally
fine.

On the swig-as-a-service front, I think the idea is interesting but there
are several practical holes with that:

* What does one do when the server is unavailable?  Needs to be a
local-only backup.  So whatever service that provides isn't a guaranteed
working solution (think on an airplane, network outage, other server
outage, etc.)

* Security: building code that has other code injected in it by another
server. Safe? Attack vector?  I could argue so is a git/svn repo
susceptible to this, so maybe this isn't a huge deal, but it's big enough
and smells enough like "introduce random unvetted code that can't be
reviewed as easily as a VCS tag" that I doubt we would ever use this in
practice.

* Security 2: what is the service really running?  Not obvious on the build
machine accessing the service.

> In all of these cases (except the proposed), the matrix choices are
justifiable because they are there to support a hard requirement of
someone's environment, and I do not think we should grow for anything that
is not also a hard requirement of someone's environment.

I'm going to call that overreaching.  We are not in the business of
dictating that one of the developers of the code "should not do something
unless there is a hard requirement."  Apple wants to eliminate the need for
people to *require* swig.  The goal there is reducing the build
requirements for the average person building lldb, not growing it.

Saying that "you hit a server for bindings as part of the build" is way
different and more onerous than saying "hey we don't want you to *require*
to have swig to build lldb."  Those are miles apart.

> And my question is, who doesn't have swig?  Maybe there is a legitimate
use case, I just want to understand what that is before we add more
different ways of building.

Swig is not a component that comes pre-installed on most public systems.
Almost nobody has it unless something they do says "you need to have this."
 So for a casual developer who is not a hardcore lldb developer, on a new
system/VM, they are not going to just have swig.  On OS X, it is a pain to
get unless you install homebrew/macports/fink.  On Linux, you'll need to do
an apt-get or yum install.

So I would flip that question around and say "who has swig unless something
requires it?"  And I'm creating a way to not require it.  That sounds a lot
more like a reduction in requirements for most build scenarios and most
people who would download and build lldb.

So for the more common, casual lldb build environment where the developer
is not touching SB API, help me understand how reducing the need for swig
(without introducing the need for hitting another server) is increasing the
requirement load?  (Especially if we --- our local dev group sitting by me
--- maintains those static bindings)?

On Wed, Nov 18, 2015 at 1:24 AM, Pavel Labath  wrote:

> On 18 November 2015 at 09:02, Zachary Turner via lldb-dev
>  wrote:
> > On Tue, Nov 17, 2015 at 8:03 PM Todd Fiala  wrote:
> >>
> >> Nothing concrete at the moment; however, it could be interesting to look
> >> at the clang community and see what could be done for llvm-based
> language
> >> implementations.  The angle that I think would be interesting would be
> if we
> >> can generate bindings more effectively based on the in-depth
> understanding
> >> of the language that is afforded by languages built on top of LLVM.
> This is
> >> probably less interesting for Python (particularly since we have a
> >> functioning solution) and more interesting for languages built on LLVM
> or
> >> clang.
> >>
> >> Honestly, though, I haven't spent much time on that.
> >>
> >> For the time being, I am going to not change the path for everyone on
> >> swig, and only use a static binding if swig cannot be found.  This will
> be
> >> minimal impact for everyone and doesn't interfere with anyone using a
> >> specific version of swig.  We can revisit larger questions about
> >> who/what/when on static bindings after we gain some experience with
> enabling
> >> them for those who don't have swig.  We can review and adjust based on
> our
> >> collective experience.  The two files this seems like it will be are the
> >> LLDBWrapPython.cpp and the lldb.py file that comes out of python.  I
> hope to
> >> have this working in the next day or so.
> >
> > To try this another way, I really would like to voice my opinion very
> very
> > strongly for moving away from having different ways of doing things for
> > different platforms / build configurations / etc unless it is *required*
> to
> > suppor

Re: [lldb-dev] [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.

[Todd Fiala via lldb-dev]

> On OS X, it is a pain to get unless you install homebrew/macports/fink.
On Linux, you'll need to do an apt-get or yum install.

And I should add that installing home-brew (the most common way of picking
up extra goodies, as I know I do it on home machines) is also one of the
most common ways to prevent lldb from building correctly and usably on OS
X.  (Particularly with the way its introduction of an alternative python
and its placement on the lookup path will totally bork a system or
hand-built lldb).

On Wed, Nov 18, 2015 at 10:00 AM, Todd Fiala  wrote:

> Checking in the static bindings is no different than projects checking in
> autoconf config baked scripts so that the vast majority of people don't
> need to run autoconf just to get a setup that rarely changes.  There is
> precedent for this going back a long way on open source projects.
>
> I'm backing off having anyone else use them if they don't want, and we
> (Apple) will keep those up to date.  Nobody else will use them.  Totally
> fine.
>
> On the swig-as-a-service front, I think the idea is interesting but there
> are several practical holes with that:
>
> * What does one do when the server is unavailable?  Needs to be a
> local-only backup.  So whatever service that provides isn't a guaranteed
> working solution (think on an airplane, network outage, other server
> outage, etc.)
>
> * Security: building code that has other code injected in it by another
> server. Safe? Attack vector?  I could argue so is a git/svn repo
> susceptible to this, so maybe this isn't a huge deal, but it's big enough
> and smells enough like "introduce random unvetted code that can't be
> reviewed as easily as a VCS tag" that I doubt we would ever use this in
> practice.
>
> * Security 2: what is the service really running?  Not obvious on the
> build machine accessing the service.
>
> > In all of these cases (except the proposed), the matrix choices are
> justifiable because they are there to support a hard requirement of
> someone's environment, and I do not think we should grow for anything that
> is not also a hard requirement of someone's environment.
>
> I'm going to call that overreaching.  We are not in the business of
> dictating that one of the developers of the code "should not do something
> unless there is a hard requirement."  Apple wants to eliminate the need for
> people to *require* swig.  The goal there is reducing the build
> requirements for the average person building lldb, not growing it.
>
> Saying that "you hit a server for bindings as part of the build" is way
> different and more onerous than saying "hey we don't want you to *require*
> to have swig to build lldb."  Those are miles apart.
>
> > And my question is, who doesn't have swig?  Maybe there is a legitimate
> use case, I just want to understand what that is before we add more
> different ways of building.
>
> Swig is not a component that comes pre-installed on most public systems.
> Almost nobody has it unless something they do says "you need to have this."
>  So for a casual developer who is not a hardcore lldb developer, on a new
> system/VM, they are not going to just have swig.  On OS X, it is a pain to
> get unless you install homebrew/macports/fink.  On Linux, you'll need to do
> an apt-get or yum install.
>
> So I would flip that question around and say "who has swig unless
> something requires it?"  And I'm creating a way to not require it.  That
> sounds a lot more like a reduction in requirements for most build scenarios
> and most people who would download and build lldb.
>
> So for the more common, casual lldb build environment where the developer
> is not touching SB API, help me understand how reducing the need for swig
> (without introducing the need for hitting another server) is increasing the
> requirement load?  (Especially if we --- our local dev group sitting by me
> --- maintains those static bindings)?
>
> On Wed, Nov 18, 2015 at 1:24 AM, Pavel Labath  wrote:
>
>> On 18 November 2015 at 09:02, Zachary Turner via lldb-dev
>>  wrote:
>> > On Tue, Nov 17, 2015 at 8:03 PM Todd Fiala 
>> wrote:
>> >>
>> >> Nothing concrete at the moment; however, it could be interesting to
>> look
>> >> at the clang community and see what could be done for llvm-based
>> language
>> >> implementations.  The angle that I think would be interesting would be
>> if we
>> >> can generate bindings more effectively based on the in-depth
>> understanding
>> >> of the language that is afforded by languages built on top of LLVM.
>> This is
>> >> probably less interesting for Python (particularly since we have a
>> >> functioning solution) and more interesting for languages built on LLVM
>> or
>> >> clang.
>> >>
>> >> Honestly, though, I haven't spent much time on that.
>> >>
>> >> For the time being, I am going to not change the path for everyone on
>> >> swig, and only use a static binding if swig cannot be found.  This
>> will be
>> >> minimal impact for everyone and doesn't interfere with

[lldb-dev] keep an eye out for broken lldb bindings

[Todd Fiala via lldb-dev]

Hi all,

I've cleaned up our swig generation scripts as used by cmake and Xcode
builds.  They're working fine on Xcode, but I've only checked them on
Ubuntu 14.04 as far as cmake builds go.  The change to switch over to the
cleaned-up script happened here:

r253478

Probably the biggest place something wrong would show up (if not the build
itself) is the running of the tests.  Let me know if you see anything
strange in the next couple hours.  r253478 would be the commit to revert as
it will switch the cmake build back to the old script.

I'd like to get the old scripts out of the repo as soon as this is verified
and/or fixed up to work.  It's a net shrinkage of code and is nearly pylint
clean.  I wanted to minimize how much I changed it while getting 90% of the
gains addressed to reduce pain of verifying the change-over.

Thanks!

-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] keep an eye out for broken lldb bindings

[Todd Fiala via lldb-dev]

FYI - This switchover change looks like it made it through the Ubuntu and
Windows builders.

On Wed, Nov 18, 2015 at 10:21 AM, Todd Fiala  wrote:

> Hi all,
>
> I've cleaned up our swig generation scripts as used by cmake and Xcode
> builds.  They're working fine on Xcode, but I've only checked them on
> Ubuntu 14.04 as far as cmake builds go.  The change to switch over to the
> cleaned-up script happened here:
>
> r253478
>
> Probably the biggest place something wrong would show up (if not the build
> itself) is the running of the tests.  Let me know if you see anything
> strange in the next couple hours.  r253478 would be the commit to revert as
> it will switch the cmake build back to the old script.
>
> I'd like to get the old scripts out of the repo as soon as this is
> verified and/or fixed up to work.  It's a net shrinkage of code and is
> nearly pylint clean.  I wanted to minimize how much I changed it while
> getting 90% of the gains addressed to reduce pain of verifying the
> change-over.
>
> Thanks!
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] keep an eye out for broken lldb bindings

[Todd Fiala via lldb-dev]

BTW this change only affected the swig wrapper generation.  There's a
separate "finish" script that I have not yet touched nor have I adopted on
Xcode.  I'll look at that as some near-term clean-up, I'd like to get Xcode
using what everyone else is using there.

-Todd

On Wed, Nov 18, 2015 at 10:32 AM, Todd Fiala  wrote:

> FYI - This switchover change looks like it made it through the Ubuntu and
> Windows builders.
>
> On Wed, Nov 18, 2015 at 10:21 AM, Todd Fiala  wrote:
>
>> Hi all,
>>
>> I've cleaned up our swig generation scripts as used by cmake and Xcode
>> builds.  They're working fine on Xcode, but I've only checked them on
>> Ubuntu 14.04 as far as cmake builds go.  The change to switch over to the
>> cleaned-up script happened here:
>>
>> r253478
>>
>> Probably the biggest place something wrong would show up (if not the
>> build itself) is the running of the tests.  Let me know if you see anything
>> strange in the next couple hours.  r253478 would be the commit to revert as
>> it will switch the cmake build back to the old script.
>>
>> I'd like to get the old scripts out of the repo as soon as this is
>> verified and/or fixed up to work.  It's a net shrinkage of code and is
>> nearly pylint clean.  I wanted to minimize how much I changed it while
>> getting 90% of the gains addressed to reduce pain of verifying the
>> change-over.
>>
>> Thanks!
>>
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.

[Todd Fiala via lldb-dev]

On Wed, Nov 18, 2015 at 10:34 AM, Zachary Turner  wrote:

>
>
> On Wed, Nov 18, 2015 at 10:00 AM Todd Fiala  wrote:
>
>> Checking in the static bindings is no different than projects checking in
>> autoconf config baked scripts so that the vast majority of people don't
>> need to run autoconf just to get a setup that rarely changes.  There is
>> precedent for this going back a long way on open source projects.
>>
>> I'm backing off having anyone else use them if they don't want, and we
>> (Apple) will keep those up to date.  Nobody else will use them.  Totally
>> fine.
>>
>> On the swig-as-a-service front, I think the idea is interesting but there
>> are several practical holes with that:
>>
>> * What does one do when the server is unavailable?  Needs to be a
>> local-only backup.  So whatever service that provides isn't a guaranteed
>> working solution (think on an airplane, network outage, other server
>> outage, etc.)
>>
>
>> * Security: building code that has other code injected in it by another
>> server. Safe? Attack vector?  I could argue so is a git/svn repo
>> susceptible to this, so maybe this isn't a huge deal, but it's big enough
>> and smells enough like "introduce random unvetted code that can't be
>> reviewed as easily as a VCS tag" that I doubt we would ever use this in
>> practice.
>>
>> * Security 2: what is the service really running?  Not obvious on the
>> build machine accessing the service.
>>
> The end result of the service is a copy of LLDBWrapPython.cpp and lldb.py
> that you check into the repo.
>

Ah, I see.  That definitely sounds like an improvement from my
interpretation :-)


> You still have a chance to diff this source code against the repository's
> copy before committing, same as you would if you had swig locally.
>

Yeah, non-issue at that point.


> Vast majority of changes to SB interfaces are going to be the addition of
> a couple methods, or maybe a class, and the diff should be very easy to
> look at and understand.
>

Yes.


>
>
>>
>> > In all of these cases (except the proposed), the matrix choices are
>> justifiable because they are there to support a hard requirement of
>> someone's environment, and I do not think we should grow for anything that
>> is not also a hard requirement of someone's environment.
>>
>> I'm going to call that overreaching.  We are not in the business of
>> dictating that one of the developers of the code "should not do something
>> unless there is a hard requirement."
>>
> I'm not saying you shouldn't do *anything* if there's not a hard
> requirement.  I agree that's overreaching.  I'm saying we should not
> increase the number of ways of doing *the same thing* unless there is a
> hard requirement.  Especially if one of the ways of doing the same thing
> exists solely to save someone from running one command to install the
> package (sorry if I'm not doing justice to how difficult it is on OSX, the
> last time I did something on OSX it seemed fairly easy to install macports,
> and I thought it's a wildly common thing for people to already have
> installed).  For example, wouldn't people need to already have macports in
> order to install CMake -- a necessary component of building LLVM?
>

In the case of cmake, no. The cmake website provides a simple-to-add binary
that doesn't depend on anything else.  (Okay, I'm long winded here so
Enrico just beat me to the punch).

For swig, the solution is homebrew/macports.  The challenge with systems
like homebrew and macports is that they will install other packages that
are dependencies (sometimes build-time, sometimes runtime), typically get
them added to various paths, and (during the process), unintentionally hide
system libraries.  It is not uncommon for us to see crash reports on OS X
where Xcode or lldb is picking up the wrong python or other component that
was built by one of these package managers.  They're often good about
saying "this might cause trouble" in some obvious cases, but less so in
less obvious cases (like, a swig installing python bindings, which installs
a new shiny python, which hides the system one, which breaks the lldb
bindings).  It really is messier there.


>
>
>>  Apple wants to eliminate the need for people to *require* swig.  The
>> goal there is reducing the build requirements for the average person
>> building lldb, not growing it.
>>
> I agree, which is why it is so important to keep the number of different
> ways of building to a minimum.
>

Glad to hear the intent here.


>   It's the same reason that the autoconf build is being removed wholesale
> from LLVM and people have decided that CMake is the one true way.
>

(Coincidentally - I am in the process of trying to eliminate the old
make-swig-bindings.sh, which is still used by the Makefile/autoconf build
of lldb, and I saw the nice banner from llvm saying autoconf support is
going out for 3.9 while testing the replacement).


>   Because even if it isn't perfect for everyone, it works for everyone.
>

Unless

Re: [lldb-dev] [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.

[Todd Fiala via lldb-dev]

On Wed, Nov 18, 2015 at 12:45 PM, Zachary Turner  wrote:

>
>
> On Wed, Nov 18, 2015 at 11:15 AM Todd Fiala  wrote:
>
>> On Wed, Nov 18, 2015 at 10:34 AM, Zachary Turner 
>> wrote:
>>
>>>   Because even if it isn't perfect for everyone, it works for everyone.
>>>
>>
>> Unless you can't get swig on your system in a way that doesn't break
>> other items.  And then it doesn't work (as things stand now).
>>
> Is that a real issue that people on OSX are facing now?  Because I thought
> having SWIG installed on your system has been a requirement for the past N
> years.  Did something change  recently that now makes it impossible to
> install swig in a way that doesn't break something?
>
>

This is changing for a large class of people.


>
>>
>>> And there is inherent simplicity in having fewer ways to do things, as
>>> well as reducing maintenance cost.
>>>
>>>

 So for the more common, casual lldb build environment where the
 developer is not touching SB API, help me understand how reducing the need
 for swig (without introducing the need for hitting another server) is
 increasing the requirement load?  (Especially if we --- our local dev group
 sitting by me --- maintains those static bindings)?

>>>
>>> Well, we would need to disable static bindings on the OSX buildbots for
>>> starters, otherwise when someone not using static bindings makes a change,
>>> the buildbots break, and we cannot leave buildbots in a broken state.  So I
>>> assume that will still be possible.  So now you don't have a buildbot
>>> testing the static binding configuration.
>>>
>>
>> I was actually going to add a verifier stage to the public OS X buildbot.
>>
> I'm not sure I follow this point.  What would your verifier stage do?  I'm
> imagining the following scenario:
>
> I check in some changes to SWIG interface files at like 10PM.  Ignoring
> the fact that you and Jason are usually up firing commits away until the
> wee hours of the morning, let's pretend nobody sees this until the
> morning.  So the build bot has been broken all night.  Is this something
> that is going to happen under this scenario?
>
>>
You can more or less ignore that point.

The intent is to alert those who update the static bindings (i.e. Apple
LLDB team) that the bindings generated and post processed from swig are not
matching the static ones.  This would just get sent to an internal group
over here since I'm not hearing definite resistance to moving to a
static-by-default model.  (If we had gone static-by-default, and made it
clean and explicit when modifying the SB API surface area by requiring an
update-the-binding step, there would have been no way to test the bindings
without having done the "update the bindings" step.  Since we won't be
doing that, then I need some way to ensure I know bindings are stale).

That's really a detail you won't need to worry about, though, since you
will not be using static bindings ever, and won't have to address any time
they go stale.  The fact that it happens on a public builder is not very
interesting either.  I worded it in a way that made it sound like others
externally would see the staleness alert, but that's not what I meant.

(I may automate that later, but not until I make sure it works manually up
front).
-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.

[Todd Fiala via lldb-dev]

On Wed, Nov 18, 2015 at 12:53 PM, Todd Fiala  wrote:

>
>
> On Wed, Nov 18, 2015 at 12:45 PM, Zachary Turner 
> wrote:
>
>>
>>
>> On Wed, Nov 18, 2015 at 11:15 AM Todd Fiala  wrote:
>>
>>> On Wed, Nov 18, 2015 at 10:34 AM, Zachary Turner 
>>> wrote:
>>>
   Because even if it isn't perfect for everyone, it works for everyone.

>>>
>>> Unless you can't get swig on your system in a way that doesn't break
>>> other items.  And then it doesn't work (as things stand now).
>>>
>> Is that a real issue that people on OSX are facing now?  Because I
>> thought having SWIG installed on your system has been a requirement for the
>> past N years.  Did something change  recently that now makes it impossible
>> to install swig in a way that doesn't break something?
>>
>>
>
> This is changing for a large class of people.
>

But more importantly, swig has never been easy to get on OS X except via
macports/fink.  Apple's internal toolchain has an ancient swig available
that we use, but we don't distribute that one.


>
>
>>
>>>
 And there is inherent simplicity in having fewer ways to do things, as
 well as reducing maintenance cost.


>
> So for the more common, casual lldb build environment where the
> developer is not touching SB API, help me understand how reducing the need
> for swig (without introducing the need for hitting another server) is
> increasing the requirement load?  (Especially if we --- our local dev 
> group
> sitting by me --- maintains those static bindings)?
>

 Well, we would need to disable static bindings on the OSX buildbots for
 starters, otherwise when someone not using static bindings makes a change,
 the buildbots break, and we cannot leave buildbots in a broken state.  So I
 assume that will still be possible.  So now you don't have a buildbot
 testing the static binding configuration.

>>>
>>> I was actually going to add a verifier stage to the public OS X buildbot.
>>>
>> I'm not sure I follow this point.  What would your verifier stage do?
>> I'm imagining the following scenario:
>>
>> I check in some changes to SWIG interface files at like 10PM.  Ignoring
>> the fact that you and Jason are usually up firing commits away until the
>> wee hours of the morning, let's pretend nobody sees this until the
>> morning.  So the build bot has been broken all night.  Is this something
>> that is going to happen under this scenario?
>>
>>>
> You can more or less ignore that point.
>
> The intent is to alert those who update the static bindings (i.e. Apple
> LLDB team) that the bindings generated and post processed from swig are not
> matching the static ones.  This would just get sent to an internal group
> over here since I'm not hearing definite resistance to moving to a
> static-by-default model.  (If we had gone static-by-default, and made it
> clean and explicit when modifying the SB API surface area by requiring an
> update-the-binding step, there would have been no way to test the bindings
> without having done the "update the bindings" step.  Since we won't be
> doing that, then I need some way to ensure I know bindings are stale).
>
> That's really a detail you won't need to worry about, though, since you
> will not be using static bindings ever, and won't have to address any time
> they go stale.  The fact that it happens on a public builder is not very
> interesting either.  I worded it in a way that made it sound like others
> externally would see the staleness alert, but that's not what I meant.
>
> (I may automate that later, but not until I make sure it works manually up
> front).
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] [Lldb-commits] [lldb] r253317 - Add Pythonic language binding wrapper generation script.

[Todd Fiala via lldb-dev]

On Wed, Nov 18, 2015 at 12:51 PM, Zachary Turner  wrote:

> (This is originally from a thread on lldb-commits, but it seems more
> appropriate here, so I'm responding here.
>
> On Wed, Nov 18, 2015 at 12:47 PM Todd Fiala  wrote:
>
>> BTW if you cook up something on the swig-as-a-service end that ends up
>> working to eliminate the need for swig, I'll be happy to remove the static
>> binding support at that point.
>>
>> -Todd
>>
>
>
> Err, rewind.  If we have the swig as a service, then I think the static
> binding does have value.
>

Yes, but in the context of what is useful for that workflow.  Not
necessarily the way I'm doing it.  (Or maybe so).


> Because I don't want to hit the network every single time I build, so it
> mostly solves the issue you mentioned about network connectivity, because
> building LLDB doesn't require a network connection unless you touch a swig
> interface file.
>
> The thing I would like some guidance on from the Apple side is this: If I
> make the swig service, can you (and will you) use code generated by swig
> 3.x?  If not, there's no value in the swig service.
>

We have no issue using code that has no additional licensing requirements
last time I verified.  And there are no additional licensing requirements
added by swig 3.x generation from what their website says (and my
non-official interpretation of it).  So I am pretty confident we can get
the answer here to be yes.



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] if you look at binding-generation-as-service element...

[Todd Fiala via lldb-dev]

Hey Zachary,

Please make sure to ping Greg Clayton, particularly if it does anything
that would make the binding no longer be C-linkage-compatible.  He can tell
you all the bits we are counting on.  I think the keys are:

* we don't ever have there be vtables.
* we never remove API surface area.
* we don't change signatures of existing API.

but I don't think that's exhaustive and certainly not authoritative.

Anyways, just make sure you sync up with him so you don't end up wasting
any energy on something that will break some kind of requirement we have.

Thanks!
-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] if you look at binding-generation-as-service element...

[Todd Fiala via lldb-dev]

Sure.

On Wed, Nov 18, 2015 at 1:09 PM, Zachary Turner  wrote:

> Can I send you a sample set of bindings generated by 3.x as a fail-fast
> test?  Try them out, see if anything breaks, etc?
>
> On Wed, Nov 18, 2015 at 1:07 PM Todd Fiala  wrote:
>
>> Hey Zachary,
>>
>> Please make sure to ping Greg Clayton, particularly if it does anything
>> that would make the binding no longer be C-linkage-compatible.  He can tell
>> you all the bits we are counting on.  I think the keys are:
>>
>> * we don't ever have there be vtables.
>> * we never remove API surface area.
>> * we don't change signatures of existing API.
>>
>> but I don't think that's exhaustive and certainly not authoritative.
>>
>> Anyways, just make sure you sync up with him so you don't end up wasting
>> any energy on something that will break some kind of requirement we have.
>>
>> Thanks!
>> --
>> -Todd
>>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] bindings as service idea

[Todd Fiala via lldb-dev]

Hey Zachary,

I think the time pressure has gotten the better of me, so I want to
apologize for getting snippy about the static bindings of late.  I am
confident we will get to a good solution for removing that dependency, but
I can certainly wait for a solution (using an alternate approach in our
branch) until we arrive at something more palatable to everyone.

Regarding the bindings as service idea:

How quickly do you think you could flesh out the bindings as a service
idea?  With a relatively strong dislike for the static approach I'm taking,
I can back off that and just use my current code here in a downstream
branch for now.  Ultimately I want to remove the requirement for swig, but
I can probably achieve that without doing it in upstream if we're going to
have some solution there at some point ideally sooner than later.

Also - I think you were going to send me a swig 3.x binding to try out (I'd
need the LLDBWrapPythoh.cpp and the lldb.py, and you'd just need to let me
know if it still needs to be post-processed or it would need to be done).
Can we shoot for trying that out maybe tomorrow?

Thanks!
-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] if you look at binding-generation-as-service element...

[Todd Fiala via lldb-dev]

Ah good got it.  Thanks, somehow didn't see this earlier.  Thanks!

On Wed, Nov 18, 2015 at 1:27 PM, Zachary Turner  wrote:

> Google Drive is being bad right now.  Hopefully this works better :-/
>
>
>
> On Wed, Nov 18, 2015 at 1:20 PM Zachary Turner  wrote:
>
>> Err, links aren't showing up for me.  Gonna try this again.
>>
>> lldb.py   -
>> https://drive.google.com/file/d/0Bzpf8QJC6OGOaEVaNF9GY0tPXzg/view?usp=sharing
>> LLDBWrapPython.cpp-
>> https://drive.google.com/file/d/0Bzpf8QJC6OGOaEVaNF9GY0tPXzg/view?usp=sharing
>>
>>
>> On Wed, Nov 18, 2015 at 1:19 PM Zachary Turner 
>> wrote:
>>
>>> LLDBWrapPython.cpp
>>> lldb.py
>>>
>>> Double check the top of lldb.py to make sure swig_version = (3,0,x)
>>>
>>>
>>> On Wed, Nov 18, 2015 at 1:14 PM Todd Fiala  wrote:
>>>
 Sure.

 On Wed, Nov 18, 2015 at 1:09 PM, Zachary Turner 
 wrote:

> Can I send you a sample set of bindings generated by 3.x as a
> fail-fast test?  Try them out, see if anything breaks, etc?
>
> On Wed, Nov 18, 2015 at 1:07 PM Todd Fiala 
> wrote:
>
>> Hey Zachary,
>>
>> Please make sure to ping Greg Clayton, particularly if it does
>> anything that would make the binding no longer be C-linkage-compatible.  
>> He
>> can tell you all the bits we are counting on.  I think the keys are:
>>
>> * we don't ever have there be vtables.
>> * we never remove API surface area.
>> * we don't change signatures of existing API.
>>
>> but I don't think that's exhaustive and certainly not authoritative.
>>
>> Anyways, just make sure you sync up with him so you don't end up
>> wasting any energy on something that will break some kind of requirement 
>> we
>> have.
>>
>> Thanks!
>> --
>> -Todd
>>
>


 --
 -Todd

>>>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] bindings as service idea

[Todd Fiala via lldb-dev]

On Wed, Nov 18, 2015 at 10:06 PM, Todd Fiala  wrote:

> Hey Zachary,
>
> I think the time pressure has gotten the better of me, so I want to
> apologize for getting snippy about the static bindings of late.  I am
> confident we will get to a good solution for removing that dependency, but
> I can certainly wait for a solution (using an alternate approach in our
> branch) until we arrive at something more palatable to everyone.
>
> Regarding the bindings as service idea:
>
> How quickly do you think you could flesh out the bindings as a service
> idea?  With a relatively strong dislike for the static approach I'm taking,
> I can back off that and just use my current code here in a downstream
> branch for now.  Ultimately I want to remove the requirement for swig, but
> I can probably achieve that without doing it in upstream if we're going to
> have some solution there at some point ideally sooner than later.
>
> Also - I think you were going to send me a swig 3.x binding to try out
> (I'd need the LLDBWrapPythoh.cpp and the lldb.py, and you'd just need to
> let me know if it still needs to be post-processed or it would need to be
> done).  Can we shoot for trying that out maybe tomorrow?
>
>
Hey I got these, Zachary.  They just didn't go in my inbox.


> Thanks!
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] bindings as service idea

[Todd Fiala via lldb-dev]

For the benefit of continuity in conversation, here is what you had to say
about it before:

> One possibility (which I mentioned to you offline, but I'll put it here for
others to see) is that we make a swig bot which is hosted in the cloud much
like our public build bots.  We provide a Python script that can be run on
your machine, which sends requests over to the swig bot to run swig and
send back the results.  Availability of the service would be governed by
the SLA of Google Compute Engine, viewable
here:https://cloud.google.com/compute/sla?hl=en

> If we do something like this, it would allow us to raise the SWIG version
in the upstream, and at that point I can see some benefit in checking the
bindings in.  Short of that, I still dont' see the value proposition in
checking bindings in to the repo.  [bits deleted]

> If it means we can get off of SWIG 1.x in the upstream, I will do the work
to make remote swig generation service and get it up and running.


I'd like feedback from others on this.  Is this something we want to
consider doing?

>From my perspective, this seems reasonable to look into doing if we:

(a) have the service code available, and

(b) if we so choose, we can readily have the script hit another server
(so that a consumer can have the entire setup on an internal network),
and

(c: option 1) be able to fall back to generate with swig locally as we
do now in the event that we can't hit the server

(c: option 2) rather than fall back to swig generation, use swig
generation as primary (as it is now) but, if a swig is not found, then
do the get-bindings-as-a-service flow.

This does open up multiple ways to do something, but I think we need
to avoid a failure mode that says "Oops, you can't hit the network.
Sorry, no lldb build for you."


Reasoning:

For (a): just so we all know what we're using.

For (b): I can envision production build flows that will not want to
be hitting a third-party server.  We shouldn't require that.

For (c): we don't want to prevent building in scenarios that can't hit
a network during the build.


-Todd


On Wed, Nov 18, 2015 at 10:58 PM, Todd Fiala  wrote:

>
>
> On Wed, Nov 18, 2015 at 10:06 PM, Todd Fiala  wrote:
>
>> Hey Zachary,
>>
>> I think the time pressure has gotten the better of me, so I want to
>> apologize for getting snippy about the static bindings of late.  I am
>> confident we will get to a good solution for removing that dependency, but
>> I can certainly wait for a solution (using an alternate approach in our
>> branch) until we arrive at something more palatable to everyone.
>>
>> Regarding the bindings as service idea:
>>
>> How quickly do you think you could flesh out the bindings as a service
>> idea?  With a relatively strong dislike for the static approach I'm taking,
>> I can back off that and just use my current code here in a downstream
>> branch for now.  Ultimately I want to remove the requirement for swig, but
>> I can probably achieve that without doing it in upstream if we're going to
>> have some solution there at some point ideally sooner than later.
>>
>> Also - I think you were going to send me a swig 3.x binding to try out
>> (I'd need the LLDBWrapPythoh.cpp and the lldb.py, and you'd just need to
>> let me know if it still needs to be post-processed or it would need to be
>> done).  Can we shoot for trying that out maybe tomorrow?
>>
>>
> Hey I got these, Zachary.  They just didn't go in my inbox.
>
>
>> Thanks!
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Auditing dotest's command line options

[Todd Fiala via lldb-dev]

We're reviewing these.  I've added the ones that I use or need for
infrastructure.  I've pinged everyone else internally and we'll be sure to
update any others over today and tomorrow.

-Todd

On Wed, Nov 18, 2015 at 2:32 PM, Zachary Turner via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> I would like to do a complete audit of dotest's command line options, find
> out who's using what, and then potentially delete anything that isn't being
> used.  There's a mess of command line options in use, to the point that
> it's often hard to find free letters to use for new options.
>
> I created this spreadsheet with a complete list of command line options,
> their descriptions, and a place for people to enter what options they're
> using or do not want to be deleted.
>
>
> https://docs.google.com/spreadsheets/d/1wkxAY7l0_cJOHhhsSlh3aKKlQShlX1D7X1Dn8kpqxy4/edit?usp=sharing
>
> If someone has already written YES in the box that indicates they need the
> option, please don't overwrite it.  If you write YES in a box, please
> provide at least a small rationale for why this option is useful to you.
> Feel free to add additional rationale if someone has already added some
> rationale.
>
> I'm going to have a couple days in mid-December and do this cleanup, so
> I'd like to get a solid picture of what options are not needed before
> then.  After people have had some time to look over this, I'll go through
> the results and decide what to do with each one, and then send out another
> email with a proposed action column for each command line option.
>
> Please do take the time to have a look at this, because any option that
> doesn't have a YES in it after a couple of weeks I'm going to assume is a
> candidate for deletion.
>
>
>
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] bindings as service idea

[Todd Fiala via lldb-dev]

On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner  wrote:

> Just to re-iterate, if we use the bindings as a service, then I envision
> checking the bindings in.  This addresses a lot of the potential pitfalls
> you point out, such as the "oops, you can't hit the network, no build for
> you" and the issue of production build flows not wanting to hit a third
> party server, etc.
>
> So if we do that, then I don't think falling back to local generation will
> be an issue (or important) in practice.  i.e. it won't matter if you can't
> hit the network.  The reason I say this is that if you can't hit the
> network you can't check in code either.  So, sure, there might be a short
> window where you can't do a local build , but that would only affect you if
> you were actively modifying a swig interface file AND you were actively
> without a network connection.  The service claims 99.95% uptime, and it's
> safe to say we are looking at significantly less than 100% usage of the
> server (given checked in bindings), so I think we're looking at once a year
> -- if that -- that anyone anywhere has an issue with being able to access
> the service.
>
>
That seems fine.


> And, as you said, the option can be provided to change the host that the
> service runs on, so someone could run one internally.
>
> But do note, that if the goal here is to get the SWIG version bumped in
> the upstream, then we will probably take advantage of some of these new
> SWIG features, which may not work in earlier versions of SWIG.  So you
> should consider how useful it will be to be able to run this server
> internally, because if you can't run a new version of swig locally, then
> can you run it internally anywhere?  I don't know, I'll leave that for you
> to figure out.
>
>
That also seems fine.  And yes, we can work it out on our end.

We'd need to make sure that developer flows would pick up the need to
generate the bindings again if binding surface area changed, but that is no
different than now.


> Either way, it will definitely have the ability to use a different host,
> because that's the easiest way to debug theclient and server (i.e. run them
> on the same machine with 127.0.0.1)
>
>
Yep, sounds right.


> On Thu, Nov 19, 2015 at 8:00 AM Todd Fiala  wrote:
>
>> For the benefit of continuity in conversation, here is what you had to
>> say about it before:
>>
>> > One possibility (which I mentioned to you offline, but I'll put it here for
>> others to see) is that we make a swig bot which is hosted in the cloud much
>> like our public build bots.  We provide a Python script that can be run on
>> your machine, which sends requests over to the swig bot to run swig and
>> send back the results.  Availability of the service would be governed by
>> the SLA of Google Compute Engine, viewable 
>> here:https://cloud.google.com/compute/sla?hl=en
>>
>> > If we do something like this, it would allow us to raise the SWIG version
>> in the upstream, and at that point I can see some benefit in checking the
>> bindings in.  Short of that, I still dont' see the value proposition in
>> checking bindings in to the repo.  [bits deleted]
>>
>> > If it means we can get off of SWIG 1.x in the upstream, I will do the work
>> to make remote swig generation service and get it up and running.
>>
>>
>> I'd like feedback from others on this.  Is this something we want to 
>> consider doing?
>>
>> From my perspective, this seems reasonable to look into doing if we:
>>
>> (a) have the service code available, and
>>
>> (b) if we so choose, we can readily have the script hit another server (so 
>> that a consumer can have the entire setup on an internal network), and
>>
>> (c: option 1) be able to fall back to generate with swig locally as we do 
>> now in the event that we can't hit the server
>>
>> (c: option 2) rather than fall back to swig generation, use swig generation 
>> as primary (as it is now) but, if a swig is not found, then do the 
>> get-bindings-as-a-service flow.
>>
>> This does open up multiple ways to do something, but I think we need to 
>> avoid a failure mode that says "Oops, you can't hit the network.  Sorry, no 
>> lldb build for you."
>>
>>
>> Reasoning:
>>
>> For (a): just so we all know what we're using.
>>
>> For (b): I can envision production build flows that will not want to be 
>> hitting a third-party server.  We shouldn't require that.
>>
>> For (c): we don't want to prevent building in scenarios that can't hit a 
>> network during the build.
>>
>>
>> -Todd
>>
>>
>> On Wed, Nov 18, 2015 at 10:58 PM, Todd Fiala 
>> wrote:
>>
>>>
>>>
>>> On Wed, Nov 18, 2015 at 10:06 PM, Todd Fiala 
>>> wrote:
>>>
 Hey Zachary,

 I think the time pressure has gotten the better of me, so I want to
 apologize for getting snippy about the static bindings of late.  I am
 confident we will get to a good solution for removing that dependency, but
 I can certainly wait for a solution (using an alternate approach in our
 branch)

Re: [lldb-dev] bindings as service idea

[Todd Fiala via lldb-dev]

I'm out next week, but I can help if needed after that.

Related to all this, you have mentioned a few times that there are newer
swig features you want to use.

Can you enumerate the features not present in 1.x but present in 3.x that
you want to take advantage of, and what benefits they will bring us?  (I'm
not referring to bug fixes in bindings, but actual features that bring
something new that we didn't have before).

Thanks!

-Todd

On Thu, Nov 19, 2015 at 10:14 AM, Zachary Turner  wrote:

> I wasn't planning on working on this immediately, but given the outcome of
> the recent static bindings work, I can re-prioritize.  I don't know how
> long it will take, because honestly writing this kind of thing in Python is
> new to me.. to make an understatement.  But I'll get it done.  Give me
> until mid next week and I'll post an update.
>
> On Thu, Nov 19, 2015 at 10:12 AM Todd Fiala  wrote:
>
>> On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner 
>> wrote:
>>
>>> Just to re-iterate, if we use the bindings as a service, then I envision
>>> checking the bindings in.  This addresses a lot of the potential pitfalls
>>> you point out, such as the "oops, you can't hit the network, no build for
>>> you" and the issue of production build flows not wanting to hit a third
>>> party server, etc.
>>>
>>> So if we do that, then I don't think falling back to local generation
>>> will be an issue (or important) in practice.  i.e. it won't matter if you
>>> can't hit the network.  The reason I say this is that if you can't hit the
>>> network you can't check in code either.  So, sure, there might be a short
>>> window where you can't do a local build , but that would only affect you if
>>> you were actively modifying a swig interface file AND you were actively
>>> without a network connection.  The service claims 99.95% uptime, and it's
>>> safe to say we are looking at significantly less than 100% usage of the
>>> server (given checked in bindings), so I think we're looking at once a year
>>> -- if that -- that anyone anywhere has an issue with being able to access
>>> the service.
>>>
>>>
>> That seems fine.
>>
>>
>>> And, as you said, the option can be provided to change the host that the
>>> service runs on, so someone could run one internally.
>>>
>>> But do note, that if the goal here is to get the SWIG version bumped in
>>> the upstream, then we will probably take advantage of some of these new
>>> SWIG features, which may not work in earlier versions of SWIG.  So you
>>> should consider how useful it will be to be able to run this server
>>> internally, because if you can't run a new version of swig locally, then
>>> can you run it internally anywhere?  I don't know, I'll leave that for you
>>> to figure out.
>>>
>>>
>> That also seems fine.  And yes, we can work it out on our end.
>>
>> We'd need to make sure that developer flows would pick up the need to
>> generate the bindings again if binding surface area changed, but that is no
>> different than now.
>>
>>
>>> Either way, it will definitely have the ability to use a different host,
>>> because that's the easiest way to debug theclient and server (i.e. run them
>>> on the same machine with 127.0.0.1)
>>>
>>>
>> Yep, sounds right.
>>
>>
>>> On Thu, Nov 19, 2015 at 8:00 AM Todd Fiala  wrote:
>>>
 For the benefit of continuity in conversation, here is what you had to
 say about it before:

 > One possibility (which I mentioned to you offline, but I'll put it here 
 > for
 others to see) is that we make a swig bot which is hosted in the cloud much
 like our public build bots.  We provide a Python script that can be run on
 your machine, which sends requests over to the swig bot to run swig and
 send back the results.  Availability of the service would be governed by
 the SLA of Google Compute Engine, viewable 
 here:https://cloud.google.com/compute/sla?hl=en

 > If we do something like this, it would allow us to raise the SWIG version
 in the upstream, and at that point I can see some benefit in checking the
 bindings in.  Short of that, I still dont' see the value proposition in
 checking bindings in to the repo.  [bits deleted]

 > If it means we can get off of SWIG 1.x in the upstream, I will do the 
 > work
 to make remote swig generation service and get it up and running.


 I'd like feedback from others on this.  Is this something we want to 
 consider doing?

 From my perspective, this seems reasonable to look into doing if we:

 (a) have the service code available, and

 (b) if we so choose, we can readily have the script hit another server (so 
 that a consumer can have the entire setup on an internal network), and

 (c: option 1) be able to fall back to generate with swig locally as we do 
 now in the event that we can't hit the server

 (c: option 2) rather than fall back to swig generation, use swig 
 genera

Re: [lldb-dev] bindings as service idea

[Todd Fiala via lldb-dev]

Some other points we need to consider on the bindings-as-service idea:

* The service should be exposed via secure connection (https/ssl/etc.)
 This might already be guaranteed on the Google end by virtue of the
endpoint, but we'll want to make sure we can have a secure connection.
 (This will be a non-issue for standing up a custom server, but the
official one should have this taken care of).

* The method behind how/when the service is updated needs to be clear to
everyone.  This is both a transparency item and affects how changes to the
service code get to the online service.

We don't have to work those out immediately, but they are things we need to
consider.

-Todd


On Thu, Nov 19, 2015 at 10:17 AM, Todd Fiala  wrote:

> I'm out next week, but I can help if needed after that.
>
> Related to all this, you have mentioned a few times that there are newer
> swig features you want to use.
>
> Can you enumerate the features not present in 1.x but present in 3.x that
> you want to take advantage of, and what benefits they will bring us?  (I'm
> not referring to bug fixes in bindings, but actual features that bring
> something new that we didn't have before).
>
> Thanks!
>
> -Todd
>
> On Thu, Nov 19, 2015 at 10:14 AM, Zachary Turner 
> wrote:
>
>> I wasn't planning on working on this immediately, but given the outcome
>> of the recent static bindings work, I can re-prioritize.  I don't know how
>> long it will take, because honestly writing this kind of thing in Python is
>> new to me.. to make an understatement.  But I'll get it done.  Give me
>> until mid next week and I'll post an update.
>>
>> On Thu, Nov 19, 2015 at 10:12 AM Todd Fiala  wrote:
>>
>>> On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner 
>>> wrote:
>>>
 Just to re-iterate, if we use the bindings as a service, then I
 envision checking the bindings in.  This addresses a lot of the potential
 pitfalls you point out, such as the "oops, you can't hit the network, no
 build for you" and the issue of production build flows not wanting to hit a
 third party server, etc.

 So if we do that, then I don't think falling back to local generation
 will be an issue (or important) in practice.  i.e. it won't matter if you
 can't hit the network.  The reason I say this is that if you can't hit the
 network you can't check in code either.  So, sure, there might be a short
 window where you can't do a local build , but that would only affect you if
 you were actively modifying a swig interface file AND you were actively
 without a network connection.  The service claims 99.95% uptime, and it's
 safe to say we are looking at significantly less than 100% usage of the
 server (given checked in bindings), so I think we're looking at once a year
 -- if that -- that anyone anywhere has an issue with being able to access
 the service.


>>> That seems fine.
>>>
>>>
 And, as you said, the option can be provided to change the host that
 the service runs on, so someone could run one internally.

 But do note, that if the goal here is to get the SWIG version bumped in
 the upstream, then we will probably take advantage of some of these new
 SWIG features, which may not work in earlier versions of SWIG.  So you
 should consider how useful it will be to be able to run this server
 internally, because if you can't run a new version of swig locally, then
 can you run it internally anywhere?  I don't know, I'll leave that for you
 to figure out.


>>> That also seems fine.  And yes, we can work it out on our end.
>>>
>>> We'd need to make sure that developer flows would pick up the need to
>>> generate the bindings again if binding surface area changed, but that is no
>>> different than now.
>>>
>>>
 Either way, it will definitely have the ability to use a different
 host, because that's the easiest way to debug theclient and server (i.e.
 run them on the same machine with 127.0.0.1)


>>> Yep, sounds right.
>>>
>>>
 On Thu, Nov 19, 2015 at 8:00 AM Todd Fiala 
 wrote:

> For the benefit of continuity in conversation, here is what you had to
> say about it before:
>
> > One possibility (which I mentioned to you offline, but I'll put it here 
> > for
> others to see) is that we make a swig bot which is hosted in the cloud 
> much
> like our public build bots.  We provide a Python script that can be run on
> your machine, which sends requests over to the swig bot to run swig and
> send back the results.  Availability of the service would be governed by
> the SLA of Google Compute Engine, viewable 
> here:https://cloud.google.com/compute/sla?hl=en
>
> > If we do something like this, it would allow us to raise the SWIG 
> > version
> in the upstream, and at that point I can see some benefit in checking the
> bindings in.  Short of that, I still dont' see the v

Re: [lldb-dev] bindings as service idea

[Todd Fiala via lldb-dev]

>> If so, does this mean everyone needs to generate a cert locally?

Generally not - as long as the server is dishing out something over https,
the server will be signed with a certificate that is going to be in the
local OS's set of trusted root certificates (particularly if this is
provided by Google).  It is true that a die-hard self OS builder can
totally build their own trusted set of roots, but that's going beyond.  (It
is possible to buy/procure very cheap certificates that come from
Certificate Authorities that are generally not popular or well known enough
to be in the stock set of Microsoft/OS X/Ubuntu trusted roots, but this is
totally avoidable.)



On Thu, Nov 19, 2015 at 11:48 AM, Jim Ingham  wrote:

> The server is sending back code.  I'd want to know I can trust whoever is
> sending me back code that I plan to build and run locally.
>
> Jim
>
> > On Nov 19, 2015, at 11:40 AM, Zachary Turner via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
> >
> >
> >
> > On Thu, Nov 19, 2015 at 10:28 AM Todd Fiala 
> wrote:
> > Some other points we need to consider on the bindings-as-service idea:
> >
> > * The service should be exposed via secure connection (https/ssl/etc.)
> This might already be guaranteed on the Google end by virtue of the
> endpoint, but we'll want to make sure we can have a secure connection.
> (This will be a non-issue for standing up a custom server, but the official
> one should have this taken care of).
> >
> > If the only thing we're sending from client -> server is packaged up
> source code which is already available on the open source repository, and
> the server doesn't require authentication, is this necessary?
> >
> > If so, does this mean everyone needs to generate a cert locally?
> > ___
> > lldb-dev mailing list
> > lldb-dev@lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Auditing dotest's command line options

[Todd Fiala via lldb-dev]

Yeah that would be awesome, and also would be a lot of work.  Good luck!


On Fri, Nov 20, 2015 at 10:37 AM, Zachary Turner via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> Seems reasonable.  I will make a best effort to get as many of them as I
> can.
>
> On Fri, Nov 20, 2015 at 10:34 AM Greg Clayton  wrote:
>
>> Zach, I would also like to get rid of all global variables in the process
>> of this change. The history goes like this: a long time ago someone wrote
>> the initial dotest.py and parsed the options manually and stored results in
>> global variables. Later, someone converted the options over to use a python
>> library to parse the options, but we mostly copied the options from the
>> options dictionary over into the globals and still use the globals all over
>> the code. It would be great if we had at most one global variable that is
>> something like "g_options" and anyone that was using any global variables
>> will switch over to use the "g_options." instead. Then we don't have to
>> make copies and we can let the g_options contain all settings that are
>> required.
>>
>> > On Nov 18, 2015, at 2:32 PM, Zachary Turner via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>> >
>> > I would like to do a complete audit of dotest's command line options,
>> find out who's using what, and then potentially delete anything that isn't
>> being used.  There's a mess of command line options in use, to the point
>> that it's often hard to find free letters to use for new options.
>> >
>> > I created this spreadsheet with a complete list of command line
>> options, their descriptions, and a place for people to enter what options
>> they're using or do not want to be deleted.
>> >
>> >
>> https://docs.google.com/spreadsheets/d/1wkxAY7l0_cJOHhhsSlh3aKKlQShlX1D7X1Dn8kpqxy4/edit?usp=sharing
>> >
>> > If someone has already written YES in the box that indicates they need
>> the option, please don't overwrite it.  If you write YES in a box, please
>> provide at least a small rationale for why this option is useful to you.
>> Feel free to add additional rationale if someone has already added some
>> rationale.
>> >
>> > I'm going to have a couple days in mid-December and do this cleanup, so
>> I'd like to get a solid picture of what options are not needed before
>> then.  After people have had some time to look over this, I'll go through
>> the results and decide what to do with each one, and then send out another
>> email with a proposed action column for each command line option.
>> >
>> > Please do take the time to have a look at this, because any option that
>> doesn't have a YES in it after a couple of weeks I'm going to assume is a
>> candidate for deletion.
>> >
>> >
>> > ___
>> > lldb-dev mailing list
>> > lldb-dev@lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] lldb-server/debugserver tests and debuginfo build type

[Todd Fiala via lldb-dev]

Hi all,

I think the vast majority of those likely aren't concerned with debug info
format.  Most of us are off next week, but when we get back I'll look into
getting those to run without debuginfo variants except where needed.

-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] serialized, low-load test pass in parallel test runner

[Todd Fiala via lldb-dev]

Hi all,

On OS X (and frankly on Linux sometimes as well, but predominently OS X),
we have tests that will sometimes fail when under significant load (e.g.
running the concurrent test suite, exacerbated if we crank up the number of
threads, but bad enough if we run at "number of concurrent workers ==
number of logical cores").

I'm planning on adding a serialized, one-worker-only phase to the end of
the concurrent test run, where the load is much lighter since only one
worker will be processing at that phase.  Then, for tests that fail in the
first run, I'd re-run them in the serialized, single worker test run
phase.  On the OS X side, this would eliminate a significant number of test
failures that are both hard to diagnose and hard to justify spending
significant amounts of time on in the short run.  (There's a whole other
conversation to have about fixing them for real, i.e. working through all
the race and/or faulty test logic assumptions that are stressed to the max
under heavier load, but practically speaking, there are so many of them
that this is going to be impractical to address in the short/mid term.).

My question to all of you is if we'd want this functionality in top of tree
llvm.org lldb.  If not, I'll do it in one of our branches.  If so, we can
talk about possibly having a category or some other mechanism if we want to
mark those tests that are eligible to be run in the follow-up serialized,
low-load pass.  Up front I was just going to allow any test to fall into
that bucket.  The one benefit to having it in top of tree llvm.org is that,
once I enable test reporting on the green dragon public llvm.org OS X LLDB
builder, that builder will be able to take advantage of this, and will most
certainly tag fewer changes as breaking a test (in the case where the test
is just one of the many that fail under high load).

Let me know your thoughts either way.

Thanks!
-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] serialized, low-load test pass in parallel test runner

[Todd Fiala via lldb-dev]

Note this is similar to the flakey test mechanism, with the primary
difference being that the re-run is done in a minimal CPU load environment
rather than wherever the failure first occurred.  The existing flakey test
rerun logic is not helpful for the high-load-induced failures that I'm
looking to handle.

On Fri, Nov 27, 2015 at 10:56 AM, Todd Fiala  wrote:

> Hi all,
>
> On OS X (and frankly on Linux sometimes as well, but predominently OS X),
> we have tests that will sometimes fail when under significant load (e.g.
> running the concurrent test suite, exacerbated if we crank up the number of
> threads, but bad enough if we run at "number of concurrent workers ==
> number of logical cores").
>
> I'm planning on adding a serialized, one-worker-only phase to the end of
> the concurrent test run, where the load is much lighter since only one
> worker will be processing at that phase.  Then, for tests that fail in the
> first run, I'd re-run them in the serialized, single worker test run
> phase.  On the OS X side, this would eliminate a significant number of test
> failures that are both hard to diagnose and hard to justify spending
> significant amounts of time on in the short run.  (There's a whole other
> conversation to have about fixing them for real, i.e. working through all
> the race and/or faulty test logic assumptions that are stressed to the max
> under heavier load, but practically speaking, there are so many of them
> that this is going to be impractical to address in the short/mid term.).
>
> My question to all of you is if we'd want this functionality in top of
> tree llvm.org lldb.  If not, I'll do it in one of our branches.  If so,
> we can talk about possibly having a category or some other mechanism if we
> want to mark those tests that are eligible to be run in the follow-up
> serialized, low-load pass.  Up front I was just going to allow any test to
> fall into that bucket.  The one benefit to having it in top of tree
> llvm.org is that, once I enable test reporting on the green dragon public
> llvm.org OS X LLDB builder, that builder will be able to take advantage
> of this, and will most certainly tag fewer changes as breaking a test (in
> the case where the test is just one of the many that fail under high load).
>
> Let me know your thoughts either way.
>
> Thanks!
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] serialized, low-load test pass in parallel test runner

[Todd Fiala via lldb-dev]

Thanks, Reid and Pavel!

On Fri, Nov 27, 2015 at 1:21 PM, Pavel Labath  wrote:

> I think it sounds like something that would be useful in general. I'd
> even go a step further and say that we can replace the current flakey
> test mechanism with your proposed solution.


Okay, I like that idea.


> If we do that (remove the
> current flakey mechanism when this is in place), then I think it would
> be super-great as we don't increase the number of moving parts and we
> can think of this as just an upgrade of an inferior solution (the
> current flakey mechanism has always felt like a hack to me) with a
> better one.
>

Sounds great.


>
> If you want to automatically re-run tests, then we can have a mode
> that does that, but I'd like to have it off by default. I have several
> reasons for this:
> - you get to feel bad for having to add flakey decorators, which may
> encourage you to fix things
> - if you make a change (hopefully only locally :) ) which breaks a lot
> of tests, you want this to fail quickly instead of waiting for reruns
> - if you make a change that makes things flakey (!), you may not
> actually notice it because of the reruns
>
>
I'm fine with that.  The only caveat I see is that it appears we have a
largish number of potential-failing tests under high load, so we may end up
(at least on OS X) marking quite a few up this way.  But, that's also
helpful since it lets us see which of the tests really are failing under
load.  So this is all likely for the best, with a small ramp-up time to
while we "discover" which tests are hitting this.


> cheers,
> pl
>

Thanks!


>
>
>
>
>
> On 27 November 2015 at 18:58, Todd Fiala via lldb-dev
>  wrote:
> > Note this is similar to the flakey test mechanism, with the primary
> > difference being that the re-run is done in a minimal CPU load
> environment
> > rather than wherever the failure first occurred.  The existing flakey
> test
> > rerun logic is not helpful for the high-load-induced failures that I'm
> > looking to handle.
> >
> > On Fri, Nov 27, 2015 at 10:56 AM, Todd Fiala 
> wrote:
> >>
> >> Hi all,
> >>
> >> On OS X (and frankly on Linux sometimes as well, but predominently OS
> X),
> >> we have tests that will sometimes fail when under significant load (e.g.
> >> running the concurrent test suite, exacerbated if we crank up the
> number of
> >> threads, but bad enough if we run at "number of concurrent workers ==
> number
> >> of logical cores").
> >>
> >> I'm planning on adding a serialized, one-worker-only phase to the end of
> >> the concurrent test run, where the load is much lighter since only one
> >> worker will be processing at that phase.  Then, for tests that fail in
> the
> >> first run, I'd re-run them in the serialized, single worker test run
> phase.
> >> On the OS X side, this would eliminate a significant number of test
> failures
> >> that are both hard to diagnose and hard to justify spending significant
> >> amounts of time on in the short run.  (There's a whole other
> conversation to
> >> have about fixing them for real, i.e. working through all the race
> and/or
> >> faulty test logic assumptions that are stressed to the max under heavier
> >> load, but practically speaking, there are so many of them that this is
> going
> >> to be impractical to address in the short/mid term.).
> >>
> >> My question to all of you is if we'd want this functionality in top of
> >> tree llvm.org lldb.  If not, I'll do it in one of our branches.  If
> so, we
> >> can talk about possibly having a category or some other mechanism if we
> want
> >> to mark those tests that are eligible to be run in the follow-up
> serialized,
> >> low-load pass.  Up front I was just going to allow any test to fall into
> >> that bucket.  The one benefit to having it in top of tree llvm.org is
> that,
> >> once I enable test reporting on the green dragon public llvm.org OS X
> LLDB
> >> builder, that builder will be able to take advantage of this, and will
> most
> >> certainly tag fewer changes as breaking a test (in the case where the
> test
> >> is just one of the many that fail under high load).
> >>
> >> Let me know your thoughts either way.
> >>
> >> Thanks!
> >> --
> >> -Todd
> >
> >
> >
> >
> > --
> > -Todd
> >
> > ___
> > lldb-dev mailing list
> > lldb-dev@lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> >
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] serialized, low-load test pass in parallel test runner

[Todd Fiala via lldb-dev]

On Fri, Nov 27, 2015 at 12:34 PM, Reid Kleckner  wrote:

> Chromium's test framework uses the same technique. It has the potential to
> really slow things down if you have a lot of failing tests. You might want
> some kind of threshold for giving up, I.e. here's 50 failures, I'll stop
> running the rest so devs see
>
results sooner.
>
> Otherwise, yeah, this seems reasonable for lldb.
>
>
Thanks, Reid!  The max failure threshold seems like a good idea if/when I
put in an auto re-run-under-low-load mechanism.

I think I will start with that (the auto mode + threshold) to get it up and
running, then add the test markers and the required opt-in mode that we'll
default with per Pavel's comments.

-Todd

>

> Sent from phone
> On Nov 27, 2015 10:57 AM, "Todd Fiala via lldb-dev" <
> lldb-dev@lists.llvm.org> wrote:
>
>> Hi all,
>>
>> On OS X (and frankly on Linux sometimes as well, but predominently OS X),
>> we have tests that will sometimes fail when under significant load (e.g.
>> running the concurrent test suite, exacerbated if we crank up the number of
>> threads, but bad enough if we run at "number of concurrent workers ==
>> number of logical cores").
>>
>> I'm planning on adding a serialized, one-worker-only phase to the end of
>> the concurrent test run, where the load is much lighter since only one
>> worker will be processing at that phase.  Then, for tests that fail in the
>> first run, I'd re-run them in the serialized, single worker test run
>> phase.  On the OS X side, this would eliminate a significant number of test
>> failures that are both hard to diagnose and hard to justify spending
>> significant amounts of time on in the short run.  (There's a whole other
>> conversation to have about fixing them for real, i.e. working through all
>> the race and/or faulty test logic assumptions that are stressed to the max
>> under heavier load, but practically speaking, there are so many of them
>> that this is going to be impractical to address in the short/mid term.).
>>
>> My question to all of you is if we'd want this functionality in top of
>> tree llvm.org lldb.  If not, I'll do it in one of our branches.  If so,
>> we can talk about possibly having a category or some other mechanism if we
>> want to mark those tests that are eligible to be run in the follow-up
>> serialized, low-load pass.  Up front I was just going to allow any test to
>> fall into that bucket.  The one benefit to having it in top of tree
>> llvm.org is that, once I enable test reporting on the green dragon
>> public llvm.org OS X LLDB builder, that builder will be able to take
>> advantage of this, and will most certainly tag fewer changes as breaking a
>> test (in the case where the test is just one of the many that fail under
>> high load).
>>
>> Let me know your thoughts either way.
>>
>> Thanks!
>> --
>> -Todd
>>
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Exclusively build and install LLDB?

[Todd Fiala via lldb-dev]

Sorry for being late the the party here.

Sean Callanan and some of the other members can comment more on this, but
LLDB's expression parser for C/C++ is going to need access to the clang
include headers, so somehow lldb has to be able to find them.  Out of tree
llvm/clang usage is certainly possible as others have pointed out.  Using
that as the one way it is done, though, is likely to lead to pain.  Parts
of lldb's source will adjust as needed when the API surface area of LLVM or
clang changes.  It may not be happening quite as frequently as it had say 2
or 3 years ago, but it definitely happens.  So my expectation would be that
if you decouple lldb from llvm/clang (i.e. let them drift), sooner or later
you will get bitten by that.  Particularly when things like clang modules
and whatnot come along and actually require different logic on the lldb
side to deal with content generated on the clang/llvm side.  Once
expression evaluation is potentially compromised (due to the drift), I
suspect the lldb experience will degrade significantly.

On Sun, Nov 29, 2015 at 9:28 PM, Kamil Rytarowski via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> On 27.11.2015 00:57, Kamil Rytarowski via lldb-dev wrote:
> > On 11/23/15 10:28, Pavel Labath wrote:
> >> I believe that for purposes of building distribution packages you
> >>  should use the out-of-tree mode of building lldb. This means,
> >> you build llvm and clang separately, and then point your LLDB
> >> build to their installation path with LLDB_PATH_TO_LLVM_BUILD and
> >>  LLDB_PATH_TO_CLANG_BUILD variables. This way you can avoid
> >> building llvm/clang twice, you can have a separate package for
> >> each logical component of llvm and you can make lldb optional for
> >> your users (e.g. have only clang installed normally, if user
> >> chooses to install lldb, it will automatically pull in clang if
> >> needed). In this mode "make install" should install only the lldb
> >> components, which should be correctly linked to the
> >> already-installed llvm libraries.
> >
> >> That said, I can't guarantee that this mode will work for you
> >> out-of-the-box. We occasionally get patches to fix it up, but I
> >> don't know anyone who is using it extensively. However, I think
> >> this would be the best way forward for you and I'm prepared yo
> >> help you out if you choose to go that way.
> >
> >> What do you think about that?
> >
> >> pl
> >
> >
> > Thank you for your note on this mode. I was trying to prototype a
> > set of packages with: sources of llvm and clang, build dirs of llvm
> > and clang and installations of llvm and clang.
> >
> > Badly this approach doesn't work with pkgsrc, as this framework
> > contains various checks against using sources, headers, executables
> > or other files out of the build tree. Packaging sources and build
> > tree triggers errors with moving invalid files into ${DESTDIR}.
> > Everything is wolkaroundable, but I think it's not the correct way
> > of handling it.
> >
> > I've checked that libcxx, cfe and compiler-rt ship with mechanism
> > to build against preinstalled LLVM. I will try them out and I'm
> > going to prepare new pkgsrc packages using this approach. Then I
> > will try to research doing the same with LLDB, exporting needed
> > libraries and headers for the compiler withing llvm and lldb.
>
> For the cross reference.
>
> A patch allowing to build (tested on NetBSD) out of sources pushed to
> review: reviews.llvm.org/D15067
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2
>
> iQIcBAEBCAAGBQJWW96JAAoJEEuzCOmwLnZsCLAP/0LrhGlzivOjtykjW3ywvXia
> wtZRLYsPwsNBJJERdOGVOJVPovnT02H+Bf1a4eDf0dJXbecklyfiupNfthlvFr9l
> PxCLZI4GQPLcP+jqWbhcFRdhzFeyEnLLy0Wjt1MNYG0s3m2u4jJM2ViNLA10/kwS
> XX82e2K7q5MUb51MMEJ09ufpYyGff7XmjVE78w1ekfNSRlKFMc0DNBsaIx4oKZfM
> G2IUtRNL59ad2pkw/xA3D2OPtoTk7+a2jjF8Z4nYY6kUSyBPUlCYrjyfavVCreR6
> 6Zo2E3lkkEb6PSIfb57RlMtxBIfmIBjv5w6OcFjSK6aYvffY9IMgyBJvGbdA+Ee7
> DlCFZax25eJPglEnfzAI2XOHOUQJtDwhb45N+XWshLfUax2e52KJvj9nq9J8pOse
> AC00qRQN+KTZsil43dlOfEn5m18mJ9o+CohK5eLMoTnS9QdtP8OEv72zGjOsSqrx
> vXDx01ziuQRCgsJ+niZXHgRLA65hxD5XgSGEBzr5prRLtU6q6V/HpYOsC46+pySc
> ibLrRWHnaeBVJknwz11iBo4gBZRk3lGhi5aTfu9+kcX6ylKSn2nn34+HGHr//FZi
> SrKcb3z7WikAR0c9cHBNOnwbro6o08j8zUE2l2S08risLRDDu01KBo3yFebjHz8D
> vQqFJNDkRLywQbXezcjB
> =7C7K
> -END PGP SIGNATURE-
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] static swig bindings and xcode workspace

[Todd Fiala via lldb-dev]

Hi Zachary,

On Mon, Nov 30, 2015 at 9:23 AM, Zachary Turner  wrote:

> Has the xcode build been changed to use static bindings yet?
>

It is only in our downstream branches.  I stripped out support in llvm.org
lldb per our other threads.


>   I got to thinking that maybe it would make sense to put them alongside
> the xcode workspace folders, just to emphasize that the static bindings
> were an artifact of how the xcode build works.
>

We could do that.  Internally we also use that for builds other than Xcode,
so whatever solution I use (which is currently what I had proposed earlier
but now have only in our branches) really needs to work for more than Xcode.


>   This way we just say "Xcode build uses static bindings" and "CMake build
> needs an installed swig", and this is enforced at the directory level.
>
>
That's a great compromise, I appreciate your thoughts on that.  Since I
need it for more than Xcode, right now the solution I have in our branch is
working okay.


> In order to do this you'd have to probably make a new toplevel folder to
> house both the lldb.xcodeproj and lldb.xcworkspace folders, but I think
> that would be useful for other reasons as well.  For example, I want to
> check in a visual studio python solution for the test suite at some point,
> and it would make sense if all of this additional stuff was in one place.
> So perhaps something like:
>
> lldb
> |__ contrib
> |__ xcode
> |__ LLDBWrapPython.cpp
> |__ lldb.py
> |__ lldb.xcodeproj
> |__ lldb.xcworkspace
> |__ msvc
>
>
That structure may make sense.  That could live in llvm.org.  Then for
other OSes where I want similar behavior, I could just keep those parts in
our branch.  Ultimately I'd end up with multiple copies of the wrapper (for
any OS we may build for internally), but I could symlink so that's not
really any kind of issue.

This might work.


> I have been thinking about this idea of a contrib folder for a while
> anyway, but wanted to have more reasons to make use of it before I brought
> it up.
>
> Good idea?  Bad idea?  Thoughts?
>

I could see that layout making sense.  If we did something like that, I
think I'd separate moving the lldb.xcodeproj and lldb.xcworkspace from the
creation of the contrib folder.  (i.e. I'd start with the wrapper part in
there, and have the others move there at lower priority as a scheduling
thing --- there's a bit of work to make the workspace/project change but
should be totally doable).

I think I like the idea since it reduces the number of merge issues I'd
have to deal with.

-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Linux core dump doesn't show listing when loaded

[Todd Fiala via lldb-dev]

Does our init file mechanism have the ability to do something conditionally
if it's a core file?  (i.e. do we already have a way to get Ted's desired
behavior via an inserted call to "thread backtrace all" that somehow gets
triggered by the init, but only when we're talking about a core file?)

Alternatively, Ted, you could have a wrapper script of some sort (think
lldb-core.{sh,bat} or something) that you call that sources an lldb
core-file-specific init file that sets up aliases and the like to start up
lldb how you want, maybe?

On Mon, Nov 30, 2015 at 9:32 AM, Greg Clayton via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> "thread list" should just list the threads and their stop reasons (no
> backtraces). If you want backtraces just do "thread backtrace all".
>
>
> On Nov 24, 2015, at 1:09 PM, Ted Woodward via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
> >
> > I’ve been working on an old rev that we’d released on; now I’m much
> closer to ToT as we move towards our next major Hexagon release.
> >
> > Core dumps on the old rev would print out a listing/disassembly for each
> thread in the core dump. Now it doesn’t.
> >
> > ToT does this, on x86 Linux:
> >
> > >bin/lldb ~/lldb_test/coredump/lincrash -c ~/lldb_test/coredump/lincore
> > (lldb) target create "/usr2/tedwood/lldb_test/coredump/lincrash" --core
> "/usr2/tedwood/lldb_test/coredump/lincore"
> > Core file '/usr2/tedwood/lldb_test/coredump/lincore' (x86_64) was loaded.
> > (lldb) thread list
> > Process 0 stopped
> > * thread #1: tid = 0, 0x00401190 lincrash`main + 16 at
> lincrash.c:5, name = 'lincrash', stop reason = signal SIGSEGV
> > (lldb)
> >
> > I can see the listing by going up and down the stack, but I’d like to
> see the listing on load. Is the no listing intended?
> >
> > Ted
> >
> > --
> > Qualcomm Innovation Center, Inc.
> > The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a
> Linux Foundation Collaborative Project
> >
> > ___
> > lldb-dev mailing list
> > lldb-dev@lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Exclusively build and install LLDB?

[Todd Fiala via lldb-dev]

Yes, that concept came out in the thread.  I just wanted to make sure there
wasn't also a desire to park on a version of llvm/clang, and if so, that
the path there is not pleasant and definitely not intended to be supported
on top of tree svn/trunk.

Thanks for clarifying!

-Todd

On Wed, Dec 2, 2015 at 8:34 AM, Pavel Labath  wrote:

> On 2 December 2015 at 16:19, Todd Fiala  wrote:
> > Sorry for being late the the party here.
> >
> > Sean Callanan and some of the other members can comment more on this, but
> > LLDB's expression parser for C/C++ is going to need access to the clang
> > include headers, so somehow lldb has to be able to find them.  Out of
> tree
> > llvm/clang usage is certainly possible as others have pointed out.  Using
> > that as the one way it is done, though, is likely to lead to pain.
> Parts of
> > lldb's source will adjust as needed when the API surface area of LLVM or
> > clang changes.  It may not be happening quite as frequently as it had
> say 2
> > or 3 years ago, but it definitely happens.  So my expectation would be
> that
> > if you decouple lldb from llvm/clang (i.e. let them drift), sooner or
> later
> > you will get bitten by that.  Particularly when things like clang modules
> > and whatnot come along and actually require different logic on the lldb
> side
> > to deal with content generated on the clang/llvm side.  Once expression
> > evaluation is potentially compromised (due to the drift), I suspect the
> lldb
> > experience will degrade significantly.
>
> I think you have misunderstood our intentions here.
>
> Kamil, correct me if I am wrong, but I don't think we are talking
> about building lldb against a different version of clang. What we want
> is just to be able to build and link lldb against an already-built
> clang (of the same version). This is quite useful when you (as a
> distribution maintainer) want to provide prebuilt packages. So, for
> example you can have a "clang" and an "lldb" package. Users wishing to
> install clang, just get the first one, while someone installing lldb
> will get the correct clang package pulled automatically. I believe the
> easiest way to build these packages is to use the standalone mode of
> lldb (which already exists, and some people use that).
>
> hope that makes sense,
> pl
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

Hi all,

I just put up an optional test results formatter that is a prototype of
what we may move towards for our default test summary results.  It went in
here:

r254530

and you can try it out with something like:

time test/dotest.py --executable `pwd`/build/Debug/lldb --results-formatter
lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
st
out

Let me know if this satisfies the basic needs of counts and whatnot.  It
counts test method runs rather than all the oddball "file, class, etc."
counts we had before.

It prints out the Details section when there are details, and keeps it nice
and clean when there are none.

It also mentions a bit about test reruns up top, but that won't come into
play until I get the multi-test-pass, single-worker/low-load mechanism in
place, which will depend on newer rerun count awareness support.

The change also cleans up places where the test event framework was using
string codes and replaces them with symbolic constants.

Let me know what you think.  I can tweak it as needed to address testbot
and other needs.  Once it looks reasonable, I'd like to move over to using
it by default in the parallel test runner rather than the legacy support.

Thanks!
-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala  wrote:

> Hi all,
>
> I just put up an optional test results formatter that is a prototype of
> what we may move towards for our default test summary results.  It went in
> here:
>
> r254530
>
> and you can try it out with something like:
>
> time test/dotest.py --executable `pwd`/build/Debug/lldb
> --results-formatter
> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
> st
> out
>
>
I cut and paste my line, but more than likely for most people you'd just
want this:

test/dotest.py --results-formatter
lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
stdout

The other stuff was specific to my setup.  That line assumes you run from
the lldb source dir root.


Let me know if this satisfies the basic needs of counts and whatnot.  It
> counts test method runs rather than all the oddball "file, class, etc."
> counts we had before.
>
> It prints out the Details section when there are details, and keeps it
> nice and clean when there are none.
>
> It also mentions a bit about test reruns up top, but that won't come into
> play until I get the multi-test-pass, single-worker/low-load mechanism in
> place, which will depend on newer rerun count awareness support.
>
> The change also cleans up places where the test event framework was using
> string codes and replaces them with symbolic constants.
>
> Let me know what you think.  I can tweak it as needed to address testbot
> and other needs.  Once it looks reasonable, I'd like to move over to using
> it by default in the parallel test runner rather than the legacy support.
>
> Thanks!
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

Also, all the text in the summary is fixed-width lined up nicely, which may
not show in the commit message description if you're using a variable-width
font.  On a terminal it looks nice.

On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala  wrote:

>
>
> On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala  wrote:
>
>> Hi all,
>>
>> I just put up an optional test results formatter that is a prototype of
>> what we may move towards for our default test summary results.  It went in
>> here:
>>
>> r254530
>>
>> and you can try it out with something like:
>>
>> time test/dotest.py --executable `pwd`/build/Debug/lldb
>> --results-formatter
>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>> st
>> out
>>
>>
> I cut and paste my line, but more than likely for most people you'd just
> want this:
>
> test/dotest.py --results-formatter
> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
> stdout
>
> The other stuff was specific to my setup.  That line assumes you run from
> the lldb source dir root.
>
>
> Let me know if this satisfies the basic needs of counts and whatnot.  It
>> counts test method runs rather than all the oddball "file, class, etc."
>> counts we had before.
>>
>> It prints out the Details section when there are details, and keeps it
>> nice and clean when there are none.
>>
>> It also mentions a bit about test reruns up top, but that won't come into
>> play until I get the multi-test-pass, single-worker/low-load mechanism in
>> place, which will depend on newer rerun count awareness support.
>>
>> The change also cleans up places where the test event framework was using
>> string codes and replaces them with symbolic constants.
>>
>> Let me know what you think.  I can tweak it as needed to address testbot
>> and other needs.  Once it looks reasonable, I'd like to move over to using
>> it by default in the parallel test runner rather than the legacy support.
>>
>> Thanks!
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] static swig bindings and xcode workspace

[Todd Fiala via lldb-dev]

On Wed, Dec 2, 2015 at 8:28 AM, Todd Fiala  wrote:

> Hi Zachary,
>
> On Mon, Nov 30, 2015 at 9:23 AM, Zachary Turner 
> wrote:
>
>> Has the xcode build been changed to use static bindings yet?
>>
>
> It is only in our downstream branches.  I stripped out support in llvm.org
> lldb per our other threads.
>
>
>>   I got to thinking that maybe it would make sense to put them alongside
>> the xcode workspace folders, just to emphasize that the static bindings
>> were an artifact of how the xcode build works.
>>
>
> We could do that.  Internally we also use that for builds other than
> Xcode, so whatever solution I use (which is currently what I had proposed
> earlier but now have only in our branches) really needs to work for more
> than Xcode.
>
>
>>   This way we just say "Xcode build uses static bindings" and "CMake
>> build needs an installed swig", and this is enforced at the directory
>> level.
>>
>>
> That's a great compromise, I appreciate your thoughts on that.  Since I
> need it for more than Xcode, right now the solution I have in our branch is
> working okay.
>
>
>> In order to do this you'd have to probably make a new toplevel folder to
>> house both the lldb.xcodeproj and lldb.xcworkspace folders, but I think
>> that would be useful for other reasons as well.  For example, I want to
>> check in a visual studio python solution for the test suite at some point,
>> and it would make sense if all of this additional stuff was in one place.
>> So perhaps something like:
>>
>> lldb
>> |__ contrib
>> |__ xcode
>> |__ LLDBWrapPython.cpp
>> |__ lldb.py
>> |__ lldb.xcodeproj
>> |__ lldb.xcworkspace
>> |__ msvc
>>
>>
> That structure may make sense.  That could live in llvm.org.  Then for
> other OSes where I want similar behavior, I could just keep those parts in
> our branch.  Ultimately I'd end up with multiple copies of the wrapper (for
> any OS we may build for internally), but I could symlink so that's not
> really any kind of issue.
>
> This might work.
>
>
>> I have been thinking about this idea of a contrib folder for a while
>> anyway, but wanted to have more reasons to make use of it before I brought
>> it up.
>>
>> Good idea?  Bad idea?  Thoughts?
>>
>
> I could see that layout making sense.  If we did something like that, I
> think I'd separate moving the lldb.xcodeproj and lldb.xcworkspace from the
> creation of the contrib folder.
>

It looks like we may have some reasons why we need the Xcode
workspace/project files at the top of the lldb source tree.  I'm not sure
we'll able to move those.  But the rest of it looks like a reasonable way
to go.


>  (i.e. I'd start with the wrapper part in there, and have the others move
> there at lower priority as a scheduling thing --- there's a bit of work to
> make the workspace/project change but should be totally doable).
>
> I think I like the idea since it reduces the number of merge issues I'd
> have to deal with.
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

I think it might be already - let me check.

The longer-term goal would be to get this without specifying anything (i.e.
does what we want by default).  If stdout is not already being used by
default when a formatter is specified, that would be an easy fix.

Checking now...

On Wed, Dec 2, 2015 at 11:04 AM, Zachary Turner  wrote:

> Can --results-file=stdout be the default so that we don't have to specify
> that?
>
> On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> Also, all the text in the summary is fixed-width lined up nicely, which
>> may not show in the commit message description if you're using a
>> variable-width font.  On a terminal it looks nice.
>>
>> On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala  wrote:
>>
>>>
>>>
>>> On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala 
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I just put up an optional test results formatter that is a prototype of
>>>> what we may move towards for our default test summary results.  It went in
>>>> here:
>>>>
>>>> r254530
>>>>
>>>> and you can try it out with something like:
>>>>
>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb
>>>> --results-formatter
>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>>>> st
>>>> out
>>>>
>>>>
>>> I cut and paste my line, but more than likely for most people you'd just
>>> want this:
>>>
>>> test/dotest.py --results-formatter
>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>>> stdout
>>>
>>> The other stuff was specific to my setup.  That line assumes you run
>>> from the lldb source dir root.
>>>
>>>
>>> Let me know if this satisfies the basic needs of counts and whatnot.  It
>>>> counts test method runs rather than all the oddball "file, class, etc."
>>>> counts we had before.
>>>>
>>>> It prints out the Details section when there are details, and keeps it
>>>> nice and clean when there are none.
>>>>
>>>> It also mentions a bit about test reruns up top, but that won't come
>>>> into play until I get the multi-test-pass, single-worker/low-load mechanism
>>>> in place, which will depend on newer rerun count awareness support.
>>>>
>>>> The change also cleans up places where the test event framework was
>>>> using string codes and replaces them with symbolic constants.
>>>>
>>>> Let me know what you think.  I can tweak it as needed to address
>>>> testbot and other needs.  Once it looks reasonable, I'd like to move over
>>>> to using it by default in the parallel test runner rather than the legacy
>>>> support.
>>>>
>>>> Thanks!
>>>> --
>>>> -Todd
>>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

Nope, it didn't work without the results-file entry.

I can get that fixed up.  I'll look at that now.

On Wed, Dec 2, 2015 at 11:05 AM, Todd Fiala  wrote:

> I think it might be already - let me check.
>
> The longer-term goal would be to get this without specifying anything
> (i.e. does what we want by default).  If stdout is not already being used
> by default when a formatter is specified, that would be an easy fix.
>
> Checking now...
>
> On Wed, Dec 2, 2015 at 11:04 AM, Zachary Turner 
> wrote:
>
>> Can --results-file=stdout be the default so that we don't have to specify
>> that?
>>
>> On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> Also, all the text in the summary is fixed-width lined up nicely, which
>>> may not show in the commit message description if you're using a
>>> variable-width font.  On a terminal it looks nice.
>>>
>>> On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala 
>>> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala 
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I just put up an optional test results formatter that is a prototype
>>>>> of what we may move towards for our default test summary results.  It went
>>>>> in here:
>>>>>
>>>>> r254530
>>>>>
>>>>> and you can try it out with something like:
>>>>>
>>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb
>>>>> --results-formatter
>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter 
>>>>> --results-file
>>>>> st
>>>>> out
>>>>>
>>>>>
>>>> I cut and paste my line, but more than likely for most people you'd
>>>> just want this:
>>>>
>>>> test/dotest.py --results-formatter
>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>>>> stdout
>>>>
>>>> The other stuff was specific to my setup.  That line assumes you run
>>>> from the lldb source dir root.
>>>>
>>>>
>>>> Let me know if this satisfies the basic needs of counts and whatnot.
>>>>> It counts test method runs rather than all the oddball "file, class, etc."
>>>>> counts we had before.
>>>>>
>>>>> It prints out the Details section when there are details, and keeps it
>>>>> nice and clean when there are none.
>>>>>
>>>>> It also mentions a bit about test reruns up top, but that won't come
>>>>> into play until I get the multi-test-pass, single-worker/low-load 
>>>>> mechanism
>>>>> in place, which will depend on newer rerun count awareness support.
>>>>>
>>>>> The change also cleans up places where the test event framework was
>>>>> using string codes and replaces them with symbolic constants.
>>>>>
>>>>> Let me know what you think.  I can tweak it as needed to address
>>>>> testbot and other needs.  Once it looks reasonable, I'd like to move over
>>>>> to using it by default in the parallel test runner rather than the legacy
>>>>> support.
>>>>>
>>>>> Thanks!
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>> ___
>>> lldb-dev mailing list
>>> lldb-dev@lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>
>>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

Yeah I'd be good with that.  I can change that as well.

-Todd

On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner  wrote:

> Also another stylistic suggestion.  I've been thinking about how to more
> logically organize all the source files now that we have a package.  So it
> makes sense conceptually to group all of the different result formatters
> under a subpackage called formatters.  So right now you've got
> lldbsuite.test.basic_results_formatter.BasicResultsFormatter but it might
> make sense for this to be
> lldbsuite.test.formatters.basic.BasicResultsFormatter.  If you do things
> this way, it can actually result in a substantially shorter command line,
> because the --results-formatter option can use lldbsuite.test.formatters as
> a starting point.  So you could instead write:
>
> test/dotest.py --results-formatter basic
>
> dotest then looks for a `basic.py` module in the
> `lldbsuite.test.formatters` package, looks for a class inside with a
> @result_formatter decorator, and instantiates that.
>
> This has the advantage of making the command line shorter *and* a more
> logical source file organization.
>
> On Wed, Dec 2, 2015 at 11:04 AM Zachary Turner  wrote:
>
>> Can --results-file=stdout be the default so that we don't have to specify
>> that?
>>
>> On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> Also, all the text in the summary is fixed-width lined up nicely, which
>>> may not show in the commit message description if you're using a
>>> variable-width font.  On a terminal it looks nice.
>>>
>>> On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala 
>>> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala 
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I just put up an optional test results formatter that is a prototype
>>>>> of what we may move towards for our default test summary results.  It went
>>>>> in here:
>>>>>
>>>>> r254530
>>>>>
>>>>> and you can try it out with something like:
>>>>>
>>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb
>>>>> --results-formatter
>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter 
>>>>> --results-file
>>>>> st
>>>>> out
>>>>>
>>>>>
>>>> I cut and paste my line, but more than likely for most people you'd
>>>> just want this:
>>>>
>>>> test/dotest.py --results-formatter
>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>>>> stdout
>>>>
>>>> The other stuff was specific to my setup.  That line assumes you run
>>>> from the lldb source dir root.
>>>>
>>>>
>>>> Let me know if this satisfies the basic needs of counts and whatnot.
>>>>> It counts test method runs rather than all the oddball "file, class, etc."
>>>>> counts we had before.
>>>>>
>>>>> It prints out the Details section when there are details, and keeps it
>>>>> nice and clean when there are none.
>>>>>
>>>>> It also mentions a bit about test reruns up top, but that won't come
>>>>> into play until I get the multi-test-pass, single-worker/low-load 
>>>>> mechanism
>>>>> in place, which will depend on newer rerun count awareness support.
>>>>>
>>>>> The change also cleans up places where the test event framework was
>>>>> using string codes and replaces them with symbolic constants.
>>>>>
>>>>> Let me know what you think.  I can tweak it as needed to address
>>>>> testbot and other needs.  Once it looks reasonable, I'd like to move over
>>>>> to using it by default in the parallel test runner rather than the legacy
>>>>> support.
>>>>>
>>>>> Thanks!
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>> ___
>>> lldb-dev mailing list
>>> lldb-dev@lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>
>>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

I'll also move those couple counts (total tests run and the rerun count)
under the summary.  I missed that when evolving the code.

On Wed, Dec 2, 2015 at 11:20 AM, Todd Fiala  wrote:

> Yeah I'd be good with that.  I can change that as well.
>
> -Todd
>
> On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner 
> wrote:
>
>> Also another stylistic suggestion.  I've been thinking about how to more
>> logically organize all the source files now that we have a package.  So it
>> makes sense conceptually to group all of the different result formatters
>> under a subpackage called formatters.  So right now you've got
>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter but it
>> might make sense for this to be
>> lldbsuite.test.formatters.basic.BasicResultsFormatter.  If you do things
>> this way, it can actually result in a substantially shorter command line,
>> because the --results-formatter option can use lldbsuite.test.formatters as
>> a starting point.  So you could instead write:
>>
>> test/dotest.py --results-formatter basic
>>
>> dotest then looks for a `basic.py` module in the
>> `lldbsuite.test.formatters` package, looks for a class inside with a
>> @result_formatter decorator, and instantiates that.
>>
>> This has the advantage of making the command line shorter *and* a more
>> logical source file organization.
>>
>> On Wed, Dec 2, 2015 at 11:04 AM Zachary Turner 
>> wrote:
>>
>>> Can --results-file=stdout be the default so that we don't have to
>>> specify that?
>>>
>>> On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev <
>>> lldb-dev@lists.llvm.org> wrote:
>>>
>>>> Also, all the text in the summary is fixed-width lined up nicely, which
>>>> may not show in the commit message description if you're using a
>>>> variable-width font.  On a terminal it looks nice.
>>>>
>>>> On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala 
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala 
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I just put up an optional test results formatter that is a prototype
>>>>>> of what we may move towards for our default test summary results.  It 
>>>>>> went
>>>>>> in here:
>>>>>>
>>>>>> r254530
>>>>>>
>>>>>> and you can try it out with something like:
>>>>>>
>>>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb
>>>>>> --results-formatter
>>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter 
>>>>>> --results-file
>>>>>> st
>>>>>> out
>>>>>>
>>>>>>
>>>>> I cut and paste my line, but more than likely for most people you'd
>>>>> just want this:
>>>>>
>>>>> test/dotest.py --results-formatter
>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter 
>>>>> --results-file
>>>>> stdout
>>>>>
>>>>> The other stuff was specific to my setup.  That line assumes you run
>>>>> from the lldb source dir root.
>>>>>
>>>>>
>>>>> Let me know if this satisfies the basic needs of counts and whatnot.
>>>>>> It counts test method runs rather than all the oddball "file, class, 
>>>>>> etc."
>>>>>> counts we had before.
>>>>>>
>>>>>> It prints out the Details section when there are details, and keeps
>>>>>> it nice and clean when there are none.
>>>>>>
>>>>>> It also mentions a bit about test reruns up top, but that won't come
>>>>>> into play until I get the multi-test-pass, single-worker/low-load 
>>>>>> mechanism
>>>>>> in place, which will depend on newer rerun count awareness support.
>>>>>>
>>>>>> The change also cleans up places where the test event framework was
>>>>>> using string codes and replaces them with symbolic constants.
>>>>>>
>>>>>> Let me know what you think.  I can tweak it as needed to address
>>>>>> testbot and other needs.  Once it looks reasonable, I'd like to move over
>>>>>> to using it by default in the parallel test runner rather than the legacy
>>>>>> support.
>>>>>>
>>>>>> Thanks!
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>> ___
>>>> lldb-dev mailing list
>>>> lldb-dev@lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>
>>>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] static swig bindings and xcode workspace

[Todd Fiala via lldb-dev]

I'll dig in more when I get to it, but yeah I like the concept for sure.

On Wed, Dec 2, 2015 at 11:12 AM, Zachary Turner  wrote:

> Ahh, that's a bummer if true.  Because `contrib` is a nice way to group
> together all the things that all the things that have specific maintainers
> and so everyone else is allowed to break.
>
> On Wed, Dec 2, 2015 at 11:04 AM Todd Fiala  wrote:
>
>> On Wed, Dec 2, 2015 at 8:28 AM, Todd Fiala  wrote:
>>
>>> Hi Zachary,
>>>
>>> On Mon, Nov 30, 2015 at 9:23 AM, Zachary Turner 
>>> wrote:
>>>
 Has the xcode build been changed to use static bindings yet?

>>>
>>> It is only in our downstream branches.  I stripped out support in
>>> llvm.org lldb per our other threads.
>>>
>>>
   I got to thinking that maybe it would make sense to put them
 alongside the xcode workspace folders, just to emphasize that the static
 bindings were an artifact of how the xcode build works.

>>>
>>> We could do that.  Internally we also use that for builds other than
>>> Xcode, so whatever solution I use (which is currently what I had proposed
>>> earlier but now have only in our branches) really needs to work for more
>>> than Xcode.
>>>
>>>
   This way we just say "Xcode build uses static bindings" and "CMake
 build needs an installed swig", and this is enforced at the directory
 level.


>>> That's a great compromise, I appreciate your thoughts on that.  Since I
>>> need it for more than Xcode, right now the solution I have in our branch is
>>> working okay.
>>>
>>>
 In order to do this you'd have to probably make a new toplevel folder
 to house both the lldb.xcodeproj and lldb.xcworkspace folders, but I think
 that would be useful for other reasons as well.  For example, I want to
 check in a visual studio python solution for the test suite at some point,
 and it would make sense if all of this additional stuff was in one place.
 So perhaps something like:

 lldb
 |__ contrib
 |__ xcode
 |__ LLDBWrapPython.cpp
 |__ lldb.py
 |__ lldb.xcodeproj
 |__ lldb.xcworkspace
 |__ msvc


>>> That structure may make sense.  That could live in llvm.org.  Then for
>>> other OSes where I want similar behavior, I could just keep those parts in
>>> our branch.  Ultimately I'd end up with multiple copies of the wrapper (for
>>> any OS we may build for internally), but I could symlink so that's not
>>> really any kind of issue.
>>>
>>> This might work.
>>>
>>>
 I have been thinking about this idea of a contrib folder for a while
 anyway, but wanted to have more reasons to make use of it before I brought
 it up.

 Good idea?  Bad idea?  Thoughts?

>>>
>>> I could see that layout making sense.  If we did something like that, I
>>> think I'd separate moving the lldb.xcodeproj and lldb.xcworkspace from the
>>> creation of the contrib folder.
>>>
>>
>> It looks like we may have some reasons why we need the Xcode
>> workspace/project files at the top of the lldb source tree.  I'm not sure
>> we'll able to move those.  But the rest of it looks like a reasonable way
>> to go.
>>
>>
>>>  (i.e. I'd start with the wrapper part in there, and have the others
>>> move there at lower priority as a scheduling thing --- there's a bit of
>>> work to make the workspace/project change but should be totally doable).
>>>
>>> I think I like the idea since it reduces the number of merge issues I'd
>>> have to deal with.
>>>
>>> --
>>> -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

On Wed, Dec 2, 2015 at 11:20 AM, Todd Fiala  wrote:

> Yeah I'd be good with that.  I can change that as well.
>
> -Todd
>
> On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner 
> wrote:
>
>> Also another stylistic suggestion.  I've been thinking about how to more
>> logically organize all the source files now that we have a package.  So it
>> makes sense conceptually to group all of the different result formatters
>> under a subpackage called formatters.  So right now you've got
>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter but it
>> might make sense for this to be
>> lldbsuite.test.formatters.basic.BasicResultsFormatter.  If you do things
>> this way, it can actually result in a substantially shorter command line,
>> because the --results-formatter option can use lldbsuite.test.formatters as
>> a starting point.  So you could instead write:
>>
>> test/dotest.py --results-formatter basic
>>
>> dotest then looks for a `basic.py` module in the
>> `lldbsuite.test.formatters` package, looks for a class inside with a
>> @result_formatter decorator, and instantiates that.
>>
>> This has the advantage of making the command line shorter *and* a more
>> logical source file organization.
>>
>
The other thing that could allow me to do is possibly short-circuit the
results formatter specifier so that, if just the module is specified, and
if the module only has one ResultsFormatter-derived class, I can probably
rig up code that figures out the right results formatter, shortening the
required discriminator to something even shorter (i.e. module.classname
becomes just module.)


>
>> On Wed, Dec 2, 2015 at 11:04 AM Zachary Turner 
>> wrote:
>>
>>> Can --results-file=stdout be the default so that we don't have to
>>> specify that?
>>>
>>> On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev <
>>> lldb-dev@lists.llvm.org> wrote:
>>>
>>>> Also, all the text in the summary is fixed-width lined up nicely, which
>>>> may not show in the commit message description if you're using a
>>>> variable-width font.  On a terminal it looks nice.
>>>>
>>>> On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala 
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala 
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> I just put up an optional test results formatter that is a prototype
>>>>>> of what we may move towards for our default test summary results.  It 
>>>>>> went
>>>>>> in here:
>>>>>>
>>>>>> r254530
>>>>>>
>>>>>> and you can try it out with something like:
>>>>>>
>>>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb
>>>>>> --results-formatter
>>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter 
>>>>>> --results-file
>>>>>> st
>>>>>> out
>>>>>>
>>>>>>
>>>>> I cut and paste my line, but more than likely for most people you'd
>>>>> just want this:
>>>>>
>>>>> test/dotest.py --results-formatter
>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter 
>>>>> --results-file
>>>>> stdout
>>>>>
>>>>> The other stuff was specific to my setup.  That line assumes you run
>>>>> from the lldb source dir root.
>>>>>
>>>>>
>>>>> Let me know if this satisfies the basic needs of counts and whatnot.
>>>>>> It counts test method runs rather than all the oddball "file, class, 
>>>>>> etc."
>>>>>> counts we had before.
>>>>>>
>>>>>> It prints out the Details section when there are details, and keeps
>>>>>> it nice and clean when there are none.
>>>>>>
>>>>>> It also mentions a bit about test reruns up top, but that won't come
>>>>>> into play until I get the multi-test-pass, single-worker/low-load 
>>>>>> mechanism
>>>>>> in place, which will depend on newer rerun count awareness support.
>>>>>>
>>>>>> The change also cleans up places where the test event framework was
>>>>>> using string codes and replaces them with symbolic constants.
>>>>>>
>>>>>> Let me know what you think.  I can tweak it as needed to address
>>>>>> testbot and other needs.  Once it looks reasonable, I'd like to move over
>>>>>> to using it by default in the parallel test runner rather than the legacy
>>>>>> support.
>>>>>>
>>>>>> Thanks!
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>> ___
>>>> lldb-dev mailing list
>>>> lldb-dev@lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>
>>>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

I think I can fix the issue without you debugging.

Getting the single pass test runner to use it isn't impossible but will
take some work.  Can you direct-send me the backtrace from the point of
failure from your system?  Thanks!

-Todd

On Wed, Dec 2, 2015 at 12:34 PM, Zachary Turner  wrote:

> Is there any way to force the single process test runner to use this same
> system?  I'm trying to debug the problem, but this codepath doesn't execute
> in the single process test runner, and it executes in the child process in
> the multiprocess test runner.  Basically I need the following callstack to
> execute in the single process test runner:
>
> Command invoked: C:\Python35\python_d.exe
> D:\src\llvm\tools\lldb\test\dotest.py -q --arch=i686 --executable
> D:/src/llvmbuild/ninja_py35/bin/lldb.exe -s
> D:/src/llvmbuild/ninja_py35/lldb-test-traces -u CXXFLAGS -u CFLAGS
> --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe
> --results-port 60794 --inferior -p TestIntegerTypesExpr.py
> D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test --event-add-entries
> worker_index=7:int
> 411 out of 412 test suites processed - TestIntegerTypesExpr.py
> Traceback (most recent call last):
>   File "D:\src\llvm\tools\lldb\test\dotest.py", line 7, in 
> lldbsuite.test.run_suite()
>   File "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest.py",
> line 1476, in run_suite
> setupTestResults()
>   File "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest.py",
> line 982, in setupTestResults
> results_formatter_object.handle_event(initialize_event)
>   File
> "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\test_results.py",
> line 1033, in handle_event
> "{}#{}".format(len(pickled_message), pickled_message))
> TypeError: a bytes-like object is required, not 'str'
>
> On Wed, Dec 2, 2015 at 11:40 AM Zachary Turner  wrote:
>
>> When I run this under Python 3 I get "A bytes object is used like a
>> string" on Line 1033 of test_results.py.  I'm going to dig into it a little
>> bit, but maybe you know off the top of your head the right way to fix it.
>>
>> On Wed, Dec 2, 2015 at 11:32 AM Zachary Turner 
>> wrote:
>>
>>> Oh yea, I made up that decorator idea because I didn't know all the
>>> formatters were derived from a common base.  But your idea is better if
>>> everything is derived from a common base.  To be honest you could even just
>>> generate an error if there are two ResultsFormatter derived classes in the
>>> same module.  We should be encouraging more smaller files with single
>>> responsibility.  One of the things I plan to do as part of some cleanup in
>>> a week or two is to split up dotest, dosep, and lldbtest.py into a couple
>>> different files by breaking out things like TestBase, etc into separate
>>> files.  So that it's easier to keep a mental map of where different code is.
>>>
>>> On Wed, Dec 2, 2015 at 11:26 AM Todd Fiala  wrote:
>>>
>>>> On Wed, Dec 2, 2015 at 11:20 AM, Todd Fiala 
>>>> wrote:
>>>>
>>>>> Yeah I'd be good with that.  I can change that as well.
>>>>>
>>>>> -Todd
>>>>>
>>>>> On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner 
>>>>> wrote:
>>>>>
>>>>>> Also another stylistic suggestion.  I've been thinking about how to
>>>>>> more logically organize all the source files now that we have a package.
>>>>>> So it makes sense conceptually to group all of the different result
>>>>>> formatters under a subpackage called formatters.  So right now you've got
>>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter but it
>>>>>> might make sense for this to be
>>>>>> lldbsuite.test.formatters.basic.BasicResultsFormatter.  If you do things
>>>>>> this way, it can actually result in a substantially shorter command line,
>>>>>> because the --results-formatter option can use lldbsuite.test.formatters 
>>>>>> as
>>>>>> a starting point.  So you could instead write:
>>>>>>
>>>>>> test/dotest.py --results-formatter basic
>>>>>>
>>>>>> dotest then looks for a `basic.py` module in the
>>>>>> `lldbsuite.test.formatters` package, looks for a class inside with a
>>>>>> @result_formatter decorator, and instantiates that.
>>>>>>
&

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

On Wed, Dec 2, 2015 at 11:04 AM, Zachary Turner  wrote:

> Can --results-file=stdout be the default so that we don't have to specify
> that?
>
>
I've adjusted the code here:
r254546

to support dropping the --results-file=stdout part if a --results-formatter
is specified and no results-file is specified.  Good idea, thanks!


> On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> Also, all the text in the summary is fixed-width lined up nicely, which
>> may not show in the commit message description if you're using a
>> variable-width font.  On a terminal it looks nice.
>>
>> On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala  wrote:
>>
>>>
>>>
>>> On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala 
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I just put up an optional test results formatter that is a prototype of
>>>> what we may move towards for our default test summary results.  It went in
>>>> here:
>>>>
>>>> r254530
>>>>
>>>> and you can try it out with something like:
>>>>
>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb
>>>> --results-formatter
>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>>>> st
>>>> out
>>>>
>>>>
>>> I cut and paste my line, but more than likely for most people you'd just
>>> want this:
>>>
>>> test/dotest.py --results-formatter
>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter --results-file
>>> stdout
>>>
>>> The other stuff was specific to my setup.  That line assumes you run
>>> from the lldb source dir root.
>>>
>>>
>>> Let me know if this satisfies the basic needs of counts and whatnot.  It
>>>> counts test method runs rather than all the oddball "file, class, etc."
>>>> counts we had before.
>>>>
>>>> It prints out the Details section when there are details, and keeps it
>>>> nice and clean when there are none.
>>>>
>>>> It also mentions a bit about test reruns up top, but that won't come
>>>> into play until I get the multi-test-pass, single-worker/low-load mechanism
>>>> in place, which will depend on newer rerun count awareness support.
>>>>
>>>> The change also cleans up places where the test event framework was
>>>> using string codes and replaces them with symbolic constants.
>>>>
>>>> Let me know what you think.  I can tweak it as needed to address
>>>> testbot and other needs.  Once it looks reasonable, I'd like to move over
>>>> to using it by default in the parallel test runner rather than the legacy
>>>> support.
>>>>
>>>> Thanks!
>>>> --
>>>> -Todd
>>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

can use 
>>>>>>> lldbsuite.test.formatters as
>>>>>>> a starting point.  So you could instead write:
>>>>>>>
>>>>>>> test/dotest.py --results-formatter basic
>>>>>>>
>>>>>>> dotest then looks for a `basic.py` module in the
>>>>>>> `lldbsuite.test.formatters` package, looks for a class inside with a
>>>>>>> @result_formatter decorator, and instantiates that.
>>>>>>>
>>>>>>> This has the advantage of making the command line shorter *and* a
>>>>>>> more logical source file organization.
>>>>>>>
>>>>>>
>>>>> The other thing that could allow me to do is possibly short-circuit
>>>>> the results formatter specifier so that, if just the module is specified,
>>>>> and if the module only has one ResultsFormatter-derived class, I can
>>>>> probably rig up code that figures out the right results formatter,
>>>>> shortening the required discriminator to something even shorter (i.e.
>>>>> module.classname becomes just module.)
>>>>>
>>>>>
>>>>>>
>>>>>>> On Wed, Dec 2, 2015 at 11:04 AM Zachary Turner 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Can --results-file=stdout be the default so that we don't have to
>>>>>>>> specify that?
>>>>>>>>
>>>>>>>> On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev <
>>>>>>>> lldb-dev@lists.llvm.org> wrote:
>>>>>>>>
>>>>>>>>> Also, all the text in the summary is fixed-width lined up nicely,
>>>>>>>>> which may not show in the commit message description if you're using a
>>>>>>>>> variable-width font.  On a terminal it looks nice.
>>>>>>>>>
>>>>>>>>> On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala >>>>>>>>> > wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I just put up an optional test results formatter that is a
>>>>>>>>>>> prototype of what we may move towards for our default test summary
>>>>>>>>>>> results.  It went in here:
>>>>>>>>>>>
>>>>>>>>>>> r254530
>>>>>>>>>>>
>>>>>>>>>>> and you can try it out with something like:
>>>>>>>>>>>
>>>>>>>>>>> time test/dotest.py --executable `pwd`/build/Debug/lldb
>>>>>>>>>>> --results-formatter
>>>>>>>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter 
>>>>>>>>>>> --results-file
>>>>>>>>>>> st
>>>>>>>>>>> out
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> I cut and paste my line, but more than likely for most people
>>>>>>>>>> you'd just want this:
>>>>>>>>>>
>>>>>>>>>> test/dotest.py --results-formatter
>>>>>>>>>> lldbsuite.test.basic_results_formatter.BasicResultsFormatter 
>>>>>>>>>> --results-file
>>>>>>>>>> stdout
>>>>>>>>>>
>>>>>>>>>> The other stuff was specific to my setup.  That line assumes you
>>>>>>>>>> run from the lldb source dir root.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Let me know if this satisfies the basic needs of counts and
>>>>>>>>>>> whatnot.  It counts test method runs rather than all the oddball 
>>>>>>>>>>> "file,
>>>>>>>>>>> class, etc." counts we had before.
>>>>>>>>>>>
>>>>>>>>>>> It prints out the Details section when there are details, and
>>>>>>>>>>> keeps it nice and clean when there are none.
>>>>>>>>>>>
>>>>>>>>>>> It also mentions a bit about test reruns up top, but that won't
>>>>>>>>>>> come into play until I get the multi-test-pass, 
>>>>>>>>>>> single-worker/low-load
>>>>>>>>>>> mechanism in place, which will depend on newer rerun count awareness
>>>>>>>>>>> support.
>>>>>>>>>>>
>>>>>>>>>>> The change also cleans up places where the test event framework
>>>>>>>>>>> was using string codes and replaces them with symbolic constants.
>>>>>>>>>>>
>>>>>>>>>>> Let me know what you think.  I can tweak it as needed to address
>>>>>>>>>>> testbot and other needs.  Once it looks reasonable, I'd like to 
>>>>>>>>>>> move over
>>>>>>>>>>> to using it by default in the parallel test runner rather than the 
>>>>>>>>>>> legacy
>>>>>>>>>>> support.
>>>>>>>>>>>
>>>>>>>>>>> Thanks!
>>>>>>>>>>> --
>>>>>>>>>>> -Todd
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> -Todd
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> -Todd
>>>>>>>>> ___
>>>>>>>>> lldb-dev mailing list
>>>>>>>>> lldb-dev@lists.llvm.org
>>>>>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

>> just
>>>>>> generate an error if there are two ResultsFormatter derived classes in 
>>>>>> the
>>>>>> same module.  We should be encouraging more smaller files with single
>>>>>> responsibility.  One of the things I plan to do as part of some cleanup 
>>>>>> in
>>>>>> a week or two is to split up dotest, dosep, and lldbtest.py into a couple
>>>>>> different files by breaking out things like TestBase, etc into separate
>>>>>> files.  So that it's easier to keep a mental map of where different code 
>>>>>> is.
>>>>>>
>>>>>> On Wed, Dec 2, 2015 at 11:26 AM Todd Fiala 
>>>>>> wrote:
>>>>>>
>>>>>>> On Wed, Dec 2, 2015 at 11:20 AM, Todd Fiala 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yeah I'd be good with that.  I can change that as well.
>>>>>>>>
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>> On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner >>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Also another stylistic suggestion.  I've been thinking about how
>>>>>>>>> to more logically organize all the source files now that we have a
>>>>>>>>> package.  So it makes sense conceptually to group all of the different
>>>>>>>>> result formatters under a subpackage called formatters.  So right now
>>>>>>>>> you've got lldbsuite.test.basic_results_formatter.
>>>>>>>>> BasicResultsFormatter but it might make sense for this to be
>>>>>>>>> lldbsuite.test.formatters.basic.BasicResultsFormatter.  If you do 
>>>>>>>>> things
>>>>>>>>> this way, it can actually result in a substantially shorter command 
>>>>>>>>> line,
>>>>>>>>> because the --results-formatter option can use 
>>>>>>>>> lldbsuite.test.formatters as
>>>>>>>>> a starting point.  So you could instead write:
>>>>>>>>>
>>>>>>>>> test/dotest.py --results-formatter basic
>>>>>>>>>
>>>>>>>>> dotest then looks for a `basic.py` module in the
>>>>>>>>> `lldbsuite.test.formatters` package, looks for a class inside with a
>>>>>>>>> @result_formatter decorator, and instantiates that.
>>>>>>>>>
>>>>>>>>> This has the advantage of making the command line shorter *and* a
>>>>>>>>> more logical source file organization.
>>>>>>>>>
>>>>>>>>
>>>>>>> The other thing that could allow me to do is possibly short-circuit
>>>>>>> the results formatter specifier so that, if just the module is 
>>>>>>> specified,
>>>>>>> and if the module only has one ResultsFormatter-derived class, I can
>>>>>>> probably rig up code that figures out the right results formatter,
>>>>>>> shortening the required discriminator to something even shorter (i.e.
>>>>>>> module.classname becomes just module.)
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>> On Wed, Dec 2, 2015 at 11:04 AM Zachary Turner 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Can --results-file=stdout be the default so that we don't have to
>>>>>>>>>> specify that?
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev <
>>>>>>>>>> lldb-dev@lists.llvm.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Also, all the text in the summary is fixed-width lined up
>>>>>>>>>>> nicely, which may not show in the commit message description if 
>>>>>>>>>>> you're
>>>>>>>>>>> using a variable-width font.  On a terminal it looks nice.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala <
>>&g

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

;> in
>>>>>> a week or two is to split up dotest, dosep, and lldbtest.py into a couple
>>>>>> different files by breaking out things like TestBase, etc into separate
>>>>>> files.  So that it's easier to keep a mental map of where different code 
>>>>>> is.
>>>>>>
>>>>>> On Wed, Dec 2, 2015 at 11:26 AM Todd Fiala 
>>>>>> wrote:
>>>>>>
>>>>>>> On Wed, Dec 2, 2015 at 11:20 AM, Todd Fiala 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Yeah I'd be good with that.  I can change that as well.
>>>>>>>>
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>> On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner >>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Also another stylistic suggestion.  I've been thinking about how
>>>>>>>>> to more logically organize all the source files now that we have a
>>>>>>>>> package.  So it makes sense conceptually to group all of the different
>>>>>>>>> result formatters under a subpackage called formatters.  So right now
>>>>>>>>> you've got lldbsuite.test.basic_results_formatter.
>>>>>>>>> BasicResultsFormatter but it might make sense for this to be
>>>>>>>>> lldbsuite.test.formatters.basic.BasicResultsFormatter.  If you do 
>>>>>>>>> things
>>>>>>>>> this way, it can actually result in a substantially shorter command 
>>>>>>>>> line,
>>>>>>>>> because the --results-formatter option can use 
>>>>>>>>> lldbsuite.test.formatters as
>>>>>>>>> a starting point.  So you could instead write:
>>>>>>>>>
>>>>>>>>> test/dotest.py --results-formatter basic
>>>>>>>>>
>>>>>>>>> dotest then looks for a `basic.py` module in the
>>>>>>>>> `lldbsuite.test.formatters` package, looks for a class inside with a
>>>>>>>>> @result_formatter decorator, and instantiates that.
>>>>>>>>>
>>>>>>>>> This has the advantage of making the command line shorter *and* a
>>>>>>>>> more logical source file organization.
>>>>>>>>>
>>>>>>>>
>>>>>>> The other thing that could allow me to do is possibly short-circuit
>>>>>>> the results formatter specifier so that, if just the module is 
>>>>>>> specified,
>>>>>>> and if the module only has one ResultsFormatter-derived class, I can
>>>>>>> probably rig up code that figures out the right results formatter,
>>>>>>> shortening the required discriminator to something even shorter (i.e.
>>>>>>> module.classname becomes just module.)
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>> On Wed, Dec 2, 2015 at 11:04 AM Zachary Turner 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Can --results-file=stdout be the default so that we don't have to
>>>>>>>>>> specify that?
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 2, 2015 at 11:02 AM Todd Fiala via lldb-dev <
>>>>>>>>>> lldb-dev@lists.llvm.org> wrote:
>>>>>>>>>>
>>>>>>>>>>> Also, all the text in the summary is fixed-width lined up
>>>>>>>>>>> nicely, which may not show in the commit message description if 
>>>>>>>>>>> you're
>>>>>>>>>>> using a variable-width font.  On a terminal it looks nice.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Dec 2, 2015 at 11:01 AM, Todd Fiala <
>>>>>>>>>>> todd.fi...@gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Dec 2, 2015 at 10:57 AM, Todd Fiala <
>>>>>>>>>>>> todd.fi...@gmail.com> wrote:
>>>

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

> Oh yea, I made up that decorator idea because I didn't know all the
>>>>>>> formatters were derived from a common base.  But your idea is better if
>>>>>>> everything is derived from a common base.  To be honest you could even 
>>>>>>> just
>>>>>>> generate an error if there are two ResultsFormatter derived classes in 
>>>>>>> the
>>>>>>> same module.  We should be encouraging more smaller files with single
>>>>>>> responsibility.  One of the things I plan to do as part of some cleanup 
>>>>>>> in
>>>>>>> a week or two is to split up dotest, dosep, and lldbtest.py into a 
>>>>>>> couple
>>>>>>> different files by breaking out things like TestBase, etc into separate
>>>>>>> files.  So that it's easier to keep a mental map of where different 
>>>>>>> code is.
>>>>>>>
>>>>>>> On Wed, Dec 2, 2015 at 11:26 AM Todd Fiala 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Wed, Dec 2, 2015 at 11:20 AM, Todd Fiala 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Yeah I'd be good with that.  I can change that as well.
>>>>>>>>>
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>> On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner <
>>>>>>>>> ztur...@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Also another stylistic suggestion.  I've been thinking about how
>>>>>>>>>> to more logically organize all the source files now that we have a
>>>>>>>>>> package.  So it makes sense conceptually to group all of the 
>>>>>>>>>> different
>>>>>>>>>> result formatters under a subpackage called formatters.  So right now
>>>>>>>>>> you've got lldbsuite.test.basic_results_formatter.
>>>>>>>>>> BasicResultsFormatter but it might make sense for this to be
>>>>>>>>>> lldbsuite.test.formatters.basic.BasicResultsFormatter.  If you do 
>>>>>>>>>> things
>>>>>>>>>> this way, it can actually result in a substantially shorter command 
>>>>>>>>>> line,
>>>>>>>>>> because the --results-formatter option can use 
>>>>>>>>>> lldbsuite.test.formatters as
>>>>>>>>>> a starting point.  So you could instead write:
>>>>>>>>>>
>>>>>>>>>> test/dotest.py --results-formatter basic
>>>>>>>>>>
>>>>>>>>>> dotest then looks for a `basic.py` module in the
>>>>>>>>>> `lldbsuite.test.formatters` package, looks for a class inside with a
>>>>>>>>>> @result_formatter decorator, and instantiates that.
>>>>>>>>>>
>>>>>>>>>> This has the advantage of making the command line shorter *and* a
>>>>>>>>>> more logical source file organization.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> The other thing that could allow me to do is possibly short-circuit
>>>>>>>> the results formatter specifier so that, if just the module is 
>>>>>>>> specified,
>>>>>>>> and if the module only has one ResultsFormatter-derived class, I can
>>>>>>>> probably rig up code that figures out the right results formatter,
>>>>>>>> shortening the required discriminator to something even shorter (i.e.
>>>>>>>> module.classname becomes just module.)
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> On Wed, Dec 2, 2015 at 11:04 AM Zachary Turner <
>>>>>>>>>> ztur...@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Can --results-file=stdout be the default so that we don't have
>>>>>>>>>>> to specify that?
>>>>>>>>>>>
>>&

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

>>>>>>
>>>>>>> When I run this under Python 3 I get "A bytes object is used like a
>>>>>>> string" on Line 1033 of test_results.py.  I'm going to dig into it a 
>>>>>>> little
>>>>>>> bit, but maybe you know off the top of your head the right way to fix 
>>>>>>> it.
>>>>>>>
>>>>>>> On Wed, Dec 2, 2015 at 11:32 AM Zachary Turner 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Oh yea, I made up that decorator idea because I didn't know all the
>>>>>>>> formatters were derived from a common base.  But your idea is better if
>>>>>>>> everything is derived from a common base.  To be honest you could even 
>>>>>>>> just
>>>>>>>> generate an error if there are two ResultsFormatter derived classes in 
>>>>>>>> the
>>>>>>>> same module.  We should be encouraging more smaller files with single
>>>>>>>> responsibility.  One of the things I plan to do as part of some 
>>>>>>>> cleanup in
>>>>>>>> a week or two is to split up dotest, dosep, and lldbtest.py into a 
>>>>>>>> couple
>>>>>>>> different files by breaking out things like TestBase, etc into separate
>>>>>>>> files.  So that it's easier to keep a mental map of where different 
>>>>>>>> code is.
>>>>>>>>
>>>>>>>> On Wed, Dec 2, 2015 at 11:26 AM Todd Fiala 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On Wed, Dec 2, 2015 at 11:20 AM, Todd Fiala 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Yeah I'd be good with that.  I can change that as well.
>>>>>>>>>>
>>>>>>>>>> -Todd
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 2, 2015 at 11:10 AM, Zachary Turner <
>>>>>>>>>> ztur...@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Also another stylistic suggestion.  I've been thinking about how
>>>>>>>>>>> to more logically organize all the source files now that we have a
>>>>>>>>>>> package.  So it makes sense conceptually to group all of the 
>>>>>>>>>>> different
>>>>>>>>>>> result formatters under a subpackage called formatters.  So right 
>>>>>>>>>>> now
>>>>>>>>>>> you've got lldbsuite.test.basic_results_formatter.
>>>>>>>>>>> BasicResultsFormatter but it might make sense for this to be
>>>>>>>>>>> lldbsuite.test.formatters.basic.BasicResultsFormatter.  If you do 
>>>>>>>>>>> things
>>>>>>>>>>> this way, it can actually result in a substantially shorter command 
>>>>>>>>>>> line,
>>>>>>>>>>> because the --results-formatter option can use 
>>>>>>>>>>> lldbsuite.test.formatters as
>>>>>>>>>>> a starting point.  So you could instead write:
>>>>>>>>>>>
>>>>>>>>>>> test/dotest.py --results-formatter basic
>>>>>>>>>>>
>>>>>>>>>>> dotest then looks for a `basic.py` module in the
>>>>>>>>>>> `lldbsuite.test.formatters` package, looks for a class inside with a
>>>>>>>>>>> @result_formatter decorator, and instantiates that.
>>>>>>>>>>>
>>>>>>>>>>> This has the advantage of making the command line shorter *and*
>>>>>>>>>>> a more logical source file organization.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> The other thing that could allow me to do is possibly
>>>>>>>>> short-circuit the results formatter specifier so that, if just the 
>>>>>>>>> module
>>>>>>>>> is specified, and if the module only has one ResultsFormatter-derived
>&g

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

rom your system?  Thanks!
>>>>>>>>>
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>> On Wed, Dec 2, 2015 at 12:34 PM, Zachary Turner <
>>>>>>>>> ztur...@google.com> wrote:
>>>>>>>>>
>>>>>>>>>> Is there any way to force the single process test runner to use
>>>>>>>>>> this same system?  I'm trying to debug the problem, but this codepath
>>>>>>>>>> doesn't execute in the single process test runner, and it executes 
>>>>>>>>>> in the
>>>>>>>>>> child process in the multiprocess test runner.  Basically I need the
>>>>>>>>>> following callstack to execute in the single process test runner:
>>>>>>>>>>
>>>>>>>>>> Command invoked: C:\Python35\python_d.exe
>>>>>>>>>> D:\src\llvm\tools\lldb\test\dotest.py -q --arch=i686 --executable
>>>>>>>>>> D:/src/llvmbuild/ninja_py35/bin/lldb.exe -s
>>>>>>>>>> D:/src/llvmbuild/ninja_py35/lldb-test-traces -u CXXFLAGS -u CFLAGS
>>>>>>>>>> --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe
>>>>>>>>>> --results-port 60794 --inferior -p TestIntegerTypesExpr.py
>>>>>>>>>> D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test 
>>>>>>>>>> --event-add-entries
>>>>>>>>>> worker_index=7:int
>>>>>>>>>> 411 out of 412 test suites processed - TestIntegerTypesExpr.py
>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>   File "D:\src\llvm\tools\lldb\test\dotest.py", line 7, in
>>>>>>>>>> 
>>>>>>>>>> lldbsuite.test.run_suite()
>>>>>>>>>>   File
>>>>>>>>>> "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest.py", 
>>>>>>>>>> line
>>>>>>>>>> 1476, in run_suite
>>>>>>>>>> setupTestResults()
>>>>>>>>>>   File
>>>>>>>>>> "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest.py", 
>>>>>>>>>> line
>>>>>>>>>> 982, in setupTestResults
>>>>>>>>>> results_formatter_object.handle_event(initialize_event)
>>>>>>>>>>   File
>>>>>>>>>> "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\test_results.py",
>>>>>>>>>> line 1033, in handle_event
>>>>>>>>>> "{}#{}".format(len(pickled_message), pickled_message))
>>>>>>>>>> TypeError: a bytes-like object is required, not 'str'
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 2, 2015 at 11:40 AM Zachary Turner <
>>>>>>>>>> ztur...@google.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> When I run this under Python 3 I get "A bytes object is used
>>>>>>>>>>> like a string" on Line 1033 of test_results.py.  I'm going to dig 
>>>>>>>>>>> into it a
>>>>>>>>>>> little bit, but maybe you know off the top of your head the right 
>>>>>>>>>>> way to
>>>>>>>>>>> fix it.
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Dec 2, 2015 at 11:32 AM Zachary Turner <
>>>>>>>>>>> ztur...@google.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Oh yea, I made up that decorator idea because I didn't know all
>>>>>>>>>>>> the formatters were derived from a common base.  But your idea is 
>>>>>>>>>>>> better if
>>>>>>>>>>>> everything is derived from a common base.  To be honest you could 
>>>>>>>>>>>> even just
>>>>>>>>>>>> generate an error if there are two ResultsFormatter derived 
>>>>>>&g

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

e the single process test runner to use
>>>>>>>>>>>> this same system?  I'm trying to debug the problem, but this 
>>>>>>>>>>>> codepath
>>>>>>>>>>>> doesn't execute in the single process test runner, and it executes 
>>>>>>>>>>>> in the
>>>>>>>>>>>> child process in the multiprocess test runner.  Basically I need 
>>>>>>>>>>>> the
>>>>>>>>>>>> following callstack to execute in the single process test runner:
>>>>>>>>>>>>
>>>>>>>>>>>> Command invoked: C:\Python35\python_d.exe
>>>>>>>>>>>> D:\src\llvm\tools\lldb\test\dotest.py -q --arch=i686 --executable
>>>>>>>>>>>> D:/src/llvmbuild/ninja_py35/bin/lldb.exe -s
>>>>>>>>>>>> D:/src/llvmbuild/ninja_py35/lldb-test-traces -u CXXFLAGS -u CFLAGS
>>>>>>>>>>>> --enable-crash-dialog -C 
>>>>>>>>>>>> d:\src\llvmbuild\ninja_release\bin\clang.exe
>>>>>>>>>>>> --results-port 60794 --inferior -p TestIntegerTypesExpr.py
>>>>>>>>>>>> D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test 
>>>>>>>>>>>> --event-add-entries
>>>>>>>>>>>> worker_index=7:int
>>>>>>>>>>>> 411 out of 412 test suites processed - TestIntegerTypesExpr.py
>>>>>>>>>>>> Traceback (most recent call last):
>>>>>>>>>>>>   File "D:\src\llvm\tools\lldb\test\dotest.py", line 7, in
>>>>>>>>>>>> 
>>>>>>>>>>>> lldbsuite.test.run_suite()
>>>>>>>>>>>>   File
>>>>>>>>>>>> "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest.py", 
>>>>>>>>>>>> line
>>>>>>>>>>>> 1476, in run_suite
>>>>>>>>>>>> setupTestResults()
>>>>>>>>>>>>   File
>>>>>>>>>>>> "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\dotest.py", 
>>>>>>>>>>>> line
>>>>>>>>>>>> 982, in setupTestResults
>>>>>>>>>>>> results_formatter_object.handle_event(initialize_event)
>>>>>>>>>>>>   File
>>>>>>>>>>>> "D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\test_results.py",
>>>>>>>>>>>> line 1033, in handle_event
>>>>>>>>>>>> "{}#{}".format(len(pickled_message), pickled_message))
>>>>>>>>>>>> TypeError: a bytes-like object is required, not 'str'
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Dec 2, 2015 at 11:40 AM Zachary Turner <
>>>>>>>>>>>> ztur...@google.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> When I run this under Python 3 I get "A bytes object is used
>>>>>>>>>>>>> like a string" on Line 1033 of test_results.py.  I'm going to dig 
>>>>>>>>>>>>> into it a
>>>>>>>>>>>>> little bit, but maybe you know off the top of your head the right 
>>>>>>>>>>>>> way to
>>>>>>>>>>>>> fix it.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Dec 2, 2015 at 11:32 AM Zachary Turner <
>>>>>>>>>>>>> ztur...@google.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Oh yea, I made up that decorator idea because I didn't know
>>>>>>>>>>>>>> all the formatters were derived from a common base.  But your 
>>>>>>>>>>>>>> idea is
>>>>>>>>>>>>>> better if everything is derived from a common base.  To be 
>>>>>>>>>>>>>&g

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

On Wed, Dec 2, 2015 at 9:48 PM, Zachary Turner  wrote:

>
>
> On Wed, Dec 2, 2015 at 9:44 PM Todd Fiala  wrote:
>
>>
>>
>>> and the classname could be dropped (there's only one class per file
>>> anyway, so the classname is just wasted space)
>>>
>>
>> Part of the reason I included that is I've hit several times where copy
>> and paste errors lead to the same class name, method name or even file name
>> being used for a test.  I think, though, that most of those are addressed
>> by having the path (relative is fine) to the python test file.  I think we
>> can probably get by with classname.methodname (relative test path).  (From
>> your other email, I think you nuke the classname and keep the module name,
>> but I'd probably do the reverse, keeping the class name and getting rid of
>> the module name since it can be derived from the filename).
>>
> I don't think the filename can be the same anymore, as things will break
> if two filenames are the same.
>

Maybe, but that wasn't my experience as of fairly recently.  When tracking
failures sometime within the last month, I tracked something down in a
downstream branch with two same-named files that (with the legacy output)
made it hard to track down what was actually failing given the limited info
of the legacy test summary output.  Maybe that has changed since then, but
I'm not aware of anything that would have prohibited that.


>   We could go one step further and enforce this in the part where it scans
> for all the tests.
>

I think I can come up with a valid counterargument to doing that.  I could
imagine some python .py files being organized hierarchically, where some of
the context of what is being tested clearly comes from the directory
structure.

Something like (I'm making this up):

lang/c/const/TestConst.py
lang/c++/const/TestConst.py

where it seems totally reasonable to me to have things testing const
support (in this example) but being very different things for C and C++,
being totally uniqued by path rather than the .py file.  I'd prefer not to
require something like this to say:
lang/c/const/TestConstC.py
lang/c++/const/TestConstC++.py

as it is redundant (at least via the path hierarchy).

The other reason I could see avoiding that
unique-test-basenames-across-test-suite restriction is that it can become
somewhat of an unnecessary burden on downstream branches.  Imagine somebody
has a branch and has a test that happens to be running fine, then somebody
in llvm.org lldb adds a test with the same name.  Downstream breaks.  We
could choose to not care about that, but given that a lot of our tests will
revolve around language features accessed/provided by the debugger, and a
number of language features pull out of a limited set of feature names
(e.g. const above), I could see us sometimes hitting this.

Just one take on it.  I'm not particularly wedded to it (I probably would
avoid the confusion by doing something exactly like what I said above with
regards to tacking on the language to the test name), but I have hit this
in similar form across different language tests.


>   If it finds two test files with the same name we could just generate an
> error.  I think that's a good idea anyway, because if two test files have
> the same name, then the tests inside must be similar enough to warrant
> merging them into the same file.
>

Maybe, but not in the real cases I saw across different languages.  I think
for other areas of the debugger, this isn't an issue.  So maybe language
feature tests just have to know to append their language (whether it be C,
C++, ObjC, etc.)


>
> If no two filenames are the same, and if there's only 1 class per file,
> then filename + method name should uniquely identify a single test, and so
> you could omit the class name and show a relative path to the filename.
>
>>
I think we currently have some tests that have multiple test classes in the
test file.  We could certainly verify that in TOT, and we could certainly
undo that which seems reasonable.

I'd be interested in what other people think here on restricting test names
to be unique across the repo.  I could be convinced either way on allowing
two tests with the same name, but I'd probably avoid layering on a
restriction if it is entirely artificial and requires longer test names
that are otherwise uniqued by path.

Thanks for the feedback!
-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

On Thu, Dec 3, 2015 at 1:41 AM, Pavel Labath  wrote:

> There is already code that enforces unique names (see dotest.py:1255).
>

How long ago was that added?  I literally hit this within the last 6-8
weeks on something internal.


> I added this a while back because we were getting test flakyness
> because of that. I don't remember the exact reason but I think it had
> something to do with the log file names clashes, as all logs are
> placed in the same folder.
>
> I agree that it is possible to have multiple test files with the same
> name, and still have the names meaningful, but then we need to make
> sure that the it actually works reliably.
>
>
I really have no problem with this being forced.  With regards to other
repos, this is no different than using a shared library where a base class
may (in the future) add a method name that perturbs a downstream user
(subclass).  Removal of confusion trumps inconvenience in this case.  I'm
sold.


>
> As for the results formatter, I like the new output a lot.


Glad to hear that.  I'm looking forward to making it better.


> For me, the
> inability to recognize crashes is a blocker for using it, but once
> that works, I would definitely be in favor of making the default.
>
>
Yep, that and timeouts.  For us, we catch both on Jenkins and the Xunit
formatter only because the Xunit results gatherer also checks the output,
where the legacy summary support displays this information.  (I have the
new summary output wiring in the timeout data, but not the exceptional exit
(i.e. signal-based exit)) bits.

I'll be addressing both of those in the extremely near future.

I'll be depending on the test event infrastructure's ability to handle
re-runs to do the low-load, single-worker test pass, so I'm working on this
test summary bit first.  (And it's nice to have!).


>
> I would welcome shortening of the test names, but that is of secondary
> significance for me.
>
>
Ultimately I think it's fair to say "hey just put one test case class in a
file", and with a "test file base names must be unique" policy, I think we
can even get away with "test_method_name (relative filename)" to identify
the test method.  That should be even shorter than what Zachary and I were
discussing last night.


> great job,
> pl
>
>
Thanks for the feedback!

Hopefully this serves you as well, Dawn?


>
> On 3 December 2015 at 06:20, Todd Fiala via lldb-dev
>  wrote:
> >
> >
> > On Wed, Dec 2, 2015 at 9:48 PM, Zachary Turner 
> wrote:
> >>
> >>
> >>
> >> On Wed, Dec 2, 2015 at 9:44 PM Todd Fiala  wrote:
> >>>
> >>>
> >>>>
> >>>> and the classname could be dropped (there's only one class per file
> >>>> anyway, so the classname is just wasted space)
> >>>
> >>>
> >>> Part of the reason I included that is I've hit several times where copy
> >>> and paste errors lead to the same class name, method name or even file
> name
> >>> being used for a test.  I think, though, that most of those are
> addressed by
> >>> having the path (relative is fine) to the python test file.  I think
> we can
> >>> probably get by with classname.methodname (relative test path).  (From
> your
> >>> other email, I think you nuke the classname and keep the module name,
> but
> >>> I'd probably do the reverse, keeping the class name and getting rid of
> the
> >>> module name since it can be derived from the filename).
> >>
> >> I don't think the filename can be the same anymore, as things will break
> >> if two filenames are the same.
> >
> >
> > Maybe, but that wasn't my experience as of fairly recently.  When
> tracking
> > failures sometime within the last month, I tracked something down in a
> > downstream branch with two same-named files that (with the legacy output)
> > made it hard to track down what was actually failing given the limited
> info
> > of the legacy test summary output.  Maybe that has changed since then,
> but
> > I'm not aware of anything that would have prohibited that.
> >
> >>
> >>   We could go one step further and enforce this in the part where it
> scans
> >> for all the tests.
> >
> >
> > I think I can come up with a valid counterargument to doing that.  I
> could
> > imagine some python .py files being organized hierarchically, where some
> of
> > the context of what is being tested clearly comes from the directory
> > structure.
> >
> > Something like (I'm making

[lldb-dev] LLDB and Swift

[Todd Fiala via lldb-dev]

Hi all,

Earlier today, you may have heard that Swift went open source over at
swift.org.  I just wanted to take a moment to mention the Swift debugger
and REPL and how they relate to LLDB.

Swift’s Debugger and REPL are built on LLDB’s source-level plug-in
architecture.  As such, the Swift Debugger repository at
github.com/apple/swift-lldb naturally contains the LLDB source from llvm.org’s
LLDB repository, plus additions for Swift language support. We merge
regularly and make every attempt to minimize our differences with llvm.org’s
LLDB.  For more information on how we’re handling this, have a look at
swift.org/contributing/#llvm-and-swift.

As we’ve worked hard to make it straightforward to develop additive-only
language support in LLDB, the Swift support can readily be found by finding
the new files in the swift-lldb repository vs. those found at
llvm.org/svn/llvm-project/lldb/trunk.  For the rest of the LLDB files in
common, we do still have a small number of diffs in
github.com/apple/swift-lldb vs. llvm.org TOT.  We will work through
upstreaming these quickly.  I’ll touch on some of those differences briefly
here:

* Several minor places where full language abstraction hasn’t yet occurred,
where we’re explicitly checking for Swift-related details.  Abstracting out
those remaining places and providing the hooks in llvm.org LLDB will
benefit all languages.

* Printed-form version string handling.  The ‘lldb -v’ and ‘(lldb) version’
commands create a different version string in both Xcode and cmake-based
Swift LLDB.  We will work to incorporate this into llvm.org LLDB once the
language/component version support info is properly abstracted out.

* Test infrastructure.  There are a few places where Swift language support
(e.g. swift compiler flags, runtime support directories, etc.) are added in
order to enable building Swift-based test inferiors.  We may be able to
rearrange things to make those language-specific additions more readily
pluggable in the core LLDB test runner.

We look forward to upstreaming the differences in common files in the
coming days and weeks.

Please feel free to contact me if you have any questions.

Thanks!

-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] LLDB and Swift

[Todd Fiala via lldb-dev]

Thanks, Kamil!

-Todd

> On Dec 3, 2015, at 5:02 PM, Kamil Rytarowski  wrote:
> 
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
> 
> Very nice. Congrats on your release!
> 
>> On 04.12.2015 00:03, Todd Fiala via lldb-dev wrote:
>> Hi all,
>> 
>> Earlier today, you may have heard that Swift went open source over 
>> at swift.org <http://swift.org/>.  I just wanted to take a moment
>> to mention the Swift debugger and REPL and how they relate to
>> LLDB.
>> 
>> Swift’s Debugger and REPL are built on LLDB’s source-level plug-in 
>> architecture.  As such, the Swift Debugger repository at
>> github.com/apple/swift-lldb <http://github.com/apple/swift-lldb>
>> naturally contains the LLDB source from llvm.org
>> <http://llvm.org/>’s LLDB repository, plus additions for Swift
>> language support. We merge regularly and make every attempt to 
>> minimize our differences with llvm.org <http://llvm.org/>’s LLDB.
>> For more information on how we’re handling this, have a look at
>> swift.org/contributing/#llvm-and-swift 
>> <http://swift.org/contributing/#llvm-and-swift>.
>> 
>> As we’ve worked hard to make it straightforward to develop
>> additive-only language support in LLDB, the Swift support can
>> readily be found by finding the new files in the swift-lldb
>> repository vs. those found at llvm.org/svn/llvm-project/lldb/trunk 
>> <http://llvm.org/svn/llvm-project/lldb/trunk/>.  For the rest of
>> the LLDB files in common, we do still have a small number of diffs 
>> in github.com/apple/swift-lldb <http://github.com/apple/swift-lldb>
>> vs. llvm.org <http://llvm.org/> TOT.  We will work through
>> upstreaming these quickly.  I’ll touch on some of those differences
>> briefly here:
>> 
>> * Several minor places where full language abstraction hasn’t yet 
>> occurred, where we’re explicitly checking for Swift-related
>> details. Abstracting out those remaining places and providing the
>> hooks in llvm.org <http://llvm.org/> LLDB will benefit all
>> languages.
>> 
>> * Printed-form version string handling.  The ‘lldb -v’ and ‘(lldb) 
>> version’ commands create a different version string in both Xcode
>> and cmake-based Swift LLDB.  We will work to incorporate this into
>> llvm.org <http://llvm.org/> LLDB once the language/component
>> version support info is properly abstracted out.
>> 
>> * Test infrastructure.  There are a few places where Swift
>> language support (e.g. swift compiler flags, runtime support
>> directories, etc.) are added in order to enable building
>> Swift-based test inferiors.  We may be able to rearrange things to
>> make those language-specific additions more readily pluggable in
>> the core LLDB test runner.
>> 
>> We look forward to upstreaming the differences in common files in
>> the coming days and weeks.
>> 
>> Please feel free to contact me if you have any questions.
>> 
>> Thanks!
>> 
>> -Todd
>> 
>> 
>> ___ lldb-dev mailing
>> list lldb-dev@lists.llvm.org 
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> 
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2
> 
> iQIcBAEBCAAGBQJWYOYqAAoJEEuzCOmwLnZsyVMP/0pC9Vahn0ErYFeBQoaXA3Qx
> FFr7dAIMAyC+tH0Rb2virNLsrozgNXAnV06vfS2fdt1nB2sVwFDeZvtscjRCY0BC
> w8v5N0joKb0Ao2RJcazCLJYOEihTc9thsBSQFDzQ3UIMJ5f5FIhykcSDecIh3OTQ
> Hb2FGzTsFbdRLQvu6XwagaxT0n5PL3IG7BIRVLgQl988ICJvGNFDub7/7Ylee52b
> oLtkxRhMMn9n2UXGPahQ6WozKfjc/l5s6isAp3bdkH4GEyTIv+D7/CKUmvLyZxaP
> L75JS0g/bb++uMY+2naKzCrTYm7Se2hopIvbvgf7vkTIrLBUZt8JtJ7qkKkiwTL3
> iW4oOiXUTz0pFQ6g2vdCGBM1263iPxS816JxLtW+aB4Gj/qhuzoTTseb7+KvFCVs
> 5PG87p7L6pm5TKswX+Cf6Di0O5fqyUFwk06hB9wuck6iCbT5dl4Zkty0OKsh/mnb
> RSztbQn9BpCbMDiZe2wv5y8H8kaMvaNvnODqNdK6C94M4km6AD7YNx0WFMPkPrL5
> IfLzGZau89ejrmJIU9SKW7HJRn+luIxjr2sa3BVGV+cZP4wUm9Z+d91Q6DPXwKv0
> MBdb7ISPGqW13yYHYJ9dK/pKFjHiMFMjIzBsMvItqvN3Xy3GmHDIm80G6jzu7Wj0
> VipucL5yeiHkTIJNccs8
> =qRiS
> -END PGP SIGNATURE-
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

That seems reasonable. I'll work that in.

-Todd

> On Dec 3, 2015, at 4:55 PM, Zachary Turner  wrote:
> 
> It would also be nice if the summary statistics were printed after the list 
> of failing / errored tests.  The reason is that it involves a fixed number of 
> lines to print the table, but the list of failures and errors is a variable 
> number of lines which could potentially be very long and push the statistics 
> off the screen.
> 
>> On Thu, Dec 3, 2015 at 10:08 AM Zachary Turner  wrote:
>> Ahh I read further and see this was already mentioned by Pavel.
>> 
>>> On Thu, Dec 3, 2015 at 10:06 AM Zachary Turner  wrote:
 On Wed, Dec 2, 2015 at 10:20 PM Todd Fiala  wrote:
> On Wed, Dec 2, 2015 at 9:48 PM, Zachary Turner  wrote:
> 
> 
>> On Wed, Dec 2, 2015 at 9:44 PM Todd Fiala  wrote:
>>  
>>> and the classname could be dropped (there's only one class per file 
>>> anyway, so the classname is just wasted space)
>> 
>> Part of the reason I included that is I've hit several times where copy 
>> and paste errors lead to the same class name, method name or even file 
>> name being used for a test.  I think, though, that most of those are 
>> addressed by having the path (relative is fine) to the python test file. 
>>  I think we can probably get by with classname.methodname (relative test 
>> path).  (From your other email, I think you nuke the classname and keep 
>> the module name, but I'd probably do the reverse, keeping the class name 
>> and getting rid of the module name since it can be derived from the 
>> filename).
> 
> I don't think the filename can be the same anymore, as things will break 
> if two filenames are the same.
 
 Maybe, but that wasn't my experience as of fairly recently.  When tracking 
 failures sometime within the last month, I tracked something down in a 
 downstream branch with two same-named files that (with the legacy output) 
 made it hard to track down what was actually failing given the limited 
 info of the legacy test summary output.  Maybe that has changed since 
 then, but I'm not aware of anything that would have prohibited that.
>>> 
>>> Well I only said "things" will break, not everything will break.  Most 
>>> likely you just didn't notice the problem or it didn't present itself in 
>>> your scenario.  There are definitely bugs surrounding multiple files with 
>>> the same name, because of some places where we use a dictionary keyed on 
>>> filename.
>>>  
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

One thing I excluded from the newer test results detail info is the
architecture.  I personally haven't ever needed that.  I'd be happy to
leave that out until we find someone who really needs it, just to keep it
shorter.

On Thu, Dec 3, 2015 at 5:14 PM, Todd Fiala  wrote:

> That seems reasonable. I'll work that in.
>
> -Todd
>
> On Dec 3, 2015, at 4:55 PM, Zachary Turner  wrote:
>
> It would also be nice if the summary statistics were printed after the
> list of failing / errored tests.  The reason is that it involves a fixed
> number of lines to print the table, but the list of failures and errors is
> a variable number of lines which could potentially be very long and push
> the statistics off the screen.
>
> On Thu, Dec 3, 2015 at 10:08 AM Zachary Turner  wrote:
>
>> Ahh I read further and see this was already mentioned by Pavel.
>>
>> On Thu, Dec 3, 2015 at 10:06 AM Zachary Turner 
>> wrote:
>>
>>> On Wed, Dec 2, 2015 at 10:20 PM Todd Fiala  wrote:
>>>
 On Wed, Dec 2, 2015 at 9:48 PM, Zachary Turner 
 wrote:

>
>
> On Wed, Dec 2, 2015 at 9:44 PM Todd Fiala 
> wrote:
>
>>
>>
>>> and the classname could be dropped (there's only one class per file
>>> anyway, so the classname is just wasted space)
>>>
>>
>> Part of the reason I included that is I've hit several times where
>> copy and paste errors lead to the same class name, method name or even 
>> file
>> name being used for a test.  I think, though, that most of those are
>> addressed by having the path (relative is fine) to the python test file. 
>>  I
>> think we can probably get by with classname.methodname (relative test
>> path).  (From your other email, I think you nuke the classname and keep 
>> the
>> module name, but I'd probably do the reverse, keeping the class name and
>> getting rid of the module name since it can be derived from the 
>> filename).
>>
> I don't think the filename can be the same anymore, as things will
> break if two filenames are the same.
>

 Maybe, but that wasn't my experience as of fairly recently.  When
 tracking failures sometime within the last month, I tracked something down
 in a downstream branch with two same-named files that (with the legacy
 output) made it hard to track down what was actually failing given the
 limited info of the legacy test summary output.  Maybe that has changed
 since then, but I'm not aware of anything that would have prohibited that.

>>> Well I only said "things" will break, not everything will break.  Most
>>> likely you just didn't notice the problem or it didn't present itself in
>>> your scenario.  There are definitely bugs surrounding multiple files with
>>> the same name, because of some places where we use a dictionary keyed on
>>> filename.
>>>
>>>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] New test summary results formatter

[Todd Fiala via lldb-dev]

Hi all,

r254890 moves the test summary counts to the end.  It also greatly cleans
up the issue detail line to be:

ISSUE_TYPE: test_method_name (test relative path)

I put a sample output in the revision comment.  I think it looks much
cleaner with the tweaks we discussed, and I really like the look of the
counts at the very end.

I'll work on getting the timeouts and exceptional exits fed into the
output, in addition to moving those two straggling counts from above the
issue details and moved into the summary counts.

After that, we can evaluate if we want to switch over to this as the
default output.  Then I can get the low-load, single worker test pass to be
added to the end of the test run.

-Todd



On Fri, Dec 4, 2015 at 5:33 PM, Todd Fiala  wrote:

> One thing I excluded from the newer test results detail info is the
> architecture.  I personally haven't ever needed that.  I'd be happy to
> leave that out until we find someone who really needs it, just to keep it
> shorter.
>
> On Thu, Dec 3, 2015 at 5:14 PM, Todd Fiala  wrote:
>
>> That seems reasonable. I'll work that in.
>>
>> -Todd
>>
>> On Dec 3, 2015, at 4:55 PM, Zachary Turner  wrote:
>>
>> It would also be nice if the summary statistics were printed after the
>> list of failing / errored tests.  The reason is that it involves a fixed
>> number of lines to print the table, but the list of failures and errors is
>> a variable number of lines which could potentially be very long and push
>> the statistics off the screen.
>>
>> On Thu, Dec 3, 2015 at 10:08 AM Zachary Turner 
>> wrote:
>>
>>> Ahh I read further and see this was already mentioned by Pavel.
>>>
>>> On Thu, Dec 3, 2015 at 10:06 AM Zachary Turner 
>>> wrote:
>>>
 On Wed, Dec 2, 2015 at 10:20 PM Todd Fiala 
 wrote:

> On Wed, Dec 2, 2015 at 9:48 PM, Zachary Turner 
> wrote:
>
>>
>>
>> On Wed, Dec 2, 2015 at 9:44 PM Todd Fiala 
>> wrote:
>>
>>>
>>>
 and the classname could be dropped (there's only one class per file
 anyway, so the classname is just wasted space)

>>>
>>> Part of the reason I included that is I've hit several times where
>>> copy and paste errors lead to the same class name, method name or even 
>>> file
>>> name being used for a test.  I think, though, that most of those are
>>> addressed by having the path (relative is fine) to the python test 
>>> file.  I
>>> think we can probably get by with classname.methodname (relative test
>>> path).  (From your other email, I think you nuke the classname and keep 
>>> the
>>> module name, but I'd probably do the reverse, keeping the class name and
>>> getting rid of the module name since it can be derived from the 
>>> filename).
>>>
>> I don't think the filename can be the same anymore, as things will
>> break if two filenames are the same.
>>
>
> Maybe, but that wasn't my experience as of fairly recently.  When
> tracking failures sometime within the last month, I tracked something down
> in a downstream branch with two same-named files that (with the legacy
> output) made it hard to track down what was actually failing given the
> limited info of the legacy test summary output.  Maybe that has changed
> since then, but I'm not aware of anything that would have prohibited that.
>
 Well I only said "things" will break, not everything will break.  Most
 likely you just didn't notice the problem or it didn't present itself in
 your scenario.  There are definitely bugs surrounding multiple files with
 the same name, because of some places where we use a dictionary keyed on
 filename.


>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] LLDB/NetBSD test suite results as of 8 Dec 2015

[Todd Fiala via lldb-dev]

Sounds like you're making some good progress, Kamil!

-Todd

On Mon, Dec 7, 2015 at 4:28 PM, Kamil Rytarowski via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> I ran the LLDB test suite of the LLDB-current version on NetBSD/amd64
> (v. 7.99.21).
>
> The results are as follows:
>
> 415 out of 415 test suites processed - TestRecursiveTypes.py
>
> Ran 415 test suites (266 failed) (64.096386%)
> Ran 222 test cases (145 failed) (65.315315%)
> Failing Tests (266)
>
> The major missing piece is native process tracking support (with the
> ptrace(2) interface). Once added, LLDB should start to be usable on
> this platform.
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v2
>
> iQIcBAEBCAAGBQJWZiQdAAoJEEuzCOmwLnZsHNUQALe8F61h9fSyaw0l6nWOIpHs
> THgeCWPKMWsf4z82wb4kFWY6TKroUkzl4BiizBUMs+m3fMJLSOMuTp6/kOCOiXM7
> DyRbCkVqXOn/qXg8rocf/mpYmxDRyWM+GcNmwCY/lXp6eYmB67PN4ymxb5mpc1z1
> 27L//+BsKW+EoGT5diTyDzVgpb9U2MZk2nR4lbimuJpmhzEgtFiMLT7GX6BCwoAA
> IWCMmO6FKiE5e5PiDoH8j+stMD5+NN2K6l6rS7JrTPdi0TwGSkIH8/DJL5Nob0lv
> lyr74B9vVIbsNk4gjzdOgOBOPIt8OVPL4m2GtBQcrCXNbsh88EZnUT+jIHCe3XjT
> p8lyrwVrzJBtijlYHJjbyRBzJtUoJ0fVFEiKI/S9Wdki77sQ1Mz48Mfa/+hgrnuW
> oc/qUyeNTTj9chjd3NpTQht5H3IF8xDIjqLmLdnYgDwnsZFd8BhHz3QGqbbP0Aa8
> jNy2NCF0+Gog1tF6xSOhBINXFF8qZJOiDOZleUc95wTz+Aw/0NLNqkq9JqPB0s6t
> KDM/ykNA0MtJD2b9u8tOFDqVQTJQdiwTKpg6BfdBxBE0R4O9zxb2XjP5fylU+y7Y
> TywBSauQ0UnqAe/nXCtWFHFEpzzuNt0D4RWzlO25WaehVbrbC+DyNuC9IKufYiGp
> uVIOFiM8loMJFCiQm+DL
> =Egm3
> -END PGP SIGNATURE-
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Auditing dotest's command line options

[Todd Fiala via lldb-dev]

I think it's a nice improvement.

Passing the options around via the argparse results (as I do in many
programs) makes it easier to unit test, but having configuration variables
all in a module makes it really simple to find and use everywhere without
having them as globals.

Thanks for cleaning that up, Zachary!

-Todd



On Tue, Dec 8, 2015 at 11:31 AM, Greg Clayton via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> Sounds good, looks good then.
>
> > On Dec 8, 2015, at 11:09 AM, Zachary Turner  wrote:
> >
> > One advantage of this approach is that it makes the options available to
> the entire test suite.  Even if we have no transferring going on, and we
> get argparse to return us a perfectly organized structure with everything
> in the right format, in order to make all the options accessible to the
> rest of the test suite, we still need to stick it in a global module
> somewhere.  And then you would write
> `configuration.options.test_categories`, whereas with this approach we just
> write `configuration.test_categories`.  It's a minor point, but I like the
> shorter member access personally.
> >
> > On Tue, Dec 8, 2015 at 11:07 AM Zachary Turner 
> wrote:
> > There's no way to avoid doing a transfer out of the options dictionary
> at some level, because it's not a straight transfer.  There's a ton of
> post-processing that gets done on the options dictionary in order to
> convert the raw options into a useful format.
> >
> > That might be solvable with more advanced use of argparse.  This
> approach does get rid of one level of option transfer though.  Because we
> would transfer
> > 1. From the class returned by argparse into the global
> > 2. From the global into the lldb module
> >
> > Now we only transfer from the argparse class into the `configuration`
> module, and everything else just uses that.
> >
> >
> > On Tue, Dec 8, 2015 at 10:52 AM Greg Clayton  wrote:
> > Do we not want to have an "options" global variable in this module that
> contains everything instead of having separate global variables in this
> file? The idea would be that you could assign directly when parsing
> arguments:
> >
> > (configuration.options, args) = parser.parse_args(sys.argv[1:])
> >
> > Its OK if we don't do this, but this is what I was originally thinking.
> Then we don't need to do any transfer out of the options dictionary that is
> returned by the option parser. The drawback with this approach is the
> "configuration.options" would probably need to be initialized in case
> someone tries to access the "configuration.options" without first parsing
> arguments. So in that respect the global approach is nicer.
> >
> > Greg
> >
> > > On Dec 8, 2015, at 10:45 AM, Zachary Turner 
> wrote:
> > >
> > > Hi Greg,
> > >
> > > Take a look at dotest.py next time you get some free time and let me
> know what you think.  There should be no more globals.  Everything that
> used to be a global is now stored in its own module `configuration.py`, and
> everything in `configuration.py` can be referenced from everywhere in the
> entire test suite.
> > >
> > > On Fri, Nov 20, 2015 at 10:34 AM Greg Clayton 
> wrote:
> > > Zach, I would also like to get rid of all global variables in the
> process of this change. The history goes like this: a long time ago someone
> wrote the initial dotest.py and parsed the options manually and stored
> results in global variables. Later, someone converted the options over to
> use a python library to parse the options, but we mostly copied the options
> from the options dictionary over into the globals and still use the globals
> all over the code. It would be great if we had at most one global variable
> that is something like "g_options" and anyone that was using any global
> variables will switch over to use the "g_options." instead. Then we
> don't have to make copies and we can let the g_options contain all settings
> that are required.
> > >
> > > > On Nov 18, 2015, at 2:32 PM, Zachary Turner via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
> > > >
> > > > I would like to do a complete audit of dotest's command line
> options, find out who's using what, and then potentially delete anything
> that isn't being used.  There's a mess of command line options in use, to
> the point that it's often hard to find free letters to use for new options.
> > > >
> > > > I created this spreadsheet with a complete list of command line
> options, their descriptions, and a place for people to enter what options
> they're using or do not want to be deleted.
> > > >
> > > >
> https://docs.google.com/spreadsheets/d/1wkxAY7l0_cJOHhhsSlh3aKKlQShlX1D7X1Dn8kpqxy4/edit?usp=sharing
> > > >
> > > > If someone has already written YES in the box that indicates they
> need the option, please don't overwrite it.  If you write YES in a box,
> please provide at least a small rationale for why this option is useful to
> you.  Feel free to add additional rationale if someone has already added
> some rationale.
> > > >
> > >

[lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

Hi all,

Per a previous thread on this, I've made all the changes I intended to make
last night to get the intended replacement of test run results meet or
exceed current requirements.

I'd like to switch over to that by default.  I'm depending on the test
event system to be able to handle test method reruns in test results
accounting.

The primary thing missing before was that timeouts were not routed through
the test events system, nor were exception process exits (i.e. test
inferiors exiting with a signal on POSIX systems).  Those were added last
night so that test events are generated for those, and the
BasicResultsFormatter presents that information properly.

I will switch it over to being the default output in a bit here.  Please
let me know if you have any concerns once I flip it on by default.

Thanks!
-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

That's a good point, Tamas.

I use (so I claim) the same all upper-case markers for the test result
details.  Including, not using XPASS but rather UNEXPECTED SUCCESS for
unexpected successes.  (The former would trigger the lit script IIRC to
parse that as a failing-style result).

The intent is this is a no-op on the test runner.

On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer 
wrote:

> +Ying Chen 
>
> Ying, what do we have to do on the build bot side to support a change in
> the default test result summary formatter?
>
> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> Hi all,
>>
>> Per a previous thread on this, I've made all the changes I intended to
>> make last night to get the intended replacement of test run results meet or
>> exceed current requirements.
>>
>> I'd like to switch over to that by default.  I'm depending on the test
>> event system to be able to handle test method reruns in test results
>> accounting.
>>
>> The primary thing missing before was that timeouts were not routed
>> through the test events system, nor were exception process exits (i.e. test
>> inferiors exiting with a signal on POSIX systems).  Those were added last
>> night so that test events are generated for those, and the
>> BasicResultsFormatter presents that information properly.
>>
>> I will switch it over to being the default output in a bit here.  Please
>> let me know if you have any concerns once I flip it on by default.
>>
>> Thanks!
>> --
>> -Todd
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

Specifically, the markers for issue details are:

FAIL
ERROR
UNEXPECTED SUCCESS
TIMEOUT

(These are the fourth field in the array entries (lines 275 - 290) of
packages/Python/lldbsuite/test/basic_results_formatter.py).

-Todd

On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala  wrote:

> That's a good point, Tamas.
>
> I use (so I claim) the same all upper-case markers for the test result
> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS for
> unexpected successes.  (The former would trigger the lit script IIRC to
> parse that as a failing-style result).
>
> The intent is this is a no-op on the test runner.
>
> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer 
> wrote:
>
>> +Ying Chen 
>>
>> Ying, what do we have to do on the build bot side to support a change in
>> the default test result summary formatter?
>>
>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> Hi all,
>>>
>>> Per a previous thread on this, I've made all the changes I intended to
>>> make last night to get the intended replacement of test run results meet or
>>> exceed current requirements.
>>>
>>> I'd like to switch over to that by default.  I'm depending on the test
>>> event system to be able to handle test method reruns in test results
>>> accounting.
>>>
>>> The primary thing missing before was that timeouts were not routed
>>> through the test events system, nor were exception process exits (i.e. test
>>> inferiors exiting with a signal on POSIX systems).  Those were added last
>>> night so that test events are generated for those, and the
>>> BasicResultsFormatter presents that information properly.
>>>
>>> I will switch it over to being the default output in a bit here.  Please
>>> let me know if you have any concerns once I flip it on by default.
>>>
>>> Thanks!
>>> --
>>> -Todd
>>> ___
>>> lldb-dev mailing list
>>> lldb-dev@lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>
>>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

Here's what I can do.

Put in the change (setting the default to use the new format).

Separately, put in a trial balloon commit with one failing test, one
exceptional exit test, and one timeout test, and watch the ubuntu 14.04
buildbot catch it and fail.  Then reverse this out.  That should show
beyond a reasonable doubt whether the buildbot catches new failures and
errors.  (I think this is a noisy way to accomplish this, but it certainly
would validate if its working).

-Todd

On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala  wrote:

> Specifically, the markers for issue details are:
>
> FAIL
> ERROR
> UNEXPECTED SUCCESS
> TIMEOUT
>
> (These are the fourth field in the array entries (lines 275 - 290) of
> packages/Python/lldbsuite/test/basic_results_formatter.py).
>
> -Todd
>
> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala  wrote:
>
>> That's a good point, Tamas.
>>
>> I use (so I claim) the same all upper-case markers for the test result
>> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS for
>> unexpected successes.  (The former would trigger the lit script IIRC to
>> parse that as a failing-style result).
>>
>> The intent is this is a no-op on the test runner.
>>
>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer 
>> wrote:
>>
>>> +Ying Chen 
>>>
>>> Ying, what do we have to do on the build bot side to support a change in
>>> the default test result summary formatter?
>>>
>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev <
>>> lldb-dev@lists.llvm.org> wrote:
>>>
>>>> Hi all,
>>>>
>>>> Per a previous thread on this, I've made all the changes I intended to
>>>> make last night to get the intended replacement of test run results meet or
>>>> exceed current requirements.
>>>>
>>>> I'd like to switch over to that by default.  I'm depending on the test
>>>> event system to be able to handle test method reruns in test results
>>>> accounting.
>>>>
>>>> The primary thing missing before was that timeouts were not routed
>>>> through the test events system, nor were exception process exits (i.e. test
>>>> inferiors exiting with a signal on POSIX systems).  Those were added last
>>>> night so that test events are generated for those, and the
>>>> BasicResultsFormatter presents that information properly.
>>>>
>>>> I will switch it over to being the default output in a bit here.
>>>> Please let me know if you have any concerns once I flip it on by default.
>>>>
>>>> Thanks!
>>>> --
>>>> -Todd
>>>> ___
>>>> lldb-dev mailing list
>>>> lldb-dev@lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>
>>>
>>
>>
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

It is a small change.

I almost have all the trial tests ready, so I'll just commit both changes
at the same time (the flip on, and the trial balloon issues).

If all goes well and the three types of issue show up, then the last of the
two will get reverted (the one with the failures).

If none (or only some) of the issues show up, they'll both get reverted.

-Todd

On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath  wrote:

> If it's not too much work, I think the extra bit of noise will not be
> a problem. But I don't think it is really necessary either.
>
> I assume the actual flip will be a small change that we can back out
> easily if we notice troubles... After a sufficient grace period we can
> remove the old formatter altogether and hopefully simplify the code
> somewhat.
>
> pl
>
> On 9 December 2015 at 17:08, Todd Fiala  wrote:
> > Here's what I can do.
> >
> > Put in the change (setting the default to use the new format).
> >
> > Separately, put in a trial balloon commit with one failing test, one
> > exceptional exit test, and one timeout test, and watch the ubuntu 14.04
> > buildbot catch it and fail.  Then reverse this out.  That should show
> beyond
> > a reasonable doubt whether the buildbot catches new failures and
> errors.  (I
> > think this is a noisy way to accomplish this, but it certainly would
> > validate if its working).
> >
> > -Todd
> >
> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala  wrote:
> >>
> >> Specifically, the markers for issue details are:
> >>
> >> FAIL
> >> ERROR
> >> UNEXPECTED SUCCESS
> >> TIMEOUT
> >>
> >> (These are the fourth field in the array entries (lines 275 - 290) of
> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
> >>
> >> -Todd
> >>
> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala 
> wrote:
> >>>
> >>> That's a good point, Tamas.
> >>>
> >>> I use (so I claim) the same all upper-case markers for the test result
> >>> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS for
> >>> unexpected successes.  (The former would trigger the lit script IIRC to
> >>> parse that as a failing-style result).
> >>>
> >>> The intent is this is a no-op on the test runner.
> >>>
> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
> tbergham...@google.com>
> >>> wrote:
> >>>>
> >>>> +Ying Chen
> >>>>
> >>>> Ying, what do we have to do on the build bot side to support a change
> in
> >>>> the default test result summary formatter?
> >>>>
> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
> >>>>  wrote:
> >>>>>
> >>>>> Hi all,
> >>>>>
> >>>>> Per a previous thread on this, I've made all the changes I intended
> to
> >>>>> make last night to get the intended replacement of test run results
> meet or
> >>>>> exceed current requirements.
> >>>>>
> >>>>> I'd like to switch over to that by default.  I'm depending on the
> test
> >>>>> event system to be able to handle test method reruns in test results
> >>>>> accounting.
> >>>>>
> >>>>> The primary thing missing before was that timeouts were not routed
> >>>>> through the test events system, nor were exception process exits
> (i.e. test
> >>>>> inferiors exiting with a signal on POSIX systems).  Those were added
> last
> >>>>> night so that test events are generated for those, and the
> >>>>> BasicResultsFormatter presents that information properly.
> >>>>>
> >>>>> I will switch it over to being the default output in a bit here.
> >>>>> Please let me know if you have any concerns once I flip it on by
> default.
> >>>>>
> >>>>> Thanks!
> >>>>> --
> >>>>> -Todd
> >>>>> ___
> >>>>> lldb-dev mailing list
> >>>>> lldb-dev@lists.llvm.org
> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> -Todd
> >>
> >>
> >>
> >>
> >> --
> >> -Todd
> >
> >
> >
> >
> > --
> > -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

These went in as:

r255130 - turn it on by default
r255131 - create known issues.  This one is to be reverted if all 3 types
show up properly.

On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala  wrote:

> It is a small change.
>
> I almost have all the trial tests ready, so I'll just commit both changes
> at the same time (the flip on, and the trial balloon issues).
>
> If all goes well and the three types of issue show up, then the last of
> the two will get reverted (the one with the failures).
>
> If none (or only some) of the issues show up, they'll both get reverted.
>
> -Todd
>
> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath  wrote:
>
>> If it's not too much work, I think the extra bit of noise will not be
>> a problem. But I don't think it is really necessary either.
>>
>> I assume the actual flip will be a small change that we can back out
>> easily if we notice troubles... After a sufficient grace period we can
>> remove the old formatter altogether and hopefully simplify the code
>> somewhat.
>>
>> pl
>>
>> On 9 December 2015 at 17:08, Todd Fiala  wrote:
>> > Here's what I can do.
>> >
>> > Put in the change (setting the default to use the new format).
>> >
>> > Separately, put in a trial balloon commit with one failing test, one
>> > exceptional exit test, and one timeout test, and watch the ubuntu 14.04
>> > buildbot catch it and fail.  Then reverse this out.  That should show
>> beyond
>> > a reasonable doubt whether the buildbot catches new failures and
>> errors.  (I
>> > think this is a noisy way to accomplish this, but it certainly would
>> > validate if its working).
>> >
>> > -Todd
>> >
>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala 
>> wrote:
>> >>
>> >> Specifically, the markers for issue details are:
>> >>
>> >> FAIL
>> >> ERROR
>> >> UNEXPECTED SUCCESS
>> >> TIMEOUT
>> >>
>> >> (These are the fourth field in the array entries (lines 275 - 290) of
>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>> >>
>> >> -Todd
>> >>
>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala 
>> wrote:
>> >>>
>> >>> That's a good point, Tamas.
>> >>>
>> >>> I use (so I claim) the same all upper-case markers for the test result
>> >>> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS for
>> >>> unexpected successes.  (The former would trigger the lit script IIRC
>> to
>> >>> parse that as a failing-style result).
>> >>>
>> >>> The intent is this is a no-op on the test runner.
>> >>>
>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
>> tbergham...@google.com>
>> >>> wrote:
>> >>>>
>> >>>> +Ying Chen
>> >>>>
>> >>>> Ying, what do we have to do on the build bot side to support a
>> change in
>> >>>> the default test result summary formatter?
>> >>>>
>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
>> >>>>  wrote:
>> >>>>>
>> >>>>> Hi all,
>> >>>>>
>> >>>>> Per a previous thread on this, I've made all the changes I intended
>> to
>> >>>>> make last night to get the intended replacement of test run results
>> meet or
>> >>>>> exceed current requirements.
>> >>>>>
>> >>>>> I'd like to switch over to that by default.  I'm depending on the
>> test
>> >>>>> event system to be able to handle test method reruns in test results
>> >>>>> accounting.
>> >>>>>
>> >>>>> The primary thing missing before was that timeouts were not routed
>> >>>>> through the test events system, nor were exception process exits
>> (i.e. test
>> >>>>> inferiors exiting with a signal on POSIX systems).  Those were
>> added last
>> >>>>> night so that test events are generated for those, and the
>> >>>>> BasicResultsFormatter presents that information properly.
>> >>>>>
>> >>>>> I will switch it over to being the default output in a bit here.
>> >>>>> Please let me know if you have any concerns once I flip it on by
>> default.
>> >>>>>
>> >>>>> Thanks!
>> >>>>> --
>> >>>>> -Todd
>> >>>>> ___
>> >>>>> lldb-dev mailing list
>> >>>>> lldb-dev@lists.llvm.org
>> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> -Todd
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> -Todd
>> >
>> >
>> >
>> >
>> > --
>> > -Todd
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

I forced a build on the ubuntu 14.04 cmake builder.  The build _after_ 9292
will contain the two changes (and we will expect failures on it).

On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala  wrote:

> These went in as:
>
> r255130 - turn it on by default
> r255131 - create known issues.  This one is to be reverted if all 3 types
> show up properly.
>
> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala  wrote:
>
>> It is a small change.
>>
>> I almost have all the trial tests ready, so I'll just commit both changes
>> at the same time (the flip on, and the trial balloon issues).
>>
>> If all goes well and the three types of issue show up, then the last of
>> the two will get reverted (the one with the failures).
>>
>> If none (or only some) of the issues show up, they'll both get reverted.
>>
>> -Todd
>>
>> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath  wrote:
>>
>>> If it's not too much work, I think the extra bit of noise will not be
>>> a problem. But I don't think it is really necessary either.
>>>
>>> I assume the actual flip will be a small change that we can back out
>>> easily if we notice troubles... After a sufficient grace period we can
>>> remove the old formatter altogether and hopefully simplify the code
>>> somewhat.
>>>
>>> pl
>>>
>>> On 9 December 2015 at 17:08, Todd Fiala  wrote:
>>> > Here's what I can do.
>>> >
>>> > Put in the change (setting the default to use the new format).
>>> >
>>> > Separately, put in a trial balloon commit with one failing test, one
>>> > exceptional exit test, and one timeout test, and watch the ubuntu 14.04
>>> > buildbot catch it and fail.  Then reverse this out.  That should show
>>> beyond
>>> > a reasonable doubt whether the buildbot catches new failures and
>>> errors.  (I
>>> > think this is a noisy way to accomplish this, but it certainly would
>>> > validate if its working).
>>> >
>>> > -Todd
>>> >
>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala 
>>> wrote:
>>> >>
>>> >> Specifically, the markers for issue details are:
>>> >>
>>> >> FAIL
>>> >> ERROR
>>> >> UNEXPECTED SUCCESS
>>> >> TIMEOUT
>>> >>
>>> >> (These are the fourth field in the array entries (lines 275 - 290) of
>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>>> >>
>>> >> -Todd
>>> >>
>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala 
>>> wrote:
>>> >>>
>>> >>> That's a good point, Tamas.
>>> >>>
>>> >>> I use (so I claim) the same all upper-case markers for the test
>>> result
>>> >>> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS
>>> for
>>> >>> unexpected successes.  (The former would trigger the lit script IIRC
>>> to
>>> >>> parse that as a failing-style result).
>>> >>>
>>> >>> The intent is this is a no-op on the test runner.
>>> >>>
>>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
>>> tbergham...@google.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> +Ying Chen
>>> >>>>
>>> >>>> Ying, what do we have to do on the build bot side to support a
>>> change in
>>> >>>> the default test result summary formatter?
>>> >>>>
>>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
>>> >>>>  wrote:
>>> >>>>>
>>> >>>>> Hi all,
>>> >>>>>
>>> >>>>> Per a previous thread on this, I've made all the changes I
>>> intended to
>>> >>>>> make last night to get the intended replacement of test run
>>> results meet or
>>> >>>>> exceed current requirements.
>>> >>>>>
>>> >>>>> I'd like to switch over to that by default.  I'm depending on the
>>> test
>>> >>>>> event system to be able to handle test method reruns in test
>>> results
>>> >>>>> accounting.
>>> >>>>>
>>> >>>>> The primary thing missing before was that timeouts were not routed
>>> >>>>> through the test events system, nor were exception process exits
>>> (i.e. test
>>> >>>>> inferiors exiting with a signal on POSIX systems).  Those were
>>> added last
>>> >>>>> night so that test events are generated for those, and the
>>> >>>>> BasicResultsFormatter presents that information properly.
>>> >>>>>
>>> >>>>> I will switch it over to being the default output in a bit here.
>>> >>>>> Please let me know if you have any concerns once I flip it on by
>>> default.
>>> >>>>>
>>> >>>>> Thanks!
>>> >>>>> --
>>> >>>>> -Todd
>>> >>>>> ___
>>> >>>>> lldb-dev mailing list
>>> >>>>> lldb-dev@lists.llvm.org
>>> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> --
>>> >>> -Todd
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> -Todd
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

I am going to stop the current build on that builder.  There was one change
in it, and it will be another 20 minutes before it completes.  I don't want
the repo in a known broken state that long.

On Wed, Dec 9, 2015 at 10:07 AM, Todd Fiala  wrote:

> I forced a build on the ubuntu 14.04 cmake builder.  The build _after_
> 9292 will contain the two changes (and we will expect failures on it).
>
> On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala  wrote:
>
>> These went in as:
>>
>> r255130 - turn it on by default
>> r255131 - create known issues.  This one is to be reverted if all 3 types
>> show up properly.
>>
>> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala  wrote:
>>
>>> It is a small change.
>>>
>>> I almost have all the trial tests ready, so I'll just commit both
>>> changes at the same time (the flip on, and the trial balloon issues).
>>>
>>> If all goes well and the three types of issue show up, then the last of
>>> the two will get reverted (the one with the failures).
>>>
>>> If none (or only some) of the issues show up, they'll both get reverted.
>>>
>>> -Todd
>>>
>>> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath  wrote:
>>>
>>>> If it's not too much work, I think the extra bit of noise will not be
>>>> a problem. But I don't think it is really necessary either.
>>>>
>>>> I assume the actual flip will be a small change that we can back out
>>>> easily if we notice troubles... After a sufficient grace period we can
>>>> remove the old formatter altogether and hopefully simplify the code
>>>> somewhat.
>>>>
>>>> pl
>>>>
>>>> On 9 December 2015 at 17:08, Todd Fiala  wrote:
>>>> > Here's what I can do.
>>>> >
>>>> > Put in the change (setting the default to use the new format).
>>>> >
>>>> > Separately, put in a trial balloon commit with one failing test, one
>>>> > exceptional exit test, and one timeout test, and watch the ubuntu
>>>> 14.04
>>>> > buildbot catch it and fail.  Then reverse this out.  That should show
>>>> beyond
>>>> > a reasonable doubt whether the buildbot catches new failures and
>>>> errors.  (I
>>>> > think this is a noisy way to accomplish this, but it certainly would
>>>> > validate if its working).
>>>> >
>>>> > -Todd
>>>> >
>>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala 
>>>> wrote:
>>>> >>
>>>> >> Specifically, the markers for issue details are:
>>>> >>
>>>> >> FAIL
>>>> >> ERROR
>>>> >> UNEXPECTED SUCCESS
>>>> >> TIMEOUT
>>>> >>
>>>> >> (These are the fourth field in the array entries (lines 275 - 290) of
>>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>>>> >>
>>>> >> -Todd
>>>> >>
>>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala 
>>>> wrote:
>>>> >>>
>>>> >>> That's a good point, Tamas.
>>>> >>>
>>>> >>> I use (so I claim) the same all upper-case markers for the test
>>>> result
>>>> >>> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS
>>>> for
>>>> >>> unexpected successes.  (The former would trigger the lit script
>>>> IIRC to
>>>> >>> parse that as a failing-style result).
>>>> >>>
>>>> >>> The intent is this is a no-op on the test runner.
>>>> >>>
>>>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
>>>> tbergham...@google.com>
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> +Ying Chen
>>>> >>>>
>>>> >>>> Ying, what do we have to do on the build bot side to support a
>>>> change in
>>>> >>>> the default test result summary formatter?
>>>> >>>>
>>>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
>>>> >>>>  wrote:
>>>> >>>>>
>>>> >>>>> Hi all,
>>>> >>>>>
>>>> >>>&

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

The reports look good at the test level:

http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9294

I'd say the buildbot reflection script missed the ERROR, so that is
something maybe Ying can look at (the summary line in the build run), but
that is unrelated AFAICT.

I'm going to move aside the failures.

On Wed, Dec 9, 2015 at 10:13 AM, Todd Fiala  wrote:

> I am going to stop the current build on that builder.  There was one
> change in it, and it will be another 20 minutes before it completes.  I
> don't want the repo in a known broken state that long.
>
> On Wed, Dec 9, 2015 at 10:07 AM, Todd Fiala  wrote:
>
>> I forced a build on the ubuntu 14.04 cmake builder.  The build _after_
>> 9292 will contain the two changes (and we will expect failures on it).
>>
>> On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala  wrote:
>>
>>> These went in as:
>>>
>>> r255130 - turn it on by default
>>> r255131 - create known issues.  This one is to be reverted if all 3
>>> types show up properly.
>>>
>>> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala  wrote:
>>>
>>>> It is a small change.
>>>>
>>>> I almost have all the trial tests ready, so I'll just commit both
>>>> changes at the same time (the flip on, and the trial balloon issues).
>>>>
>>>> If all goes well and the three types of issue show up, then the last of
>>>> the two will get reverted (the one with the failures).
>>>>
>>>> If none (or only some) of the issues show up, they'll both get reverted.
>>>>
>>>> -Todd
>>>>
>>>> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath  wrote:
>>>>
>>>>> If it's not too much work, I think the extra bit of noise will not be
>>>>> a problem. But I don't think it is really necessary either.
>>>>>
>>>>> I assume the actual flip will be a small change that we can back out
>>>>> easily if we notice troubles... After a sufficient grace period we can
>>>>> remove the old formatter altogether and hopefully simplify the code
>>>>> somewhat.
>>>>>
>>>>> pl
>>>>>
>>>>> On 9 December 2015 at 17:08, Todd Fiala  wrote:
>>>>> > Here's what I can do.
>>>>> >
>>>>> > Put in the change (setting the default to use the new format).
>>>>> >
>>>>> > Separately, put in a trial balloon commit with one failing test, one
>>>>> > exceptional exit test, and one timeout test, and watch the ubuntu
>>>>> 14.04
>>>>> > buildbot catch it and fail.  Then reverse this out.  That should
>>>>> show beyond
>>>>> > a reasonable doubt whether the buildbot catches new failures and
>>>>> errors.  (I
>>>>> > think this is a noisy way to accomplish this, but it certainly would
>>>>> > validate if its working).
>>>>> >
>>>>> > -Todd
>>>>> >
>>>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala 
>>>>> wrote:
>>>>> >>
>>>>> >> Specifically, the markers for issue details are:
>>>>> >>
>>>>> >> FAIL
>>>>> >> ERROR
>>>>> >> UNEXPECTED SUCCESS
>>>>> >> TIMEOUT
>>>>> >>
>>>>> >> (These are the fourth field in the array entries (lines 275 - 290)
>>>>> of
>>>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>>>>> >>
>>>>> >> -Todd
>>>>> >>
>>>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala 
>>>>> wrote:
>>>>> >>>
>>>>> >>> That's a good point, Tamas.
>>>>> >>>
>>>>> >>> I use (so I claim) the same all upper-case markers for the test
>>>>> result
>>>>> >>> details.  Including, not using XPASS but rather UNEXPECTED SUCCESS
>>>>> for
>>>>> >>> unexpected successes.  (The former would trigger the lit script
>>>>> IIRC to
>>>>> >>> parse that as a failing-style result).
>>>>> >>>
>>>>> >>> The intent is this is a no-op on the test runner.
>>>>> >>>
>>>>> 

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

Verification tests parked (i.e. disabled) here:
r255134

I decided to leave them in the repo so it is faster/easier to do this in
the future.

-Todd

On Wed, Dec 9, 2015 at 10:26 AM, Todd Fiala  wrote:

> The reports look good at the test level:
>
>
> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake/builds/9294
>
> I'd say the buildbot reflection script missed the ERROR, so that is
> something maybe Ying can look at (the summary line in the build run), but
> that is unrelated AFAICT.
>
> I'm going to move aside the failures.
>
> On Wed, Dec 9, 2015 at 10:13 AM, Todd Fiala  wrote:
>
>> I am going to stop the current build on that builder.  There was one
>> change in it, and it will be another 20 minutes before it completes.  I
>> don't want the repo in a known broken state that long.
>>
>> On Wed, Dec 9, 2015 at 10:07 AM, Todd Fiala  wrote:
>>
>>> I forced a build on the ubuntu 14.04 cmake builder.  The build _after_
>>> 9292 will contain the two changes (and we will expect failures on it).
>>>
>>> On Wed, Dec 9, 2015 at 10:05 AM, Todd Fiala 
>>> wrote:
>>>
>>>> These went in as:
>>>>
>>>> r255130 - turn it on by default
>>>> r255131 - create known issues.  This one is to be reverted if all 3
>>>> types show up properly.
>>>>
>>>> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala 
>>>> wrote:
>>>>
>>>>> It is a small change.
>>>>>
>>>>> I almost have all the trial tests ready, so I'll just commit both
>>>>> changes at the same time (the flip on, and the trial balloon issues).
>>>>>
>>>>> If all goes well and the three types of issue show up, then the last
>>>>> of the two will get reverted (the one with the failures).
>>>>>
>>>>> If none (or only some) of the issues show up, they'll both get
>>>>> reverted.
>>>>>
>>>>> -Todd
>>>>>
>>>>> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath 
>>>>> wrote:
>>>>>
>>>>>> If it's not too much work, I think the extra bit of noise will not be
>>>>>> a problem. But I don't think it is really necessary either.
>>>>>>
>>>>>> I assume the actual flip will be a small change that we can back out
>>>>>> easily if we notice troubles... After a sufficient grace period we can
>>>>>> remove the old formatter altogether and hopefully simplify the code
>>>>>> somewhat.
>>>>>>
>>>>>> pl
>>>>>>
>>>>>> On 9 December 2015 at 17:08, Todd Fiala  wrote:
>>>>>> > Here's what I can do.
>>>>>> >
>>>>>> > Put in the change (setting the default to use the new format).
>>>>>> >
>>>>>> > Separately, put in a trial balloon commit with one failing test, one
>>>>>> > exceptional exit test, and one timeout test, and watch the ubuntu
>>>>>> 14.04
>>>>>> > buildbot catch it and fail.  Then reverse this out.  That should
>>>>>> show beyond
>>>>>> > a reasonable doubt whether the buildbot catches new failures and
>>>>>> errors.  (I
>>>>>> > think this is a noisy way to accomplish this, but it certainly would
>>>>>> > validate if its working).
>>>>>> >
>>>>>> > -Todd
>>>>>> >
>>>>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala 
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Specifically, the markers for issue details are:
>>>>>> >>
>>>>>> >> FAIL
>>>>>> >> ERROR
>>>>>> >> UNEXPECTED SUCCESS
>>>>>> >> TIMEOUT
>>>>>> >>
>>>>>> >> (These are the fourth field in the array entries (lines 275 - 290)
>>>>>> of
>>>>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>>>>>> >>
>>>>>> >> -Todd
>>>>>> >>
>>>>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala 
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> That's a good point, 

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

t; > -Todd
>>>>>>> >
>>>>>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala 
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> Specifically, the markers for issue details are:
>>>>>>> >>
>>>>>>> >> FAIL
>>>>>>> >> ERROR
>>>>>>> >> UNEXPECTED SUCCESS
>>>>>>> >> TIMEOUT
>>>>>>> >>
>>>>>>> >> (These are the fourth field in the array entries (lines 275 -
>>>>>>> 290) of
>>>>>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>>>>>>> >>
>>>>>>> >> -Todd
>>>>>>> >>
>>>>>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala 
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> That's a good point, Tamas.
>>>>>>> >>>
>>>>>>> >>> I use (so I claim) the same all upper-case markers for the test
>>>>>>> result
>>>>>>> >>> details.  Including, not using XPASS but rather UNEXPECTED
>>>>>>> SUCCESS for
>>>>>>> >>> unexpected successes.  (The former would trigger the lit script
>>>>>>> IIRC to
>>>>>>> >>> parse that as a failing-style result).
>>>>>>> >>>
>>>>>>> >>> The intent is this is a no-op on the test runner.
>>>>>>> >>>
>>>>>>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
>>>>>>> tbergham...@google.com>
>>>>>>> >>> wrote:
>>>>>>> >>>>
>>>>>>> >>>> +Ying Chen
>>>>>>> >>>>
>>>>>>> >>>> Ying, what do we have to do on the build bot side to support a
>>>>>>> change in
>>>>>>> >>>> the default test result summary formatter?
>>>>>>> >>>>
>>>>>>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
>>>>>>> >>>>  wrote:
>>>>>>> >>>>>
>>>>>>> >>>>> Hi all,
>>>>>>> >>>>>
>>>>>>> >>>>> Per a previous thread on this, I've made all the changes I
>>>>>>> intended to
>>>>>>> >>>>> make last night to get the intended replacement of test run
>>>>>>> results meet or
>>>>>>> >>>>> exceed current requirements.
>>>>>>> >>>>>
>>>>>>> >>>>> I'd like to switch over to that by default.  I'm depending on
>>>>>>> the test
>>>>>>> >>>>> event system to be able to handle test method reruns in test
>>>>>>> results
>>>>>>> >>>>> accounting.
>>>>>>> >>>>>
>>>>>>> >>>>> The primary thing missing before was that timeouts were not
>>>>>>> routed
>>>>>>> >>>>> through the test events system, nor were exception process
>>>>>>> exits (i.e. test
>>>>>>> >>>>> inferiors exiting with a signal on POSIX systems).  Those were
>>>>>>> added last
>>>>>>> >>>>> night so that test events are generated for those, and the
>>>>>>> >>>>> BasicResultsFormatter presents that information properly.
>>>>>>> >>>>>
>>>>>>> >>>>> I will switch it over to being the default output in a bit
>>>>>>> here.
>>>>>>> >>>>> Please let me know if you have any concerns once I flip it on
>>>>>>> by default.
>>>>>>> >>>>>
>>>>>>> >>>>> Thanks!
>>>>>>> >>>>> --
>>>>>>> >>>>> -Todd
>>>>>>> >>>>> ___
>>>>>>> >>>>> lldb-dev mailing list
>>>>>>> >>>>> lldb-dev@lists.llvm.org
>>>>>>> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> >>> --
>>>>>>> >>> -Todd
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>
>>>>>>> >> --
>>>>>>> >> -Todd
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > -Todd
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>
>>
>> --
>> -Todd
>>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Benchmark tests

[Todd Fiala via lldb-dev]

Hey Jason,

Are you the benchmark user?

-Todd

On Wed, Dec 9, 2015 at 12:32 PM, Zachary Turner via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> Is anyone using the benchmark tests?  None of the command line options
> 
> related to the benchmark tests were claimed as being used by anyone.  Which
> makes me wonder if the tests are even being used by anyone.
>
> What I really want to know is: Is it really ok to delete the -x and -y
> command line options?  And what is the status of these tests?  Does anyone
> use them?
>
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

>>>>>>>>
>>>>>>>> On Wed, Dec 9, 2015 at 9:41 AM, Todd Fiala 
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> It is a small change.
>>>>>>>>>
>>>>>>>>> I almost have all the trial tests ready, so I'll just commit both
>>>>>>>>> changes at the same time (the flip on, and the trial balloon issues).
>>>>>>>>>
>>>>>>>>> If all goes well and the three types of issue show up, then the
>>>>>>>>> last of the two will get reverted (the one with the failures).
>>>>>>>>>
>>>>>>>>> If none (or only some) of the issues show up, they'll both get
>>>>>>>>> reverted.
>>>>>>>>>
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>> On Wed, Dec 9, 2015 at 9:39 AM, Pavel Labath 
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> If it's not too much work, I think the extra bit of noise will
>>>>>>>>>> not be
>>>>>>>>>> a problem. But I don't think it is really necessary either.
>>>>>>>>>>
>>>>>>>>>> I assume the actual flip will be a small change that we can back
>>>>>>>>>> out
>>>>>>>>>> easily if we notice troubles... After a sufficient grace period
>>>>>>>>>> we can
>>>>>>>>>> remove the old formatter altogether and hopefully simplify the
>>>>>>>>>> code
>>>>>>>>>> somewhat.
>>>>>>>>>>
>>>>>>>>>> pl
>>>>>>>>>>
>>>>>>>>>> On 9 December 2015 at 17:08, Todd Fiala 
>>>>>>>>>> wrote:
>>>>>>>>>> > Here's what I can do.
>>>>>>>>>> >
>>>>>>>>>> > Put in the change (setting the default to use the new format).
>>>>>>>>>> >
>>>>>>>>>> > Separately, put in a trial balloon commit with one failing
>>>>>>>>>> test, one
>>>>>>>>>> > exceptional exit test, and one timeout test, and watch the
>>>>>>>>>> ubuntu 14.04
>>>>>>>>>> > buildbot catch it and fail.  Then reverse this out.  That
>>>>>>>>>> should show beyond
>>>>>>>>>> > a reasonable doubt whether the buildbot catches new failures
>>>>>>>>>> and errors.  (I
>>>>>>>>>> > think this is a noisy way to accomplish this, but it certainly
>>>>>>>>>> would
>>>>>>>>>> > validate if its working).
>>>>>>>>>> >
>>>>>>>>>> > -Todd
>>>>>>>>>> >
>>>>>>>>>> > On Wed, Dec 9, 2015 at 8:06 AM, Todd Fiala <
>>>>>>>>>> todd.fi...@gmail.com> wrote:
>>>>>>>>>> >>
>>>>>>>>>> >> Specifically, the markers for issue details are:
>>>>>>>>>> >>
>>>>>>>>>> >> FAIL
>>>>>>>>>> >> ERROR
>>>>>>>>>> >> UNEXPECTED SUCCESS
>>>>>>>>>> >> TIMEOUT
>>>>>>>>>> >>
>>>>>>>>>> >> (These are the fourth field in the array entries (lines 275 -
>>>>>>>>>> 290) of
>>>>>>>>>> >> packages/Python/lldbsuite/test/basic_results_formatter.py).
>>>>>>>>>> >>
>>>>>>>>>> >> -Todd
>>>>>>>>>> >>
>>>>>>>>>> >> On Wed, Dec 9, 2015 at 8:04 AM, Todd Fiala <
>>>>>>>>>> todd.fi...@gmail.com> wrote:
>>>>>>>>>> >>>
>>>>>>>>>> >>> That's a good point, Tamas.
>>>>>>>>>> >>>
>>>>>>>>>> >>> I use (so I claim) the same all upper-case markers for the
>>>>>>>>>> test result
>>>>>>>>>> >>> details.  Including, not using XPASS but rather UNEXPECTED
>>>>>>>>>> SUCCESS for
>>>>>>>>>> >>> unexpected successes.  (The former would trigger the lit
>>>>>>>>>> script IIRC to
>>>>>>>>>> >>> parse that as a failing-style result).
>>>>>>>>>> >>>
>>>>>>>>>> >>> The intent is this is a no-op on the test runner.
>>>>>>>>>> >>>
>>>>>>>>>> >>> On Wed, Dec 9, 2015 at 8:02 AM, Tamas Berghammer <
>>>>>>>>>> tbergham...@google.com>
>>>>>>>>>> >>> wrote:
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> +Ying Chen
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> Ying, what do we have to do on the build bot side to support
>>>>>>>>>> a change in
>>>>>>>>>> >>>> the default test result summary formatter?
>>>>>>>>>> >>>>
>>>>>>>>>> >>>> On Wed, Dec 9, 2015 at 4:00 PM Todd Fiala via lldb-dev
>>>>>>>>>> >>>>  wrote:
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> Hi all,
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> Per a previous thread on this, I've made all the changes I
>>>>>>>>>> intended to
>>>>>>>>>> >>>>> make last night to get the intended replacement of test run
>>>>>>>>>> results meet or
>>>>>>>>>> >>>>> exceed current requirements.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> I'd like to switch over to that by default.  I'm depending
>>>>>>>>>> on the test
>>>>>>>>>> >>>>> event system to be able to handle test method reruns in
>>>>>>>>>> test results
>>>>>>>>>> >>>>> accounting.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> The primary thing missing before was that timeouts were not
>>>>>>>>>> routed
>>>>>>>>>> >>>>> through the test events system, nor were exception process
>>>>>>>>>> exits (i.e. test
>>>>>>>>>> >>>>> inferiors exiting with a signal on POSIX systems).  Those
>>>>>>>>>> were added last
>>>>>>>>>> >>>>> night so that test events are generated for those, and the
>>>>>>>>>> >>>>> BasicResultsFormatter presents that information properly.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> I will switch it over to being the default output in a bit
>>>>>>>>>> here.
>>>>>>>>>> >>>>> Please let me know if you have any concerns once I flip it
>>>>>>>>>> on by default.
>>>>>>>>>> >>>>>
>>>>>>>>>> >>>>> Thanks!
>>>>>>>>>> >>>>> --
>>>>>>>>>> >>>>> -Todd
>>>>>>>>>> >>>>> ___
>>>>>>>>>> >>>>> lldb-dev mailing list
>>>>>>>>>> >>>>> lldb-dev@lists.llvm.org
>>>>>>>>>> >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>>
>>>>>>>>>> >>> --
>>>>>>>>>> >>> -Todd
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >>
>>>>>>>>>> >> --
>>>>>>>>>> >> -Todd
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > -Todd
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> -Todd
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> -Todd
>>>>>
>>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

Sure, I can do that.

Tamas, okay to give more detail on -v?  I'll give it a shot to see what
else comes out if we do that.

-Todd

On Thu, Dec 10, 2015 at 12:58 PM, Zachary Turner  wrote:

>
>
> On Thu, Dec 10, 2015 at 12:54 PM Todd Fiala  wrote:
>
>> Hi Tamas,
>>
>>
>>
>> On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer > > wrote:
>>
>>> HI Todd,
>>>
>>> You changed the way the test failure list is printed in a way that now
>>> we only print the name of the test function failing with the name of the
>>> test file in parenthesis. Can we add back the name of the test class to
>>> this list?
>>>
>>
>> Sure.  I originally planned to have that in there but there was some
>> discussion about it being too much info.  I'm happy to add that back.
>>
> Can we have it tied to verbosity level?  We have -t and -v, maybe one of
> those could trigger more detail in the summary view.
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

Checked this in as r255310.  Let me know if you find any issues with that,
Tamas.

You will need '-v' to enable it.  Otherwise, it will just print the method
name.

-Todd

On Thu, Dec 10, 2015 at 2:39 PM, Todd Fiala  wrote:

> Sure, I can do that.
>
> Tamas, okay to give more detail on -v?  I'll give it a shot to see what
> else comes out if we do that.
>
> -Todd
>
> On Thu, Dec 10, 2015 at 12:58 PM, Zachary Turner 
> wrote:
>
>>
>>
>> On Thu, Dec 10, 2015 at 12:54 PM Todd Fiala  wrote:
>>
>>> Hi Tamas,
>>>
>>>
>>>
>>> On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer <
>>> tbergham...@google.com> wrote:
>>>
 HI Todd,

 You changed the way the test failure list is printed in a way that now
 we only print the name of the test function failing with the name of the
 test file in parenthesis. Can we add back the name of the test class to
 this list?

>>>
>>> Sure.  I originally planned to have that in there but there was some
>>> discussion about it being too much info.  I'm happy to add that back.
>>>
>> Can we have it tied to verbosity level?  We have -t and -v, maybe one of
>> those could trigger more detail in the summary view.
>>
>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Separating test runner and tests

[Todd Fiala via lldb-dev]

I'm fine with the idea.

FWIW the test events model will likely shift a bit, as it is currently a
single sink, whereas I am likely to turn it into a test event filter chain
shortly here.  Formatters still make sense as they'll be the things at the
end of the chain.

Minor detail, result_formatter.py should be results_formatter.py - they are
ResultsFormatter instances (plural on Results since it transforms a series
of results into coherent reported output).  I'll rename that at some point
in the near future, but if you shift a number of things around, you can do
that.

I'm just about done with the multi-pass running.  I expect to get an opt-in
version of that running end of day today or worst case on Sunday.  It would
be awesome if you can hold off on any significant change like that until
this little bit is done as I'm sure we'll collide, particularly since this
hits dosep.py pretty significantly.

Thanks!

-Todd

On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
lldb-dev@lists.llvm.org> wrote:

> Sounds like a reasonable thing to do. A couple of tiny remarks:
> - when you do the move, you might as well rename dotest into something
> else, just to avoid the "which dotest should I run" type of
> questions...
> - there is nothing that makes it obvious that "engine" is actually a
> "test running engine", as it sits in a sibling folder. OTOH,
> "test_engine" might be too verbose, and messes up tab completion, so
> that might not be a good idea either...
>
> pl
>
>
> On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
>  wrote:
> > Currently our folder structure looks like this:
> >
> > lldbsuite
> > |-- test
> > |-- dotest.py
> > |-- dosep.py
> > |-- lldbtest.py
> > |-- ...
> > |-- functionalities
> > |-- lang
> > |-- expression_command
> > |-- ...
> > etc
> >
> > I've been thinking about organizing it like this instead:
> >
> > lldbsuite
> > |-- test
> > |-- functionalities
> > |-- lang
> > |-- expression_command
> > |-- ...
> > |-- engine
> > |-- dotest.py
> > |-- dosep.py
> > |-- lldbtest.py
> > |-- ...
> >
> > Anybody have any thoughts on this?  Good idea or bad idea?  The main
> reason
> > I want to do this is because as we start breaking up some of the code, it
> > makes sense to start having some subpackages under the `engine` folder
> (or
> > the `test` folder in our current world).  For example, Todd and I have
> > discussed the idea of putting formatter related stuff under a
> `formatters`
> > subpackage.  In the current world, there's no way to differentiate
> between
> > folders which contain tests and folders which contain test
> infrastructure,
> > so when we walk the directory tree looking for tests we end up walking a
> > bunch of directories that are used for test infrastructure code and not
> > actual tests.  So I like the logical separation this provides -- having
> the
> > tests themselves all under a single subpackage.
> >
> > Thoughts?
> >
> > ___
> > lldb-dev mailing list
> > lldb-dev@lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
> >
> ___
> lldb-dev mailing list
> lldb-dev@lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

On Fri, Dec 11, 2015 at 3:26 AM, Pavel Labath  wrote:

> Todd, I've had to disable the new result formatter as it was not
> working with the expected timeout logic we have for the old one. The
> old XTIMEOUT code is a massive hack and I will be extremely glad when
> we get rid of it, but we can't keep our buildbot red until then, so
> I've switched it off.
>
>
Ah, sorry my comments on the check-in precede me reading this.  Glad you
see this as a hack :-)

No worries on shutting it off.  I can get the expected timeout as currently
written working with the updated summary results.


> I am ready to start working on this, but I wanted to run this idea
> here first. I thought we could have a test annotation like:
> @expectedTimeout(oslist=["linux"], ...)
>
> Then, when the child runner would encounter this annotation, it would
> set a flag in the "test is starting" message indicating that this test
> may time out. Then if the test really times out, the parent would know
> about this, and it could avoid flagging the test as error.
>
>
Yes, the idea seems reasonable.  The actual implementation will end up
being slightly different as the ResultsFormatter will receive the test
start event (where the timeout is expected comes from), whereas the
reporter of the timeout (the test worker) will not know anything about that
data.  It will still generate the timeout, but then the ResultsFormatter
can deal with transforming this into the right event when a timeout is
"okay".


> Alternatively, if we want to avoid the proliferation test result
> states, we could key this off the standard @expectedFailure
> annotation, then a "time out" would become just another way it which a
> test can fail, and XTIMEOUT would become XFAIL.
>
> What do you think ?
>
>
Even though the above would work, if the issue here ultimately is that a
larger timeout is needed, we can avoid all this by increasing the timeout.
Probably more effective, though, is going to be running it in the
follow-up, low-load, single worker pass, where presumably we would not hit
the timeout.  If you think that would work, I'd say:

(1) short term (like in the next hour or so), I get the expected timeout
working in the summary results.

(2) longer term (like by end of weekend or maybe Monday at worst), we have
the second pass test run at lower load (i.e. single worker thread), which
should prevent these things from timing out in the first place.

If the analysis of the cause of the timeout is incorrect, then really we'll
want to do your initial proposal in the earlier paragraphs, though.

What do you think about any of that?




> pl
>
> PS: I am pretty new to this part of code, so any pointers you have
> towards implementing this would be extremely helpful.
>
>
>
> On 10 December 2015 at 23:20, Todd Fiala  wrote:
> > Checked this in as r255310.  Let me know if you find any issues with
> that,
> > Tamas.
> >
> > You will need '-v' to enable it.  Otherwise, it will just print the
> method
> > name.
> >
> > -Todd
> >
> > On Thu, Dec 10, 2015 at 2:39 PM, Todd Fiala 
> wrote:
> >>
> >> Sure, I can do that.
> >>
> >> Tamas, okay to give more detail on -v?  I'll give it a shot to see what
> >> else comes out if we do that.
> >>
> >> -Todd
> >>
> >> On Thu, Dec 10, 2015 at 12:58 PM, Zachary Turner 
> >> wrote:
> >>>
> >>>
> >>>
> >>> On Thu, Dec 10, 2015 at 12:54 PM Todd Fiala 
> wrote:
> 
>  Hi Tamas,
> 
> 
> 
>  On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer
>   wrote:
> >
> > HI Todd,
> >
> > You changed the way the test failure list is printed in a way that
> now
> > we only print the name of the test function failing with the name of
> the
> > test file in parenthesis. Can we add back the name of the test class
> to this
> > list?
> 
> 
>  Sure.  I originally planned to have that in there but there was some
>  discussion about it being too much info.  I'm happy to add that back.
> >>>
> >>> Can we have it tied to verbosity level?  We have -t and -v, maybe one
> of
> >>> those could trigger more detail in the summary view.
> >>
> >>
> >>
> >>
> >> --
> >> -Todd
> >
> >
> >
> >
> > --
> > -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

Merging threads.

> The concept is not there to protect against timeouts, which are caused
by processes being too slow, for these we have been increasing
timeouts where necessary.

Okay, I see.  If that's the intent, then expected timeout sounds
reasonable.  (My abhorrence was against the idea of using that as a
replacement for increasing a timeout that was too short under load).

I would go with your original approach (the marking as expected timeout).
We can either have that generate a new event (much like a change I'm about
to put in that has flakey tests send and event indicating that they are
eligible for rerun) or annotate the start message.  FWIW, the startTest()
call on the LLDBTestResults gets called before decorators have a chance to
execute, which is why I'm going with the 'send an enabling event' approach.
 (I'll be checking that in shortly here, like when I'm done writing this
email, so you'll see what I did there).

On Fri, Dec 11, 2015 at 9:41 AM, Todd Fiala  wrote:

>
>
> On Fri, Dec 11, 2015 at 3:26 AM, Pavel Labath  wrote:
>
>> Todd, I've had to disable the new result formatter as it was not
>> working with the expected timeout logic we have for the old one. The
>> old XTIMEOUT code is a massive hack and I will be extremely glad when
>> we get rid of it, but we can't keep our buildbot red until then, so
>> I've switched it off.
>>
>>
> Ah, sorry my comments on the check-in precede me reading this.  Glad you
> see this as a hack :-)
>
> No worries on shutting it off.  I can get the expected timeout as
> currently written working with the updated summary results.
>
>
>> I am ready to start working on this, but I wanted to run this idea
>> here first. I thought we could have a test annotation like:
>> @expectedTimeout(oslist=["linux"], ...)
>>
>> Then, when the child runner would encounter this annotation, it would
>> set a flag in the "test is starting" message indicating that this test
>> may time out. Then if the test really times out, the parent would know
>> about this, and it could avoid flagging the test as error.
>>
>>
> Yes, the idea seems reasonable.  The actual implementation will end up
> being slightly different as the ResultsFormatter will receive the test
> start event (where the timeout is expected comes from), whereas the
> reporter of the timeout (the test worker) will not know anything about that
> data.  It will still generate the timeout, but then the ResultsFormatter
> can deal with transforming this into the right event when a timeout is
> "okay".
>
>
>> Alternatively, if we want to avoid the proliferation test result
>> states, we could key this off the standard @expectedFailure
>> annotation, then a "time out" would become just another way it which a
>> test can fail, and XTIMEOUT would become XFAIL.
>>
>> What do you think ?
>>
>>
> Even though the above would work, if the issue here ultimately is that a
> larger timeout is needed, we can avoid all this by increasing the timeout.
> Probably more effective, though, is going to be running it in the
> follow-up, low-load, single worker pass, where presumably we would not hit
> the timeout.  If you think that would work, I'd say:
>
> (1) short term (like in the next hour or so), I get the expected timeout
> working in the summary results.
>
> (2) longer term (like by end of weekend or maybe Monday at worst), we have
> the second pass test run at lower load (i.e. single worker thread), which
> should prevent these things from timing out in the first place.
>
> If the analysis of the cause of the timeout is incorrect, then really
> we'll want to do your initial proposal in the earlier paragraphs, though.
>
> What do you think about any of that?
>
>
>
>
>> pl
>>
>> PS: I am pretty new to this part of code, so any pointers you have
>> towards implementing this would be extremely helpful.
>>
>>
>>
>> On 10 December 2015 at 23:20, Todd Fiala  wrote:
>> > Checked this in as r255310.  Let me know if you find any issues with
>> that,
>> > Tamas.
>> >
>> > You will need '-v' to enable it.  Otherwise, it will just print the
>> method
>> > name.
>> >
>> > -Todd
>> >
>> > On Thu, Dec 10, 2015 at 2:39 PM, Todd Fiala 
>> wrote:
>> >>
>> >> Sure, I can do that.
>> >>
>> >> Tamas, okay to give more detail on -v?  I'll give it a shot to see what
>> >> else comes out if we do that.
>> >>
>> >> -Todd
>> >>
>> >> On Thu, Dec 10, 2015 at 12:58 PM, Zachary Turner 
>> >> wrote:
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Dec 10, 2015 at 12:54 PM Todd Fiala 
>> wrote:
>> 
>>  Hi Tamas,
>> 
>> 
>> 
>>  On Thu, Dec 10, 2015 at 2:52 AM, Tamas Berghammer
>>   wrote:
>> >
>> > HI Todd,
>> >
>> > You changed the way the test failure list is printed in a way that
>> now
>> > we only print the name of the test function failing with the name
>> of the
>> > test file in parenthesis. Can we add back the name of the test
>> class to this
>> > list?
>> 
>> 
>>  Sure.  I originally planned to have t

Re: [lldb-dev] BasicResultsFormatter - new test results summary

[Todd Fiala via lldb-dev]

 >  (btw, I haven't checked, is it possible to XFAIL crashes now?

This currently doesn't work.  We'd need a smarter rerun mechanism
(something I do intend to do at some point), where we (1) know all the
tests that should run from a given test file before any are run, and (2)
when a timeout or exceptional exit takes out the whole file's worth of
tests, if there are any other tests that didn't get to run, we rerun just
those.

That is out of scope for the second pass low-load, single worker work I'm
doing now.  But it is something I'd like to have in the future.  The first
phase of that would be to list all the tests that didn't run because of a
timeout or exceptional exit.  The second phase is to enable only running a
named set of tests from a test file, and then rerunning ones that never had
a chance to run.  (That I could imagine rolling into the implementation of
the second test run pass fairly nicely).

On Fri, Dec 11, 2015 at 9:54 AM, Todd Fiala  wrote:

> Merging threads.
>
> > The concept is not there to protect against timeouts, which are caused
> by processes being too slow, for these we have been increasing
> timeouts where necessary.
>
> Okay, I see.  If that's the intent, then expected timeout sounds
> reasonable.  (My abhorrence was against the idea of using that as a
> replacement for increasing a timeout that was too short under load).
>
> I would go with your original approach (the marking as expected timeout).
> We can either have that generate a new event (much like a change I'm about
> to put in that has flakey tests send and event indicating that they are
> eligible for rerun) or annotate the start message.  FWIW, the startTest()
> call on the LLDBTestResults gets called before decorators have a chance to
> execute, which is why I'm going with the 'send an enabling event' approach.
>  (I'll be checking that in shortly here, like when I'm done writing this
> email, so you'll see what I did there).
>
> On Fri, Dec 11, 2015 at 9:41 AM, Todd Fiala  wrote:
>
>>
>>
>> On Fri, Dec 11, 2015 at 3:26 AM, Pavel Labath  wrote:
>>
>>> Todd, I've had to disable the new result formatter as it was not
>>> working with the expected timeout logic we have for the old one. The
>>> old XTIMEOUT code is a massive hack and I will be extremely glad when
>>> we get rid of it, but we can't keep our buildbot red until then, so
>>> I've switched it off.
>>>
>>>
>> Ah, sorry my comments on the check-in precede me reading this.  Glad you
>> see this as a hack :-)
>>
>> No worries on shutting it off.  I can get the expected timeout as
>> currently written working with the updated summary results.
>>
>>
>>> I am ready to start working on this, but I wanted to run this idea
>>> here first. I thought we could have a test annotation like:
>>> @expectedTimeout(oslist=["linux"], ...)
>>>
>>> Then, when the child runner would encounter this annotation, it would
>>> set a flag in the "test is starting" message indicating that this test
>>> may time out. Then if the test really times out, the parent would know
>>> about this, and it could avoid flagging the test as error.
>>>
>>>
>> Yes, the idea seems reasonable.  The actual implementation will end up
>> being slightly different as the ResultsFormatter will receive the test
>> start event (where the timeout is expected comes from), whereas the
>> reporter of the timeout (the test worker) will not know anything about that
>> data.  It will still generate the timeout, but then the ResultsFormatter
>> can deal with transforming this into the right event when a timeout is
>> "okay".
>>
>>
>>> Alternatively, if we want to avoid the proliferation test result
>>> states, we could key this off the standard @expectedFailure
>>> annotation, then a "time out" would become just another way it which a
>>> test can fail, and XTIMEOUT would become XFAIL.
>>>
>>> What do you think ?
>>>
>>>
>> Even though the above would work, if the issue here ultimately is that a
>> larger timeout is needed, we can avoid all this by increasing the timeout.
>> Probably more effective, though, is going to be running it in the
>> follow-up, low-load, single worker pass, where presumably we would not hit
>> the timeout.  If you think that would work, I'd say:
>>
>> (1) short term (like in the next hour or so), I get the expected timeout
>> working in the summary results.
>>
>> (2) longer term (like by end of weekend or maybe Monday at worst), we
>> have the second pass test run at lower load (i.e. single worker thread),
>> which should prevent these things from timing out in the first place.
>>
>> If the analysis of the cause of the timeout is incorrect, then really
>> we'll want to do your initial proposal in the earlier paragraphs, though.
>>
>> What do you think about any of that?
>>
>>
>>
>>
>>> pl
>>>
>>> PS: I am pretty new to this part of code, so any pointers you have
>>> towards implementing this would be extremely helpful.
>>>
>>>
>>>
>>> On 10 December 2015 at 23:20, Todd Fiala  wrote:
>>> > Checke

Re: [lldb-dev] Separating test runner and tests

[Todd Fiala via lldb-dev]

I like it.

On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner  wrote:

> Yea wasn't planning on doing this today, just throwing the idea out there.
>
> On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala  wrote:
>
>> I'm fine with the idea.
>>
>> FWIW the test events model will likely shift a bit, as it is currently a
>> single sink, whereas I am likely to turn it into a test event filter chain
>> shortly here.  Formatters still make sense as they'll be the things at the
>> end of the chain.
>>
>> Minor detail, result_formatter.py should be results_formatter.py - they
>> are ResultsFormatter instances (plural on Results since it transforms a
>> series of results into coherent reported output).  I'll rename that at some
>> point in the near future, but if you shift a number of things around, you
>> can do that.
>>
>> I'm just about done with the multi-pass running.  I expect to get an
>> opt-in version of that running end of day today or worst case on Sunday.
>> It would be awesome if you can hold off on any significant change like that
>> until this little bit is done as I'm sure we'll collide, particularly since
>> this hits dosep.py pretty significantly.
>>
>> Thanks!
>>
>> -Todd
>>
>> On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> Sounds like a reasonable thing to do. A couple of tiny remarks:
>>> - when you do the move, you might as well rename dotest into something
>>> else, just to avoid the "which dotest should I run" type of
>>> questions...
>>> - there is nothing that makes it obvious that "engine" is actually a
>>> "test running engine", as it sits in a sibling folder. OTOH,
>>> "test_engine" might be too verbose, and messes up tab completion, so
>>> that might not be a good idea either...
>>>
>>> pl
>>>
>>>
>>> On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
>>>  wrote:
>>> > Currently our folder structure looks like this:
>>> >
>>> > lldbsuite
>>> > |-- test
>>> > |-- dotest.py
>>> > |-- dosep.py
>>> > |-- lldbtest.py
>>> > |-- ...
>>> > |-- functionalities
>>> > |-- lang
>>> > |-- expression_command
>>> > |-- ...
>>> > etc
>>> >
>>> > I've been thinking about organizing it like this instead:
>>> >
>>> > lldbsuite
>>> > |-- test
>>> > |-- functionalities
>>> > |-- lang
>>> > |-- expression_command
>>> > |-- ...
>>> > |-- engine
>>> > |-- dotest.py
>>> > |-- dosep.py
>>> > |-- lldbtest.py
>>> > |-- ...
>>> >
>>> > Anybody have any thoughts on this?  Good idea or bad idea?  The main
>>> reason
>>> > I want to do this is because as we start breaking up some of the code,
>>> it
>>> > makes sense to start having some subpackages under the `engine` folder
>>> (or
>>> > the `test` folder in our current world).  For example, Todd and I have
>>> > discussed the idea of putting formatter related stuff under a
>>> `formatters`
>>> > subpackage.  In the current world, there's no way to differentiate
>>> between
>>> > folders which contain tests and folders which contain test
>>> infrastructure,
>>> > so when we walk the directory tree looking for tests we end up walking
>>> a
>>> > bunch of directories that are used for test infrastructure code and not
>>> > actual tests.  So I like the logical separation this provides --
>>> having the
>>> > tests themselves all under a single subpackage.
>>> >
>>> > Thoughts?
>>> >
>>> > ___
>>> > lldb-dev mailing list
>>> > lldb-dev@lists.llvm.org
>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>> >
>>> ___
>>> lldb-dev mailing list
>>> lldb-dev@lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>>>
>>
>>
>>
>> --
>> -Todd
>>
>


-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

[lldb-dev] marking new summary output for expected timeouts

[Todd Fiala via lldb-dev]

Hi Pavel,

I'm going to adjust the new summary output for expected timeouts.  I hope
to do that in the next hour or less.  I'll put that in and flip the default
back on for using the new summary output.

I'll do those two changes separately, so you can revert the flip back on to
flip it back off if we still have an issue.

Sound good?

(This can be orthogonal to the new work to mark up expected timeouts).
-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Separating test runner and tests

[Todd Fiala via lldb-dev]

One thing I want to make sure we can do is have a sane way of storing and
running tests that  test the test execution engine.  Those are tests that
should not run as part of an "lldb test run".  These are tests that
maintainers of the test system run to make sure we're not breaking stuff
when we touch the test system.

I would be writing more of those if I had a semi-sane way of doing it.
 (Part of the reason I broke out the python-based timeout logic the way I
did, before the major packaging changes, was so I had an obvious spot to
add tests for the process runner logic).

On Fri, Dec 11, 2015 at 10:03 AM, Todd Fiala  wrote:

> I like it.
>
> On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner 
> wrote:
>
>> Yea wasn't planning on doing this today, just throwing the idea out there.
>>
>> On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala  wrote:
>>
>>> I'm fine with the idea.
>>>
>>> FWIW the test events model will likely shift a bit, as it is currently a
>>> single sink, whereas I am likely to turn it into a test event filter chain
>>> shortly here.  Formatters still make sense as they'll be the things at the
>>> end of the chain.
>>>
>>> Minor detail, result_formatter.py should be results_formatter.py - they
>>> are ResultsFormatter instances (plural on Results since it transforms a
>>> series of results into coherent reported output).  I'll rename that at some
>>> point in the near future, but if you shift a number of things around, you
>>> can do that.
>>>
>>> I'm just about done with the multi-pass running.  I expect to get an
>>> opt-in version of that running end of day today or worst case on Sunday.
>>> It would be awesome if you can hold off on any significant change like that
>>> until this little bit is done as I'm sure we'll collide, particularly since
>>> this hits dosep.py pretty significantly.
>>>
>>> Thanks!
>>>
>>> -Todd
>>>
>>> On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
>>> lldb-dev@lists.llvm.org> wrote:
>>>
 Sounds like a reasonable thing to do. A couple of tiny remarks:
 - when you do the move, you might as well rename dotest into something
 else, just to avoid the "which dotest should I run" type of
 questions...
 - there is nothing that makes it obvious that "engine" is actually a
 "test running engine", as it sits in a sibling folder. OTOH,
 "test_engine" might be too verbose, and messes up tab completion, so
 that might not be a good idea either...

 pl


 On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
  wrote:
 > Currently our folder structure looks like this:
 >
 > lldbsuite
 > |-- test
 > |-- dotest.py
 > |-- dosep.py
 > |-- lldbtest.py
 > |-- ...
 > |-- functionalities
 > |-- lang
 > |-- expression_command
 > |-- ...
 > etc
 >
 > I've been thinking about organizing it like this instead:
 >
 > lldbsuite
 > |-- test
 > |-- functionalities
 > |-- lang
 > |-- expression_command
 > |-- ...
 > |-- engine
 > |-- dotest.py
 > |-- dosep.py
 > |-- lldbtest.py
 > |-- ...
 >
 > Anybody have any thoughts on this?  Good idea or bad idea?  The main
 reason
 > I want to do this is because as we start breaking up some of the
 code, it
 > makes sense to start having some subpackages under the `engine`
 folder (or
 > the `test` folder in our current world).  For example, Todd and I have
 > discussed the idea of putting formatter related stuff under a
 `formatters`
 > subpackage.  In the current world, there's no way to differentiate
 between
 > folders which contain tests and folders which contain test
 infrastructure,
 > so when we walk the directory tree looking for tests we end up
 walking a
 > bunch of directories that are used for test infrastructure code and
 not
 > actual tests.  So I like the logical separation this provides --
 having the
 > tests themselves all under a single subpackage.
 >
 > Thoughts?
 >
 > ___
 > lldb-dev mailing list
 > lldb-dev@lists.llvm.org
 > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
 >
 ___
 lldb-dev mailing list
 lldb-dev@lists.llvm.org
 http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

>>>
>>>
>>>
>>> --
>>> -Todd
>>>
>>
>
>
> --
> -Todd
>



-- 
-Todd
___
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

Re: [lldb-dev] Separating test runner and tests

[Todd Fiala via lldb-dev]

Unittest.

Comes with Python.

On Fri, Dec 11, 2015 at 11:07 AM, Zachary Turner  wrote:

> Presumably those tests use an entirely different, hand-rolled test running
> infrastructure?
>
> On Fri, Dec 11, 2015 at 10:52 AM Todd Fiala  wrote:
>
>> One thing I want to make sure we can do is have a sane way of storing and
>> running tests that  test the test execution engine.  Those are tests that
>> should not run as part of an "lldb test run".  These are tests that
>> maintainers of the test system run to make sure we're not breaking stuff
>> when we touch the test system.
>>
>> I would be writing more of those if I had a semi-sane way of doing it.
>>  (Part of the reason I broke out the python-based timeout logic the way I
>> did, before the major packaging changes, was so I had an obvious spot to
>> add tests for the process runner logic).
>>
>> On Fri, Dec 11, 2015 at 10:03 AM, Todd Fiala 
>> wrote:
>>
>>> I like it.
>>>
>>> On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner 
>>> wrote:
>>>
 Yea wasn't planning on doing this today, just throwing the idea out
 there.

 On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala 
 wrote:

> I'm fine with the idea.
>
> FWIW the test events model will likely shift a bit, as it is currently
> a single sink, whereas I am likely to turn it into a test event filter
> chain shortly here.  Formatters still make sense as they'll be the things
> at the end of the chain.
>
> Minor detail, result_formatter.py should be results_formatter.py -
> they are ResultsFormatter instances (plural on Results since it transforms
> a series of results into coherent reported output).  I'll rename that at
> some point in the near future, but if you shift a number of things around,
> you can do that.
>
> I'm just about done with the multi-pass running.  I expect to get an
> opt-in version of that running end of day today or worst case on Sunday.
> It would be awesome if you can hold off on any significant change like 
> that
> until this little bit is done as I'm sure we'll collide, particularly 
> since
> this hits dosep.py pretty significantly.
>
> Thanks!
>
> -Todd
>
> On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
> lldb-dev@lists.llvm.org> wrote:
>
>> Sounds like a reasonable thing to do. A couple of tiny remarks:
>> - when you do the move, you might as well rename dotest into something
>> else, just to avoid the "which dotest should I run" type of
>> questions...
>> - there is nothing that makes it obvious that "engine" is actually a
>> "test running engine", as it sits in a sibling folder. OTOH,
>> "test_engine" might be too verbose, and messes up tab completion, so
>> that might not be a good idea either...
>>
>> pl
>>
>>
>> On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
>>  wrote:
>> > Currently our folder structure looks like this:
>> >
>> > lldbsuite
>> > |-- test
>> > |-- dotest.py
>> > |-- dosep.py
>> > |-- lldbtest.py
>> > |-- ...
>> > |-- functionalities
>> > |-- lang
>> > |-- expression_command
>> > |-- ...
>> > etc
>> >
>> > I've been thinking about organizing it like this instead:
>> >
>> > lldbsuite
>> > |-- test
>> > |-- functionalities
>> > |-- lang
>> > |-- expression_command
>> > |-- ...
>> > |-- engine
>> > |-- dotest.py
>> > |-- dosep.py
>> > |-- lldbtest.py
>> > |-- ...
>> >
>> > Anybody have any thoughts on this?  Good idea or bad idea?  The
>> main reason
>> > I want to do this is because as we start breaking up some of the
>> code, it
>> > makes sense to start having some subpackages under the `engine`
>> folder (or
>> > the `test` folder in our current world).  For example, Todd and I
>> have
>> > discussed the idea of putting formatter related stuff under a
>> `formatters`
>> > subpackage.  In the current world, there's no way to differentiate
>> between
>> > folders which contain tests and folders which contain test
>> infrastructure,
>> > so when we walk the directory tree looking for tests we end up
>> walking a
>> > bunch of directories that are used for test infrastructure code and
>> not
>> > actual tests.  So I like the logical separation this provides --
>> having the
>> > tests themselves all under a single subpackage.
>> >
>> > Thoughts?
>> >
>> > ___
>> > lldb-dev mailing list
>> > lldb-dev@lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
>> >
>> ___
>> lldb-dev mailing list
>> lldb-dev@lists.llvm.org
>> http://lists.llvm.org/cgi-bin/ma

Re: [lldb-dev] Separating test runner and tests

[Todd Fiala via lldb-dev]

It just requires running the test file as a python script.

The runner is fired off like this:

if __name__ == "__main__":
unittest.main()

which is typically added to the bottom of all test files so you can call it
directly.

-Todd

On Fri, Dec 11, 2015 at 11:12 AM, Todd Fiala  wrote:

> Unittest.
>
> Comes with Python.
>
> On Fri, Dec 11, 2015 at 11:07 AM, Zachary Turner 
> wrote:
>
>> Presumably those tests use an entirely different, hand-rolled test
>> running infrastructure?
>>
>> On Fri, Dec 11, 2015 at 10:52 AM Todd Fiala  wrote:
>>
>>> One thing I want to make sure we can do is have a sane way of storing
>>> and running tests that  test the test execution engine.  Those are tests
>>> that should not run as part of an "lldb test run".  These are tests that
>>> maintainers of the test system run to make sure we're not breaking stuff
>>> when we touch the test system.
>>>
>>> I would be writing more of those if I had a semi-sane way of doing it.
>>>  (Part of the reason I broke out the python-based timeout logic the way I
>>> did, before the major packaging changes, was so I had an obvious spot to
>>> add tests for the process runner logic).
>>>
>>> On Fri, Dec 11, 2015 at 10:03 AM, Todd Fiala 
>>> wrote:
>>>
 I like it.

 On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner 
 wrote:

> Yea wasn't planning on doing this today, just throwing the idea out
> there.
>
> On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala 
> wrote:
>
>> I'm fine with the idea.
>>
>> FWIW the test events model will likely shift a bit, as it is
>> currently a single sink, whereas I am likely to turn it into a test event
>> filter chain shortly here.  Formatters still make sense as they'll be the
>> things at the end of the chain.
>>
>> Minor detail, result_formatter.py should be results_formatter.py -
>> they are ResultsFormatter instances (plural on Results since it 
>> transforms
>> a series of results into coherent reported output).  I'll rename that at
>> some point in the near future, but if you shift a number of things 
>> around,
>> you can do that.
>>
>> I'm just about done with the multi-pass running.  I expect to get an
>> opt-in version of that running end of day today or worst case on Sunday.
>> It would be awesome if you can hold off on any significant change like 
>> that
>> until this little bit is done as I'm sure we'll collide, particularly 
>> since
>> this hits dosep.py pretty significantly.
>>
>> Thanks!
>>
>> -Todd
>>
>> On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
>> lldb-dev@lists.llvm.org> wrote:
>>
>>> Sounds like a reasonable thing to do. A couple of tiny remarks:
>>> - when you do the move, you might as well rename dotest into
>>> something
>>> else, just to avoid the "which dotest should I run" type of
>>> questions...
>>> - there is nothing that makes it obvious that "engine" is actually a
>>> "test running engine", as it sits in a sibling folder. OTOH,
>>> "test_engine" might be too verbose, and messes up tab completion, so
>>> that might not be a good idea either...
>>>
>>> pl
>>>
>>>
>>> On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
>>>  wrote:
>>> > Currently our folder structure looks like this:
>>> >
>>> > lldbsuite
>>> > |-- test
>>> > |-- dotest.py
>>> > |-- dosep.py
>>> > |-- lldbtest.py
>>> > |-- ...
>>> > |-- functionalities
>>> > |-- lang
>>> > |-- expression_command
>>> > |-- ...
>>> > etc
>>> >
>>> > I've been thinking about organizing it like this instead:
>>> >
>>> > lldbsuite
>>> > |-- test
>>> > |-- functionalities
>>> > |-- lang
>>> > |-- expression_command
>>> > |-- ...
>>> > |-- engine
>>> > |-- dotest.py
>>> > |-- dosep.py
>>> > |-- lldbtest.py
>>> > |-- ...
>>> >
>>> > Anybody have any thoughts on this?  Good idea or bad idea?  The
>>> main reason
>>> > I want to do this is because as we start breaking up some of the
>>> code, it
>>> > makes sense to start having some subpackages under the `engine`
>>> folder (or
>>> > the `test` folder in our current world).  For example, Todd and I
>>> have
>>> > discussed the idea of putting formatter related stuff under a
>>> `formatters`
>>> > subpackage.  In the current world, there's no way to differentiate
>>> between
>>> > folders which contain tests and folders which contain test
>>> infrastructure,
>>> > so when we walk the directory tree looking for tests we end up
>>> walking a
>>> > bunch of directories that are used for test infrastructure code
>>> and not
>>> > actual tests.  So I like the logical separation this provides --
>>> having 

Re: [lldb-dev] Separating test runner and tests

[Todd Fiala via lldb-dev]

The tests end up looking substantially similar to our lldb test suite
tests, as they were based on unittest2, which is/was a relative of unittest
that now lives in Python.  The docs for unittest in python 2.x have
generally been accurate for the unittest2 lib we use.  At least, for the
areas I use.

On Fri, Dec 11, 2015 at 11:13 AM, Todd Fiala  wrote:

> It just requires running the test file as a python script.
>
> The runner is fired off like this:
>
> if __name__ == "__main__":
> unittest.main()
>
> which is typically added to the bottom of all test files so you can call
> it directly.
>
> -Todd
>
> On Fri, Dec 11, 2015 at 11:12 AM, Todd Fiala  wrote:
>
>> Unittest.
>>
>> Comes with Python.
>>
>> On Fri, Dec 11, 2015 at 11:07 AM, Zachary Turner 
>> wrote:
>>
>>> Presumably those tests use an entirely different, hand-rolled test
>>> running infrastructure?
>>>
>>> On Fri, Dec 11, 2015 at 10:52 AM Todd Fiala 
>>> wrote:
>>>
 One thing I want to make sure we can do is have a sane way of storing
 and running tests that  test the test execution engine.  Those are tests
 that should not run as part of an "lldb test run".  These are tests that
 maintainers of the test system run to make sure we're not breaking stuff
 when we touch the test system.

 I would be writing more of those if I had a semi-sane way of doing it.
  (Part of the reason I broke out the python-based timeout logic the way I
 did, before the major packaging changes, was so I had an obvious spot to
 add tests for the process runner logic).

 On Fri, Dec 11, 2015 at 10:03 AM, Todd Fiala 
 wrote:

> I like it.
>
> On Fri, Dec 11, 2015 at 9:51 AM, Zachary Turner 
> wrote:
>
>> Yea wasn't planning on doing this today, just throwing the idea out
>> there.
>>
>> On Fri, Dec 11, 2015 at 9:35 AM Todd Fiala 
>> wrote:
>>
>>> I'm fine with the idea.
>>>
>>> FWIW the test events model will likely shift a bit, as it is
>>> currently a single sink, whereas I am likely to turn it into a test 
>>> event
>>> filter chain shortly here.  Formatters still make sense as they'll be 
>>> the
>>> things at the end of the chain.
>>>
>>> Minor detail, result_formatter.py should be results_formatter.py -
>>> they are ResultsFormatter instances (plural on Results since it 
>>> transforms
>>> a series of results into coherent reported output).  I'll rename that at
>>> some point in the near future, but if you shift a number of things 
>>> around,
>>> you can do that.
>>>
>>> I'm just about done with the multi-pass running.  I expect to get an
>>> opt-in version of that running end of day today or worst case on Sunday.
>>> It would be awesome if you can hold off on any significant change like 
>>> that
>>> until this little bit is done as I'm sure we'll collide, particularly 
>>> since
>>> this hits dosep.py pretty significantly.
>>>
>>> Thanks!
>>>
>>> -Todd
>>>
>>> On Fri, Dec 11, 2015 at 1:33 AM, Pavel Labath via lldb-dev <
>>> lldb-dev@lists.llvm.org> wrote:
>>>
 Sounds like a reasonable thing to do. A couple of tiny remarks:
 - when you do the move, you might as well rename dotest into
 something
 else, just to avoid the "which dotest should I run" type of
 questions...
 - there is nothing that makes it obvious that "engine" is actually a
 "test running engine", as it sits in a sibling folder. OTOH,
 "test_engine" might be too verbose, and messes up tab completion, so
 that might not be a good idea either...

 pl


 On 10 December 2015 at 23:30, Zachary Turner via lldb-dev
  wrote:
 > Currently our folder structure looks like this:
 >
 > lldbsuite
 > |-- test
 > |-- dotest.py
 > |-- dosep.py
 > |-- lldbtest.py
 > |-- ...
 > |-- functionalities
 > |-- lang
 > |-- expression_command
 > |-- ...
 > etc
 >
 > I've been thinking about organizing it like this instead:
 >
 > lldbsuite
 > |-- test
 > |-- functionalities
 > |-- lang
 > |-- expression_command
 > |-- ...
 > |-- engine
 > |-- dotest.py
 > |-- dosep.py
 > |-- lldbtest.py
 > |-- ...
 >
 > Anybody have any thoughts on this?  Good idea or bad idea?  The
 main reason
 > I want to do this is because as we start breaking up some of the
 code, it
 > makes sense to start having some subpackages under the `engine`
 folder (or
 > the `test` folder in our current world).  For example, Todd and I
 have
 > discussed the idea of