Re: [PATCH RFC] fork: reduce chances for "address space is already occupied" errors
Brian Inglis writes: > File list my-dlls.txt is your local test rebase db listing all your > test dlls. I think Michael got confused by your usage of "db" here. This is in fact just a listing of all the DLL to operate on, not the rebase database (which won't be changed at all by an oblivious rebase, only read in order to not collide the new rebase with the already existing ones). > If you are packaging your own exes and dlls with your own local Cygwin distro, > you should point to your local utility directory with a path in a file under > /var/lib/rebase/user.d/$USER for each Cygwin userid on each system, or perhaps > you might also need to add your own production exes and dlls into > /var/cache/rebase/rebase_user and /var/cache/rebase/rebase_user_exe: see > /usr/share/doc/Cygwin/_autorebase.README. What Michael is using is a fairly complex build system that would indeed benefit from a layered rebase database, i.e. the one for the base system providing the substrate for the build system and then at leat on other one that collects the information from inside the build system (maybe even a third layer for tests). How to deal with the complexities of when you want to push information down to a previous layer would likely be a main point of contention, so you'd probably best skip it in the beginning. SHTDI, PTC, etc.pp. With the current rebase, you'll have to use "--oblivious" (which, again, doesn't remember any data for the newly rebased objects) and those non-existing upper layers will have to be provided by side-channel information that the build system has to collect and maintain itself, then feed to the rebase command. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf rackAttack: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Re: [PATCH fifo 0/2] Add support for duplex FIFOs
Ken Brown writes: >> I'm pretty sure Ken would be happy about an STC. > > Yes, please. Barring that, is there any chance I could see the relevant > code, > or at least enough of it so that I can see how FIFOs are being used? Well, I'm trying -- but got nothing so far. As the individual FIFO seem to work and the error happens pretty early, I think it has something to do with either switching between different FIFO (which the original code does) or some race between fill and drain, possibly around a buffer boundary (in my limited testing it always seemed to happen in the same place for the same data). Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf rackAttack: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Re: [PATCH RFC] fork: reduce chances for "address space is already occupied" errors
On 2019-03-29 01:15, Achim Gratz wrote: > Brian Inglis writes: >> File list my-dlls.txt is your local test rebase db listing all your >> test dlls. > > I think Michael got confused by your usage of "db" here. This is in > fact just a listing of all the DLL to operate on, not the rebase > database (which won't be changed at all by an oblivious rebase, only > read in order to not collide the new rebase with the already existing > ones). > >> If you are packaging your own exes and dlls with your own local Cygwin >> distro, >> you should point to your local utility directory with a path in a file under >> /var/lib/rebase/user.d/$USER for each Cygwin userid on each system, or >> perhaps >> you might also need to add your own production exes and dlls into >> /var/cache/rebase/rebase_user and /var/cache/rebase/rebase_user_exe: see >> /usr/share/doc/Cygwin/_autorebase.README. > > What Michael is using is a fairly complex build system that would indeed > benefit from a layered rebase database, i.e. the one for the base system > providing the substrate for the build system and then at leat on other > one that collects the information from inside the build system (maybe > even a third layer for tests). How to deal with the complexities of > when you want to push information down to a previous layer would likely > be a main point of contention, so you'd probably best skip it in the > beginning. > > SHTDI, PTC, etc.pp. > > With the current rebase, you'll have to use "--oblivious" (which, again, > doesn't remember any data for the newly rebased objects) and those > non-existing upper layers will have to be provided by side-channel > information that the build system has to collect and maintain itself, > then feed to the rebase command. Achim, thanks for the clarifications; could you please comment on the suggested approach for handling local production dlls and exes, or explain the best approach for migrating from test to prod and handling rebase on target systems? -- Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada This email may be disturbing to some readers as it contains too much technical detail. Reader discretion is advised.
Re: [PATCH fifo 0/2] Add support for duplex FIFOs
On 3/29/2019 3:17 AM, Achim Gratz wrote: > Ken Brown writes: >>> I'm pretty sure Ken would be happy about an STC. >> >> Yes, please. Barring that, is there any chance I could see the relevant >> code, >> or at least enough of it so that I can see how FIFOs are being used? > > Well, I'm trying -- but got nothing so far. As the individual FIFO seem > to work and the error happens pretty early, I think it has something to > do with either switching between different FIFO (which the original code > does) or some race between fill and drain, possibly around a buffer > boundary (in my limited testing it always seemed to happen in the same > place for the same data). I found a bug in my fhandler_fifo::raw_write code that could explain the problem. The call to NtWriteFile in that function always returns immediately because the Windows named pipe underlying the FIFO is non-blocking. If it can't write because the pipe buffer is full, raw_write returns -1 with error EAGAIN. That's wrong if the FIFO was opened in blocking mode. I'll have to think about how to best handle this. I think I might be able to imitate what's done in fhandler_socket_unix::sendmsg in the topic/af_unix branch. Ken
Re: [PATCH fifo 0/2] Add support for duplex FIFOs
Ken Brown writes: > I found a bug in my fhandler_fifo::raw_write code that could explain the > problem. The call to NtWriteFile in that function always returns immediately > because the Windows named pipe underlying the FIFO is non-blocking. If it > can't > write because the pipe buffer is full, raw_write returns -1 with error > EAGAIN. > That's wrong if the FIFO was opened in blocking mode. Sounds like a clue or at least it doesn#t contradict what I'm seeing. I have no idea if that explains all the problems I'm seeing, I'll describe them in more detail below. > I'll have to think about how to best handle this. I think I might be able to > imitate what's done in fhandler_socket_unix::sendmsg in the topic/af_unix > branch. OK, a bit more info: The whole thing runs from a perl script (actually a module) that opens pipes to gnuplot and ghostscript. This code is _really_ old and has seen a lot of Cygwin releases, so it has options to either use temporary files, named pipes aka FIFO or direct pipes. Using temporary files serializes the execution and using a pipe chain is _really_ slow (like a hundred times, which is mostly tied up in system for a reason that I don't understand), so using FIFO is the default. Your new FIFO code increases the system time by about a factor of 10 in my tests, btw. The error with the FIFO to gnuplot is that some data that was written into the FIFO already doesn't show up at the reader end, but later data written into it does. Here somewhere around a few kiB go missing and gnuplot runs into a syntax error when it happens. If I run gnuplot through a plain pipe to skip that error, but keep ghostscript on FIFO, then it _almost_ works correctly. Except that once gnuplot has finished writing to the FIFO I need to write the bookmarks dictionary before closing the output file and that write again overwrites data that should already have been present at the other side. I've played a bit with putting in flushes and sleeps and there's between one to three pages at the end of the document that go missing, so again a handful of KiB. So either of these two errors indiqcates problems with synchronizing the reader/writer side, which makes some of the written data disappear on the receiving end of the FIFO. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Q+, Q and microQ: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Re: [PATCH RFC] fork: reduce chances for "address space is already occupied" errors
Brian Inglis writes: > Achim, thanks for the clarifications; could you please comment on the > suggested > approach for handling local production dlls and exes, or explain the best > approach for migrating from test to prod and handling rebase on target > systems? I'm not quite sure what you want to know. As I said before oblivious rebase was invented for running tests that use freshly built DLL (I usually package them before running the tests, so the package will have the un-rebased DLL from before the test was run). For this it suffices to simply feed in all new DLL names to rebase. If you were to build in stages and/or combine different builds then you'd somehow have to remember the DLL from each stage or build, or just collect all the DLL names again each time you change something. The important thing is that each oblivious rebase needs to get the list of _all_ DLL that need to get rebased, since the database only knows about the host system (i.e. you can't rebase incrementally with --oblivious). Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptation for Waldorf rackAttack V1.04R1: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Re: [PATCH RFC] fork: reduce chances for "address space is already occupied" errors
On 2019-03-29 14:23, Achim Gratz wrote:
> Brian Inglis writes:
>>> If you are packaging your own exes and dlls with your own local Cygwin
>>> distro,
>>> you should point to your local utility directory with a path in a file under
>>> /var/lib/rebase/user.d/$USER for each Cygwin userid on each system, or
>>> perhaps
>>> you might also need to add your own production exes and dlls into
>>> /var/cache/rebase/rebase_user and /var/cache/rebase/rebase_user_exe: see
>>> /usr/share/doc/Cygwin/_autorebase.README.
>> Achim, thanks for the clarifications; could you please comment on the
>> suggested
>> approach for handling local production dlls and exes, or explain the best
>> approach for migrating from test to prod and handling rebase on target
>> systems?
> I'm not quite sure what you want to know. As I said before oblivious
> rebase was invented for running tests that use freshly built DLL (I
> usually package them before running the tests, so the package will have
> the un-rebased DLL from before the test was run). For this it suffices
> to simply feed in all new DLL names to rebase. If you were to build in
> stages and/or combine different builds then you'd somehow have to
> remember the DLL from each stage or build, or just collect all the DLL
> names again each time you change something. The important thing is that
> each oblivious rebase needs to get the list of _all_ DLL that need to
> get rebased, since the database only knows about the host system
> (i.e. you can't rebase incrementally with --oblivious).
I was wondering as my first para above stated, whether rebase_user{,_exe} would
be the proper place to add 3rd party Cygwin dlls and exes, that are distributed
with Cygwin (internally)?
--
Take care. Thanks, Brian Inglis, Calgary, Alberta, Canada
This email may be disturbing to some readers as it contains
too much technical detail. Reader discretion is advised.
