Re: [lttng-dev] Large number of stream files in CTF trace -- too many file handles

2020-03-16 Thread Rocky Dunlap via lttng-dev
Jonathan, Increasing the soft FD limit worked great, both for the command line and the Python script. Thanks for the help! Rocky On Mon, Mar 16, 2020 at 8:51 AM Jonathan Rajotte-Julien < jonathan.rajotte-jul...@efficios.com> wrote: > Hi, > > > If this is not the right approach, how should I pr

Re: [lttng-dev] Large number of stream files in CTF trace -- too many file handles

2020-03-16 Thread Jonathan Rajotte-Julien via lttng-dev
Hi, > If this is not the right approach, how should I proceed? E.g., should the > source-ctf-fs manage a limited pool of file handles? I would think this > would be pretty inefficient as you would need to constantly open/close > files--expensive. I would probably start looking at the soft and h

[lttng-dev] Large number of stream files in CTF trace -- too many file handles

2020-03-13 Thread Rocky Dunlap via lttng-dev
I am attempting to use babeltrace2 to read a CTF trace that has ~2000 stream files. This is a custom trace collected from an MPI application on an HPC platform. In this case, each MPI process opens and writes to its own stream file, so you end up with one file per MPI task. When I attempt to rea