Mathieu, Thank you for your response. see inline ...
On Wed, Dec 19, 2018 at 4:20 PM Mathieu Desnoyers < mathieu.desnoy...@efficios.com> wrote: > Hi Yonghong, > > ----- On Dec 19, 2018, at 1:19 PM, Yonghong Yan <yany...@gmail.com> wrote: > > We are experimenting LTTng for tracing multi-threaded program, it works > very well for us. Thank you for having this great tool. But we have some > concerns about the overhead and scalability of the tracing. Could you share > some insight of the following questions? > 1. The session domain communicates with the user application via Unix > domain socket, from LTTng document. is the communication frequent, such as > each event requires communication, or the communication just happens at the > beginning to configure user space tracing? > > This Unix socket is only for "control" of tracing (infrequent > communication). The high-throughput tracing data goes through a shared > memory map (per-cpu buffers). > > 2. For the consumer domain, is the consumer domain has a thread per > CPU/channel to write to disk or relay the traces, or it is a single > threaded-process handling all the channels and ring buffers, which could > become a bottleneck if we have large number of user threads all feeding > traces? > > Each consumer daemon is a single thread at the moment. It could be > improved by implementing a multithreaded design in the future. It should > help especially in NUMA setups, where having the consumer daemon on the > same NUMA node as the ring buffer it reads from would minimize the amount > of remote NUMA accesses. > > Another point is cases where I/O is performed to various target locations > (different network interfaces or disks). When all I/O goes through the same > interface, the bottleneck becomes the block device or the network > interface. However, for scenarios involving many network interfaces or > block devices, then multithreading the consumer daemon could become useful. > > This has not been a priority for anyone so far though. > That makes sense. > 3. In the one channel/ring buffer per CPU setting, if a user thread > migrates from one CPU to another, are the traces generated by that user > thread fed to the two channels/buffers for the two CPUs? > > The event generated will belong to the per-cpu buffer on which the > "sched_getcpu()" invocation occurs for the event. It is only saved into a > single per-cpu buffer, even if the thread is migrated to a different CPU > before it completes writing the event. This effectively creates infrequent > situations where threads write into other cpu's per-cpu buffers. Note that > the "reserve" and "commit" operations are smp-safe in lttng-ust for that > reason. > so after the migration, new events will still be written to another CPU's buffer? > 4. So far, the events for tracing can be enabled and disabled from command > line, are you considering to have runtime options (APIs) to enable or > disable certain events? Or this is the feature that already in or can be > implemented in different way? > > We already expose a public LGPLv2.1 API for this. See lttng-tools: > > include/lttng/event.h: lttng_enable_event() > > It is implemented by liblttng-ctl.so > Great and will try that. > 5. For context field, from the document, context fields cannot be removed > from a channel once you add it. I would like to request a feature to allow > removing context fields in the user program. > > That's unfortunately not that simple. The channel context field belongs to > the channel, which maps to the "stream" description in the resulting CTF > metadata in the output trace. That stream description is invariant once it > has been created. > That makes senses. We use babeltrace to process trace records. So far, it presents a single-ordered event sequence and we have to use a dedicated event record filed to separate the traces what are from different channels/threads. Is there library or API (C or python-based) that allows us to process traces channel by channel? I saw the traces are saved in files of channels. Thank you. Yonghong > > So currently the only way to remove a context would be to destroy your > tracing session and create a new one. > > Thanks for your interest in LTTng! > > Mathieu > > > > Thank you very much. > Yonghong > > > > _______________________________________________ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > > > -- > Mathieu Desnoyers > EfficiOS Inc. > http://www.efficios.com >
_______________________________________________ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
