Re: [lttng-dev] reading context fields causes syscalls
- On Jul 14, 2022, at 9:10 AM, Norbert Lange nolang...@gmail.com wrote: > Hello Mathieu, > > Am Do., 20. Mai 2021 um 16:16 Uhr schrieb Mathieu Desnoyers > : >> >> - On May 20, 2021, at 9:42 AM, Norbert Lange nolang...@gmail.com wrote: >> >> > Am Do., 20. Mai 2021 um 15:28 Uhr schrieb Mathieu Desnoyers >> > : >> >> >> >> - On May 20, 2021, at 8:46 AM, Norbert Lange nolang...@gmail.com >> >> wrote: >> >> >> >> > Am Mi., 19. Mai 2021 um 20:52 Uhr schrieb Mathieu Desnoyers >> >> > : >> >> >> >> >> >> - On May 19, 2021, at 8:11 AM, lttng-dev lttng-dev@lists.lttng.org >> >> >> wrote: >> >> >> >> >> >> > Hello, >> >> >> > >> >> >> > Several context fields will cause a syscall atleast the first time a >> >> >> > tracepoint is >> >> >> > recorded. For example all of the following: >> >> >> > >> >> >> > `lttng add-context -c chan --userspace --type=vpid --type=vtid >> >> >> > --type=procname` >> >> >> > >> >> >> > Each of them seems cached in TLS however, and most should never >> >> >> > change >> >> >> > after startup. >> >> >> > >> >> >> > As I am using Lttng over Xenomai, syscalls are strictly forbidden, I >> >> >> > would like to have some function that prepares all data, which I can >> >> >> > call on each thread before it switches to realtime work. >> >> >> > >> >> >> > Kinda similar to urcu_bp_register_thread, I'd like to have some >> >> >> > `lttng_ust_warmup_thread` function that fetches the context values >> >> >> > that can be cached. (urcu_bp_register_thread should be called there >> >> >> > aswell) >> >> >> > I considered just doing a tracepoint, but AFAIK the channel can be >> >> >> > changed/configured after the process is running. So this is not >> >> >> > robust >> >> >> > enough. >> >> >> >> >> >> The new lttng_ust_init_thread() API in lttng-ust 2.13 would be the >> >> >> right >> >> >> place to do this I think: >> >> >> >> >> >> /* >> >> >> * Initialize this thread's LTTng-UST data structures. There is >> >> >> * typically no need to call this, because LTTng-UST initializes its >> >> >> * per-thread data structures lazily, but it should be called >> >> >> explicitly >> >> >> * upon creation of each thread before signal handlers nesting over >> >> >> * those threads use LTTng-UST tracepoints. >> >> >> */ >> >> >> >> >> >> It would make sense that this new initialization helper also >> >> >> initializes >> >> >> all contexts which cache the result of a system call. Considering that >> >> >> contexts can be used from the filter and capture bytecode interpreter, >> >> >> as >> >> >> well as contexts added to channels, I think we'd need to simply >> >> >> initialize >> >> >> them all. > > Any update on that, is there an list of open points/features for lttng? Hi Norbert, You can find a feature wish list for the LTTng-UST project here: https://bugs.lttng.org/projects/lttng-ust One option you have is to implement a patch and send it our way for review. We will then review it as time/bandwidth allows. However, if you would like the EfficiOS team to do the implementation, testing, and integration, please contact us in private so we can discuss our consulting services. Thanks, Mathieu -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] LTTng on RHEL 7.9 with real time kernel patch
- On Jul 15, 2022, at 9:19 AM, Pennese Marco via lttng-dev wrote: > Hi, > I’m trying to use LTTng on a Red Hat Enterprise Linux 7.9 (with the real time > kernel patch). > I installed LTTng on the machine following the EfficiOS guide linked by lttng > official website. > In my configuration, lttng manages to record user space events. > I’d like to record also kernel space events, so I’m following the quickstart > guide on the website. > As soon as I run “ lttng list –-kernel” , I obtain the error: > Error: Unable to list kernel events: Kernel tracer not available > Error: Command error > If I repeat the procedure on a machine without the real time kernel patch, it > works as it should. > The real time kernel I’m using is the following > 3.10.0-1160.62.1.rt56.1203.el7.x86_64 > Do you think the error is due to the real time patch? Can LTTng work with a > real > time kernel? > Do I need to follow a different procedure for installing LTTng on a RHEL 7.9 > real time machine? Hi Marco, Specific packaging needs around RHEL 7, including the real-time variant, are part of EfficiOS packaging, support and feature development services, which allow us to develop and maintain the entire LTTng open source ecosystem. Please contact us in private to discuss the available options. Thanks for reaching out! Mathieu > Thank you, > Marco > Il presente messaggio e-mail e ogni suo allegato devono intendersi indirizzati > esclusivamente al destinatario indicato e considerarsi dal contenuto > strettamente riservato e confidenziale. Se non siete l'effettivo destinatario > o > avete ricevuto il messaggio e-mail per errore, siete pregati di avvertire > immediatamente il mittente e di cancellare il suddetto messaggio e ogni suo > allegato dal vostro sistema informatico. Qualsiasi utilizzo, diffusione, copia > o archiviazione del presente messaggio da parte di chi non ne è il > destinatario > è strettamente proibito e può dar luogo a responsabilità di carattere civile e > penale punibili ai sensi di legge. > Questa e-mail ha valore legale solo se firmata digitalmente ai sensi della > normativa vigente. > The contents of this email message and any attachments are intended solely for > the addressee(s) and contain confidential and/or privileged information. > If you are not the intended recipient of this message, or if this message has > been addressed to you in error, please immediately notify the sender and then > delete this message and any attachments from your system. If you are not the > intended recipient, you are hereby notified that any use, dissemination, > copying, or storage of this message or its attachments is strictly prohibited. > Unauthorized disclosure and/or use of information contained in this email > message may result in civil and criminal liability. “ > This e-mail has legal value according to the applicable laws only if it is > digitally signed by the sender > ___ > lttng-dev mailing list > lttng-dev@lists.lttng.org > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] [PATCH] Improve tracef/tracelog to use the stack for small strings
Support two common cases, one being that the resulting message is
small enough to fit into a on-stack buffer.
The seconds being the common 'printf("%s", "Message")' scheme.
Unfortunately, iterating a va_list is destructive,
so it has to be copied before calling vprintf.
The implementation was moved to a separate file,
used by both tracef.c and tracelog.c.
Signed-off-by: Norbert Lange
---
v2:
- move define into src/common/tracer.h
see https://lists.lttng.org/pipermail/lttng-dev/2021-May/029977.html
- moved macro magic into common tracelog-internal.h header
- rebased onto master
---
src/common/tracer.h | 2 +
src/lib/lttng-ust/tracef.c| 32 +++
src/lib/lttng-ust/tracelog-internal.h | 83 +++
src/lib/lttng-ust/tracelog.c | 40 +++--
4 files changed, 102 insertions(+), 55 deletions(-)
create mode 100644 src/lib/lttng-ust/tracelog-internal.h
diff --git a/src/common/tracer.h b/src/common/tracer.h
index 2affd6ab..8e18c9b5 100644
--- a/src/common/tracer.h
+++ b/src/common/tracer.h
@@ -26,6 +26,8 @@
#define LTTNG_RFLAG_EXTENDED RING_BUFFER_RFLAG_END
#define LTTNG_RFLAG_END(LTTNG_RFLAG_EXTENDED << 1)
+#define LTTNG_TRACE_PRINTF_BUFSIZE 512
+
/*
* LTTng client type enumeration. Used by the consumer to map the
* callbacks from its own address space.
diff --git a/src/lib/lttng-ust/tracef.c b/src/lib/lttng-ust/tracef.c
index c05c7811..92911e1d 100644
--- a/src/lib/lttng-ust/tracef.c
+++ b/src/lib/lttng-ust/tracef.c
@@ -7,6 +7,7 @@
#define _LGPL_SOURCE
#include
#include "common/macros.h"
+#include "common/tracer.h"
/* The tracepoint definition is public, but the provider definition is hidden.
*/
#define LTTNG_UST_TRACEPOINT_PROVIDER_HIDDEN_DEFINITION
@@ -15,39 +16,22 @@
#define LTTNG_UST_TRACEPOINT_DEFINE
#include "lttng-ust-tracef-provider.h"
-static inline
-void lttng_ust___vtracef(const char *fmt, va_list ap)
- __attribute__((always_inline, format(printf, 1, 0)));
-static inline
-void lttng_ust___vtracef(const char *fmt, va_list ap)
-{
- char *msg;
- const int len = vasprintf(&msg, fmt, ap);
-
- /* len does not include the final \0 */
- if (len < 0)
- goto end;
- lttng_ust_tracepoint_cb_lttng_ust_tracef___event(msg, len,
- LTTNG_UST_CALLER_IP());
- free(msg);
-end:
- return;
-}
+#include "tracelog-internal.h"
void lttng_ust__vtracef(const char *fmt, va_list ap)
__attribute__((format(printf, 1, 0)));
void lttng_ust__vtracef(const char *fmt, va_list ap)
{
- lttng_ust___vtracef(fmt, ap);
+ LTTNG_UST_TRACELOG_VALIST(fmt, ap,
+ lttng_ust_tracepoint_cb_lttng_ust_tracef___event,
+ msg, len, LTTNG_UST_CALLER_IP());
}
void lttng_ust__tracef(const char *fmt, ...)
__attribute__((format(printf, 1, 2)));
void lttng_ust__tracef(const char *fmt, ...)
{
- va_list ap;
-
- va_start(ap, fmt);
- lttng_ust___vtracef(fmt, ap);
- va_end(ap);
+ LTTNG_UST_TRACELOG_VARARG(fmt,
+ lttng_ust_tracepoint_cb_lttng_ust_tracef___event,
+ msg, len, LTTNG_UST_CALLER_IP());
}
diff --git a/src/lib/lttng-ust/tracelog-internal.h
b/src/lib/lttng-ust/tracelog-internal.h
new file mode 100644
index ..291ff27c
--- /dev/null
+++ b/src/lib/lttng-ust/tracelog-internal.h
@@ -0,0 +1,83 @@
+/*
+ * SPDX-License-Identifier: MIT
+ *
+ * Copyright (C) 2013-2014 Mathieu Desnoyers
+ * Copyright (C) 2021 Norbert Lange
+ *
+ * Shared helper macro for tracelog and tracef
+ */
+
+#define LTTNG_UST_TRACELOG_VARARG(fmt, callback, ...) \
+ va_list ap; \
+ char local_buf[LTTNG_TRACE_PRINTF_BUFSIZE]; \
+ int len; \
+ char *msg = local_buf; \
+ size_t buflen = sizeof(local_buf); \
+ char *alloc_buff = NULL; \
+\
+ if (caa_unlikely(fmt[0] == '%' && fmt[1] == 's' && fmt[2] == '\0')) { \
+ va_start(ap, fmt); \
+ msg = va_arg(ap, char *); \
+ va_end(ap); \
+ len = strlen(msg); \
+ } else \
+ for(;;) { \
+ va_start(ap, fmt); \
+ len = vsnprintf(msg, buflen, fmt, ap); \
+ va_end(ap); \
+\
+ if (caa_unlikely(len >= (int)sizeof(local_buf) &&
!alloc_buff)) { \
+ buflen = (size_t)len + 1U; \
+ len = -1; \
+ alloc_buff = (char *)malloc(buflen); \
+ msg = alloc_buff; \
+ if (!alloc_buff) \
+ break; \
+ } else \
+ break; \
+ } \
+\
+ /* len does not include the final \0 */ \
+ if (caa_likely(len >= 0)) { \
+ callback(__VA_ARGS__); \
+ } \
+ /*
[lttng-dev] [PATCH 2/2] lttng_ust_init_thread: call urcu_register_thread
Eagerly register the thread, and avoid taking mutex during the first tracepoint. Signed-off-by: Norbert Lange --- src/lib/lttng-ust/lttng-ust-comm.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/lttng-ust/lttng-ust-comm.c b/src/lib/lttng-ust/lttng-ust-comm.c index ba0bd985..3ff6b086 100644 --- a/src/lib/lttng-ust/lttng-ust-comm.c +++ b/src/lib/lttng-ust/lttng-ust-comm.c @@ -447,6 +447,8 @@ void lttng_ust_init_thread(void) * this thread attempts to use them. */ lttng_ust_alloc_tls(true); + + lttng_ust_urcu_register_thread(); } int lttng_get_notify_socket(void *owner) -- 2.35.1 ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] [PATCH 1/2] lttng_ust_init_thread: initialise cached context values
Modify all relevant *_alloc_tls functions so that they take an
argument for 'init'. Setting this to 'true' will read
the actual context value and store it into a thread local
cache.
The function 'lttng_ust_init_thread' will use this to
precalculate context values. Tracepoints can then avoid
systemcalls.
Signed-off-by: Norbert Lange
---
* see
https://lore.kernel.org/all/CADYdroN3=pae66crtsvt9ahe4t+bt61-tvhfkcuwzykhhuy...@mail.gmail.com/
* Will update docs after review, and do some more tests
during the week
---
src/lib/lttng-ust/lttng-context-cgroup-ns.c | 4 +++-
src/lib/lttng-ust/lttng-context-ipc-ns.c| 4 +++-
src/lib/lttng-ust/lttng-context-net-ns.c| 4 +++-
src/lib/lttng-ust/lttng-context-procname.c | 4 +++-
src/lib/lttng-ust/lttng-context-provider.c | 4 ++--
src/lib/lttng-ust/lttng-context-time-ns.c | 4 +++-
src/lib/lttng-ust/lttng-context-uts-ns.c| 4 +++-
src/lib/lttng-ust/lttng-context-vtid.c | 4 +++-
src/lib/lttng-ust/lttng-probes.c| 4 ++--
src/lib/lttng-ust/lttng-tracer-core.h | 16 +++---
src/lib/lttng-ust/lttng-ust-comm.c | 24 ++---
src/lib/lttng-ust/lttng-ust-statedump.c | 2 +-
12 files changed, 46 insertions(+), 32 deletions(-)
diff --git a/src/lib/lttng-ust/lttng-context-cgroup-ns.c
b/src/lib/lttng-ust/lttng-context-cgroup-ns.c
index 34523ea1..3eb5ab5b 100644
--- a/src/lib/lttng-ust/lttng-context-cgroup-ns.c
+++ b/src/lib/lttng-ust/lttng-context-cgroup-ns.c
@@ -155,7 +155,9 @@ error_find_context:
/*
* Force a read (imply TLS allocation for dlopen) of TLS variables.
*/
-void lttng_cgroup_ns_alloc_tls(void)
+void lttng_cgroup_ns_alloc_tls(bool init)
{
asm volatile ("" : : "m" (URCU_TLS(cached_cgroup_ns)));
+ if (init)
+ (void)get_cgroup_ns();
}
diff --git a/src/lib/lttng-ust/lttng-context-ipc-ns.c
b/src/lib/lttng-ust/lttng-context-ipc-ns.c
index 63401e8d..b68b939f 100644
--- a/src/lib/lttng-ust/lttng-context-ipc-ns.c
+++ b/src/lib/lttng-ust/lttng-context-ipc-ns.c
@@ -153,7 +153,9 @@ error_find_context:
/*
* Force a read (imply TLS allocation for dlopen) of TLS variables.
*/
-void lttng_ipc_ns_alloc_tls(void)
+void lttng_ipc_ns_alloc_tls(bool init)
{
asm volatile ("" : : "m" (URCU_TLS(cached_ipc_ns)));
+ if (init)
+ (void)get_ipc_ns();
}
diff --git a/src/lib/lttng-ust/lttng-context-net-ns.c
b/src/lib/lttng-ust/lttng-context-net-ns.c
index 960591c2..93e574ad 100644
--- a/src/lib/lttng-ust/lttng-context-net-ns.c
+++ b/src/lib/lttng-ust/lttng-context-net-ns.c
@@ -153,7 +153,9 @@ error_find_context:
/*
* Force a read (imply TLS allocation for dlopen) of TLS variables.
*/
-void lttng_net_ns_alloc_tls(void)
+void lttng_net_ns_alloc_tls(bool init)
{
asm volatile ("" : : "m" (URCU_TLS(cached_net_ns)));
+ if (init)
+ (void)get_net_ns();
}
diff --git a/src/lib/lttng-ust/lttng-context-procname.c
b/src/lib/lttng-ust/lttng-context-procname.c
index b5bf77be..95f6fe42 100644
--- a/src/lib/lttng-ust/lttng-context-procname.c
+++ b/src/lib/lttng-ust/lttng-context-procname.c
@@ -122,7 +122,9 @@ error_find_context:
/*
* Force a read (imply TLS allocation for dlopen) of TLS variables.
*/
-void lttng_procname_alloc_tls(void)
+void lttng_procname_alloc_tls(bool init)
{
asm volatile ("" : : "m" (URCU_TLS(cached_procname)[0]));
+ if (init)
+ (void)wrapper_getprocname();
}
diff --git a/src/lib/lttng-ust/lttng-context-provider.c
b/src/lib/lttng-ust/lttng-context-provider.c
index 4e7e429f..ba24aad7 100644
--- a/src/lib/lttng-ust/lttng-context-provider.c
+++ b/src/lib/lttng-ust/lttng-context-provider.c
@@ -67,7 +67,7 @@ struct lttng_ust_registered_context_provider
*lttng_ust_context_provider_registe
size_t name_len = strlen(provider->name);
uint32_t hash;
- lttng_ust_alloc_tls();
+ lttng_ust_alloc_tls(false);
/* Provider name starts with "$app.". */
if (strncmp("$app.", provider->name, strlen("$app.")) != 0)
@@ -101,7 +101,7 @@ end:
void lttng_ust_context_provider_unregister(struct
lttng_ust_registered_context_provider *reg_provider)
{
- lttng_ust_alloc_tls();
+ lttng_ust_alloc_tls(false);
if (ust_lock())
goto end;
diff --git a/src/lib/lttng-ust/lttng-context-time-ns.c
b/src/lib/lttng-ust/lttng-context-time-ns.c
index ec32a1de..a957409b 100644
--- a/src/lib/lttng-ust/lttng-context-time-ns.c
+++ b/src/lib/lttng-ust/lttng-context-time-ns.c
@@ -152,7 +152,9 @@ error_find_context:
/*
* Force a read (imply TLS allocation for dlopen) of TLS variables.
*/
-void lttng_time_ns_alloc_tls(void)
+void lttng_time_ns_alloc_tls(bool init)
{
asm volatile ("" : : "m" (URCU_TLS(cached_time_ns)));
+ if (init)
+ (void)get_time_ns();
}
diff --git a/src/lib/lttng-ust/lttng-context-uts-ns.c
b/src/lib/lttng-ust/lttng-context-uts-ns.c
index 06a978fd..0c802188 100644
--- a/s
