[lttng-dev] [PATCH lttng-tools] fix: Re-add link to urcu-bp for _LGPL_SOURCE tests
Tests with tracepoints defined with _LGPL_SOURCE require to be explicitly linked against urcu-bp. Signed-off-by: Michael Jeanson --- tests/regression/ust/multi-session/Makefile.am | 2 +- tests/utils/testapp/gen-ust-events/Makefile.am | 2 +- tests/utils/testapp/gen-ust-nevents-str/Makefile.am | 2 +- tests/utils/testapp/gen-ust-nevents/Makefile.am | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/tests/regression/ust/multi-session/Makefile.am b/tests/regression/ust/multi-session/Makefile.am index 0d807a878..676c26f21 100644 --- a/tests/regression/ust/multi-session/Makefile.am +++ b/tests/regression/ust/multi-session/Makefile.am @@ -2,7 +2,7 @@ AM_CFLAGS += -I$(srcdir) noinst_PROGRAMS = gen-nevents gen_nevents_SOURCES = gen-nevents.c tp.c ust_gen_nevents.h -gen_nevents_LDADD = $(UST_LIBS) $(DL_LIBS) +gen_nevents_LDADD = $(UST_LIBS) -lurcu-bp $(DL_LIBS) noinst_SCRIPTS = test_multi_session EXTRA_DIST = test_multi_session diff --git a/tests/utils/testapp/gen-ust-events/Makefile.am b/tests/utils/testapp/gen-ust-events/Makefile.am index 38e620d75..8918fc80d 100644 --- a/tests/utils/testapp/gen-ust-events/Makefile.am +++ b/tests/utils/testapp/gen-ust-events/Makefile.am @@ -4,7 +4,7 @@ AM_CPPFLAGS += -I$(top_srcdir)/tests/utils -I$(srcdir) \ if HAVE_LIBLTTNG_UST_CTL noinst_PROGRAMS = gen-ust-events gen_ust_events_SOURCES = gen-ust-events.c tp.c tp.h -gen_ust_events_LDADD = $(UST_LIBS) \ +gen_ust_events_LDADD = $(UST_LIBS) -lurcu-bp \ $(top_builddir)/tests/utils/libtestutils.la \ $(DL_LIBS) endif diff --git a/tests/utils/testapp/gen-ust-nevents-str/Makefile.am b/tests/utils/testapp/gen-ust-nevents-str/Makefile.am index 0e961b4c4..27cb9efa7 100644 --- a/tests/utils/testapp/gen-ust-nevents-str/Makefile.am +++ b/tests/utils/testapp/gen-ust-nevents-str/Makefile.am @@ -4,7 +4,7 @@ AM_CPPFLAGS += -I$(srcdir) -I$(top_srcdir)/tests/utils \ if HAVE_LIBLTTNG_UST_CTL noinst_PROGRAMS = gen-ust-nevents-str gen_ust_nevents_str_SOURCES = gen-ust-nevents-str.c tp.c tp.h -gen_ust_nevents_str_LDADD = $(UST_LIBS) \ +gen_ust_nevents_str_LDADD = $(UST_LIBS) -lurcu-bp \ $(top_builddir)/tests/utils/libtestutils.la \ $(DL_LIBS) endif diff --git a/tests/utils/testapp/gen-ust-nevents/Makefile.am b/tests/utils/testapp/gen-ust-nevents/Makefile.am index c20910bd8..f85d35a13 100644 --- a/tests/utils/testapp/gen-ust-nevents/Makefile.am +++ b/tests/utils/testapp/gen-ust-nevents/Makefile.am @@ -4,7 +4,7 @@ AM_CPPFLAGS += -I$(srcdir) -I$(top_srcdir)/tests/utils \ if HAVE_LIBLTTNG_UST_CTL noinst_PROGRAMS = gen-ust-nevents gen_ust_nevents_SOURCES = gen-ust-nevents.c tp.c tp.h -gen_ust_nevents_LDADD = $(UST_LIBS) \ +gen_ust_nevents_LDADD = $(UST_LIBS) -lurcu-bp \ $(top_builddir)/tests/utils/libtestutils.la \ $(DL_LIBS) endif -- 2.17.1 ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
Re: [lttng-dev] [PATCH lttng-tools] fix: Re-add link to urcu-bp for _LGPL_SOURCE tests
Merged in master. Thanks! Jérémie On Tue, Nov 05, 2019 at 11:30:24AM -0500, Michael Jeanson wrote: > Tests with tracepoints defined with _LGPL_SOURCE require to be > explicitly linked against urcu-bp. > > Signed-off-by: Michael Jeanson > --- > tests/regression/ust/multi-session/Makefile.am | 2 +- > tests/utils/testapp/gen-ust-events/Makefile.am | 2 +- > tests/utils/testapp/gen-ust-nevents-str/Makefile.am | 2 +- > tests/utils/testapp/gen-ust-nevents/Makefile.am | 2 +- > 4 files changed, 4 insertions(+), 4 deletions(-) > > diff --git a/tests/regression/ust/multi-session/Makefile.am > b/tests/regression/ust/multi-session/Makefile.am > index 0d807a878..676c26f21 100644 > --- a/tests/regression/ust/multi-session/Makefile.am > +++ b/tests/regression/ust/multi-session/Makefile.am > @@ -2,7 +2,7 @@ AM_CFLAGS += -I$(srcdir) > > noinst_PROGRAMS = gen-nevents > gen_nevents_SOURCES = gen-nevents.c tp.c ust_gen_nevents.h > -gen_nevents_LDADD = $(UST_LIBS) $(DL_LIBS) > +gen_nevents_LDADD = $(UST_LIBS) -lurcu-bp $(DL_LIBS) > > noinst_SCRIPTS = test_multi_session > EXTRA_DIST = test_multi_session > diff --git a/tests/utils/testapp/gen-ust-events/Makefile.am > b/tests/utils/testapp/gen-ust-events/Makefile.am > index 38e620d75..8918fc80d 100644 > --- a/tests/utils/testapp/gen-ust-events/Makefile.am > +++ b/tests/utils/testapp/gen-ust-events/Makefile.am > @@ -4,7 +4,7 @@ AM_CPPFLAGS += -I$(top_srcdir)/tests/utils -I$(srcdir) \ > if HAVE_LIBLTTNG_UST_CTL > noinst_PROGRAMS = gen-ust-events > gen_ust_events_SOURCES = gen-ust-events.c tp.c tp.h > -gen_ust_events_LDADD = $(UST_LIBS) \ > +gen_ust_events_LDADD = $(UST_LIBS) -lurcu-bp \ > $(top_builddir)/tests/utils/libtestutils.la \ > $(DL_LIBS) > endif > diff --git a/tests/utils/testapp/gen-ust-nevents-str/Makefile.am > b/tests/utils/testapp/gen-ust-nevents-str/Makefile.am > index 0e961b4c4..27cb9efa7 100644 > --- a/tests/utils/testapp/gen-ust-nevents-str/Makefile.am > +++ b/tests/utils/testapp/gen-ust-nevents-str/Makefile.am > @@ -4,7 +4,7 @@ AM_CPPFLAGS += -I$(srcdir) -I$(top_srcdir)/tests/utils \ > if HAVE_LIBLTTNG_UST_CTL > noinst_PROGRAMS = gen-ust-nevents-str > gen_ust_nevents_str_SOURCES = gen-ust-nevents-str.c tp.c tp.h > -gen_ust_nevents_str_LDADD = $(UST_LIBS) \ > +gen_ust_nevents_str_LDADD = $(UST_LIBS) -lurcu-bp \ > $(top_builddir)/tests/utils/libtestutils.la \ > $(DL_LIBS) > endif > diff --git a/tests/utils/testapp/gen-ust-nevents/Makefile.am > b/tests/utils/testapp/gen-ust-nevents/Makefile.am > index c20910bd8..f85d35a13 100644 > --- a/tests/utils/testapp/gen-ust-nevents/Makefile.am > +++ b/tests/utils/testapp/gen-ust-nevents/Makefile.am > @@ -4,7 +4,7 @@ AM_CPPFLAGS += -I$(srcdir) -I$(top_srcdir)/tests/utils \ > if HAVE_LIBLTTNG_UST_CTL > noinst_PROGRAMS = gen-ust-nevents > gen_ust_nevents_SOURCES = gen-ust-nevents.c tp.c tp.h > -gen_ust_nevents_LDADD = $(UST_LIBS) \ > +gen_ust_nevents_LDADD = $(UST_LIBS) -lurcu-bp \ > $(top_builddir)/tests/utils/libtestutils.la \ > $(DL_LIBS) > endif > -- > 2.17.1 > ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] [RFC PATCH lttng-tools 2/3] align.h: Implement ALIGN_FLOOR macro
Implement the ALIGN_FLOOR macro which aligns the given value to the previous alignment boundary, or keeps the value as-is if it is already aligned. Signed-off-by: Mathieu Desnoyers --- src/common/align.h | 6 ++ 1 file changed, 6 insertions(+) diff --git a/src/common/align.h b/src/common/align.h index ed353206..c8d9d517 100644 --- a/src/common/align.h +++ b/src/common/align.h @@ -35,6 +35,12 @@ #define ALIGN(v, align)__ALIGN_MASK(v, (__typeof__(v)) (align) - 1) #endif +#define __ALIGN_FLOOR_MASK(v, mask)((v) & ~(mask)) + +#ifndef ALIGN_FLOOR +#define ALIGN_FLOOR(v, align) __ALIGN_FLOOR_MASK(v, (__typeof__(v)) (align) - 1) +#endif + #define PAGE_ALIGN(addr) ALIGN(addr, PAGE_SIZE) /** -- 2.17.1 ___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
[lttng-dev] [RFC PATCH lttng-tools 3/3] Fix: consumerd: use packet sequence number for rotation position
Refer to "Fix: relayd: use packet sequence number for rotation position"
for context of this change.
This commit introduces the changes required in the consumerd.
Some notable points related to this commit:
- The per-stream rotation position within the consumerd is now the
very start of the first packet to go into the next trace chunk,
which makes consumerd consistent with the semantic of the relayd
and of the rotate_at_seq_num field of the protocol between those two
daemons.
- Internally, the rotate_position (per-stream) is now a 64-bit
value rather than an unsigned long.
- The scheme to rotate a stream is changed to allow using the
backward-compatible lttng_consumer_take_snapshot() rather than
the newer lttng_consumer_get_produced_snapshot(), thus allowing
backward compatibility of the implicit rotation on destroy with
pre-2.10 lttng-modules.
- The rotate position used as pivot point for the rotation is
based on the packet_seq_num of the last packet that has been
send over the network by consumerd, incremented by the number of
packets between the sampled produced_pos and the consumed_pos.
In the worse case scenario where an overwrite mode ring buffer
overwrites its contents enough to trigger a 4GB overflow on a
32-bit producer since the last packet was sent (e.g. due to a
slow network), the difference between produced_pos and
consumed_pos will be lower that what would have been expected.
However, because this pivot position is used as a lower bound,
being smaller than the real value is fine: the data that would
have been misplaced in the wrong trace chunk were actually
overwritten, and will therefore never be consumed.
(limitations)
- When interacting with pre-2.8 lttng-modules, the packet sequence
number is not available. The current approach is to send -1ULL as
rotation position to the relayd, but it is probably not expected.
Signed-off-by: Mathieu Desnoyers
---
src/common/consumer/consumer.c | 142 ++-
src/common/consumer/consumer.h | 15 +-
src/common/kernel-consumer/kernel-consumer.c | 1 +
3 files changed, 85 insertions(+), 73 deletions(-)
diff --git a/src/common/consumer/consumer.c b/src/common/consumer/consumer.c
index 09f0f098..3d764f92 100644
--- a/src/common/consumer/consumer.c
+++ b/src/common/consumer/consumer.c
@@ -603,6 +603,7 @@ struct lttng_consumer_stream
*consumer_allocate_stream(uint64_t channel_key,
stream->endpoint_status = CONSUMER_ENDPOINT_ACTIVE;
stream->index_file = NULL;
stream->last_sequence_number = -1ULL;
+ stream->rotate_position = -1ULL;
pthread_mutex_init(&stream->lock, NULL);
pthread_mutex_init(&stream->metadata_timer_lock, NULL);
@@ -4081,7 +4082,7 @@ int lttng_consumer_rotate_channel(struct
lttng_consumer_channel *channel,
ht->hash_fct(&channel->key, lttng_ht_seed),
ht->match_fct, &channel->key, &iter.iter,
stream, node_channel_id.node) {
- unsigned long consumed_pos;
+ unsigned long produced_pos = 0, consumed_pos = 0;
health_code_update();
@@ -4094,65 +4095,77 @@ int lttng_consumer_rotate_channel(struct
lttng_consumer_channel *channel,
rotating_to_new_chunk = false;
}
- ret = lttng_consumer_sample_snapshot_positions(stream);
+ /*
+* Active flush; has no effect if the production position
+* is at a packet boundary.
+*/
+ ret = consumer_flush_buffer(stream, 1);
if (ret < 0) {
- ERR("Failed to sample snapshot position during channel
rotation");
+ ERR("Failed to flush stream %" PRIu64 " during channel
rotation",
+ stream->key);
goto end_unlock_stream;
}
- ret = lttng_consumer_get_produced_snapshot(stream,
- &stream->rotate_position);
- if (ret < 0) {
- ERR("Failed to sample produced position during channel
rotation");
+ ret = lttng_consumer_take_snapshot(stream);
+ if (ret < 0 && ret != -ENODATA && ret != -EAGAIN) {
+ ERR("Failed to sample snapshot position during channel
rotation");
goto end_unlock_stream;
}
+ if (!ret) {
+ ret = lttng_consumer_get_produced_snapshot(stream,
+ &produced_pos);
+ if (ret < 0) {
+ ERR("Failed to sample produced position during
channel rotation");
+ goto end_unlock_stream;
+ }
- lttng_consumer_get_consumed_snapshot(stream,
-
[lttng-dev] [RFC PATCH lttng-tools 1/3] Fix: relayd: use packet sequence number for rotation position
The "network" sequence number (net_seq_num) is a 64-bit sequence number
tagging each packet sent over the network. The net_seq_num increments
monotonically (+1) for each packet sent from consumer daemon to relay
daemon, on a per-stream basis. It is tagged by the consumer daemon when
sending a trace packet to the relay daemon.
The LTTng kernel and user-space ring buffer "consumed position"
(consumed_pos) and "produced position" (produced_pos) are free-running
counters counting the number of bytes consumed and produced so far by
each stream. Because those counters are updated atomically, they are
limited to a size of 32-bit on 32-bit architectures.
The "packet" sequence number (packet_seq_num) is a sequence number
found in the packet header starting from LTTng 2.8. It is a 64-bit
sequence number assigned by the lttng-modules and lttng-ust ring
buffers. It increments monotonically (+1) for each packet produced
within a given ring buffer (stream).
Using produced_pos as rotation position and comparing it to the
net_seq_num has a few issues:
1) It breaks on 32-bit producers after generating more than 4GB of
data per stream, due to overflow. The net_seq_num is a 64-bit
counter, which does not overflow, but the produced_pos overflows
after 4GB on 32-bit architectures. This can lead to never-completing
rotations.
2) It breaks scenarios where ring buffers are configured in
overwrite mode, and streaming to a relay daemon. Indeed, when
the ring buffer moves the consumed_pos ahead, actually overwriting
data within the ring buffer, it introduces an offset between the
produced_pos and the net_seq_num. Therefore, if producers are
generating a low- (or no-) throughput in some streams, the
rotation may never complete, even on 64-bit architectures.
The solution proposed for this issue is to use the packet_seq_num as
rotation position rather than the net_seq_num. It takes care of
the two problematic scenarios, since the counter is always 64-bit
(even on 32-bit architectures), and because the counter is managed
by the producer, which therefore tracks progress of the ring buffer
overwrites.
This commit introduces changes required at the relayd side. A
separate commit introduces the changes required in the consumerd.
In relayd, one major restriction is the fact that the packet_seq_num
is not sent over the data socket, only through the control socket
receiving the indexes.
Therefore, in order to figure out the pivot position for the data
socket for a given stream, the associated index first needs to be
received. At that point, the corresponding net_seq_num is known,
which provides the pivot position for the data stream. Given that
the data and index sockets provide no ordering guarantees with
respect to their arrival, we handle the fact that data might have
been saved to disk in the wrong (previous) trace chunk by moving
it to the next trace chunk when the pivot position is known.
In order to allow "jumps" in the sequence numbers produced by
overwrite mode buffers, try_rotate_stream_index(), which previously
asserted that each sequence number was received in sequence, now
uses the packet_seq_num pivot position as a lower (inclusive) bound.
Signed-off-by: Mathieu Desnoyers
---
src/bin/lttng-relayd/index.c | 5 +++
src/bin/lttng-relayd/main.c | 3 ++
src/bin/lttng-relayd/stream.c | 61 ++-
src/bin/lttng-relayd/stream.h | 18 +--
src/common/relayd/relayd.h| 2 +-
5 files changed, 69 insertions(+), 20 deletions(-)
diff --git a/src/bin/lttng-relayd/index.c b/src/bin/lttng-relayd/index.c
index 3cae94e8..bdbd1133 100644
--- a/src/bin/lttng-relayd/index.c
+++ b/src/bin/lttng-relayd/index.c
@@ -433,6 +433,11 @@ int relay_index_set_control_data(struct relay_index *index,
if (minor_version >= 8) {
index->index_data.stream_instance_id =
htobe64(data->stream_instance_id);
index->index_data.packet_seq_num =
htobe64(data->packet_seq_num);
+ } else {
+ uint64_t unset_value = -1ULL;
+
+ index->index_data.stream_instance_id = htobe64(unset_value);
+ index->index_data.packet_seq_num = htobe64(unset_value);
}
return relay_index_set_data(index, &index_data);
diff --git a/src/bin/lttng-relayd/main.c b/src/bin/lttng-relayd/main.c
index 5f72c32a..09ba4e9d 100644
--- a/src/bin/lttng-relayd/main.c
+++ b/src/bin/lttng-relayd/main.c
@@ -2121,6 +2121,9 @@ static int relay_recv_index(const struct
lttcomm_relayd_hdr *recv_hdr,
index_info.stream_instance_id =
be64toh(index_info.stream_instance_id);
index_info.packet_seq_num = be64toh(index_info.packet_seq_num);
+ } else {
+ index_info.stream_instance_id = -1ULL;
+ index_info.packet_seq_num = -1ULL;
}
stream = stream_get_by_id(index_info.relay_stream_id);
diff --git a/src/bin/lttng-relayd/stream.c b
Re: [lttng-dev] 回复:Re: 回复:Re: 回复:Re: Pros and Cons of LTTng
Thanks Jonathan. Regarding to the CI MTTR/MTTF test results, it varies from time to time, and on master/stable branches. 1. it monitors new Linux kernels, so the CI job may use newer kernel version than months ago? 2. What would be the criteria of MTBF here? MTTRLast 7 Days0 ms Last 30 Days12 hr All Time3 days 21 hr MTTFLast 7 Days0 ms Last 30 Days12 days All Time29 days Standard DeviationLast 7 Days21 min Last 30 Days30 min All Time44 min Regards Hai -- Original -- From: "Jonathan Rajotte-Julien"http://packages.efficios.com/. > Does LTTng have roadmap for upcoming new kernel versions? I presume you are explicitly asking for RHEL & SUSE kernels. The current packages for these distro are maintained on a best effort and commercial incentive of EfficiOS. If you are interested in supporting this effort please contact sa...@efficios.com. As for the actual upstream kernel, we are monitoring [1] all new kernels and stable branches. [1] https://ci.lttng.org/view/LTTng-modules/ https://ci.lttng.org/view/LTTng-modules/job/lttng-modules_stable-2.9_build-vanilla/ https://ci.lttng.org/view/LTTng-modules/job/lttng-modules_stable-2.10_build-vanilla/ https://ci.lttng.org/view/LTTng-modules/job/lttng-modules_stable-2.11_build-vanilla/ https://ci.lttng.org/view/LTTng-modules/job/lttng-modules_master_build-vanilla/ Cheers___ lttng-dev mailing list lttng-dev@lists.lttng.org https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
