Re: lttng-dev Digest, Vol 203, Issue 7

Kienan Stewart via lttng-dev Fri, 04 Apr 2025 11:38:05 -0700

Hi Lakshya,

I did some digging around. What you are seeing is the result of theswitching to MAP_POPULATE by default in LTTng-UST 2.12[1] in commit4d4838b ("Use MAP_POPULATE to reduce pagefault when available").

The purpose of this change is to avoid taking page faults which tracing,reducing first-event in a page latency.

In the master branch, this feature has been made configurable for userswho don't want to pre-populate the pages and would rather take pagefaults while tracing[2].


Here is an example from LTTng master with map populate per possible CPU:

```
export LTTNG_UST_MAP_POPULATE_POLICY=cpu_possible

# Create session, channels, start tracing, and run test app
# top -n1 -b | grep -E '(MiB|COMMAND|lttng)'

MiB Mem : 32768.0 total, 21883.7 free, 1456.0 used, 9428.3buff/cacheMiB Swap: 0.0 total, 0.0 free, 0.0 used. 31312.0 availMemPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+COMMAND301117 debian 20 0 880176 2760 2760 S 0.0 0.0 0:00.04lttng-sessiond301118 debian 20 0 43856 1376 1376 S 0.0 0.0 0:00.01lttng-runas301133 debian 20 0 718616 263456 263456 S 0.0 0.8 0:00.17lttng-consumerd301135 debian 20 0 6996 0 0 S 0.0 0.0 0:00.05lttng-runas


# cat /proc/$(pgrep lttng-sessiond)/statm

lttng-sessiond: 220044 690 690 345 0 29900 0

# pmap $(pgrep lttng-sessiond) | grep total

 total           880176K

# smem -P lttng-sessiondPID User Command Swap USS PSSRSS301118 debian lttng-sessiond --daemonize 0 344 8812236301117 debian lttng-sessiond --daemonize 0 5676 66839276301201 debian /usr/bin/python3 /usr/bin/s 0 8636 1008612936

# /proc/PID/statm for lttng-consumerd

lttng-consumerd: 1749 0 0 129 0 130 0

# pmap lttng-consumerd-pid | grep total

total kB 6996 1700 472


# smem -P lttng-consumerd

PID User Command Swap USS PSSRSS301135 debian lttng-consumerd -u --consu 0 280 5631700301211 debian /usr/bin/python3 /usr/bin/s 0 10048 1150114404301133 debian lttng-consumerd -u --consu 0 262376 263177265480

# smem -m | grep -i ust

/dev/shm/lttng-ust-wait-8-1000               1        4        4
/dev/shm/shm-ust-consumer-301133             1   260756   260756
```

When using the none policy:

```
# export LTTNG_UST_MAP_POPULATE_POLICY=none
# as above

Running test app UID 0procs -----------memory---------- ---swap-- -----io---- -system--------cpu-----r b swpd free buff cache si so bi bo in cs us syid wa st1 0 0 21875 0 9434 0 0 39 636 1105 2496 0 199 0 0MiB Mem : 32768.0 total, 21875.0 free, 1458.2 used, 9434.7buff/cacheMiB Swap: 0.0 total, 0.0 free, 0.0 used. 31309.8 availMemPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+COMMAND301616 debian 20 0 880176 2756 2756 S 0.0 0.0 0:00.04lttng-sessiond301617 debian 20 0 43856 1392 1392 S 0.0 0.0 0:00.01lttng-runas301632 debian 20 0 718612 5416 5416 S 0.0 0.0 0:00.17lttng-consumerd301634 debian 20 0 6992 0 0 S 0.0 0.0 0:00.05lttng-runaslttng-sessiond: 220044 689 689 345 0 29900 0total 880176KPID User Command Swap USS PSSRSS301617 debian lttng-sessiond --daemonize 0 344 8622188301616 debian lttng-sessiond --daemonize 0 5784 67599328301700 debian /usr/bin/python3 /usr/bin/s 0 8632 1007912928


lttng-consumerd: 1748 0 0 129 0 129 0
total kB            6992    1580     468

PID User Command Swap USS PSSRSS301634 debian lttng-consumerd -u --consu 0 276 5361580301632 debian lttng-consumerd -u --consu 0 5672 64338652301710 debian /usr/bin/python3 /usr/bin/s 0 9996 1144914328


/dev/shm/lttng-ust-wait-8-1000               1        4        4
/dev/shm/shm-ust-consumer-301632             1     4048     4048
```

thanks,
kienan

[1]:https://github.com/lttng/lttng-ust/commit/4d4838bad480d48424bddc686f5ad0089e28ac94[2]:https://github.com/lttng/lttng-ust/commit/97572c0438845cee953ebd3e39615f78bfa405a7


On 3/17/25 2:29 AM, Gour DEV wrote:

Hi, Kienan

Sorry for the late reply.

Looks like in buster the memory is allocated by lttng-consumerd reserved

I buster, the rss is less than the VIRT
root@localhost:~#  COLUMNS=500 top  -b -n 1 | grep lttng
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
  4095 root      20   0 1003188  31256   4660 S   0.0   0.1   0:03.81
lttng-sessiond
  4096 root      20   0   44260    796      0 S   0.0   0.0   0:00.01
lttng-runas
  4440 root      20   0 5236020  10224   8756 S   0.0   0.0   2:56.25
lttng-consumerd -- here the VIRT is much more higher than RSS
  4443 root      20   0   48048    540      0 S   0.0   0.0   0:00.12
lttng-runas



In bookworm the VIRT and RES are nearly the same only.
root@edgecore-40XKE-j2-101-32:~# COLUMNS=500 top  -b -n 1 | grep lttng
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    4382 root      20   0 1098824  42600   8436 S   0.0   0.1   0:08.87
lttng-sessiond
    4403 root      20   0   48928   2116    996 S   0.0   0.0   0:00.00
lttng-runas
    5171 root      20   0 9879764   8.9g   8.9g S   0.0  28.7 108:23.53
lttng-consumerd -- here the VRIT is nearly equal to RSS
    5173 root      20   0    3680   1028    680 S   0.0   0.0   0:00.88
lttng-runas


Looks like lttng consumerd is allocating and reserving those pages, when
any instrumented application starts.

I am attaching the lttng status output in the mail, please do tell me if
you need any more information regarding this.


These is how we used to create the lttng channels and enable event which is
same in both buster and bookworm, (number of channels might differ)

def enable_channel(channels, session, subbuf_size, subbuf_num):
for c in channels:
call(['lttng', 'enable-channel', '-u', c, '-s', session, '--subbuf-size',
str(subbuf_size), '--num-subbuf', str(subbuf_num),],
stdout=devnull, stderr=subprocess.STDOUT)


def enable_events(traces, session):
for t in traces:
if 'log-level-only' in t:
log_opt = '--loglevel-only=' + t['log-level-only']
elif 'log-level' in t:
log_opt = '--loglevel=' + t['log-level']
else:
log_opt = ''

else:
call(['lttng', 'enable-event', '-u', t['name'], '-c', t['channel'],
'-s', session], stdout=devnull, stderr=subprocess.STDOUT)


Thank You.
Lakshya




On Wed, Mar 12, 2025 at 8:06 PM <lttng-dev-requ...@lists.lttng.org> wrote:

Send lttng-dev mailing list submissions to
         lttng-dev@lists.lttng.org

To subscribe or unsubscribe via the World Wide Web, visit
         https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
or, via email, send a message with subject or body 'help' to
         lttng-dev-requ...@lists.lttng.org

You can reach the person managing the list at
         lttng-dev-ow...@lists.lttng.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of lttng-dev digest..."

Today's Topics:

    1. Re: Memory Consumption High After Upgrading to 2.13 from 2.10
       (Kienan Stewart)
    2. Re: Memory Consumption High After Upgrading to 2.13 from 2.10
       (Gour DEV)
    3. Re: Memory Consumption High After Upgrading to 2.13 from 2.10
       (Kienan Stewart)

----------------------------------------------------------------------

Message: 1
Date: Tue, 11 Mar 2025 14:55:21 -0400
From: Kienan Stewart <kstew...@efficios.com>
To: Gour DEV <lakshyagou...@gmail.com>, lttng-dev@lists.lttng.org
Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10
Message-ID: <38dab5ef-f106-4e57-9e36-b4b30015c...@efficios.com>
Content-Type: text/plain; charset=UTF-8; format=flowed

Hi Lakshya,

On 3/11/25 12:25 PM, Gour DEV wrote:
  > Hi, Kienan
  >
  > here is the requested output
  >
  > root@localhost:~# top -b -n 1 | grep  lttng
  >     4841 root      20   0   11.5g  11.0g  11.0g S   5.9  35.4   8:39.93
  > lttng-c+
  >     4824 root      20   0 1098824  26456   5380 S   0.0   0.1   0:07.25
  > lttng-s+
  >     4825 root      20   0   48872   2188   1012 S   0.0   0.0   0:00.00
  > lttng-r+
  >     4843 root      20   0    3680   1160    816 S   0.0   0.0   0:00.23

This top output for `localhost` seems very different than the output for
`localhost` in your previous message.

  > lttng-r+
  > root@localhost:~# nrpco
  > bash: nrpco: command not found
  > root@localhost:~# nproc
  > 16
  > root@localhost:~# cat /sys/devices/system/cpu/possible
  > 0-15
  >

You indicated the bookworm machine has 32 cores, this is showing 16. If
you're comparing a 16 core machine to a 32 core machine, it is very
normal that the memory usage is higher on the 32 core machine.

  >
  > Most of the process are running as asorcs user but some are running
as root.

So you have two users with instrumented applications.

Given the discrepancies in the information provided I'm finding it a bit
hard to understand what you're looking at.

In general, a channel's shared memory footprint can be estimated with[1]:

    (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
(nUIDs or nPIDs)

Note that the sub-buffer sizes you are using get rounded to the nearest
larger power of 2. See [2].

thanks,
kienan

[1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes
[2]:
https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size

------------------------------

Message: 2
Date: Wed, 12 Mar 2025 14:49:07 +0530
From: Gour DEV <lakshyagou...@gmail.com>
To: Kienan Stewart <kstew...@efficios.com>
Cc: lttng-dev@lists.lttng.org
Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10
Message-ID:
         <CAE9Jrzg7qsabhPiO-0=B1DY3bVo-3FYu2tiJR2Bmb=
nqohn...@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi, Kienan

I am attaching an screen recording of the behaviour I am seeing in this
mail. The behaviour is same irrespective of the device i use, sorry for
miscommunication in the npocs output (I assumed it was 32), but other than
that all outputs are same (except the hostname as there are multiple
devices with same lttng config but this memory cosumption is seen on all
the devices).

I had few question

1. Does lltng allocated all the memory it needs and mark it as dirty in ram
when any process which links/uses lttng-ust runs? (here i tried with one
process but it is same for any of my process)
2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
(nUIDs or nPIDs)

How do we calculate uid in the system is it all uids in the system? is it
equal to `cat /etc/passwd | wc -l` ?

I will put my calculations according to the above estimate based on all the
channel i am creating

(4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal to
`cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on the
start_lttng.py please do correct me if am wrong here.]

But since there are only two users which uses lttng i think the correct
estimate would be
(4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB

Please do correct me If I am wrong calculations here.

Now, there are a few things here, according to my output lttng is using 11G
which is much more higher than the what is configured.

I am attaching the lttng status and the file which is uses to create the
lttng sessions.

Thank You.

https://drive.google.com/file/d/1tS_ZWEsXDpHZXfWzZHXmWcT0igiIOIaa/view?usp=sharing
-- recording of the behaviour which is seen

https://drive.google.com/file/d/1PrU31oyEw1n9tKETlUtmNGO50s6ywx7p/view?usp=sharing
-- the file which is used to create lttng sessions

On Wed, Mar 12, 2025 at 12:25?AM Kienan Stewart <kstew...@efficios.com>
wrote:

Hi Lakshya,

On 3/11/25 12:25 PM, Gour DEV wrote:
  > Hi, Kienan
  >
  > here is the requested output
  >
  > root@localhost:~# top -b -n 1 | grep  lttng
  >     4841 root      20   0   11.5g  11.0g  11.0g S   5.9  35.4

  8:39.93

  > lttng-c+
  >     4824 root      20   0 1098824  26456   5380 S   0.0   0.1

  0:07.25

  > lttng-s+
  >     4825 root      20   0   48872   2188   1012 S   0.0   0.0

  0:00.00

  > lttng-r+
  >     4843 root      20   0    3680   1160    816 S   0.0   0.0

  0:00.23


This top output for `localhost` seems very different than the output for
`localhost` in your previous message.


  > lttng-r+
  > root@localhost:~# nrpco
  > bash: nrpco: command not found
  > root@localhost:~# nproc
  > 16
  > root@localhost:~# cat /sys/devices/system/cpu/possible
  > 0-15
  >

You indicated the bookworm machine has 32 cores, this is showing 16. If
you're comparing a 16 core machine to a 32 core machine, it is very
normal that the memory usage is higher on the 32 core machine.

  >
  > Most of the process are running as asorcs user but some are running
as root.

So you have two users with instrumented applications.


Given the discrepancies in the information provided I'm finding it a bit
hard to understand what you're looking at.


In general, a channel's shared memory footprint can be estimated with[1]:

    (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
(nUIDs or nPIDs)

Note that the sub-buffer sizes you are using get rounded to the nearest
larger power of 2. See [2].

thanks,
kienan

[1]: https://lttng.org/docs/v2.13/#doc-channel-buffering-schemes
[2]:
https://lttng.org/man/1/lttng-enable-channel/v2.13/#doc-opt--subbuf-size

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
https://lists.lttng.org/pipermail/lttng-dev/attachments/20250312/57f240d8/attachment-0001.htm


------------------------------

Message: 3
Date: Wed, 12 Mar 2025 10:36:28 -0400
From: Kienan Stewart <kstew...@efficios.com>
To: Gour DEV <lakshyagou...@gmail.com>, lttng-dev@lists.lttng.org
Subject: Re: Memory Consumption High After Upgrading to 2.13 from 2.10
Message-ID: <0f819583-ea8e-468e-9102-e1410d886...@efficios.com>
Content-Type: text/plain; charset=UTF-8; format=flowed

Hi Lakshya,

On 3/12/25 5:03 AM, Gour DEV wrote:

Hi, Kienan

I am attaching an screen recording of the behaviour I am seeing in this
mail. The behaviour is same irrespective of the device i use, sorry for
miscommunication in the npocs output (I assumed it was 32), but other

than

that all outputs are same (except the hostname as there are multiple
devices with same lttng config but this memory cosumption is seen on all
the devices).

I had few question

1. Does lltng allocated all the memory it needs and mark it as dirty in

ram

when any process which links/uses lttng-ust runs? (here i tried with one
process but it is same for any of my process)


I believe the shared memory for per-CPU data structures is allocated
when an instrumented application connects. There is no pre-allocation
for each possible UID on the system.

You can run your instrumented applications with `LTTNG_UST_DEBUG=1` to
see when the connection happens[1].

2. (nSubbuf * subbufSize) * (nCPUs + 1 iff snapshot mode is enabled) *
(nUIDs or nPIDs)

How do we calculate uid in the system is it all uids in the system? is it
equal to `cat /etc/passwd | wc -l` ?


nUIDs is the number of distinct UIDs running instrumented applications.


I will put my calculations according to the above estimate based on all

the

channel i am creating

(4194304*4 + 262144*4 + 16384*4) * (16) * (30 if number user are equal to
`cat /etc/passwd | wc -l`)B = 7.998046875 GB approx [this is based on the
start_lttng.py please do correct me if am wrong here.]

But since there are only two users which uses lttng i think the correct
estimate would be
(4194304*4 + 262144*4 + 16384*4) * (16) * (2)B = 546MB


The estimate I gave is per-channel.

small channel: (0.015625 MiB * 4) * (16 + 1) = 1.0625 MiB per-channel
per-UID
medium channel: (0.250 MiB * 4) * (16 + 1) = 17.0 MiB per-channel per-UID
large channel: (4 MiB * 4) * (16 + 1) = 27 2MiB per-channel per-UID

Now, you said you have 0 small channels, 6 medium channels, and 16 large
channels in your session. (Note: I see your script differs from these
stated channel counts).

small: 0 * 1.0625 MiB = 0 MiB per-UID
medium: 6 * 17 MiB = 102 MiB per-UID
large: 16 * 272 MiB = 4352 MiB per-UID

And if you're running instrumented applications with 2 users:

small: 0 MiB * 2 = 0 MiB with 2 UIDs
medium: 102 MiB * 2 = 204 MiB with 2 UIDs
large: 4352 MiB * 2 = 8704 MiB with 2 UIDs

Now this is just an estimation for the per-CPU ring buffers only, and
you numbers aren't hugely off so without analyzing your specific system
it doesn't seem to be that strange to me.

If I take the number of channels I see in your script, it becomes:

small: 0 MiB with 2 UIDs
medium: 136 MiB with 2 UIDs
large: 7616 MiB with 2 UIDs

total: 7.57 GiB with 2 UIDs


Please do correct me If I am wrong calculations here.

Now, there are a few things here, according to my output lttng is using

11G

which is much more higher than the what is configured.


I have no idea what 'service start spyder' is doing. Maybe it's running
instrumented applications with an extra user that you didn't expect? I
can't help you with that aspect of your system.

The above estimated 7.57 GiB with 2 UIDs would be 11.35 GiB with 3 UIDs
so maybe?

I'd recommend you read your verbose sessiond log so see which
applications are connecting and with which UIDs.

I am attaching the lttng status and the file which is uses to create the
lttng sessions.



Thank You.


In any case, the information you have given to date hasn't demonstrated
to me in a tangible manner that you are seeing a difference related to
the version of LTTng being used.

thanks,
kienan

[1]: https://lttng.org/man/3/lttng-ust/v2.13/#doc-_environment_variables


------------------------------

Subject: Digest Footer

_______________________________________________
lttng-dev mailing list
lttng-dev@lists.lttng.org
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


------------------------------

End of lttng-dev Digest, Vol 203, Issue 7
*****************************************

Re: lttng-dev Digest, Vol 203, Issue 7

Reply via email to