stateful caches (was Re: OBS Studio memory leak)

[Giovanni Biscuolo]

Hi Guillaume Le Vaillant and Guix Devels,

sorry for cross-posting but IMHO the workaround you found [1] for the memory
leak affecting a number of media processing applications is of interest
for many people potentially not subscribed to help-guix

AFAIK this was not filed as a Guix bug

Guillaume Le Vaillant  writes:

> Ott Joon  skribis:
>
>> Hey
>>
>> Tried the same thing in VLC and it freezes on GPU accel and starts
>> leaking memory while also becoming hard to kill.  Maybe this also
>> explains why some mpv GPU accel settings don't work also in the exact
>> same way.  I have an AMD RX 6900 XT on this machine.

[...]

> It looks like an issue with the shader cache of mesa.
> After clearing it, I don't see the memory leak anymore.

good catch: please can you tell us how you managed to spot that problem?
Did you straced it or did yoy find a related mesa bug report?

do you think this bug (is it a bug, right?) needs to be reported
upstream?

I'm asking this because I "feel" we (I mean Guix users) could do
something to help upstream removing this "status mismanagement"

> Could you try doing a "rm -r $HOME/.cache/mesa_shader_cache/*" and see
> if it also solves the issue for you?

AFAIU this is "just" another instance of the "mismanaged state" error
class, like the one(s) discussed back in Oct 2019 [2] and probably
periodically recurring since the beginning of some (many) the upstream
applications lifecycle.

Back then, Efraim Flashner was using this snippet [2] in his OS-config:

--8<---cut here---start->8---

;; This directory shouldn't exist
(file-system
  (device "none")
  (mount-point "/var/cache/fontconfig")
  (type "tmpfs")
  (flags '(read-only))
  (check? #f))

--8<---cut here---end--->8---

It seems that a similar snippet could also be useful for all
"~/.cache/*" :-O

Happy hacking! Gio'



[1] message id:87y1kozvny@robbyzambito.me

[2] message id:20191018073501.GB1224@E5400

-- 
Giovanni Biscuolo

Xelera IT Infrastructures


signature.asc
Description: PGP signature

Re: stateful caches (was Re: OBS Studio memory leak)

[Guillaume Le Vaillant]

Giovanni Biscuolo  skribis:

> Hi Guillaume Le Vaillant and Guix Devels,
>
> sorry for cross-posting but IMHO the workaround you found [1] for the memory
> leak affecting a number of media processing applications is of interest
> for many people potentially not subscribed to help-guix
>
> AFAIK this was not filed as a Guix bug
>
> Guillaume Le Vaillant  writes:
>
>> Ott Joon  skribis:
>>
>>> Hey
>>>
>>> Tried the same thing in VLC and it freezes on GPU accel and starts
>>> leaking memory while also becoming hard to kill.  Maybe this also
>>> explains why some mpv GPU accel settings don't work also in the exact
>>> same way.  I have an AMD RX 6900 XT on this machine.
>
> [...]
>
>> It looks like an issue with the shader cache of mesa.
>> After clearing it, I don't see the memory leak anymore.
>
> good catch: please can you tell us how you managed to spot that problem?
> Did you straced it or did yoy find a related mesa bug report?

I used gdb on versions of mesa and vlc with debug symbols:

--8<---cut here---start->8---
guix build --with-debug-info=mesa --with-debug-info=vlc vlc

gdb /gnu/store/...-vlc-3.0.18/bin/.vlc-real
(gdb) run some-video.mkv
--8<---cut here---end--->8---

Then I sent a SIGSTOP signal to the vlc process, and in gdb I looked at
the backtrace of all the threads of vlc. There was only one thread
allocating things:

--8<---cut here---start->8---
Thread 34

#0  0x77e127c7 in sysmalloc () from 
/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6
#1  0x77e13cfb in _int_malloc () from 
/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6
#2  0x77e145b5 in malloc () from 
/gnu/store/gsjczqir1wbz8p770zndrpw4rnppmxi3-glibc-2.35/lib/libc.so.6
#3  0x7fff810822b1 in ralloc_size () from 
/gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#4  0x7fff81710b2c in read_constant () from 
/gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#5  0x7fff817110f2 in read_var_list () from 
/gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#6  0x7fff81711865 in nir_deserialize () from 
/gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#7  0x7fff8162c874 in tgsi_to_nir () from 
/gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#8  0x7fff8146cba1 in si_create_compute_state () from 
/gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#9  0x7fff8118e1ba in vl_compositor_cs_create_shader () from 
/gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#10 0x7fff8118e800 in vl_compositor_cs_init_shaders () from 
/gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#11 0x7fff811882af in vl_compositor_init () from 
/gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#12 0x7fff810692a8 in __vaDriverInit_1_18 () from 
/gnu/store/5gwcwadzfm48j9cm2b7imw9qcywjyzdz-mesa-23.0.3/lib/dri/radeonsi_drv_video.so
#13 0x7fffacfacdb8 in ?? () from 
/gnu/store/xf6x5qmp75vdnvph041zjdz5779l5q7p-libva-2.18.0/lib/libva.so.2
#14 0x7fffacfadf26 in vaInitialize () from 
/gnu/store/xf6x5qmp75vdnvph041zjdz5779l5q7p-libva-2.18.0/lib/libva.so.2
#15 0x7fff942a7657 in vlc_vaapi_InitializeInstance (o=0x7fff7c001ee0, 
dpy=0x7fff7c21d6c0, native=native@entry=0x7fff7c2137d0, 
native_destroy_cb=native_destroy_cb@entry=0x7fff942a64a0 
) at hw/vaapi/vlc_vaapi.c:95
#16 0x7fff942a7198 in x11_init_vaapi_instance (priv=0x7fff7c2135c0, 
tc=0x7fff7c212e00) at video_output/opengl/converter_vaapi.c:438
#17 Open (obj=0x7fff7c212e00) at video_output/opengl/converter_vaapi.c:521
#18 0x77c9b3f8 in module_load (obj=obj@entry=0x7fff7c212e00, 
m=m@entry=0x4bb570, init=init@entry=0x77c9b320 , 
args=args@entry=0x7fff9478a688)
at modules/modules.c:183
#19 0x77c9b9a4 in vlc_module_load (obj=obj@entry=0x7fff7c212e00, 
capability=capability@entry=0x7fff94685f34 "glconv", name=0x77d38487 "", 
name@entry=0x7fff94685f33 "$glconv", 
strict=strict@entry=true, probe=probe@entry=0x77c9b320 ) 
at modules/modules.c:280
#20 0x77c9bfa4 in module_need (obj=obj@entry=0x7fff7c212e00, 
cap=cap@entry=0x7fff94685f34 "glconv", name=name@entry=0x7fff94685f33 
"$glconv", strict=strict@entry=true)
at modules/modules.c:372
#21 0x7fff9467b46b in opengl_init_program (vgl=vgl@entry=0x7fff7c210690, 
prgm=prgm@entry=0x7fff7c210998, 
glexts=glexts@entry=0x7fff7c210d80 "GL_ARB_multisample GL_EXT_abgr 
GL_EXT_bgra GL_EXT_blend_color GL_EXT_blend_minmax GL_EXT_blend_subtract 
GL_EXT_copy_texture GL_EXT_subtexture GL_EXT_texture_object GL_EXT_vertex_array 
GL_EXT_compiled_"..., fmt=fmt@entry=0x7fff7c0014a8, 
subpics=subpics@entry=f

Re: stateful caches (was Re: OBS Studio memory leak)

[Giovanni Biscuolo]

Hi!

Guillaume Le Vaillant  writes:

[...]

> I used gdb on versions of mesa and vlc with debug symbols:
>
> --8<---cut here---start->8---
> guix build --with-debug-info=mesa --with-debug-info=vlc vlc
>
> gdb /gnu/store/...-vlc-3.0.18/bin/.vlc-real
> (gdb) run some-video.mkv
> --8<---cut here---end--->8---
>
> Then I sent a SIGSTOP signal to the vlc process, and in gdb I looked at
> the backtrace of all the threads of vlc.

got it, thanks!

[...]

>> do you think this bug (is it a bug, right?) needs to be reported
>> upstream?
>
> I guess it would be better if the code reading the shader cache was more
> robust when reading possibly incompatible or corrupted data. However
> I have not tried more recent versions of mesa, maybe they are better at
> it...
>
> And it seems that Maxim has already reported the issue upstream,
> see 

oh I missed it: I'll make my comments in that issue then, thanks!

> and 

I see

Happy hacking. Gio'


-- 
Giovanni Biscuolo

Xelera IT Infrastructures


signature.asc
Description: PGP signature

Re: FSDG issues of SCUMMVM-based games

[Denis 'GNUtoo' Carikli]

Hi,

About the license it's still unclear if it's free or not as some people
pointed on the gnu-linux-libre mailing[1] list that it's a bit similar
to the open font license, but that is a license for fonts not code
though. So I've looked at the other aspects (missing source code, etc).

On Sat, 6 Aug 2022 17:24:52 +0200
Maxime Devos  wrote:
> As such, I agree they should not be distributed (unless the actual 
> sources are found, but even then there is (1.)), as per the first 
> sentence of ‘(guix)GNU Distribution’.

For the lack of missing source code I've looked a bit more in depth
about Drascula and the other games mentioned and here's what I found
below.

Drascula:
-
The file used by ScummVM is Packet.001.

Apparently it's an ARJ archive:
> $ file Packet.001
> Packet.001: ARJ archive data, v7, slash-switched, created 11 dec
> 1980+18, original name: PACKET.ARJ, os: MS-DOS

And binwalk can manage to extract the files from it:
> $ binwalk -e Packet.001

There might be way better tools to do that but I don't know them.

That extracted 1146 files.

One of these files is DRASCULA.COM, and it's safe to assume that it
should be a dos executable because many of these games were also
released for DOS. There are other .COM files too.

File also seems to think that DRASCULA.COM is a dos executable (file is
not 100% reliable).
> $ file DRASCULA.COM 
> DRASCULA.COM: DOS executable (COM), start instruction 0xeb3a9043
> 6f6d7069

I've no idea what most of the files are for but here's their extensions:
> $ ls | sed 's#.*\.##' | sort -u
> ALD
> ALG
> ALS
> arj (that could be the header of the Packet.001 file)
> BIN
> CAL
> CFG
> COM
> DEV
> DRV
> EPA
> EXE
[GSAVE* (GSAVE00, GSAVE01, .. GSAVE10)]
> RCT
> VOC

So:
- We lack the source code for DRASCULA.COM
- If people re-make ARJ archives without any of the executables, and
  that the game still works, then still have issues about the other
  files that might contain code that lack corresponding source code.
  More on that below (in the part about draci-historie).

Lure:
-
The files used by ScummVM are: Disk1.vga Disk2.vga Disk3.vga
and Disk4.vga I don't know much more about the format.

I've also found an executable without source code (Lure.exe) in one of
the source pacakge, but Guix probably has a mechanism to take care of
that to republishing modified source code without Lure.exe.

queen:
--
The file used by ScummVM is queen.1c, but I don't know more.

sky:

The following 3 files are used by ScummVM:
- sky.cpt: Targa image data - Map - RLE 487 x 608 x 1 +353 +412 - 1-bit
   alpha
- sky.dnr: DOS executable (COM), start instruction 0xe913 cfea
- sky.dsk: data

File seems wrong with sky.cpt, and I've no idea if file is write with
sky.dnr.

There are also deeper issues:
--
- All these files are not built from source, so it's complicated to
  understand the provenance of what's inside. And so it's probably
  safer not to ship them (unless we build everything from source)
  because we're unsure that there is some corresponding source code
  inside. 
- Even if there was complete source code inside, we need to be able to
  modify that source code somehow, so doing that might require a lot of
  work to research or build the tools for that.
- ScummVM might contain checksums (See the "4. Play the recompiled
  game" of the draci-historie build documentation [3]) but if someone
  manages to build a game for ScummVM, it could be added as a
  dependency of ScummVM and its checksum could be added inside ScummVM
  at compilation time. 

Also if:
- There are no free programs for ScummVM (a hello world under
  a free license would could count as a free program) (we don't know if
  it's the case or not)
- ScummVM needs to be patched to run modified games (this is very
  likely)
- We don't know if it's possible to build a game for ScummVM
  with only free software (the game doesn't necessarily need to be
  public but free software tools would need to exist to build it).

Then it would clearly steers users toward nonfree software. 

Though here we don't know if there are free programs or not but we also
cannot tell users that scummVM is usefull with a specific free program
or show them a use case that doesn't require nonfree software, so we
might also have a problem with that.

A way forward:
--
A possibility here could be to remove ScummVM and the games that run in
it. Another would be to find a 100% free program for ScummVM (that
doesn't have nonfree dependencies for being modified or rebuilt) or to
find a way to make your own program for ScummVM with 100% free software.

I've looked a bit at another game (draci-historie[2]) that has some
source code published. This game is not packaged nor redistributed by
Guix but it looks way better than the other freedom wise and it can
teach us how ScummVM games are made.

Its probably not good enough as-is as its source code also also relies
on a tarball that contains exec

Re: FSDG issues of SCUMMVM-based games

[Liliana Marie Prikler]

Am Donnerstag, dem 15.06.2023 um 18:30 +0200 schrieb Denis 'GNUtoo'
Carikli:
> Also if:
> - There are no free programs for ScummVM (a hello world under
>   a free license would could count as a free program) (we don't know 
>   if it's the case or not)
> - ScummVM needs to be patched to run modified games (this is very
>   likely)
> - We don't know if it's possible to build a game for ScummVM
>   with only free software (the game doesn't necessarily need to be
>   public but free software tools would need to exist to build it).
> 
> Then it would clearly steers users toward nonfree software. 
I don't think we need to be that harsh on ScummVM itself, it being a
virtual machine.  Compare it to Wine: the tools to create Windows
binaries with free software only are limited (albeit existing if we
discount the necessity for system headers), but it still serves a
purpose by enabling you to run said programs without resorting to a
Windows machine.  Even if no such toolchain exists for ScummVM, you
need ScummVM as a testbed to write one :)

The only limiting factor here is your point (2), i.e. it being able to
run arbitrary games compiled for the VM.  I don't think that weird
checksums ought to be enforced if they're not baked into the program
itself.

> I've looked a bit at another game (draci-historie[2]) that has some
> source code published. This game is not packaged nor redistributed by
> Guix but it looks way better than the other freedom wise and it can
> teach us how ScummVM games are made.
> 
> Its probably not good enough as-is as its source code also also
> relies on a tarball that contains executable to build the game and I
> also didn't manage to build it with Guix yet (I've attached a file
> with my attempt) but maybe it's possible to get it to build and maybe
> we can build a 100% free software version of it.
You might be able to bootstrap parts of it with fpc, i.e. the Free
Pascal Compiler.  I'm not sure whether you'll encounter the necessity
for Borland Pascal, as we package version 3.2.2, which is somewhat
newer than the mentioned 2.4.

Cheers

Re: bug#62264: [PATCH] Add 'guix locate' command

[Ludovic Courtès]

Hi Ryan,

Ryan Prior  skribis:

> It would be helpful to provide a link to any documentation that's part of 
> this work, as neither "guix locate" nor "guix index" have been discussed on 
> guix-devel previously. I look forward to learning more about this feature!

You can find the discussion leading to this patch series at:

  https://issues.guix.gnu.org/62264

Current documentation:

  https://issues.guix.gnu.org/62264#12-lineno60

Before that, there was a preliminary discussion at:

  https://lists.gnu.org/archive/html/guix-devel/2022-01/msg00354.html

HTH!

Ludo’.

Ideas for ocaml-team

[pukkamustard]

Hello Guix,

I think it's time to start an `ocaml-team` (or `ocaml-updates`) branch
to collect some bigger updates and changes to the OCaml packages in
Guix.

Some things that I can think of:

* Update OCaml from 4.14.0 to 4.14.1

* Update OPAM from 2.1.3 to 2.1.5
  - Requires a major update of ocaml-dose3 from 5.0.1 to 7.0.0

* Update Dune from 3.6.1 to 3.8.1

* Update Jane Street packages from 0.15.0 to 0.16.0

* Remove most ocaml4.07-* and ocaml4.09 packages
  - We only want to keep the compiler around for bootstrapping purposes.
  - Update unison 2.51.2 to 2.53.3: This makes it buildable with OCaml
4.14 or even 5.0.  (see
https://lists.gnu.org/archive/html/guix-devel/2023-02/msg00253.html).

* Split packages from (gnu packages ocaml) into multiple modules. Maybe
  in following modules:

  - (gnu packages ocaml): For the compiler and core dev packages (opam,
dune, merlin)
  - (gnu packages ocaml-boot): For the 4.07 and 4.09 compilers
  - (gnu packages ocaml-xyz): Everything else

Thoughts? Any other things? How do we get started with such a branch? 

Cheers,
pukkamustard