Em Tue, 15 Apr 2025 10:03:47 +0300
Andy Shevchenko <andriy.shevche...@intel.com> escreveu:
> On Tue, Apr 15, 2025 at 10:01:04AM +0300, Andy Shevchenko wrote:
> > On Mon, Apr 14, 2025 at 09:17:51AM -0600, Jonathan Corbet wrote:
> > > Andy Shevchenko <andriy.shevche...@intel.com> writes:
> > > > On Wed, Apr 09, 2025 at 12:30:00PM -0600, Jonathan Corbet wrote:
> > > >> Mauro Carvalho Chehab <mchehab+hua...@kernel.org> writes:
> > > >>
> > > >> > This changeset contains the kernel-doc.py script to replace the
> > > >> > verable
> > > >> > kernel-doc originally written in Perl. It replaces the first version
> > > >> > and the
> > > >> > second series I sent on the top of it.
> > > >>
> > > >> OK, I've applied it, looked at the (minimal) changes in output, and
> > > >> concluded that it's good - all this stuff is now in docs-next. Many
> > > >> thanks for doing this!
> > > >>
> > > >> I'm going to hold off on other documentation patches for a day or two
> > > >> just in case anything turns up. But it looks awfully good.
> > > >
> > > > This started well, until it becomes a scripts/lib/kdoc.
> > > > So, it makes the `make O=...` builds dirty *). Please make sure this
> > > > doesn't leave
> > > > "disgusting turd" )as said by Linus) in the clean tree.
> > > >
> > > > *) it creates that __pycache__ disaster. And no, .gitignore IS NOT a
> > > > solution.
> > >
> > > If nothing else, "make cleandocs" should clean it up, certainly.
Not sure about that, as __pycache__ is completely managed by Python: it
will not only create it for scripts/lib, but also for all Python libraries,
including the Sphinx ones.
IMO, it makes more sense, instead, to ensure that __pycache__ won't be
created at the sourcedir if O= is used, but ignore it if this is created.
Btw, the same problem should already happen with get_abi.py, as it also
uses "import" from scripts/lib. So, we need a more generic solution. See
below.
> > >
> > > We can also tell CPython to not create that directory at all. I'll run
> > > some tests to see what the effect is on the documentation build times;
> > > I'm guessing it will not be huge...
I doubt it would have much impact for kernel-doc, but it can have some impact
for Sphinx, as disabling Python JIT to store bytecode would affect it too.
-
Andy,
Could you please remove __pycache__ and set this env:
PYTHONDONTWRITEBYTECODE=1
before building the Kernel? If this works, one alternative would be to
set it when O= is used.
> > I do not build documentation at all, it's just a regular code build that
> > leaves
> > tree dirty.
> >
> > $ python3 --version
> > Python 3.13.2
> >
> > It's standard Debian testing distribution, no customisation in the code.
> >
> > To reproduce.
> > 1) I have just done a new build to reduce the churn, so, running make again
> > does nothing;
> > 2) The following snippet in shell shows the issue
> >
> > $ git clean -xdf
> > $ git status --ignored
> > On branch ...
> > nothing to commit, working tree clean
> >
> > $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64
> > make[1]: Entering directory '...'
> > GEN Makefile
> > DESCEND objtool
> > CALL .../scripts/checksyscalls.sh
> > INSTALL libsubcmd_headers
> > .pylintrc: warning: ignored by one of the .gitignore files
> > Kernel: arch/x86/boot/bzImage is ready (#23)
> > make[1]: Leaving directory '...'
> >
> > $ touch drivers/gpio/gpiolib-acpi.c
> >
> > $ make LLVM=-19 O=.../out W=1 C=1 CF=-D__CHECK_ENDIAN__ -j64
> > make[1]: Entering directory '...'
> > GEN Makefile
> > DESCEND objtool
> > CALL .../scripts/checksyscalls.sh
> > INSTALL libsubcmd_headers
> > ...
> > OBJCOPY arch/x86/boot/setup.bin
> > BUILD arch/x86/boot/bzImage
> > Kernel: arch/x86/boot/bzImage is ready (#24)
> > make[1]: Leaving directory '...'
> >
> > $ git status --ignored
> > On branch ...
> > Untracked files:
> > (use "git add <file>..." to include in what will be committed)
> > scripts/lib/kdoc/__pycache__/
> >
> > nothing added to commit but untracked files present (use "git add" to
> > track)
>
> FWIW, I repeated this with removing the O=.../out folder completely, so it's
> fully clean build. Still the same issue.
>
> And it appears at the very beginning of the build. You don't need to wait to
> have the kernel to be built actually.
Depending on your .config, kernel-doc will be called even without building
documentation to check for some problems at kernel-doc tags.
>
> > It's 100% reproducible on my side. I am happy to test any patches to fix
> > this.
> > It's really annoying "feature" for `make O=...` builds. Also note that
> > theoretically the Git worktree may be located on read-only storage / media
> > and this can induce subtle issues.
Python's JIT compiler automatically creates __pycache__ whenever it
encounters an "import" and the *.pyc is older than the script (or doesn't
exist). This happens with external libraries, and also with the internal
ones, like the ones we now have at the Kernel.
I dunno what happens if the FS is read-only. I would expect that the JIT
compiler would just work as if bytecode creation is disabled.
That's said, I never played myself with __pycache__.
Yet, I have some raw ideas about how to deal with that. This requires
more tests, though. I can see some possible solutions for that:
1. Assuming that PYTHONDONTWRITEBYTECODE=1 works, the build system could
set it if O= is used. This would have some performance impact for both
Kernel compilation (because kernel-doc is called to check doc issues),
and for Kernel compilation itself. I dunno how much it would impact,
but this is probably the quickest solution to implement;
2. when O=<targetdir> is used, copy scripts/lib/*/*.py to the target
directory and change kernel-doc.py to use <targetdir> for library search
on such case. This way, the __pycache__ would be created at the
<targetdir>. This might work as well with symlinks. The performance
impact here would be minimal, but it will require an extra step for
O= to copy data (or to create symlinks);
3. eventually there is a way to teach Python to place the __pycache__
at <targetdir>, instead of <sourcedir>. If supported, this would
be the cleanest solution.
Regards,
Mauro
