Am 24.10.19 um 17:13 schrieb Jean-Marie LEMETAYER:
Hi Stefan,
On Oct 24, 2019, at 2:01 PM, Stefan Herbrechtsmeier ste...@herbrechtsmeier.net
wrote:
Am 22.10.19 um 13:22 schrieb Richard Purdie:
On Tue, 2019-10-22 at 11:03 +0200, Jean-Marie LEMETAYER wrote:
The current NPM support have several issues:
- The current NPM fetcher downloads the dependency tree but not the other
fetchers. The 'subdir' parameter was used to fix this issue.
- They are multiple issues with package names (uppercase, exotic characters,
scoped packages) even if they are inside the dependencies.
- The lockdown file generation have issues. When a package depends on
multiple version of the same package (all versions have the same checksum).
This patchset refactors the NPM support in Yocto:
- As the NPM algorithm for dependency management is hard to handle, the new
NPM fetcher downloads only the package source (and not the dependencies,
like the other fetchers) (patch submitted in the bitbake-devel list).
What make the new fetcher different from the simple wget fetcher?
The fetcher asks the registry for the tarball url and its integrity. Then it
uses wget to download the tarball and then check the integrity given by the
registry. Finally it unpacks the tarball according to the tarball format (the
root directory is called 'package' which must be renamed).
Does this means you don't add the SRC_URI checksum to the recipe?
- The NPM class handles the dependencies using NPM (and not manually).
Is this really an improvement? NPM will do the cross compile during
fetch, loads additionally archives (not packages) from the internet and
doesn't reuse dependencies.
In the next version of this patchset the dependencies are cached in a npm_cache
directory stored in the DL_DIR during the do_fetch_append task (using 'npm
cache --cache=${DL_DIR}/npm_cache'). The 'npm install' doesn't require any
internet accesses because it is run with '--offline' option.
Does the npm cache works together with a download proxy?
- The NPM recipe creation is simplified to avoid issues.
We create new not obvious issues. How you would handle prebuild binaries?
I have no case of using prebuild binaries. After checking some informations it
seems that it is handled the same way as node-gyp (by using scripts in the
package.json). A patchset to support that is welcome.
The problem is that prebuild is only one example and you detect the
issue only if you remove the internet connection.
- The lockdown file is no more used as it is no longer relevant compared to
the
latest shrinkwrap file format.
This patchset may remove some features (lockdown file, license management for
dependencies)
You really remove the license management of the dependencies? I think a
main feature of OE is the license management.
The previous npm support was creating subpackages to handle the license
management of the dependencies. This is a good idea but it fails with package
with exotic names. A work will be needed to cleanly remake it.
The license was independent of the sub packages. You need to parse all
the package.json and dependency folders.
but fixes the majority of the NPM issues. All of these issues
from the bugzilla.yoctoproject.org are resolved by this patchset:
#10237, #10760, #11028, #11728, #11902, #12534
One key requirement which many of our users have from the fetcher is
that its deterministic and allows for "offline" builds.
I think this is impossible with npm because every dependency could run a
script and download additional files (ex. prebuild).
I have no case of using prebuild binaries. Can you give me an example of a
package using prebuild?
You could use the pouchdb package as an example and test it with an ARM
soft float machine.
Do you know about the '--offline' and the '--ignore-scripts' options? A
patchset to support that is welcome.
Can you point me to the '--offline' option.
The '--ignore-scripts' option will disable all scripts and breaks binary
packages.
What this means is that should I have a populated DL_DIR, the build
should not need to touch the network. Also, only do_fetch tasks would
make network accesses.
@Richard: What is your opinion about the per recipe dependency?
Typically OE use one recipe per project. The NPM based solution handle a
project and all dependencies via one recipe.
@Jean-Marie: Do you know PNPM? They use a different node_modules layout
which allows the reuse of dependencies.
Thanks for the link. What is the value in our scope ?
They use a different node_modules layout which avoid duplicates. Maybe
it more interesting for my prototype.
Regards
Stefan
--
_______________________________________________
Openembedded-core mailing list
Openembedded-core@lists.openembedded.org
http://lists.openembedded.org/mailman/listinfo/openembedded-core