Alex, I just saw you posted this last message. This will need more
tweaking over time it seems. It's not clear to me if we can have a
commons-parent that works generically for both single and multi-module
projects for CycloneDX and/or SPDX.
Gary
On Tue, Sep 20, 2022 at 7:56 AM Alex Herbert <alex.d.herb...@gmail.com> wrote:
>
> Hi Gary,
>
> I have found part of the issue. I copied the default configuration for
> cyclone DX maven plugin [1] into the statistics project. This worked.
>
> The only difference between this config and the config in parent is:
>
> Default:
>
> <outputName>bom</outputName>
>
> CP:
>
> <outputName>${project.artifactId}-${project.version}-bom</outputName>
>
> Unfortunately the documentation on what this value is used for is lacking
> (see [2]). With the default setting the BOM is simply named bom.xml in the
> target directory. Previously there had been a bom created for each module
> (parent and dependencies) in the module target directory. I presume now
> each of these is overwriting the previous one as they are all named
> bom.xml. When a later install mojo comes along and tries to find the file
> it will pick up bom.xml even if the correct file it is looking for
> (identified by artifact-version was never generated).
>
> When the bom is installed it is correctly named, e.g.
> commons-statistics-distribution-1.0-SNAPSHOT-cyclonedx.xml.
> However the files are different. The installed bom has dependency
> information collated from other modules which are not actually
> dependencies. So the aggregation is bringing in dependencies incorrectly.
> This makes the BOM incorrect.
>
> Statistics has this structure:
>
> - commons-statistics-parent
> + commons-statistics-distribution
> + commons-statistics-regression
>
> regression does not depend on distribution. Both child modules have
> external dependencies. Unfortunately the dependencies for regression are a
> subset of distribution and so it is impossible to identify the difference
> between a union of them and simply all the dependencies from distribution.
>
> The target/bom.xml for the parent and regression modules matches that
> installed into the repo. The target/bom.xml and the installed distribution
> module bom do not match. The installed bom adds information on the
> regression module (on which it does not depend). However it does at least
> create a bom and allow the install. Why the CP configuration works for the
> outputName property for VFS and not Statistics I do not know.
>
> A quick check for VFS shows that the mangling of dependencies occurs there.
> For example the installed bom for commons-vfs-jackrabbit1 has a component
> entry for commons-vfs2-jackrabbit2. This seems to be a mistake. I have not
> delved much deeper into the generated boms. It appears that the bom for
> each module is being generated with the dependencies of the entire project
> reactor.
>
> If I update the cycloneDX configuration to use:
>
> <outputReactorProjects>false</outputReactorProjects>
>
> Then the target/bom.xml from each module exactly matches that installed in
> the maven repo. However they still include more components than the true
> dependencies of the individual module. For example the statistics
> distribution and regression modules include each other. All the rest of the
> dependencies are identical, which is not the case for the actual modules.
> So the entire set of dependencies for the reactor are still ending up in
> the installed bom.
>
> Alex
>
> [1] https://github.com/CycloneDX/cyclonedx-maven-plugin
> [2]
> https://cyclonedx.github.io/cyclonedx-maven-plugin/makeAggregateBom-mojo.html#outputName
>
> On Tue, 20 Sept 2022 at 11:39, Alex Herbert <alex.d.herb...@gmail.com>
> wrote:
>
> > Hi Gary,
> >
> > I tried VFS. On my mac it did not pass the unit tests:
> >
> > [*ERROR*] *Errors: *
> >
> > [*ERROR*] *
> > AbstractSftpProviderTestCase$SftpProviderTestSuite>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->setUp:235->AbstractTestSuite.setUp:268
> > » FileSystem Could not connect to SFTP server at
> > "sftp://testtest@localhost:51426/".*
> >
> > [*ERROR*] *
> > SftpPermissionExceptionTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> > » FileSystem Could not connect to SFTP server at
> > "sftp://testtest@localhost:51426/".*
> >
> > [*ERROR*] *
> > SftpProviderClosedExecChannelTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> > » FileSystem Could not connect to SFTP server at
> > "sftp://testtest@localhost:51426/".*
> >
> > [*ERROR*] *
> > SftpProviderStreamProxyModeTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> > » FileSystem Could not connect to SFTP server at
> > "sftp://testtest@localhost:51426/".*
> >
> > I've never built this project before so I do not know if this is just a
> > flaky build. FYI:
> >
> > *Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)*
> >
> > Maven home: /usr/local/apache-maven-3.6.3
> >
> > Java version: 11.0.12, vendor: Eclipse Foundation, runtime:
> > /Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home
> >
> > Default locale: en_GB, platform encoding: UTF-8
> >
> > OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac"
> >
> > I tried on linux where 'mvn install' ran OK (it took ~14 minutes). Here it
> > worked OK. The CycloneDX plugin creates a bom for each project module in
> > every module target directory, e.g.
> >
> > ./commons-vfs2-jackrabbit2/target/commons-vfs2-2.10.0-SNAPSHOT-bom.xml
> > ./commons-vfs2-jackrabbit2/target/commons-vfs2-project-2.10.0-SNAPSHOT-bom.xml
> >
> > ./commons-vfs2-jackrabbit2/target/commons-vfs2-examples-2.10.0-SNAPSHOT-bom.xml
> >
> > ./commons-vfs2-jackrabbit2/target/commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-bom.xml
> >
> > ./commons-vfs2-jackrabbit2/target/commons-vfs2-jackrabbit1-2.10.0-SNAPSHOT-bom.xml
> >
> >
> > ./commons-vfs2-jackrabbit2/target/commons-vfs2-distribution-2.10.0-SNAPSHOT-bom.xml
> >
> > When installed the local maven repository only contains:
> >
> > commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-cyclonedx.json
> > commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-cyclonedx.xml
> >
> > The installed file matches
> > commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-bom.xml. A quick check in the
> > other modules and it is the same. The bom matching the module name matches
> > the installed cylonedx file in the maven repo. So here I think the
> > plugin is working correctly.
> >
> > I tried Commons Numbers again on linux and got the same result (an error
> > installing on the first module). So this may require some work on a minimal
> > multi-module project to find out what is causing the issue. Note that on
> > the projects I tried (RNG, Numbers, Statistics) they all have a first
> > module that does not include any dependencies. I added one with a test case
> > to exercise the code using the dependency but the install error still
> > occurred. All these projects have the same multi-module structure and so I
> > can investigate what is different between these and VFS.
> >
> > Alex
> >
> >
> > On Tue, 20 Sept 2022 at 00:52, Gary Gregory <garydgreg...@gmail.com>
> > wrote:
> >
> >> Hi Alex,
> >>
> >> Thank you for the review.
> >>
> >> - .gitattributes: Yes let's do that for the next release. In addition,
> >> there has been talk about this and recent changes around these types of
> >> files on the Maven mailing list but we can and should handle these in our
> >> parent POM for now.
> >>
> >> - CycloneDX: At the time I integrated this, I tested with Commons VFS and
> >> nothing broke but it is unfortunate that the plugin does some odd things
> >> in
> >> a multi module project. Would report this as an issue to CycloneDX?
> >>
> >> In general, and in light of security issues in the software ecosystem, I
> >> think that providing these metadata is important, so I am willing to go
> >> through some of the growing pains but handling multi-module projects needs
> >> to get fixed upstream in CycloneDX.
> >>
> >> Gary
> >>
> >>
> >> On Mon, Sep 19, 2022, 17:07 Alex Herbert <alex.d.herb...@gmail.com>
> >> wrote:
> >>
> >> > Cloned and installed locally from the git tag.
> >> >
> >> > I updated Commons RNG to use parent 54 and tested with:
> >> >
> >> > mvn clean package
> >> >
> >> > I had to add '.gitattributes' to a list of excluded files for the
> >> > apache-rat plugin. Not a blocker but this could be moved to
> >> commons-parent.
> >> >
> >> > The new bill of materials generated by CycloneDX is generated for all
> >> > modules and appears in the target directory. But there seems to be an
> >> issue
> >> > with this process.
> >> >
> >> > I tested a release:
> >> >
> >> > mvn -Dcommons.release.dryRun=true -Ptest-deploy -Prelease clean verify
> >> > deploy
> >> >
> >> > Here I get an error message from the install for the CycloneDX bom.
> >> >
> >> > [*ERROR*] Failed to execute goal
> >> > org.apache.maven.plugins:maven-install-plugin:2.5.2:install
> >> > *(default-install)* on project commons-rng-client-api: *Failed to
> >> install
> >> > artifact
> >> > org.apache.commons:commons-rng-client-api:xml:cyclonedx:1.5-SNAPSHOT:
> >> >
> >> >
> >> /Users/ah403/git/commons-rng/commons-rng-client-api/target/commons-rng-parent-1.5-SNAPSHOT-bom.xml
> >> > (No such file or directory)* -> *[Help 1]*
> >> >
> >> > The bom files are:
> >> >
> >> > ./target/commons-rng-parent-1.5-SNAPSHOT-bom.xml
> >> >
> >> >
> >> ./commons-rng-client-api/target/commons-rng-client-api-1.5-SNAPSHOT-bom.xml
> >> >
> >> > ./commons-rng-core/target/commons-rng-simple-1.5-SNAPSHOT-bom.xml
> >> > ./commons-rng-core/target/commons-rng-core-1.5-SNAPSHOT-bom.xml
> >> > ./commons-rng-core/target/commons-rng-client-api-1.5-SNAPSHOT-bom.xml
> >> > ./commons-rng-core/target/commons-rng-sampling-1.5-SNAPSHOT-bom.xml
> >> > ./commons-rng-core/target/commons-rng-parent-1.5-SNAPSHOT-bom.xml
> >> >
> >> > ./commons-rng-simple/target/commons-rng-simple-1.5-SNAPSHOT-bom.xml
> >> > ./commons-rng-simple/target/commons-rng-client-api-1.5-SNAPSHOT-bom.xml
> >> > ./commons-rng-simple/target/commons-rng-sampling-1.5-SNAPSHOT-bom.xml
> >> > ./commons-rng-simple/target/commons-rng-parent-1.5-SNAPSHOT-bom.xml
> >> >
> >> >
> >> ./commons-rng-sampling/target/commons-rng-client-api-1.5-SNAPSHOT-bom.xml
> >> > ./commons-rng-sampling/target/commons-rng-sampling-1.5-SNAPSHOT-bom.xml
> >> > ./commons-rng-sampling/target/commons-rng-parent-1.5-SNAPSHOT-bom.xml
> >> >
> >> >
> >> > For some reason the CycloneDX bom for commons-rng-parent is placed in
> >> the
> >> > target directory for all the child modules except
> >> commons-rng-client-api.
> >> > So the install fails on this module.
> >> >
> >> > I do not know what is different about this module. It has no
> >> dependencies
> >> > other than commons-rng-parent. The other modules are all dependent on
> >> it.
> >> > It seems to be the issue that it is the first child module.
> >> >
> >> > A simpler multi-module project is Commons Statistics. It only has one
> >> child
> >> > module. This works with CP 54 for 'mvn verify -Dspotbugs.skip
> >> -Dpmd.skip'
> >> > (I did not suppress 'new' bugs found by an upgraded spotbugs and a weird
> >> > PMD runtime error) but fails for 'mvn install -Dspotbugs.skip
> >> -Dpmd.skip'
> >> > for the same error. So it seems in the first child module of the
> >> > multi-module project the parent bom is not copied by cyclone DX.
> >> >
> >> > As a final test I tried with Commons Numbers. This again works for 'mvn
> >> > verify' but not 'mvn install' with the same issue. The first child
> >> module
> >> > is missing the bom for the parent module. Other child modules appear to
> >> > have a bom for all their dependencies.
> >> >
> >> > I see that you did revert the Cyclone DX version to an earlier version
> >> due
> >> > to issues with building the parent POM. So perhaps this is another bug
> >> in
> >> > CycloneDX for multi-module builds.
> >> >
> >> > This is not a blocker as the plugin can simply be disabled. However it
> >> is
> >> > not ideal as this plugin is meant to add traceability to the build and
> >> > currently it does not work for multi-module projects as configured.
> >> >
> >> > Alex
> >> >
> >> >
> >> > On Sun, 18 Sept 2022 at 22:39, Bruno Kinoshita <ki...@apache.org>
> >> wrote:
> >> >
> >> > > [x] +1 Release these artifacts
> >> > >
> >> > > Thanks!
> >> > >
> >> > > On Mon, 19 Sept 2022 at 03:47, Gary Gregory <garydgreg...@gmail.com>
> >> > > wrote:
> >> > >
> >> > > > We have fixed a few bugs and added enhancements since Apache Commons
> >> > > > Parent 53 was released, so I would like to release Apache Commons
> >> > > > Parent 54.
> >> > > >
> >> > > > Apache Commons Parent 54 RC1 is available for review here:
> >> > > >
> >> > https://dist.apache.org/repos/dist/dev/commons/commons-parent/54-RC1
> >> > > > (svn revision 56878)
> >> > > >
> >> > > > The Git tag commons-parent-54-RC1 commit for this RC is
> >> > > > efd8232f4811706ad21bc3583e32d2473256b8d8 which you can browse here:
> >> > > >
> >> > > >
> >> > >
> >> >
> >> https://gitbox.apache.org/repos/asf?p=commons-parent.git;a=commit;h=efd8232f4811706ad21bc3583e32d2473256b8d8
> >> > > > You may checkout this tag using:
> >> > > > git clone
> >> https://gitbox.apache.org/repos/asf/commons-parent.git
> >> > > > --branch <
> >> > https://gitbox.apache.org/repos/asf/commons-parent.git--branch
> >> > > >
> >> > > > commons-parent-54-RC1 commons-parent-54-RC1
> >> > > >
> >> > > > Maven artifacts are here:
> >> > > >
> >> > > >
> >> > >
> >> >
> >> https://repository.apache.org/content/repositories/orgapachecommons-1594/org/apache/commons/commons-parent/54/
> >> > > >
> >> > > > These are the artifacts and their hashes:
> >> > > >
> >> > > > #Release SHA-512s
> >> > > > #Sun Sep 18 11:32:16 EDT 2022
> >> > > > Apache\ Commons\
> >> > > >
> >> > > >
> >> > >
> >> >
> >> Parent-54.spdx.rdf.xml=a5ca11505acdfefabc2bff44f52566220929d3f1b4b7164c9fea0adf4fcb8c04223f5e27089698615264e89a071400a72b19ffc54516343cacbfdeffcf3a7776
> >> > > >
> >> > > >
> >> > >
> >> >
> >> commons-parent-54-bom.json=ce0bf440d926a725e840459034d59cfe9f9bfc5b9131bee087ed2e80859a8064a5efb2c8abeb9997b08ad8fe693b1a8587c38721cca7ff63701e1ee1407ac17c
> >> > > >
> >> > > >
> >> > >
> >> >
> >> commons-parent-54-bom.xml=2e2f29e1d26d9f5493ea83ea9707109f755fea41a16949f56438338875ee3e21c44a362d9f58c265bf43adb7a250647c463faa3275ba042eb8673686f6a29adf
> >> > > >
> >> > > >
> >> > >
> >> >
> >> commons-parent-54-site.xml=735ffceca46a0574d430b4e1213a2462b9475143c0788913312b8af117eaf3b7c02a075aaf6d9b30d2560822339651cb511b838f6c9f2bced46de1fc1227c5ff
> >> > > >
> >> > > >
> >> > >
> >> >
> >> commons-parent-54-src.tar.gz=7b800ea9fcb607e2e44dea906d203abdc4452872b207b4ae4229090c3e9dc471f53dea6515c487453eeb17aef833b7394ee00cb1a9edd424cfc7bb6860841e07
> >> > > >
> >> > > >
> >> > >
> >> >
> >> commons-parent-54-src.zip=9b3674b54052c7b56e9f3b1fe5a8bdf6673007e2c1e9a9aff2491fefdc04554550a6725bc58fe92f3b417e1284e5a61b20004fbcf514f9df0e1ef832a56bc890
> >> > > >
> >> > > > I have tested this with 'mvn -V -Duser.name=$my_apache_id
> >> > > > -Ddoclint=none -Prelease -Ptest-deploy clean package site deploy'
> >> > > > using:
> >> > > >
> >> > > > Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
> >> > > > Maven home: /usr/local/Cellar/maven/3.8.6/libexec
> >> > > > Java version: 1.8.0_345, vendor: Homebrew, runtime:
> >> > > > /usr/local/Cellar/openjdk@8
> >> > > > /1.8.0+345/libexec/openjdk.jdk/Contents/Home/jre
> >> > > > Default locale: en_US, platform encoding: UTF-8
> >> > > > OS name: "mac os x", version: "12.6", arch: "x86_64", family: "mac"
> >> > > >
> >> > > > Darwin ***.local 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22
> >> > > > 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64 x86_64
> >> > > >
> >> > > > Details of changes since 53 are in the release notes:
> >> > > >
> >> > > >
> >> > >
> >> >
> >> https://dist.apache.org/repos/dist/dev/commons/commons-parent/54-RC1/RELEASE-NOTES.txt
> >> > > >
> >> > > >
> >> > >
> >> >
> >> https://dist.apache.org/repos/dist/dev/commons/commons-parent/54-RC1/site/changes-report.html
> >> > > >
> >> > > > Site:
> >> > > >
> >> > > >
> >> > >
> >> >
> >> https://dist.apache.org/repos/dist/dev/commons/commons-parent/54-RC1/site/index.html
> >> > > > (note some *relative* links are broken and the 54 directories
> >> are
> >> > > > not yet created - these will be OK once the site is deployed.)
> >> > > >
> >> > > > RAT Report:
> >> > > >
> >> > > >
> >> > >
> >> >
> >> https://dist.apache.org/repos/dist/dev/commons/commons-parent/54-RC1/site/rat-report.html
> >> > > >
> >> > > > KEYS:
> >> > > > https://www.apache.org/dist/commons/KEYS
> >> > > >
> >> > > > Please review the release candidate and vote.
> >> > > > This vote will close no sooner than 72 hours from now.
> >> > > >
> >> > > > [ ] +1 Release these artifacts
> >> > > > [ ] +0 OK, but...
> >> > > > [ ] -0 OK, but really should fix...
> >> > > > [ ] -1 I oppose this release because...
> >> > > >
> >> > > > Thank you,
> >> > > >
> >> > > > Gary Gregory,
> >> > > > Release Manager (using key 86fdc7e2a11262cb)
> >> > > >
> >> > > > For following is intended as a helper and refresher for reviewers.
> >> > > >
> >> > > > Validating a release candidate
> >> > > > ==============================
> >> > > >
> >> > > > These guidelines are NOT complete.
> >> > > >
> >> > > > Requirements: Git, Java, Maven.
> >> > > >
> >> > > > You can validate a release from a release candidate (RC) tag as
> >> > follows.
> >> > > >
> >> > > > 1) Clone and checkout the RC tag
> >> > > >
> >> > > > git clone https://gitbox.apache.org/repos/asf/commons-parent.git
> >> > > > --branch commons-parent-54-RC1 commons-parent-54-RC1
> >> > > > cd commons-parent-54-RC1
> >> > > >
> >> > > > 2) Check Apache licenses
> >> > > >
> >> > > > This step is not required if the site includes a RAT report page
> >> which
> >> > > > you then must check.
> >> > > >
> >> > > > mvn apache-rat:check
> >> > > >
> >> > > > 3) Build the package
> >> > > >
> >> > > > mvn -V clean verify
> >> > > >
> >> > > > You can record the Maven and Java version produced by -V in your
> >> VOTE
> >> > > > reply.
> >> > > > To gather OS information from a command line:
> >> > > > Windows: ver
> >> > > > Linux: uname -a
> >> > > >
> >> > > > 4) Build the site for a single module project
> >> > > >
> >> > > > Note: Some plugins require the components to be installed instead of
> >> > > > packaged.
> >> > > >
> >> > > > mvn site
> >> > > > Check the site reports in:
> >> > > > - Windows: target\site\index.html
> >> > > > - Linux: target/site/index.html
> >> > > >
> >> > > > -the end-
> >> > > >
> >> > > >
> >> ---------------------------------------------------------------------
> >> > > > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> >> > > > For additional commands, e-mail: dev-h...@commons.apache.org
> >> > > >
> >> > > >
> >> > >
> >> >
> >>
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org
