Re: [VOTE][LAZY] Release Apache Commons Parent 54 based on RC1
Hi Gary, I tried VFS. On my mac it did not pass the unit tests: [*ERROR*] *Errors: * [*ERROR*] * AbstractSftpProviderTestCase$SftpProviderTestSuite>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->setUp:235->AbstractTestSuite.setUp:268 » FileSystem Could not connect to SFTP server at "sftp://testtest@localhost:51426/".* [*ERROR*] * SftpPermissionExceptionTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268 » FileSystem Could not connect to SFTP server at "sftp://testtest@localhost:51426/".* [*ERROR*] * SftpProviderClosedExecChannelTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268 » FileSystem Could not connect to SFTP server at "sftp://testtest@localhost:51426/".* [*ERROR*] * SftpProviderStreamProxyModeTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268 » FileSystem Could not connect to SFTP server at "sftp://testtest@localhost:51426/".* I've never built this project before so I do not know if this is just a flaky build. FYI: *Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)* Maven home: /usr/local/apache-maven-3.6.3 Java version: 11.0.12, vendor: Eclipse Foundation, runtime: /Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home Default locale: en_GB, platform encoding: UTF-8 OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac" I tried on linux where 'mvn install' ran OK (it took ~14 minutes). Here it worked OK. The CycloneDX plugin creates a bom for each project module in every module target directory, e.g. ./commons-vfs2-jackrabbit2/target/commons-vfs2-2.10.0-SNAPSHOT-bom.xml ./commons-vfs2-jackrabbit2/target/commons-vfs2-project-2.10.0-SNAPSHOT-bom.xml ./commons-vfs2-jackrabbit2/target/commons-vfs2-examples-2.10.0-SNAPSHOT-bom.xml ./commons-vfs2-jackrabbit2/target/commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-bom.xml ./commons-vfs2-jackrabbit2/target/commons-vfs2-jackrabbit1-2.10.0-SNAPSHOT-bom.xml ./commons-vfs2-jackrabbit2/target/commons-vfs2-distribution-2.10.0-SNAPSHOT-bom.xml When installed the local maven repository only contains: commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-cyclonedx.json commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-cyclonedx.xml The installed file matches commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-bom.xml. A quick check in the other modules and it is the same. The bom matching the module name matches the installed cylonedx file in the maven repo. So here I think the plugin is working correctly. I tried Commons Numbers again on linux and got the same result (an error installing on the first module). So this may require some work on a minimal multi-module project to find out what is causing the issue. Note that on the projects I tried (RNG, Numbers, Statistics) they all have a first module that does not include any dependencies. I added one with a test case to exercise the code using the dependency but the install error still occurred. All these projects have the same multi-module structure and so I can investigate what is different between these and VFS. Alex On Tue, 20 Sept 2022 at 00:52, Gary Gregory wrote: > Hi Alex, > > Thank you for the review. > > - .gitattributes: Yes let's do that for the next release. In addition, > there has been talk about this and recent changes around these types of > files on the Maven mailing list but we can and should handle these in our > parent POM for now. > > - CycloneDX: At the time I integrated this, I tested with Commons VFS and > nothing broke but it is unfortunate that the plugin does some odd things in > a multi module project. Would report this as an issue to CycloneDX? > > In general, and in light of security issues in the software ecosystem, I > think that providing these metadata is important, so I am willing to go > through some of the growing pains but handling multi-module projects needs > to get fixed upstream in CycloneDX. > > Gary > > > On Mon, Sep 19, 2022, 17:07 Alex Herbert wrote: > > > Cloned and installed locally from the git tag. > > > > I updated Commons RNG to use parent 54 and tested with: > > > > mvn clean package > > > > I had to add '.gitattributes' to a list of excluded files for the > > apache-rat plugin. Not a blocker but this could be moved to > commons-parent. > > > > The new bill of materials generated by CycloneDX is generated for all > > modules and appears in the target directory. But there seems to be an > issue > > with this process. > > > > I tested a release: > > > > mvn -Dcommons.release.dryRun=true -Ptest-deploy -Prelease clean verify > > deploy > > > > Here I get an error message from the install for the CycloneDX bom. > > > > [*ERROR*] Failed to execute goal > > org.apache.maven.plugins:maven-install-plug
Re: [VOTE][LAZY] Release Apache Commons Parent 54 based on RC1
Hi Gary,
I have found part of the issue. I copied the default configuration for
cyclone DX maven plugin [1] into the statistics project. This worked.
The only difference between this config and the config in parent is:
Default:
bom
CP:
${project.artifactId}-${project.version}-bom
Unfortunately the documentation on what this value is used for is lacking
(see [2]). With the default setting the BOM is simply named bom.xml in the
target directory. Previously there had been a bom created for each module
(parent and dependencies) in the module target directory. I presume now
each of these is overwriting the previous one as they are all named
bom.xml. When a later install mojo comes along and tries to find the file
it will pick up bom.xml even if the correct file it is looking for
(identified by artifact-version was never generated).
When the bom is installed it is correctly named, e.g.
commons-statistics-distribution-1.0-SNAPSHOT-cyclonedx.xml.
However the files are different. The installed bom has dependency
information collated from other modules which are not actually
dependencies. So the aggregation is bringing in dependencies incorrectly.
This makes the BOM incorrect.
Statistics has this structure:
- commons-statistics-parent
+ commons-statistics-distribution
+ commons-statistics-regression
regression does not depend on distribution. Both child modules have
external dependencies. Unfortunately the dependencies for regression are a
subset of distribution and so it is impossible to identify the difference
between a union of them and simply all the dependencies from distribution.
The target/bom.xml for the parent and regression modules matches that
installed into the repo. The target/bom.xml and the installed distribution
module bom do not match. The installed bom adds information on the
regression module (on which it does not depend). However it does at least
create a bom and allow the install. Why the CP configuration works for the
outputName property for VFS and not Statistics I do not know.
A quick check for VFS shows that the mangling of dependencies occurs there.
For example the installed bom for commons-vfs-jackrabbit1 has a component
entry for commons-vfs2-jackrabbit2. This seems to be a mistake. I have not
delved much deeper into the generated boms. It appears that the bom for
each module is being generated with the dependencies of the entire project
reactor.
If I update the cycloneDX configuration to use:
false
Then the target/bom.xml from each module exactly matches that installed in
the maven repo. However they still include more components than the true
dependencies of the individual module. For example the statistics
distribution and regression modules include each other. All the rest of the
dependencies are identical, which is not the case for the actual modules.
So the entire set of dependencies for the reactor are still ending up in
the installed bom.
Alex
[1] https://github.com/CycloneDX/cyclonedx-maven-plugin
[2]
https://cyclonedx.github.io/cyclonedx-maven-plugin/makeAggregateBom-mojo.html#outputName
On Tue, 20 Sept 2022 at 11:39, Alex Herbert
wrote:
> Hi Gary,
>
> I tried VFS. On my mac it did not pass the unit tests:
>
> [*ERROR*] *Errors: *
>
> [*ERROR*] *
> AbstractSftpProviderTestCase$SftpProviderTestSuite>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->setUp:235->AbstractTestSuite.setUp:268
> » FileSystem Could not connect to SFTP server at
> "sftp://testtest@localhost:51426/".*
>
> [*ERROR*] *
> SftpPermissionExceptionTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> » FileSystem Could not connect to SFTP server at
> "sftp://testtest@localhost:51426/".*
>
> [*ERROR*] *
> SftpProviderClosedExecChannelTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> » FileSystem Could not connect to SFTP server at
> "sftp://testtest@localhost:51426/".*
>
> [*ERROR*] *
> SftpProviderStreamProxyModeTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> » FileSystem Could not connect to SFTP server at
> "sftp://testtest@localhost:51426/".*
>
> I've never built this project before so I do not know if this is just a
> flaky build. FYI:
>
> *Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)*
>
> Maven home: /usr/local/apache-maven-3.6.3
>
> Java version: 11.0.12, vendor: Eclipse Foundation, runtime:
> /Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home
>
> Default locale: en_GB, platform encoding: UTF-8
>
> OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac"
>
> I tried on linux where 'mvn install' ran OK (it took ~14 minutes). Here it
> worked OK. The CycloneDX plugin creates a bom for each proj
Re: [VOTE][LAZY] Release Apache Commons Parent 54 based on RC1
Hi Alex,
Thank you for the additional testing and reporting.
As a baseline, the VFS git master build is green on macOS, Windows,
and Linux using Java 8, 11, and 17:
https://github.com/apache/commons-vfs/actions where the current latest
CI build for git master is
https://github.com/apache/commons-vfs/actions/runs/3068521283
With VFS git master plus a local change to update to commons-parent
from 53 to 54, I ran the default Maven goal from the command line
('mvn') on macOS using:
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /usr/local/Cellar/maven/3.8.6/libexec
Java version: 1.8.0_345, vendor: Homebrew, runtime:
/usr/local/Cellar/openjdk@8/1.8.0+345/libexec/openjdk.jdk/Contents/Home/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "12.6", arch: "x86_64", family: "mac"
Darwin gdg-mac-mini.local 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug
22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64 x86_64
My local build passed in about 12 minutes.
Maybe you had a random failure or were not running from the command
line. Some VFS tests won't run properly from IDEs because they depend
on the old VFS testing framework still in place that relies on some
JUnit 3 patterns.
WRT SBOMS like CycloneDX and multi-module projects, I think we need to
live with the growing pains for now.
I just tested a single module component -- Commons Text -- and that
worked and produced and installed the right files.
TY!
Gary
On Tue, Sep 20, 2022 at 6:40 AM Alex Herbert wrote:
>
> Hi Gary,
>
> I tried VFS. On my mac it did not pass the unit tests:
>
> [*ERROR*] *Errors: *
>
> [*ERROR*] *
> AbstractSftpProviderTestCase$SftpProviderTestSuite>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->setUp:235->AbstractTestSuite.setUp:268
> » FileSystem Could not connect to SFTP server at
> "sftp://testtest@localhost:51426/".*
>
> [*ERROR*] *
> SftpPermissionExceptionTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> » FileSystem Could not connect to SFTP server at
> "sftp://testtest@localhost:51426/".*
>
> [*ERROR*] *
> SftpProviderClosedExecChannelTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> » FileSystem Could not connect to SFTP server at
> "sftp://testtest@localhost:51426/".*
>
> [*ERROR*] *
> SftpProviderStreamProxyModeTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> » FileSystem Could not connect to SFTP server at
> "sftp://testtest@localhost:51426/".*
>
> I've never built this project before so I do not know if this is just a
> flaky build. FYI:
>
> *Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)*
>
> Maven home: /usr/local/apache-maven-3.6.3
>
> Java version: 11.0.12, vendor: Eclipse Foundation, runtime:
> /Library/Java/JavaVirtualMachines/temurin-11.jdk/Contents/Home
>
> Default locale: en_GB, platform encoding: UTF-8
>
> OS name: "mac os x", version: "11.5", arch: "x86_64", family: "mac"
>
> I tried on linux where 'mvn install' ran OK (it took ~14 minutes). Here it
> worked OK. The CycloneDX plugin creates a bom for each project module in
> every module target directory, e.g.
>
> ./commons-vfs2-jackrabbit2/target/commons-vfs2-2.10.0-SNAPSHOT-bom.xml
> ./commons-vfs2-jackrabbit2/target/commons-vfs2-project-2.10.0-SNAPSHOT-bom.xml
>
> ./commons-vfs2-jackrabbit2/target/commons-vfs2-examples-2.10.0-SNAPSHOT-bom.xml
>
> ./commons-vfs2-jackrabbit2/target/commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-bom.xml
>
> ./commons-vfs2-jackrabbit2/target/commons-vfs2-jackrabbit1-2.10.0-SNAPSHOT-bom.xml
>
> ./commons-vfs2-jackrabbit2/target/commons-vfs2-distribution-2.10.0-SNAPSHOT-bom.xml
>
> When installed the local maven repository only contains:
>
> commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-cyclonedx.json
> commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-cyclonedx.xml
>
> The installed file matches
> commons-vfs2-jackrabbit2-2.10.0-SNAPSHOT-bom.xml. A quick check in the
> other modules and it is the same. The bom matching the module name matches
> the installed cylonedx file in the maven repo. So here I think the
> plugin is working correctly.
>
> I tried Commons Numbers again on linux and got the same result (an error
> installing on the first module). So this may require some work on a minimal
> multi-module project to find out what is causing the issue. Note that on
> the projects I tried (RNG, Numbers, Statistics) they all have a first
> module that does not include any dependencies. I added one with a test case
> to exercise the code using the dependency but the install error still
> occurred. All these projects have the same multi-module structure and so I
> can investigate what is different between these and VF
Re: [VOTE][LAZY] Release Apache Commons Parent 54 based on RC1
Alex, I just saw you posted this last message. This will need more
tweaking over time it seems. It's not clear to me if we can have a
commons-parent that works generically for both single and multi-module
projects for CycloneDX and/or SPDX.
Gary
On Tue, Sep 20, 2022 at 7:56 AM Alex Herbert wrote:
>
> Hi Gary,
>
> I have found part of the issue. I copied the default configuration for
> cyclone DX maven plugin [1] into the statistics project. This worked.
>
> The only difference between this config and the config in parent is:
>
> Default:
>
> bom
>
> CP:
>
> ${project.artifactId}-${project.version}-bom
>
> Unfortunately the documentation on what this value is used for is lacking
> (see [2]). With the default setting the BOM is simply named bom.xml in the
> target directory. Previously there had been a bom created for each module
> (parent and dependencies) in the module target directory. I presume now
> each of these is overwriting the previous one as they are all named
> bom.xml. When a later install mojo comes along and tries to find the file
> it will pick up bom.xml even if the correct file it is looking for
> (identified by artifact-version was never generated).
>
> When the bom is installed it is correctly named, e.g.
> commons-statistics-distribution-1.0-SNAPSHOT-cyclonedx.xml.
> However the files are different. The installed bom has dependency
> information collated from other modules which are not actually
> dependencies. So the aggregation is bringing in dependencies incorrectly.
> This makes the BOM incorrect.
>
> Statistics has this structure:
>
> - commons-statistics-parent
> + commons-statistics-distribution
> + commons-statistics-regression
>
> regression does not depend on distribution. Both child modules have
> external dependencies. Unfortunately the dependencies for regression are a
> subset of distribution and so it is impossible to identify the difference
> between a union of them and simply all the dependencies from distribution.
>
> The target/bom.xml for the parent and regression modules matches that
> installed into the repo. The target/bom.xml and the installed distribution
> module bom do not match. The installed bom adds information on the
> regression module (on which it does not depend). However it does at least
> create a bom and allow the install. Why the CP configuration works for the
> outputName property for VFS and not Statistics I do not know.
>
> A quick check for VFS shows that the mangling of dependencies occurs there.
> For example the installed bom for commons-vfs-jackrabbit1 has a component
> entry for commons-vfs2-jackrabbit2. This seems to be a mistake. I have not
> delved much deeper into the generated boms. It appears that the bom for
> each module is being generated with the dependencies of the entire project
> reactor.
>
> If I update the cycloneDX configuration to use:
>
> false
>
> Then the target/bom.xml from each module exactly matches that installed in
> the maven repo. However they still include more components than the true
> dependencies of the individual module. For example the statistics
> distribution and regression modules include each other. All the rest of the
> dependencies are identical, which is not the case for the actual modules.
> So the entire set of dependencies for the reactor are still ending up in
> the installed bom.
>
> Alex
>
> [1] https://github.com/CycloneDX/cyclonedx-maven-plugin
> [2]
> https://cyclonedx.github.io/cyclonedx-maven-plugin/makeAggregateBom-mojo.html#outputName
>
> On Tue, 20 Sept 2022 at 11:39, Alex Herbert
> wrote:
>
> > Hi Gary,
> >
> > I tried VFS. On my mac it did not pass the unit tests:
> >
> > [*ERROR*] *Errors: *
> >
> > [*ERROR*] *
> > AbstractSftpProviderTestCase$SftpProviderTestSuite>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->setUp:235->AbstractTestSuite.setUp:268
> > » FileSystem Could not connect to SFTP server at
> > "sftp://testtest@localhost:51426/".*
> >
> > [*ERROR*] *
> > SftpPermissionExceptionTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> > » FileSystem Could not connect to SFTP server at
> > "sftp://testtest@localhost:51426/".*
> >
> > [*ERROR*] *
> > SftpProviderClosedExecChannelTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> > » FileSystem Could not connect to SFTP server at
> > "sftp://testtest@localhost:51426/".*
> >
> > [*ERROR*] *
> > SftpProviderStreamProxyModeTestCase$1>AbstractTestSuite.run:239->AbstractTestSuite.lambda$run$0:234->AbstractSftpProviderTestCase$SftpProviderTestSuite.setUp:235->AbstractTestSuite.setUp:268
> > » FileSystem Could not connect to SFTP server at
> > "sftp://testtest@localhost:51426/".*
> >
> > I've never built this project before so I do not know if this is just a
> > flaky build. FYI:
> >
> > *Apach
Re: [VOTE][LAZY] Release Apache Commons Parent 54 based on RC1
Hi Gary, On Tue, 20 Sept 2022 at 12:59, Gary Gregory wrote: > > Maybe you had a random failure or were not running from the command > line. Some VFS tests won't run properly from IDEs because they depend > on the old VFS testing framework still in place that relies on some > JUnit 3 patterns. > The failed tests are trying to connect to a local FTP server. It could be due to an issue with firewall configuration on my macbook preventing them from starting. Or perhaps just a flaky test. I'll run it again later to check. I can also try a different Maven and JDK. > WRT SBOMS like CycloneDX and multi-module projects, I think we need to > live with the growing pains for now. > On Tue, 20 Sept 2022 at 13:09, Gary Gregory wrote: > Alex, I just saw you posted this last message. This will need more tweaking over time it seems. It's not clear to me if we can have a commons-parent that works generically for both single and multi-module projects for CycloneDX and/or SPDX. IIUC to release the projects I tested I would simply have to update the property for CycloneDX back to the default. The installed BOM for each module will then contain information from the entire project reactor. This will at least contain information on the true dependencies for the module. I am not sure what effect having the extra redundant information will have for users of this feature. It is a pity that the documentation for CycloneDX is basically absent. Some of the settings are not entirely self-documenting. I think this should be reported as a bug to CycloneDX. I will look into that. It should require a simple project with 2 modules, each with different dependencies. IIUC the default config for the plugin will create a bom for each module with too much information when installed. Alex
Re: [VOTE][LAZY] Release Apache Commons Parent 54 based on RC1
Hello. > [...] The installed bom has dependency > information collated from other modules which are not actually > dependencies. So the aggregation is bringing in dependencies incorrectly. > This makes the BOM incorrect. > [...] If that's the case, I suggest that this feature is disabled by default in CP. RM should be aware that the release could contain wrong information (which IMHO is worse than no information). Gilles - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [VOTE][LAZY] Release Apache Commons Parent 54 based on RC1
Hi, I have put together a simple project with a parent and two modules, each with their own dependencies. This has the same result in that the installed bom for each module includes the dependencies of the entire project reactor. When I change the goal from 'makeAggregateBom' to 'makeBom' then I see the behaviour I expect. Each module has a bom that only includes the direct dependencies of the project module. This holds for the installed bom that is attached during install. I think the goal we require when building separate installed jar files in a multi module project is 'makeBom' and not 'makeAggregateBom'. The lack of documentation on the Cyclone DX website does not help distinguish the two. The fact that the default execution is 'makeAggregateBom' also does not help. If I directly add the Cyclone DX plugin config from CP 54 to Commons Statistics (but not via CP 54) but change the default execution from makeAggregateBom to makeBom, then the plugin works as I would expect. I have not tested this with a single module commons project. Alex On Tue, 20 Sept 2022 at 14:22, Gilles Sadowski wrote: > Hello. > > > [...] The installed bom has dependency > > information collated from other modules which are not actually > > dependencies. So the aggregation is bringing in dependencies incorrectly. > > This makes the BOM incorrect. > > [...] > > If that's the case, I suggest that this feature is disabled by default > in CP. RM should be aware that the release could contain wrong > information (which IMHO is worse than no information). > > Gilles > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > >
