Issues with how Ivy processes maven dependency management dependencies compared to adding a classifier key
Hello, At the Howard Hughes Medical Institute Janelia Research Campus, we encountered an issue with transitive dependencies when using the IJava [1] Jupyter kernel that uses Apache Ivy for dependency resolution. When adding the dependency org.janelia.saalfeld.lab.n5-ij [2] Ivy does not correctly recognize net.imglib2.imglib2 [3] as a compile dependency. This is because when processing the parent POM (pom-scijava [4]), Ivy only stores extra information with a groupId__artifactId key without the classifier. The initial fix I have produced is to modify the key to include the classifier: groupId__artifiactId__classifierId. I have prototyped these changes on a Github fork [5]. I would like to discuss how we might upstream these changes to Apache Ivy. Sincerely, Mark Kittisopikul, Ph.D. Software Engineer II Computational Methods, Scientific Computing Janelia Research Campus Howard Hughes Medical Institute [1] https://github.com/SpencerPark/IJava [2] https://github.com/saalfeldlab/n5-ij [3] https://github.com/imglib/imglib2 [4] https://github.com/scijava/pom-scijava [5] https://github.com/mkitti/ant-ivy/commits/mkitti-maven-classifier P.S. Some details on n5-ij to imglib2 dependency chain. n5-ij declares a dependency on n5-imglib2 https://github.com/saalfeldlab/n5-ij/blob/86ac6eb7c2dd77f57cb145902a974f6dc55038f3/pom.xml#L109 n5-imglib2 declares a dependency on imglib2 https://github.com/saalfeldlab/n5-imglib2/blob/20d5051e01aa88afafc040293813fb85a1bd1761/pom.xml#L173-L176 The parent POM for both above projects is pom-scijava which contains a maven dependency management section. The dependency management section contains two entries for net.imglib2.imglib2 Unclassified dependency setting the imglib2 version to 6.1.0: https://github.com/scijava/pom-scijava/blob/9faccfc99fe969bf1b59b5b24d5f446b7f782a14/pom.xml#L2807-L2811 Dependency with tests classifier adding net.imglib2.imglib2 as a dependency with a test scope. https://github.com/scijava/pom-scijava/blob/9faccfc99fe969bf1b59b5b24d5f446b7f782a14/pom.xml#L6623-L6629 I have posted some minimal configuration files here for testing: https://github.com/mkitti/minimal_n5_ij_ivy_test The abbreviated output of `mvn dependency:tree` is as follows. [INFO] \- org.janelia.saalfeldlab:n5-ij:jar:3.2.6:compile [INFO]+- org.janelia.saalfeldlab:n5-imglib2:jar:7.0.0:compile [INFO]| +- net.imglib2:imglib2:jar:6.1.0:compile What I would expect to see from Ivy but I only do after applying my patches: [ivy:dependencytree] \- org.janelia.saalfeldlab#n5-ij;3.2.6 [ivy:dependencytree] | +- org.janelia.saalfeldlab#n5-imglib2;7.0.0 [ivy:dependencytree] | | +- net.imglib2#imglib2;6.1.0 Appling the patches requires that the Ivy cache (default ~/.ivy2/cache) be invalidated. The current dependency management entries from pom-scijava are currently parsed by Ivy as follows, which causes imglib2 to be wrongly classified as a test rather than compile dependency: 6.1.0 test After my patches the dependency management entries are as follows with a classifier entry. With my changes, under the default classifier, imglib2 is now correctly scoped as a compile dependency. 6.1.0 6.1.0 test
Re: Issues with how Ivy processes maven dependency management dependencies compared to adding a classifier key
Hi Stefan, I submitted a cleaner pull request here to add the maven classifier to the extra info key. https://github.com/apache/ant-ivy/pull/99 I am not clear how to create an account to file the issue. When attempting to set up an account via the self service and selecting ant it says " The project you have selected does not use Jira for issue tracking. Please contact the project at dev@ant.apache.org to find out where to submit issues. " https://selfserve.apache.org/jira-account.html Sincerely, Mark From: Stefan Bodewig Sent: Sunday, November 19, 2023 5:04 AM To: dev@ant.apache.org Subject: Re: Issues with how Ivy processes maven dependency management dependencies compared to adding a classifier key External Email: Use Caution Hi Mark first of all please go and read https://urldefense.com/v3/__https://lists.apache.org/thread/h372vt1ztd6gfmgmfkmqzrksx6fpw97g__;!!Eh6p8Q!GgynwlKf6QW3Ni-XDPMGKmczFHKLN_uAlLFMNWuYa6hCRFUwgy7ITEzA4_otBW07MduVoBD-ieBEm5kt19GVUSFtsQ$ I very much doubt you will find any people who know Ivy's internals better than you do (by now) on this list. On 2023-11-17, Kittisopikul, Mark Andrew wrote: > Hello, > At the Howard Hughes Medical Institute Janelia Research Campus, we > encountered an issue with transitive dependencies when using the IJava [1] > Jupyter kernel that uses Apache Ivy for dependency resolution. > When adding the dependency org.janelia.saalfeld.lab.n5-ij [2] Ivy does not > correctly recognize net.imglib2.imglib2 [3] as a compile dependency. This is > because when processing the parent POM (pom-scijava [4]), Ivy only stores > extra information with a groupId__artifactId key without the classifier. > The initial fix I have produced is to modify the key to include the > classifier: groupId__artifiactId__classifierId. I have prototyped these > changes on a Github fork [5]. I would like to discuss how we might upstream > these changes to Apache Ivy. The changes are probably the same as in https://urldefense.com/v3/__https://github.com/mkitti/ant-ivy/pull/1/files__;!!Eh6p8Q!GgynwlKf6QW3Ni-XDPMGKmczFHKLN_uAlLFMNWuYa6hCRFUwgy7ITEzA4_otBW07MduVoBD-ieBEm5kt19EGrRpHlw$ Before I'd try to understand the changes you've made I'd ask you to back out all changes that are not relevant to what you want to achieve. All changes to whitespace, ordering of imports or changes to the eclipse settings for example as they (1) are just noise when it comes to understanding your changes and (2) may hide changes that are overlooked when a reviewer thinks they can skip the changes as they "are just noise". This is not because I wouldn't trust you but rather because I don't trust myself as a reviewer. We've had more than one changeset in Ant that was big, looked good, and introduced subtle bugs that we overlooked. I vaguely recall having to change some classifier stuff in order to fix a bug a few months ago[A] so I am not surprised there are more issues lurking. Personally I'll be glad to review a PR and merge it - please open a JIRA issue as well. As for upstreaming the patch, even after we merged a PR we'd still have to cut a release. Even if we managed to get enough people interested in voting on that, I still want to point you at the thread linked from the top of the mail. Cheers Stefan [A] https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IVY-1642__;!!Eh6p8Q!GgynwlKf6QW3Ni-XDPMGKmczFHKLN_uAlLFMNWuYa6hCRFUwgy7ITEzA4_otBW07MduVoBD-ieBEm5kt19Fy1q8wZA$ - To unsubscribe, e-mail: dev-unsubscr...@ant.apache.org For additional commands, e-mail: dev-h...@ant.apache.org
RE: Issues with how Ivy processes maven dependency management dependencies compared to adding a classifier key
Hi Stefan, I applied for an account again. -Mark -Original Message- From: Stefan Bodewig Sent: Monday, February 5, 2024 12:57 PM To: dev@ant.apache.org Cc: Kittisopikul, Mark Andrew Subject: Re: Issues with how Ivy processes maven dependency management dependencies compared to adding a classifier key External Email: Use Caution Mark can you please try to create a JIRA account again? I've been told it should work now. Thanks Stefan On 2024-02-03, Kittisopikul, Mark Andrew wrote: > Hi Stefan, > I submitted a cleaner pull request here to add the maven classifier to the > extra info key. > https://urldefense.com/v3/__https://github.com/apache/ant-ivy/pull/99_ > _;!!Eh6p8Q!BN7M48YY6oaCJQfxxIbT9-P0vrbwogDDO78bDO_CXYNJkBZm8udLn1Vk-sE > Q0yU8xcv_MJRxEmkm6EDN50csFdhC5Q$ > I am not clear how to create an account to file the issue. When > attempting to set up an account via the self service and selecting ant it > says " > The project you have selected does not use Jira for issue tracking. Please > contact the project at dev@ant.apache.org to find out where to submit issues. > " > https://urldefense.com/v3/__https://selfserve.apache.org/jira-account. > html__;!!Eh6p8Q!BN7M48YY6oaCJQfxxIbT9-P0vrbwogDDO78bDO_CXYNJkBZm8udLn1 > Vk-sEQ0yU8xcv_MJRxEmkm6EDN50dIfjEdGw$ > Sincerely, > Mark > > From: Stefan Bodewig > Sent: Sunday, November 19, 2023 5:04 AM > To: dev@ant.apache.org > Subject: Re: Issues with how Ivy processes maven dependency management > dependencies compared to adding a classifier key > External Email: Use Caution > Hi Mark > first of all please go and read > https://urldefense.com/v3/__https://lists.apache.org/thread/h372vt1ztd > 6gfmgmfkmqzrksx6fpw97g__;!!Eh6p8Q!GgynwlKf6QW3Ni-XDPMGKmczFHKLN_uAlLFM > NWuYa6hCRFUwgy7ITEzA4_otBW07MduVoBD-ieBEm5kt19GVUSFtsQ$ > I very much doubt you will find any people who know Ivy's internals > better than you do (by now) on this list. > On 2023-11-17, Kittisopikul, Mark Andrew wrote: >> Hello, >> At the Howard Hughes Medical Institute Janelia Research Campus, we >> encountered an issue with transitive dependencies when using the IJava [1] >> Jupyter kernel that uses Apache Ivy for dependency resolution. >> When adding the dependency org.janelia.saalfeld.lab.n5-ij [2] Ivy does not >> correctly recognize net.imglib2.imglib2 [3] as a compile dependency. This is >> because when processing the parent POM (pom-scijava [4]), Ivy only stores >> extra information with a groupId__artifactId key without the classifier. >> The initial fix I have produced is to modify the key to include the >> classifier: groupId__artifiactId__classifierId. I have prototyped these >> changes on a Github fork [5]. I would like to discuss how we might upstream >> these changes to Apache Ivy. > The changes are probably the same as in > https://urldefense.com/v3/__https://github.com/mkitti/ant-ivy/pull/1/f > iles__;!!Eh6p8Q!GgynwlKf6QW3Ni-XDPMGKmczFHKLN_uAlLFMNWuYa6hCRFUwgy7ITE > zA4_otBW07MduVoBD-ieBEm5kt19EGrRpHlw$ > Before I'd try to understand the changes you've made I'd ask you to > back out all changes that are not relevant to what you want to > achieve. All changes to whitespace, ordering of imports or changes to > the eclipse settings for example as they (1) are just noise when it > comes to understanding your changes and (2) may hide changes that are > overlooked when a reviewer thinks they can skip the changes as they > "are just noise". > This is not because I wouldn't trust you but rather because I don't > trust myself as a reviewer. We've had more than one changeset in Ant > that was big, looked good, and introduced subtle bugs that we > overlooked. > I vaguely recall having to change some classifier stuff in order to > fix a bug a few months ago[A] so I am not surprised there are more > issues lurking. Personally I'll be glad to review a PR and merge it - > please open a JIRA issue as well. > As for upstreaming the patch, even after we merged a PR we'd still > have to cut a release. Even if we managed to get enough people > interested in voting on that, I still want to point you at the thread > linked from the top of the mail. > Cheers > Stefan > [A] > https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IVY- > 1642__;!!Eh6p8Q!GgynwlKf6QW3Ni-XDPMGKmczFHKLN_uAlLFMNWuYa6hCRFUwgy7ITE > zA4_otBW07MduVoBD-ieBEm5kt19Fy1q8wZA$ > - > To unsubscribe, e-mail: dev-unsubscr...@ant.apache.org For additional > commands, e-mail: dev-h...@ant.apache.org --
