Hello Loren,
thanks for digging into all this.
It might not be a good thing for softwares like ivy which have a lot of users
in the wild to change defaults - because people like to be able to upgrade and
still have an old behavior, but making the needed behaviors possible at least
should be done.
Your point 2 makes sense to me though, if two artifacts have as version
1.0-SNAPSHOT we should compare their timestamp.
Point 3 your help will be welcome.
Best regards,
Antoine
On Mar 25, 2015, at 3:00 PM, Martin Gainty <mgai...@hotmail.com> wrote:
>
>
>
>> From: lkrat...@blueorigin.com
>> To: dev@ant.apache.org
>> Subject: RE: Possible Ivy bug (and suggested fix) in ChainResolver
>> Date: Wed, 25 Mar 2015 18:16:08 +0000
>>
>> I found the root cause of the fishiness. It was not a bug (and was not
>> actually in ChainResolver) but is in my opinion a rather unfortunate and
>> poorly chosen default setting that is to blame (the latest-strategy).
>>
>> During resolution in the chain resolver, the current sub-resolver tries to
>> determine whether it should be skipped and the previous artifact be selected
>> over the current artifact. Assuming that the "force" and "dynamic" tests do
>> not result in an early rejection of the current artifact, the "latest" of
>> the two artifacts is selected by sorting the artifacts in a List using a
>> Comparator suited to the current latest-strategy (latest-revision,
>> latest-time, etc).
>>
>> The problem here is quite simple. The default latest-strategy is
>> "latest-revision". So when a second artifact has been resolved, and it is of
>> the same revision as the first, nothing gets sorted. The result is that you
>> will get the first artifact found based upon the ordering of your
>> repositories in the chain instead of the newer of the two artifacts.
>>
>> I think that either latest-time needs to be the default strategy or the
>> latest revision comparator needs to do a secondary sort to sort by
>> lastModified time.
>>
>> Possibly allow configuration of this behavior (should there be a secondary
>> sort by time to avoid stale artifacts, or not so that repository order
>> breaks the tie). This is important (critical) behavior and should be
>> configurable.
>>
>> Defaulting to latest-revision will not only deliver undesired stale
>> artifacts, but it is unclear to the user why they are getting stale
>> artifacts or how to make it stop happening. The latest-time strategy will
>> give you the latest revision 99% of the time and the latest artifact 100% of
>> the time. But the latest-revision strategy will give you the latest artifact
>> 100%, 50%, 33%, or 25% of the time when the revision numbers are the same,
>> depending upon how many resolvers you have (1/n), and assuming that any
>> repository may contain the latest artifact.
>>
>> Furthermore, the docs do not give a great description of "force". I learned
>> much about the actual behavior of this attribute while debugging. First
>> thing I learned was that it is not a good name.
>>
>> What force actually does is it allows a resolver to be considered when a
>> previous resolver has found an artifact. After the first artifact has been
>> found, only repositories with force=true will have a chance of competing
>> (for instance, they might have a newer version of 1.0.0-SNAPSHOT).
>> Otherwise, they are discarded immediately and no date comparison is
>> attempted.
>>
>> Force should actually be named "considerAlways" or "considerAnyway". That
>> seems to be a more suitable name. No action requested here, but pointing out
>> that this attribute has a misleading name.
>>
>> Summary:
>> 1 - Please reconsider changing the default latest-strategy to be latest-time.
>>
>> 2 - Please consider adding a secondary "lastModified" sort to
>> LatestRevisionStrategy.ArtifactInfoComparator whether or not you change the
>> default latest-strategy to latest-time or not.
>>
>> 3 - Please document, illustrate, and demonstrate in one place the behaviors
>> of chain resolver in combination with force, returnFirst,
>> defaultLatestStrategy, ivy.resolver.default.check.modified, useOrigin, and
>> other settings and attributes that affect resolution behavior. (I am working
>> on this document now.)
>>
>> 4 - Please consider creating independent caches by default for each
>> repository. I have not drilled down on this issue yet, but I suspect that it
>> fixes serious cache collision issues that I think I saw while debugging
>> (found and selected local repo artifact, checked cache before delivery,
>> ended up delivering cached stale artifact that came from a totally different
>> repo :( ).
>>
>> Thanks,
>>
>> L.K.
>
> MG>i think we can take hints from maven brothers on a tested strategy
> MG>to referencing dev jars during development..their solution is to employ
> SNAPSHOT version during CI cycles
> MG>SNAPSHOTs are available until the jar is promoted to RELEASE at which
> point a tag is assigned to version
> MG>SNAPSHOT delivers ${project.id}-YYYYMMDD.hhmmss.jar so unless you have
> multiple machines able to gen
> MG>deployables within a second the last second is the arbiter which clearly
> identifies the latest jar
> MG>Snapshot versions are ephemeral until the next snapshot build so remote
> lookup would not be implemented
> http://books.sonatype.com/mvnref-book/reference/pom-relationships-sect-pom-syntax.html
>
> MG>repository caches when stored within a regular Nexus Repository are typed
> as Proxy/Hosted/Virtual
> MG>ProxyApache,ProxyCentral or ProxyCodehaus
> MG>Hosted3rdParty,HostedRelease,HostedSnapshot
> MG>VirtualRepo (Virtual repos are generally for OSGI bundles)
> MG>Once you know the general type Proxy or Hosted or Virtual you can then
> select sub-type (such as Hosted3rdParty,HostedRelease,HostedSnapshot)
> https://books.sonatype.com/nexus-book/reference/confignx-sect-manage-repo.html
> MG>thank you for taking the necessary time to think this through
>
>>
>> From: Loren Kratzke
>> Sent: Tuesday, March 24, 2015 1:09 PM
>> To: 'dev@ant.apache.org'
>> Subject: Possible Ivy bug (and suggested fix) in ChainResolver
>>
>> I have a some observations about how the chain resolver selects a
>> dependency. I think this may be a bug but I am not sure because the intent
>> of the source code is not entirely clear. It reads one way, but behaves in a
>> different way. I have pinpointed the exact spots in code where this happens.
>>
>> Here is my simple test setup used to debug this issue. I have two resolvers
>> (Filesystem and URL) configured in a ChainResolver in that order. I publish
>> to one resolver and then the other repeatedly and consume the result in
>> another project. I use checkModified=true and changingPattern=".*" on both
>> resolvers.
>>
>> My artifact is simply a text file with the current date and time so it is
>> easy to see whether you get fresh or stale artifacts from the repos.
>>
>> When I consume the published artifact from the other project, I will get the
>> artifact from the first configured resolver in the chain (Filesystem in this
>> case). But I know from debugging that the second resolver is also evaluated.
>> So as an experiment, I added force="true" on the second resolver to see if I
>> could force Ivy to ignore the first result and favor an artifact returned by
>> the second resolver. Instead, Ivy returned the artifact from the first
>> resolver even though the second artifact was newer AND the second resolver
>> had force="true".
>>
>> When I debugged this to see why the first artifact was chosen over the
>> second artifact, I found something very fishy.
>>
>> ChainResolver.getDependency() iterates over each resolver in the chain.
>> First it found the Filesystem resolver and the artifact and next it found
>> the URL resolver and artifact. Next it calls BasicResolver.getDependency()
>> which will compare the previously resolved artifact with the current
>> artifact.
>>
>> This is where it gets very fishy. At the end of the getDependency() method
>> it calls AbstractResolver.checkLatest() which I assume is intended to return
>> the latest of the two artifacts. But that comparison never happens.
>> AbstractResolver.isAfter is invoked with two artifacts to be compared and a
>> null Date. Since the date is null, the two artifacts are never compared and
>> no matter what, the first artifact will be returned and the second one
>> discarded and a verbose message will be emitted stating that the second
>> artifact is older than the first artifact, every time. The message is on
>> line 533 of AbstractResolver. (I am looking at Ivy-2.3.0 so if that line
>> does not make sense on trunk then let me know.)
>>
>> Message.debug("\tmodule revision kept as younger: " + newModuleDesc);
>> saveModuleRevisionIfNeeded(dd, newModuleFound);
>> return newModuleFound;
>>
>> The message is not true. The artifact that was kept was the older of the two
>> and a comparison of lastModified never happened (and never can happen in the
>> current code as far as I can tell).
>>
>> So the actual logic in AbstractResolver.checkLatest() simply returns the
>> first artifact found. While this is not a bad behavior, it does not seem
>> like it is the intended behavior. I mean, why go through all the trouble of
>> pretending to compare two artifacts using date methods when the logic never
>> executes because the passed in Date object is null. And why emit a message
>> stating that one was determined to be older than the other. That is super
>> fishy.
>>
>> Furthermore, the next line in ChainResolver.getDependency() after
>> resolver.getDependency() is called (ChainResolver line105) references
>> isReturnFirst(). That is fishy because none of that matters any more. The
>> current artifact was rejected on the previous line of code and the previous
>> (aka first) artifact is now the current artifact and is the one that will be
>> returned (without a date comparison, and for the arbitrary reason that is
>> was found before the other one).
>>
>> I think that the intent of the null Date object is to compare each artifact
>> to a static Date configured elsewhere (I have no idea where), but if the
>> code were to actually compare the lastModified dates of the two artifacts, a
>> useful result would happen - Ivy would return the latest artifact from
>> across multiple repositories.
>>
>> That is huge because I have never been able to get Ivy to do this. I have
>> never seen anybody get Ivy to search multiple repositories and return the
>> latest artifact. This is useful for local development when you publish
>> locally to consume locally modified artifacts. It would be nice to have the
>> option of picking up newer artifacts from a central repo when those occur
>> without having to blow away a local repository and its cache.
>>
>> (By the way, giving my local repo its own cache seems to have solved some
>> other strange issues I was having. I recommend this to everybody and I think
>> it should be a default in Ivy, but that is debatable and would need some
>> more research and concensus.)
>>
>> I think that this is a good feature and should be configurable. I think
>> possibly it was intended to be configured via
>> ChainResolver.returnFirst="true|false" but that code executed when it was
>> too late and the decision had already been made. If I were to make this a
>> feature, and make it configurable, I would configure this using an attribute
>> named returnFirst because that is the exact facet of functionality that we
>> are talking about here.
>>
>> Thanks for your attention. Hope I am helping here. I am considering coding
>> this to see if it works as expected. I would be happy to report my results
>> and provide a patch if anybody is interested in evaluating this.
>>
>> L.K.
