On Mon, Apr 28, 2014 at 10:13 AM, Nicolas Lalevée < nicolas.lale...@hibnet.org> wrote:
> > Le 28 avr. 2014 à 13:59, Josh Suereth <joshua.suer...@gmail.com> a écrit : > > > On a whim, I implemented parallel download of artifacts by hooking the > > downloadArtifacts > > method of Resolve engine. While this can potentially speed up download > > performance, the resolution times (getDepedencies) still dominates (of > > course). Parallel downloads is an often requested feature of sbt, so we > > may still release this hook for our users, as they request it, but I > think > > the *really* just want faster resolution times. > > > > We may be doing some investigation into improving the performance of > > getDependencies (and most importantly fetchDependencies). I was curious > if > > anyone else in the Ivy community has attempted this before and what sorts > > of guidance they could offer before we dig in. > > This has been discussed, but AFAIR nothing has been done other than > discussing the potential issues to tackle. > > Things which would require some attention would be to find a way to > properly report the progress of the parallel download: in the console when > it's done via Ant or via the listeners for IvyDE. > A thing that is worrying me, is that Ivy is quite sensitive to which > thread is actually working on the resolve, see IvyContext's thread local. > So a parallelization may not be a very upper level than the artifact > downloading. > > Totally understand. I haven't had time to check if my resolution on different threads has mucked up the ivy context, but I know I can propagate context as needed, as long as downloading artifacts is idempotent (i.e. no changes to artifacts). The first implementation just defers reporting all progress until AFTER the entire download, then notifies, which probably breaks IvyDE, except I'm only overriding this in sbt itself, so there is no IvyDE there :). So, if this was something to contribute back to ivy (which I'm not sure it is, given the lack of real speed-up we're seeing), I'd have to figure out how to maintain the progress reporting notifications with the IvyContext, possibly adapting my worker pool such that we register a thread-safe IvyContext locally for the reporting events to get fired out of? Does the event model, as it exist, prevent parallelism? In terms of resolution improvements, mostly I was hoping that some memoization/cache of immutable graph pieces could help avoid re-resolving the same bits of the graph repeatedly. Again, I haven't dug fully into the details of IvyNode, ResolveData + VisitNode, but it seems likely to me that we should be able to avoid calling "resolve" for an artifact (the same ModuleRevisionId) more than once during full graph resolution, which (and I may be wrong) appears to be the case when we profile Ivy. I'm mostly concerned with this aspect, and was wondering if anyone else has seen this or if we're doing something very wrong in sbt.
