Hi Russ - thanks, this is great. I think all of this will be much easier with these discussions happening in the open.
On Monday, March 13, 2017 at 5:09:19 PM UTC-4, Russ Cox wrote: > > On Tue, Mar 7, 2017 at 6:47 AM, Sam Boyer <samuel....@gmail.com > <javascript:>> wrote: > >> Hi folks, >> >> Since we opened up the dep project <https://github.com/golang/dep> in >> late January, there's been flurry of contributions, which has been great. >> However, the plans and end goal of dep haven't been entirely clear to >> everyone. >> >> To address that, and help guide contributors, we've put out a roadmap >> <https://github.com/golang/dep/wiki/Roadmap>. >> >> We're hoping to have dep ready to merge into the toolchain at the >> beginning of the 1.10 cycle. But that'll only happen if the community steps >> up even more to help! >> > > Hi Sam, > > You make it sound like dep is just going to become 'go dep' in Go 1.10. > That's not the plan I thought we discussed. > Ahh, if that's the way it's coming across, I apologize! I've tried to adhere as closely to the plans we've discussed as possible. At times, in some places (particularly high-level docs like roadmaps), I've omitted some detail, in favor of making the project approachable for new contributors. For example, I can see how the wording I used in the OP here could be read as suggesting that it would become `go dep` in 1.10, which would violate your subcommand complexity requirement. That wasn't my intent; I most recently reiterated that requirement earlier today <https://github.com/golang/dep/issues/168#issuecomment-286148978>. My guess is that my general enthusiasm about all of this may be obscuring some of the uncertainty. I mean...I'm enthusiastic about this effort, what can I say :) More on this at the bottom. > > As I wrote to you and the package management group in January: > > I think it's very important instead to weave package and dependency > management into the go command so that you don't notice it as a separate > thing in day-to-day work. As a result, I see dep as an important > improvement over current practice that's usable immediately, an important > step toward a final solution, and also an experiment that helps us learn > more about what structure works for people, but I don't see it, for > example, as a prototype of the eventual go command integration (since it's > not integrated at all). > > In particular, I agree strongly with all of these decisions in dep: > > - Manifest file listing version requirements. > - Lock file listing exact versions for a reproducible build. > - Version control repo as project granularity is good. > - Calling the group of packages released as a unit a "project" is a good > name to establish. > - Implementing coordination with the build by populating vendor is clearly > the right implementation. > > For eventual integration with the go command and the broader go toolchain, > some of the decisions are likely not optimal. In particular: > > - go get vs not. Dep creates a parallel world separate from 'go get'. > Having two is not great. Ideally, I'd like 'go get' to improve instead of > adding a second mechanism. > - Subcommand complexity. As mentioned in #25, if dep foo became 'go dep > foo', then it would be the first sub-sub-commands and likely indicates a > less than clean integration. > - Vendor directories. As an implementation mechanism today, these make > sense. They stop making sense if we can change the toolchain. Instead of > translating the lock file into an ephemeral vendor directory, the tooling > can just read the lock file directly. > - Multiple packages. At scale, I think allowing v1 and v2 of a single > package to be linked into the final binary is unavoidable. It's also not > possible to express in the "top-level vendor directory" model. Another > reason to eliminate the "translate lock file to vendor directory" step. > - GOPATH vs not. As I understand it, dep continues to assume GOPATH, but > in a multiversion world that may not be appropriate. We need to understand > the implications here. GOPATH has to keep working well, but I think we > probably want 'git clone foo && cd foo && do work' outside GOPATH to work > equally well, and package management has an important role to work out here. > > > I understand that not changing the go command was a constraint from the > discussions at Gophercon and also from the point of view of making > something people can use today. That's absolutely appropriate and correct > for dep. But for long-term integration, I think we do need to rethink these > aspects of the go command itself. > > I realize we've discussed these a bit further since that email, but for everyone's benefit: All of these are things that, as you note, can only really be addressed once we're seriously looking at moving directly into the toolchain. It's felt to me like there's a catch-22 in bringing up such things up publicly; doing so would provoke (reasonable) responses of "we're getting ahead of ourselves," but not bringing them up leaves us open to being blindsided when we start seriously looking at toolchain integration. (Subcommand complexity is sort of an exception; we're at least trying to design <https://github.com/golang/dep/issues/277> for minimizing changes when integrating into the toolchain. This seems tricky to me in general, as lessons learned in dep about tool ergonomics may not be ones that transfer well directly into the toolchain...unless we're explicitly thinking ahead to that now.) In any case, I *think* the proto-proposal I wrote <https://gist.github.com/sdboyer/def9b138af448f39300cb078b0e94cc3> addresses everything (except subcommands) in this list. The only thing we differ on there is the future of `go get`, which I think makes sense to focus on as a tool for users, not developers. In general, that proto-proposal is the loosely-held goal I'm keeping in mind, and the capabilities described therein are a pretty straight shot from our current trajectory. > > By far the most important things to do with dep right now are (1) build > something people can use and get benefit from today and eliminate blockers > to adoption and (2) based on experience with that usage, learn what the > eventual design and go command integration should look like. > I think it's certainly the case that we're doing 2) right now, as well as the second half of 1) - eliminating adoption blockers. I did not realize from our discussions, though, that your intent was for dep to live for some time as a standalone tool on which the ecosystem operates. I see it in retrospect, thinking over our discussions, and it does seem like a less risky way of going about this whole transition. Happily, I don't think it necessarliy changes much in terms of our short or medium term priorities, as we've already been moving in that general direction. > > It's important to note that I do not expect projects using dep to > automatically work with the new go command integration too. In particular, > I expect that the lock and manifest files will change in syntax and > therefore file name during integration into the go command, so that it is > possible for projects to work with both dep and the go command during a > transition. > This is not something I recall from earlier discussions. It would be very helpful to hear more about how you imagine that process going. Are there changes you're imagining outside of the multi-version lock (above) and more restricted constraint operations (below)? I had nascent plans to add the former already, but it seems worthwhile to me to at least _try_ to minimize pain during the transition into the toolchain, if we can. Either way, it seems I overreached with the idea of a into-toolchain compatibility guarantee <https://github.com/golang/dep/issues/276>. Sorry. We can walk that back. > I also expect that we need to have a discussion about supported version > constraints, to avoid forcing our tools to solve NP-complete problems > <https://research.swtch.com/version-sat> (it's not just crazy hard, it's > crazy hard to explain to users when it breaks). The best input to that > discussion will be examination of what dep users use. > > I also was hoping to have more thoughts fleshed out more than a month ago, > but I've had a sequence of higher-priority interrupts and haven't had a > chance to do that yet. That's on me, and I apologize for that. (I'm not > going to be around a lot for the next three weeks either, so likely not > much will happen on my end until April.) > Not a problem! While this is a major question and certainly at the heart of how the system works/people will use it, in terms of practical engineering, it's quite contained. The only user-facing effect is, as you noted, the version constraints we allow people to declare. Beyond that, I'm gonna leave this topic for another discussion elsewhere :) > > Dep is another important step - just as vendor dirs were - in the effort > toward proper package management within the go command. It's a way for us > to learn more about what works and what doesn't. There will be iteration > and revision and almost certainly rethinking on the way into the go command > proper. > Of course - I'd have to imagine that everyone expects toolchain integration to be an iterative process. For me at least, the big question here is, "how much of that iteration and revision should we try to do now?" This highlights the tricky position that dep is in. If we limit the scope of our planning with dep to problems that can be solved when outside the toolchain, then we might create a tool that works well for that problem, but makes poor assumptions that will ultimately make transition into the toolchain more difficult (subcommand + flag design is probably the greatest risk here), and painful for those who've begun using dep for real. OTOH, if we *just* experiment with dep as a proto-toolchain tool and never really stabilize it, then we won't ever get that crucial feedback coming from people using dep in anger. There'd also be very little hope of getting data to inform the decision about constraint expressiveness. These are all tensions reflected in your comments - we're in the experiment phase, but also needing to build something that helps, today. The tightrope I've been trying to walk is based on the goal of create two things: 1) a working tool that people have used, designed in a considered way that makes it mostly ready for absorption into the toolchain, and 2) a list of TODOs to take care of once absorption begins. > My concern is that the roadmap and some recent Reddit comments > <https://www.google.com/url?q=https%3A%2F%2Fwww.reddit.com%2Fr%2Fgolang%2Fcomments%2F5y06i3%2Fdep_roadmap_to_becoming_the_official_dep_mgmt_tool%2Fdeo0qcp%2F%3Fst%3Dj08llk2p%26sh%3D08200fe4&sa=D&sntz=1&usg=AFQjCNGhAwz9bvV8O8BURLp8Cfsf-XTCzg> > > do not reflect that level of uncertainty. I believe we're still very much > in the "build and experiment and learn" phase, not the "polish and ship" > phase. > If I'm understanding your overall take correctly, then I'm guessing " *candidate"* was really the problem, as it suggests a direct merge of dep into the toolchain. So again, mea culpa - "the official experiment" may have been better. But, I suspect your concerns may have less to do with any specific wording, and more from the rising tidal wave of effort around this right now. I think that's just peoples' excitement about seeing this problem solved, combined with even a whiff of officialness. And maintaining that momentum seems essential to making this experimental period an effective one - we *want* people on board, we *want* people kicking the tires, because otherwise, we're not getting the information we need. But none of that means we're not maintaining an experimental outlook. I feel like my recent response to Francesc's PR <https://github.com/golang/dep/pull/313#issuecomment-285918383> is a good illustration of that. Like I said, I think we're walking a tightrope. I can certainly be more circumspect in my discussions of the future, but on the whole, IMO we're balancing on this tightrope pretty well so far. cheers s > > Russ > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
