On 6 April 2014 10:29, Fraser Adams <fraser.ad...@blueyonder.co.uk> wrote:
Hi Robbie,
Firstly thanks so much for taking the time to give such a comprehensive
response. I'll summarise my current thoughts below.
1. The directory structure in qpid/tools/src/java seems fine, I've got no
strong aversion to it. There might be some tweakage, and possibly as you've
suggested before there might be mileage in separating into console, agent,
common+util as most applications using it would only tend to need the
console, common and JMS stuff, but it's probably not worth doing too much
there at the moment 'cause I've no idea what carnage I'll cause when I
start to (eventually) get round to doing AMQP 1.0 Management - quite how I
handle any migration is very much TBD :-/
2. I can see how Maven can (sort of) simplify build scripts, though that
only seems to apply to large projects like the Java Broker, for the QMF
stuff, which is ultimately just a JMS client it seems to have gone from one
fairly simple build.xml which was pretty straightforward to something like
six pom.xml files - there's progress for you :-D
You only have to deal with the top level pom.xml generally when building
stuff, sure there may be more files 'behind' that but you basically dont
need to look at them unless you want to change them, at which point I think
it actually becomes easier to find what you want to change because things
are nicely split up into modules that deal with themselves, and a parent
that everything inherits.
We could have left the non-broker-plugin stuff as one big aggregate jar,
but the reality is that the existing 'qmf.jar' contains a lot more than a
single thing and so splitting it up to at least some extent seemed like the
correct thing to do. The python/original QMF implementation and command
line tools built around it are separated in a similar way, presumably for
similar reasons.
3. In the qpid-qmf2-parent pom I don't really understand the dependencies,
that pom seems to be responsible for the top-level building of the modules,
but why does it have dependencies of its own (log4j, slf4j, jms spec)? I
would get it if this had the common dependencies of all the sub-modules,
but the qpid-qmf2 pom has its own dependencies on slf4j and jms spec -
exposing more holes in my lack of understanding of Maven I guess.
First up, as with qpid-parent yesterday, qpid-qmf2-parent actually has no
dependencies, but it does have a 'dependency management' section, which is
used to ensure consistent alignment of dependency versions in the child
modules and make their poms slightly simpler. See
http://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html#Dependency_Management
Next up, inheritance. The parent pom uses 'pom' packaging and doesnt
produce any output except itself, while the individual modules then use it
as their parent and then inherit much of its contents - such as the
dependencyManagement section that in turn ensures they all use the same
versions for those things going forward.
I could have left out the dependencyManagement section and just repeated
the SLF4J and JMS spec dependency etries in their entirity in every module
requiring them, but that becomes harder to maintain going forward and is
more prone to becoming inconsistent.
Some of this information is overriding bits from qpid-parent pom, which
currently contains a much larger amount of dependency management, but we
are considering moving some of that back to its immediate children as it
doesnt all apply universally to them. I also now realise that a (highly
unusual going forward) change I made yesterday to qpid-parent might not sit
nice with the dependencyManagement section in qpid-qmf2-parent so I'll go
fix that if required - which yes it was, update commmited :)
4. The main benefit that I can see of Maven, aside from (sort of)
simplifying build scripts (which only seems to apply for large builds) is
that it does seem to provide useful artefacts from the perspective of
*users*. That is to say it's biggest (or only as far as I can see ;->)
benefit is from the perspective of release packaging - you end up with some
nice tar.gz files that contain what's needed, so I sort of get why it's
nice from a user perspective - maybe that's worth the price of admission
alone.
We always had the ability to make nice tar.gz files, its just much more of
a pain in the ass to do so in our Ant build, often requiring significant
hoop jumping and typically coming out during the release process that they
were wrong because we dont actually use them the rest of the time (due to
the convenient but horrible global build dir that doesnt represent what
users actually get).
For me the main benefits are the consistency (this obviously doesnt apply
to you yet, but the many people who know maven from use on other projects
will now largely be able to pick up the Qpid build and use most of it
without assistance, while the same wasn't really true of the Ant build
because so much of it is bespoke) and extending from that the ease of
integration with a lot of tooling that for the most part works simply with
or is actually built into maven builds but usually requires pulling teeth
(often mine) to get working with highly bespoke Ant builds such as ours.
It will also mean we (i.e me) dont need to make a special effort to publish
maven artifacts for components to maven central after our users request it
(which they wont need to because we just will), as we (i.e me) always have
in the past, or waste considerable amounts of [my] time verifying those
generated artifacts actually contain what they should every time we release
(which they often dont - I have yet to check them this time round) because
we dont actually use them the rest of the time.
5. From a *dev* perspective though, at the moment the bit of the Maven
rework that a dislike the most is more or less exactly the same as point 4.
above. In order to actually use the Java Broker I have to do mvn clean
install,
You just need to run mvn package currently. You only need to use install if
you want to build something entirely separate using maven which declares it
depends on the artifacts for the broker you just built. Strictly speaking
you only need actually do that if you want to modify the broker first, as
it will otherwise download the release or snapshot version (based on your
specified dependency) from the remote repositories if you dont have it in
your local repo already.
I then need to take the tar.gz from <qpid>/java/broke/target copy it
somewhere and unzip it - inconvenient!
As mentioned yesterday, its simply not finished. Removing this need is a
'nice to have' thing in my mind, which has ranked it below some other
'needs to be done before this is correct / suitable for use at all' things.
I have basically viewed this personally along the lines of....until we do
put something into the poms to start or unpack the broker, we actually have
to use/do something a user e.g. looking at the website would do? Oh the
horror ;)
But that's not the worse bit, when I need to figure out what's broken next
time the plugin API changes there's a whole world of copying and unzipping
to be had, which as well as being inconvenient is prone to finger trouble.
Build and install the broker artifacts (or dont and it will just use the
latest snapshot). Build the QMF bits, which will fail if we broke
something. Dont touch the tar file, or any jar files, or set the classpath.
This stage is far easier than it was before with the Ant build for the QMF
bits.
When it comes to running it all up to make sure it works, yes you will need
to copy some things *currently*. That could certainly be done via config in
the poms if needed to 'automate' it. Its probably worth noting that having
some/any real automated tests for these bits would mean that this step
wouldnt even be necessary much of the time.
A key thing I think is being overlooked is that we will simply try harder
not to break it in future. Because it wont be a complete pain to build as
it is now, we will far more easily be able to ensure it is kept up to date.
We would also now be able to very easily stick it in CI to make sure that
we know we broke it if we hadnt noticed in advance (which I expect we will
more, because it wont be an ordeal to do so anymore).
It's unfortunate that Maven install only "installs" things to the
repository and that there doesn't seem to be a way to install assembly
artefacts onto my system. I guess I could write my own private script that
does all that - but it is as I say inconvenient from a dev perspective.
Thats the primary purpose of mvn install. We can however bind anything we
like to any of the build phases, perhaps adding a profile that unpacks to a
specified directory as a 'system installation' type command etc etc. As I
said yesterday, there are a multitude of options here, it just isnt
something we have got to *yet* because it really is only a convenience
whereas other things are not.
6. Hmmm so I say item 5 was what I dislike the most, actually one thing I
dislike just a bit more is what it does for WebApp development. What I
really like about programming in JavaScript is that I can just make a
change, hit refresh and bingo, none of this build/compile malarkey. Now
from a user perspective it's great that the QMF WebApp is neatly packaged
in qpid-qmf2-tools.tar.gz but from a dev perspective it's a car crash
waiting to happen :-( In order to run QpidRestAPI I have to unpack the
tools, but that will then use the packaged WebApp by default, now there is
a command line option so I can point webroot to
<qpid>/tools/src/java/qpid-qmf2-tools/bin/qpid-web
on trunk, so it's not like I can't work around it, but how long do you give
me before I start accidentally making changes in the "installed" version -
then accidentally lose them all next time I update the QpidRestAPI. I might
be lucky, but I have a very simple brain :-)
You could probably make a profile to it run the GUI straight out of the src
tree without ever building the package. Thats basically how the 'test'
classes work for example since it doesnt even need to build the jar files
to run them.
7. I'm a bit worried about the Maven version stuff. I can see it being a
good thing wrt. being able to do reproducible builds, but does it really
mean that every time a new Qpid release comes out there will be a need to
try and remember to go through dozens of pom.xml files to update various
artefact versions? Given that there are six poms for the QMF stuff alone
that doesn't feel brilliant.
There are plugins that can help with this.
8. It feels like I'm pushed into Mavenising every other little Qpid test
app I've got on my system. Before I had things (albeit perhaps messily) on
my CLASSPATH, but now not.
As you say, this new way is cleaner, but you cancertainly do basically what
you did before if you like, you will just have to specify more locations in
your classpath.
For production things that's fine and the Maven packaging does seem quite
good, but for all the other hacky little things .... sigh, I really do
think Maven is a virus, once one thing catches it you have to "update"
everything else.
You arent really forced in any way in my mind. Sure, it makes something you
are doing slightly uglier, but you are entirely free to keep doing that
hacky thing if you like.
I feel a related consideration also needs to be applied in the opposite
direction, i.e the folks (lots of them) that have long had maven builds and
might come along and want to use the output of our Ant build. Poor sods.
9. All that said, there seems to be an inexorable trend towards Maven, so
from my perspective you might as well move forward on the great Qpid
Mavenisation. I'm not going to stand in the way even if I might moan a bit
until I get used to it :-D.
Feel free to moan away :)
It'd be nice though if there were some solutions to the productivity issues
I realise that this transition might involve a temporary productivity dip
for some who arent as used to it already, but also consider the
productivity increase others will get from it (I'll be honest: whilst I
think that does apply generally, I do primarily mean me, who wont have to
spend so much time rewriting chunks the Ant build to do really simple
things, or waste as much time verifying the output we rarely generate is
actually correct, or <insert time sinks here>..) and that a key driver is
allowing us to progress towards making the project as a whole more
component focused, which will itself improve productivity and wouldnt be
possible without rewriting what we had anyway. We will get to the little
things like starting the broker one its built.
There is obviously a learning curve, but its not that big, once you are
over it I actually think you would actually come to see increased
productivity from using it generally instead of writing Ant scripts (or
dealing directly with javac) and tinkering with the CLASSPATH for example.
- I already struggle with productivity given limited coding time between
work and family stuff, some weekends I feel like I've wasted it all chasing
my tail rather than doing anything useful.
Regards,
Frase
On 05/04/14 18:57, Robbie Gemmell wrote:
On 4 April 2014 14:41, Fraser Adams <fraser.ad...@blueyonder.co.uk>
wrote:
On 04/04/14 13:30, Rob Godfrey wrote:
No... "install in the local repo" is what the mvn clean install will
do... So, you literally just have to follow Robbie's instructions: check
out the qpid-parent-pom/trunk directory, run mvn clean install from
wherever you checked it out to... then go back to the broker and the
maven
build should complete ok. Hope this helps, Rob
Cheers Rob,
I ended up figuring out that myself after staring at the instructions for
a while and deciding that I was reading way too much into things :-)
I've now got the Java Broker etc. built using Maven but before I play
with
the QMF things I wouldn't mind answers to a few Maven questions - I'm
still
very much at the "Burn the Witch" stage wrt. my trust of Maven ;-)
I've now ended up with a directory
/home/fadams/.m2/repository
Filled with stuff.
I'm not clear by what witchcraft Maven decides what to shove there -
Maven has the concept of a local repo and a remote repo, analogous to e.g.
your installed packages in a linux distro, and the repository the package
manager got them from. The default local repo localtion is
~/.m2/repository, and the default remote repo is maven central.
Any dependencies it needs to download will go in the local repo, so it
doenst necessarily need to download them again next time they are used
(unles e.g they are snapshots and new versions came out). Additionally,
anything you run "mvn install" on will go in the local repo configured at
the time (can be overriden per-command for example) to allow it to be used
by other things you build later.
'Any dependencies' above includes the dependencies of a particular
artifact
you are using, the [plugin] dependencies of anything needed by your build
process to do what you are asking, and any [plugin] dependencies needed by
maven itself.
(Additional reading, there are different dependency scopes that can
influence if/when they are needed/used etc:
http://maven.apache.org/guides/introduction/introduction-to-dependency-
mechanism.html#Dependency_Scope
)
for example I eyeballed the parent pom.xml and saw the dependencies,
The parent pom actually has no real dependencies (except on its parent,
the
apache pom).
What you likely saw was the pluginManagement and dependencyManagement
sections, which help fix down and/or consistently specify the versions of
particular dependencies and plugins that might be used by the child
modules
and their build steps. These dont actually mean things have a dependency
on
those artifacts, just which version etc they would get by default if they
should happen to specify such a dependency or plugin to use. This helps
keep things using consistent versions of artifacts, and in the case of
plugins helps ensure reproducable builds by locking down the versions in
use.
but when I looked at the repository after running mvn clean install on
that I was kind of expecting the directories in the repository to pretty
much match what was in the dependencies, but it definitely didn't after
running mvn on the parent pom.xml, though to be fair after building the
main things it does all appear to match (though with a ton of other stuff
too).
What you would have got after doing 'mvn install' on the qpid-parent pom
would be mostly only the plugins needed by maven itself to carry out those
actions. What they ship in the core maven release is augmented by
downloading/installing things to the local repo as they are used both by
itself and by people declaring they want to use a specific plugin in their
build.
When I build the Java Broker etc. it seemed to take an age, when I was
using ant it took under a minute on my system, but with Maven it reports
a
total time of 5:18 min and it looks like it was downloading half the
Internet :-D I'm *guessing* that this is a one-off cost as it fills up my
local repo with stuff?
Yes, the first time you would have incurred the cost of grabbing any
plugins the build needed that you didnt already have and also the
dependencies of the things being built (the later of which happens on your
first Ant build too - go look in ~/.ivy2/cache, then try deleting it and
see what happens to the Ant build time). Run it again and it will be a lot
faster going forward.
On my machine using the head of trunk (I just committed a bunch of tweaks
earlier), 'ant clean build' vs 'mvn clean package -DskipTests=true' has
the
maven build being 1 or 2sec slower (32 vs 33/34), which is actually
surprisingly close since using those commands we currenty have the maven
build doing a bunch of extra packaging and enforcement that the ant build
doesnt. It may also be quicker if told to build 'offline' to ensure it
didnt check any repos for updates.
(It takes a further 2mins 21sec on top of that for me to use the Ant build
to generate the maven artifacts we currently publish to central...a
generation step we obviously wont need to do for the maven build as they
are already there inherantly.)
Where do you set CLASSPATH when you build using Maven? With the ant build
I used to have:
<qpid>qpid/java/build/lib/qpid-all.jar
You dont set CLASSPATH, basically. Supplying dependency information is
all
handled through the poms, which maven uses to build the classpath for each
module as it builds. If you want to build something else that uses e.g the
qpid client, the thing your are building should have a dependency for
qpid-client in its pom. You can make that dependency be either a published
release [or snapshot] version available in central [or the apache
snapshots
repo], or you can 'mvn install' your own modified copy if you want to use
an unpublished client release. If you dont want to use maven / want to use
basic javac commands for the other thing you are compiling against the
client, then you would need to make the client jars available somewhere
and
add the individual jars to your classpath currently.
We aren't recreating qpid-all at this time because while it may be
convenient in cases, it is actually a horrible horrible thing, which I'll
talk a bit more to later when covering the build directory :)
We have a qpid-all file in every release artifact made by the ant build,
each having different content, and all different from the one that gets
created in the build dir. Even if we wanted to recreate it in the maven
build, we would need to give it a unique name for every component to ever
publish it anyway.
We could instead do something like add the classpath manifest entries
previously contained in qpid-all.jar into the main qpid-client,
qpid-broker
etc jars so you could do 'java -jar qpid-foo' etc on them instead.
Which was nice and convenient, if possibly a bit sloppy, from what I can
see each qpid/java subdirectory seems to have a target directory e.g.
qpid/java/client/target though in there there is
qpid-client-0.28-SNAPSHOT.jar which seems less convenient that
qpid-client.jar if I want to set up CLASSPATH in my .bashrc. Am I missing
something?
Another thing that I'm not clear on is that in the Maven repository there
seems to be a bunch of jars installed - as an example
org/eclipse/jetty/jetty-websocket/8.1.14.v20131031/
jetty-websocket-8.1.14.v20131031.jar.
In the "olden days" Qpid pulled in jetty and the Jar was available in
qpid/java/build/lib so when I was messing around with Jetty on another
project I had it handily on my CLASSPATH - I can't seem to see jetty
anywhere now except in the Maven repository? So how does the Broker see
it?
The broker sees it because it says it has a runtime dependency on the
management-http module in its pom, and the management-http module says it
has a compile time dependency on Jetty in its pom, and so maven then knows
the broker transitively needs jetty. The actual jars never leave the local
repo unless we instruct the build to do something that requires that (e.g.
package a binary assembly containing it, or copying it somewhere) in the
pom or you execute a command manually that does (e.g. a quick Google says
'mvn dependency:copy-dependencies' requests the dependency plugin to run
its copy-dependencies goal, which create a dir in target/dependency with
everything in it).
The only reason the Ant build still puts things in the lib dir (by using
Ivy to grab thigns from the maven central repo, i.e roughly the same thing
Maven is doing to download things) is so that I didnt need to rewrite even
more of the Ant build than I already was when I modified it to allow
removing the jars from the lib dir, where they were actually committed
into
the svn repo historically (I refer you to my previous mails comment about
the Ant build being a pain in the ass and me being the one best placed to
speak best to this point :P).
In the olden days when I just did "ant" at the end of that I ended up
with
something that would run 'cause I had
<qpid-trunk>qpid/java/build/bin
on my path and could simply do "qpid-server" but now there's no nice
convenient build directory.
The existing build directory is horrible and should burn :P
I'll admit, it can be convenient, but its also a complete jumble with a
little bit of extra mess on top, mixing up all the different components
together in lib so you cant tell which bits need what (qpid-all.jar in the
build directory being worst of all, as you have everything on the
classpath
manifest in there) and doing ugly copies of stuff into subdirs of lib to
seperate them for later use precisely because you cant tell which bits
need
what. I would have tried to kill it a long time ago if it didnt involve
yet
more 'rewriting the ant build' fun, which I just couldnt bring myself to
subject me to :)
Maven deals with modules, which each have their own target dir for their
intermediate work and final output, and if you want something that
aggregates multiple modules then there are varous ways to do that,
avoiding
the global build dir mess while still letting you compose a larger unit
from the individual bits. It is certainly different than what our Ant
build
does, but in this case I think its a good thing. We can always do things
to
make life easier, more below.
I noticed in
qpid/java/broker/target/qpid-broker-0.28-SNAPSHOT-bin.tar.gz that
archive
seems to contain qpid-server, do I *really* have to now have to faff
around
copying and unpacking archives everytime I want to update a build? That
seems an awful lot less convenient than simply doing "ant clean" then
"ant".
Have I missed something? I hope so 'cause if not it seems a step
backwards
from a "just works" POV
What you say is true *currently*, to run the broker built by the maven
build you would need to unpack that tar file. We can certainly add
functionality to remove that need, we just havent got to those final few
'nice to have' things yet. There are a multitude of ways we could do it,
for example we could use a plugin and have the maven build itself start
the
broker (e.g see what i did with the 'tests' in the QMF tree: those
actually
run without even creating a jar file), or add some build config to allow
output of an unpackaged version of it during the main build, or..<insert
idea>.
This might all be second nature to folks familiar with Maven, but I'm a
bit "old skool" and I quite like knowing what's installed on my system,
where it is installed, and why it's there so I'd quite like a bit of
reassurance :-)
Frase
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
