Hey Gavin!
Sorry for not replying back...I forgot to write back...
For Hive we can just copy the existing Hive-Precommit job over to the new place.
Actual testing happens outside of builds.apache.org - on a set of cloud nodes -
the job provides a way to keep track of it from the usual location.
And as such we can't just move it and let running the jobs there and start at
the new place as well - it could be probably done on the weekend..
There is one thing which Hive-Precommit relies on - and it's the Precommit-Admin job; I don't know if it will be retained in the new jenkins instance; I don't yet see
anything like that...
As a matter of fact right now I'm working on rebuilding this stuff which will most probably also move out hive precommit runs from the asf infra - it will run a small
jenkins instance in the cloud to service these precommit runs based on github PR..or at least that's how I imagine things going forward...
cheers,
Zoltan
On 4/29/20 3:05 PM, Gavin McDonald wrote:
Hi All,
Following on from the below email I sent *11 DAYS ago now*, so far we have
had *one reply *from mahout cc:d to me (thank you Trevor) , and had *ONE
PERSON* sign up to the new hadoop-migrati...@infra.apache.org -
that is out of a total of *OVER 7000* people signed up to the 13 mailing
lists emailed.
To recap what I asked for:-
"...What I would like from each community, is to decide who is going to
help with their project in performing these migrations - ideally 2 or 3
folks who use the current builds.a.o regularly. Those folks should then
subscribe to the new dedicated hadoop-migrati...@infra.apache.org mailing
lists as soon as possible so we can get started..."
This will be last email I sent to your dev list directly. I am now building
a new Jenkins Master, and as soon as it is ready I will start to migrate
the Jenkins Nodes/Agents over to the new system.
And; when I am done, the existing builds.apache.org *WILL BE TURNED OFF*.
I am now going to continue all conversations on the
hadoop-migrati...@infra.apache.org list *only.*
Thanks
Gavin McDonald (ASF Infra)
On Sat, Apr 18, 2020 at 4:21 PM Gavin McDonald <gmcdon...@apache.org> wrote:
Hi All,
A couple of months ago, I wrote to a few project private lists mentioning
the need to migrate Hadoop labelled nodes (H0-H21) over to a new dedicated
Jenkins Master [1] (a Cloudbees Client Master.).
I'd like to revisit this now that I have more time to dedicate to getting
this done. However, keeping track across multiple mailing lists,
separate conversations that spring up in various places is cumbersome and
not realistic. To that end, I have created a new specific mailing list
dedicated to the migrations of these nodes, and the projects that use them,
over to the new system.
The mailing list 'hadoop-migrati...@infra.apache.org' is up and running
now (and this will be the first post to it). Previous discussions were on
the private PMC lists, (there was some debate about that but I wanted the
PMCs initially to be aware of the change,) this new list is public and
archived.
This email is BCC'd to 13 projects dev lists [2] determined by the https:/
hadoop.apache.org list of Related projects, minus Cassandra whom already
have their own dedicated client master [3] and I added Yetus as I think
they cross collaborate with many Hadoop based projects. If anyone thinks a
project is missing, or should not be on the list, let me know.
What I would like from each community, is to decide who is going to help
with their project in performing these migrations - ideally 2 or 3 folks
who use the current builds.a.o regularly. Those folks should then subscribe
to the new dedicated hadoop-migrati...@infra.apache.org mailing lists as
soon as possible so we can get started.
About the current setup - and I hope this answers previously asked
questions on private lists - the new dedicated master is a Cloudbees Client
Master 2.204.3.7-rolling. It is not the same setup as the current Jenkins
master on builds.a.o - it is not intended to be. It is more or less a
'clean install' in that I have not installed over 500 plugins as is the
case on builds.a.o , I would rather we install plugins as we find we need
them. So yes, there may be some features missing - the point of having
people sign up to the new list is to find out what those are, get them
installed, and get your builds to at least the same state they are in
currently.
We have 2 nodes on there currently for testing, as things progress we can
transfer over a couple more, projects can start to migrate their jobs over
at any time they are happy , until done. We also need to test auth - the
master; and its nodes will be restricted to just Hadoop + Related projects
(which is important this list of related projects is correct). No longer
will other projects be able to hop on to Hadoop nodes, and no longer will
Hadoop related projects be able to hop onto other folks nodes. This is a
good thing, and may encourage some providers to donate a few more VMs for
dedicated use.
For now then, decide who will help with this process, and sign up to the
new mailing list, and lets get started!
Note I am NOT subscribed to any of your dev lists, so replies please cc
the new list. and I will await your presence there to get started.
Thanks all.
Gavin McDonald (ASF Infra)
[1] - https://ci-hadoop.apache.org
[2] -
hadoop,chukwa,avro,ambari,hbase,hive,mahout,pig,spark,submarine,tez,zookeeper,yetus
[3] - https://ci-cassandra.apache.org
