Re: [OpenStack-Infra] [gear] Making Gear easier to consume ( less .encode() and .decode() )

2016-06-21 Thread James E. Blair
Morgan Fainberg writes: > As I have been converting Zuul and NodePool to python3, I have had to do a > bunch of changes around encode() and decode() of strings since gear is > (properly) an implementation of a protocol that requires binary data > (rather than text_strings). > > What this has high

Re: [OpenStack-Infra] Problem with Cross-Repo Dependencies in Zuul

2016-06-22 Thread James E. Blair
Artur Zarzycki writes: > ... > And now as I understand zuul should recognize dependencies between patches. > So I created patch1 in repo1 and patch2 in repo2 with Depends-On: > I45137d1186caeafda0cee3504370d01ef3d9d271 (patch1) > and I'm trying to merge patch2. I see that zuul run gate jobs for

[OpenStack-Infra] Nodepool schema change

2016-06-22 Thread James E. Blair
Hi, In nodepool master we just merged a change (add "--reason" to 'nodepool hold' command) which will require a schema change. To do so, run the following: mysql> alter table node add column comment varchar(255) after state_time; -Jim ___ OpenStack-I

Re: [OpenStack-Infra] Switching test jobs to Xenial and the Trusty, Xenial split

2016-06-28 Thread James E. Blair
Clark Boylan writes: > The downside here is we double our total number of jobs (though we do > not double the number of gearman job registrations since gearman will > register a job per node type regardless of option used). It is also much > more explicit and will likely require a greater underst

Re: [OpenStack-Infra] Switching test jobs to Xenial and the Trusty, Xenial split

2016-06-28 Thread James E. Blair
Clark Boylan writes: > Could we do this by slightly changing the zuul gearman protocol to drop > the "label" suffix and expect that all jobs have this hardcoded? Then we > could do the second option and we would have: > > build:tempest-trusty > build:tempest-xenial > > This should come out to rou

[OpenStack-Infra] Announcing Gertty 1.4.0

2016-07-26 Thread James E. Blair
Announcing Gertty 1.4.0 === Gertty is a console-based interface to the Gerrit Code Review system. Gertty is designed to support a workflow similar to reading network news or mail. It syncs information from Gerrit to local storage to support disconnected operation and easy man

Re: [OpenStack-Infra] [openstack-dev] Announcing Gertty 1.4.0

2016-07-27 Thread James E. Blair
MichaƂ Dulko writes: > Just wondering - were there tries to implement syntax highlighting in > diff view? I think that's the only thing that keeps me from switching to > Gertty. I don't know of anyone working on that, but I suspect it could be done using the pygments library. -Jim

[OpenStack-Infra] New IRC channel: #zuul

2016-08-09 Thread James E. Blair
Hi, In consultation with Jeremy and some other Zuul contributors, I have established the #zuul channel on Freenode. As intensive development of Zuulv3 gets underway, this is a place where folks working on that effort can discuss details without being overwhelmed by the traffic in #openstack-infra

Re: [OpenStack-Infra] Development environments for infra's puppet modules

2016-08-25 Thread James E. Blair
Joshua Hesketh writes: > There's probably further discussions here but I don't have enough knowledge > in this area to comment further. The aim though should be to make it easy > to bootstrap a server with the module you're developing on so you can > easily verify and debug your changes. Don't f

Re: [OpenStack-Infra] Fwd: Zuul-Cloner failing with missing origin/HEAD

2016-09-27 Thread James E. Blair
"Philipp, Damian" writes: > Hello, > > yup, this did help. It would be great if I could just pip install the > working version of zuul to my build hosts. Sounds good. I released Zuul 2.5.1 with this change. Thanks, Jim ___ OpenStack-Infra mailing l

Re: [OpenStack-Infra] ZUUL query: instead of layout.yaml, parameter-function can be use to control job flow ?

2016-10-06 Thread James E. Blair
Bhavik Bavishi writes: > Hi, > > In our project we have started using ZUUL. Our project is stack based > and more of branch oriented, in other words in stack almost all > components (each component is separate git repo) should follow same > workflow but for few are exceptions and this also change

[OpenStack-Infra] Scheduling a Zuul meeting

2016-11-02 Thread James E. Blair
Hi, Recently, a bunch of folks have expressed an interest in helping with Zuul v3. With more than a handful of people engaged in active development, it seems like it might be time to adopt some useful development practices. One of those is scheduling a weekly meeting, the other is task tracking.

Re: [OpenStack-Infra] Scheduling a Zuul meeting

2016-11-07 Thread James E. Blair
Ricardo Carrillo Cruz writes: > 22:00 UTC works for me too, even if it's a bit late for me. Okay, it seems like Mondays at 22:00 works for most folks. We can change it later if it's too much of a stretch. I know a lot of people are traveling this week, so let's plan on having the first meeting

Re: [OpenStack-Infra] Scheduling a Zuul meeting

2016-11-18 Thread James E. Blair
Hi, In our first Zuul meeting, we discussed how we wanted to run the meeting and agreed that we would like to have an agenda page in the wiki, as one does. I have created it here: https://wiki.openstack.org/wiki/Meetings/Zuul -Jim ___ OpenStack-Infra

[OpenStack-Infra] Upcoming Nodepool changes

2016-12-12 Thread James E. Blair
Hi, I'm happy to report that we have made significant progress on adding ZooKeeper support to Nodepool, in preparation for supporting Zuul v3. You may recall that a while back, we made a release of Nodepool (0.3.0) before we started this work so that folks who didn't want to live on the bleeding

Re: [OpenStack-Infra] Upcoming Nodepool changes

2016-12-13 Thread James E. Blair
Darragh Bailey writes: > On 13 December 2016 at 00:25, James E. Blair wrote: > >> Hi, >> >> I'm happy to report that we have made significant progress on adding >> ZooKeeper support to Nodepool, in preparation for supporting Zuul v3. >> > I keep mean

[OpenStack-Infra] Nodepool config file structure

2016-12-20 Thread James E. Blair
Hi, I've been working on reconciling a confusing part of the Nodepool config file: the mapping between labels, provider images, and diskimages. Broadly speaking the three constructs are: diskimages: the file(s) that DIB produces provider images: a disk image uploaded to a provider, combined with

Re: [OpenStack-Infra] Nodepool config file structure

2016-12-20 Thread James E. Blair
Clark Boylan writes: > This appears to currently be called "meta". Let's change it while we're at it. :) >> That also lets us remove the 'providers' section from each label >> definition. That is used to indicate which providers should be used to >> create nodes of each label, but by associat

Re: [OpenStack-Infra] Nodepool config file structure

2016-12-20 Thread James E. Blair
Paul Belanger writes: > What are you visioning for the image-type key? I only bring it up since we've > dropped 'images' here. The thing that says "qcow2"? That's an attribute of the provider and doesn't need to change (we also usually get it via OSCC anyway -- maybe we can remove it). > On th

Re: [OpenStack-Infra] [OpenStack-docs] [infra][docs] Steps to migrate docs.o.o to new AFS based server

2017-01-06 Thread James E. Blair
Andreas Jaeger writes: > we discussed in today's infra meeting to move forward as follows: > > 1) update some missing content since a few projects haven't published >content since we started: >http://users.suse.com/~aj/missing-docs.tar.bz2 is a copy from >docs.o.o that needs to be add

Re: [OpenStack-Infra] Nodepool config file structure

2017-01-09 Thread James E. Blair
cor...@inaugust.com (James E. Blair) writes: >> Yup, I think this makes sense and avoids duplicate image data. One other >> similarish use case that I don't think this addresses that we should >> consider is the one we had in hpcloud and what we do in osic-cloud1 >>

[OpenStack-Infra] Announcing Nodepool 0.4.0

2017-01-11 Thread James E. Blair
Hi, In December we released version 0.3.1 of Nodepool which is the last version that doesn't use ZooKeeper[1]. Today we're releasing 0.4.0 which includes major changes to the image building component of Nodepool. We have been using this in production for OpenStack for about a month and have been

Re: [OpenStack-Infra] Nodepool v3 shim for talking to v2 zuul

2017-01-16 Thread James E. Blair
Clint Byrum writes: > Are the jobs so complicated that you can't just write a dedicated gearman > worker to do the translation to/from zk? I mean, gearman is really really > really simple for a reason. > > Just saying.. might be simpler to write a throw-away 500 line python gear > worker, than mo

[OpenStack-Infra] Upcoming database maintenance affecting wiki

2017-01-18 Thread James E. Blair
Rackspace writes: >Hello, >A maintenance event has been scheduled. Details below: > >Affected device(s) >Wiki-MySQL >Wiki-Dev-MySQL > >Maintenance window > >2017-01-19 0300-0400 CST > >Expected disruption > >Loss of access to database for 10-15 minutes > >T

Re: [OpenStack-Infra] Zuul DependentPipeline conceptual questions

2017-01-23 Thread James E. Blair
"Karpenko, Oleksandr (Nokia - DE/Ulm)" writes: > Hallo together, > > Sorry for bothering you but I am not able to find any zuul specific mailing > list. > First of all if there is some better place to ask questions, please > let me know and I will do it there. You've come to the right place. >

Re: [OpenStack-Infra] Zuul DependentPipeline conceptual questions

2017-01-27 Thread James E. Blair
Clint Byrum writes: > Excerpts from Oleksandr Karpenko's message of 2017-01-27 16:46:24 +0100: >> Hello together, >> >> Thanks for your answers. >> >> Yes, we are interested in this feature and ready to contribute. But we >> will need a support from your side for clear understanding how to >>

[OpenStack-Infra] Zuul info for PTG

2017-02-10 Thread James E. Blair
Hi! We're meeting in person at the Project Team Gathering in Atlanta soon, so I wanted to share the current state of Zuul and how we're thinking about setting up jobs so we're all able to contribute to our goal of actually running Zuul v3 when we are there. Also, we might borrow some of these wor

[OpenStack-Infra] Multi-connection support in Zuul

2017-03-02 Thread James E. Blair
Hi, For a while, Zuul has had support for connections to multiple services for use in triggering jobs, fetching changes, and reporting results. This has been in use (the github patches work with github, we do support reporting to gerrit as two different users, we send email over SMTP), but we have

Re: [OpenStack-Infra] Adding projects to zuulv3-dev.o.o

2017-03-02 Thread James E. Blair
Paul Belanger writes: > Greetings! > > I wanted to start a thread about what people thought about expanding our > coverage of projects a little. I've been working on ansible roles for CI > things > for a while, and figure it might be a good first step to have zuulv3 test a > role[1] to install z

Re: [OpenStack-Infra] [zuul] Feedback requested for tox job definition

2017-03-09 Thread James E. Blair
Clark Boylan writes: > Also reading these job defs and comparing against the zuulv3 spec it > isn't clear to me what the expected behavior for inheriting pre and post > playbooks is. Seems like maybe pre is a queue so parent pre roles run > first and post is a stack so parent post roles run last?

[OpenStack-Infra] [infra][security] Encryption in Zuul v3

2017-03-21 Thread James E. Blair
Hi, In working on the implementation of the encrypted secrets feature of Zuul v3, I have found some things that warrant further discussion. It's important to be deliberate about this and I welcome any feedback. For reference, here is the relevant portion of the Zuul v3 spec: http://specs.openst

Re: [OpenStack-Infra] [openstack-dev] [infra][security] Encryption in Zuul v3

2017-03-21 Thread James E. Blair
David Moreau Simard writes: > I don't have a horse in this race or a strong opinion on the topic, in > fact I'm admittedly not very knowledgeable when it comes to low-level > encryption things. > > However, I did have a question, even if just to generate discussion. > Did we ever consider simply

Re: [OpenStack-Infra] [openstack-dev] [infra][security] Encryption in Zuul v3

2017-03-22 Thread James E. Blair
Ian Cordasco writes: > On Tue, Mar 21, 2017 at 6:10 PM, James E. Blair wrote: >> We did talk about some other options, though unfortunately it doesn't >> look like a lot of that made it into the spec reviews. Among them, it's >> probably worth noting that th

Re: [OpenStack-Infra] [openstack-dev] [infra][security] Encryption in Zuul v3

2017-03-22 Thread James E. Blair
Darragh Bailey writes: > On 22 March 2017 at 15:02, James E. Blair wrote: > >> Ian Cordasco writes: >> >> > >> > I suppose Barbican doesn't meet those requirements either, then, yes? >> >> Right -- we don't want to require ano

[OpenStack-Infra] Test message

2017-03-31 Thread James E. Blair
Hi, This is a test message to verify mailing list functionality after the upgrade. -Jim ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

[OpenStack-Infra] Skipping 2017-04-10 Zuul meeting

2017-04-07 Thread James E. Blair
Hi, We're going to have several absences next Monday, the 10th, so let's skip the Zuul meeting that day. -Jim ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Ask.o.o Email not getting through?

2017-05-01 Thread James E. Blair
Tom Fifield writes: > Hello infra, > > Exploratory question here, no idea what's actually going on. > > We have a prospective new user on Ask OpenStack, who despite trying > multiple auth methods (Launchpad & Google) multiple times did not > receive a confirmation email. > > I checked and there h

[OpenStack-Infra] Canceling 2017-05-15 Zuul meeting

2017-05-11 Thread James E. Blair
Hi, I think enough of us have post-summit plans/chores/burnout that it makes sense to cancel the May 15, 2017 Zuul meeting. See you on May 22. -Jim ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin

[OpenStack-Infra] Zuul v3: proposed new Depends-On syntax

2017-05-24 Thread James E. Blair
Hi, As part of Zuul v3, we're adding support for GitHub (and later possibly other systems). We want these systems to have access to the full power of cross-project-dependencies in the same way as Gerrit. However, the current syntax for the Depends-On footer is currently the Gerrit-specific chang

Re: [OpenStack-Infra] Zuul v3: proposed new Depends-On syntax

2017-05-25 Thread James E. Blair
Jeremy Stanley writes: > On 2017-05-25 00:33:10 + (+), Tristan Cacqueray wrote: >> On May 24, 2017 11:04 pm, James E. Blair wrote: >> [...] >> >How does that sound? >> >> Thinking about further connections support, could this also works >>

Re: [OpenStack-Infra] Zuul v3: proposed new Depends-On syntax

2017-05-25 Thread James E. Blair
Sean Dague writes: > On 05/24/2017 07:04 PM, James E. Blair wrote: > >> The natural way to identify a GitHub pull request is with its URL. >> >> This can be used to identify Gerrit changes as well, and will likely be >> well supported by other systems. Theref

Re: [OpenStack-Infra] Zuul v3: proposed new Depends-On syntax

2017-05-25 Thread James E. Blair
Jeremy Stanley writes: > On 2017-05-25 08:50:14 -0500 (-0500), Kevin L. Mitchell wrote: >> Can I suggest that, for OpenStack purposes, we also deploy some sort of >> bot that comments on reviews using the old syntax, to at least alert >> developers to the pending deprecation? If it had the smart

Re: [OpenStack-Infra] Zuul v3: proposed new Depends-On syntax

2017-05-26 Thread James E. Blair
Joshua Hesketh writes: > On Fri, May 26, 2017 at 1:09 AM, James E. Blair wrote: >> So I think that we should start out by simply silently ignoring any >> patchset elements in the URL. We could consider having Zuul leave a >> note indicating that the patchset component is

[OpenStack-Infra] Canceling Zuul meeting 2017-05-29

2017-05-26 Thread James E. Blair
Hi, Due to the US holiday on Monday, the Zuul meeting on May 29, 2017 is canceled. Thanks, Jim ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] Zuul V3: Behavior of change related requirements on push like events

2017-05-30 Thread James E. Blair
Jeremy Stanley writes: > On 2017-05-30 12:53:15 -0700 (-0700), Jesse Keating wrote: > [...] >> Github labels: This is like approvals/reviews. > [...] > > Perhaps an interesting aside, Gerrit uses the same term (labels) for > how we're doing approvals and review voting. Yeah, or at least, related

Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul

2017-06-09 Thread James E. Blair
Monty Taylor writes: > We should use aiohttp with no extra REST framework. > > Meaning: > > - aiohttp serving REST and websocket streaming in a scale-out tier > - talking RPC to the scheduler over gear or zk > - possible in-process aiohttp endpoints for k8s style health endpoints ... > Since we

Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul

2017-06-09 Thread James E. Blair
Clint Byrum writes: > Your words are more succinct than what I wrote, which is nice. I think > we agree on the general direction for the time being. > > However, I don't think ZK will be a good choice for async event > handling. I'd sooner expect MQTT to replace gear for that. It's worth > noting

Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul

2017-06-09 Thread James E. Blair
Clark Boylan writes: > I'm wary of this simply because it looks a lot like repeating > OpenStack's (now failed) decision to stick web servers in a bunch of > python processes then do cooperative multithreading with them along with > all your application logic. It just gets complicated. I also thi

Re: [OpenStack-Infra] About aarch64 third party CI

2017-06-12 Thread James E. Blair
Ricardo Carrillo Cruz writes: > This is a nodepool.yaml that can help you get going: > > http://paste.openstack.org/show/612191/ Glad it worked! You can drop 'zmq-publishers' from the config entirely. If 'images-dir' and 'diskimages' are required, then I would consider that a bug; we should ha

Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul

2017-06-12 Thread James E. Blair
Clint Byrum writes: > Excerpts from corvus's message of 2017-06-09 13:11:00 -0700: >> Clark Boylan writes: >> >> > I'm wary of this simply because it looks a lot like repeating >> > OpenStack's (now failed) decision to stick web servers in a bunch of >> > python processes then do cooperative mu

Re: [OpenStack-Infra] Nodepool drivers

2017-06-14 Thread James E. Blair
Tristan Cacqueray writes: > Hi, > > With the nodepool-drivers[0] spec approved, I started to hack a quick > implementation[1]. Well I am not very familiar with the nodepool/zookeeper > architecture, thus this implementation may very well be missing important > bits... The primary goal is to be ab

[OpenStack-Infra] [infra] Status of Zuul v3

2017-06-14 Thread James E. Blair
Greetings! This periodic update is primarily intended as a way to keep contributors to the OpenStack community apprised of Zuul v3 project status, including future changes and milestones on our way to use in production. Additionally, the numerous existing and future users of Zuul outside of the Op

[OpenStack-Infra] Canceling June 19, 2017 Zuul meeting

2017-06-14 Thread James E. Blair
Hi, Several of us will be traveling to and attending AnsibleFest in London next week, so I propose we cancel the Monday meeting. Also, expect review bandwidth to be considerably reduced next week. Thanks, Jim ___ OpenStack-Infra mailing list OpenStac

Re: [OpenStack-Infra] On the subject of HTTP interfaces and Zuul

2017-06-28 Thread James E. Blair
Monty Taylor writes: > With that in mind, I believe the path should be for the new server > being written for the console-log streaming to be called "zuul-web" > and that it should serve the console-log websocket at /console-stream > or /console-log or something. > > We can then use our Apache fr

[OpenStack-Infra] Announcing Gertty 1.5.0

2017-07-30 Thread James E. Blair
Announcing Gertty 1.5.0 === Gertty is a console-based interface to the Gerrit Code Review system. Gertty is designed to support a workflow similar to reading network news or mail. It syncs information from Gerrit to local storage to support disconnected operation and easy man

Re: [OpenStack-Infra] Blogging Question

2017-08-03 Thread James E. Blair
Jeremy Stanley writes: > On 2017-08-03 20:34:14 + (+), Morgenstern, Chad wrote: >> I'm wondering, now that the ini is corrected (I hope), will our older >> posts show up or will only the blogs we post after the change is >> picked up show up on your site? > > The cache directory for it al

[OpenStack-Infra] Zuul v3: some layout checks disabled in project-config

2017-08-10 Thread James E. Blair
Hi, With https://review.openstack.org/492697 we are moving gating of Zuul itself and some related job repos from Zuul v2 to Zuul v3. As part of this, we need to disable some of the checks that we perform on the layout file. That change disables the following checks for the openstack-infra/* repo

[OpenStack-Infra] Plan for devstack Zuul v3 jobs

2017-08-25 Thread James E. Blair
Hi, As many of you may recall, our plan to migrate the devstack jobs to Zuul v3 was to gradually convert pieces of the existing devstack-gate script to Ansible roles which would be shared between the Ansible run inside the devstack-gate script and the Ansible run by Zuul v3. Eventually, the idea

Re: [OpenStack-Infra] Plan for devstack Zuul v3 jobs

2017-08-25 Thread James E. Blair
David Moreau Simard writes: > I'm OK with this. > Should we get to a point where the scripts can be used by both Zuul v2 > and Zuul v3 simultaneously so that there is no "migration" or "cutover" so > to speak ? > > It might mean some amount of work but it shouldn't be too bad, I think ? > Ansible

Re: [OpenStack-Infra] [zuul] v3 UI enquiry

2017-09-05 Thread James E. Blair
Darragh Bailey writes: > Hi, > > Currently the main issue from end users, is the user experience around the > UI when looking at job results once the run is complete, and to a lesser > extend looking at your jobs in the zuul status page when running. > > I'm wondering if there is a plan to have a

[OpenStack-Infra] Upcoming backwards incompatible change on Zuul master

2017-09-18 Thread James E. Blair
Hi, We need to merge a backwards incompatible change to Zuul master. The change is: https://review.openstack.org/482856 and it makes Gerrit label entries case sensitive. Unfortunately, some combinations of Gerrit versions and underlying database configurations make this both necessary and diffic

Re: [OpenStack-Infra] [zuul] v3 UI enquiry

2017-09-19 Thread James E. Blair
Darragh Bailey writes: > On 6 September 2017 at 01:26, James E. Blair wrote: > >> Darragh Bailey writes: >> >> > Hi, >> > >> > Currently the main issue from end users, is the user experience around >> the >> > UI when looking at job r

[OpenStack-Infra] Moving docs-draft to logs.o.o

2017-09-20 Thread James E. Blair
Hi, We originally created the docs-draft site (and filesystem partition) because doc builds were *big* and we had to expire them much more quickly than build logs. The expiration times were 21 days for docs-draft and 6 months for build logs. The tables have turned. Docs-draft expiration is stil

[OpenStack-Infra] Zuul feature/zuulv3 branch to be rewound

2017-09-21 Thread James E. Blair
Hi, A change recently landed on the feature/zuulv3 branch of Zuul with a Sem-Ver commit message footer. This is used by PBR to alter the way it constructs version numbers. It's pretty nifty, but in my opinion, it has a fundamental flaw: it can't be undone. I think use of this by a project shoul

Re: [OpenStack-Infra] Zuul feature/zuulv3 branch to be rewound

2017-09-21 Thread James E. Blair
Darragh Bailey writes: >> The abandon/restore steps are required by Gerrit in order to delete the >> branch. We could force-push the branch tip, but this is the procedure >> we have asked and would ask any other project to use in a similar >> situation, in order to reduce the risk of error. > >

Re: [OpenStack-Infra] Upcoming backwards incompatible change on Zuul master

2017-09-27 Thread James E. Blair
cor...@inaugust.com (James E. Blair) writes: > Hi, > > We need to merge a backwards incompatible change to Zuul master. > > The change is: https://review.openstack.org/482856 and it makes Gerrit > label entries case sensitive. Unfortunately, some combinations of > Gerrit ve

Re: [OpenStack-Infra] Nominating new project-config and zuul job cores

2017-10-17 Thread James E. Blair
Clark Boylan writes: > Hello everyone, > > I'd like to nominate a few people to be core on our job related config > repos. Dmsimard, mnaser, and jlk have been doing some great reviews > particularly around the Zuul v3 transition. In recognition of this work > I propose that we give them even more

[OpenStack-Infra] Zuul v3 tag checkouts

2017-10-24 Thread James E. Blair
Hi, It turns out that there are some jobs in OpenStack's Zuul that were relying on a behavior in devstack-gate and zuul-cloner that would check out a tag when given an override branch. Zuul v3 is a bit more literal -- the 'override-branch' option (for both jobs and projects) will only checkout a

[OpenStack-Infra] Some branch issues with Zuul

2017-10-24 Thread James E. Blair
Hi, A number of issues related to how jobs are defined and run on projects with stable branches have come up recently. I believe they are all related, and they, as well as the solutions to them, must be considered together. The Problems I've identified the following five problems:

[OpenStack-Infra] Some guidance on job variants

2017-10-24 Thread James E. Blair
At the infra meeting today, we discussed how to handle job variants. I will try to summarize the discussion and extrapolate some things. The Zuul v3 migration doc in the infra manual is very clear that some project-templates should only be added in project-config, rather than in-repo[1]. However

[OpenStack-Infra] Ignoring pep8 E741

2017-10-24 Thread James E. Blair
Hi, I discussed this briefly with some folks in IRC and received support, but I thought it wise to bring to the mailing list. I think we should add E741 to the list of pep8 errors that we ignore as a matter of course in infra projects. This is a recently added change which forbids the use of var

Re: [OpenStack-Infra] Some branch issues with Zuul

2017-10-25 Thread James E. Blair
Andrea Frittoli writes: > We will need somewhere in the logs a description of the traversal that was > done to build the final job. I believe that would help debugging issues that > may arise from unexpected inheritance behaviour. > > Andrea Frittoli (andreaf) Yes! We have that to some degree t

Re: [OpenStack-Infra] Some branch issues with Zuul

2017-10-25 Thread James E. Blair
Doug Hellmann writes: > In the discussion yesterday, and in the emails today, you've implied > that there is an ordering to job definitions beyond inheritance. Is that > discovery order documented somewhere? If not, is it simple enough to > describe in a few sentences here? Are repositories scann

[OpenStack-Infra] Zuul v3 questions (was: Re: [openstack-dev] Update on Zuul v3 Migration - and what to do about issues)

2017-10-27 Thread James E. Blair
Rikimaru Honjo writes: > Hello, > > (Can I still use this thread?) In the future, you may want to start a new thread on openstack-infra@lists.openstack.org for general Zuul questions. I've changed the CC list and subject of this message to redirect the conversation there. > Excuse me, I'm tryi

Re: [OpenStack-Infra] Some branch issues with Zuul

2017-10-27 Thread James E. Blair
cor...@inaugust.com (James E. Blair) writes: ... > The Changes > === > > I believe the following changes will address all five problems and > achieve both design goals: > > a) Apply inheritance at the same time as variance > > Rather than applying inheritance a

Re: [OpenStack-Infra] Zuul v3 questions

2017-10-30 Thread James E. Blair
Rikimaru Honjo writes: > I confirmed the below PPA, but my version is the latest. > > https://launchpad.net/~ansible/+archive/ubuntu/bubblewrap > > Should I use ubuntu higher than 16.04...? We are running on 16.04. But it looks like this is the PPA we're using: deb http://ppa.launchpad.net/ope

[OpenStack-Infra] Zuul roadmap

2017-11-01 Thread James E. Blair
Hi, At the PTG we brainstormed a road map for Zuul once we completed the infra cutover. I think we're in a position now that we can get back to thinking about this, so I've (slightly) cleaned it up and organized it here. I've grouped into a number of sections. First: Very Near Term ---

Re: [OpenStack-Infra] Zuul roadmap

2017-11-21 Thread James E. Blair
Paul Belanger writes: >> * demonstrate openstack-infra reporting on github > I can start working on this one, is there any objections if we use > gtest-org/ansible first? Thanks! This may involve continuing to debug some github rate limit issues which we've been ignoring since we aren't doing a

[OpenStack-Infra] Merging Zuul v3 into master

2017-11-27 Thread James E. Blair
Hi, A while back we created a feature branch in git for Zuul and Nodepool v3 (feature/zuulv3). Now that OpenStack-Infra is running it, and all of our development focus is on it, we are unlikely to merge any more changes to Zuul v2 or issue another Zuul v2 release. In fact, we are approaching the

Re: [OpenStack-Infra] [zuul] third-party CI for zuul-jobs

2017-11-28 Thread James E. Blair
Jens Harbott writes: > 2017-11-23 5:28 GMT+00:00 Tristan Cacqueray : > ... >> TL;DR; Is it alright if we re-enable this CI and report those tests on >> zuul-jobs patchsets? > > I like the general idea, but please wait for more feedback until doing so. I am in favor of the idea in general,

Re: [OpenStack-Infra] Nodepool drivers

2017-12-01 Thread James E. Blair
Tristan Cacqueray writes: > Hi, > > Now that the zuulv3 release is approaching, please find below a > follow-up on this spec. > > The current code could use one more patch[0] to untangle the common > config from the openstack provider specific bits. The patch often needs > to be manualy rebased.

Re: [OpenStack-Infra] Zuul roadmap

2017-12-01 Thread James E. Blair
Fabien Boucher writes: >> * finish git driver >> > > If ok for you, I want to propose myself to work on that git driver topic. > I'll try to > provide a first patch asap. That's great, thanks! There's some code there already, but it has no tests and hasn't been used in a while -- I don't know i

Re: [OpenStack-Infra] Zuul roadmap

2017-12-06 Thread James E. Blair
Clint Byrum writes: > I know a bunch of this stuff is janky as all get out, because as much of the > jankiness is my own fault as anybody else's. But so much work has gone into > zuulv3 beyond what OpenStack needs, I am still not convinced we need to wait > for any of this. Maybe the zuul-web stu

Re: [OpenStack-Infra] Nodepool drivers

2017-12-07 Thread James E. Blair
Tristan Cacqueray writes: > Hi, > > Top posting here to raise another complication. > James mentioned an API problem regarding the NodeRequestHandler > interface. Indeed the run_handler method should actually be part of the > generic code so that the driver's handler only implements the 'launch'

Re: [OpenStack-Infra] plan for Zuul and Nodepool to support Python3.x ?

2017-12-07 Thread James E. Blair
"Apua A.Aa" writes: > Hi, > > As title, is there a plan or road map for Zuul and Nodepool to > support/migrate to Python3.x currently? Yes, the next versions, Zuul and Nodepool v3.0, will support python 3 only. Note that they will be very different from the current versions and are not backward

Re: [OpenStack-Infra] Nodepool drivers

2017-12-08 Thread James E. Blair
Tristan Cacqueray writes: >>> I also proposed a 'plugin' interface so that driver are fully contained >>> in their namespace, which seems like another legitimate addition to this >>> feature: >>> https://review.openstack.org/524620 >> >> I think that will be a nice thing to have, but I'd like to

Re: [OpenStack-Infra] Xenial Upgrade Sprint Recap

2017-12-15 Thread James E. Blair
Clark Boylan writes: > Hello everyone, > > Just wanted to quickly recap what we got done this week during our > control plane upgrade to Xenial sprint. Thanks! Now that this is over, I wonder if we should start a 'marathon' (as opposed to a sprint) to finish the rest of the servers? Now that w

Re: [OpenStack-Infra] Xenial Upgrade Sprint Recap

2017-12-18 Thread James E. Blair
Ian Wienand writes: > There's a bunch of stuff that wouldn't show up until live, but we > probably could have got a lot of prep work out of the way if the > integration tests were doing something. I didn't realise that although > we run the tests, most of our modules don't actually have any test

[OpenStack-Infra] Test message

2017-12-22 Thread James E. Blair
Hi, This is a test message following some maintenance to the mailing list server. There is no need to reply. Thanks, Jim ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac

[OpenStack-Infra] Hostnames

2018-01-06 Thread James E. Blair
Hi, It seems that every time we boot a new server, it either randomly has a hostname of foo, or foo.openstack.org. And maybe that changes between the first boot and second. The result of this is that our services which require that they know their hostname (which is a lot, especially the complic

[OpenStack-Infra] Zuul mailing lists

2018-01-08 Thread James E. Blair
Hi, We recently completed the work needed to host mailing lists for projects at their own domains. With our expanded focus in Zuul v3 on users beyond those related to the OpenStack project, now seems like a good time to create dedicated Zuul mailing lists. I'd like to create the following two li

Re: [OpenStack-Infra] Hostnames

2018-01-08 Thread James E. Blair
Clark Boylan writes: > On Sun, Jan 7, 2018, at 2:30 PM, David Moreau Simard wrote: >> When I compared ze10 with ze09 today, I noticed that ze09's "hostname" >> command returned "ze09" while ze10 had "ze10.openstack.org". >> >> However, both nodes had the full fqdn when doing "hostname -f". >> >

Re: [OpenStack-Infra] Merging feature/zuulv3 into master in Zuul and Nodepool repos

2018-01-15 Thread James E. Blair
Clark Boylan writes: > Hello, > > I think we are very close to being ready to merge the zuulv3 feature > branch into master in both the Zuul and Nodepool repos. In particular > we merged https://review.openstack.org/#/c/523951/ which should > prevent breakages for anyone using that deployment met

[OpenStack-Infra] New Zuul mailing lists

2018-01-16 Thread James E. Blair
Hi, We've created two new mailing lists for Zuul. If you run an instance of Zuul, or are a user of Zuul, you should subscribe. The new lists are: zuul-annou...@lists.zuul-ci.org --- http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-announce This will be a low-

Re: [OpenStack-Infra] RFC: Zuul executor congestion control

2018-01-16 Thread James E. Blair
writes: > Hi zuulers, > > the zuul-executor resource governor topic seems to be a recurring now > and we might want take the step and make it a bit smarter. To be honest, it keeps coming up because we haven't gotten around to finishing the work already in progress on this. We're not done with t

[OpenStack-Infra] Merging feature/zuulv3 into master

2018-01-16 Thread James E. Blair
Hi, On Thursday, January 18, 2018, we will merge the feature/zuulv3 branches of both Zuul and Nodepool into master. If you continuously deploy Zuul or Nodepool from master, you should make sure you are prepared for this. The current version of the single_node_ci pattern in puppet-openstackci sho

[OpenStack-Infra] [infra][all] New Zuul Depends-On syntax

2018-01-24 Thread James E. Blair
Hi, We recently introduced a new URL-based syntax for Depends-On: footers in commit messages: Depends-On: https://review.openstack.org/535851 The old syntax will continue to work for a while, but please begin using the new syntax on new changes. Why are we changing this? Zuul has grown the a

Re: [OpenStack-Infra] [nodepool] Restricting images to specific nodepool builders

2018-02-19 Thread James E. Blair
Paul Belanger writes: > On Mon, Feb 19, 2018 at 08:28:27AM -0500, David Shrewsbury wrote: >> Hi, >> >> On Sun, Feb 18, 2018 at 10:25 PM, Ian Wienand wrote: >> >> > Hi, >> > >> > How should we go about restricting certain image builds to specific >> > nodepool builder instances? My immediate i

[OpenStack-Infra] Zuul project evolution

2018-03-15 Thread James E. Blair
Hi, To date, Zuul has (perhaps rightly) often been seen as an OpenStack-specific tool. That's only natural since we created it explicitly to solve problems we were having in scaling the testing of OpenStack. Nevertheless, it is useful far beyond OpenStack, and even before v3, it has found adopte

Re: [OpenStack-Infra] Public numbers about the scale of the infrastructure/CI ?

2018-03-26 Thread James E. Blair
David Moreau Simard writes: > Unless there's any objection, I'd have a slide with up to date numbers such > as: I don't have any objection to making them public (I believe nearly all, if not all, of these are public already). But I would like them to be as accurate as possible :). > - # of pr

  1   2   3   4   >