Summary of IRC Meeting in #aurora at Mon Aug 4 18:06:09 2014: Attendees: jfarrell, mchucarroll, wfarner, kts, jaybuff, mkhutornenko
- Preface - Action: jfarrell: grant meeting control privileges to kts - Action: website update for 0.5.0 release - Docker support - Job Update orchestration in the scheduler - jcenter IRC log follows: ## Preface ## [Mon Aug 4 18:07:17 2014] <wfarner>: #action jfarrell: grant meeting control privileges to kts [Mon Aug 4 18:07:31 2014] <jfarrell>: ASFBot: karma add kts 3 [Mon Aug 4 18:07:40 2014] <jfarrell>: done [Mon Aug 4 18:08:04 2014] <wfarner>: Morning, everyone! Quick attendance? [Mon Aug 4 18:08:04 2014] <wfarner>: here [Mon Aug 4 18:08:08 2014] <jfarrell>: here [Mon Aug 4 18:08:09 2014] <jaybuff>: here [Mon Aug 4 18:08:11 2014] <mkhutornenko>: here [Mon Aug 4 18:08:21 2014] <kts>: here [Mon Aug 4 18:08:22 2014] <mchucarroll>: here [Mon Aug 4 18:09:04 2014] <wfarner>: Wonderful. Does anyone have a topic they would like to start with? [Mon Aug 4 18:09:54 2014] <wfarner>: How about https://issues.apache.org/jira/browse/AURORA-633 [Mon Aug 4 18:09:59 2014] <jfarrell>: #action website update for 0.5.0 release [Mon Aug 4 18:09:59 2014] <jaybuff>: :) [Mon Aug 4 18:10:15 2014] <jaybuff>: i'd love to get some feedback from everyone on the interfaces [Mon Aug 4 18:10:29 2014] <jaybuff>: i want to change the DSL and the thrift messages to support that ## Docker support ## [Mon Aug 4 18:12:08 2014] <wfarner>: jaybuff: do you see the changes going far beyond the ExecutorConfig? [Mon Aug 4 18:12:49 2014] <jaybuff>: wfarner: not sure I know what you mean [Mon Aug 4 18:13:06 2014] <jaybuff>: you mean in the protobuf message the scheduler sends to the mesos master? [Mon Aug 4 18:13:58 2014] <wfarner>: nope, the ExecutorConfig struct defined in our api.thrift [Mon Aug 4 18:14:10 2014] <wfarner>: but sounds like changes in the protobuf level are required as well? [Mon Aug 4 18:14:33 2014] <jaybuff>: ah, i see. so I think there are changes required in four areas: [Mon Aug 4 18:14:41 2014] <jaybuff>: the DSL [Mon Aug 4 18:14:47 2014] <jaybuff>: thrift messages from client to scheduler [Mon Aug 4 18:14:55 2014] <jaybuff>: protobuf message from scheduler to mesos master [Mon Aug 4 18:14:59 2014] <jaybuff>: and thermos observer [Mon Aug 4 18:15:18 2014] <jaybuff>: the DSL one I detailed in AURORA-633 [Mon Aug 4 18:15:39 2014] <wfarner>: any light at the end of the ICLA tunnel for you? :-) [Mon Aug 4 18:15:40 2014] <jaybuff>: the thrift message I'd just be adding an optional DockerConfig to TaskConfig [Mon Aug 4 18:16:26 2014] <jaybuff>: wfarner: its always so hard to tell with these things. I've done it before with two other projects. I'm guessing one more month, but I said that a month ago [Mon Aug 4 18:16:49 2014] <wfarner>: :-) [Mon Aug 4 18:17:37 2014] <wfarner>: welp, until then, documenting your findings sounds like a good trade-off [Mon Aug 4 18:17:52 2014] <jaybuff>: dockerconfig in the DSL and the thrift message will basically be whatever ends up in Mesos's DockerInfo which is still be implmeneted, but benh and tnachen tell me that it will go into 0.20 which is targeting release by this friday [Mon Aug 4 18:18:47 2014] <jaybuff>: right now I can do it with ContainerInfo instead of DockerInfo, but ContainerInfo is just the image name plus raw command line args that are opaquely passed to the docker CLI [Mon Aug 4 18:19:21 2014] <jaybuff>: and then the thermos observer will have to be changed to ask docker what is happening [Mon Aug 4 18:21:03 2014] <kts>: seems like there's interest in building a "universal" executor [Mon Aug 4 18:21:16 2014] <kts>: that will run in any docker containers [Mon Aug 4 18:21:45 2014] <jaybuff>: kts: that's what tnachen has been doing, which I'm going to use [Mon Aug 4 18:21:54 2014] <jaybuff>: its not an executor though, its a containerizer [Mon Aug 4 18:22:23 2014] <jaybuff>: and the change to allow multiple containerizers has already gone into mesos master [Mon Aug 4 18:22:41 2014] <jaybuff>: the impact on aurora is actually pretty small [Mon Aug 4 18:22:57 2014] <kts>: sorry, I'm not up-to-speed on the latest mesos terminology [Mon Aug 4 18:23:12 2014] <kts>: I was talking about adding the ability to run the thermos executor inside docker containers [Mon Aug 4 18:23:55 2014] <jaybuff>: kts: that support is in mesos master today, i've hacked aurora to send the correct protobuf messages and it successully started the docker container and ran the executor in the container and executed my process [Mon Aug 4 18:25:03 2014] <kts>: jaybuff: awesome, sounds like you're most of the way there [Mon Aug 4 18:25:34 2014] <jaybuff>: kts: all the hardwork is being done in mesos itself [Mon Aug 4 18:28:03 2014] <wfarner>: any final words on that topic? [Mon Aug 4 18:28:19 2014] <jfarrell>: jaybuff: AURORA-633 if yours now ## Job Update orchestration in the scheduler ## [Mon Aug 4 18:30:36 2014] <wfarner>: work is being tracked at AURORA-610 [Mon Aug 4 18:31:30 2014] <wfarner>: this is proceeding along nicely, we expect to have something usable within the next ~2 weeks [Mon Aug 4 18:32:13 2014] <jaybuff>: will the existing hooks that allow you to write an "external updater" ever be ripped out? [Mon Aug 4 18:32:37 2014] <wfarner>: they will not [Mon Aug 4 18:32:41 2014] <jaybuff>: +1 [Mon Aug 4 18:33:04 2014] <wfarner>: however, in the current shape of the API, the scheduler will provide a much richer experience for the built-in updates [Mon Aug 4 18:33:33 2014] <wfarner>: that could change in the future, but is not currently planned [Mon Aug 4 18:34:44 2014] <wfarner>: any other topics? [Mon Aug 4 18:35:28 2014] <jaybuff>: wfarner: just a thank you for the rest API stuff. We've got a guy using it and he was happy to find out about it after he struggled for two days to get perl thrift bindings building [Mon Aug 4 18:36:00 2014] <kts>: jaybuff: great to hear! [Mon Aug 4 18:36:10 2014] <wfarner>: awesome! please do encourage him to give feedback. did he discover the auto-doc pages? [Mon Aug 4 18:36:15 2014] <kts>: oh one more thing [Mon Aug 4 18:36:36 2014] <jaybuff>: wfarner: not sure, i'll ask him. what's the auto-doc pages? [Mon Aug 4 18:36:49 2014] <kts>: unofficial but I threw up some 0.5.0 artifacts here: http://people.apache.org/~kevints/aurora/dist/0.5.0-incubating/ [Mon Aug 4 18:37:06 2014] <kts>: *https [Mon Aug 4 18:37:34 2014] <wfarner>: jaybuff: visit a scheduler's /apibeta in a web browser [Mon Aug 4 18:37:43 2014] <wfarner>: it has docs of all API methods, and structure of all objects [Mon Aug 4 18:38:04 2014] <wfarner>: primitive, but better than trial and error [Mon Aug 4 18:38:19 2014] <jaybuff>: ah, great, thanks [Mon Aug 4 18:39:16 2014] <kts>: there are some bugs with the pip install method of installation, filed some issues to try to fix that for 0.6.0 [Mon Aug 4 18:39:29 2014] <kts>: one more thing ## jcenter ## [Mon Aug 4 18:40:08 2014] <wfarner>: relevant: sonatype just blogged about HTTPS availability today [Mon Aug 4 18:40:16 2014] <jfarrell>: maven central has all the certs and is making it available today [Mon Aug 4 18:40:21 2014] <wfarner>: http://central.sonatype.org/articles/2014/Aug/03/https-support-launching-now/ [Mon Aug 4 18:43:02 2014] <kts>: ok, want me to update the patch to point to that? [Mon Aug 4 18:43:56 2014] <kts>: tangent as I'm ignorant to the history: why do we prefer Central over JCenter [Mon Aug 4 18:45:20 2014] <jaybuff>: excuse my lack of java knowledge, are you talking about where gradel fetches build deps from? [Mon Aug 4 18:46:24 2014] <kts>: as far as I can tell we use maven central because it's the first example listed in the gradle tutorial for remote dependency fetching [Mon Aug 4 18:46:31 2014] <kts>: jcenter is the second example [Mon Aug 4 18:46:36 2014] <kts>: jaybuff: yeah [Mon Aug 4 18:46:55 2014] <kts>: internally we run a mirror so we override anyway [Mon Aug 4 18:48:19 2014] <jaybuff>: yeah, i've had to modify build.gradle so I could start using my locally built 0.20 libmesos. I guess the mesos guys push the java bindings to maven central whenever they do a release? do you know if they push nighties? [Mon Aug 4 18:48:27 2014] <jaybuff>: err, nightlys [Mon Aug 4 18:48:32 2014] <jfarrell>: its automatic [Mon Aug 4 18:48:51 2014] <jaybuff>: jfarrell: automatically kicked off when? Every commit? [Mon Aug 4 18:49:36 2014] <jfarrell>: repository.apache.org is the primary for the ASF which handles all releases or snapshots [Mon Aug 4 18:50:42 2014] <jfarrell>: that is what gets mirrored to maven central and what Aurora would be releasing any artifacts to [Mon Aug 4 18:50:45 2014] <kts>: jfarrell: looks like mesos just pushes released and -rcs [Mon Aug 4 18:50:56 2014] <jaybuff>: jfarrell: i see what you mean [Mon Aug 4 18:54:31 2014] <kts>: so do we keep using central but switch to https? [Mon Aug 4 18:55:24 2014] <wfarner>: that's the most obvious change to me [Mon Aug 4 18:55:33 2014] <jfarrell>: kts: details for maven staging artifacts is https://github.com/apache/mesos/blob/master/src/java/MESOS-MAVEN-README [Mon Aug 4 18:56:15 2014] <jfarrell>: +1 to switching to https, there are discussions in maven right now to switching to making that the default in an upcoming release [Mon Aug 4 18:57:51 2014] <kts>: right, I assume gradle will change as well since they already accepted a patch to change jcenter to https [Mon Aug 4 18:58:03 2014] <kts>: I think that's it for that topic [Mon Aug 4 18:58:22 2014] <wfarner>: popping the stack once more, i think that's it for today's meeting [Mon Aug 4 18:58:37 2014] <jfarrell>: thanks everyone [Mon Aug 4 18:58:38 2014] <wfarner>: ASFBot: meeting stop Meeting ended at Mon Aug 4 18:58:38 2014
