Re: Review Request 16015: Add sample workflow

tps://reviews.apache.org/r/16015/#comment57261> this logic needs to be moved to the virtualenv bootstrapper - Brian Wickman On Dec. 5, 2013, 12:09 a.m., Kevin Sweeney wrote: > > --- > This is an automatically generated e-mail.

Re: Review Request 16015: Add sample workflow

015/#comment57301> nope -- don't do this. mesos doesn't work with python 3. - Brian Wickman On Dec. 5, 2013, 12:39 a.m., Kevin Sweeney wrote: > > --- > This is an automatically generated e-mail. To reply, visit: >

Re: Review Request 16015: Add sample workflow

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16015/#review29830 --- Ship it! Ship It! - Brian Wickman On Dec. 5, 2013, 7 p.m

Re: Review Request 16144: ZookeeperSchedulerClient url property errored when no proxy_url was set, and the client was not connected.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16144/#review30116 --- Ship it! seems reasonable to me. - Brian Wickman On Dec. 10

Re: Review Request 16130: First steps towards aurora client v2

thon/twitter/aurora/client/cli/jobs.py <https://reviews.apache.org/r/16130/#comment57686> any reason not to make parse_shards an argument action? just to keep things simpler? - Brian Wickman On Dec. 10, 2013, 1:26 a.m., Mark Chu-Carroll wrote: > > ---

Re: Review Request 16228: Pin master to -SNAPSHOT and -dev versions.

for python and .upper() for java. src/main/python/twitter/aurora/client/BUILD <https://reviews.apache.org/r/16228/#comment58019> To stay consistent, let's call these 0.2.0-snapshot. - Brian Wickman On Dec. 13, 2013, 12:05

Re: Review Request 16228: Pin master to -SNAPSHOT and -dev versions.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16228/#review30322 --- Ship it! Ship It! - Brian Wickman On Dec. 13, 2013, 1:27 a.m

Re: Review Request 16266: Bumping up python thrift version to 0.9.1

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16266/#review30387 --- Ship it! Ship It! - Brian Wickman On Dec. 14, 2013, 12:26 a.m

Re: Review Request 16130: First steps towards aurora client v2

m., Mark Chu-Carroll wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/16130/ > --- > > (Updated Dec. 10, 2013, 10:14 p.m.) > > > Revi

Re: Tagging process

+1 On Fri, Dec 13, 2013 at 6:26 PM, Bill Farner wrote: > This sounds good to me. > > -=Bill > > > On Fri, Dec 13, 2013 at 6:20 PM, Kevin Sweeney wrote: > >> Hey all, >> >> I think figuring out the tagging process sooner than later is in >> everyone's best interest so that we can get out of che

Re: Review Request 16130: First steps towards aurora client v2

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16130/#review30453 --- Ship it! Ship It! - Brian Wickman On Dec. 16, 2013, 4:45 p.m

Re: Review Request 16130: First steps towards aurora client v2

> > On Mon, Dec 16, 2013 at 12:43 PM, Brian Wickman wrote: > >>This is an automatically generated e-mail. To reply, visit: >> https://reviews.apache.org/r/16130/ >> >> Ship it! >> >> Ship It! >> >> >> - Brian Wickman >> >>

Re: Review Request 16310: Dropping duplicate LOCK error message.

<https://reviews.apache.org/r/16310/#comment58436> when deleting deps like this always doublecheck the BUILD -- you can remove the src/main/python/twitter/aurora/client:base dep from the updater target - Brian Wickman On Dec. 17, 2013, 2:02 a.m., Maxim Khutornenko

Re: Review Request 16326: Fix thermos glitch in aurora-12.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16326/#review30552 --- Ship it! Ship It! - Brian Wickman On Dec. 17, 2013, 6:15 p.m

Re: Review Request 16326: Fix thermos glitch in aurora-12.

merged On Tue, Dec 17, 2013 at 11:08 AM, Mark Chu-Carroll wrote: > Thanks for the review. I still can't ship it myself - so can you either > push it, or ask someone in SF with commit rights to do it? > > Thanks! > > -Mark > > > On Tue, Dec 17, 2013

Re: Review Request 16311: Fixing updater diff.

rt serialize from thrift.protocol import TJSONProtocol return serialize(config, protocol_factory=TJSONProtocol.TSimpleJSONProtocolFactory()) - Brian Wickman On Dec. 17, 2013, 2:34 a.m., Maxim Khutornenko wrote: > > --

Re: Review Request 16306: "job status" command, with wildcard support.

tter/aurora/client/cli/jobs.py <https://reviews.apache.org/r/16306/#comment58559> meta-question: does it make sense to put a logger on the context instead of using print? (even if the logger _is_ print?) - Brian Wickman On Dec. 16, 2013, 10:19 p.m., Mark Chu-Carroll wrote: >

Re: Review Request 16325: Client v2: add support for options shared between verbs.

a/client/cli/__init__.py <https://reviews.apache.org/r/16325/#comment58562> how is this different from an argparse.Action? - Brian Wickman On Dec. 17, 2013, 4:46 p.m., Mark Chu-Carroll wrote: > > --- > This is an automatica

Re: Review Request 16325: Client v2: add support for options shared between verbs.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16325/#review30715 --- Ship it! Ship It! - Brian Wickman On Dec. 17, 2013, 11:37 p.m

Re: Instruction on setting up dev environment on my Mac machine

There is an executor_id in the state there, so it looks like it is managing to launch the tasks. Since you've listed your job as a Service(), it means that every task will be rescheduled after it exits, and since it looks like your task just immediately returns, it's probably launching dozens of t

Re: Instruction on setting up dev environment on my Mac machine

why it says "Not matching hosts"? > I will try it to see if it works. > > > > > On Fri, Dec 20, 2013 at 9:40 AM, Brian Wickman >wrote: > > > There is an executor_id in the state there, so it looks like it is > managing > > to launch the tasks

Re: Review Request 16311: Fixing updater diff.

;requirements', ((None, False, u'ansicolors'), (None, False, u'elementtree'), (None, False, u'mako'), (None, False, u'markdown'), (None, False, u'psutil==1.1.2'), (None, False, u'pygments

Re: Review Request 16311: Fixing updater diff.

> On Jan. 2, 2014, 9:11 p.m., Brian Wickman wrote: > > src/main/python/twitter/aurora/client/api/updater.py, lines 170-191 > > <https://reviews.apache.org/r/16311/diff/3/?file=413507#file413507line170> > > > > this seems a bit ela

Re: Review Request 16311: Fixing updater diff.

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16311/#review31077 --- Ship it! Ship It! - Brian Wickman On Jan. 2, 2014, 10:09 p.m

Re: Review Request 16444: Client side changes for the client quota check.

do the fluent + and -. src/test/python/apache/aurora/client/api/test_quota_check.py <https://reviews.apache.org/r/16444/#comment59491> 2nl - Brian Wickman On Jan. 3, 2014, 1:23 a.m., Maxim Khutornenko wrote: > > ---

Re: Review Request 16444: Client side changes for the client quota check.

er.py <https://reviews.apache.org/r/16444/#comment59537> or CapacityRequest() - Brian Wickman On Jan. 3, 2014, 9:21 p.m., Maxim Khutornenko wrote: > > --- > This is an automatically generated e-mail. To reply, visit: >

Re: Review Request 16306: "job status" command, with wildcard support.

lazy in python3, so len() won't work on it. (render_tasks_pretty calls len() on the iterable) - Brian Wickman On Dec. 18, 2013, 9:27 p.m., Mark Chu-Carroll wrote: > > --- > This is an automatically generated e-ma

Re: Review Request 16615: Client command hooks doc

, bar) docs/configurationtutorial.md <https://reviews.apache.org/r/16615/#comment59573> This is actually not true -- we do not implement the Mustache specification, we just use mustaches {{}} for indicating which variables should be bound. - Brian Wickman On Jan. 3, 2014, 7:

Re: Inheritance in .aurora

Inheritance in pystachio is not pythonic, so what you're trying to do won't work. I consider this a broken window. We've been discussing how to improve this for a while in a backwards compatible fashion and will be tracking ideas in an epic centered around AURORA-188

Re: Client Configuration Plugin Extension Proposal

Sounds good to me. If I understand correctly, this models the API hooks fairly closely (I think they have before/after/on error) and this is before parse / before command / after command, which seems right. On Fri, Apr 18, 2014 at 2:28 PM, Mark Chu-Carroll wrote: > No, I don't think that there

Re: Python versions

Please use twitter.common.lang Compatibility.exec_function: https://github.com/twitter/commons/blob/master/src/python/twitter/common/lang/__init__.py#L122 Happy on 2.x and 3.x (even 2.6.x) On Tue, Apr 29, 2014 at 6:31 AM, Mark Chu-Carroll wrote: > Folks: > > For one of the client changes I'm w

Re: Adding shorthands, defaults, and initialization files to the aurora client

Sending again -- earlier try bounced. On Mon, May 19, 2014 at 10:25 AM, Brian Wickman wrote: > Perhaps a stop-gap would be to require explicit aliases, e.g. > > [aliases] > test-west = wickman/west/test/service > test-east = wickman/east/test/service > prod-west =

Re: timeline on aurora-515: kerberos support

The first step of client support is there -- we created the TRequestsTransport on the client, which uses the python requests library to do thrift-over-HTTP. There is a python package called 'requests-kerberos' that has a class called HTTPKerberosAuth that can be passed into TRequestsTransport and

Re: timeline on aurora-515: kerberos support

m not totally clear if this even > possible based on my brief understanding of kerberos. > > thanks > > > > On Thu, Jun 26, 2014 at 2:25 PM, Brian Wickman wrote: > > > The first step of client support is there -- we created the > > TRequestsTransport on the client,

Re: Task Constraints

host/rack should not be treated specially. If you go to the "/slaves" endpoint on the scheduler UI, what does it report as attributes being exported by your slaves? You might want to validate there that the "staging" attribute got picked up properly. If it's not getting picked up (e.g. the attri

Re: [VOTE] Release Apache Aurora 0.5.0 (incubating) RC1

+1 On Fri, Jul 18, 2014 at 5:38 AM, Mark Chu-Carroll wrote: > Downloaded, created the vagrant cluster, ran end-to-end tests. Looks good! > > +1! > > > > On Thu, Jul 17, 2014 at 4:53 PM, Bill Farner wrote: > > > +1 > > > > -=Bill > > > > > > On Thu, Jul 17, 2014 at 4:09 PM, Bill Farner wrote:

Re: Propsal: Centralizing update orchestration in Aurora

Will the API for client-side updates still exist? Will the client continue to have its own implementation of 'update' (or perhaps an 'update --local' flag?) The reason I ask is whether customers should continue to have the flexbility to implement their own update algorithms (e.g. 1% -> 10% -> 25%

Re: Propsal: Centralizing update orchestration in Aurora

side to satisfy visibility requirements Bill outlined above, which may > not > > worth the effort. That would also create ground for inconsistent update > > expectations and experience. > > > > > > On Fri, Jul 25, 2014 at 1:34 PM, Brian Wickman > wrote: > >

Re: Build failed in Jenkins: Aurora #497

I'm unable to reproduce this either in the vagrant image or on my Mac laptop. Do we have any access to a similar build environment or have the ability to launch ad hoc jenkins jobs? Without this ability I'm not sure I can properly develop a fix. On Wed, Aug 6, 2014 at 9:34 AM, Apache Jenkins Se

announcer implementation now available

An example implementation of an announcer is now available. It can be enabled with the --announcer-enable and --announce-ensemble flags passed to the Aurora executor. There is a ticket open to make passing co

Re: Service Discovery with Announcer a la AURORA-587

Your understanding is correct. The znodes will be standard sequential/ephemerals in the paths you mention, starting with "member_" and ending in the sequence id. You can either roll your own code to read the znodes or derive from one of the implementations referenced in the user guide. ~brian

Re: "Could not deserialize AssignedTask" error

That's a new one! What do your 'constraints' look like on your Job? It's possible Aurora is not doing doing proper input validation somewhere. Could be related to this: https://issues.apache.org/jira/browse/THRIFT-162 On Tue, Aug 12, 2014 at 6:04 PM, Josh Adams wrote: > We're trying to upgra

Re: Authentication for thermos webui

There is an existing kerberos plugin that works with twitter.common.http: https://github.com/twitter/commons/blob/master/src/python/twitter/common/http/plugins/kerberos.py I only bring this up as a reference to prior art. There's nothing that requires this or other plugins to be within the common

Re: Problems building Thermos

Not a silly question at all. The error messages could definitely be more helpful. Out of curiosity, do you have the python-dev package and compiler toolchain installed into this image? psutil has C extensions and it's possible that pants simply can't get python to build them. For mesos, one pos

Re: Aurora roles vs Slave roles and how that relates to slave configuration --resource

They are unrelated. That is not to say they shouldn't be supported. Any particular use-cases that come to mind? On Tue, Sep 23, 2014 at 3:44 PM, Joe Stein wrote: > Hi, I was looking at resource role allocation (slave configuration > --resource) and was wondering how this relates to Aurora and

Re: Error handling in the aurora client

+1 as well. On Fri, Oct 3, 2014 at 9:13 AM, Maxim Khutornenko wrote: > +1 on that. Hides away the ugliness yet more accessible than searching > through client logs. > > On Fri, Oct 3, 2014 at 5:51 AM, Mark Chu-Carroll > wrote: > > I like the proposal from John. Any objections to implementing th

Re: Build failed in Jenkins: Aurora #636

Yep, and the pex http code on master has been refactored substantially recently to make this quite a lot easier to do. Mind filing an issue on pantsbuild/pex to get this implemented? On Thu, Oct 9, 2014 at 11:48 AM, Zameer Manji wrote: > I think to fix this problem we might need to get pex to d

Re: build errors

You should probably update thrift_wrapper_codegen.py to #!/usr/bin/env python2.7 The 'flags=' keyword argument was added in 2.7, and I believe 2.6 is the default for CentOS 6. On Wed, Oct 15, 2014 at 11:09 AM, Bill Farner wrote: > What's your python version? Presumably this would fail on any s

Re: build errors

; / > > > > On Wed, Oct 15, 2014 at 2:19 PM, Kevin Sweeney > wrote: > > > > > +1 to that - python2.7: command not found is a much better error > message > > > here. > > > > > > On Wed, Oct 15, 20

Re: [DISCUSS] Deprecate use of mock.patch?

I sent an e-mail about this a couple days but it got swallowed since I hadn't updated my Apache mail settings. My opinion is the same as David's. Explicitly calling out every stubbable method in constructors / functions is just Guiceifying our code. mock wouldn't be part of the Python standard l

Re: [DISCUSS] Features for 0.7.0

In the deprecated behavior category, I'd like to get feedback on https://issues.apache.org/jira/browse/AURORA-915 as possible candidate for inclusion to 0.7.0 (at the very least, warn-on-import and providing a binding helper for os.environ.) On Mon, Dec 1, 2014 at 3:04 PM, Bill Farner wrote: > I

Re: override python interpreter when building pex using pants

You can use ./pants build -i to build with a specific interpreter, but it will still revert to a #!/usr/bin/env pythonX.X hashbang line. There is not yet a pants environment variable to switch interpreters at runtime but it would be a straightforward feature to add. I've filed https://github.co

Re: aurora watch_secs change

Refactoring the executor to allow (B) is also probably easiest, but per Farner's caveat -- it should only send framework messages on status transitions. There is something called the StatusManager that takes in input from any status checker (such as the health checker, announcer, resource manager,

Re: aurora watch_secs change

get lost in the cracks. > > > > As a reminder, the previous emails in this thread were: > > Bill Farner > > < > http://mail-archives.apache.org/mod_mbox/incubator-aurora-dev/201412.mbox/ajax/%3CCAGRA8uMpWyhcV-hxLU%3Dw7twDD7jbffu39TmbX5MPiXQE8jextA%40mail.gmail.com%3E &g

Re: Should we continue to maintain fine-grained python BUILD targets?

4) Enforcing sensible software hygiene. The pants project itself for the longest time had a single root rglobs python_library target. I spent a few days splitting up that project into fine-grained targets and it uncovered a number of cyclical dependencies and abstraction leaks that simply would'v

upgrading to pants 0.0.27

For those of you who hack on the Python components of Aurora, we're upgrading to pants 0.0.27 here: https://reviews.apache.org/r/29866/ The major change is that typing "goal" is no longer a thing. To run tests: ./pants test and to build PEX binaries: ./pants binary Let me know if you run into

Thermos external component deprecation plan

Thermos is a standalone task execution system that is not coupled to Aurora or Mesos. This is why by default, Thermos writes out of the sandbox (/var/run/thermos), has a separate observability system (Thermos observer), and CLI (thermos.) Aurora built a Thermos executor as its default executor, b

Re: Seeking volunteers to create scheduler REST API RFC

+1 On Mon, Feb 9, 2015 at 1:55 PM, Joshua Cohen wrote: > I'd love to help out with this as well. > > On Mon, Feb 9, 2015 at 1:25 PM, Bill Farner wrote: > > > Hi folks, > > > > Before embarking on work [1] to build a REST* API for the scheduler, it's > > important that we come up with a sound de

thermos observer resource monitoring

Does anybody rely upon the resource monitoring feature in the Thermos observer UI (the UI on port :1338)? Specifically the cpu/ram/disk columns. As far as I know, nobody is using Thermos in the absence of Mesos, and because the Mesos slave provides the same information more accurately, I'm inclin

Re: thermos observer resource monitoring

Manji wrote: > > > Would removing this also remove the psutil dependency from the observer? > > > > On Mon, Feb 9, 2015 at 3:59 PM, Brian Wickman > wrote: > > > > > Does anybody rely upon the resource monitoring feature in the Thermos > > > observer UI (the U

Re: Speeding up Aurora client job creation

To echo what bill said, pro-tip: Pystachio objects have .json_dump and .json_dumps methods that serialize to file objects and strings respectively. Analogously, they have classmethods .json_load and .json_loads that can deserialize this data (these behave exactly like the python json module method

Re: [proposal] Deprecate the Thermos CLI

I agree it is a valuable component. However, I think that until it has test coverage we should consider it an unsupported tool. Filed AURORA-1131 . This is already on my radar as part of AURORA-1027

Re: Ubuntu 12.04 - ImportError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.16' not found

Yes, it's very likely that the bad python dependency was cached in .pants.d. Unfortunately there's no way to differentiate between linux ABI in python package naming so if something is already in .pants.d/python/eggs that matches the requirement (mesos.native-0.21.1-linux-x86_64) then it will like

Re: [VOTE] Graduate Apache Aurora as a TLP

ly below be and > hereby are appointed to serve as the initial members of the > Apache Aurora Project: > >* Mark Chu-Carroll >* William Farner >* Jake Farrell >* Benjamin Hindman >* Suman Karumuri >* Maxim Khutornenko >* Dave Lester

Re: Build failed in Jenkins: Aurora #889

Looking. On Fri, Feb 20, 2015 at 4:17 PM, Zameer Manji wrote: > The health checker test seems flaky or incorrect. Wickman, can you please > take a look? > > On Fri, Feb 20, 2015 at 4:15 PM, Apache Jenkins Server < > jenk...@builds.apache.org> wrote: > >> See

Re: Build failed in Jenkins: Aurora #889

_health_checks.pop(0) IndexError: pop from empty list WARNING:root:Health check failure: reason WARNING:root:Health check failure: reason On Fri, Feb 20, 2015 at 4:42 PM, Brian Wickman wrote: > Looking. > > On Fri, Feb 20,

Re: [proposal] Deprecate the Thermos CLI

t; > > > On Wed, Feb 18, 2015 at 12:04 PM, Bill Farner > > wrote: > > >> Can either of you elaborate on the type of debugging you currently > > >> accomplish with this tool? > > >> > > >> On Wednesday, February 18, 2015, Brian Wickman &

Re: time constraint on aurora jobs?

Another option is to just add a process to your task that looks something like Process(name = 'timeout', cmdline = 'sleep 3600; false'). If the task runs for 3600 seconds, then that process will exit with a failure causing the whole task to fail. The only issue I can think of is that it will also

[jira] [Commented] (AURORA-44) aurora python code should be exported as publishable sdists

[ https://issues.apache.org/jira/browse/AURORA-44?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13874081#comment-13874081 ] brian wickman commented on AURORA-44: - They already are publishable as sdists. Sh

[jira] [Created] (AURORA-56) executor logs way too much

brian wickman created AURORA-56: --- Summary: executor logs way too much Key: AURORA-56 URL: https://issues.apache.org/jira/browse/AURORA-56 Project: Aurora Issue Type: Bug Components

[jira] [Commented] (AURORA-117) Scheduler performance issues with very large jobs

[ https://issues.apache.org/jira/browse/AURORA-117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13882006#comment-13882006 ] brian wickman commented on AURORA-117: -- One option is to use a simple limit inde

[jira] [Commented] (AURORA-125) "aurora ssh" and "aurora run" fail with a pystachio-related error.

[ https://issues.apache.org/jira/browse/AURORA-125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13883414#comment-13883414 ] brian wickman commented on AURORA-125: -- Feature not a bug. Perhaps not

[jira] [Reopened] (AURORA-56) executor logs way too much

[ https://issues.apache.org/jira/browse/AURORA-56?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] brian wickman reopened AURORA-56: - > executor logs way too much > -- > > Ke

[jira] [Resolved] (AURORA-56) executor logs way too much

[ https://issues.apache.org/jira/browse/AURORA-56?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] brian wickman resolved AURORA-56. - Resolution: Fixed {noformat} commit 3e59dfb19c738bc50f73ccb0cf768ab36dddee29 Author: Brian

[jira] [Commented] (AURORA-37) investigate src.test.python.apache.aurora.executor.thermos_task_runner test timeout on builds.a.o

[ https://issues.apache.org/jira/browse/AURORA-37?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13884917#comment-13884917 ] brian wickman commented on AURORA-37: - python_test targets take a timeout kwarg so