+2
On Fri, 2016-08-12 at 00:56 +0800, lu jander wrote:
> +2 from me thx Telles
>
> 2016-08-12 0:20 GMT+08:00 Sergey Reshetnyak
> :
> +2 from me
>
> 2016-08-11 19:15 GMT+03:00 Sergey Lukjanov
> :
> +2
>
> On Thu, Au
g this,
> has there been any progress to deprecate sahara jobs -> internal db mechanism?
> and/or the config option to disable internal db storage?
>
> Regards,
>
> Jerico
>
>
>
> > On 18 Mar 2016, at 12:55 AM, Trevor McKay wrote:
> >
> > Hi
+2 for all
On Fri, 2016-05-13 at 18:33 +0300, Vitaly Gridnev wrote:
> Hello Sahara core folks!
>
>
> I'd like to bring the following folks to Sahara Core:
>
>
> 1. Lu Huichun
> 2. Nikita Konovalov
> 3. Chad Roberts
>
>
> Let's vote with +2/-2 for additions above.
>
>
> [0] http://stackalyt
Thanks Luigi,
sounds good to me. I'll be happy to help with reviews/approvals as
needed.
Trevor
On Mon, 2016-03-21 at 12:05 +0100, Luigi Toscano wrote:
> On Monday 21 of March 2016 10:50:30 Evgeny Sikachev wrote:
> > Hi, Luigi!
> >
> > Thanks for this short spec :)
> > Changes looking good fo
Hi Jerico,
Internal db storage for job binaries was added at
the start of EDP as an alternative for sites that do
not have swift running. Since then, we've also added
integration with manila so that job binaries can be
stored in manila shares.
You are correct, storing lots of binaries in the
Hi Jerico,
Internal db storage for job binaries was added at
the start of EDP as an alternative for sites that do
not have swift running. Since then, we've also added
integration with manila so that job binaries can be
stored in manila shares.
You are correct, storing lots of binaries in the
approved, I the
FFE should be granted.
Best,
Trev
On Mon, 2016-03-07 at 09:07 -0500, Trevor McKay wrote:
> For some reason the link below is wrong for me, it goes to a different
> review. Here is a good one (I hope!):
>
> https://review.openstack.org/#/c/285839/
>
> Trev
>
&g
For some reason the link below is wrong for me, it goes to a different
review. Here is a good one (I hope!):
https://review.openstack.org/#/c/285839/
Trev
On Mon, 2016-03-07 at 14:28 +0800, lu jander wrote:
> Hi folks,
>
> I would like to request a FFE for the feature “Resume EDP job”:
>
>
And of course, don't forget the sahara specs repo:
https://github.com/openstack/sahara-specs
This is always a good way to propose a new idea.
Create a high-level blueprint, then submit an associated spec
through gerrit in the openstack/sahara-specs project.
Trev
On Mon, 2016-02-01 at 18:59 +03
Vitaly,
yes, welcome! I would have voted +1 but I was on PTO :)
Trev
On Thu, 2015-10-15 at 10:51 -0400, michael mccune wrote:
> congrats Vitaly!
>
> On 10/15/2015 10:38 AM, Sergey Lukjanov wrote:
> > I think we have a quorum.
> >
> > Vitaly, congrats!
> >
> > On Tue, Oct 13, 2015 at 6:39 PM, M
+1 from me as well. It would be a shame to see this go to the next
cycle.
On Fri, 2015-09-04 at 10:40 -0400, Ethan Gafford wrote:
> Hello all,
>
> I request a FFE for the change at: https://review.openstack.org/#/c/209683/
>
> This change enables a significant improvement to UX in Sahara's elas
Flavio,
thanks, bad joke on my part. I work with Telles on Sahara, just poking
him in jest. Apologies, didn't mean to create an issue on the list.
Trev
On Fri, 2015-08-14 at 17:30 +0200, Flavio Percoco wrote:
> On 14/08/15 09:29 -0400, Trevor McKay wrote:
> >Hi Telle
Hi Telles,
you technically don't get a vote, but thanks anyway :)
Trev
On Fri, 2015-08-14 at 12:14 +, Telles Nobrega wrote:
> +1
>
> On Fri, Aug 14, 2015 at 7:11 AM Alexander Ignatov
> wrote:
>
> +1
>
> Regards,
> Alexander Ignatov
>
>
+1, welcome addition
On Thu, 2015-08-13 at 17:56 +0300, Sergey Lukjanov wrote:
> Hi folks,
>
>
> I'd like to propose Ethan Gafford as a member of the Sahara core
> reviewer team.
>
>
> Ethan contributing to Sahara for a long time and doing a great job on
> reviewing and improving Sahara. Here
Hi Lu,
yes, you're right. Return is a dictionary and for the other EDP
engines only "status" is returned (and we primarily care about
"status"). For Oozie, there is more information.
I'm fine with changing the name to "get_job_info()" throughout the
job_manager and EDP.
It actually raise
Kilo is closed to everything but critical bugs at this point.
Going forward, this change probably counts as a new feature and is
forbidden as a backport to Kilo too. So Liberty is the earliest
opportunity.
https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes
Trev
On Wed, 2015-04-29 at
On Wed, 2015-04-22 at 12:36 +, Chen, Ken wrote:
> o more complex workflows (job dependencies, DAGs, etc. Do we rely on
> Oozie, or something else?
> >> Huichun is now figuring this. I am not whether you guys already have
> some detail ideas about this? If needed we can contribute some
Hi Ken,
responses inline
On Wed, 2015-04-22 at 12:36 +, Chen, Ken wrote:
> Hi Trevor,
> I saw below items in Proposed Sprint Topics of sahara liberty.
> https://etherpad.openstack.org/p/sahara-liberty-proposed-sessions. I
> guess these are the EDP ideas we want to discuss on Vancouver desi
Thanks, Sergey. I agree -- my first thought was "default templates plus
new ACLs!"
Yanchao, note that with the current default template mechanism, the
templates can be added per tenant -- you can run the script multiple
times and change the tenant id. We thought this provided enough
functionality
Weiting, Andrew,
Agreed, great ideas! As Andrew noted, we have discussed some of these
things before and it would be great to discuss them in Vancouver.
I think that a Sahara-side workflow manager is the right approach. Oozie
has a lot of capability for job coordination, but it won't work for al
Hi Li,
I am using a fresh devstack with Horizon deployed as part of devstack.
I am running Sahara separately from the command line from the git
sources (master branch).
I use a little script to register the Sahara endpoint so that Horizon
sees it.
The only change I had to make was to register
Hi Sahara folks,
please checkout
https://review.openstack.org/#/c/159872/
and respond there in comments, or here on the email thread. We have some
things to figure out for this new CLI, and I want to make sure that we
make
sane choices and set a good precedent.
Once we have a consensus we can
Sean,
thanks! I feel better already. I'll check out the review.
Trevor
On Tue, 2015-02-24 at 14:39 -0500, Sean Dague wrote:
> On 02/24/2015 02:34 PM, Trevor McKay wrote:
> > Hi folks,
> >
> > I've just joined the stable maintenance team for Sahara.
> >
Hi folks,
I've just joined the stable maintenance team for Sahara.
We have this review here, from OpenStack proposal bot:
https://review.openstack.org/158775/
Since it came from the proposal bot, there's no justification in the
commit message and no cherry pick.
I didn't see this case cove
(wf:lastErrorNode())}]
On Thu, 2015-02-12 at 17:15 -0500, Trevor McKay wrote:
> Hi folks,
>
> Here is another way to do this. Lu had mentioned Oozie shell actions
> previously.
> Sahara doesn't support them, but I played with it from the Oozie
> command line
>
ie -config job.xml -run
Best,
Trev
On Thu, 2015-02-12 at 08:39 -0500, Trevor McKay wrote:
> Hi Lu, folks,
>
> I've been investigating how to run Java actions in Sahara EDP that
> depend on
> HBase libraries (see snippet from the original question by Lu below).
>
>
Hi Lu, folks,
I've been investigating how to run Java actions in Sahara EDP that
depend on
HBase libraries (see snippet from the original question by Lu below).
In a nutshell, we need to use Oozie sharelibs for this. I am working on
a spec now, thinking
about the best way to support this in Sah
Answers to other questions:
2) (first part) Yes, I think Oozie shell actions are a great idea. I can
help work on a spec for this.
In general, Sahara should be able to support any kind of Oozie action.
Each will require a new job type, changes to the Oozie engine, and a UI
form to handle submissi
Hi,
Thanks for your patience. I have been consumed with spark-swift, but
I can start to address these questions now :)
On (1) (a) below, I will try to reproduce and look at how we can better
support classpath in EDP. I'll let you know what I find.
We may need to add some configuration options
4 but without other changes and
address the above issues in L, or do we push on and try to resolve it
all for Kilo?
Best regards,
Trevor
On Wed, 2015-01-28 at 11:57 -0500, Trevor McKay wrote:
> Daniele,
>
> Excellent! I'll have to keep a closer eye on bigfoot activity :) I'll
gt; The image generated with this code runs in Sahara without any further
> changes. Feel free to take the code, clean it up and submit for review.
>
> Dan
>
> On Wed, Jan 28, 2015 at 10:43:30AM -0500, Trevor McKay wrote:
> > Intel folks,
> >
> > Belated welcome
Intel folks,
Belated welcome to Sahara! Thank you for your recent commits.
Moving this thread to openstack-dev so others may contribute, cc'ing
Daniele and Pietro who pioneered the Spark plugin.
I'll respond with another email about Oozie work, but I want to
address the Spark/Swift issue in CDH
+1 for me, too. But I agree with Mike -- we need input from Saratov and
similar time zones.
I think though that the folks in Saratov generally try to shift hours
anyway, to better align with the rest of us. So I think something that
works for the US will likely work for them.
Trev
On Tue, 2014
Hello all,
at our last Sahara IRC meeting we started discussing whether or not to add
a global requirement for cm_api.py https://review.openstack.org/#/c/130153/
One issue (but not the only issue) is that cm_api is not packaged for Fedora,
Centos, or Ubuntu currently. The global requirements
+ 2
On 11/11/2014 12:37 PM, Sergey Lukjanov wrote:
Hi folks,
I'd like to propose Michael McCune to sahara-core. He has a good
knowledge of codebase and implemented important features such as Swift
auth using trusts. Mike has been consistently giving us very well
thought out and constructive
+2
On 11/11/2014 12:35 PM, Sergey Lukjanov wrote:
Hi folks,
I'd like to propose Sergey to sahara-core. He's made a lot of work on
different parts of Sahara and he has a very good knowledge of
codebase, especially in plugins area. Sergey has been consistently
giving us very well thought out
Zhidong,
Thanks for your question. I personally don't have an answer, but I
think we definitely should bring up the possibility of dockerization for
Sahara at the design summit next week. It may be something we want to
formalize for Kilo. Will you be at the summit?
Just to be clear, are you r
Not sure how this is done, but I'm a core member for Sahara, and I
hereby sponsor it.
On Fri, 2014-09-05 at 09:57 -0400, Michael McCune wrote:
> hey folks,
>
> I am requesting an exception for the Swift trust authentication blueprint[1].
> This blueprint addresses a security bug in Sahara and re
by the way, what typo?
Trev
On Wed, 2014-09-03 at 14:58 -0700, Andrew Lazarev wrote:
> Hi team,
>
>
> Today I've realized that we have some tests called 'integration'
> in python-saharaclient. Also I've found out that Jenkins doesn't use
> them and they can't be run starting from April because
Yes, I wrote them. I use them all the time -- no typo that I know of.
They are great for spinning up a cluster and running EDP jobs.
They may need some polish, but the point is to test the whole chain of
operations from the CLI. This is contrary to what most OpenStack
projects traditionally do -
ll/1010
>
> I sent information about this patch to this mailing list about two
> months ago.
>
> All the best,
> Gil.
>
>
>
>
>
> From:Trevor McKay
> To:OpenStack Development Mailing List
>
> Date:28/08/2014 06:22 P
Hi folks,
I've updated this etherpad with notes from an investigation of
Spark/Swift and the hadoop-openstack plugin carried in the sahara-extra
repo.
Following the notes there, I was able to access swift:// paths from
Spark jobs on a Spark standalone cluster launched from Sahara and then
f
Thoughts, rapidfire :)
In short, I think we should plan on backward compat unless some stubborn
technical problem gets in our way
I think backward compatibility is a good idea. We can make the
user/pass inputs for data objects optional (they are required
currently), maybe even gray them out
Hi,
I believe if you look in the sahara log output you may see more
messages about this. There should be some indicator of what the
validation error was.
Trevor
On Wed, 2014-07-09 at 23:44 +0700, Dat Tran wrote:
> Thank matt,
>
>
> It's worked. Can i ask you a question, please?
>
>
> When
e master node of the cluster.
> One last observation: currently, spark in standalone mode (that we use
> in the plugin) does not support other schedulers than FIFO, when
> multiple spark applications/jobs are submitted to the cluster. Hence,
> the spark job-server could be a good plac
last observation: currently, spark in standalone mode (that we use
> in the plugin) does not support other schedulers than FIFO, when
> multiple spark applications/jobs are submitted to the cluster. Hence,
> the spark job-server could be a good place to integrate a better job
> scheduler.
Hi folks,
We've just started an investigation of spark EDP for Sahara. Please
visit the etherpad and share your insights. Spark experts especially
welcome!
https://etherpad.openstack.org/p/sahara_spark_edp
There are some design/roadmap decisions we need to make -- how are we
going to do th
Hi folks,
Here is a summary of priorities from Summit and some action items for
high the high priority issues. The link to the pad is here:
https://etherpad.openstack.org/p/juno-summit-sahara-edp
We really did not have any leftover questions from summit, but we need
investigation and develo
below, sahara-extra
On Wed, 2014-05-28 at 20:02 +0400, Sergey Lukjanov wrote:
> Hey folks,
>
> it's a small wrap-up for the topic "Sahara subprojects releasing and
> versioning" that was discussed partially on summit and requires some
> more discussions. You can find details in [0].
>
> > common
Catching up...
On Thu, 2014-05-29 at 15:59 +0400, Alexander Ignatov wrote:
> On 28 May 2014, at 17:14, Sergey Lukjanov wrote:
> > 1. How should we handle addition of new functionality to the API,
> > should we bump minor version and just add new endpoints?
>
> Agree with most of folks. No new ve
wrote:
> Trevor, congrats!
>
> welcome to the sahara-core.
>
> On Thu, May 15, 2014 at 11:41 AM, Matthew Farrellee wrote:
> > On 05/12/2014 05:31 PM, Sergey Lukjanov wrote:
> >>
> >> Hey folks,
> >>
> >> I'd like to nominate T
I am having a very similar issue with horizon, just today. I cloned the
repo and started from scratch on master.
tools/install_venv.py is trying to install cffi as a depdendency,
ultimately fails with
ImportError: cannot import name Feature
This is Fedora 19. I know some folks on Fedora 20 who
> Regards,
> Alexander Ignatov
>
>
>
> On 05 Feb 2014, at 01:01, Trevor McKay
> wrote:
>
> > This brin
it'll not require changes in all Savanna subprojects;
> * eventually we'd like to use not only Oozie for EDP (for example, if
> we'll support Twitter Storm) and this new tools could require
> additional 'subtypes'.
>
>
> Thanks for catching this.
&g
to ensure that DB
> >> schema created by migration & models are same (actually they are not same).
> >>
> >> So before dropping support of migrations for sqlite & switching to model
> >> based created schema we should add tests that will check that mode
type' field will solve both problems. Having it optional
> will not break backward compatibility. Adding database migration
> script is also pretty straightforward.
>
>
> Summarizing, my vote is on "subtype" field.
>
>
> Thanks,
> Andrew.
>
>
I was trying my best to avoid adding extra job types to support
mapreduce variants like streaming or mapreduce with pipes, but it seems
that adding the types is the simplest solution.
On the API side, Savanna can live without a specific job type by
examining the data in the job record. Presence/
"Database migrations are not supported for sqlite"
Because, as a developer, when I see a sql error trace as the result of
an operation I assume it's broken :)
Best,
Trevor
On Thu, 2014-01-30 at 15:04 -0500, Jay Pipes wrote:
> On Thu, 2014-01-30 at 14:51 -0500, Trevor McKay w
I was playing with alembic migration and discovered that
op.drop_column() doesn't work with sqlite. This is because sqlite
doesn't support dropping a column (broken imho, but that's another
discussion). Sqlite throws a syntax error.
To make this work with sqlite, you have to copy the table to a
+1
How about an undocumented config?
Trev
On Thu, 2014-01-30 at 09:24 -0500, Matthew Farrellee wrote:
> i imagine this is something that can be useful in a development and
> testing environment, especially during the transition period from direct
> to heat. so having the ability is not unreaso
My mistake, it's already there. I missed the distinction between set on
startup and set per cluster.
Trev
On Thu, 2014-01-30 at 10:50 -0500, Trevor McKay wrote:
> +1
>
> How about an undocumented config?
>
> Trev
>
> On Thu, 2014-01-30 at 09:24 -0500, Matthew Farr
Hi Sergey,
In https://review.openstack.org/#/c/69982/1 we are moving the
'main_class' and 'java_opts' fields for a job execution into the
job_configs['configs'] dictionary. This means that 'main_class' and
'java_opts' don't need to be in the database anymore.
These fields were just added in
So, assuming we go forward with this, the followup question is whether
or not to move "main_class" and "java_opts" for Java actions into
"edp.java.main_class" and "edp.java.java_opts" configs.
I think yes.
Best,
Trevor
On Wed, 2014-01-29 at 09:15 -0500,
t new blueprint. Thanks.
> [1] https://review.openstack.org/#/c/69712
> [2]
> https://blueprints.launchpad.net/openstack/?searchtext=edp-oozie-streaming-mapreduce
>
> Regards,
> Alexander Ignatov
>
>
>
> On 28 Jan 2014, at 20:47, Trevor McKay wrote:
>
> > Hell
Hello all,
In our first pass at EDP, the model for job settings was very consistent
across all of our job types. The execution-time settings fit into this
(superset) structure:
job_configs = {'configs': {}, # config settings for oozie and hadoop
'params': {}, # substitution values
We should consider turning "mains" into a string instead of a list for
v2.
Hive and Pig Oozie actions use mains, and each may only specify a single
Matt et al,
Yes, "swift-internal" was meant as a marker to distinguish it from
"swift-external" someday. I agree, this could be indicated by setting
other fields.
Little bit of implementation detail for scope:
In the current EDP implementation, SWIFT_INTERNAL_PREFIX shows up in
essentially
fyi, updates to the diagram based on feedback
On Thu, 2013-07-18 at 13:49 -0400, Trevor McKay wrote:
> Hi all,
>
> Here is a page to hold sequence diagrams for Savanna EDP,
> based on current launchpad blueprints. We thought it might be helpful to
> create some diagrams fo
Hi all,
Here is a page to hold sequence diagrams for Savanna EDP,
based on current launchpad blueprints. We thought it might be helpful to
create some diagrams for discussion as the component specs are written and the
API is worked out:
https://wiki.openstack.org/wiki/Savanna/EDP_Sequences
69 matches
Mail list logo
