[Spark on mesos] Spark framework not re-registered and lost after mesos master restarted

[Yu Wei]

Hi guys,

I encountered a problem about spark on mesos.

I setup mesos cluster and launched spark framework on mesos successfully.

Then mesos master was killed and started again.

However, spark framework couldn't be re-registered again as mesos agent does. I 
also couldn't find any error logs.

And MesosClusterDispatcher is still running there.


I suspect this is spark framework issue.

What's your opinion?



Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux

Re: planning & discussion for larger scheduler changes

[Tom Graves]

If we are worried about major changes destabilizing current code (which I can 
understand) only way around that is to make it pluggable or configurable.  For 
major changes it seems like making it pluggable is cleaner from a code being 
cluttered point of view. But it also means you may have to make the same or 
similar change in 2 places.We could make the interfaces more well defined but 
if the major changes would require interfaces changes that doesn't help.  It 
still seems like if we had a list of things we would like to accomplish and get 
an idea of the rough overall design we could see if defining the interfaces 
better or making them pluggable would help.
There seem to be 3 jiras all related to handling fetch failures: 
https://issues.apache.org/jira/browse/SPARK-20091,  SPARK-14649 , and 
SPARK-19753.  It might be nice to create one epic jira where we think about a 
design as a whole and discuss that more.  Any objections to this?  If not I'll 
create an epic and link the others to it.

Tom
On Monday, March 27, 2017 9:01 PM, Kay Ousterhout  
wrote:
 

 (1) I'm pretty hesitant to merge these larger changes, even if they're feature 
flagged, because:   (a) For some of these changes, it's not obvious that 
they'll always improve performance. e.g., for SPARK-14649, it's possible that 
the tasks that got re-started (and temporarily are running in two places) are 
going to fail in the first attempt (because they haven't read the missing map 
output yet).  In that case, not re-starting them will lead to worse 
performance.   (b) The scheduler already has some secret flags that aren't 
documented and are used by only a few people.  I'd like to avoid adding more of 
these (e.g., by merging these features, but having them off by default), 
because very few users use them (since it's hard to learn about them), they add 
complexity to the scheduler that we have to maintain, and for users who are 
considering using them, they often hide advanced behavior that's hard to reason 
about anyway (e.g., the point above for SPARK-14649).    (c) The worst 
performance problem is when jobs just hang or crash; we've seen a few cases of 
that in recent bugs, and I'm worried that merging these complex performance 
improvements trades better performance in a small number of cases for the 
possibility of worse performance via job crashes/hangs in other cases.
Roughly I think our standards for merging performance fixes to the scheduler 
should be that the performance improvement either (a) is simple / easy to 
reason about or (b) unambiguously fixes a serious performance problem.  In the 
case of SPARK-14649, for example, it is complex, and improves performance in 
some cases but hurts it in others, so doesn't fit either (a) or (b).
(2) I do think there are some scheduler re-factorings that would improve 
testability and our ability to reason about correctness, but think there are 
some what surgical, smaller things we could do in the vein of Imran's comment 
about reducing shared state.  Right now we have these super wide interfaces 
between different components of the scheduler, and it means you have to reason 
about the TSM, TSI, CGSB, and DAGSched to figure out whether something works.  
I think we could have an effort to make each component have a much narrower 
interface, so that each part hides a bunch of complexity from other components. 
 The most obvious place to do this in the short term is to remove a bunch of 
info tracking from the DAGScheduler; I filed a JIRA for that here.  I suspect 
there are similar things that could be done in other parts of the scheduler.
Tom's comments re: (2) are more about performance improvements rather than 
readability / testability / debuggability, but also seem important and it does 
seem useful to have a JIRA tracking those.
-Kay
On Mon, Mar 27, 2017 at 11:06 AM, Tom Graves  wrote:

1) I think this depends on individual case by case jira.  I haven't looked in 
detail at spark-14649 seems much larger although more the way I think we want 
to go. While SPARK-13669 seems less risky and easily configurable.
2) I don't know whether it needs an entire rewrite but I think there need to be 
some major changes made especially in the handling of reduces and fetch 
failures.  We could do a much better job of not throwing away existing work and 
more optimally handling the failure cases.  For this would it make sense for us 
to start with a jira that has a bullet list of things we would like to improve 
and get more cohesive view and see really how invasive it would be?
Tom 

On Friday, March 24, 2017 10:41 AM, Imran Rashid  
wrote:
 

 Kay and I were discussing some of the  bigger scheduler changes getting 
proposed lately, and realized there is a broader discussion to have with the 
community, outside of any single jira.  I'll start by sharing my initial 
thoughts, I know Kay has thoughts on this too, but it would be good to input 
from everyone.
In particular, SPARK-14649 & SPARK

Re: [Spark on mesos] Spark framework not re-registered and lost after mesos master restarted

[Timothy Chen]

I think failover isn't enabled on regular Spark job framework, since we assume 
jobs are more ephemeral.

It could be a good setting to add to the Spark framework to enable failover.

Tim

> On Mar 30, 2017, at 10:18 AM, Yu Wei  wrote:
> 
> Hi guys,
> 
> I encountered a problem about spark on mesos.
> 
> I setup mesos cluster and launched spark framework on mesos successfully.
> 
> Then mesos master was killed and started again.
> 
> However, spark framework couldn't be re-registered again as mesos agent does. 
> I also couldn't find any error logs.
> 
> And MesosClusterDispatcher is still running there.
> 
> I suspect this is spark framework issue. 
> What's your opinion?
> 
> 
> Thanks,
> 
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux

Re: Outstanding Spark 2.1.1 issues

[Holden Karau]

Hi All,

Just circling back to see if there is anything blocking the RC that isn't
being tracked in JIRA?

The current in progress list from ((affectedVersion = 2.1.1 AND
cf[12310320] is Empty) OR fixVersion = 2.1.1 OR cf[12310320] = "2.1.1") AND
project = spark AND resolution = Unresolved ORDER BY priority DESC is only
4 elements:

   1. SPARK-19690  - Join
   a streaming DataFrame with a batch DataFrame may not work (PR
   https://github.com/apache/spark/pull/17052
    ) - some discussion around
   re-targeting exists on the PR
   2.
  1.
 1. SPARK-19522 
  - --executor-memory flag doesn't work in local-cluster mode (PR
 https://github.com/apache/spark/pull/16975
 
 2. SPARK-19035 
  -rand() function in case when cause failed - no PR exists and it
 isn't a blocker so I'd suggest we consider re-targetting
1. SPARK-19759
    -
ALSModel.predict
   on Dataframes : potential optimization by not using
blas - not explicitly
   targeted but I'd suggest targeting for 2.3 if people agree



Cheers,

Holden :)

On Tue, Mar 28, 2017 at 2:07 PM, Xiao Li  wrote:

> Hi, Michael,
>
> Since Daniel Siegmann asked for a bug fix backport in the previous email,
> I just merged https://issues.apache.org/jira/browse/SPARK-14536 into
> Spark 2.1 branch.
>
> If this JIRA is not part of Spark 2.1.1 release, could you help me correct
> the fix version from 2.1.1. to the next release number.
>
> Thanks,
>
> Xiao
>
> 2017-03-28 8:33 GMT-07:00 Michael Armbrust :
>
>> We just fixed the build yesterday.  I'll kick off a new RC today.
>>
>> On Tue, Mar 28, 2017 at 8:04 AM, Asher Krim  wrote:
>>
>>> Hey Michael,
>>> any update on this? We're itching for a 2.1.1 release (specifically
>>> SPARK-14804 which is currently blocking us)
>>>
>>> Thanks,
>>> Asher Krim
>>> Senior Software Engineer
>>>
>>> On Wed, Mar 22, 2017 at 7:44 PM, Michael Armbrust <
>>> mich...@databricks.com> wrote:
>>>
 An update: I cut the tag for RC1 last night.  Currently fighting with
 the release process.  Will post RC1 once I get it working.

 On Tue, Mar 21, 2017 at 2:16 PM, Nick Pentreath <
 nick.pentre...@gmail.com> wrote:

> As for SPARK-19759 ,
> I don't think that needs to be targeted for 2.1.1 so we don't need to 
> worry
> about it
>
>
> On Tue, 21 Mar 2017 at 13:49 Holden Karau 
> wrote:
>
>> I agree with Michael, I think we've got some outstanding issues but
>> none of them seem like regression from 2.1 so we should be good to start
>> the RC process.
>>
>> On Tue, Mar 21, 2017 at 1:41 PM, Michael Armbrust <
>> mich...@databricks.com> wrote:
>>
>> Please speak up if I'm wrong, but none of these seem like critical
>> regressions from 2.1.  As such I'll start the RC process later today.
>>
>> On Mon, Mar 20, 2017 at 9:52 PM, Holden Karau 
>> wrote:
>>
>> I'm not super sure it should be a blocker for 2.1.1 -- is it a
>> regression? Maybe we can get TDs input on it?
>>
>> On Mon, Mar 20, 2017 at 8:48 PM Nan Zhu 
>> wrote:
>>
>> I think https://issues.apache.org/jira/browse/SPARK-19280 should be
>> a blocker
>>
>> Best,
>>
>> Nan
>>
>> On Mon, Mar 20, 2017 at 8:18 PM, Felix Cheung <
>> felixcheun...@hotmail.com> wrote:
>>
>> I've been scrubbing R and think we are tracking 2 issues
>>
>> https://issues.apache.org/jira/browse/SPARK-19237
>>
>> https://issues.apache.org/jira/browse/SPARK-19925
>>
>>
>>
>>
>> --
>> *From:* holden.ka...@gmail.com  on behalf of
>> Holden Karau 
>> *Sent:* Monday, March 20, 2017 3:12:35 PM
>> *To:* dev@spark.apache.org
>> *Subject:* Outstanding Spark 2.1.1 issues
>>
>> Hi Spark Developers!
>>
>> As we start working on the Spark 2.1.1 release I've been looking at
>> our outstanding issues still targeted for it. I've tried to break it down
>> by component so that people in charge of each component can take a quick
>> look and see if any of these things can/should be re-targeted to 2.2 or
>> 2.1.2 & the overall list is pretty short (only 9 items - 5 if we only 
>> look
>> at explicitly tagged) :)
>>
>> If your working on something for Spark 2.1.1 and it doesn't show up
>> in this list please speak up now :) We have a lot of issues (including 
>> "in
>> progress") that are listed as impacting 2.1.0, but they aren't targeted 
>> for
>> 2.1.1 - if there is something you a

[VOTE] Apache Spark 2.1.1 (RC2)

[Michael Armbrust]

Please vote on releasing the following candidate as Apache Spark version
2.1.0. The vote is open until Sun, April 2nd, 2018 at 16:30 PST and passes
if a majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Spark 2.1.1
[ ] -1 Do not release this package because ...


To learn more about Apache Spark, please see http://spark.apache.org/

The tag to be voted on is v2.1.1-rc2
 (
02b165dcc2ee5245d1293a375a31660c9d4e1fa6)

List of JIRA tickets resolved can be found with this filter

.

The release files, including signatures, digests, etc. can be found at:
http://home.apache.org/~pwendell/spark-releases/spark-2.1.1-rc2-bin/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/pwendell.asc

The staging repository for this release can be found at:
https://repository.apache.org/content/repositories/orgapachespark-1227/

The documentation corresponding to this release can be found at:
http://people.apache.org/~pwendell/spark-releases/spark-2.1.1-rc2-docs/


*FAQ*

*How can I help test this release?*

If you are a Spark user, you can help us test this release by taking an
existing Spark workload and running on this release candidate, then
reporting any regressions.

*What should happen to JIRA tickets still targeting 2.1.1?*

Committers should look at those and triage. Extremely important bug fixes,
documentation, and API tweaks that impact compatibility should be worked on
immediately. Everything else please retarget to 2.1.2 or 2.2.0.

*But my bug isn't fixed!??!*

In order to make timely releases, we will typically not hold the release
unless the bug in question is a regression from 2.1.0.

*What happened to RC1?*

There were issues with the release packaging and as a result was skipped.

Predicate not getting pusdhown to PrunedFilterScan

[Hanumath Rao Maduri]

Hello All,

I am working on creating a new PrunedFilteredScan operator which has the
ability to execute the predicates pushed to this operator.

However What I observed is that if column with deep in the hierarchy is
used then it is not getting pushed down.

SELECT tom._id, tom.address.city from tom where tom.address.city = "Peter"

Here predicate tom.address.city = "Peter" is not getting pushed down.

However if the first level column name is used then it is getting pushed
down.

SELECT tom._id, tom.address.city from tom where tom.first_name = "Peter"


Please let me know what is the issue in this case.

Thanks,

Re: [Spark on mesos] Spark framework not re-registered and lost after mesos master restarted

[Yu Wei]

Hi Tim,

I tested the scenario again with settings as below,

[dcos@agent spark-2.0.2-bin-hadoop2.7]$ cat conf/spark-defaults.conf
spark.deploy.recoveryMode  ZOOKEEPER
spark.deploy.zookeeper.url 192.168.111.53:2181
spark.deploy.zookeeper.dir /spark
spark.executor.memory 512M
spark.mesos.principal agent-dev-1


However, the case still failed. After master restarted, spark framework did not 
re-register.
From spark framework log, it seemed that below method in MesosClusterScheduler 
was not called.
override def reregistered(driver: SchedulerDriver, masterInfo: MasterInfo): Unit

Did I miss something? Any advice?



Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux



From: Timothy Chen 
Sent: Friday, March 31, 2017 5:13 AM
To: Yu Wei
Cc: us...@spark.apache.org; dev
Subject: Re: [Spark on mesos] Spark framework not re-registered and lost after 
mesos master restarted

I think failover isn't enabled on regular Spark job framework, since we assume 
jobs are more ephemeral.

It could be a good setting to add to the Spark framework to enable failover.

Tim

On Mar 30, 2017, at 10:18 AM, Yu Wei 
mailto:yu20...@hotmail.com>> wrote:


Hi guys,

I encountered a problem about spark on mesos.

I setup mesos cluster and launched spark framework on mesos successfully.

Then mesos master was killed and started again.

However, spark framework couldn't be re-registered again as mesos agent does. I 
also couldn't find any error logs.

And MesosClusterDispatcher is still running there.


I suspect this is spark framework issue.

What's your opinion?



Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux

Re: [Spark on mesos] Spark framework not re-registered and lost after mesos master restarted

[Timothy Chen]

Hi Yu,

As mentioned earlier, currently the Spark framework will not
re-register as the failover_timeout is not set and there is no
configuration available yet.
It's only enabled in MesosClusterScheduler since it's meant to be a HA
framework.

We should add that configuration for users that want their Spark
frameworks to be able to failover in case of Master failover or
network disconnect, etc.

Tim

On Thu, Mar 30, 2017 at 8:25 PM, Yu Wei  wrote:
> Hi Tim,
>
> I tested the scenario again with settings as below,
>
> [dcos@agent spark-2.0.2-bin-hadoop2.7]$ cat conf/spark-defaults.conf
> spark.deploy.recoveryMode  ZOOKEEPER
> spark.deploy.zookeeper.url 192.168.111.53:2181
> spark.deploy.zookeeper.dir /spark
> spark.executor.memory 512M
> spark.mesos.principal agent-dev-1
>
>
> However, the case still failed. After master restarted, spark framework did
> not re-register.
> From spark framework log, it seemed that below method in
> MesosClusterScheduler was not called.
> override def reregistered(driver: SchedulerDriver, masterInfo: MasterInfo):
> Unit
>
> Did I miss something? Any advice?
>
>
> Thanks,
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux
>
>
>
> 
> From: Timothy Chen 
> Sent: Friday, March 31, 2017 5:13 AM
> To: Yu Wei
> Cc: us...@spark.apache.org; dev
> Subject: Re: [Spark on mesos] Spark framework not re-registered and lost
> after mesos master restarted
>
> I think failover isn't enabled on regular Spark job framework, since we
> assume jobs are more ephemeral.
>
> It could be a good setting to add to the Spark framework to enable failover.
>
> Tim
>
> On Mar 30, 2017, at 10:18 AM, Yu Wei  wrote:
>
> Hi guys,
>
> I encountered a problem about spark on mesos.
>
> I setup mesos cluster and launched spark framework on mesos successfully.
>
> Then mesos master was killed and started again.
>
> However, spark framework couldn't be re-registered again as mesos agent
> does. I also couldn't find any error logs.
>
> And MesosClusterDispatcher is still running there.
>
>
> I suspect this is spark framework issue.
>
> What's your opinion?
>
>
>
> Thanks,
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

Predicate not getting pusdhown to PrunedFilterScan

[Hanumath Rao Maduri]

Hello All,

I am working on creating a new PrunedFilteredScan operator which has the
ability to execute the predicates pushed to this operator.

However What I observed is that if column with deep in the hierarchy is
used then it is not getting pushed down.

SELECT tom._id, tom.address.city from tom where tom.address.city = "Peter"

Here predicate tom.address.city = "Peter" is not getting pushed down.

However if the first level column name is used then it is getting pushed
down.

SELECT tom._id, tom.address.city from tom where tom.first_name = "Peter"


Please let me know what is the issue in this case.

Thanks,

Spark - Kinesis integration needs improvements

[Yash Sharma]

Hello fellow spark devs, hope you are doing fabulous,

Dropping a brain dump here about the Spark kinesis integration. I am able
to get spark kinesis to work perfectly under ideal conditions, but see a
lot of open ends when things are not so ideal. I feel there are lot of open
ends and are specific to Kinesis (and need attention).

Unlike Kafka, which works flawlessly with Spark, Kinesis has its own sets
of issues involving Throttling and recovery. I would like to volunteer to
fix few such issues faced by us. Proposing some critical improvements which
would make Kinesis more reliable with Spark:

1. Kinesis Retries : Right now kinesis retries are hard coded with a very
small value of 100 ms. While this is ok for normal errors while fetching,
its not ok for Throttling errors. The Kinesis throttles are lot more
frequent in a recovery situation (will talk more on it in point 3).The
Kinesis throttles are per second, hence a retry in 100 ms will not take it
any far. We should have these values configurable to have more control over
the retries. https://github.com/apache/spark/pull/17467

2. Kinesis recovery from provided timestamp : Kinesis client lib has the
capability of resuming from a TIMESTAMP and this is very useful when we
would want to start from a specified time in past rather than reading from
INITIAL position. We have few scenarios where we need to read a huge stream
and filter out all data before a timestamp. This change will push this to
kinesis client and get only relevant data. I have got a half-baked patch
for this. https://issues.apache.org/jira/browse/SPARK-20168

3. Kinesis recovery : This issue is a bit tricky and I would need help of
experienced contributors for a proper design.
Kinesis in ideal scenarios uses the kinesis consumer library, and the
library takes care of the requests made to shards. But in case of recovery
the Kinesis consumer makes direct requests to Kinesis from all the tasks.
Thousands of tasks request to the all the shards (multiple requests per
shards). Unlike Kafka where the requests are just network bound, Kinesis
requests are Read throttled if lot of requests are made in a second. Not
sure what would be the best way to make these requests organized but we
should have some way of limiting number of request per shard. We use
workaround of increasing the fail tasks tolerance to a ridiculously high
value and the job completes after a thousand failures. Point 1 mentioned
above ^ can also be used in this workaround to delay the retries so that
the request volume decreases per shard. I don't have a ticket for it yet
but will get one soon.

4. Spark doesn't work with AWS temporary tokens : Currently spark just
reads the aws tokens using various strategies. Temporary session tokens are
more secure and (probably) encouraged way of using the AWS credentials.
Temporary tokens are also useful for accessing cross account aws resources.
Currently not lot of usecases are using temporary tokens, but soon this
will be a common scenario. We are already facing some minor challenges and
resorting to workarounds until its supported in Spark. We should have a
mechanism to make Spark work with the temporary aws tokens. Thoughts ?

I am trying to get familiar to Spark's code base along the process. I will
be primarily focussing on the spark-kinesis bit for now. I would like to
have your valuable inputs on these.

Cheers,
Yash

Re: [Spark on mesos] Spark framework not re-registered and lost after mesos master restarted

[Yu Wei]

Got that.


Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux


From: Timothy Chen 
Sent: Friday, March 31, 2017 11:33:42 AM
To: Yu Wei
Cc: dev; us...@spark.apache.org
Subject: Re: [Spark on mesos] Spark framework not re-registered and lost after 
mesos master restarted

Hi Yu,

As mentioned earlier, currently the Spark framework will not
re-register as the failover_timeout is not set and there is no
configuration available yet.
It's only enabled in MesosClusterScheduler since it's meant to be a HA
framework.

We should add that configuration for users that want their Spark
frameworks to be able to failover in case of Master failover or
network disconnect, etc.

Tim

On Thu, Mar 30, 2017 at 8:25 PM, Yu Wei  wrote:
> Hi Tim,
>
> I tested the scenario again with settings as below,
>
> [dcos@agent spark-2.0.2-bin-hadoop2.7]$ cat conf/spark-defaults.conf
> spark.deploy.recoveryMode  ZOOKEEPER
> spark.deploy.zookeeper.url 192.168.111.53:2181
> spark.deploy.zookeeper.dir /spark
> spark.executor.memory 512M
> spark.mesos.principal agent-dev-1
>
>
> However, the case still failed. After master restarted, spark framework did
> not re-register.
> From spark framework log, it seemed that below method in
> MesosClusterScheduler was not called.
> override def reregistered(driver: SchedulerDriver, masterInfo: MasterInfo):
> Unit
>
> Did I miss something? Any advice?
>
>
> Thanks,
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux
>
>
>
> 
> From: Timothy Chen 
> Sent: Friday, March 31, 2017 5:13 AM
> To: Yu Wei
> Cc: us...@spark.apache.org; dev
> Subject: Re: [Spark on mesos] Spark framework not re-registered and lost
> after mesos master restarted
>
> I think failover isn't enabled on regular Spark job framework, since we
> assume jobs are more ephemeral.
>
> It could be a good setting to add to the Spark framework to enable failover.
>
> Tim
>
> On Mar 30, 2017, at 10:18 AM, Yu Wei  wrote:
>
> Hi guys,
>
> I encountered a problem about spark on mesos.
>
> I setup mesos cluster and launched spark framework on mesos successfully.
>
> Then mesos master was killed and started again.
>
> However, spark framework couldn't be re-registered again as mesos agent
> does. I also couldn't find any error logs.
>
> And MesosClusterDispatcher is still running there.
>
>
> I suspect this is spark framework issue.
>
> What's your opinion?
>
>
>
> Thanks,
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux