Re: Spark History Server does not redirect to Yarn aggregated logs for container logs

Thanks Sai ! I have created a Jira Bug: https://issues.apache.org/jira/browse/SPARK-21013 Regards n Thanks Hari mob: +1 813-510-8215 On Wed, Jun 7, 2017 at 11:44 PM, Saisai Shao wrote: > Yes, currently if log is aggregated, then accessing through UI is not > worked, you can create a JIRA to i

Re: [VOTE] Apache Spark 2.2.0 (RC4)

+1 non-binding Tested on macOS Sierra, Ubuntu 16.04 test suite includes various test cases including Spark SQL, ML, GraphFrames, Structured Streaming On Wed, Jun 7, 2017 at 9:40 PM vaquar khan wrote: > +1 non-binding > > Regards, > vaquar khan > > On Jun 7, 2017 4:32 PM, "Ricardo Almeida" > w

Re: Spark History Server does not redirect to Yarn aggregated logs for container logs

Yes, currently if log is aggregated, then accessing through UI is not worked, you can create a JIRA to improve this if you would like to. On Thu, Jun 8, 2017 at 1:43 PM, ckhari4u wrote: > Hey Guys, > > I am hitting the below issue when trying to access the STDOUT/STDERR logs > in > Spark History

SPARK-19547

Hi, I have been trying to distribute Kafka topics among different instances of same consumer group. I am using KafkaDirectStream API for creating DStreams. After the second consumer group comes up, Kafka does partition rebalance and then Spark driver of the first consumer dies with the followin

Spark History Server does not redirect to Yarn aggregated logs for container logs

Hey Guys, I am hitting the below issue when trying to access the STDOUT/STDERR logs in Spark History Server for the executors of a Spark application executed in Yarn mode. I have enabled Yarn log aggregation. Repro Steps: 1) Run the spark-shell in yarn client mode. Or run Pi job in Yarn mode.

Re: [VOTE] Apache Spark 2.2.0 (RC4)

+1 non-binding Regards, vaquar khan On Jun 7, 2017 4:32 PM, "Ricardo Almeida" wrote: +1 (non-binding) Built and tested with -Phadoop-2.7 -Dhadoop.version=2.7.3 -Pyarn -Phive -Phive-thriftserver -Pscala-2.11 on - Ubuntu 17.04, Java 8 (OpenJDK 1.8.0_111) - macOS 10.12.5 Java 8 (build 1.8.

Re: [VOTE] Apache Spark 2.2.0 (RC4)

+1 (non-binding) Built and tested with -Phadoop-2.7 -Dhadoop.version=2.7.3 -Pyarn -Phive -Phive-thriftserver -Pscala-2.11 on - Ubuntu 17.04, Java 8 (OpenJDK 1.8.0_111) - macOS 10.12.5 Java 8 (build 1.8.0_131) On 5 June 2017 at 21:14, Michael Armbrust wrote: > Please vote on releasing th

Will higher order functions in spark SQL be pushed upstream?

Hey guys, Databricks released higher order functions as part of their runtime 3.0 beta (https://databricks.com/blog/2017/05/24/working-with-nested-data-using-higher-order-functions-in-sql-on-databricks.html), which helps working with array within SQL statements. * As a heavy user of complex data

Re: Is static volatile variable different with static variable in the closure?

static and volatile are unrelated. Being volatile doesn't change the properties of the variable with respect to being static. On Wed, Jun 7, 2017 at 4:01 PM Chang Chen wrote: > Static variable will be initialized in worker node JVM, will not be > serialized from master. But how about static vol

Is static volatile variable different with static variable in the closure?

Static variable will be initialized in worker node JVM, will not be serialized from master. But how about static volatile variable? Recently I read the beam spark runner code, and i find that they use static volatile Broadcast variable. See https://github.com/apache/beam/blob/master/runners/spark