Re: Memory leak warnings in Spark 2.0.1

See https://issues.apache.org/jira/browse/SPARK-18557 On Mon, Nov 21, 2016 at 1:16 PM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > I'm also curious about this. Is there something we can do to help > troubleshoot these leaks and file

Is it possible to pass "-javaagent=customAgent.jar" into spark as a JAVA_OPTS

Hi, I'm interested in passing an agent that will expose jmx metrics from spark to my agent. I wanted to know if anyone has tried this and if so what environment variable do I need to set ? Do I set: $SPARK_DAEMON_JAVA_OPTS ?? http://docs.oracle.com/javase/7/docs/api/java/lang/instrument/package-

Re: [SparkStreaming] 1 SQL tab for each SparkStreaming batch in SparkUI

If you create a HiveContext before starting StreamingContext, then `SQLContext.getOrCreate` in foreachRDD will return the HiveContext you created. You can just call asInstanceOf[HiveContext] to convert it to HiveContext. On Tue, Nov 22, 2016 at 8:25 AM, Dirceu Semighini Filho < dirceu.semigh...@gm

Re: Please limit commits for branch-2.1

Thanks, this was another message that went to spam for me: http://apache-spark-developers-list.1001551.n3.nabble.com/ANNOUNCE-Apache-Spark-branch-2-1-td19688.html Looks great -- cutting branch = in RC period. On Tue, Nov 22, 2016 at 5:31 PM Reynold Xin wrote: > I did send an email out with tho

Re: Please limit commits for branch-2.1

I did send an email out with those information on Nov 1st. It is not meant to be in new feature development mode anymore. FWIW, I will cut an RC today to remind people of that. The RC will fail, but it can serve as a good reminder. On Tue, Nov 22, 2016 at 1:53 AM Sean Owen wrote: > Maybe I miss

Re: [SparkStreaming] 1 SQL tab for each SparkStreaming batch in SparkUI

Hi Koert, Certainly it's not a good idea, I was trying to use SQLContext.getOrCreate but it will return a SQLContext and not a HiveContext. As I'm using a checkpoint, whenever I start the context by reading the checkpoint it didn't create my hive context, unless I create it foreach microbach. I did

Re: [SPARK-16654][CORE][WIP] Add UI coverage for Application Level Blacklisting

Hi Reynold - Alright, if that makes things easier, here is the same patch set against Imran's squito:blacklist-SPARK-8425 branch. https://github.com/squito/spark/pull/6 Cheers, --José On Mon, Nov 21, 2016 at 10:22 PM, Reynold Xin wrote: > You can submit a pull request against Imran's branch fo

Re: [SparkStreaming] 1 SQL tab for each SparkStreaming batch in SparkUI

you are creating a new hive context per microbatch? is that a good idea? On Tue, Nov 22, 2016 at 8:51 AM, Dirceu Semighini Filho < dirceu.semigh...@gmail.com> wrote: > Has anybody seen this behavior (see tha attached picture) in Spark > Streaming? > It started to happen here after I changed the H

[SparkStreaming] 1 SQL tab for each SparkStreaming batch in SparkUI

Has anybody seen this behavior (see tha attached picture) in Spark Streaming? It started to happen here after I changed the HiveContext creation to stream.foreachRDD { rdd => val hiveContext = new HiveContext(rdd.sparkContext) } Is this expected? Kind Regards, Dirceu

Re: How is the order ensured in the jdbc relation provider when inserting data from multiple executors

On 11/22/2016 12:11 PM, nirandap wrote: > Hi Maciej, > > Thank you for your reply. > > I have 2 queries. > 1. I can understand your explanation. But in my experience, when I > check the final RDBMS table, I see that the results follow the > expected order, without an issue. Is this just a coinci

Re: How is the order ensured in the jdbc relation provider when inserting data from multiple executors

Hi Maciej, Thank you for your reply. I have 2 queries. 1. I can understand your explanation. But in my experience, when I check the final RDBMS table, I see that the results follow the expected order, without an issue. Is this just a coincidence? 2. I was further looking into this. So, say I run

Re: Please limit commits for branch-2.1

Maybe I missed it, but did anyone declare a QA period? In the past I've not seen this, and just seen people start talking retrospectively about how "we're in QA now" until it stops. We have https://cwiki.apache.org/confluence/display/SPARK/Wiki+Homepage saying it is already over, but clearly we're