style checker will be turned on for test code

2015-04-12 Thread Reynold Xin
FYI once this pull request goes in, scalastyle will be run against all test code as well. Previously we didn't run the style checker on test code. https://github.com/apache/spark/pull/5486 We might see the master build breaking tomorrow if a PR has style violations in test code and is tested by J

Re: How is hive-site.xml loaded?

2015-04-12 Thread Raunak Jhawar
The most obvious path being /etc/hive/conf, but this can be changed to lookup for any other path. -- Thanks, Raunak Jhawar On Mon, Apr 13, 2015 at 11:22 AM, Dean Chen wrote: > Ah ok, thanks! > > -- > Dean Chen > > On Apr 12, 2015, at 10:45 PM, Reynold Xin wrote: > > It is loaded by Hive's

Re: How is hive-site.xml loaded?

2015-04-12 Thread Dean Chen
Ah ok, thanks! -- Dean Chen On Apr 12, 2015, at 10:45 PM, Reynold Xin wrote: It is loaded by Hive's HiveConf, which simply searches for hive-site.xml on the classpath. On Sun, Apr 12, 2015 at 10:41 PM, Dean Chen wrote: > The docs state that: > Configuration of Hive is done by placing your `

Re: How is hive-site.xml loaded?

2015-04-12 Thread Reynold Xin
It is loaded by Hive's HiveConf, which simply searches for hive-site.xml on the classpath. On Sun, Apr 12, 2015 at 10:41 PM, Dean Chen wrote: > The docs state that: > Configuration of Hive is done by placing your `hive-site.xml` file in > `conf/`. > > I've searched the codebase for hive-site.xm

How is hive-site.xml loaded?

2015-04-12 Thread Dean Chen
The docs state that: Configuration of Hive is done by placing your `hive-site.xml` file in `conf/`. I've searched the codebase for hive-site.xml and didn't find code that specifically loaded it anywhere so it looks like there is some magic to autoload *.xml files in /conf? I've skimmed through Hiv

Re: Integrating D3 with Spark

2015-04-12 Thread anshu shukla
Hey Ousterhout , I found its amazing .Before this i used to use my own D3.js files that subscribes to the redis pub-shub database where output tuples are being published to the DB . So it was already including latency to push data to redis ,although it was very less. Once again tha

Re: [VOTE] Release Apache Spark 1.3.1 (RC3)

2015-04-12 Thread Mark Hamstra
+1 On Fri, Apr 10, 2015 at 11:05 PM, Patrick Wendell wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.3.1! > > The tag to be voted on is v1.3.1-rc2 (commit 3e83913): > > https://git-wip-us.apache.org/repos/asf?p=spark.git;a=commit;h=3e8391327ba586eaf54447043b

Re: wait time between start master and start slaves

2015-04-12 Thread Nicholas Chammas
Oh, good point. So I guess I should be able to query the master via code like this before any slaves are started. On Sat, Apr 11, 2015 at 7:52 PM Ted Yu wrote: > From SparkUI.scala : > > def getUIPort(conf: SparkConf): Int = { > conf.getInt("spark.ui.port", SparkUI.DEFAULT_PORT) > } > Be

Re: SPARK-5364

2015-04-12 Thread Reynold Xin
I closed it. Thanks. On Sun, Apr 12, 2015 at 11:08 AM, Sree V wrote: > Hi, > I was browsing through the JIRAs and found this can be closed.If anyone > who has edit permissions on Spark JIRA, please close this. > https://issues.apache.org/jira/browse/SPARK-5364 > It is OpenIts Pull Request alrea

Re: SPARK-5364

2015-04-12 Thread Sree V
Hi, I was browsing through the JIRAs and found this can be closed.If anyone who has edit permissions on Spark JIRA, please close this. https://issues.apache.org/jira/browse/SPARK-5364 It is OpenIts Pull Request already merged Its parent and grand parent Resolved Thanking you. With Regards Sree

Re: Integrating D3 with Spark

2015-04-12 Thread Kay Ousterhout
Hi Pradyumn, Take a look at this pull request, which does something similar: https://github.com/apache/spark/pull/2342/files You can put JavaScript in

Re: Parquet File Binary column statistics error when reuse byte[] among rows

2015-04-12 Thread Cheng Lian
Thanks for reporting this! Would you mind to open JIRA tickets for both Spark and Parquet? I'm not sure whether Parquet declares somewhere the user mustn't reuse byte arrays when using binary type. If it does, then it's a Spark bug. Anyway, this should be fixed. Cheng On 4/12/15 1:50 PM, Yi