Anyone has perfect solution for spark source code compilation issue on intellij

2015-11-03 Thread canan chen
Hi folks, I often meet the spark compilation issue on intellij. It wastes me lots of time. I googled it and found someone else also meet similar issue, but seems no perfect solution for now. but still wondering anyone here has perfect solution for that. The issue happens sometimes, I don't know wh

Re: PMML export for LinearRegressionModel

2015-10-15 Thread canan chen
The method toPMML is in trait PMMLExportable *LinearRegressionModel has this trait, you should be able to call * *LinearRegressionModel#toPMML* On Thu, Oct 15, 2015 at 5:25 PM, Fazlan Nazeem wrote: > Hi > > I am trying to export a LinearRegressionModel in PMML format. According to > the followi

Re: When does python program started in pyspark

2015-10-13 Thread canan chen
Runner @ > https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/PythonRunner.scala > > On Tue, Oct 13, 2015 at 7:50 PM, canan chen wrote: > >> I look at the source code of spark, but didn't find where python program >> is started in pytho

When does python program started in pyspark

2015-10-13 Thread canan chen
I look at the source code of spark, but didn't find where python program is started in python. It seems spark-submit will call PythonGatewayServer, but where is python program started ? Thanks

Re: What's the best practice for developing new features for spark ?

2015-08-19 Thread canan chen
gt; http://search-hadoop.com/m/q3RTtdZv0d1btRHl/Spark+build+module&subj=Building+Spark+Building+just+one+module+ > > > > > On Aug 19, 2015, at 1:44 AM, canan chen wrote: > > > > I want to work on one jira, but it is not easy to do unit test, because > it involves di

What's the best practice for developing new features for spark ?

2015-08-19 Thread canan chen
I want to work on one jira, but it is not easy to do unit test, because it involves different components especially UI. spark building is pretty slow, I don't want to build it each time to test my code change. I am wondering how other people do ? Is there any experience can share ? Thanks