Re: spark-deployer 3.0.1 released

Grt will test and share feedback Sent from my iPhone > On 09-Sep-2016, at 9:37 PM, pishen tsai wrote: > > spark-deployer is a sbt plugin that help deploying a Spark stand-alone > cluster on EC2 and submit your Spark jobs. All the works are done in sbt. > > We just released a new version with

Please help: Spark job hung/stop writing after exceeding the folder size

Dear All, I have struggling with an issue where spark steam job gets hung after exceeding size of output folder path. here is more details: I have Flume sending and configuration agent1.sources = source1 agent1.sinks = sink1 agent1.channels = channel2 # Describe/configure source1 agent1.sources

flume.thrift.queuesize

Please suggest me where/in which file I should set/configure " flume.thrift.queuesize" Many Thanks!

Spark steaming with Flume jobs failing

Hi team, I have integrated SparkSteam with Flume and my flume as well spark job gets failed and getting following error. Your help will be highly appreciative. Many Thanks my flume configuration is as follows flume.conf *** agent1.sources = source1 agent1.sinks = sink1 agent1.channels = ch

How to read content of hdfs files

I have hdfs data in zip formate which includes data, name and nameseconday folder. Pretty much structure is like datanode, name node and secondary node. How to read the content of data. would be great if some can suggest tips/steps. Thanks

Re: ImportError: No module named numpy

lled on all machines? >> >> To check that the environment is set-up correctly, you can try something >> like >> >> import os >> pythonpaths = sc.range(10).map(lambda i: >> os.environ.get("PYTHONPATH")).collect() >> print(pythonpaths) >>

Re: ImportError: No module named numpy

did not resolved. :( On Thu, Jun 2, 2016 at 3:01 PM, Sergio Fernández wrote: > > On Thu, Jun 2, 2016 at 9:59 AM, Bhupendra Mishra < > bhupendra.mis...@gmail.com> wrote: >> >> and i have already exported environment variable in spark-env.sh as >> follows.. erro

Re: ImportError: No module named numpy

ry adding to spark-env.sh (renaming if you still have it with .template > at the end): > > PYSPARK_PYTHON=/path/to/your/bin/python > > Where your bin/python is your actual Python environment with Numpy > installed. > > > El 1 jun 2016, a las 20:16, Bhupendra Mishra > e

Re: ImportError: No module named numpy

I have numpy installed but where I should setup PYTHONPATH? On Wed, Jun 1, 2016 at 11:39 PM, Sergio Fernández wrote: > sudo pip install numpy > > On Wed, Jun 1, 2016 at 5:56 PM, Bhupendra Mishra < > bhupendra.mis...@gmail.com> wrote: > >> Thanks . >> How can th

Re: ImportError: No module named numpy

Thanks . How can this be resolved? On Wed, Jun 1, 2016 at 9:02 PM, Holden Karau wrote: > Generally this means numpy isn't installed on the system or your > PYTHONPATH has somehow gotten pointed somewhere odd, > > On Wed, Jun 1, 2016 at 8:31 AM, Bhupendra Mishra < > bhu

ImportError: No module named numpy

If any one please can help me with following error. File "/opt/mapr/spark/spark-1.6.1/python/lib/pyspark.zip/pyspark/mllib/__init__.py", line 25, in ImportError: No module named numpy Thanks in advance!

spark-ts

Guys, please help me with following question on Spark-TS liabrary You’ve just acquired a new dataset showing the purchases of stock from market resellers during the day over a ten month period. You’ve looked at the daily data and have decided that you can model this using a time series analysis. Y

Re: PySpark without PySpark

Very interesting and well organized post. Thanks for sharing On Wed, Jul 8, 2015 at 10:29 PM, Sujit Pal wrote: > Hi Julian, > > I recently built a Python+Spark application to do search relevance > analytics. I use spark-submit to submit PySpark jobs to a Spark cluster on > EC2 (so I don't use th