Re: Reading lzo+index with spark-csv (Splittable reads)

Well looking at the src it look like its not implemented: https://github.com/databricks/spark-csv/blob/master/src/main/scala/com/databricks/spark/csv/util/TextFile.scala#L34-L36 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Reading-lzo-index-with-spark-

Reading lzo+index with spark-csv (Splittable reads)

Hello, ​ I have managed to speed up the read stage when loading CSV files using the classic "newAPIHadoopFile" method, the issue is that I would like to use the spark-csv package and it seams that its not taking into consideration the LZO Index file / Splittable reads. /# Using the classic method

​Spark 1.6 - YARN Cluster Mode

Hello, This week I have been testing 1.6 (#d509194b) in our HDP 2.3 platform and its been working pretty ok, at the exception of the YARN cluster deployment mode. Note that with 1.5 using the same "spark-props.conf" and "spark-env.sh" config files the cluster mode works as expected. Has anyone el

Re: [Yarn-Client]Can not access SparkUI

Hello Earthson, Is you cluster multihom​ed​? If yes, try setting the variables SPARK_LOCAL_{IP,HOSTNAME} I had this issue before: https://issues.apache.org/jira/browse/SPARK-11147 -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Yarn-Client-Can-not-access-S

Re: Kafka createDirectStream ​issue

Hello, Thanks for all the help on resolving this issue, especially to Cody who guided me to the solution. For other facing similar issues, basically the issue was that I was running Spark Streaming jobs from the spark-shell and this is not supported. Running the same job through spark-submit work

Re: Kafka createDirectStream ​issue

yes, I have two clusters one standalone an another using Mesos Sebastian YEPES http://sebastian-yepes.com On Wed, Jun 24, 2015 at 12:37 AM, drarse [via Apache Spark User List] < ml-node+s1001560n23457...@n3.nabble.com> wrote: > Hi syepes, > Are u run the application in sta

Kafka createDirectStream ​issue

Hello, I ​am trying ​use the new Kafka ​consumer ​​"KafkaUtils.createDirectStream"​ but I am having some issues making it work. I have tried different versions of Spark v1.4.0 and branch-1.4 #8d6e363 and I am still getting the same strange exception "ClassNotFoundException: $line49.$read$$iwC$$i..

Re: spark job progress-style report on console ?

Just add the following line "spark.ui.showConsoleProgress true" do your conf/spark-defaults.conf file. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/spark-job-progress-style-report-on-console-tp22440p22506.html Sent from the Apache Spark User List mailing

EventLog / Timeline calculation - Optimization

Hello, For the past days I have been trying to process and analyse with Spark a Cassandra eventLog table similar to the one shown here. Basically what I want to calculate is the delta time "epoch" between each event type for all the device id's in the table. Currently its working as expected but I