Unsubscribe

Unsubscribe

Re: Spark SQL query for List

arcs31> On Tue, Apr 26, 2016 at 4:41 PM, Hyukjin Kwon wrote: > Doesn't get(0) give you the Array[String] for CITY (am I missing > something?) > On 26 Apr 2016 11:02 p.m., "Ramkumar V" wrote: > > JavaSparkContext ctx = new JavaSparkContext(sparkConf); >

Re: Spark SQL query for List

din.com/in/ramkumarcs31> On Tue, Apr 26, 2016 at 3:48 PM, Hyukjin Kwon wrote: > Could you maybe share your codes? > On 26 Apr 2016 9:51 p.m., "Ramkumar V" wrote: > >> Hi, >> >> I had loaded JSON file in parquet format into SparkSQL. I can't

Spark SQL query for List

Hi, I had loaded JSON file in parquet format into SparkSQL. I can't able to read List which is inside JSON. Sample JSON { "TOUR" : { "CITIES" : ["Paris","Berlin","Prague"] }, "BUDJET" : 100 } I want to read value of CITIES. *Thanks*,

Re: Read Parquet in Java Spark

HI, Any idea on this ? *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Mon, Apr 4, 2016 at 2:47 PM, Akhil Das wrote: > I wasn't knowing you have a parquet file containing json data. > > Thanks > Best Regards > > On Mon, Apr 4, 2016 at 2:44 PM, Ramku

Re: Read Parquet in Java Spark

ng() Operation) > > Thanks > Best Regards > > On Mon, Apr 4, 2016 at 12:02 PM, Ramkumar V > wrote: > >> Any idea on this ? How to convert parquet file into JavaRDD ? >> >> *Thanks*, >> <https://in.linkedin.com/in/ramkumarcs31> >> >> >&g

Re: Read Parquet in Java Spark

Any idea on this ? How to convert parquet file into JavaRDD ? *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Thu, Mar 31, 2016 at 4:30 PM, Ramkumar V wrote: > Hi, > > Thanks for the reply. I tried this. It's returning JavaRDD instead > of JavaRDD. How to

Re: Read Parquet in Java Spark

On Thu, Mar 31, 2016 at 2:57 PM, UMESH CHAUDHARY wrote: > From Spark Documentation: > > DataFrame parquetFile = sqlContext.read().parquet("people.parquet"); > > JavaRDD jRDD= parquetFile.javaRDD() > > javaRDD() method will convert the DF to RDD > > On Thu, Mar 3

Read Parquet in Java Spark

Hi, I'm trying to read parquet log files in Java Spark. Parquet log files are stored in hdfs. I want to read and convert that parquet file into JavaRDD. I could able to find Sqlcontext dataframe api. How can I read if it is sparkcontext and rdd ? what is the best way to read it ? *Thanks*,

Re: Spark with MapDB

/in/ramkumarcs31> On Tue, Dec 8, 2015 at 5:30 PM, Jörn Franke wrote: > You may want to use a bloom filter for this, but make sure that you > understand how it works > > On 08 Dec 2015, at 09:44, Ramkumar V wrote: > > Im running spark batch job in cluster mode every hour and it runs

Re: Spark with MapDB

qlContext.read.orc(“your previous batch data > path”).select($”uniq_key”).collect > val broadCast = sc.broadCast(keys) > > val rdd = your_current_batch_data > rdd.filter( line => line.key not in broadCase.value) > > > > > > > On Dec 8, 2015, at 4:44 PM, Ramkumar

Re: Spark with MapDB

wrote: > Can you detail your question? what looks like your previous batch and the > current batch? > > > > > > On Dec 8, 2015, at 3:52 PM, Ramkumar V wrote: > > Hi, > > I'm running java over spark in cluster mode. I want to apply filter on > javaRDD bas

Spark with MapDB

Hi, I'm running java over spark in cluster mode. I want to apply filter on javaRDD based on some previous batch values. if i store those values in mapDB, is it possible to apply filter during the current batch ? *Thanks*,

Re: Distinct on key-value pair of JavaRDD

I thought some specific function would be there but I'm using reducebykey now. Its working fine. Thanks a lot. *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Tue, Nov 17, 2015 at 6:21 PM, ayan guha wrote: > How about using reducebykey? > On 17 Nov 2015 22:00, &q

Distinct on key-value pair of JavaRDD

Hi, I have JavaRDD. I would like to do distinct only on key but the normal distinct applies on both key and value. i want to apply only on key. How to do that ? Any help is appreciated. *Thanks*,

Re: Exception while reading from kafka stream

m call. > Then you can use hasOffsetRanges to see what the topic for a given > partition is. > > On Mon, Nov 2, 2015 at 7:26 AM, Ramkumar V > wrote: > >> if i try like below code snippet , it shows exception , how to avoid this >> exception ? how to switch processing

Re: Exception while reading from kafka stream

arkDriver@10.125.4.200:34251/user/CoarseGrainedScheduler] swallowing exception during message send (akka.remote.RemoteTransportExceptionNoStackTrace) *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Fri, Oct 30, 2015 at 7:34 PM, Cody Koeninger wrote: > Just put them all in one s

Re: Exception while reading from kafka stream

you cannot use in such way. Instead you could run multiple streaming > applications, since you're using Yarn. > > 2015年10月30日星期五，Ramkumar V 写道： > >> I found NPE is mainly because of im using the same JavaStreamingContext >> for some other kafka stream. if i change the

Re: Exception while reading from kafka stream

In general , i need to consume five different type of logs from kafka in spark. I have different set of topics for each log. How to start five different stream in spark ? *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Fri, Oct 30, 2015 at 4:40 PM, Ramkumar V wrote: > I fou

Re: Exception while reading from kafka stream

start, unless some place changes it's value in > the runtime that makes it null, but I cannot image how this happened. Maybe > you could add some logs around the place where exception happens if you > could reproduce it. > > On Fri, Oct 30, 2015 at 5:31 PM, Ramkumar V > wro

Re: Exception while reading from kafka stream

gt; On Fri, Oct 30, 2015 at 5:13 PM, Ramkumar V > wrote: > >> No, i dont have any special settings. if i keep only reading line in my >> code, it's throwing NPE. >> >> *Thanks*, >> <https://in.linkedin.com/in/ramkumarcs31> >> >> >>

Re: Exception while reading from kafka stream

will > incur NPE at that place. > > On Fri, Oct 30, 2015 at 4:32 PM, Ramkumar V > wrote: > >> spark version - spark 1.4.1 >> >> my code snippet: >> >> String brokers = "ip:port,ip:port"; >> String topics = "x,y,z"; >> Ha

Re: Exception while reading from kafka stream

n Fri, Oct 30, 2015 at 1:57 PM, Saisai Shao wrote: > What Spark version are you using, also a small code snippet of how you use > Spark Streaming would be greatly helpful. > > On Fri, Oct 30, 2015 at 3:57 PM, Ramkumar V > wrote: > >> I can able to read and print fe

Re: Exception while reading from kafka stream

I can able to read and print few lines. Afterthat i'm getting this exception. Any idea for this ? *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Thu, Oct 29, 2015 at 6:14 PM, Ramkumar V wrote: > Hi, > > I'm trying to read from kafka stream and printing it t

Exception while reading from kafka stream

Hi, I'm trying to read from kafka stream and printing it textfile. I'm using java over spark. I dont know why i'm getting the following exception. Also exception message is very abstract. can anyone please help me ? Log Trace : 15/10/29 12:15:09 ERROR scheduler.JobScheduler: Error in job genera

Saving offset while reading from kafka

Hi, I had written spark streaming application using kafka stream and its writing to hdfs for every hour(batch time). I would like to know how to get offset or commit offset of kafka stream while writing to hdfs so that if there is any issue or redeployment, i'll start from the point where i did a

Conf setting for Java Spark

Hi, I'm using java over spark for processing 30 GB of data every hour. I'm doing spark-submit in cluster mode. I have a cluster of 11 machines (9 - 64 GB memory and 2 - 32 GB memory ) but it takes 30 mins to process 30 GB of data every hour. How can i optimize this ? How to compute the driver and

Re: OutOfMemoryError

How to increase the Xmx of the workers ? *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Mon, Oct 5, 2015 at 3:48 PM, Ramkumar V wrote: > No. I didn't try to increase xmx. > > *Thanks*, > <https://in.linkedin.com/in/ramkumarcs31> > > > On Mon,

Re: OutOfMemoryError

No. I didn't try to increase xmx. *Thanks*, <https://in.linkedin.com/in/ramkumarcs31> On Mon, Oct 5, 2015 at 1:36 PM, Jean-Baptiste Onofré wrote: > Hi Ramkumar, > > did you try to increase Xmx of the workers ? > > Regards > JB > > On 10/05/2015 08:56 AM, Ra

OutOfMemoryError

Hi, When i submit java spark job in cluster mode, i'm getting following exception. *LOG TRACE :* INFO yarn.ExecutorRunnable: Setting up executor with commands: List({{JAVA_HOME}}/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms1024m, -Xmx1024m, -Djava.io.tmpdir={{PWD}}/tmp, '-Dspark.ui

Re: Combine key-value pair in spark java

ng > > On Wed, Sep 30, 2015 at 10:34 AM, Ramkumar V > wrote: > >> Hi, >> >> I have key value pair of JavaRDD (JavaPairRDD rdd) but i >> want to concatenate into one RDD String (JavaRDD result ). >> >> How can i do that ? What i have to use (map,flat

Combine key-value pair in spark java

Hi, I have key value pair of JavaRDD (JavaPairRDD rdd) but i want to concatenate into one RDD String (JavaRDD result ). How can i do that ? What i have to use (map,flatmap)? can anyone please give me the syntax for this in java ? *Thanks*,

GroupBy Java objects in Java Spark

Hi, I want to know whether grouping by java class objects is possible or not in java Spark. I have Tuple2< JavaObject, JavaObject>. i want to groupbyKey and then i'll do some operations in values after grouping. *Thanks*,

Re: Spark Streaming failing on YARN Cluster

k the yarn app logs ? You can use > command "yarn logs -" to get the yarn app logs. > > > > On Thu, Aug 20, 2015 at 1:15 AM, Ramkumar V > wrote: > >> I'm getting some spark exception. Please look this log trace ( >> *http://pastebin.com/xL9jaRUa >

Re: Spark Streaming failing on YARN Cluster

> It looks like you are having issues with the files getting distributed to > the cluster. What is the exception you are getting now? > > > On Wednesday, August 19, 2015, Ramkumar V wrote: > >> Thanks a lot for your suggestion. I had modified HADOOP_CONF_DIR in >> spark-

Re: Spark Streaming failing on YARN Cluster

CONF_DIR is the environment variable point to the hadoop conf > directory. Not sure how CDH organize that, make sure core-site.xml is > under HADOOP_CONF_DIR. > > On Wed, Aug 19, 2015 at 4:06 PM, Ramkumar V > wrote: > >> We are using Cloudera-5.3.1. since it is one of the earlie

Re: Spark Streaming failing on YARN Cluster

9/spark.py > > 15/08/14 10:48:50 INFO yarn.Client: Uploading resource > file:/Users/abc/github/spark/python/lib/pyspark.zip -> hdfs:// > 0.0.0.0:9000/user/abc/.sparkStaging/application_1439432662178_0019/pyspark.zip > > On Thu, Aug 13, 2015 at 2:50 PM, Ramkumar V > wrote: > >

Re: Spark Streaming failing on YARN Cluster

17 PM, Akhil Das wrote: > Just make sure this file is available: > > appattempt_1437639737006_3808_02 exited with exitCode: -1000 due to: > File *file:/home/hdfs/spark-1.4.1/python/lib/pyspark.zip* does not exist > > Thanks > Best Regards > > On Thu, Aug 13, 201

Spark Streaming failing on YARN Cluster

t, it gives the following error. Log link : *http://pastebin.com/kfyVWDGR <http://pastebin.com/kfyVWDGR>* How to fix this issue ? Please help me if i'm doing wrong. *Thanks*, Ramkumar V