Exception when using HttpSolrServer (httpclient) from within Spark Streaming: java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/conn/sc

Hello, I'm using *Spark 1.1.0* and *Solr 4.10.3*. I'm getting an exception when using *HttpSolrServer* from within Spark Streaming: 15/01/28 13:42:52 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0) java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSyste

Re: Exception when using HttpSolrServer (httpclient) from within Spark Streaming: java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/con

the newer version is incompatible you'll have to > perform some magic with the Maven shade plugin. > > > On Wed Jan 28 2015 at 8:00:22 AM Emre Sevinc > wrote: > >> Hello, >> >> I'm using *Spark 1.1.0* and *Solr 4.10.3*. I'm getting an exceptio

Re: Exception when using HttpSolrServer (httpclient) from within Spark Streaming: java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/con

uced in version 4.3.1 > of httpcomponents so even if by chance one of them did rely on > commons-httpclient there wouldn't be a class conflict. > > > > On Wed Jan 28 2015 at 9:19:20 AM Emre Sevinc > wrote: > >> This is what I get: >> >> ./bigcontent-1.0

Re: Exception when using HttpSolrServer (httpclient) from within Spark Streaming: java.lang.NoSuchMethodError: org.apache.http.impl.conn.SchemeRegistryFactory.createSystemDefault()Lorg/apache/http/con

ude-a-maven-dependency-globally > > (I don't know if a provided dependency will work without a specific > version number so I'm just making a guess here.) > > > On Wed Jan 28 2015 at 11:24:02 AM Emre Sevinc > wrote: > >> When I examine the dependencies again, I

How to define a file filter for file name patterns in Apache Spark Streaming in Java?

Hello, I'm using Apache Spark Streaming 1.2.0 and trying to define a file filter for file names when creating an InputDStream by invoking the fileStream

Re: Spark streaming - tracking/deleting processed files

-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Emre Sevinc

Re: How to define a file filter for file name patterns in Apache Spark Streaming in Java?

Format.class)); > > > ​ > > > Thanks > Best Regards > > On Mon, Feb 2, 2015 at 6:34 PM, Emre Sevinc wrote: > >> Hello, >> >> I'm using Apache Spark Streaming 1.2.0 and trying to define a file filter >> for file names when creating an InputDStre

Re: Get filename in Spark Streaming

Hi All, > > We have filename with timestamp say ABC_1421893256000.txt and the > timestamp needs to be extracted from file name for further processing.Is > there a way to get input file name picked up by spark streaming job? > > Thanks in advance > > Subacini > -- Emre Sevinc

Re: Spark certifications

tSERV Limited has its registered office located at Level 4, Ropemaker > Place, 25 Ropemaker Street, London, EC2Y 9LY and is authorized and > regulated by the Financial Conduct Authority with registration number 207294 > -- Emre Sevinc

How to log using log4j to local file system inside a Spark application that runs on YARN?

Hello, I'm building an Apache Spark Streaming application and cannot make it log to a file on the local filesystem *when running it on YARN*. How can achieve this? I've set log4.properties file so that it can successfully write to a log file in /tmp directory on the local file system (shown below

Re: How to log using log4j to local file system inside a Spark application that runs on YARN?

ooking at it. > > > On Wed, Feb 11, 2015 at 4:29 AM, Emre Sevinc > wrote: > > Hello, > > > > I'm building an Apache Spark Streaming application and cannot make it > log to > > a file on the local filesystem when running it on YARN. How can achieve > &g

Documentation error in MLlib - Clustering?

Hello, I was trying the streaming kmeans clustering example in the official documentation at: http://spark.apache.org/docs/1.2.0/mllib-clustering.html But I've got a type error when I tried to compile the code: [error] found : org.apache.spark.streaming.dstream.DStream[org.apache.spark.ml

Magic number 16: Why doesn't Spark Streaming process more than 16 files?

Hello, I have an application in Java that uses Spark Streaming 1.2.1 in the following manner: - Listen to the input directory. - If a new file is copied to that input directory process it. - Process: contact a RESTful web service (running also locally and responsive), send the contents of the

Re: Magic number 16: Why doesn't Spark Streaming process more than 16 files?

16. -- Emre On Mon, Feb 16, 2015 at 12:56 PM, Emre Sevinc wrote: > Hello, > > I have an application in Java that uses Spark Streaming 1.2.1 in the > following manner: > > - Listen to the input directory. > - If a new file is copied to that input directory process it. &

Re: Magic number 16: Why doesn't Spark Streaming process more than 16 files?

wen wrote: > How are you deciding whether files are processed or not? It doesn't seem > possible from this code. Maybe it just seems so. > On Feb 16, 2015 12:51 PM, "Emre Sevinc" wrote: > >> I've managed to solve this, but I still don't know exactly why m

Re: Magic number 16: Why doesn't Spark Streaming process more than 16 files?

ar on what the real code does and what about the > output of that code tells you only 16 files were processed. > On Feb 16, 2015 1:18 PM, "Emre Sevinc" wrote: > >> Hello Sean, >> >> I did not understand your question very well, but what I do is checking >> t

Can't I mix non-Spark properties into a .properties file and pass it to spark-submit via --properties-file?

Hello, I'm using Spark 1.2.1 and have a module.properties file, and in it I have non-Spark properties, as well as Spark properties, e.g.: job.output.dir=file:///home/emre/data/mymodule/out I'm trying to pass it to spark-submit via: spark-submit --class com.myModule --master local[4] --dep

Re: Can't I mix non-Spark properties into a .properties file and pass it to spark-submit via --properties-file?

Spark > mechanisms for your configuration, and you can use any config > mechanism you like to retain your own properties. > > On Mon, Feb 16, 2015 at 3:26 PM, Emre Sevinc > wrote: > > Hello, > > > > I'm using Spark 1.2.1 and have a module.properties file, and in it I ha

Re: Can't I mix non-Spark properties into a .properties file and pass it to spark-submit via --properties-file?

;; though I'd point you to [3] below if you want to make your > development life easier. > > 1. https://github.com/typesafehub/config > 2. https://github.com/ceedubs/ficus > 3. > http://deploymentzone.com/2015/01/27/spark-ec2-and-easy-spark-shell-deployment/ > > > >

Re: Magic number 16: Why doesn't Spark Streaming process more than 16 files?

? That might give us more details about > what the difference is. > > I can't see how .count.println() would be different than just println(), > but maybe I am missing something also. > > Imran > > On Mon, Feb 16, 2015 at 7:49 AM, Emre Sevinc > wrote: > >> S

Re: Spark Streaming output cannot be used as input?

equires that you delete it without > acting upon or copying any of its contents, and we further request that you > advise us. > > SDL PLC is a public limited company registered in England and Wales. > Registered number: 02675207. > Registered address: Globe House, Clivemont Road, Maidenhead, Berkshire SL6 > 7DY, UK. > > This message has been scanned for malware by Websense. www.websense.com > -- Emre Sevinc

[POWERED BY] Can you add Big Industries to the Powered by Spark page?

Hello, Could you please add Big Industries to the Powered by Spark page at https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark ? Company Name: Big Industries URL: http://http://www.bigindustries.be/ Spark Components: Spark Streaming Use Case: Big Content Platform Summary: The

Re: Re: Problem with 1 master + 2 slaves cluster

On Wed, Feb 18, 2015 at 10:23 AM, bit1...@163.com wrote: > Sure, thanks Akhil. > A further question : Is local file system(file:///) not supported in > standalone cluster? > FYI: I'm able to write to local file system (via HDFS API and using file:/// notation) when using Spark. -- Emre Sevinç

Re: Can't I mix non-Spark properties into a .properties file and pass it to spark-submit via --properties-file?

All(configToStringSeq(config.getConfig("spark").atPath("spark"))) >> >> And we can make use of the config object everywhere else. >> >> We use the override model of the typesafe config: reasonable defaults go >> in the reference.conf (with

Re: Class loading issue, spark.files.userClassPathFirst doesn't seem to be working

er.(HttpSolrServer.java:168) >>> at >>> >>> org.apache.solr.client.solrj.impl.HttpSolrServer.(HttpSolrServer.java:141) >>> ... >>> >>> >>> >>> >>> >>> -- >>> View this message in co

Re: Class loading issue, spark.files.userClassPathFirst doesn't seem to be working

On Wed, Feb 18, 2015 at 4:54 PM, Dmitry Goldenberg wrote: > Thank you, Emre. It seems solrj still depends on HttpClient 4.1.3; would > that not collide with Spark/Hadoop's default dependency on HttpClient set > to 4.2.6? If that's the case that might just solve the problem. > > Would Solrj 4.0.0

In a Spark Streaming application, what might be the potential causes for "util.AkkaUtils: Error sending message in 1 attempts" and "java.util.concurrent.TimeoutException: Futures timed out and"

Hello, We have a Spark Streaming application that watches an input directory, and as files are copied there the application reads them and sends the contents to a RESTful web service, receives a response and write some contents to an output directory. When testing the application by copying a few

Re: In a Spark Streaming application, what might be the potential causes for "util.AkkaUtils: Error sending message in 1 attempts" and "java.util.concurrent.TimeoutException: Futures timed out and"

On Thu, Feb 19, 2015 at 12:27 PM, Tathagata Das wrote: > What version of Spark are you using? > > TD > Spark version is 1.2.0 (running on Cloudera CDH 5.3.0) -- Emre Sevinç

Re: Streaming Linear Regression

--- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Emre Sevinc

Can you add Big Industries to the Powered by Spark page?

Hello, Could you please add Big Industries to the Powered by Spark page at https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark ? Company Name: Big Industries URL: http://http://www.bigindustries.be/ Spark Components: Spark Streaming Use Case: Big Content Platform Summary: The

Re: Streaming Linear Regression

-- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Emre Sevinc

Where to look for potential causes for Akka timeout errors in a Spark Streaming Application?

Hello, We are building a Spark Streaming application that listens to a directory on HDFS, and uses the SolrJ library to send newly detected files to a Solr server. When we put 10.000 files to the directory it is listening to, it starts to process them by sending the files to our Solr server but af

Re: Where to look for potential causes for Akka timeout errors in a Spark Streaming Application?

Todd Nist wrote: > Hi Emre, > > Have you tried adjusting these: > > .set("spark.akka.frameSize", "500").set("spark.akka.askTimeout", > "30").set("spark.core.connection.ack.wait.timeout", "600") > > -Todd >

Re: Get filename in Spark Streaming

it this into Dstream RDD. > > val inputStream = ssc.textFileStream("/hdfs Path/") > > inputStream is Dstreamrdd and in foreachrdd , am doing my processing > > inputStream.foreachRDD(rdd => { >* //how to get filename here??* > }) > > > Can you pleas

Re: Can you add Big Industries to the Powered by Spark page?

I've added it, thanks! > > On Fri, Feb 20, 2015 at 12:22 AM, Emre Sevinc > wrote: > > > > Hello, > > > > Could you please add Big Industries to the Powered by Spark page at > > https://cwiki.apache.org/confluence/display/SPARK/Powered+By+Spark ? > >

Which one is faster / consumes less memory: collect() or count()?

Hello, I have a piece of code to force the materialization of RDDs in my Spark Streaming program, and I'm trying to understand which method is faster and has less memory consumption: javaDStream.foreachRDD(new Function, Void>() { @Override public Void call(JavaRDD stringJavaRDD) thr

Re: Which one is faster / consumes less memory: collect() or count()?

ledge-base/content/best_practices/dont_call_collect_on_a_very_large_rdd.html > > — > FG > > > On Thu, Feb 26, 2015 at 2:28 PM, Emre Sevinc > wrote: > >> Hello, >> >> I have a piece of code to force the materialization of RDDs in my Spark >> Streamin

Re: Which one is faster / consumes less memory: collect() or count()?

, Sean Owen wrote: > Those do quite different things. One counts the data; the other copies > all of the data to the driver. > > The fastest way to materialize an RDD that I know of is > foreachPartition(i => None) (or equivalent no-op VoidFunction in > Java) > > On Thu

Re: Which one is faster / consumes less memory: collect() or count()?

On Thu, Feb 26, 2015 at 4:20 PM, Sean Owen wrote: > Yea we discussed this on the list a short while ago. The extra > overhead of count() is pretty minimal. Still you could wrap this up as > a utility method. There was even a proposal to add some 'materialize' > method to RDD. > I definitely woul

Re: Issues reading in Json file with spark sql

According to Spark SQL Programming Guide: jsonFile - loads data from a directory of JSON files where each line of the files is a JSON object. Note that the file that is offered as jsonFile is not a typical JSON file. Each line must contain a separate, self-contained valid JSON object. As a conseq

Why can't Spark Streaming recover from the checkpoint directory when using a third party library for processingmulti-line JSON?

Hello, I have a Spark Streaming application (that uses Spark 1.2.1) that listens to an input directory, and when new JSON files are copied to that directory processes them, and writes them to an output directory. It uses a 3rd party library to process the multi-line JSON files ( https://github.co

Re: Why can't Spark Streaming recover from the checkpoint directory when using a third party library for processingmulti-line JSON?

n, factory, false); but I still get the same exception. Why doesn't getOrCreate ignore that Hadoop configuration part (which normally works, e.g. when not recovering)? -- Emre On Tue, Mar 3, 2015 at 3:36 PM, Emre Sevinc wrote: > Hello, > > I have a Spark Streaming application (t

Is FileInputDStream returned by fileStream method a reliable receiver?

Is FileInputDStream returned by fileStream method a reliable receiver? In the Spark Streaming Guide it says: "There can be two kinds of data sources based on their *reliability*. Sources (like Kafka and Flume) allow the transferred data to be acknowledged. If the system receiving data from

Re: Why can't Spark Streaming recover from the checkpoint directory when using a third party library for processingmulti-line JSON?

uld you give the > command you used? > > TD > > On Wed, Mar 4, 2015 at 12:42 AM, Emre Sevinc > wrote: > >> I've also tried the following: >> >> Configuration hadoopConfiguration = new Configuration(); >> hadoopConfiguration.set("multiline

Re: Writing Spark Streaming Programs

f programming Scala >>> code can become unreadable, but I do like the fact it seems to be possible >>> to do so much work with so much less code, that's a strong selling point >>> for me. Also it could be that the type of programming done in Spark is best >>> implemented in Scala as FP language, not sure though. >>> >>> The question I would like your good help with is are there any other >>> considerations I need to think about when deciding this? are there any >>> recommendations you can make in regards to this? >>> >>> Regards >>> jk >>> >>> >>> >>> >>> >>> >>> >> > -- Emre Sevinc

Re: Reducing Spark's logging verbosity

Hello Edmon, Does the following help? http://stackoverflow.com/questions/27248997/how-to-suppress-spark-logging-in-unit-tests/2736#2736 -- Emre Sevinç http://www.bigindustries.be On Mar 22, 2015 1:44 AM, "Edmon Begoli" wrote: > Hi, > Does anyone have concrete recommendations how to red

Re: log files of failed task

er-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > -- Emre Sevinc

Why doesn't the --conf parameter work in yarn-cluster mode (but works in yarn-client and local)?

Hello, According to Spark Documentation at https://spark.apache.org/docs/1.2.1/submitting-applications.html : --conf: Arbitrary Spark configuration property in key=value format. For values that contain spaces wrap “key=value” in quotes (as shown). And indeed, when I use that parameter, in my S

Re: Why doesn't the --conf parameter work in yarn-cluster mode (but works in yarn-client and local)?

i Emre, > > The --conf property is meant to work with yarn-cluster mode. > System.getProperty("key") isn't guaranteed, but new SparkConf().get("key") > should. Does it not? > > -Sandy > > On Mon, Mar 23, 2015 at 8:39 AM, Emre Sevinc > wrote: > >> H

Re: Why doesn't the --conf parameter work in yarn-cluster mode (but works in yarn-client and local)?

lue pairs to > the JVM system properties. > > -Sandy > > On Tue, Mar 24, 2015 at 4:25 AM, Emre Sevinc > wrote: > >> Hello Sandy, >> >> Your suggestion does not work when I try it locally: >> >> When I pass >> >> --conf "key

Re: log4j.properties in jar

; > Is it possible to put the log4j.properties in the application jar such > that the driver and the executors use this log4j file. Do I need to specify > anything while submitting my app so that this file is used? > > Thanks, > Udit > -- Emre Sevinc

Re: Query REST web service with Spark?

data, the total number of calls to the service is expected to be > low, so it would be ideal to do the whole job in Spark as we scour the data. > > I don't see anything obvious in the API or on Google relating to making > REST calls from a Spark job. Is it possible? > > Thanks, > > Alec > -- Emre Sevinc

Re: override log4j.properties

override log4j.properties for a specific spark job? > > BR, > Patcharee > > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > > -- Emre Sevinc

Re: Spark Unit Testing

ce that covers an approach (or approaches) for > unit testing using Java. > > Regards > jk > -- Emre Sevinc

How to deal with code that runs before foreach block in Apache Spark?

I'm trying to deal with some code that runs differently on Spark stand-alone mode and Spark running on a cluster. Basically, for each item in an RDD, I'm trying to add it to a list, and once this is done, I want to send this list to Solr. This works perfectly fine when I run the following code in

Re: How to deal with code that runs before foreach block in Apache Spark?

>> SolrIndexerDriver.solrServer.commit >> >> is happening on the driver. >> >> In practical terms, the lists on the executors are being filled-in but >> they are never committed and on the driver the opposite is happening. >> >> -kr, Gerard >>

Re: Building Spark

; resources. Is there any efficient method to build Spark. >> >> Thanks >> Akhil >> > > -- Emre Sevinc

How can I compile only the core and streaming (so that I can get test utilities of streaming)?

Hello, I'm currently developing a Spark Streaming application and trying to write my first unit test. I've used Java for this application, and I also need use Java (and JUnit) for writing unit tests. I could not find any documentation that focuses on Spark Streaming unit testing, all I could find

Re: How can I compile only the core and streaming (so that I can get test utilities of streaming)?

Hello, Specifying '-DskipTests' on commandline worked, though I can't be sure whether first running 'sbt assembly' also contributed to the solution. (I've tried 'sbt assembly' because branch-1.1's README says to use sbt). Thanks for the answer. Kind regards, Emre Sevinç

How can I make Spark Streaming count the words in a file in a unit test?

Hello, I've successfully built a very simple Spark Streaming application in Java that is based on the HdfsCount example in Scala at https://github.com/apache/spark/blob/branch-1.1/examples/src/main/scala/org/apache/spark/examples/streaming/HdfsWordCount.scala . When I submit this application to m

Re: Unit testing and Spark Streaming

On Fri, Dec 12, 2014 at 2:17 PM, Eric Loots wrote: > How can the log level in test mode be reduced (or extended when needed) ? Hello Eric, The following might be helpful for reducing the log messages during unit testing: http://stackoverflow.com/a/2736/236007 -- Emre Sevinç https://be.lin

Re: Tuning Spark Streaming jobs

similar issues. > > http://www.virdata.com/tuning-spark/ > > Your feedback is welcome. > > With kind regards, > > Gerard. > Data Processing Team Lead > Virdata.com > @maasg > > > -- Emre Sevinc

Why does consuming a RESTful web service (using javax.ws.rs.* and Jsersey) work in unit test but not when submitted to Spark?

Hello, I have a piece of code that runs inside Spark Streaming and tries to get some data from a RESTful web service (that runs locally on my machine). The code snippet in question is: Client client = ClientBuilder.newClient(); WebTarget target = client.target("http://localhost:/res

Re: Why does consuming a RESTful web service (using javax.ws.rs.* and Jsersey) work in unit test but not when submitted to Spark?

On Wed, Dec 24, 2014 at 1:46 PM, Sean Owen wrote: > I'd take a look with 'mvn dependency:tree' on your own code first. > Maybe you are including JavaEE 6 for example? > For reference, my complete pom.xml looks like: http://maven.apache.org/POM/4.0.0"; xmlns:xsi=" http://www.w3.org/2001/XMLSchem

Re: Why does consuming a RESTful web service (using javax.ws.rs.* and Jsersey) work in unit test but not when submitted to Spark?

ey in Spark's dependency tree except from HBase tests, > which in turn only appear in examples, so that's unlikely to be it. > I'd take a look with 'mvn dependency:tree' on your own code first. > Maybe you are including JavaEE 6 for example? > > On Wed, Dec 2

Re: Why does consuming a RESTful web service (using javax.ws.rs.* and Jsersey) work in unit test but not when submitted to Spark?

; Your guess is right, that there are two incompatible versions of > >> Jersey (or really, JAX-RS) in your runtime. Spark doesn't use Jersey, > >> but its transitive dependencies may, or your transitive dependencies > >> may. > >> > >> I don't see