date:20140429

Issue during Spark streaming with ZeroMQ source

2014-04-29 Thread Francis . Hu

Hi, all

 

I installed spark-0.9.1 and zeromq 4.0.1 , and then run below example:

 

./bin/run-example org.apache.spark.streaming.examples.SimpleZeroMQPublisher
tcp://127.0.1.1:1234 foo.bar`

./bin/run-example org.apache.spark.streaming.examples.ZeroMQWordCount
local[2] tcp://127.0.1.1:1234 foo`

 

No any message was received in ZeroMQWordCount side. 

 

Does anyone know what the issue is ? 

 

 

Thanks,

Francis

RE: questions about debugging a spark application

2014-04-29 Thread wxhsdp

Hi Liu, 
is it the feature of spark 0.9.1?
my version is 0.9.0, it has no effect when i set spark.eventLog.enabled



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/questions-about-debugging-a-spark-application-tp4891p5028.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Issue during Spark streaming with ZeroMQ source

2014-04-29 Thread Prashant Sharma

Unfortunately zeromq 4.0.1 is not supported.
https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/streaming/examples/ZeroMQWordCount.scala#L63Says
about the version. You will need that version of zeromq to see it
work. Basically I have seen it working nicely with zeromq 2.2.0 and if you
have jzmq libraries installed performance is much better.

Prashant Sharma

On Tue, Apr 29, 2014 at 12:29 PM, Francis.Hu
wrote:

>  Hi, all
>
>
>
> I installed spark-0.9.1 and zeromq 4.0.1 , and then run below example:
>
>
>
> ./bin/run-example
> org.apache.spark.streaming.examples.SimpleZeroMQPublisher *tcp*://
> 127.0.1.1:1234 foo.bar`
>
> ./bin/run-example org.apache.spark.streaming.examples.ZeroMQWordCount
> local[2] *tcp*://127.0.1.1:1234 *foo*`
>
>
>
> No any message was received in ZeroMQWordCount side.
>
>
>
> Does anyone know what the issue is ?
>
>
>
>
>
> Thanks,
>
> Francis
>
>
>

Re: File list read into single RDD

2014-04-29 Thread Christophe Préaud

Hi,

You can also use any path pattern as defined here:
http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html#globStatus%28org.apache.hadoop.fs.Path%29

e.g.:

sc.textFile('{/path/to/file1,/path/to/file2}')

Christophe.

On 29/04/2014 05:07, Nicholas Chammas wrote:
Not that I know of. We were discussing it on another thread and it came up.

I think if you look up the Hadoop FileInputFormat API (which Spark uses) you'll
see it mentioned there in the docs.

http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/mapred/FileInputFormat.html

But that's not obvious.

Nick

2014년 4월 28일 월요일, Pat Ferrelmailto:pat.fer...@gmail.com>>
님이 작성한 메시지:
Perfect.

BTW just so I know where to look next time, was that in some docs?

On Apr 28, 2014, at 7:04 PM, Nicholas Chammas
>
wrote:

Yep, as I just found out, you can also provide sc.textFile() with a
comma-delimited string of all the files you want to load.

For example:

sc.textFile('/path/to/file1,/path/to/file2')

So once you have your list of files, concatenate their paths like that and pass
the single string to textFile().

Nick

On Mon, Apr 28, 2014 at 7:23 PM, Pat Ferrel
>
wrote:
sc.textFile(URI) supports reading multiple files in parallel but only with a
wildcard. I need to walk a dir tree, match a regex to create a list of files,
then I’d like to read them into a single RDD in parallel. I understand these
could go into separate RDDs then a union RDD can be created. Is there a way to
create a single RDD from a URI list?

Kelkoo SAS
Société par Actions Simplifiée
Au capital de € 4.168.964,30
Siège social : 8, rue du Sentier 75002 Paris
425 093 069 RCS Paris

Ce message et les pièces jointes sont confidentiels et établis à l'attention
exclusive de leurs destinataires. Si vous n'êtes pas le destinataire de ce
message, merci de le détruire et d'en avertir l'expéditeur.

65 matches

Mail list logo