Re: Spark Direct Streaming With ZK Updates

2015-08-24 Thread Cody Koeninger
s"${topicDirs.consumerOffsetDir}/${osr.partition}" > ZkUtils.updatePersistentPath(zkClient, zkPath, > osr.untilOffset.toString) > } > } > > Why would there be serialization issues when I'm not trying to pass > ZkClient > to any of the workers? >

Re: Spark Direct Streaming With ZK Updates

2015-08-24 Thread suchenzang
ls.updatePersistentPath(zkClient, zkPath, osr.untilOffset.toString) } } Why would there be serialization issues when I'm not trying to pass ZkClient to any of the workers? Thanks -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Direct-Streami

Re: Spark Direct Streaming With ZK Updates

2015-08-24 Thread Susan Zhang
itoring purposes, that should be fine. > > > On Mon, Aug 24, 2015 at 12:11 PM, suchenzang wrote: > >> Forgot to include the PR I was referencing: >> https://github.com/apache/spark/pull/4805/ >> >> >> >> -- >> View this message in context: &g

Re: Spark Direct Streaming With ZK Updates

2015-08-24 Thread Cody Koeninger
lude the PR I was referencing: > https://github.com/apache/spark/pull/4805/ > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Direct-Streaming-With-ZK-Updates-tp24423p24424.html > Sent from the Apache Spark User Li

Re: Spark Direct Streaming With ZK Updates

2015-08-24 Thread suchenzang
Forgot to include the PR I was referencing: https://github.com/apache/spark/pull/4805/ -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Direct-Streaming-With-ZK-Updates-tp24423p24424.html Sent from the Apache Spark User List mailing list archive at

Spark Direct Streaming With ZK Updates

2015-08-24 Thread suchenzang
missing something about ZK here, please let me know too. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Direct-Streaming-With-ZK-Updates-tp24423.html Sent from the Apache Spark User