}
>
>
> Best Regards,
> Shixiong Zhu
>
> 2015-09-24 17:42 GMT+08:00 Bin Wang :
>
>> It seems like a work around. But I don't know how to get the database
>> connection from the working nodes.
>>
>> Shixiong Zhu 于2015年9月24日周四 下午5:37写道：
>>
}
>
> and use this overload:
>
> def updateStateByKey[S: ClassTag](
> updateFunc: (Seq[V], Option[S]) => Option[S],
> partitioner: Partitioner
> ): DStream[(K, S)]
>
> There is a JIRA: https://issues.apache.org/jira/browse/SPARK-2629 but
> doesn't
livered soon.
>
> Best Regards,
> Shixiong Zhu
>
> 2015-09-24 13:45 GMT+08:00 Bin Wang :
>
>> I've read the source code and it seems to be impossible, but I'd like to
>> confirm it.
>>
>> It is a very useful feature. For example, I need to store the state
I've read the source code and it seems to be impossible, but I'd like to
confirm it.
It is a very useful feature. For example, I need to store the state of
DStream into my database, in order to recovery them from next redeploy. But
I only need to save the updated ones. Save all keys into database
vide the logs on when and how you are seeing this error?
>
> On Wed, Sep 23, 2015 at 6:32 PM, Bin Wang wrote:
>
>> BTW, I just kill the application and restart it. Then the application
>> cannot recover from checkpoint because of some lost of RDD. So I'm wonder,
&
BTW, I just kill the application and restart it. Then the application
cannot recover from checkpoint because of some lost of RDD. So I'm wonder,
if there are some failure in the application, won't it possible not be able
to recovery from checkpoint?
Bin Wang 于2015年9月23日周三 下午6:58写道：
&g
I find the checkpoint directory structure is like this:
-rw-r--r-- 1 root root 134820 2015-09-23 16:55
/user/root/checkpoint/checkpoint-144299850
-rw-r--r-- 1 root root 134768 2015-09-23 17:00
/user/root/checkpoint/checkpoint-144299880
-rw-r--r-- 1 root root 134895 2015-0
Azuryy Yu
>
>
>
> On Sep 22, 2015, at 13:36, Bin Wang wrote:
>
> However I find some scripts in dev/audit-release, can I use them?
>
> Bin Wang 于2015年9月22日周二 下午1:34写道：
>
>> No, I mean push spark to my private repository. Spark don't have a
>> build.sbt
However I find some scripts in dev/audit-release, can I use them?
Bin Wang 于2015年9月22日周二 下午1:34写道：
> No, I mean push spark to my private repository. Spark don't have a
> build.sbt as far as I see.
>
> Fengdong Yu 于2015年9月22日周二 下午1:29写道：
>
>> Do you mean you want to pu
ld.sb:
>
> publishTo := {
> val nexus = "https://YOUR_PRIVATE_REPO_HOSTS/";
> if (version.value.endsWith("SNAPSHOT"))
> Some("snapshots" at nexus + "content/repositories/snapshots")
> else
> Some("releases" at
f exactly what is in that build readily available,
> not just somewhat arbitrary JARs.
>
> On Mon, Sep 21, 2015 at 9:57 PM, Bin Wang wrote:
>
>> But I cannot find 1.5.1-SNAPSHOT either at
>> https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-core
APSHOT -- soon to be 1.5.1 release
> candidates and then the 1.5.1 release.
>
> On Mon, Sep 21, 2015 at 9:51 PM, Bin Wang wrote:
>
>> I'd like to use some important bug fixes in 1.5 branch and I look for the
>> apache maven host, but don't find any snapshot for 1.5
I'd like to use some important bug fixes in 1.5 branch and I look for the
apache maven host, but don't find any snapshot for 1.5 branch.
https://repository.apache.org/content/groups/snapshots/org/apache/spark/spark-core_2.10/1.5.0-SNAPSHOT/
I can find 1.4.X and 1.6.0 versions, why there is no snap
Never mind. I've found a PR and it merged:
https://github.com/apache/spark/pull/8624/commits
Bin Wang 于2015年9月17日周四 下午4:50写道：
> I'm using spark streaming and use updateStateByKey, which forced to use
> checkpoint. In my unit test, I create a queueStream to test. But in spark
&g
I'm using spark streaming and use updateStateByKey, which forced to use
checkpoint. In my unit test, I create a queueStream to test. But in spark
1.5, QueueStream will throw an exception while use it with checkpoint, it
makes difficult to do unit test. Is there an option to disable this? Though
I k
Hungary, 2475 Kápolnásnyék, Kossuth 6/a
>
> elte: HSKSJZ (ZVZOAAI.ELTE)
>
> 2015-03-24 11:55 GMT+01:00 Arush Kharbanda :
>
>> The block size is configurable and that way I think you can reduce the
>> block interval, to keep the block in memory only for the limiter interv
o keep all the batch data in the memory. Something
like a pipeline should be OK.
Is it difficult to implement on top of the current implementation?
Thanks.
---
Bin Wang
17 matches
Mail list logo
