et this working.
>>>
>>> @Stefan or @Stephan : can you please help in resolving this issue
>>>
>>> Regards,
>>> Vinay Patil
>>>
>>> On Thu, Jun 29, 2017 at 6:01 PM, gerryzhou [via Apache Flink User Mailing
>>> List archive.]
>> Hi, Vinay,
>>>> I observed a similar problem in flink 1.3.0 with rocksdb. I wonder
>>>> how to use FRocksDB as you mentioned above. Thanks.
>>>>
>>>> --
>>>> If you reply to this email, your message
a similar problem in flink 1.3.0 with rocksdb. I wonder
>>> how to use FRocksDB as you mentioned above. Thanks.
>>>
>>> --
>>> If you reply to this email, your message will be added to the discussion
>>> below:
>>> http
ailing
>> List archive.] > <mailto:ml+s2336050n1406...@n4.nabble.com>> wrote:
>> Hi, Vinay,
>> I observed a similar problem in flink 1.3.0 with rocksdb. I wonder how
>> to use FRocksDB as you mentioned above. Thanks.
>>
>> If you reply to th
ioned above. Thanks.
>>
>> --
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend
sdb. I wonder how
> to use FRocksDB as you mentioned above. Thanks.
>
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p14063.
d above. Thanks.
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p14063.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at
> Nabble.com.
e. Thanks.
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-
> Checkpointing-with-RocksDB-as-statebackend-tp11752p14063.html
> To start
Hi, Vinay,
I observed a similar problem in flink 1.3.0 with rocksdb. I wonder how
to use FRocksDB as you mentioned above. Thanks.
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend
disk.
>>> >
>>> > I have attached the snapshot for reference.
>>> >
>>> > Also the data processed till now is only 17GB and above 120GB memory is
>>> > getting used.
>>> >
>>> > Is there any change wrt RocksDB configurations
>>> >
>>> > <http://apache-flink-user-mailing-list-archive.2336050.n4.na
>>> bble.com/file/n14013/TM_Memory_Usage.png>
>>> >
>>> > Regards,
>>> > Vinay Patil
>>> >
>>> >
>>> >
>>> > --
>>> > View this message in context: http://apache-flink-user-maili
>>> ng-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-
>>> RocksDB-as-statebackend-tp11752p14013.html
>>> > Sent from the Apache Flink User Mailing List archive. mailing list
>>> archive at Nabble.com.
>>>
>>>
>>
>
> Also the data processed till now is only 17GB and above 120GB memory is
>> > getting used.
>> >
>> > Is there any change wrt RocksDB configurations
>> >
>> > <http://apache-flink-user-mailing-list-archive.2336050.n4.na
>> bble.com/file/n14013/TM_Mem
4013/TM_Memory_Usage.png>
> >
> > Regards,
> > Vinay Patil
> >
> >
> >
> > --
> > View this message in context: http://apache-flink-user-maili
> ng-list-archive.2336050.n4.nabble.com/Re-Checkpointing-
> with-RocksDB-as-statebackend-tp11752p14013.html
> > Sent from the Apache Flink User Mailing List archive. mailing list
> archive at Nabble.com.
>
>
3/TM_Memory_Usage.png>
>
>
> Regards,
> Vinay Patil
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p14013.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at
> Nabble.com.
RocksDB configurations
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n14013/TM_Memory_Usage.png>
Regards,
Vinay Patil
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-stateb
;>
>>>>>>>>>>> Best,
>>>>>>>>>>> Stefan
>>>>>>>>>>>
>>>>>>>>>>> Am 14.03.2017 um 15:31 schrieb Vishnu Viswanath <[hidden email
t would cause a
>> FileNotFound exception which would fail the checkpoint.
>>
>>
>>
>> Stephan,
>>
>>
>>
>> Currently my aws fork contains some very specific assumptions about the
>> pipeline that will in general only hold for my pipeline. Th
>>>>
>>>>>>>>> A certain access pattern in RocksDB starts being so slow after a
>>>>>>>>> certain size-per-key that it basically brings down the streaming
>>>>>>>>> program
>>>&g
ings,
>>>>>>>> Stephan
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 1, 2017 at 12:10 PM, Stephan Ewen <[hidden email]
>>>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&
;>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Vinay Patil
>>>>>>>>>
>>&
gt;>>> Stephan
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Mar 1, 2017 at 12:10 PM, Stephan Ewen <[hidden email]
>>>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&i=3>> wrote:
>>>>&
;
>>>>>>>> @vinay Can you try to not set the buffer timeout at all? I am
>>>>>>>> actually not sure what would be the effect of setting it to a negative
>>>>>>>> value, that can be a cause of problems...
>>>>>&g
gt; On Mon, Feb 27, 2017 at 7:44 PM, Seth Wiesman <[hidden email]
>>>>>>> <http:///user/SendEmail.jtp?type=node&node=12209&i=4>> wrote:
>>>>>>>
>>>>>>>> Vinay,
>>>>>>>>
>>>>>>&
;>>
>>>>>>> Vinay,
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> The bucketing sink performs rename operations during the checkpoint
>>>>>>> and if it tries to rename
>>>>>>
>>>>>>
>>>>>>
>>>>>> Currently my aws fork contains some very specific assumptions about
>>>>>> the pipeline that will in general only hold for my pipeline. This is
>>>>>> because there we
>>>>
>>>>>
>>>>>
>>>>> Currently my aws fork contains some very specific assumptions about
>>>>> the pipeline that will in general only hold for my pipeline. This is
>>>>> because there were still some
;>> pipeline that will in general only hold for my pipeline. This is because
>>>> there were still some open questions that I had about how to solve
>>>> consistency issues in the general case. I will comment on the Jira issue
>>>> with more specific.
>>
to:vinay18.pa...@gmail.com>>
> Reply-To: "user@flink.apache.org <mailto:user@flink.apache.org>"
> mailto:user@flink.apache.org>>
> Date: Monday, February 27, 2017 at 1:05 PM
> To: "user@flink.apache.org <mailto:user@flink.apache.org>"
>
he
>>> pipeline that will in general only hold for my pipeline. This is because
>>> there were still some open questions that I had about how to solve
>>> consistency issues in the general case. I will comment on the Jira issue
>>> with more specific.
>>
many minutes to
>> rename which would stall the entire pipeline. The only viable solution I
>> could find was to write a custom sink which understands S3. Each writer
>> will write file locally and then copy it to S3 on checkpoint. By only
>> interacting with S3 once per file it c
gt;
>
> *From: *vinay patil
> *Reply-To: *"user@flink.apache.org"
> *Date: *Monday, February 27, 2017 at 1:05 PM
> *To: *"user@flink.apache.org"
>
> *Subject: *Re: Checkpointing with RocksDB as statebackend
>
>
>
> Hi Seth,
>
> Thank you fo
;user@flink.apache.org"
Date: Monday, February 27, 2017 at 1:05 PM
To: "user@flink.apache.org"
Subject: Re: Checkpointing with RocksDB as statebackend
Hi Seth,
Thank you for your suggestion.
But if the issue is only related to S3, then why does this happen when I
replace the S3 sink
>> *From: *vinay patil <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=11943&i=1>>
>> *Reply-To: *"[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=11943&i=2>" <[hidden email]
>> <http:///user/SendEmail.jt
By only
> interacting with S3 once per file it can circumvent consistency issues all
> together.
>
>
>
> Hope this helps,
>
>
>
> Seth Wiesman
>
>
>
> *From: *vinay patil
> *Reply-To: *"user@flink.apache.org"
> *Date: *Saturday, February 25, 2017 at
,
Seth Wiesman
From: vinay patil
Reply-To: "user@flink.apache.org"
Date: Saturday, February 25, 2017 at 10:50 AM
To: "user@flink.apache.org"
Subject: Re: Checkpointing with RocksDB as statebackend
HI Stephan,
Just to avoid the confusion here, I am using S3 sink for writing
next
>>>>> 16minutes the
>>>>> pipeline is stuck , I don't see any progress beyond 15M because of
>>>>> checkpoints getting failed consistently.
>>>>>
>>>>> <http://apache-flink-user-mailing-list-archive.2336050.n
t;>> 3 it
>>>> is stuck at the Kafka source after 50%
>>>> (The data sent till now by Kafka source 1 is 65GB and sent by source 2
>>>> is
>>>> 15GB )
>>>>
>>>> Within 10minutes 15M records were processed, and for the nex
15GB )
>>>
>>> Within 10minutes 15M records were processed, and for the next 16minutes
>>> the
>>> pipeline is stuck , I don't see any progress beyond 15M because of
>>> checkpoints getting failed consistently.
>>>
>>> <http://apach
336050.n4.
>> nabble.com/file/n11882/Checkpointing_Failed.png>
>>
>>
>>
>> --
>> View this message in context: http://apache-flink-user-maili
>> ng-list-archive.2336050.n4.nabble.com/Re-Checkpointing-
>> with-RocksDB-as-statebackend-tp11752p11882.
ly.
>
> <http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/file/n11882/Checkpointing_Failed.png>
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Re-
> Checkpointing-with-RocksDB
pache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11882.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
y are done asynchronously.
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11879.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
--
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-
>> tp11752p11831.html
>> To start a new topic under Apac
ally distributed among all TM's.
>> Why does this happen ?
>>
>> --
>> If you reply to this email, your message will be added to the discussion
>> below:
>> http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/
x27;s.
> Why does this happen ?
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-
> Checkpointing-with-RocksDB-as-statebackend-tp11752p11831.html
> To start a new topic
TM's.
Why does this happen ?
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11831.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
> Regards,
> Vinay Patil
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/Re-
> Checkpointing-with-RocksDB-as-statebackend-tp11752p11799.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>
)
Regards,
Vinay Patil
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11799.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
DB to store
>>>>>> necessary indices. To avoid the unlimited growth in the memory
>>>>>> consumption, you can put these indices into block cache (set
>>>>>> CacheIndexAndFilterBlock to true) and properly set the block cache size.
>>
FilterBlock
>>>>> to
>>>>> true) and properly set the block cache size.
>>>>>
>>>>> You can also increase the number of backgroud threads to improve the
>>>>> performance of flushes and compactions (via MaxBackgroundFlu
et the block cache size.
>>>>
>>>> You can also increase the number of backgroud threads to improve the
>>>> performance of flushes and compactions (via MaxBackgroundFlushes and
>>>> MaxBackgroudCompactions).
>>>>
>>>> In YARN clusters, task managers will be killed if their memory
>>>> util
age in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752p11759.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
does not count the
>>> memory used by RocksDB in the allocation. We are working on fine-grained
>>> resource allocation (see FLINK-5131). It may help to avoid such problems.
>>>
>>> May the information helps you.
>>>
>>> Regards,
>>> Xia
t;
>> May the information helps you.
>>
>> Regards,
>> Xiaogang
>>
>>
>> --------------
>> 发件人：Vinay Patil <[hidden email]
>> <http:///user/SendEmail.jtp?type=node&node=11731&i=0>
jtp?type=node&node=11731&i=0>>
> 发送时间：2017年2月17日(星期五) 21:19
> 收件人：user <[hidden email]
> <http:///user/SendEmail.jtp?type=node&node=11731&i=1>>
> 主 题：Re: Checkpointing with RocksDB as statebackend
>
> Hi Guys,
>
> There seems to be some issue w
rchive.2336050.n4.nabble.com/file/n11640/checkpointing_issue.PNG>
>
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Checkpointing
any acknowledgements, now during this
>>> 10minutes
>>> > the entire pipeline did not make any progress and no data was getting
>>> > processed. (For Ex : In 13minutes 20M records were processed and when
>>> the
>>> > checkpoint took time the
line did not make any progress and no data was getting
>> > processed. (For Ex : In 13minutes 20M records were processed and when
>> the
>> > checkpoint took time there was no progress for the next 10minutes)
>> >
>> > I have even tried to set max checkpoint time
; >
> > P.S. I am writing to 3 S3 sinks
> >
> > checkpointing_issue.PNG
> > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n11640/checkpointing_issue.PNG
> >
> > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.
s 20M records were processed and when
>> the
>> > checkpoint took time there was no progress for the next 10minutes)
>> >
>> > I have even tried to set max checkpoint timeout to 3min, but in that
>> case as
>> > well multiple checkpoints were gett
> >
> > I have set RocksDB FLASH_SSD_OPTION
> > What could be the issue ?
> >
> > P.S. I am writing to 3 S3 sinks
> >
> > checkpointing_issue.PNG
> > <
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n11640/checkpointing
gt;> > P.S. I am writing to 3 S3 sinks
>> >
>> > checkpointing_issue.PNG
>> > <http://apache-flink-user-mailing-list-archive.2336050.n4.
>> nabble.com/file/n11640/checkpointing_issue.PNG>
>> >
>> >
>> >
>> > --
>>
> >
> > checkpointing_issue.PNG
> > <http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/file/n11640/checkpointing_issue.PNG>
> >
> >
> >
> > --
> > View this message in context: http://apache-flink-user-
> mailing-li
gt; View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Checkpointing-with-RocksDB-as-statebackend-tp11640.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at
> Nabble.com.
.n4.nabble.com/file/n11640/checkpointing_issue.PNG>
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Checkpointing-with-RocksDB-as-statebackend-tp11640.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
64 matches
Mail list logo
