Hi Vinay! Can you start by giving us a bit of an environment spec?
- What Flink version are you using? - What is your rough topology (what operations does the program use) - Where is the state (windows, keyBy)? - What is the rough size of your checkpoints and where does the time go? Can you attach a screenshot from https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/checkpoint_monitoring.html - What is the size of the JVM? Those things would be helpful to know... Best, Stephan On Mon, Feb 20, 2017 at 7:04 PM, vinay patil <vinay18.pa...@gmail.com> wrote: > Hi Xiaogang, > > Thank you for your inputs. > > Yes I have already tried setting MaxBackgroundFlushes and > MaxBackgroundCompactions to higher value (tried with 2, 4, 8) , still not > getting expected results. > > System.getProperty("java.io.tmpdir") points to /tmp but there I could not > find RocksDB logs, can you please let me know where can I find it ? > > Regards, > Vinay Patil > > On Mon, Feb 20, 2017 at 7:32 AM, xiaogang.sxg [via Apache Flink User > Mailing List archive.] <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=11752&i=0>> wrote: > >> Hi Vinay >> >> Can you provide the LOG file in RocksDB? It helps a lot to figure out >> the problems becuse it records the options and the events happened >> during the execution. Otherwise configured, it should locate at the path >> set in System.getProperty("java.io.tmpdir"). >> >> Typically, a large amount of memory is consumed by RocksDB to store >> necessary indices. To avoid the unlimited growth in the memory consumption, >> you can put these indices into block cache (set CacheIndexAndFilterBlock to >> true) and properly set the block cache size. >> >> You can also increase the number of backgroud threads to improve the >> performance of flushes and compactions (via MaxBackgroundFlushes and >> MaxBackgroudCompactions). >> >> In YARN clusters, task managers will be killed if their memory >> utilization exceeds the allocation size. Currently Flink does not count the >> memory used by RocksDB in the allocation. We are working on fine-grained >> resource allocation (see FLINK-5131). It may help to avoid such problems. >> >> May the information helps you. >> >> Regards, >> Xiaogang >> >> >> ------------------------------------------------------------------ >> 发件人：Vinay Patil <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=11731&i=0>> >> 发送时间：2017年2月17日(星期五) 21:19 >> 收件人：user <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=11731&i=1>> >> 主 题：Re: Checkpointing with RocksDB as statebackend >> >> Hi Guys, >> >> There seems to be some issue with RocksDB memory utilization. >> >> Within few minutes of job run the physical memory usage increases by 4-5 >> GB and it keeps on increasing. >> I have tried different options for Max Buffer Size(30MB, 64MB, 128MB , >> 512MB) and Min Buffer to Merge as 2, but the physical memory keeps on >> increasing. >> >> According to RocksDB documentation, these are the main options on which >> flushing to storage is based. >> >> Can you please point me where am I doing wrong. I have tried different >> configuration options but each time the Task Manager is getting killed >> after some time :) >> >> Regards, >> Vinay Patil >> >> On Thu, Feb 16, 2017 at 6:02 PM, Vinay Patil <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=11731&i=2>> wrote: >> I think its more of related to RocksDB, I am also not aware about RocksDB >> but reading the tuning guide to understand the important values that can be >> set >> >> Regards, >> Vinay Patil >> >> On Thu, Feb 16, 2017 at 5:48 PM, Stefan Richter [via Apache Flink User >> Mailing List archive.] <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=11731&i=3>> wrote: >> What kind of problem are we talking about? S3 related or RocksDB related. >> I am not aware of problems with RocksDB per se. I think seeing logs for >> this would be very helpful. >> >> Am 16.02.2017 um 11:56 schrieb Aljoscha Krettek <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=11673&i=0>>: >> >> [hidden email] <http:///user/SendEmail.jtp?type=node&node=11673&i=1> and >> [hidden >> email] <http:///user/SendEmail.jtp?type=node&node=11673&i=2> could this >> be the same problem that you recently saw when working with other people? >> >> On Wed, 15 Feb 2017 at 17:23 Vinay Patil <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=11673&i=3>> wrote: >> Hi Guys, >> >> Can anyone please help me with this issue >> >> Regards, >> Vinay Patil >> >> On Wed, Feb 15, 2017 at 6:17 PM, Vinay Patil <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=11673&i=4>> wrote: >> Hi Ted, >> >> I have 3 boxes in my pipeline , 1st and 2nd box containing source and s3 >> sink and the 3rd box is window operator followed by chained operators and a >> s3 sink >> >> So in the details link section I can see that that S3 sink is taking time >> for the acknowledgement and it is not even going to the window operator >> chain. >> >> But as shown in the snapshot ,checkpoint id 19 did not get any >> acknowledgement. Not sure what is causing the issue >> >> Regards, >> Vinay Patil >> >> On Wed, Feb 15, 2017 at 5:51 PM, Ted Yu [via Apache Flink User Mailing >> List archive.] <[hidden email] >> <http:///user/SendEmail.jtp?type=node&node=11673&i=5>> wrote: >> What did the More Details link say ? >> >> Thanks >> >> > On Feb 15, 2017, at 3:11 AM, vinay patil <[hidden email] >> <http://user/SendEmail.jtp?type=node&node=11641&i=0>> wrote: >> > >> > Hi, >> > >> > I have kept the checkpointing interval to 6secs and minimum pause >> between >> > checkpoints to 5secs, while testing the pipeline I have observed that >> that >> > for some checkpoints it is taking long time , as you can see in the >> attached >> > snapshot checkpoint id 19 took the maximum time before it gets failed, >> > although it has not received any acknowledgements, now during this >> 10minutes >> > the entire pipeline did not make any progress and no data was getting >> > processed. (For Ex : In 13minutes 20M records were processed and when >> the >> > checkpoint took time there was no progress for the next 10minutes) >> > >> > I have even tried to set max checkpoint timeout to 3min, but in that >> case as >> > well multiple checkpoints were getting failed. >> > >> > I have set RocksDB FLASH_SSD_OPTION >> > What could be the issue ? >> > >> > P.S. I am writing to 3 S3 sinks >> > >> > checkpointing_issue.PNG >> > <http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/file/n11640/checkpointing_issue.PNG> >> > >> > >> > >> > -- >> > View this message in context: http://apache-flink-user-maili >> ng-list-archive.2336050.n4.nabble.com/Checkpointing-with- >> RocksDB-as-statebackend-tp11640.html >> > Sent from the Apache Flink User Mailing List archive. mailing list >> archive at Nabble.com. >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/Checkpointing-with-RocksDB-as-statebackend-tp11640p11641.html >> To start a new topic under Apache Flink User Mailing List archive., email >> [hidden >> email] <http:///user/SendEmail.jtp?type=node&node=11673&i=6> >> To unsubscribe from Apache Flink User Mailing List archive., click here >> <#m_6198963695418156302_m_8892162958879126193_this>. >> NAML >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> >> >> >> ------------------------------ >> If you reply to this email, your message will be added to the discussion >> below: >> http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/Checkpointing-with-RocksDB-as-statebackend-tp11640p11673.html >> To start a new topic under Apache Flink User Mailing List archive., email >> [hidden >> email] <http:///user/SendEmail.jtp?type=node&node=11731&i=4> >> To unsubscribe from Apache Flink User Mailing List archive., click here. >> NAML >> <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >> >> >> > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050.n4. > nabble.com/Checkpointing-with-RocksDB-as-statebackend-tp11640p11731.html > To start a new topic under Apache Flink User Mailing List archive., email > [hidden > email] <http:///user/SendEmail.jtp?type=node&node=11752&i=1> > To unsubscribe from Apache Flink User Mailing List archive., click here. > NAML > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> > > > ------------------------------ > View this message in context: Re: Checkpointing with RocksDB as > statebackend > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Re-Checkpointing-with-RocksDB-as-statebackend-tp11752.html> > > Sent from the Apache Flink User Mailing List archive. mailing list archive > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/> at > Nabble.com. >
