Fanbin, I have no idea now, can you created a JIRA to track it? You can describe complete SQL and some data informations.
Best, Jingsong Lee On Thu, Jan 23, 2020 at 4:13 PM Fanbin Bu <fanbin...@coinbase.com> wrote: > Jingsong, > > Do you have any suggestions to debug the above mentioned > IndexOutOfBoundsException error? > Thanks, > > Fanbin > > On Wed, Jan 22, 2020 at 11:07 PM Fanbin Bu <fanbin...@coinbase.com> wrote: > >> I got the following error when running another job. any suggestions? >> >> Caused by: java.lang.IndexOutOfBoundsException >> at >> org.apache.flink.core.memory.MemorySegment.getInt(MemorySegment.java:701) >> at org.apache.flink.table.dataformat.BinaryRow.getInt(BinaryRow.java:264) >> at HashWinAggWithKeys$538.endInput(Unknown Source) >> at >> org.apache.flink.streaming.runtime.tasks.OperatorChain.endInput(OperatorChain.java:276) >> at >> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.checkFinished(StreamOneInputProcessor.java:151) >> at >> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:138) >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask.performDefaultAction(StreamTask.java:276) >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:298) >> at >> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:403) >> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) >> at java.lang.Thread.run(Thread.java:748) >> >> On Wed, Jan 22, 2020 at 8:57 PM Fanbin Bu <fanbin...@coinbase.com> wrote: >> >>> Jingsong, >>> >>> I set the config value to be too large. After I changed it to a smaller >>> number it works now! >>> thanks you for the help. really appreciate it! >>> >>> Fanbin >>> >>> On Wed, Jan 22, 2020 at 8:50 PM Jingsong Li <jingsongl...@gmail.com> >>> wrote: >>> >>>> Fanbin, >>>> >>>> Looks like your config is wrong, can you show your config code? >>>> >>>> Best, >>>> Jingsong Lee >>>> >>>> On Thu, Jan 23, 2020 at 12:41 PM Fanbin Bu <fanbin...@coinbase.com> >>>> wrote: >>>> >>>>> Jingsong, >>>>> >>>>> Great, now i got a different error: >>>>> >>>>> java.lang.NullPointerException: Initial Segment may not be null >>>>> at >>>>> org.apache.flink.runtime.memory.AbstractPagedOutputView.<init>(AbstractPagedOutputView.java:65) >>>>> at >>>>> org.apache.flink.runtime.io.disk.SimpleCollectingOutputView.<init>(SimpleCollectingOutputView.java:49) >>>>> at >>>>> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap$RecordArea.<init>(BytesHashMap.java:522) >>>>> at >>>>> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.<init>(BytesHashMap.java:190) >>>>> at >>>>> org.apache.flink.table.runtime.operators.aggregate.BytesHashMap.<init>(BytesHashMap.java:149) >>>>> at LocalHashWinAggWithKeys$292.open(Unknown Source) >>>>> at >>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.openAllOperators(StreamTask.java:529) >>>>> at >>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:393) >>>>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) >>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) >>>>> at java.lang.Thread.run(Thread.java:748) >>>>> >>>>> >>>>> is there any other config i should add? >>>>> >>>>> thanks, >>>>> >>>>> Fanbin >>>>> >>>>> >>>>> On Wed, Jan 22, 2020 at 8:07 PM Fanbin Bu <fanbin...@coinbase.com> >>>>> wrote: >>>>> >>>>>> you beat me to it. >>>>>> let's me try that. >>>>>> >>>>>> On Wed, Jan 22, 2020 at 7:57 PM Jingsong Li <jingsongl...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Fanbin, >>>>>>> >>>>>>> Document is here: >>>>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/config.html >>>>>>> NOTE: you need configure this into TableConfig. >>>>>>> >>>>>>> Best, >>>>>>> Jingsong Lee >>>>>>> >>>>>>> On Thu, Jan 23, 2020 at 11:54 AM Fanbin Bu <fanbin...@coinbase.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Jingsong, >>>>>>>> >>>>>>>> Thank you for the response. >>>>>>>> Since I'm using flink on EMR and the latest version is 1.9 now. the >>>>>>>> second option is ruled out. but will keep that in mind for future >>>>>>>> upgrade. >>>>>>>> >>>>>>>> I'm going to try the first option. It's probably a good idea to add >>>>>>>> that in the doc for example: >>>>>>>> https://ci.apache.org/projects/flink/flink-docs-stable/ops/config.html >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Fanbin >>>>>>>> >>>>>>>> On Wed, Jan 22, 2020 at 7:08 PM Jingsong Li <jingsongl...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Fanbin, >>>>>>>>> >>>>>>>>> Thanks for using blink batch mode. >>>>>>>>> >>>>>>>>> The OOM is caused by the manage memory not enough in Hash >>>>>>>>> aggregation. >>>>>>>>> >>>>>>>>> There are three options you can choose from: >>>>>>>>> >>>>>>>>> 1.Is your version Flink 1.9? 1.9 still use fix memory >>>>>>>>> configuration. So you need increase hash memory: >>>>>>>>> - table.exec.resource.hash-agg.memory: 1024 mb >>>>>>>>> >>>>>>>>> 2.In 1.10, we use slot manage memory to dynamic config real >>>>>>>>> operator memory, so operator can use more manage memory, so you don't >>>>>>>>> need >>>>>>>>> configure hash agg memory anymore. You can try 1.10 RC0 [1] >>>>>>>>> >>>>>>>>> 3.We can use sort aggregation to avoid OOM too, but there is no >>>>>>>>> config option now, I created JIRA to track it. [2] >>>>>>>>> >>>>>>>>> [1] >>>>>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Apache-Flink-1-10-0-release-candidate-0-td36770.html >>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-15732 >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Jingsong Lee >>>>>>>>> >>>>>>>>> On Thu, Jan 23, 2020 at 7:40 AM Fanbin Bu <fanbin...@coinbase.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> tried to increase memory: >>>>>>>>>> flink run -m yarn-cluster -p 16 -ys 1 -ytm 200000 -yjm 8096 myjar >>>>>>>>>> >>>>>>>>>> and still got the same OOM exception. >>>>>>>>>> >>>>>>>>>> my sql is like: >>>>>>>>>> >>>>>>>>>> select id, hop_end(created_at, interval '30' second, interval '1' >>>>>>>>>> minute), sum(field)... #20 of these sums >>>>>>>>>> >>>>>>>>>> from table group by id, hop(created_at, interval '30' second, >>>>>>>>>> interval '1' minute) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Jan 22, 2020 at 3:19 PM Fanbin Bu <fanbin...@coinbase.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I have a batch job using blink planner. and got the following >>>>>>>>>>> error. I was able to successfully run the same job with flink 1.8 >>>>>>>>>>> on yarn. >>>>>>>>>>> >>>>>>>>>>> I set conf as: >>>>>>>>>>> taskmanager.heap.size: 50000m >>>>>>>>>>> >>>>>>>>>>> and flink UI gives me >>>>>>>>>>> Last Heartbeat:20-01-22 >>>>>>>>>>> 14:56:25ID:container_1579720108062_0018_01_000020Data >>>>>>>>>>> Port:41029Free Slots >>>>>>>>>>> / All Slots:1 / 0CPU Cores:16Physical Memory:62.9 GBJVM Heap >>>>>>>>>>> Size:10.3 >>>>>>>>>>> GBFlink Managed Memory:24.9 GB >>>>>>>>>>> >>>>>>>>>>> any suggestions on how to move forward? >>>>>>>>>>> Thanks, >>>>>>>>>>> Fanbin >>>>>>>>>>> >>>>>>>>>>> Caused by: >>>>>>>>>>> org.apache.flink.runtime.client.JobExecutionException: Job >>>>>>>>>>> execution failed. >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:146) >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.client.program.rest.RestClusterClient.submitJob(RestClusterClient.java:259) >>>>>>>>>>> ... 25 more >>>>>>>>>>> >>>>>>>>>>> *Caused by: java.io.IOException: Hash window aggregate map OOM.* >>>>>>>>>>> at HashWinAggWithKeys$534.processElement(Unknown Source) >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processElement(StreamOneInputProcessor.java:164) >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:143) >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.performDefaultAction(StreamTask.java:276) >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:298) >>>>>>>>>>> at >>>>>>>>>>> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:403) >>>>>>>>>>> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) >>>>>>>>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) >>>>>>>>>>> at java.lang.Thread.run(Thread.java:748) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Best, Jingsong Lee >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Best, Jingsong Lee >>>>>>> >>>>>> >>>> >>>> -- >>>> Best, Jingsong Lee >>>> >>> -- Best, Jingsong Lee
