Problem with flink-orc and hive

2020-12-04 Thread Сергей Чернов
Hello,   My situation is following: * I write data in ORC format by Flink into HDFS: * I implements   Vectorizer   interface for processing my data and converting it into   VectorizedRowBatch *   I create   OrcBulkWriter: OrcBulkWriterFactory orcBulkWriterFactory = new OrcBulkWriterFactory<>(

Re: Flink 1.9Version State TTL parameter configuration it does not work

2020-12-04 Thread Andrey Zagrebin
Hi Yang, (redirecting this to user mailing list as this is not a dev question) I am not sure why the state loading is stuck after enabling the compaction filter but the background cleanup of RocksDB state with TTL will not work without activating the filter. This happens on RocksDB opening in Fli

Re: Problem with flink-orc and hive

2020-12-04 Thread Arvid Heise
For cross-referencing, here is the SO thread[1]. Unfortunately, I don't have a good answer for you, except try to align the ORC versions somehow. [1] https://stackoverflow.com/questions/65126074/orc-files-generated-by-flink-can-not-be-used-in-hive On Fri, Dec 4, 2020 at 9:00 AM Сергей Чернов wro

Re: Problem with flink-orc and hive

2020-12-04 Thread Sivaprasanna
Let me try this out on my standalone Hive. I remember reading something similar on SO[1]. In this case, it was an external ORC generated by Spark and an external table was created using CDH. The OP answered referring to a community post[2] on Cloudera. It may be worth checking. [1] https://stackov

Flink 1.12 Release Preview Meetup

2020-12-04 Thread Ana Vasiliuk
Hi everyone! We'll be hosting a meetup on the upcoming Apache Flink 1.12 release on December 9. Join the Ask-Me-Anything (AMA) session with Aljoscha Krettek, Timo Walther, Stephan Ewen, Arvid Heise and Robert Metzger. The AMA will be live-streamed on Youtube (link TBA in the meetup groups below).

Re: Broadcasting control messages to a sink

2020-12-04 Thread aj
Hi Jafee, Can u please help me out with the sample code how you have written the custom sink and how you using this broadcast pattern to update schema at run time. It will help me. On Sat, Oct 17, 2020 at 1:55 PM Piotr Nowojski wrote: > Hi Julian, > > Glad to hear it worked! And thanks for comi

Re: Broadcasting control messages to a sink

2020-12-04 Thread Jaffe, Julian
Hey AJ, Depending on your control messages and what you’re trying to accomplish you can simplify the application even further by stripping out the second broadcast and letting operator chaining guarantee that control messages flow appropriately. This results in ___

taskmanager.cpu.cores 1.7976931348623157E308

2020-12-04 Thread Rex Fenley
Hello, I'm tuning flink for parallelism right now and when I look at the JobManager I see *taskmanager.cpu.cores* 1.7976931348623157E308 Which looks like the maximum double number. We have 8 cpu cores, so we figured we'd bump to 16 for hyper threading. We have 37 operators so we rounded up and se

Re: taskmanager.cpu.cores 1.7976931348623157E308

2020-12-04 Thread Rex Fenley
We're running this in a local environment so that may be contributing to what we're seeing. On Fri, Dec 4, 2020 at 10:41 PM Rex Fenley wrote: > Hello, > > I'm tuning flink for parallelism right now and when I look at the > JobManager I see > *taskmanager.cpu.cores* 1.7976931348623157E308 > Which