Re: How can I pass multiple java options in standalone mode ?

Hi Vishwas, I think we are on the same page now. > When you submit a job through Flink WebUI, your main code will be executed in the job manager jvm process. Since job manager is started with the java opts in flink-conf.yaml, so you could find it works. > When you submit a job in command line(fl

processing avro data source using DataSet API and output to parquet

Hi, I am using Flink 1.8.1 DataSet for a batch processing. The data source is avro files and I want to output the result into parquet. https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/batch/ only has no related information. What's the recommended way for doing this? Do I need to wri

Re: How to implement Multi-tenancy in Flink

Hi Fabian, In this case, how do we emit tumbling window when there are no events? Otherwise it’s not possible to emulate a sliding window in process function and move the buffer ring every 5 mins when there are no events. Yes I can create a periodic source function but how can it be associated

Re: Customize file assignments logic in flink application

Hi, Zhu Thanks for reply! I found using SplittableIterator is also doable to some extent. How to choose between these two? Best Lu On Wed, Aug 14, 2019 at 8:02 PM Zhu Zhu wrote: > Hi Lu, > > Implementing your own *InputFormat* and *InputSplitAssigner*(which has > the interface "InputSplit getN

Re: Understanding job flow

The encryptor will be serialized and sent with the rest of your Job Graph when the job is submitted. If it’s not serializable you get an error. Sent from my iPhone > On Aug 15, 2019, at 11:00 AM, Vishwas Siravara wrote: > > Hi guys, > I have a map job where I want to encrypt certain keys . I

Understanding job flow

Hi guys, I have a map job where I want to encrypt certain keys . I initialize the encryptor in the main method and apply it in the map function. How is this encryptor shared when I have my job running on multiple task managers with parallelism > 1 ? Thanks, Vishwas

Flink job parallelism

Hi guys, I have a flink job which I want to run with a parallelism of 2. I run it from command line like : flink run -p 2 -C file:///home/was/classpathconfig/ -c com.visa.flink.cli.Main flink-job-assembly-0.1-SNAPSHOT.jar flink druid My cluster has two task managers with only 1 task slot each. Ho

Re: How can I pass multiple java options in standalone mode ?

No this does not work , the config in flink-config.yaml applies only to job manager and task manager for env.java.options. When I say "flink run ..." from command line, there is a new jvm which is created which starts the main method and that does not get the env.java.options which is set in the fl

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

Congratulations Andrey! On Thu, Aug 15, 2019 at 3:30 PM Fabian Hueske wrote: > Congrats Andrey! > > Am Do., 15. Aug. 2019 um 07:58 Uhr schrieb Gary Yao : > > > Congratulations Andrey, well deserved! > > > > Best, > > Gary > > > > On Thu, Aug 15, 2019 at 7:50 AM Bowen Li wrote: > > > > > Congrat

Re: How to load udf jars in flink program

Hi Jiangang, Does "flink run -j jarpath ..." work for you? If that jar id deployed to the same path on each worker machine, you can try "flink run -C classpath ..." as well. Thanks, Zhu Zhu 刘建刚 于2019年8月15日周四 下午5:31写道： > We are using per-job to load udf jar when start job. Our jar file is

How to load udf jars in flink program

We are using per-job to load udf jar when start job. Our jar file is in another path but not flink's lib path. In the main function, we use classLoader to load the jar file by the jar path. But it reports the following error when job starts running. If the jar file is in lib, everything

Re: Implementing a low level join

Thanks for the advice. *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com * On Thu, Aug 15, 2019 at 9:59 AM Fabian Hueske wrote: > Hi Felipe, > > No, this is not possible (with reasonable effort). > A che

Re: Implementing a low level join

Hi Felipe, No, this is not possible (with reasonable effort). A checkpoint would be the right time to do the change, but this would be very involved, IMO. As I said, you need a global decision to switch the strategy. This could be communicated with a checkpoint barrier. Then all operator would nee

Re: Implementing a low level join

Hi Fabian, thanks for jumping within this thread. Do you think there is possible to extend any join generic operator in order to make it a little dynamic? I was thinking that after I process a checkpoint I can change the join strategy. and if you do, do you have any toy example of this? Thanks,

Re: Implementing a low level join

Hi, Just to clarify. You cannot dynamically switch the join strategy while a job is running. What Hequn suggested was to have a util method Util.joinDynamically(ds1, ds2) that chooses the join strategy when the program is generated (before it is submitted for execution). The problem is that distr

Re: Update tables env after have been register them

What do you use the data(from redis) for? If it's used for driving computation, a Redis source is what you need, but not support now[1]. If it's used for lookup something by keys, you can implement a TableFunction to fetch records from redis, or you can take a look at LookupableTableSource of the

Re: Implementing a low level join

I see, I am gonna try this. Thanks Hequn *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com * On Thu, Aug 15, 2019 at 4:01 AM Hequn Cheng wrote: > Hi Felipe, > > If I understand correctly, you also have t

Re: [ANNOUNCE] Andrey Zagrebin becomes a Flink committer

Congrats Andrey! Am Do., 15. Aug. 2019 um 07:58 Uhr schrieb Gary Yao : > Congratulations Andrey, well deserved! > > Best, > Gary > > On Thu, Aug 15, 2019 at 7:50 AM Bowen Li wrote: > > > Congratulations Andrey! > > > > On Wed, Aug 14, 2019 at 10:18 PM Rong Rong wrote: > > > >> Congratulations A

Re: Flink metrics scope for YARN single job

Hi Biao! > Do you mean "distinguish metrics from different JobManager running on same host"? Exactly. >Will give you a feedback if there is a conclusion. Thanks! On Thu, 15 Aug 2019 at 06:40, Biao Liu wrote: > Hi Vasily, > > > Is there any way to distinguish logs from different JobManager r