Re: How to share text file across tasks at run time in flink.

Hi, you could use Zookeeper if you want to dynamically change the DB name / credentials at runtime. The GlobalJobParameters are immutable at runtime, so you can not pass updates through it to the cluster. They are intended for parameters for all operators/the entire job and the web interface. Re

Re: How to share text file across tasks at run time in flink.

Thanks to all for your inputs. Yeah, I could put all these common configurations/rules in DB and workers can pick it up dynamically at run time. In this case DB configuration/connection details need to be hard coded ? Is there any way worker can pickup DB name/credentials etc at run time dynamica

Re: How to share text file across tasks at run time in flink.

Hi! 1. The community is working on adding side inputs to the DataStream API. That will allow you to easily distribute data to all of your workers. 2. In the meantime, you could use `.broadcast()` on a DataSet to broadcast data to all workers. You still have to join that data with another stream t

Re: How to share text file across tasks at run time in flink.

Hi, I think what Bswaraj want is excatly something like Storm Distributed Cache API[1] (if I’m not misunderstanding). > The distributed cache feature in storm is used to efficiently distribute > files (or blobs, which is the equivalent terminology for a file in the > distributed cache and is

Re: How to share text file across tasks at run time in flink.

Hi May be you could use Cassandra to store and fetch all such reference data. This way the reference data can be updated without restarting your application. Lohith Sent from my Sony Xperia™ smartphone Baswaraj Kasture wrote Thanks Kostas ! I am using DataStream API. I have few co

Re: How to share text file across tasks at run time in flink.

Thanks Kostas ! I am using DataStream API. I have few config/property files (key vale text file) and also have business rule files (json). These rules and configurations are needed when we process incoming event. Is there any way to share them to task nodes from driver program ? I think this is ve

Re: How to share text file across tasks at run time in flink.

Hello Baswaraj, Are you using the DataSet (batch) or the DataStream API? If you are in the first, you can use a broadcast variable for your task. If you are using the DataStream one, then there is

How to share text file across tasks at run time in flink.

Am running Flink standalone cluster. I have text file that need to be shared across tasks when i submit my application. in other words , put this text file in class path of running tasks. How can we achieve this with flink ? In spark, spark-submit has --jars option that puts all the files specif