Re: How to user ParameterTool.fromPropertiesFile() to get resource file inside my jar

2017-09-10 Thread Tony Wei
Hi Aljoscha, I found the root cause of my problem from this reference https://stackoverflow.com/questions/18151072/cant-find-resource-file-after-exporting-to-a-runnable-jar . So I changed the way to use ParameterTool. I read the configurations from InputStream, construct them as argument format an

RE: ETL with changing reference data

2017-09-10 Thread qinnchen
Hi Peter, I think what you referred is typical amendment process where partial or all results need to modified. I think it is definitely interesting topic! Here is my two cents In ideal world, reference data source can ingest updated used values as events and join with buffered events in wind

ETL with changing reference data

2017-09-10 Thread Peter Lappo
hi, We are building an ETL style application in Flink that consumes records from a file or a message bus as a DataStream. We are transforming records using SQL and UDFs. The UDF loads reference data in the open method and currently the data loaded remains in memory until the job is cancelled. Th

Best way to deriving streams from another one

2017-09-10 Thread AndreaKinn
Hi, I have a data stream resulting from an operation executed on a data stream of data. Essentially I want to obtain two different streams from that one to send their to different cassandra tables. I.e.: datastream 0 composed by Tuple3 I want to have: a datastream 1 composed by every triple o

FLIP-17: Side Inputs

2017-09-10 Thread Elias Levy
A bit late to this discussion, but I wanted to reiterate something that others also said. Side input readiness, and blocking until that is the case, is an important feature. This is specially true when the side input is used as a configuration stream. You don't want the main stream to be processe

Java heap size

2017-09-10 Thread AndreaKinn
Hi, I developed a program with Flink using OS X. Following the doc so I put in VMArguments of "Run configuration" in eclipse the value: -Xmx800m in order to increase heap memory. I'm using an external lib on flink but all worked perfectly... until now. I modified something in the use of this lib a

Is State access synchronized?

2017-09-10 Thread Federico D'Ambrosio
Hi, as per the mail subject I wanted to ask you if a State access (read and write) is synchronized. I have the following stream: val airtrafficEvents = stream .keyBy(_.flightInfo.flight) .map(new UpdateIdFunction()) where UpdateIdFunction is a RichMapFunction with a ValueState and a MapState,

Delay in Flink timers

2017-09-10 Thread Narendra Joshi
Hi, We are using Flink as a timer scheduler and delay in timer execution is a huge problem for us. What we have experienced is that as the number of Timers we register increases the timers start getting delayed (for more than 5 seconds). Can anyone point us in the right direction to figure out wha

Queryable State

2017-09-10 Thread Navneeth Krishnan
Hi All, I'm running a streaming job on flink 1.3.2 with few queryable states. There are 3 task managers and a job manager. I'm getting timeout exception when trying to query a state and also a warning message in the job manager log. *Client:* final Configuration config = new Configuration(); con