Re: Error while running flink job on local environment

Hi Andrey and Jeff, Thank you for the reply. I agree with Jeff. My concern is to use different code for local and non-local deployments. It would help if StreamExecutionEnvironment.getExecutionEnvironment() works for both local and cluster deployments. Thanks & Regards, Vinayak On Wed,

Re: Error while running flink job on local environment

@Andrey, Although your approach will work, it requires the user to write different code for local mode and other modes. This is inconvenient for users. IMHO, we should not check these kinds of memory configuration in local mode. Or implicitly set the memory of TM pretty large in local mode to avoi

issue on Flink 1.6.2 for Table API when table schema contains nested ObjectArray

hi, I Have column not found exception when issue running a simple FLink query against Flink Table API on Flink 1.6.2. exception log: Caused by: org.apache.calcite.sql.validate.SqlValidatorException: Column 'data.interaction.action_type' not found in table 'mycable' When I changed Flink version

Re: Error while running flink job on local environment

Hi Vinayak, the error message provides a hint about changing config options, you could try to use StreamExecutionEnvironment.createLocalEnvironment(2, customConfig); to increase resources. this issue might also address the problem, it will be part of 1.9 release: https://issues.apache.org/jira/bro

Re: StreamingFileSink part file count reset

Hi Sidhartha, This is a general limitation now because Flink does not keep counters for all buckets but only a global one. Flink assumes that the sink can write to any bucket any time and the counter is not reset to not rewrite the previously written file number 0. Best, Andrey On Tue, Jul 30, 2

Re: pattern to detect if event field changed

You could use ValueState (keyed by key) to store the event. And when each event arrives, check the value in the State to determine if either of "Field1" or "Field2" has changed. If either of the fields have changed, update the state with the new event and perform the trigger (emitting the modified

Checkpoints very slow with high backpressure

Hi, I'm still facing the same issue under 1.8. Our pipeline uses end-to-end exactly-once semantic, which means the consumer program cannot read the messages until they are committed. So in case of an outage, the whole runtime delay is passed over to the next stream processor application and create

pattern to detect if event field changed

Hello, I have a stream of events in the form (key, field1, field2) and I would like to detect every time field2 value changes and emit a new event on another stream.  I searched StackOverflow and [1] was the closest to my use case but not quite since I don't have a pattern to match. Is it poss

Error while running flink job on local environment

Hi, I am using Flink version: 1.7.1 I have a flink job that gets the execution environment as below and executes the job. StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); When I run the code in cluster, it runs fine. But on local machine while running the j

Re: Is Queryable State not working in standalone cluster with 1 task manager?

[image: image.png] --- Oytun Tez *M O T A W O R D* The World's Fastest Human Translation Platform. oy...@motaword.com — www.motaword.com On Tue, Jul 30, 2019 at 9:04 AM Oytun Tez wrote: > We extend the official container and the jar is available under ./lib – so > it wasn't enabled by default

Re: Is Queryable State not working in standalone cluster with 1 task manager?

We extend the official container and the jar is available under ./lib – so it wasn't enabled by default, we moved the jar under ./lib and then build the container+application. --- Oytun Tez *M O T A W O R D* The World's Fastest Human Translation Platform. oy...@motaword.com — www.motaword.com

Fwd: Why is the size of each checkpoint increasing?

Thanks Rohrmann. Your answer inspired me. CountWindow defaults to using CountTrigger， but I set the trigger again. Parallelism is 1 .trigger(DeltaTrigger.of(50,deltaFunction，stateSerializer） Through testing，I found that the data is generated much faster than tigger and countSize,slideSize

Fwd: Why is the size of each checkpoint increasing?

Thanks Rohrmann. Your answer inspired me. CountWindow defaults to using CountTrigger， but I set the trigger again. Parallelism is 1 .trigger(DeltaTrigger.of(50,deltaFunction，stateSerializer） Through testing，I found that the data is generated much faster than tigger and countSize,slideSize

Re: Is Queryable State not working in standalone cluster with 1 task manager?

Hi Oytun, Is QS enabled in your Docker image or did you enable QS by copying/moving flink-queryable-state-runtime_2.11-1.8.0.jar from ./opt to ./lib [1]? Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/queryable_state.html#activating-queryable-state Am M