We have encountered a rather rare, but very nasty bug with Flink related
to serialization of Pojos in keystate.
-- Timeline --
1) Write a specific item to keystate of class C at Time1, no read of that
key will happen until step 5.
2) Time elapses
3) class C is schema evolved to include an additio
I should probably clarify that this is intermittent and it is a different
subtask ID each time it does happen.
On Thu, May 5, 2022 at 4:25 PM Ammon Diether wrote:
> Flink Stateful Functions 3.2.0 (Flink 1.14.3)
> All java embedded code.
> Parallelism 32
> Standard Stateful Fun
Flink Stateful Functions 3.2.0 (Flink 1.14.3)
All java embedded code.
Parallelism 32
Standard Stateful Functions Tasks: router -> functions -> feedback
The Router reads from kinesis and routes to stateful functions. For some
reason, one and only one of the router subtasks will have have a start
p, while b will be cleaned in ~10min.
>
> Kind regards,
> Igal.
>
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/fault-tolerance/state/#cleanup-during-rocksdb-compaction
>
> On Wed, Jul 14, 2021 at 5:09 AM Ammon Diether
the state backend. Most
> operators will register a timer and when the timer triggers it cleans up
> that key.
>
> Ammon Diether 于2021年7月14日周三 上午5:17写道：
>
>>
>> Question
>> If the duration is 20 minutes,
>> 1) is the duration per item?
>> 2) or is the
Question
If the duration is 20 minutes,
1) is the duration per item?
2) or is the duration for the table as a whole?
Suppose the following items
("a", "a-value") 30 minutes ago
("b", "b-value") 10 minutes ago
Does "a" get cleaned up? or neither gets cleaned up yet because the most
recent item is
he/flink-statefun/blob/master/statefun-flink/statefun-flink-io/src/main/java/org/apache/flink/statefun/flink/io/datastream/SourceFunctionSpec.java
> [2] https://issues.apache.org/jira/browse/FLINK-22529
>
> On Thu, Apr 29, 2021 at 7:25 AM Ammon Diether wrote:
>
>>
>>
When using Flink Stateful Function's KinesisIngressBuilder, I do not see a
way to set things like ConsumerConfigConstants.SHARD_GETRECORDS_MAX or
ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS
Looking at KinesisSourceProvider, it appears that this is the spot that
creates the FlinkKinesi
