Hi Marco,
> Is this assumption correct?
Yes. More or else each operator is first creating a copy of its state
locally and uploading to S3 this whole file at once.
Please first take a look which part of checkpointing is taking so long.
Re backpressure. Keep in mind that Checkpoint Barriers need
Thank you.
Checkpoints timeout often, even though the timeout limit is 20 minutes. The
volume of records in our processing window that require checkpointing is
large (between 20 and 2 million). I made the assumption that Flink
would batch a blob of bytes to S3, and not create an S3 call per re
Hi,
Yes, it's working. You would need to analyse what's working slower than
expected. Checkpointing times? (Async duration? Sync duration? Start
delay/back pressure?) Throughput? Recovery/startup? Are you being rate
limited by Amazon?
Piotrek
czw., 28 sty 2021 o 03:46 Marco Villalobos
napisał(a
Just curious, has anybody had success with Amazon EMR with RocksDB and
checkpointing in S3?
That's the configuration I am trying to setup, but my system is running
more slowly than expected.
