Hi Sidhartha,
I don't think you should worry about this.
Currently the `StreamingFileSink` uses a long to keep this counter. The
maximum of long is 9,223,372,036,854,775,807. The counter would be reset if
count of files reaches that value. I don't think it should happen. WRT the
max filename leng
Thank you for the clarification Habibo and Andrey.
Is there any limitation after which the global counter will reset ? I mean
do we have to worry the counter may get too long and part file crosses the
max filename length limit set by OS or is it handled by flink.
Thanks
Sidhartha
On Tue, Jul 30,
Hi Sidhartha,
This is a general limitation now because Flink does not keep counters for
all buckets but only a global one.
Flink assumes that the sink can write to any bucket any time and the
counter is not reset to not rewrite the previously written file number 0.
Best,
Andrey
On Tue, Jul 30, 2
Hi,
We are using StreamingFileSink with a custom BucketAssigner and
DefaultRollingPolicy. The custom BucketAssigner is simply a date bucket
assigner. The StreamingFileSink creates part files with name
"part--". The
count is an integer and is incrementing on each rollover. Now my doubts
are:
1. Wh
