Hi,
The end result that I'm expecting is to only have a few files that are
filled up to a specific file size and not multiple small files.
So if I understand you correctly, when I enable the auto compaction, all
the small files that are being created will be merged into files that are
up to the specified size?
eg. (with the default 128MB max filesize)
Without autocompaction:
output/
file1.avro - 28MB
file2.avro - 100MB
file3.avro - 30MB
With autocompaction:
output/
file1.avro - 128MB
file2.avro - 30MB
Am I understanding this correctly? If so, I think autocompaction is the
exact solution to this issue.
Best Regards
Kamil
On Mon, 22 Nov 2021 at 16:43, Francesco Guardiani <france...@ververica.com>
wrote:
> Hi,
> Looking at the code, there is no ability to disable the rollover-interval.
>
> But I'm wondering, what are you trying to do? Write a file up to the
> configured file-size? Note that if you're using auto compaction, on every
> checkpoint you'll have a rollover, regardless of the rollover-interval.
>
> I cc'ed Fabian in the discussion which has a better knowledge than me on
> file sink relates topics.
>
> FG
>
> On Mon, Nov 22, 2021 at 3:51 PM Matthias Pohl <matth...@ververica.com>
> wrote:
>
>> Hi Kamil,
>> by looking at the code I'd say that the only option you have is to
>> increase the parameter you already mentioned to a very high number. But I'm
>> not sure about the side effects. I'm gonna add Francesco to this thread.
>> Maybe he has better ideas on how to answer your question.
>>
>> Best,
>> Matthias
>>
>> On Mon, Nov 22, 2021 at 10:32 AM Kamil ty <kamilt...@gmail.com> wrote:
>>
>>> Hey all,
>>>
>>> I wanted to know if there is a way to disable the interval rolling
>>> policy in the Table API filesystem connector.
>>> From flink docs: FileSystem | Apache Flink
>>> <https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/filesystem/#rolling-policy>
>>> The key to change the interval: sink.rolling-policy.rollover-interval
>>> Is it possible to fully disable this rolling policy or the only solution
>>> is to set a very big duration?
>>>
>>> Best Regards
>>> Kamil
>>>
>>
