>
> It seems to work for some jobs and not for others.  Maybe jobs with little
> or empty state don't have _entropy_ swapped out correctly?


This is done by design. As the documentation explains:

The Flink runtime currently passes the option to inject entropy only to
> checkpoint data files. All other files, including checkpoint metadata and
> external URI, do not inject entropy to keep checkpoint URIs predictable.


This means that smaller pieces of state (state smaller
than state.storage.fs.memory-threshold is written with the checkpoint
metadata) won't have entropy injected.

David

On Sat, Nov 13, 2021 at 2:33 AM Dan Hill <quietgol...@gmail.com> wrote:

> It seems to work for some jobs and not for others.  Maybe jobs with little
> or empty state don't have _entropy_ swapped out correctly?
>
> On Fri, Nov 12, 2021 at 5:31 PM Dan Hill <quietgol...@gmail.com> wrote:
>
>> Hi.  My config I was able to verify my configs work locally with using
>> minio.  When I have the same code deployed to prod, the entropy key is not
>> replaced.  Any ideas?  My logs are showing the correct entropy key.
>>
>>
>> Configs:
>> state.checkpoints.dir: s3a://my-flink-state/_entropy_/checkpoints
>> state.savepoints.dir: s3a://my-flink-state/savepoints
>> s3.entropy.key: _entropy_
>> s3.entropy.length: 4
>>
>>
>> INFO [] - Loading configuration property: state.checkpoints.dir,
>> s3a://my-flink-state/_entropy_/checkpoints
>> INFO [] - Loading configuration property: state.savepoints.dir,
>> s3a://my-flink-state/savepoints
>> INFO [] - Loading configuration property: s3.entropy.key, _entropy_
>> INFO [] - Loading configuration property: s3.entropy.length, 4
>>
>

Reply via email to