Thanks!

On Sat, Nov 13, 2021, 13:04 David Anderson <dander...@apache.org> wrote:

> It seems to work for some jobs and not for others.  Maybe jobs with little
>> or empty state don't have _entropy_ swapped out correctly?
>
>
> This is done by design. As the documentation explains:
>
> The Flink runtime currently passes the option to inject entropy only to
>> checkpoint data files. All other files, including checkpoint metadata and
>> external URI, do not inject entropy to keep checkpoint URIs predictable.
>
>
> This means that smaller pieces of state (state smaller
> than state.storage.fs.memory-threshold is written with the checkpoint
> metadata) won't have entropy injected.
>
> David
>
> On Sat, Nov 13, 2021 at 2:33 AM Dan Hill <quietgol...@gmail.com> wrote:
>
>> It seems to work for some jobs and not for others.  Maybe jobs with
>> little or empty state don't have _entropy_ swapped out correctly?
>>
>> On Fri, Nov 12, 2021 at 5:31 PM Dan Hill <quietgol...@gmail.com> wrote:
>>
>>> Hi.  My config I was able to verify my configs work locally with using
>>> minio.  When I have the same code deployed to prod, the entropy key is not
>>> replaced.  Any ideas?  My logs are showing the correct entropy key.
>>>
>>>
>>> Configs:
>>> state.checkpoints.dir: s3a://my-flink-state/_entropy_/checkpoints
>>> state.savepoints.dir: s3a://my-flink-state/savepoints
>>> s3.entropy.key: _entropy_
>>> s3.entropy.length: 4
>>>
>>>
>>> INFO [] - Loading configuration property: state.checkpoints.dir,
>>> s3a://my-flink-state/_entropy_/checkpoints
>>> INFO [] - Loading configuration property: state.savepoints.dir,
>>> s3a://my-flink-state/savepoints
>>> INFO [] - Loading configuration property: s3.entropy.key, _entropy_
>>> INFO [] - Loading configuration property: s3.entropy.length, 4
>>>
>>

Reply via email to