> > It seems to work for some jobs and not for others. Maybe jobs with little > or empty state don't have _entropy_ swapped out correctly?
This is done by design. As the documentation explains: The Flink runtime currently passes the option to inject entropy only to > checkpoint data files. All other files, including checkpoint metadata and > external URI, do not inject entropy to keep checkpoint URIs predictable. This means that smaller pieces of state (state smaller than state.storage.fs.memory-threshold is written with the checkpoint metadata) won't have entropy injected. David On Sat, Nov 13, 2021 at 2:33 AM Dan Hill <quietgol...@gmail.com> wrote: > It seems to work for some jobs and not for others. Maybe jobs with little > or empty state don't have _entropy_ swapped out correctly? > > On Fri, Nov 12, 2021 at 5:31 PM Dan Hill <quietgol...@gmail.com> wrote: > >> Hi. My config I was able to verify my configs work locally with using >> minio. When I have the same code deployed to prod, the entropy key is not >> replaced. Any ideas? My logs are showing the correct entropy key. >> >> >> Configs: >> state.checkpoints.dir: s3a://my-flink-state/_entropy_/checkpoints >> state.savepoints.dir: s3a://my-flink-state/savepoints >> s3.entropy.key: _entropy_ >> s3.entropy.length: 4 >> >> >> INFO [] - Loading configuration property: state.checkpoints.dir, >> s3a://my-flink-state/_entropy_/checkpoints >> INFO [] - Loading configuration property: state.savepoints.dir, >> s3a://my-flink-state/savepoints >> INFO [] - Loading configuration property: s3.entropy.key, _entropy_ >> INFO [] - Loading configuration property: s3.entropy.length, 4 >> >
