Thanks! On Sat, Nov 13, 2021, 13:04 David Anderson <dander...@apache.org> wrote:
> It seems to work for some jobs and not for others. Maybe jobs with little >> or empty state don't have _entropy_ swapped out correctly? > > > This is done by design. As the documentation explains: > > The Flink runtime currently passes the option to inject entropy only to >> checkpoint data files. All other files, including checkpoint metadata and >> external URI, do not inject entropy to keep checkpoint URIs predictable. > > > This means that smaller pieces of state (state smaller > than state.storage.fs.memory-threshold is written with the checkpoint > metadata) won't have entropy injected. > > David > > On Sat, Nov 13, 2021 at 2:33 AM Dan Hill <quietgol...@gmail.com> wrote: > >> It seems to work for some jobs and not for others. Maybe jobs with >> little or empty state don't have _entropy_ swapped out correctly? >> >> On Fri, Nov 12, 2021 at 5:31 PM Dan Hill <quietgol...@gmail.com> wrote: >> >>> Hi. My config I was able to verify my configs work locally with using >>> minio. When I have the same code deployed to prod, the entropy key is not >>> replaced. Any ideas? My logs are showing the correct entropy key. >>> >>> >>> Configs: >>> state.checkpoints.dir: s3a://my-flink-state/_entropy_/checkpoints >>> state.savepoints.dir: s3a://my-flink-state/savepoints >>> s3.entropy.key: _entropy_ >>> s3.entropy.length: 4 >>> >>> >>> INFO [] - Loading configuration property: state.checkpoints.dir, >>> s3a://my-flink-state/_entropy_/checkpoints >>> INFO [] - Loading configuration property: state.savepoints.dir, >>> s3a://my-flink-state/savepoints >>> INFO [] - Loading configuration property: s3.entropy.key, _entropy_ >>> INFO [] - Loading configuration property: s3.entropy.length, 4 >>> >>
