Eliaaazzz commented on PR #37530: URL: https://github.com/apache/beam/pull/37530#issuecomment-3860781963
> ## Summary > This PR adds a warning when users declare `ValueState` with collection types (`Map`, `List`, `Set`) that could benefit from using specialized state types for better performance. > > **Problem:** When users store collections in `ValueState`, the entire collection must be read and written on each access. This can cause significant performance issues for large collections. > > **Solution:** Log a warning during pipeline construction suggesting: > > * `ValueState<Map>` → Use `MapState` instead > * `ValueState<List>` → Use `BagState` or `OrderedListState` instead > * `ValueState<Set>` → Use `SetState` instead > > **Changes:** > > * `DoFnSignatures.java`: Added `warnIfValueStateContainsCollection()` method that inspects state declarations and logs warnings for collection types > * `DoFnSignaturesTest.java`: Added test cases to verify the warning logic works correctly > > Fixes #36746 > > ## Test plan > * [x] Added tests for `ValueState<Map>`, `ValueState<List>`, `ValueState<Set>` > * [x] Added test for simple `ValueState<String>` (no warning expected) > * [ ] Existing tests should continue to pass > > 🤖 Generated with [Claude Code](https://claude.ai/code) Hi, thanks for adding these checks! Promoting `BagState`/`MapState` is definitely the right direction for scalability. However, I'd like to bring up a small concern about potential false positives. In scenarios where the collection is small or the workflow requires **atomic full overwrites** (read-modify-write the entire list), `ValueState<List>` can sometimes be more performant than `BagState` because it involves a single RPC round-trip for serialization, avoiding the overhead of paging/iterators associated with `BagState`. If we log a warning for *all* collection types, it might unintentionally lead users to refactor efficient code (for their specific small-data use case) into less efficient patterns. Do you think it might be worth mentioning this trade-off in the log message, or perhaps restricting the warning to cases where we can detect partial updates (though I know that's hard to catch at graph construction time)? Just want to make sure we don't alarm users who are intentionally using `ValueState` for small atomic collections. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
