[Help Needed] Efficient Handling of Large Historical Data in RocksDB (Flink 1.17.2)

2024-09-15 Thread Owais Ansari
Dear Flink Community, We want to process user data, including historical events, stored as JSON arrays in RocksDB to keep the data localized during scaling in Flink (1.17.2). When a new event arrives, our goal is to: 1. Fetch historical data from RocksDB. 2. Run a SQL query on the data (we

Re: Is there anyway to control the size of my ListState in my KeyedProcessFunction

2025-06-03 Thread Owais Ansari
+user@flink.apache.org On Wed, 4 Jun, 2025, 9:41 am Owais Ansari, wrote: > It expires the individual key and not the entire state. For your use case > Map state is a good option. > > On Wed, 4 Jun, 2025, 8:26 am Sachin Mittal, wrote: > >> So