Hi Ufuk, Thanks for the great proposal I think this will be a very cool feature :)
Option 1: I think the main problem here is sending all the state everywhere will not scale at all. I think this will even fail for some internal Flink operators (window timers I think are kept like this, maybe Im wrong here). The general problem here what we don't have with the key-value states is that the system can't do the repartitioning automatically. I think we should try to make abstractions that would allow the system to do this. Option 2: To be honest I don't completely get this approach, what do the indices mean in the get set methods? What happens if the same index is used from multiple operators? This may also suffers in scalability like option 1 (but as I said I dont get this completely :() I think another approach could be (might be similar what option 2 is trying to achieve) to provide a Set<T> (or Map) like abstraction to keep the non partitioned states. Users could add/remove things from it at their on will, but the system would be free to redistribute the Sets between the operators. In practice this would mean for instance that the Kafka sources would store (partition, offset) tuples in the set but and every time in the open method they would check what is assigned to them (the system is free to decide). This of course would only work well if we can assume that distributing the states by equal numbers is desirable. Broadcast states: This might be a good time to think about broadcast states. Non-partitioned states that are the same at all subtasks, I think this comes up in a lot of use-cases (I know at least one myself haha) and it is pretty straight forward from a runtime perspective, the bigger question is the API. Cheers, Gyula Ufuk Celebi <u...@apache.org> ezt írta (időpont: 2016. aug. 11., Cs, 19:20): > Hey all! > > I've created a short FLIP for rescalable non-partitioned state: > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-8%3A+Rescalable+Non-Partitioned+State > > This is related to an effort led by Till, Aljoscha, Stephan, and > Stefan to allow rescaling of Flink jobs (see FLINK-3755, FLINK-4381, > and > https://docs.google.com/document/d/1G1OS1z3xEBOrYD4wSu-LuBCyPUWyFd9l3T9WyssQ63w/edit > ). > > FLIP-8 provides some explanation about why we need rescaling of > non-partitioned state, how the APIs could look like and a high-level > implementation idea. > > I think the main point for discussion should be the APIs. I would like > to start working on an initial implementation next week. I would then > extend the implementation/proposed changes part after figuring out > more details/implications. > > Ufuk >
