Igniters, During work on index rebuild bug [1] I realized that our asynchronous nature of index rebuild logic may lead to long SQL queries in user applications, indistinguishable from hangs.
Fundamental problem here is that we think that node is ready to become partition owner when partition data is rebalanced. But correct rule form user standpoint is that both partitions and indexes are ready. This is always true for in-memory mode and when fresh node with persistent is started, as indexes are filled with data during rebalance. But this is not true for several other cases: 1) When new index were created while a node was done 2) When index.bin was corrupted or removed 3) Possible in future: rebalance through partition files [2], as current IEP doesn't cover secondary indexes. Let's think on how to overcome this problem. First, we may cover it on SQL level: extract used indexes on query planning phase, attach them to query messages, check on mapper side that indexes are ready, re-try if not. Second, we may think of making index rebuild as a part of rebalance procedure, so that node do not become primary until both partitions and indexes are ready. Third, we may detect missing indexes during join. Then node may leave topology, rebuild indexes locally, then re-join. For missing index.bin case, rebuild process may be started even without joining node to the cluster, though it potentially may lead to rebuild of an index which doesn't exist in the cluster anymore. The third option looks appears to be the most promising to me. What do you think? Vladimir. [1] https://issues.apache.org/jira/browse/IGNITE-10291 [2] https://cwiki.apache.org/confluence/display/IGNITE/IEP-28%3A+Cluster+peer-2-peer+balancing
