Greetings, I am hoping for some design advice? I am new to Ignite and don’t want to start out on the wrong foot.
At my day job we have a “Rates Engine”. It computes the current Rate for a particular User request. It is a high volume/low latency system. And currently it lives within a Java Servlet Engine (jetty), but is moving to Netty. This is how it works – from 50000 ft : In the Foreground: User X requests a Rate (current price) for Product Y. (Actually it is for Products[X,Y,Z,…n], where n <= 1000) The request is routed to “Shard P” based on Y. The Rate is computed based on the cached RateInfo (essentially, a concurrent HashMap) for Y, found in Shard P. The computation is relatively expensive CPU-wise. The Rate is returned to User X. In the Background: The system listens to “Rate change events” for all Products. Rate changes are read and are aggregated over some period of time. Periodically, the appropriate, cached RateInfo is updated as required, in the appropriate Shard Simple. And it works well. Except the system relies on static host bindings. The system is showing its age. Moreover, we are moving all of this to AWS. So, since we have to refactor anyway, we decided to look at better ways. And Ignite seems a perfect fit. Ignite clustering, sharding, and replication are all quite appealing. So my question. How would y’all model this system? I would like to use Ignite as a standalone cluster (not in-memory within the Rates Engine) To basically divide the system into its component pieces; A “Rates Engine”, a Distributed HashMap, and a “Rates Injector” The things I find most puzzling are: * How does a Clustered & Partitioned IgniteCache handle a `cache.getAll(Set<> keys)` behind the scenes ?? Will it transparently fan those requests out to the appropriate Partition for me (from the Client – or is it a “double hop” — once to the Cluster, another to the appropriate shard)?? How does it handle individual failures?? I can’t seem to find much detail about how all of the internal routing is handled in Ignite. * There is currently ~30GB of Rates cached. What is an optimal “shard size”? I’ve found in other sharded systems that; (shard <= 5GB) is a good operational rule of thumb. For when data has to start moving about for maintenance, etc anything larger than 5GB is cumbersome. Is this true for Ignite? I am trying to figure out how big a cluster to build out. * Would you build your own “artificial sharding” on top of the Rates. In other words; apply “affinity collocation” and use say Geo info. Something/anything so that the Rates collocate by some auxiliary key. That builds reasonably sized shards?? This is probably a given. * Is it possible to configure Ignite to, essentially, “always serve data”. E.g. even it if split-brains?? In other words, since my RatesEngine is read-only, and really doesn’t care (Rates are assumed to be eventually consistent) – as long as it can find Rates it will be happy. Only the Rates Injector is really concerned about fidelity. And may blow up. And as we piece back together the world, the Rates Engine can keep chugging along somehow. Does that make sense?? Perhaps another way to ask this is, what are the failure scenarios for an Ignite Cluster?? Sorry if these questions are too broad. I am a total newbie. I am looking to start a POC, and need a place to begin. Thanks much, -- Chris
