Re: Partitioning a query result..

2016-12-16 Thread Giaccone, Tony
So the essential bit of this that's not perhaps been exposed is that we're publishing this data to google's pubsub. The intent is to generate the events as a result of actions taken in the main application and store the event data in a new event database Then a periodic job, reads the data from t

Re: Partitioning a query result..

2016-12-16 Thread Andrus Adamchik
Actually this is an interesting architectural discussion. Speaking for myself, I certainly like having it here. The 2 main approaches have already been mentioned: 1. Single dispatcher -> message queue -> multiple workers. 2. Multiple workers that somehow guess their part of the workload. Both c

Re: Partitioning a query result..

2016-12-16 Thread Giaccone, Tony
Right so I agree with the partitioning of the database, that's a thing that can be done. Andrus, I'm a bit less confident in the proposal you're suggesting. I want to be able to spin up new instances potentially in new containers and run them in different environments. If we're moving to a cloud b