Re: Reliably implementing global KeyValueStore#get

2017-06-07 Thread Steven Schlansker
Indeed, all good points. Thanks all for the continuing valuable feedback! > On Jun 7, 2017, at 3:07 PM, Matthias J. Sax wrote: > > If you write to remote DB, keep in mind that this will impact you > Streams app, as you loose data locality. > > Thus, populating a DB from the changelog might be

Re: Reliably implementing global KeyValueStore#get

2017-06-07 Thread Matthias J. Sax
If you write to remote DB, keep in mind that this will impact you Streams app, as you loose data locality. Thus, populating a DB from the changelog might be better. It also decouples both systems what give you the advantage that your Streams app can still run if DB has an issues. If you write dire

Re: Reliably implementing global KeyValueStore#get

2017-06-07 Thread Jan Filipiak
Depends, embedded postgress puts you into the same spot. But if you use your state store change log to materialize into a postgress; that might work out decently. Current JDBC doesn't support delete which is an issue but writing a custom sink is not to hard. Best Jan On 07.06.2017 23:47, St

Re: Reliably implementing global KeyValueStore#get

2017-06-07 Thread Steven Schlansker
I was actually considering writing my own KeyValueStore backed by e.g. a Postgres or the like. Is there some feature Connect gains me that would make it better than such an approach? thanks > On Jun 7, 2017, at 2:20 PM, Jan Filipiak wrote: > > Hi, > > have you thought about using connect to p

Re: Reliably implementing global KeyValueStore#get

2017-06-07 Thread Jan Filipiak
Hi, have you thought about using connect to put data into a store that is more reasonable for your kind of query requirements? Best Jan On 07.06.2017 00:29, Steven Schlansker wrote: On Jun 6, 2017, at 2:52 PM, Damian Guy wrote: Steven, In practice, data shouldn't be migrating that often.

Re: Reliably implementing global KeyValueStore#get

2017-06-07 Thread Eno Thereska
Hi Steven, You are right in principle. The thing is that what we shipped in Kafka is just the low level bare bones that in a sense belong to Kafka. A middle layer that keeps track of the data is absolutely needed, and it should hopefully hide the distributed system challenges from the end user.

Re: Reliably implementing global KeyValueStore#get

2017-06-06 Thread Steven Schlansker
> On Jun 6, 2017, at 2:52 PM, Damian Guy wrote: > > Steven, > > In practice, data shouldn't be migrating that often. If it is then you > probably have bigger problems. Understood and agreed, but when designing distributed systems, it usually helps to model for the worst case rather than the "w

Re: Reliably implementing global KeyValueStore#get

2017-06-06 Thread Damian Guy
Steven, In practice, data shouldn't be migrating that often. If it is then you probably have bigger problems. You should be able to use the metadata api to find the instance the key should be on and then when you check that node you can also check with the metadata api that the key should still be

Re: Reliably implementing global KeyValueStore#get

2017-06-06 Thread Steven Schlansker
> On Jun 6, 2017, at 6:16 AM, Eno Thereska wrote: > > Hi Steven, > > Do you know beforehand if a key exists? If you know that and are getting > null() the code will have to retry by refreshing the metadata and going to > the new instance. If you don’t know beforehand if a key exists or not yo

Re: Reliably implementing global KeyValueStore#get

2017-06-06 Thread Eno Thereska
Hi Steven, Do you know beforehand if a key exists? If you know that and are getting null() the code will have to retry by refreshing the metadata and going to the new instance. If you don’t know beforehand if a key exists or not you might have to check all instances of a store to make sure. E