Hi Svante, Thank you, that's cleared things up. We'll look for something where locality is more built in
Ben > -----Original Message----- > From: Svante Karlsson [mailto:svante.karls...@csi.se] > Sent: 12 November 2015 09:17 > To: users@kafka.apache.org > Subject: Re: Locality question > > If you have a kafka partition that is replicated to 3 nodes the partition > varies > (in time) thus making the colocation pointless. You can only produce and > consume to/from the leader. > > /svante > > > > 2015-11-12 9:00 GMT+01:00 Young, Ben <ben.yo...@sungard.com>: > > > Hi, > > > > Any thoughts on this? Perhaps Kafka is not the best way to go for > > this, but the docs do mention transaction/replication logs as a use > > case, and I'd have thought locality would have been important for that? > > > > Thanks, > > Ben > > > > -----Original Message----- > > From: Young, Ben [mailto:ben.yo...@sungard.com] > > Sent: 06 November 2015 08:20 > > To: users@kafka.apache.org > > Subject: Locality question > > > > Hi, > > > > I've had a look over the website and searched the archives, but I > > can't find any obvious answers to this, so apologies if it's been asked > before. > > > > I'm investigating potentially using Kafka for the transaction log for > > our in-memory database technology. The idea is the Kafka partitioning > > and replication will "automatically" give us sharding and hot-standby > > capabilities in the db (obviously with a fair amount of work). > > > > The database can ingest hundreds of gigabytes of data extremely > > quickly, easily enough to saturate any reasonable network connection, > > so I've thought about co-locating the db on the same nodes of the > > kafka cluster that actually store the data, to cut out the network > > entirely from the loading process. We'd also probably want the db > > topology to be defined first, and the kafka partitioning to follow. I > > can see how to use the partitioner class to assign a specific > > partition to a key, but I can't currently see how to assume partitions > > to known machines upfront. Is this possible? > > > > Does the plan sound reasonable in general? I've also considered a log > > shipping approach like Flume, but Kafka seems simplest all round, and > > a really like the idea of just being able to set the log offset to > > zero to reload on startup. > > > > Thanks, > > Ben Young > > > > > > Ben Young . Principal Software Engineer . Adaptiv . > > > > > >
