I think that is eating too much of your dog food :)
On Mon, Feb 11, 2013 at 3:08 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > One thought I had on some of these things was that it is actually kind of > funny that we implement them in zookeeper. Technically we could implement > them in Kafka itself which would mean less code (potentially) and also a > clean api to clients. > > The idea is that many of the admin apis are basically changing cluster > state. Examples are adding a topic, deleting a topic, altering the log > config for a topic, changing the replication factor for a topic, migrating > partitions, etc. > > One way to implement these would be to define an admin topic in kafka. Then > standardize a JSON format or something for these command type messages. The > API to make these changes would just be to publish a message to this topic. > This would allow all brokers to subscribe and carry out these commands in > order, and would allow any client that has support for the producer api to > publish commands. > > I think in addition to providing an API this is actually the functionality > needed for most of these cluster changes. For example the dynamic config > patch I posted is basically implementing a queue like this but in zk for > subscribing to and applying config changes. > > This is not fully thought through, but thought I would just toss the idea > out there. > > -Jay > > > On Fri, Feb 8, 2013 at 3:12 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > Admin functionality in 0.8 is kind of messy. What should we do? > > > > There are a bunch of misc. command line tools under kafka.admin. > > AdminUtils is the closest thing we have to an admin interface, but it is > > weirdly implemented as all static methods. Furthermore its APIs are > really > > wonky and tend to describe the operation they perform in zookeeper rather > > than the high-level thing they accomplish. > > > > Since people are going to want programmatic access to administrative > > functionality it would be good to think this through a bit. The least we > > could do would be to refactor AdminUtils into an Admin class and think > > through those APIs enough to make them usable from java/scala. This would > > mean having a clear API for each thing that the various tools currently > do > > (create/delete/alter/describe/list topics, migrate partitions, shutdown > > broker, etc). > > > > Arguably some of these should really be RPC apis so other languages can > > invoke basic operations like creating and deleting topics. But this is > not > > critical so we can probably do it later. > > > > I think it would be good to also think about how we ended up here. I > think > > we are not putting in place basic APIs that provide a layer of > > simplification for common operations. Instead we tend to just be jabbing > > directly at Zookeeper from different places using ad hoc methods in > > ZkUtils. This is really hard to understand or change. In general we seem > to > > have trouble thinking through APIs. > > > > Thoughts? > > > > -Jay > > > > >
