About serialization, I am wondering if most people would try to have a
customized partioning mainly due to logic motivations, such that they want
some messages to go to the same partition; if that is true, they do not
really care about the actually partition id but all they would do is
specify the partition key to be the same. As for performance motivations
such as sending to the least loaded broker or broker with an existing
connection already, I am not sure if it should really be exposed to users
to consider such optimizations.
On Sun, Feb 2, 2014 at 10:04 AM, Neha Narkhede <neha.narkh...@gmail.com>wrote:
> It *is* expected that the user call this method on
> every single call and because we no longer control the partitioner
> interface there will be no way to control this.
>
> Make sense. This will ensure that new partitions are detected as defined by
> topic.metadata.refresh.interval.ms.
>
> There are a quite a few things that are inconsistent between the server and
> client. For example, java vs scala.
> Regardless, if we change the client to 'org.apache.kafka', the server
> packaging would obviously have to change
> as well to point to 'org.apache.kafka'. This feedback, to remain consistent
> with other Apache projects, was punted
> on when we first entered Apache, for no particular reason.
>
>
> On Sun, Feb 2, 2014 at 9:42 AM, Jay Kreps <jay.kr...@gmail.com> wrote:
>
> > Hey Neha,
> >
> > Basically partitionsForTopic will invoke
> > Metadata.fetch(topic).partitionsFor. So it will block on the first
> request
> > for a given topic while metadata is loaded. Each subsequent request will
> > return whatever metadata is present, the logic for refreshing metadata
> will
> > remain exactly as it is. It *is* expected that the user call this method
> on
> > every single call and because we no longer control the partitioner
> > interface there will be no way to control this.
> >
> > Overall having looked at this more I think I am pretty neutral on the
> > serialization/partitioning thing. We do give up a lot by removing those
> > interfaces...
> >
> > Regardless of whether you prefer org.apache or not, the server code
> doesn't
> > have that package. So we will be inconsistent with our own code, which
> > would be extremely weird. For example the maven co-ords would be
> > different...
> >
> > -Jay
> >
> >
> >
> > On Sat, Feb 1, 2014 at 2:52 PM, Neha Narkhede <neha.narkh...@gmail.com
> > >wrote:
> >
> > > 1. Change send to use java.util.concurrent.Future in send():
> > > Future<RecordPosition> send(ProducerRecord record, Callback callback)
> > > The cancel method will always return false and not do anything.
> Callback
> > > will then be changed to
> > > interface Callback {
> > > public void onCompletion(RecordPosition position, Exception
> > exception);
> > > }
> > > so that the exception is available there too and will be null if no
> error
> > > occurred. I'm not planning on changing the name callback because I
> > haven't
> > > thought of a better one.
> > >
> > > +1
> > >
> > > 2. We will change the way serialization works to proposal 1A in the
> > > previous discussion. That is the Partitioner and Serializer interfaces
> > will
> > > disappear. ProducerRecord will change to:
> > > class ProducerRecord {
> > > public byte[] key() {...}
> > > public byte[] value() {...}
> > > public Integer partition() {...} // can be null
> > > }
> > > In order to allow correctly choosing a partition the Producer interface
> > > will include a new method:
> > > List<PartitionInfo> partitionsForTopic(String topic);
> > > PartitionInfo will be changed to include the actual Node objects not
> just
> > > the Node ids.
> > >
> > > Mostly agree with this but wanted to confirm my understanding of
> > > partitionsForTopic.
> > > So, is that something a user has to remember to invoke in order to
> detect
> > > newly
> > > added partitions to a topic? Will each invocation of partitionsForTopic
> > > lead to a
> > > TopicMetadataRequest RPC?
> > >
> > > 3. I will make the producer implement java.io.Closable but not throw
> any
> > > exceptions as there doesn't really seem to be any disadvantage to this
> > and
> > > the interface may remind people to call close.
> > >
> > > +1
> > >
> > > Non-changes
> > >
> > > 1. I don't plan to change the build system. The SBT=>gradle change is
> > > basically orthoganol and we should debate it in the context of its
> > ticket.
> > >
> > > +1
> > >
> > > 2. I'm going to stick with my oddball kafka.* rather than
> > > org.apache.kafka.* package name and non-getter methods unless everyone
> > > complains.
> > >
> > > -1. I prefer org.apache.kafka for consistency with other top level
> Apache
> > > projects
> > >
> > > 3. I'm not going to introduce a zookeeper dependency in the client as I
> > > don't see any advantage.
> > >
> > > +1. I see the point about non-java clients not having an easy way to
> talk
> > > to zookeeper,
> > > but don't really think the connections to zookeeper is the problem.
> > >
> > > 4. There were a number of reasonable suggestions on the Callback
> > execution
> > > model. I'm going to leave it as is, though. Because we are moving to
> use
> > > Java Future we can't include functionality like ListenableFuture. I
> think
> > > the simplestic callback model where callbacks are executed in the i/o
> > > thread should be good enough for most cases and other cases can use
> their
> > > own threading.
> > >
> > > +1
> > >
> > >
> > > On Sat, Feb 1, 2014 at 9:06 AM, Joel Koshy <jjkosh...@gmail.com>
> wrote:
> > >
> > > > .
> > > > >
> > > > > > In order to allow correctly choosing a partition the Producer
> > > interface
> > > > > > will include a new method:
> > > > > > List<PartitionInfo> partitionsForTopic(String topic);
> > > > > > PartitionInfo will be changed to include the actual Node objects
> > not
> > > > just
> > > > > > the Node ids.
> > > > >
> > > > > Why are the node id's alone insufficient? You can still do
> > > > > round-robin/connection-limiting, etc. with just the node id right?
> > > >
> > > >
> > > > Actually no - we do need the full node info.
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Sent from Gmail Mobile
> > > >
> > >
> >
>
--
-- Guozhang
