I think "java.lang.OutOfMemoryError: Map failed" has usually been "out of
address space for mmap" if memory serves.
If you sum the length of all .index files while the service is running (not
after stopped), do they sum to something really close to 2GB? If so it is
likely either that the OS/arch i
Hey Garry,
Super interesting. We honestly never did a ton of performance tuning on the
producer. I checked the profiles early on in development and we fixed a few
issues that popped up in deployment, but I don't think anyone has done a
really scientific look. If you (or anyone else) want to dive i
Hey Ben,
The consumer actually doesn't promise to return records on any given poll()
call and even in trunk it won't return records on the first call likely.
Internally the reason is that it basically does one or two rounds of
non-blocking actions and then returns. This could include things like
This sounds a lot like a bug we fixed in 0.8.2.0, no chance you are running
that pre-release version is there?
-Jay
On Wed, Jun 3, 2015 at 9:43 PM, Anuja Pundlik (apundlik) wrote:
> Hi,
>
> I am using Kafka 0.8.2.1.
> We have 1 zookeeper, 3 kafka brokers.
> We have 9 topics, out of which 1 topi
d to hear from you. I met you at the kafka meetup at linkedin.
> >
> >- No, I am running kafka_2.11-0.8.2.1
> >
> >
> >Are there any logs/ any info that I can provide that will help you
> >understand what could be the issue?
> >
> >Thanks,
> >Anuja
I have been thinking a little about this. I don't think CAS actually
requires any particular broker support. Rather the two writers just write
messages with some deterministic check-and-set criteria and all the
replicas read from the log and check this criteria before applying the
write. This mecha
If I recall correctly, setting log.retention.ms and log.retention.bytes to
-1 disables both.
On Fri, Jul 10, 2015 at 1:55 PM, Daniel Schierbeck <
daniel.schierb...@gmail.com> wrote:
>
> > On 10. jul. 2015, at 15.16, Shayne S wrote:
> >
> > There are two ways you can configure your topics, log co
This is almost certainly a client bug. Kafka's request format is size
delimited messages in the form
<4 byte size N>
If the client sends a request with an invalid size or sends a partial
request the server will see effectively random bytes from the next request
as the size of the next message an
Hey Sivananda,
That's actually no longer true and likely a documentation bug. Where did
you see that?
-Jay
On Fri, Jul 17, 2015 at 9:35 AM, Sivananda Reddy
wrote:
> Hi,
>
> Kafka document ion says that the new producer is in Beta state, how safe is
> it to use the new producer in production?.
It seems less weird if you think of the offset as the position of the
consumer, i.e. it is "on" record 5. In some sense the consumer is actually
in between records, i.e. if it has processed 4 and not processed 5 do you
think about your position as being on 4 or on 5? Well not on 4 because it
alread
Hey James,
You are right the intended use of that was to have a way to capture some
very small metadata about your state at the time of offset commit in an
atomic way.
That field isn't exposed but we do need to add it to the new consumer api
(I think just no one has done it yet.
-Jay
On Mon, Au
Yuheng,
>From the command you gave it looks like you are configuring the perf test
to send data as fast as possible (the -1 for target throughput). This means
it will always queue up a bunch of unsent data until the buffer is
exhausted and then block. The larger the buffer, the bigger the queue. T
The second command you give actually doesn't seem to double the memory
(maybe just a typo?). I can't explain why doubling buffer memory would
decrease throughput. The only effect of adding memory would be if you run
out, and then running out of memory would cause you to block and hence
lower throug
rtitions on 2 brokers (on different physical server too) gave me the
> reading of 25MB/sec
>
> I just wanna know how to interpret these numbers so I can draw a pattern
> but so far this is
> not very consistent (more partitions = less throughput?)
>
> Cheers,
>
> Paul
>
>
&
on
> different spindle) would lead to better read/write performance because
> of the aggregate bandwidth derived from parallel operations. But what
> I am seeing here defies my understanding. I just wonder if I did the
> benchmark tests wrong or I had the concept wrong.
>
> Kind R
Hey Dave,
We're separating the problem of getting data in and out of Kafka from the
problem of transforming it. If you think about ETL (Extract, Transform,
Load), what Kafka Connect does is E and L really really well and not T at
all; the focus in stream processing systems is T with E and L being
Actually this may be something we can improve in the documentation. Calling
poll(1000) doesn't mean "check for new messages every second" but rather
"return the instant there are new messages, but if no messages come return
after a one second timeout passes".
So in that respect both the old and ne
I think the point is that we should ideally try to cover all these in the
"upgrade" notes.
-Jay
On Tue, Dec 1, 2015 at 10:37 AM, Aditya Auradkar
wrote:
> Rajiv,
>
> By default, the quota is unlimited until you decide to configure them
> explicitly.
> And yes, we did get rid of "replica.lag.max.
It's worth noting that both the old and new consumer are identical in the
number of records fetched at once and this is bounded by the fetch size and
the number of partitions you subscribe to. The old consumer held these in
memory internally and waited for you to ask for them, the new consumer
imme
Hey Everyone,
As you may have heard, Confluent is hosting the first ever Kafka
Summit. It'll be in San Francisco on Tuesday, April 26, 2016.
We'll be announcing open registration tomorrow, but I wanted to let
everyone here know first, and also let you know there is a $50
community discount. To ge
Kafka should be robust against abrupt termination.
-Jay
On Sat, Dec 5, 2015 at 11:05 AM, Per Jahn wrote:
> Hi
>
> Are files that kafka writes to the file system robust against abrupt
> termination?
> Including fs meta data.
> And assuming it's used with a journaling fs.
> Or do I need ups and a
Hey guys,
Those docs were for a draft version of the API and I think they may be
a bit out of date. I'd stick with the javadocs linked here:
http://kafka.apache.org/documentation.html#newconsumerapi
-Jay
On Sun, Dec 6, 2015 at 7:40 AM, Dhyan Muralidharan
wrote:
> Thank you .These docs are good
If you use the perf test without any bound on throughput it will
always try to send data faster than it can go out and build up a queue
of unsent data. So e.g. if your buffer is 1MB each send will be
blocked on waiting for the full 1MB of queued data to clear out and
get sent. This makes sense if y
Hey Ewen, how come you need to get it all in memory for approach (1)? I
guess the obvious thing to do would just be to query for the record
after-image when you get the diff--e.g. just read a batch of changes and
multi-get the final values. I don't know how bad the overhead of this would
be...batch
> Kafka, you could track only offsets in memory and read back full values as
> needed to apply deltas, but this of course requires random reads into your
> Kafka topic (but may perform fine in practice depending on the workload).
>
> -Ewen
>
> On Fri, Jan 29, 2016 at 9:12 AM, Ja
Also, most database provide a "full logging" option that let's you capture
the whole row in the log (I know Oracle and MySQL have this) but it sounds
like Mongo doesn't yet. That would be the ideal solution.
-Jay
On Fri, Jan 29, 2016 at 9:38 AM, Jay Kreps wrote:
> Ah, a
Check out Kafka Connect:
http://www.confluent.io/blog/how-to-build-a-scalable-etl-pipeline-with-kafka-connect
-Jay
On Wed, Feb 10, 2016 at 5:09 PM, R P wrote:
> Hello All,
> New Kafka user here. What is the best way to write Kafka data into HDFS?
> I have looked into following options and f
The default semantics of the new consumer with auto commit are
at-least-once-delivery. Basically during the poll() call the commit will be
triggered and will commit the offset for the messages consumed during the
previous poll call. This is an advantage over the older scala consumer
where the consu
Yeah I didn't mean to imply that we committed after each poll, but rather
that when it was time to commit, this would happen on the next poll call
and hence only commit processed messages.
-Jay
On Thu, Feb 18, 2016 at 2:21 PM, Avi Flax wrote:
> On Thu, Feb 18, 2016 at 4:26 PM, Jay Krep
Hey Andrew,
Yeah I think the current state is that we did several design and prototypes
(both the transaction work and the idempotence design and the conditional
write KIP), but none of these offshoots is really fully rationalized with
the other ones. Slow progress in this area has been mainly due
Hey Gligor,
Sorry for the rough edges. I think there are a variety of rough edges in
error messages here we can improve:
1. "Error ILLEGAL_GENERATION occurred while committing offsets for group
MetadataConsumerSpout" is obviously NOT the most intuitive error message,
it doesn't really ex
Hey all,
Lot's of people have probably seen the ongoing work on Kafka Streams
happening. There is no real way to design a system like this in a vacuum,
so we put up a blog, some snapshot docs, and something you can download and
use easily to get feedback:
http://www.confluent.io/blog/introducing-
ve-stream protocol for
> interoperability with libraries akka-stream and RxJava?
>
> Thanks,
> DaVe.
>
> David Buschman
> d...@timeli.io
>
>
>
> > On Mar 10, 2016, at 2:26 PM, Jay Kreps wrote:
> >
> > Hey all,
> >
> > Lot's of people have
; On Mar 11, 2016, at 7:21 AM, Dick Davies wrote:
> >
> > Nice - I've read topics on the idea of a database as the 'now' view of a
> stream
> > of updates, it's a very powerful concept.
> >
> > Reminds me of Rich Hickeys talk on DAtomic, if anyone
This seems like a bug, no? It should just initiate the request not wait for
it to be written, there is no way for the user to reason about the state of
the send buffer.
-jay
On Monday, March 14, 2016, Jason Gustafson wrote:
> Hey Alexey,
>
> Asynchronous commit handling could probably be improv
If you hard kill the broker when it restarts it doesn't know the
status of it's on-disk files, it will need to run though the last log
segment to validate the checksums of messages and rebuild the index
off this to ensure consistency. (Why does it need to do this
validation? Because in the event of
I think the question is when do you actually *want* processing time
semantics? There are definitely times when its safe to assume the two are
close enough that a little lossiness doesn't matter much but it is pretty
hard to make assumptions about when the processing time is and has been
hard for us
I think the question is when do you actually *want* processing time
semantics? There are definitely times when its safe to assume the two are
close enough that a little lossiness doesn't matter much but it is pretty
hard to make assumptions about when the processing time is and has been
hard for us
+1
On Sat, May 27, 2017 at 11:04 AM, Gwen Shapira wrote:
> Thanks Vahid,
>
> Do you mind if we leave the command-line out of scope for this?
>
> I can see why adding confirmations, options to bypass confirmations, etc
> would be an improvement. However, I've seen no complaints about the current
I suspect this is on Linux right?
The way Linux works is it uses a percent of memory to buffer new writes, at
a certain point it thinks it has too much buffered data and it gives high
priority to writing that out. The good news about this is that the writes
are very linear, well layed out, and hig
Hey Chris,
I heard a similar complaint from a few people. I am quite ignorant about
event sourcing and don't feel I understand the relationship fully but I am
interested in understanding a little better what you are saying.
I think we see the world this way:
1. You store the log of primary ev
I think you can do this now by using a custom partitioner, no?
https://kafka.apache.org/0110/javadoc/org/apache/kafka/clients/producer/Partitioner.html
-Jay
On Mon, Oct 2, 2017 at 6:29 AM Michal Michalski
wrote:
> Hi,
>
> TL;DR: I'd love to be able to make log compaction more "granular" than j
There isn't a jdbc sink yet, though that is actually a very useful and easy
connector to write. I think it can be pretty efficient as long as it uses
prepared statements and batch insert...
-Jay
On Fri, Apr 1, 2016 at 6:06 AM, Kavitha Veluri
wrote:
> Thank you for the guidance . Does Kafka conn
Another way to think about this is that the producer allows you to PUSH
data into Kafka and the consumer allows you to PULL data out. This is what
you need to write an application.
However for an existing data system you need the opposite you need to PULL
data into Kafka from the system or PUSH it
To summarize a chat session Guozhang and I just had on slack:
We currently do dedupe the output for stateful operations (e.g. join,
aggregate). They hit an in-memory cache and only produce output to
rocksdb/kafka when either that cache fills up or the commit period occurs.
So the changelog for the
This is great!
-Jay
On Fri, Apr 22, 2016 at 2:28 PM, Surendra , Manchikanti <
surendra.manchika...@gmail.com> wrote:
> Hi,
>
> I have implemented KafkaConnector for Solr, Please find the below github
> link.
>
> https://github.com/msurendra/kafka-connect-solr
>
> The initial release having SolrS
Is it possible to make the error message give more an explanation?
-Jay
On Wed, May 4, 2016 at 8:46 AM, Matthias J. Sax
wrote:
> Hi,
>
> I am still new to Kafka Streams by myself, but from my understanding if
> you change the key, your partitioning changes, ie, is not valid anymore.
> Thus, the
can Sands, Dustin Cote,
> Eamon Zhang, edoardo, Edward Ribeiro, Eno Thereska, Ewen
> Cheslack-Postava, Flavio Junqueira, Francois Visconte, Frank Scholten,
> Gabriel Zhang, gaob13, Geoff Anderson, glikson, Grant Henke, Greg
> Fodor, Guozhang Wang, Gwen Shapira, Igor Stepanov, Ishita Mand
Can you sanity check this with the end-to-end latency test that ships with
Kafka in the tools package?
https://apache.googlesource.com/kafka/+/1769642bb779921267bd57d3d338591dbdf33842/core/src/main/scala/kafka/tools/TestEndToEndLatency.scala
On Saturday, June 25, 2016, Kafka wrote:
> Hi all，
>
Even though I'm not aware of anyone working on it, I think we'd definitely
be open to it if someone wants to take a swing at it.
-Jay
On Sun, Jun 26, 2016 at 9:13 AM, Alex Glikson wrote:
> Hi all,
>
> I wonder whether there are plans to implement Apache Beam backend based on
> Kafka Streams?
>
I think you may get this for free as Kafka Streams attempts to align
consumption across different topics/partitions by the timestamp in the
messages. So in a case where you are starting a job fresh and it has a
database changelog to consume and a event stream to consume, it will
attempt to keep the
Can you describe the behavior you saw that you didn't like?
-Jay
On Mon, Aug 22, 2016 at 12:24 AM, Jan Filipiak
wrote:
> Hello everyone,
>
> I stumbled across KIP-33 and the time based index, while briefly checking
> the wiki and commits, I fail to find a way to opt out.
> I saw it having quite
-1
I think the REST server for Kafka that already exists is quite good and
getting contributions. Moving this into the core project doesn't solve a
problem that I see.
-Jay
On Tue, Oct 25, 2016 at 2:16 PM, Harsha Chintalapani
wrote:
> Hi All,
>We are proposing to have a REST Server
I think the most common cause of rebalancing is still GC that exceeds the
consumer liveness timeout you've configured. Might be worth enabling GC
logging in java and then checking the pause times. If they exceed the
timeout you have for liveness then you will detect that as a process
failure and re
Good question! Here's my understanding.
The streams API has a config num.standby.replicas. If this value is set to
0, the default, then the local state will have to be recreated by
re-reading the relevant Kafka partition and replaying that into the state
store, and as you point out this will take
+1
On Wed, Jan 11, 2017 at 10:56 AM, Ben Stopford wrote:
> Looks like there was a good consensus on the discuss thread for KIP-106 so
> lets move to a vote.
>
> Please chime in if you would like to change the default for
> unclean.leader.election.enabled from true to false.
>
> https://cwiki.apa
Congrats Grant!
-Jay
On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shapira wrote:
> The PMC for Apache Kafka has invited Grant Henke to join as a
> committer and we are pleased to announce that he has accepted!
>
> Grant contributed 88 patches, 90 code reviews, countless great
> comments on discussion
+1
Nice improvement.
-Jay
On Tue, Feb 14, 2017 at 1:22 PM, Steven Schlansker <
sschlans...@opentable.com> wrote:
> Hi, it looks like I have 2 of the 3 minimum votes, can a third voter
> please consider this KIP?
> Thanks.
>
> (PS - new revision on GitHub PR with hopefully the last round of
> im
Two things:
1. This is a minor thing but the proposed new name for KStreamBuilder
is StreamsTopologyBuilder. I actually think we should not put topology in
the name as topology is not a concept you need to understand at the
kstreams layer right now. I'd think of three categories of con
pe for this KIP. KIP-120 has the focus on removing leaking
> internal APIs and do some cleanup how our API reflects some concepts.
>
> However, I added your idea to API discussion Wiki page and we take if
> from there:
> https://cwiki.apache.org/confluence/display/KAFKA/
> Kafka+S
think
> > > >>>>> about good names. It's just to discuss the pattern.
> > > >>>>>
> > > >>>>>
> > > >>>>>
> > > >>>>> -Matthias
> > > >
Congrats Rajini!
On Mon, Apr 24, 2017 at 2:06 PM, Gwen Shapira wrote:
> The PMC for Apache Kafka has invited Rajini Sivaram as a committer and we
> are pleased to announce that she has accepted!
>
> Rajini contributed 83 patches, 8 KIPs (all security and quota
> improvements) and a significant n
+1
On Tue, May 9, 2017 at 3:41 PM BigData dev wrote:
> Hi, Everyone,
>
> Since this is a relatively simple change, I would like to start the voting
> process for KIP-156: Add option "dry run" to Streams application reset tool
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=69
This is great!
-Jay
On Sat, May 27, 2017 at 12:47 PM, Michal Borowiecki <
michal.borowie...@openbet.com> wrote:
> Hi all,
>
> I've updated the wiki page with a draft pattern for consecutively growing
> time-windowed aggregations which was discussed some time ago on this
> mailing list.
>
> I'm y
I think setting max.block.ms=0 does what you want.
-Jay
On Sat, May 27, 2017 at 12:40 PM, Abhimanyu Nagrath <
abhimanyunagr...@gmail.com> wrote:
> HI Hans,
>
> What exactly I meant by asynchronous is that when my Kafka broker is down
> and I am trying to produce the message . It is getting stuck
Hey Eno,
I think this makes sense. I do think people who spend time running
production stream processing systems will, over time, end up strongly
preferring the current behavior of failing and fixing the root problem
rather than skipping, but we don't need to force this on people as long as
the de
.util.Try[Int] =
>>>>>> Failure(java.lang.ArithmeticException: / by zero)
>>>>>>
>>>>>> scala> computation.getOrElse(42)
>>>>>> res2: Int = 42
>>>>>>
>>>>>> Another example wi
For anyone that's interested, I did a blog post on logs, stream processing,
and distributed systems:
http://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying
FYI...
-Jay
-- Forwarded message --
From: Jakob Homan
Date: Tue, Jan 14, 2014 at 2:09 PM
Subject: Upcoming Apache Samza talk in Seattle, January 22
To: d...@samza.incubator.apache.org, users@kafka.apache.org
Hey all-
Anyone in the Seattle area interested in Apache Samza shou
As mentioned in a previous email we are working on a re-implementation of
the producer. I would like to use this email thread to discuss the details
of the public API and the configuration. I would love for us to be
incredibly picky about this public api now so it is as good as possible and
we don'
;
> --
> Andrey Yegorov
>
>
> On Fri, Jan 24, 2014 at 11:54 AM, Jay Kreps wrote:
>
> > As mentioned in a previous email we are working on a re-implementation of
> > the producer. I would like to use this email thread to discuss the
> details
> > of t
r per thread and
> reuse it. Since kafka producer already have messages in the batch and knows
> the batch that failed, it can pass the message to the onError() callback.
>
> Am I over-thinking this?
>
>
> ------
> Andrey Yegorov
>
>
> On Fri, Jan 24, 2014 at 1:15 PM,
re pushing the first message through.
>
> Should there also be a KafkaProducer.connect() or .open() method or
> connectAll()? I guess it would try to connect to all brokers in the
> BROKER_LIST_CONFIG
>
> HTH,
>
> Roger
>
>
> On Fri, Jan 24, 2014 at 11:54 AM,
t; 2) It looks like broker connections are created on demand. I'm wondering
> > if sometimes you might want to flush out config or network connectivity
> > issues before pushing the first message through.
> >
> > Should there also be a KafkaProducer.connect() or .open
>
> Cheers,
>
> Roger
>
>
> On Fri, Jan 24, 2014 at 4:13 PM, Jay Kreps wrote:
>
> > Roger,
> >
> > These are good questions.
> >
> > 1. The producer since 0.8 is actually zookeeper free, so this is not new
> to
> > this client it is true f
e a custom serializer via passing it into
> the producer constructor so I feel like having to include my own every time
> is a step backwards. I might be misunderstanding this.
>
> Joe Lawson
>
> On Jan 24, 2014 2:54 PM, Jay Kreps wrote:
> As mentioned in a previous email we are w
mer API.
> >
> > 2) It looks like broker connections are created on demand. I'm wondering
> > if sometimes you might want to flush out config or network connectivity
> > issues before pushing the first message through.
> >
> > Should there also be a Kafka
pache Kafka project and whether something is from
> another org and related to Kafka. That said, nothing really enforces it.
>
> Futures: I'll see if I can create some examples to demonstrate Future
> making interop easier.
>
> Regards,
> C
>
>
>
>
> On Fri,
cted and it's
> pretty free in most IDE's. The advantages I see is that it is clear whether
> something is from the Apache Kafka project and whether something is from
> another org and related to Kafka. That said, nothing really enforces it.
>
> Futures: I'll see if I ca
ating because what are you supposed to do then,
close again?
4. From twitter: "The name Callback is too generic, make it
CompleteCallback." I actually prefer generic names (Producer, Consumer,
etc). Hopefully the package should avoid any ambiguity?
On Sun, Jan 26, 2014 at 9:21 PM, Jay Kreps
Clark,
Yeah good point. Okay I'm sold on Closable. Autoclosable would be much
better, but for now we are retaining 1.6 compatibility and I suspect the
use case of temporarily creating a producer would actually be a more rare
case.
-Jay
On Mon, Jan 27, 2014 at 9:29 AM, Clark Breyman wrote:
> r
; > > stopped using property files. They are lingua franca. The only thought
> > > might be to separate the config interface from the implementation to
> > allow
> > > for alternatives, but that might undermine your point of "do it this
> way
> > so
>
Hmmm, I would really strongly urge us to not introduce a zk dependency just
for discovery. People who want to implement this can certainly do so by
simply looking up urls and setting them in the consumer config, but our
experience with doing this at large scale was pretty bad. Hardcoding the
discov
t; generated. You can register as many callbacks as you like, each in the
> appropriate layer of the code and have each observer get notified when the
> promised i/o is complete without any of them knowing about each other.
>
>
> On Tue, Jan 28, 2014 at 11:32
rated. You can register as many callbacks as you like, each in the
> > appropriate layer of the code and have each observer get notified when
> the
> > promised i/o is complete without any of them knowing about each other.
> >
> >
> > On Tue, Jan 28, 2014 at 11:32 AM, Jay Krep
uggested as part of
> option 1A. I like this approach since it maintains the simplicity of APIs
> by allowing us to deal with bytes and does not compromise performance in
> the custom partitioning case.
>
> Thanks,
> Neha
>
>
>
> On Tue, Jan 28, 2014 at 5:42 PM, Jay Kreps
can send individual produce requests (singular
> or batched). The "producer" layer adds an additional queue for each
> partition, allowing individual messages to be batched together.
>
> --Tom
>
>
>
> On Tue, Jan 28, 2014 at 9:10 PM, Jay Kreps wrote:
>
> >
ed
> exception, rather than wrap the useful ProduceRequestResult in a custom
> async object (RecordSend) and explain that to our many users.
>
> Thanks,
> Neha
>
>
>
>
> On Tue, Jan 28, 2014 at 8:10 PM, Jay Kreps wrote:
>
> > Hey Neha,
> >
> > Can
() operation would
>> always return false and mean nothing. But we can mention that caveat in
>> our
>> Java docs.
>>
>> To summarize, I would prefer to expose a well understood and widely
>> adopted
>> Java API and put up with the overhead of catching one
if the API under the hood?
>
> > On Jan 29, 2014, at 10:15, Jay Kreps wrote:
> >
> > The challenge of directly exposing ProduceRequestResult is that the
> offset
> > provided is just the base offset and there is no way to know for a
> > particular message where it w
I think this is a good point and you are not the first person to bring it
up.
I am not hugely knowledgable about java 8 so any feedback would be useful.
In the producer I think the biggest impact is that the Callback can be
implemented with a lambda instead of a anon class which will be much nice
I think this may be more a general java thing. Can you try running any java
class with the same command line options you are using for kafka and
confirm that that also doesn't work.
-Jay
On Thu, Jan 30, 2014 at 11:23 AM, David Montgomery <
davidmontgom...@gmail.com> wrote:
> Hi, <%20us...@kafka.
Yeah this is confusing.
What they are trying to say is that adding partitions doesn't move data
that is already in existing partitions. I.e. say you have 10 partitions
(0-9) each containing 1GB of data and you add an 11th partition. When this
is complete partitions 0-9 will contain the exact same
tream-processing use cases that read from one topic and write to
another. We would have to include rack information in our metadata.
Having this kind of functionality included is actually kind of nice.
-Jay
On Fri, Jan 24, 2014 at 5:17 PM, Jay Kreps wrote:
> Clark and all,
>
> I thought a li
t;
> Thanks,
>
> Jun
>
>
> On Fri, Jan 24, 2014 at 5:17 PM, Jay Kreps wrote:
>
> > Clark and all,
> >
> > I thought a little bit about the serialization question. Here are the
> > options I see and the pros and cons I can think of. I'd love t
> a producer config:
> default.partitioner.strategy="minimize-connections"/"roundrobin" - and
> so on; and implement those partitioners internally in the producer.
> Not as clear as a .class config, but it accomplishes the same effect
> no?
>
> On Thu, Jan 30
Oliver,
Yeah that was my original plan--allow the registration of multiple
callbacks on the future. But there is some additional implementation
complexity because then you need more synchronization variables to ensure
the callback gets executed even if the request has completed at the time
the cal
r -xzf kafka-#{version}-src.tgz
> > > cd kafka-#{version}-src
> > > ./sbt update
> > > ./sbt package
> > > ./sbt assembly-package-dependency
> > > echo 'KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote=true
> > > -Dcom.
l have to be used from outside of the core
> API.
>
> This design would make the core API less confusing (when do I use a
> partiton key instead of a partition index, does the key overwrite the
> index, can the key be null, etc...?) while still providing the flexibility
> you want.
We probably need to add a small amount of logging in the new producer and
(soon) consumer clients. I wanted to have a quick discussion on logging
libraries before I start adding this in the producer.
Previously we have been pretty verbose loggers and I think we should stop
that. For clients you mo
301 - 400 of 495 matches
Mail list logo
