The docs say: “Each task is assigned to a thread. Each task is capable of
handling multiple Kafka partitions, but a single partition must be handled by
only one task.”From what I understand additional tasks would sit idle.
From: Yeikel Santana
Date: Thursday, May 30, 2024 at 7:43 AM
To:
Consider purchasing support from Confluent to get this sort of request answered
quickly.
From: Sahil Sharma D
Date: Tuesday, May 9, 2023 at 12:40 PM
To: users@kafka.apache.org
Subject: [EXTERNAL] RE: CVEs related to Kafka
Gentle reminder-2 !
-Original Message-
From: Sahil Sharma D
Se
I don’t think Streaming is a good fit for those use cases. It is best for
asynchronous processes. For example, once something is hotel room is booked
and paid then send a message to update the rewards account. But if you need
to gate access to something (like if a room is booked or not) then
FOSS == Free Open Source Software
From: andrew davidson
Date: Wednesday, March 30, 2022 at 3:16 PM
To: users@kafka.apache.org
Subject: [EXTERNAL] Re: Newbie looking for a connector I can configure on my mac
Thanks Liam.
What is 'FOSS Kafka'? google did not find any useful definitions
A tutoria
From: Jatin Chhabriya
Date: Wednesday, March 16, 2022 at 9:20 AM
To: users@kafka.apache.org
Cc: Murali Krishna
Subject: [EXTERNAL] Apache Kafka Questions
Hello Team
Upon careful perusal of documentation and tutorials, our team has a few open
questions, and we would appreciate to have these c
PR means “Pull Request”. It is a way to have others review your code changes
and, when ready, they can merge them in.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/about-pull-requests
From: Andreas Gillmann
Date: Mon
ingly to do that
> work.
> I'm so sorry that I can't help.
>
> Best regards
> Franziska
>
> -Ursprüngliche Nachricht-
> Von: Tauzell, Dave
> Gesendet: Montag, 10. Januar 2022 14:30
> An: users@kafka.apache.org
> Betreff: Re: Log4j 1.2
>
>
Log4j 2.x isn’t a drop-in replacement for 1.x. It isn’t a difficult change
but somebody does need to go through all the source code and do the work.
-Dave
From: Brosy, Franziska
Date: Monday, January 10, 2022 at 3:16 AM
To: users@kafka.apache.org
Subject: [EXTERNAL] AW: Log4j 1.2
Hi Roger,
consumer publish notifications about messages
it has processed to a new topic (or other storage mechanism).
You may be able to use the admin api, but I don't think it's a standard use
case.
On Tue, May 25, 2021, 8:21 AM Tauzell, Dave
wrote:
> I don’t know about monitoring when
I don’t know about monitoring when a particular message is reads but you can
use something like https://github.com/linkedin/Burrow to monitor consumer lag.
Basically you can see that consumer Y has not yet read X number of messages
that are ready.
-Dave
From: Alberto Moio
Date: Tuesday, May
When you send a message the returned RecordMetadata has an offset. If you
know the name of the Consumer Group that is reading these messages there is an
API to get the latest offset of the Consumer Group so you could poll that and
wait for the latest Offset to be greater than the offset of you
Your consumer will need to put some sort of response message on a different
topic and the API will have to subscribe and wait for that response message.
You'll need some sort of key so that it can know which one was for that call.
I would avoid this type of API, however. Instead I would create
So if the stream is:
A:1:FOO
A:3:BAR
A:3:BAZ
Then A:3* must be processed after A:1 but A:3:BAR and A:3:BAZ can be processed
in any order?
I don’t think there is a way to do that with topics.
-Dave
From: Andre Mermegas
Reply-To: "users@kafka.apache.org"
Date: Wednesday, September 2, 2020 at
When clients connect the Kafka broker assigns each client a subset of the topic
partitions. When a client becomes unresponsive then the broker assigns those
partitions to another client.
The state of your connector task isn't going to be transferred but another
connector task will eventually
If you run in the cloud I think there are some solutions. If you think you
might need to add brokers in the future then make sure you create enough
partitions ahead of time. It is easier to add brokers and have them take over
existing partitions than it is to re-partition data.
I don't think
I would go with #1:
1. It will be easier to add new "batch producers" since you won't need to worry
about re-partitioning
2. You have more control over the parallelism since you can have different
numbers of partitions for each topic
3. You can easily split out your consumer into N consumers if
e:
Hi Dave,
thank you . saw some tutorial where they told it otherwise .. which
confuses me a litte.
If its done round-robin .. my "world view" makes sense again 😊
Oliver
-Ursprüngliche Nachricht-----
Von: Tauzell, Dave
Gesendet:
A null key results in the client sending to partitions in a round-robin order.
Use a key if you want to ensure that specific messages end up on the same
partition.
-Dave
On 11/8/19, 1:06 AM, "Oliver Eckle" wrote:
Hi,
Don’t get me wrong, I just want to understand what's going on.
If somebody insists on using Kafka as a database you might be able to do the
following:
1. Create a "compacted topic". The key for the topic should be the point of
sales id.
2. Create a webservice which takes a point of sale id and can read or update
the topic
3. Have the point of sale apps
We are using both and leaning towards a web service fronting Kafka because it
gives us the ability to centralize other logic. That said, I don't think the
webservice will be much more "stable" and you'll need to consider what to do
with your audit records if the web service call fails.
-Dave
It is possible that if all the nodes fail at about the same time and after the
broker acknowledged the message, then some messages will be lost because they
were in memory and not yet fully written to the disk. If you set ACKS=all
then this requires all of your replicas to fail in this way to
If you size your cluster right, you can send large messages of many megabytes.
We send lots (millions per day) of medium sized messages (5-10k) without any
issues.
-Dave
-Original Message-
From: Chanchal Chatterji [mailto:chanchal.chatte...@infosys.com]
Sent: Wednesday, September 12, 2
We use Jolokia (which has a java agent you can load with kafka to expose
metrics via HTTP) and Influx/Telegraf which has support for Jolokia. There is
a fair bit of configuration but it can be done without any coding.
-Dave
-Original Message-
From: Ted Yu [mailto:yuzhih...@gmail.com]
What does the hardware side of your brokers look like - do you have enough
memory to hold all pending messages in memory (i.e. before consumers get them).
At what rate are your clients trying to send messages?
-Dave
-Original Message-
From: Pritam Kadam [mailto:kpri...@thoughtworks.com]
Does anybody have any experience with Confluent Replicator? Has it worked
well for you?
-Dave
This e-mail and any files transmitted with it are confidential, may contain
sensitive information, and are intended solely for the use of the individual or
entity to whom they are addressed. If you
I would have a cron that runs every day but somehow tracks if it has pulled
data for the month. If it has it just does nothing. This way if you have some
sort of failure one day (website is down, etc ...) it would pull data the next
day.
You could possibly use Kaka itself to store the last mo
Whatever you use I recommend some sort of wrapper since Kafka doesn't support
any sort of metadata (like the version of the serialization format).
-Dave
-Original Message-
From: Matt Farmer [mailto:m...@frmr.me]
Sent: Thursday, January 11, 2018 8:56 AM
To: users@kafka.apache.org
Subject:
If you haven’t built in logic from the start (with micro-service version 1)
then I think you’ll need some sort of “router” in the middle that knows the
routing logic.
-Dave
From: Assaf Katz [mailto:assaf.k...@amdocs.com]
Sent: Wednesday, December 13, 2017 3:12 AM
To: Yuval Alon ; users@kafka.a
You then also need to set this up for each topic you create:
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor
> 3 --partitions 3 --topic my-replicated-topic
-Dave
-Original Message-
From: Skip Montanaro [mailto:skip.montan...@gmail.com]
Sent: Thursday, Nove
If you create a partitioned topic with at least 3 partitions then you will see
your client connect to all of the brokers. The client decides which partition
a message should go to and then sends it directly to the broker that is the
leader for that partition. If you have replicated topics, the
What are you going to do with the messages every 15 minutes?
One way I can think of is to have two consumers in your application. One of
them reads messages and just keeps track of the offsets for each hour. The
other consumer then uses this info to pull the data. You could publish these
off
Have you tried increasing max.in.flight.requests.per.connection? I wonder if
that would be similar to you having multiple producers.
Dave
Sent using OWA for iPhone
From: Sunny Kim
Sent: Wednesday, August 30, 2017 4:55:02 PM
To: users@kafka.apache.org
Su
I don't think that is possible since Kafka uses the file system cache for this.
-Dave
-Original Message-
From: Archie [mailto:anubhavnidhi1...@gmail.com]
Sent: Monday, August 28, 2017 4:14 PM
To: users@kafka.apache.org
Subject: Re: Is it possible to disable caching for some kafka topics?
, all avro messages holds same id. Then how multiple schemas on same topic
possble ?
Please clarify
Thanks,
Sreejith
On 17-Aug-2017 9:49 pm, "Tauzell, Dave"
wrote:
> > How does consumer know A is the avro class when there could be
> > other
> classes like B,C and D d
ginal Message-----
From: Tauzell, Dave [mailto:dave.tauz...@surescripts.com]
Sent: Thursday, August 17, 2017 8:30 AM
To: users@kafka.apache.org
Subject: RE: Different Schemas on same Kafka Topic
It does. The way it works is that the Avro serializer precedes each message
with a two-byte integer that
It does. The way it works is that the Avro serializer precedes each message
with a two-byte integer that references a schema id in the confluent schema
registry. The Avro deserializer looks at this value to determine which schema
to de-serialize with. In order for this to work you need to u
What sort of skew do you expect. For example do you expect one key to have
1000x as many messages as others?
The consumer API allows you to pick a partition. So if you know that you have
N partition groups then you could setup N consumers each pull from one
partition in the group. You could
I don't have any concrete numbers but the REST proxy is quite a bit slower.
That said, it can still be fast and can scale out so it might meet your needs.
-Dave
-Original Message-
From: Affan Syed [mailto:as...@an10.io]
Sent: Thursday, August 10, 2017 1:32 AM
To: users@kafka.apache.org
As others mentioned this is not a forum to discuss the works of Franz Kafka.
Here are some places to get you started:
1. The works of Franz Kafka:
https://www.vanderbilt.edu/olli/class-materials/Franz_Kafka.pdf
2. Literature stack exchange: https://literature.stackexchange.com/
-Dave
>> java.lang.NoClassDefFound Error
You are missing some dependent classes. Two questions:
1. Does the message have more information about what class it couldn't find?
2. What exactly are you putting into your jar file?
-Dave
-Original Message-
From: Rahul R04 [mailto:rahul.kuma...@mph
All the brokers write to server.log. The broker that happens to be the
controller will also write to the controller.log file.
-Dave
-Original Message-
From: karan alang [mailto:karan.al...@gmail.com]
Sent: Wednesday, June 28, 2017 6:04 PM
To: users@kafka.apache.org
Subject: Kafka logs
Losing one out of three should not impact the cluster. Losing more than a
majority means certain Kafka operations won't work. Anything that requires the
zookeeper data like electing a new leader for example.
Dave
Sent using OWA for iPhone
From: mayank r
I’m not really familiar with Netty so I won’t be of much help. Maybe try
posting on a Netty forum to see what they think?
-Dave
From: SenthilKumar K [mailto:senthilec...@gmail.com]
Sent: Wednesday, June 21, 2017 10:28 AM
To: Tauzell, Dave
Cc: users@kafka.apache.org; senthilec...@apache.org; d
seems possible with the right sort of kafka producer tuning.
-Dave
From: SenthilKumar K [mailto:senthilec...@gmail.com]
Sent: Wednesday, June 21, 2017 8:55 AM
To: Tauzell, Dave
Cc: users@kafka.apache.org; senthilec...@apache.org; d...@kafka.apache.org;
Senthil kumar
Subject: Re: Handling 2 to 3
What are your configurations?
- production
- brokers
- consumers
Is the problem that web servers cannot send to Kafka fast enough or your
consumers cannot process messages off of kafka fast enough?
What is the average size of these messages?
-Dave
-Original Message-
From: SenthilKumar
Lots of large messages will slow down throughput. From the client side you
might want to have a client for large messages and one for the others so that
they each have their own queue.
-Dave
-Original Message-
From: Ghosh, Achintya (Contractor) [mailto:achintya_gh...@comcast.com]
Sent:
Sounds like there are some issues using the Kafka java library on Android. I
think instead you should create a REST api (or use the REST proxy provided by
Confluent) and have your device make HTTP calls to something that then puts
messages onto Kafka.
-Dave
-Original Message-
From: Mi
o the disk.
It's somewhat not clear to me in the documentation, whether
log.flush.interval.messages is connected to the ACK sent to the client.
____
From: Tauzell, Dave
Sent: Tuesday, May 30, 2017 5:08:04 PM
To: users@kafka.apache.org
Subject: RE: client recordm
>>If kafka client producer gets record meta data with a valid offset, do we
>>consider that that message is indeed fsynced to the disk ?
No, it doesn't.The meaning depends on your configuration
(https://www.cloudera.com/documentation/kafka/latest/topics/kafka_ha.html).
To increase the dura
Both Confluent and Cloudera provide support.
-Dave
From: Benny Rutten [mailto:brut...@isabel.eu]
Sent: Wednesday, April 26, 2017 2:36 AM
To: users@kafka.apache.org
Subject: Kafka 24/7 support
Good morning,
I am trying to convince my company to choose Apache Kafka as our standard
messaging syst
I think because the product batches messages which could be for different
topics.
-Dave
-Original Message-
From: Nicolas MOTTE [mailto:nicolas.mo...@amadeus.com]
Sent: Wednesday, March 8, 2017 2:41 PM
To: users@kafka.apache.org
Subject: Performance and Encryption
Hi everyone,
I underst
Also, see this article on streaming changes from MySQL to kafka:
https://wecode.wepay.com/posts/streaming-databases-in-realtime-with-mysql-debezium-kafka
-Original Message-
From: Tauzell, Dave
Sent: Monday, February 27, 2017 9:07 AM
To: users@kafka.apache.org
Subject: RE: Kafka Connect
Are you specifically talking about relational databases?Kafka Connect has
a JDBC source
(http://docs.confluent.io/3.1.1/connect/connect-jdbc/docs/source_connector.html)
which can push data changes to kafka. It can only run sql queries, though, so
out of the box it will just get you update
You'll need to provide some details. At a minimum the error message that you
are getting.
-Dave
-Original Message-
From: VIVEK KUMAR MISHRA 13BIT0066 [mailto:vivekkumar.mishra2...@vit.ac.in]
Sent: Friday, February 10, 2017 4:22 AM
To: users@kafka.apache.org
Subject: about producer and c
Yes, you just need to point it to your cluster.
-Dave
-Original Message-
From: Guillermo Ortiz [mailto:konstt2...@gmail.com]
Sent: Wednesday, February 1, 2017 1:09 PM
To: users@kafka.apache.org
Subject: Kafka Connect in different nodes than Kafka.
Is it possible to use Kafka Connect in n
ious
> > >>> > email, I see GC running roughly every 10-12 seconds, with
> > >>> > total
> times
> > >>> > similar to the following:
> > >>> >
> > >>> > 2017-01-12T07:16:46.867-0500: 46891.844: Total time for which
>
When you say "high transaction" what are your estimates? ActiveMQ will be
easier to get started with. If you design your code to hide the queuing it
won't be too hard to switch between them. Kafka provides (I think) more
scalability and HA but is going to more complicated to operate if you w
-XX:MaxPermSize=48M -verbose:gc
>>> > -Xloggc:/var/log/kafka/gc.log -XX:+PrintGCDateStamps
>>> -XX:+PrintGCDetails
>>> > -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime
>>> > -XX:+PrintTLAB -XX:+DisableExplicitGC -XX:+UseGCLogFileRotati
You can set the retention for the topic to a small time and then wait for Kafka
to delete the messages before setting it back:
bin/kafka-topics.sh --zookeeper zk.prod.yoursite.com --alter --topic TOPIC_NAME
--config retention.ms=1000
-Original Message-
From: Laxmi Narayan NIT DGP [mailt
Can you collect garbage collection stats and verify there isn't a long GC
happening at the same time?
-Dave
-Original Message-
From: Stephen Powis [mailto:spo...@salesforce.com]
Sent: Thursday, January 12, 2017 8:34 AM
To: users@kafka.apache.org
Subject: Re: Taking a long time to roll a
Can you explain in more detail? Do you want to have files created in hdfs
somehow broken into records and put into Kafka?
> On Jan 9, 2017, at 19:57, Cas Apanowicz wrote:
>
> Hi,
>
> I have general understanding of main Kafka functionality as a streaming tool.
> However, I'm trying to figure out
Kafka persists messages to disk (would be SSD if that is what you have).
However, if you have a large enough memory and are pulling off messages
quickly, then the receivers will likely get the messages directly from memory
and the write to disk will happen asynchronously in the background.
If you specify a key with each message then all messages with the same key get
sent to the same partition.
> On Dec 26, 2016, at 23:32, Ali Akhtar wrote:
>
> How would I route the messages to a specific partition?
>
>> On 27 Dec 2016 10:25 a.m., "Asaf Mesika" wrote:
>>
>> There is a much easier
What is the plan for backup and recovery of the kafka data?
-Dave
-Original Message-
From: Susheel Kumar [mailto:susheel2...@gmail.com]
Sent: Thursday, December 15, 2016 12:00 PM
To: users@kafka.apache.org
Subject: Kafka as a database/repository question
Hello Folks,
I am going thru an
I don't know if any API to stream a message. I don't suggest putting lots of
large messages onto Kafka.
As far as documentation I hear that confluent is going to support a C and C#
client so you could try asking questions on the confluent mailing list.
Dave
On Dec 5, 2016, at 17:51, Doyle, Ke
Can you use the console consumer to see the messages on the other topics?
> On Dec 2, 2016, at 04:56, Vincenzo D'Amore wrote:
>
> Hi Kafka Gurus :)
>
> I'm creating process between few applications.
>
> First application create a producer and then write a message into a main
> topic (A), within t
topic and aggregated to a table.
Keyed message will be around 1 KB or so.
On Thu, Dec 1, 2016 at 9:44 PM, Tauzell, Dave
wrote:
> Do you have some idea of the size and number of messages per second
> you'll put onto the topics at peak?
>
> -Dave
>
> -Original M
h the production server architecture
And what about my brokers. Should I hedge them as well.
Like say put 2 zk on nodejs server and 1 on db server.
Put 2 brokers on db server and 1 on nodejs server, something like that.
Thanks
Sachin
On Thu, Dec 1, 2016 at 8:59 PM, Tauzell, Dave
wrote:
>
For low volume zookeeper doesn't seem to use many resources. I would put it
on nodejs server as that will have less IO and heavy IO could impact zookeeper.
Or, you could put some ZK nodes on nodejs and some on DB servers to hedge your
bets. As always, you'll find out a lot once you actually
Kafka doesn't have the concept of message headers like some other messaging
systems.
You will have to create a payload that contains these headers and whatever
bytes you are sending.
Dave
> On Nov 28, 2016, at 16:47, Prasad Dls wrote:
>
> Hi,
>
> While publishing each message (single message
.
If you have performance numbers you can share for the large messages, I think
we'll all appreciate :)
On Tue, Nov 22, 2016 at 1:04 PM, Tauzell, Dave
wrote:
> I ran tests with a mix of messages, some as large as 20MB. These large
> messages do slow down processing, but it still works
e 4.x.x
>
> On Sun, Nov 27, 2016 at 8:41 AM, Tauzell, Dave > wrote:
>
>> It looks like you are missing a spring jar. Can you google to find out
>> which jar that class is in?
>>
>> Dave
>>
>>> On Nov 27, 2016, at 01:16, Prasad Dls wrote:
>>&g
It looks like you are missing a spring jar. Can you google to find out which
jar that class is in?
Dave
> On Nov 27, 2016, at 01:16, Prasad Dls wrote:
>
> Hi users,
>
>
> My project is already developed with Spring 3.0.5.RELEASE, We are planning
> to use Kafka for new requirements. I am trying
I ran tests with a mix of messages, some as large as 20MB. These large
messages do slow down processing, but it still works.
-Dave
-Original Message-
From: h...@confluent.io [mailto:h...@confluent.io]
Sent: Tuesday, November 22, 2016 1:41 PM
To: users@kafka.apache.org
Subject: Re: Over
Do you have:
Unclean.leader.election.enable = false ?
Dave
> On Nov 17, 2016, at 19:39, Mark Smith wrote:
>
> Hey folks,
>
> I work at Dropbox and I was doing some maintenance yesterday and it
> looks like we lost some committed data during a preferred replica
> election. As far as I understand
Partitions are used to distribute the messages in a topic between several
different broker instances. This provides higher throughput. Partitions can
also be replicate which allows for high availability.
-Dave
From: Doyle, Keith [mailto:keith.do...@greenwayhealth.com]
Sent: Wednesday, Novemb
Here is a scenario where this could be useful:
Add the kafka offset as a field on the record in both Cassandra and
Elasticsearch
Now when you get search results from Elastic search and look up details in
Cassandra you can know if they come from the same kafka record. If you can
use the of
You should have one consumer pull the message and submit the data to each
storage using an XA transaction.
> On Nov 5, 2016, at 19:49, kant kodali wrote:
>
> yes this problem can definetly be approached in many ways but given the
> hard constraints by our clients we don't seem to have many optio
Is Kafka connect adding some bytes to the beginning of the avro with the scheme
registry id?
Dave
> On Nov 2, 2016, at 18:43, Will Du wrote:
>
> By using the kafka-avro-console-consumer I am able to get rich message from
> kafka connect with AvroConvert, but it got no output except schema from
You want the servers in the primary zone to put messages onto Kafka and
applications in the edge nodes to read and process them?
-Dave
This e-mail and any files transmitted with it are confidential, may contain
sensitive information, and are intended solely for the use of the individual or
enti
ut the
brokers before the message is gonna be read/written.
Thank you for support!
On Mon, 31 Oct 2016 at 02:41 Tauzell, Dave
wrote:
> Once enough failures happen the circuit is marked open. The client
> would then periodically try some messages until it works again.
> Others would
Once enough failures happen the circuit is marked open. The client would then
periodically try some messages until it works again. Others would be failed.
There are a number of existing circuit breaker libraries you can use in the
meantime like the Netflix one.
Dave
> On Oct 30, 2016, at 20:
estamp based offset lookup is necessary because
> the offset numbers for a given message will not match in both datacenters.
>
> -hans
>
> On Oct 28, 2016, at 8:08 AM, Mudit Agarwal
>
> wrote:
>
> Thanks dave.
> Any ways for how we can achieve HA/Failover in k
I don't know of anything to handle that situation for you, but your application
can be written to do that.
-Dave
-Original Message-
From: Mudit Agarwal [mailto:mudit...@yahoo.com.INVALID]
Sent: Friday, October 28, 2016 11:08 AM
To: Tauzell, Dave; users@kafka.apache.org
Subjec
...@yahoo.com.INVALID]
Sent: Friday, October 28, 2016 10:09 AM
To: users@kafka.apache.org
Subject: Re: Kafka Multi DataCenter HA/Failover
Thanks dave.
Any ways for how we can achieve HA/Failover in kafka across two DC?
Thanks,Mudit
From: "Tauzell, Dave"
To: "users@kafka.apache.org&q
>> without any lag
You are going to have some lag at some point between datacenters.
I haven't used this but from taking to them they are working or have created a
replacement for MirrorMaker using the Connect framework which will fix a number
of MirrorMaker issues. I haven't talked to anybod
There is a script kafka-topics.sh which calls kafka.admin.TopicCommand. I
think you could probably call that same class from your application if you
figure out which jars are needed. When I do testing I'll clear out queues by
setting the retention time to a second and then wait for kafka to r
For similar queue related tests we put the check in a loop. Check every second
until either the result is found or a timeout happens.
-Dave
-Original Message-
From: Ali Akhtar [mailto:ali.rac...@gmail.com]
Sent: Wednesday, October 19, 2016 3:38 PM
To: users@kafka.apache.org
Subject: Ho
I don't see why you couldn't. InfluxDB's Telegraph
(https://docs.influxdata.com/telegraf/v1.0/) has plugin to read from Kafka.
You could use Spark or Kafka streams to do custom aggregation.
-Dave
-Original Message-
From: Furkan KAMACI [mailto:furkankam...@gmail.com]
Sent: Sunday, Oct
Does anybody know if the librdkafka releases are kept in step with kafka
releases?
-Dave
This e-mail and any files transmitted with it are confidential, may contain
sensitive information, and are intended solely for the use of the individual or
entity to whom they are addressed. If you have rec
Spark Streaming needs to store the output somewhere. Cassandra is a possible
target for that.
-Dave
-Original Message-
From: Ali Akhtar [mailto:ali.rac...@gmail.com]
Sent: Thursday, September 29, 2016 9:16 AM
Cc: users@kafka.apache.org; spark users
Subject: Re: Architecture recommendati
The current converters want you to send Avro records with a "schema id"
prepended to the serialized Avro. You also need the schema registry running.
I'm guessing this is what Olivier is talking about.
I think it is possible to write your own converter that doesn't need this but
I haven't tri
If by "sync" you mean "fsync" then, no it does not. There are some properties:
log.flush.interval.messages
log.flush.interval.ms
In theory you could set log.flush.interval.messages to 1 to fsync with each
write. I haven't tried this to see what happens but I expect performance will
drop quit
Kafka writes each message but the OS is writing those to in memory disk cache.
Kafka periodically calls fsync() to tell the OS to force the disk cache to
actual disk. Kafka gets high availability by replicating messages to other
brokers so that the messages are in-memory on several machines at
Are you using the new java consumer? What method are you using to commit
offsets?
-Dave
-Original Message-
From: Ghosh, Achintya (Contractor) [mailto:achintya_gh...@comcast.com]
Sent: Tuesday, September 20, 2016 8:56 AM
To: users@kafka.apache.org
Cc: d...@kafka.apache.org
Subject: Kafk
The send() method returns a Future. You need to get the result at some point
to see what happened. A simple way would be:
m_kafkaProducer.send(prMessage).get();
-Dave
-Original Message-
From: Agostino Calamita [mailto:agostino.calam...@gmail.com]
Sent: Friday, September 9, 2016 9:33 A
The leader for each partition is on a different broker.
Example:
Three brokers
Topic has three partitions and replication of three.
In this case each broker will be a leader for one partition and a follower for
two. Three consumers would each be reading from a different topic.
Dave
> On Sep 7
Does anybody do the following in production? If so, what are your experiences?
1. Use .Net applications for producers or consumers
2. Consume messages across the WAN (across datacenters) - I'm wondering
if MirrorMaker is always a requirement for cross-WAN
-Dave
This e-mail and any
I would write a python client that writes dummy data to kafka to measure how
fast you can write to Kafka without MongoDB in the mix. I've been doing load
testing recently can with 3 brokers I can write 100MB/s (using Java clients).
-Dave
-Original Message-
From: Dominik Safaric [mailto
Take a look at jmxtrans: https://github.com/jmxtrans. There are a number of
options, including some ones you can ebmed into your java application.
-Dave
-Original Message-
From: Lawrence Weikum [mailto:lwei...@pandora.com]
Sent: Monday, August 22, 2016 2:32 PM
To: users@kafka.apache.or
1 - 100 of 148 matches
Mail list logo
