Im a bit confused on the concept of a "message" in Kafka. How does this
differ, if at all, from a message in RabbitMQ? It seems to me that Kafka is
better suited for very write intensive "messages" like log data but RabbitMQ
may be a better fit for traditional "messages"… i.e. "Product Purchase
ant to
know If its suited for other types of messages as well. Are there certain
determine factors on why one would choose Kafka over RabbitMQ? Is it mostly
scale or is it the type of messages/events/logs being produced/consumed?
On Jun 7, 2013, at 5:21 PM, Alexis Richardson
wrote:
> On
to HDFS.
>
> -Jonathan
>
>
> On Fri, Jun 7, 2013 at 5:30 PM, Mark wrote:
>
>> Like I said, Im a bit confused. I see the terms "events", "messages" and
>> "logs" and not quite sure what to make of it.
>>
>> We are trying t
age in the pipeline. Typically though, messages are produces as soon as
> they are written.
>
>
> -Jonathan
>
>
> On Fri, Jun 7, 2013 at 6:06 PM, Mark wrote:
> Ok so in your use case instead of your application(s) writing directly to
> Kafka you instead have a separa
What if you messaging requirements are in the 100's GBSs? Would you say
RabbitMQ is probably a better fit?
On Jun 8, 2013, at 4:03 PM, Jonathan Hodges wrote:
> I am not making any assumptions other than Rabbit needs to maintain the
> state of the consumers. As the Kafka docs point out this is
I've read that LinkedIn uses Avro for their message serialization. Was there
any particular reason this was chosen say over something like Thrift or
ProtocolBuffers? Was the main motivating factor the native handling of Avro in
Hadoop?
Are all Kafka clients working with the latest version of Kafka?
I tried the kafka-rb client and a simple example listed in the README but I
keep getting a nasty error
require 'kafka'
producer = Kafka::Producer.new
message = Kafka::Message.new("some random message content")
producer.push(message)
ns it's not up to date to the protocol, I'm tried the
>> java client and it was working with latest 0.8 api.
>>
>> Not sure about any other languages.
>>
>> Tim
>>
>>
>> On Sat, Aug 10, 2013 at 2:55 PM, Mark wrote:
>>
>>> Ar
Can someone break down how message serialization would work with Avro? I've
read instead of adding a schema to every single event it would be wise to add
some sort of fingerprint with each message to identify which schema it should
used. What I'm having trouble understanding is, how do we read t
So basically prepend the information in the header of each message? Wouldn't
this require changes in all readers/writers?
On Aug 20, 2013, at 9:19 AM, Guozhang Wang wrote:
> Hello Mark,
>
> The Kafka message format starts with a magic byte indicating what kind of
> serializ
ion on what we are doing at LinkedIn:
> http://sites.computer.org/debull/A12june/pipeline.pdf
>
> This Avro JIRA has a schema repository implementation similar to the one
> LinkedIn uses:
> https://issues.apache.org/jira/browse/AVRO-1124
>
> -Jay
>
>
> On Tue, A
Ok someone answered a similar question in the Avro forum.
It *sounds* like that the Avro messages sent to Kafka are wrapped and/or
prepended with the SHA which is used by the consumer to lookup the schema. That
makes more sense.
On Aug 20, 2013, at 11:09 AM, Mark wrote:
> Thanks Jay I
Does LinkedIn include the SHA of the schema into the header of each Avro
message they write or do they wrap the avro message and prepend the SHA?
In either case, how does the Hadoop consumer know what schema to read?
ma in the repository and deserializes the
> message.
>
> Thanks,
> Neha
>
>
> On Wed, Aug 21, 2013 at 8:15 PM, Mark wrote:
>
>> Does LinkedIn include the SHA of the schema into the header of each Avro
>> message they write or do they wrap the avro message and pre
e
> if I can comment on the compatibility just yet.
>
> Thanks,
> Neha
>
>
> On Wed, Aug 21, 2013 at 9:00 PM, Mark wrote:
>
>> Neha, thanks for the response.
>>
>> So the only point of the magic byte is to indicate that the rest of the
>> message is Avro e
… or is the payload of the message prepending with a magic byte followed by the
SHA?
On Aug 22, 2013, at 9:49 AM, Mark wrote:
> Are you referring to the same message class as:
> https://github.com/apache/kafka/blob/0.7/core/src/main/scala/kafka/message/Message.scala
> or are you tal
I don't think it would matter as long as you separate the types of message in
different topics. Then just add more consumers to the ones that are slow. Am I
missing something?
On Aug 25, 2013, at 8:59 AM, Ian Friedman wrote:
> What if you don't know ahead of time how long a message will take t
I imagine when you start to have dozens of different consumers and multiple
consumers in each group this gets really complicated to manage. What are people
out there using to manage, start/stop and monitor their consumer groups? Anyway
to visualize the grouping and status of each consumer or con
Is it common/necessary to load balance the connections your producers use? I'm
looking at the Ruby gem and it doesn't look like this is handled via the gem.
lived?
On Aug 29, 2013, at 7:20 AM, Jun Rao wrote:
> I think a typical setup is that you have a load balancer in front of a
> bunch of frontend services, each of which has an embedded producer.
>
> Thanks,
>
> Jun
>
>
> On Wed, Aug 28, 2013 at 9:01 PM, Mark wrote:
>
Can you explain why not?
On Aug 29, 2013, at 8:43 AM, Benjamin Black wrote:
> The LB in front of the brokers doesn't make sense.
> On Aug 29, 2013 8:42 AM, "Mark" wrote:
>
>> We have a few dozen front-end web apps running rails. Each one of these
>> rail
If I have 3 brokers with 3 partitions does that mean:
1) I have 3 partitions per broker so I can have up to 9 consumers
or
2) There is only 1 partition per brokers which means I can have only 3 consumers
Thanks
How can one configure the number of partitions per topic in 0.7? I understand
0.8 has a command line tool to help with this. Is my only option setting a
global value for numPartition in server.properties?
Thanks
1) Should a producer be aware of which broker to write to or is this somehow
managed by Kafka itself. For example, If I have 2 brokers with a configured
partition size of 1 will my messages be written in a round-robin type of
fashion to each broker or is this completed dependent on which broker
Also, are the consumer offsets store in Kafka or Zookeeper?
On Aug 29, 2013, at 11:09 AM, Mark wrote:
> 1) Should a producer be aware of which broker to write to or is this somehow
> managed by Kafka itself. For example, If I have 2 brokers with a configured
> partition size of
I'm assuming for this auto-detection of partitions is only with the
ZookeeperConsumerConnector and not the ConsumerConnector?
On Aug 29, 2013, at 1:04 PM, Philip O'Toole wrote:
> On Thu, Aug 29, 2013 at 11:09 AM, Mark wrote:
>
>> 1) Should a producer be aware of which
We are thinking about using Kafka to collect events from our Rails application
and I was hoping to get some input from the Kafka community.
Currently the only gems available are:
https://github.com/acrosa/kafka-rb
https://github.com/bpot/poseidon (Can't use since we are only running 1.8.7)
Now
Can you view Kafka metrics via JConsole? I've tried connecting to port
with no such luck?
line to see the jmx port
> usually.
>
> -Suren
>
>
>
> On Thu, Aug 29, 2013 at 6:41 PM, Mark wrote:
>
>> Can you view Kafka metrics via JConsole? I've tried connecting to port
>> with no such luck?
>
>
>
>
> --
>
I should note this is Kafka 0.7
On Aug 29, 2013, at 3:59 PM, Mark wrote:
> I tried changing the ports and still no luck. Does it work with JConsole
> and/or do I need anything in my class path?
>
>
> On Aug 29, 2013, at 3:44 PM, Surendranauth Hiraman
> wrote:
>
&
org/jmxterm/tutorial
>
>
> On Thu, Aug 29, 2013 at 7:02 PM, Mark wrote:
>
>> I should note this is Kafka 0.7
>>
>> On Aug 29, 2013, at 3:59 PM, Mark wrote:
>>
>>> I tried changing the ports and still no luck. Does it work with JConsole
>> an
I needed to add the hostname to get it working..
-Djava.rmi.server.hostname=${HOSTNAME}
On Aug 29, 2013, at 4:47 PM, Mark wrote:
> Strange.. looks like that works. Not sure if because I am using that locally
> whereas jconsole and visualvm are remote
>
> On Aug 29, 2013
What is the quickest and easiest way to write message from Kafka into HDFS?
I've come across Camus but before we go the whole route of writing Avro
messages we want to test plain old vanilla messages.
Thanks
Our 0.7.2 Kafka cluster keeps crashing with:
2013-09-24 17:21:47,513 - [kafka-acceptor:Acceptor@153] - Error in acceptor
java.io.IOException: Too many open
The obvious fix is to bump up the number of open files but I'm wondering if
there is a leak on the Kafka side and/or our applicati
er client?
>
> Thanks,
>
> Jun
>
>
>> On Tue, Sep 24, 2013 at 5:33 PM, Mark wrote:
>>
>> Our 0.7.2 Kafka cluster keeps crashing with:
>>
>> 2013-09-24 17:21:47,513 - [kafka-acceptor:Acceptor@153] - Error in
>> acceptor
>>java.i
x27;t close the old
> ones.
>
> Thanks,
>
> Jun
>
>
> On Wed, Sep 25, 2013 at 6:08 AM, Mark wrote:
>
>> No. We are using the kafka-rb ruby gem producer.
>> https://github.com/acrosa/kafka-rb
>>
>> Now that you asked that question I need to ask. Is
FYI if I kill all producers I don't see the number of open files drop. I still
see all the ESTABLISHED connections.
Is there a broker setting to automatically kill any inactive TCP connections?
On Sep 25, 2013, at 4:30 PM, Mark wrote:
> Any other ideas?
>
> On Sep 25, 2013, a
>> Sent: Thursday, September 26, 2013 12:39 PM
>> To: users@kafka.apache.org
>> Subject: Re: Too many open files
>>
>> If a client is gone, the broker should automatically close those broken
>> sockets. Are you using a hardware load balancer?
>>
eptember 26, 2013 12:39 PM
> To: users@kafka.apache.org
> Subject: Re: Too many open files
>
> If a client is gone, the broker should automatically close those broken
> sockets. Are you using a hardware load balancer?
>
> Thanks,
>
> Jun
>
>
> On Wed, Sep
are using A10's
On Sep 26, 2013, at 6:41 PM, Nicolas Berthet wrote:
> Hi Mark,
>
> I'm using centos 6.2. My file limit is something like 500k, the value is
> arbitrary.
>
> One of the thing I changed so far are the TCP keepalive parameters, it had
&g
This is in regards to consumer group consumption in 0.7.2.
Say we have 3 machines with 3 partitions in each topic totaling 9 partitions.
Now if I create a consumer group with 9 threads on the same machine then all
partitions will be read from. Now what happens if I start another 9 threads on
a
igs here - http://kafka.apache.org/07/configuration.html
>
> Thanks,
> Neha
>
>
> On Wed, Oct 9, 2013 at 10:07 AM, Mark wrote:
>
>> This is in regards to consumer group consumption in 0.7.2.
>>
>> Say we have 3 machines with 3 partitions in each topi
a: Yes, absolutely.
-Mark
On Thu, Oct 23, 2014 at 3:08 AM, Po Cheung
wrote:
> Hello,
>
> We are planning to set up a data pipeline and send periodic, incremental
> updates from Teradata to Hadoop via Kafka. For a large DW table with
> hundreds of GB of data, is it okay (in terms o
I think it will depend on how your producer application logs things, but
yes I have historically seen exceptions in the producer logs when they
exceed the max message size.
-Mark
On Mon, Oct 27, 2014 at 10:19 AM, Chen Wang
wrote:
> Hello folks,
> I recently noticed our message amount in
Just to be clear: this is going to be exposed via some Api the clients can call
at startup?
> On Nov 12, 2014, at 08:59, Guozhang Wang wrote:
>
> Sounds great, +1 on this.
>
>> On Tue, Nov 11, 2014 at 1:36 PM, Gwen Shapira wrote:
>>
>> So it looks like we can use Gradle to add properties to
ery JMX
for the server version of every server in the cluster? Is there a reason
not to include this in the API itself?
-Mark
On Wed, Nov 12, 2014 at 9:50 AM, Joel Koshy wrote:
> +1 on the JMX + gradle properties. Is there any (seamless) way of
> including the exact git hash? That would b
g [queue] rethought as a
distributed commit log.
-Mark
On Tue, Jan 6, 2015 at 3:14 PM, Joseph Pachod
wrote:
> Hi
>
> Having read a lot about kafka and its use at linkedin, I'm still unsure
> whether Kafka can be used, with some mindset change for sure, as a general
> purpose data s
>
> I don't think there are any issues.
>
+1, I've been running Kafka with Java 7 for quite some time now and haven't
experienced any issues.
Regards,
Mark
On 2 February 2015 at 19:09, Otis Gospodnetic
wrote:
> I don't think there are any issues. We use 0.8.1
Hi Saravana,
Since 0.8.1 Kafka uses Gradle, previous to that SBT was used. Here is the
JIRA which the drove the change:
https://issues.apache.org/jira/browse/KAFKA-1171
The offical docs for the latest release (0.8.2) can be found here:
http://kafka.apache.org/documentation.html
Regards,
Mark
Wouldn't it be a better choice to store the logs offline somewhere? HDFS and S3
are both good choices...
-Mark
> On Feb 27, 2015, at 16:12, Warren Kiser wrote:
>
> Does anyone know how to achieve unlimited log retention either globally or
> on a per topic basis? I tried e
Hi,
I would like to subscribe to the Kafka mailing list for general questions.
Please let me know what I need to do in order to submit questions to the Kafka
general mailing list. Thanks.
Regards,
Mark Flores
Project Manager, Enterprise Technology
Direct206-576-2675
Email mark.flo
between each
cluster?
*How do we prevent a message published at one cluster being replicated
to another cluster and into an infinite loop (assuming we use a MirrorMaker
whitelist like Global.* at each cluster)?
Regards,
Mark Flores
Project Manager, Enterprise Technology
Direct206-576
/kafka/0.8.2.1/RELEASE_NOTES.html
Mark
On 13 March 2015 at 15:10, Marc Labbe wrote:
> Hi,
>
> our cluster is deployed on AWS, we have brokers on r3.large instances, a
> decent amount of topics+partitions (+600 partitions). We're not making that
> many requests/sec, roughly 80
I found the 0.8.2.0 and 0.8.2.1 has a KafkaConsumer. But this class seems
not completed and not functional. Lots of method returns null or throws
NSM. Which version of consumer for kafka 0.8.2 broker?
Thanks!
--
Best regards!
Mike Zang
eed to sort out, so this question is specific for the Kafka portion.
Note: My main consumer from Kafka will be STORM.
Regard
Mark.
Looks like 'at' got appended to the url. The below url should get to the page,
FetchRequestBuilder is at the bottom.
https://github.com/apache/kafka/blob/0.8/core/src/main/scala/kafka/api/FetchRequest.scala
On 14 Aug 2013, at 22:34, Thomas Edison
wrote:
> Thanks, but I might not have permiss
relationship."
http://www.quora.com/What-is-the-relation-between-Kafka-the-writer-and-Apache-Kafka-the-distributed-messaging-system
Regards,
Mark
On 20 May 2015 at 15:32, András Serény wrote:
> Hi All,
>
> I wonder, how the messaging system Kafka has got its name? I believe it
> w
.
I have done the usual Google searching, and followed many threads over the past
few days, but have drawn a blank.
Any thoughts where I should start to look?
Regards
Mark
---
Mark Whalley
Principal Consultant
Actian | Services
Accelerating Big Data 2.0
O +44 01
bs/kafka_2.10-0.8.1.2.2.0.0-2041-scaladoc.jar
I will also raise this under
http://hortonworks.com/community/forums/forum/kafka/
Regards
Mark
-Original Message-
From: Harsha [mailto:ka...@harsha.io]
Sent: 04 October 2015 16:42
To: users@kafka.apache.org
Subject: Re: Kafka Broker proces
ences from the
members of this group. What is the general best practice for reusing
topics or creating new ones? What has worked well in the past? What
should we be considering while making this decision?
Thanks in advance!
Mark.
call about the best approach to take.
Thanks,
Mark.
I usually approach this questions by looking at possible consumers. You
> usually want each consumer to read from relatively few topics, use most
> of the messages it receives and have fairly cohesive logic for using these
> messages
ql would handle this
nicely as it would get the changes once they're committed.
Thanks for any insight,
Mark.
Original github issue:
https://github.com/confluentinc/kafka-connect-jdbc/issues/27
ere. I'll also
close the github issue with a link to this thread.
Thanks again,
Mark.
On Wed, Dec 16, 2015 at 9:51 PM Ewen Cheslack-Postava
wrote:
> Mark,
>
> There are definitely limitations to using JDBC for change data capture.
> Using a database-specific implementa
Kafka
rather than by the numerous proprietary systems that do this.
Mark
Thanks, Gwen.
I was testing RC2 with Spark Streaming and found an issue that has a minor
fix. I think this issue exists with RC3 as well:
https://issues.apache.org/jira/browse/KAFKA-3669
On Thu, May 5, 2016 at 1:46 PM, Gwen Shapira wrote:
> Hello Kafka users, developers and client-developers,
>
ith respect
to receiving records but I would like to understand
1. Why is the message being logged?
2. Do I need to do anything?
3. Can anything be done to stop it being logged?
Thanks,
Mark
n.id}: expected " +
>
> s"epoch ${session.epoch}, but got epoch $
> {reqMetadata.epoch()}.")
>
> new SessionErrorContext(Errors.INVALID_FETCH_SESSION_EPOCH,
> reqMetadata)
>
> Can you pastebin the log line preceding what you pas
there are always out by one significant?
Thanks,
Mark
On Wed, 13 Jun 2018 at 17:46 Ted Yu wrote:
> You would need this (plus any appender you want the log to go to):
>
> log4j.logger.kafka.server=DEBUG
>
> FYI
>
> On Wed, Jun 13, 2018 at 9:15 AM, Ted Yu wrote:
>
>>
Also in this case will it fall back to a full request?
Hence no data is lost but it might increase latency?
Thanks
Mark
On Thu, 26 Jul 2018, 12:28 Mark Anderson, wrote:
> Ted,
>
> Below are examples of the DEBUG entries from FetchSession
>
> [2018-07-26 11:14:43,461] DEBUG Crea
Have you reviewed
https://www.confluent.io/blog/getting-started-apache-kafka-kubernetes/ as a
starting point?
On Mon, 22 Oct 2018, 18:07 M. Manna, wrote:
> Thanks a lot for your prompt answer. This is what I was expecting.
>
> So, if we had three pods where volumes are mapped as the following
>
d and 2.1.x is imminent.
Thanks,
Mark
nt to the new
leader.
Could you please confirm my understanding is correct?
Thanks,
Mark
27;t seen
this issue.
Please let me know if there is any more information I can supply.
Thanks,
Mark
che/kafka/pull/6005/files
>
>
> Guozhang
>
> On Wed, Dec 5, 2018 at 6:54 AM Mark Anderson
> wrote:
>
> > Hi,
> >
> > I'm periodically seeing ConcurrentModificationExceptions in the producer
> > when records are expired e.g.
> >
> > E
at the key parameters are to tune to make this possible.
Does anyone have any pointers? Or are there any guides online?
Thanks,
Mark
twork threads (To try and work through the queued requests faster)?
Thanks,
Mark
On Thu, 6 Dec 2018 at 23:43 Mayuresh Gharat
wrote:
> Hi Mark,
>
> The leader election of a new topic partition happens once the controller
> detects that the Leader has crashed.
> This happens asynchrono
is it possible this will cause messages to be re-ordered
within the transaction?
Mark
tCnxn)
However, my zookeeper session timeout is configured as 2000ms.
Why does the log file show a session timeout for a value less than what is
configured?
Thanks,
Mark
he 'readTimeout' is defined as:
>
> readTimeout = sessionTimeout * 2 / 3;
>
> Thus, the 'actually' sessionTimeout is 1333ms while
> config:zookeeper.session.timeout=2000ms
>
>
> >-Original Message-
> >From: Mark Anderson [mailto:manderso...@gm
mpact performance or stability?
Thanks,
Mark
nyone have any experience of running Kafka with this flag and seeing
a reduction in pauses?
Thanks,
Mark
t for?
Thanks,
Mark
e was a network
problem that caused it to timeout before it could return that record?
Mark
I'm sure I initially made this assumption when trying to read all records
from a compacted topic on application startup and it was incorrect.
Due to latency, threading GC pauses etc it would return 0 when there were
still records on the topic.
Mark
On Mon, 4 Feb 2019, 18:02 Pere Urbón Baye
e any cases where it would be OK to re-use the KafkaConsumer after
poll() throws an exception?
Thanks,
Mark
ccur? On a fixed schedule? The next time
Producer.send is called for the same topic and partition? Or does something
else trigger it?
Thanks,
Mark
need to replace it with a new consumer instance?
Thanks
Mark
ception being thrown
at line 926. Can anyone shed any light?
Thanks,
Mark
Hello,
I would like to know, how many opened windows Kafka streams can hold?
Sincerely, Mark
[cid:image001.png@01D3B960.F400ABC0]
This message has been scanned for malware by Websense. www.websense.com
Further investigation has uncovered a defect when resolving a hostname
fails - https://issues.apache.org/jira/browse/KAFKA-8182
Looks like it has been present since support for resolving all DNS IPs was
added.
On Mon, 1 Apr 2019 at 15:55, Mark Anderson wrote:
> Hi list,
>
> I'
Kafka has its own version of the zookeeper client libraries that are still
3.4.13.
I'd be interested to know if it is compatible with 3.5.x now that has a
stable release.
Mark
On Wed, 5 Jun 2019, 21:27 Sebastian Schmitz, <
sebastian.schm...@propellerhead.co.nz> wrote:
> Hi,
>
you could do something similar?
Mark
On Thu, 13 Jun 2019, 17:49 Murphy, Gerard, wrote:
> Hi,
>
> I am wondering if there is something I am missing about my set up to
> facilitate long running jobs.
>
> For my purposes it is ok to have `At most once` message delivery, this
The first thing I would do is update to the latest Java 8 release. Just in
case you are hitting any G1GC bugs in such an old version.
Mark
On Thu, 22 Aug 2019, 07:17 Xiaobing Bu, wrote:
> it not a network issues, since i had capture the network packets.
> when the GC remark and unloading
Are you using Spring Boot?
I know that the recent Spring Boot 2.2.0 release specifically updates their
Kafka dependency to 2.3.0. Previous version used Kafka 2.1.x though I've
used 2.2.x with it.
Maybe running mvn dependency:tree would help see if there are multiple
Kafka versions that could conf
le to import 2.3.1-rc2 in spring boot?
>
> Thanks!
>
> On Thu, Oct 24, 2019 at 4:21 PM Mark Anderson
> wrote:
> >
> > Are you using Spring Boot?
> >
> > I know that the recent Spring Boot 2.2.0 release specifically updates
> their
> > Kafka depe
Jackson was updated to 2.10 in the latest Kafka release. The method
mentioned no longer exists in 2.10.
Do you have multiple versions of Jackson on the ckasspath?
On Thu, 12 Dec 2019, 11:09 Charles Bueche, wrote:
> Hello again,
>
> spending hours debugging this and having no clue...
>
> * Kaf
lease feel free to ask more.
I’m looking for confirmation on the above or corrections, both welcome.
Thanks,
Mark
0.10.2
Thanks!
Mark
have a bunch of consumer groups, one per topic,
> so they can rebalance independently.
>
> On Wed, Feb 26, 2020, 1:05 AM Mark Zang wrote:
>
> > Hi,
> >
> > I have a 20 brokers kafka cluster and there are about 50 topics to
> consume.
> >
> > Between creatin
08)
at
io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:447)
... 17 more
Best regards,
John Mark Causing
1 - 100 of 128 matches
Mail list logo
