Thanks for your prompt response.
In my check for topicExists i will add a check for topic in
DeleteTopicsPath.
On Mon, Mar 23, 2015 at 8:21 PM, Harsha wrote:
> Just to be clear, one needs to stop producers and consumers that
> writing/reading from a topic “test” if they are trying to delete t
Hi All,
Currently we are using kafka_2.8.0-0.8.0-beta1 in our production system. I
am using sync producer with ack=0 to send the events to broker.
but I am seeing most of my producer threads are blocked.
"jmsListnerTaskExecutor-818" prio=10 tid=0x7f3f5c05a800 nid=0x1719
waiting for monitor
Hi Jon,
It the link for the 1/27 meetup you posted works for me, but I haven't
found how to find that same link on the meetup site (there are links that
point to the live stream, which of course is no longer happening!).
Thoughts?
Thanks,
Jason
On Mon, Mar 2, 2015 at 11:31 AM, Jon Bringhurst <
If you have feedback, don't hesitate to comment on the JIRA.
On Mon, Mar 23, 2015 at 4:19 PM, Pete Wright
wrote:
> Hi Gwen - thanks for sending this along. I'll patch my local checkout
> and take a look at this.
>
> Cheers,
> -pete
>
> On 03/20/15 21:16, Gwen Shapira wrote:
> > We have a patch
Hi Gwen - thanks for sending this along. I'll patch my local checkout
and take a look at this.
Cheers,
-pete
On 03/20/15 21:16, Gwen Shapira wrote:
> We have a patch with examples:
> https://issues.apache.org/jira/browse/KAFKA-1982
>
> Unfortunately, its not committed yet.
>
> Gwen
>
> On Fri
Hey Everyone –
Just a reminder about the Meetup tomorrow night @ LinkedIn.
There will be 3 talks:
Offset management - 6:35PM Joel Koshy(LinkedIn)
The Netflix Data Pipeline - 7:05PM - Allen Wang & Steven Wu(Netflix)
Best Practices - 7:50PM - Jay Kreps(Confluent)
If you are interested in atte
We've talked about it a little bit, but part of the problem is that it is
pretty well integrated into our infrastructure, and as such it's hard to
pull it out. I illustrated this a little differently than Jon did in my
latest blog post (http://engineering.linkedin.com/kafka/running-kafka-scale),
ho
I created a wiki page that lists all the MySQL replication options that people
posted, plus a couple others. People may/may not find it useful.
https://github.com/wushujames/mysql-cdc-projects/wiki
I wasn't sure where to host it, so I put it up on a Github Wiki.
-James
On Mar 17, 2015, at 11:0
Hi Francois,
Looks like this belong storm mailing lists. Can you please send this
question on storm mailing lists.
Thanks,
Harsha
On March 23, 2015 at 11:17:47 AM, François Méthot (fmetho...@gmail.com) wrote:
Hi,
We have a storm topology that uses Kafka to read a topic with 6
p
Hi Jason,
Yes, I agree the restriction makes the usage of round-robin less flexible.
I think the focus of round-robin strategy is workload balance. If
different consumers are consuming from different topics, it is unbalanced
by nature. In that case, is it possible that you use different consumer
g
Hi,
We have a storm topology that uses Kafka to read a topic with 6
partitions. ( Kafka 0.8.2, Storm 0.9.3 )
Recently, we had to set the KafkaSpout to read from the beginning, so we
temporary configured our KafkaConfig this way:
kafkaConfig.forceFromStart=true
kafkaConfig.startOffsetTime = Of
Just to be clear, one needs to stop producers and consumers that
writing/reading from a topic “test” if they are trying to delete that specific
topic “test”. Not all producers and clients.
--
Harsha
On March 23, 2015 at 10:13:47 AM, Harsha (harsh...@fastmail.fm) wrote:
Currently we have auto.c
Currently we have auto.create.topics.enable set to true by default. If this is
set true any one who is making TopicMetadataRequest can create a topic . As
both producers and consumers can send TopicMetadataRequest which will create a
topic if the above config is true. So while doing deletion if
What happens when producers or consumers are running while the topic
deleting is going on?
On Mon, Mar 23, 2015 at 10:02 AM, Harsha wrote:
> DeleteTopic makes a node in zookeeper to let controller know that there is
> a topic up for deletion. This doesn’t immediately delete the topic it can
> ta
With MM, the source and destination cluster can choose different number of
partitions for the mirrored topic, and hence messages may be re-grouped in
the destination cluster. In addition, let's say you have two MMs piping
data to the same destination from two sources, the ordering of which
messages
DeleteTopic makes a node in zookeeper to let controller know that there is a
topic up for deletion. This doesn’t immediately delete the topic it can take
time depending if all the partitions of that topic are online and brokers are
available as well. Once all the Log files deleted zookeeper nod
Emmanuel, if it helps, here's a little more detail on the hardware spec we
are using at the moment:
12 CPU (HT enabled)
64 GB RAM
16 x 1TB SAS drives (2 are used as a RAID-1 set for the OS, 14 are a
RAID-10 set just for the Kafka log segments)
We don't colocate any other applications with Kafka e
We have make a simple web console to monitor some kafka informations like
consumer offset, logsize.
https://github.com/shunfei/DCMonitor
Hope you like it and offer your help to make it better :)
Regards
Flow
I think I worked out the answer to question 1.
java.lang.IllegalMonitorStateException
was thrown due to no ownership of ReentrantLock when trying to call await()
on the lock condition.
Here is the code snippet from the AbstractFetcherThread.scala in trunk
partitionMapLock synchronized {
On deleting a topic via TopicCommand.deleteTopic()
I get "Topic test-delete is marked for deletion."
I follow up by checking if the topic exists by using
AdminUtils.topicExists()
which suprisingly returns true.
I expected AdminUtils.TopicExists() to check both BrokerTopicsPath
and DeleteTopicsPa
Are there any plans to open source the same? What alternates do we have
here?
We are building an internal auditing framework for our entire big data
pipeline. Kafka is one of the data sources we have (ingested data).
On Mon, Mar 23, 2015 at 1:03 PM, tao xiao wrote:
> Linkedin has an excellent t
Linkedin has an excellent tool that monitors lag/data loss/data duplication
and etc. Here is the reference
http://www.slideshare.net/JonBringhurst/kafka-audit-kafka-meetup-january-27th-2015
it is not open sourced though.
On Mon, Mar 23, 2015 at 3:26 PM, sunil kalva wrote:
> Hi
> What is best p
Hi
What is best practice for adding audit feature in kafka, Is there any
framework available for enabling audit feature at producer and consumer
level and any UI frameworks for monitoring.
tx
SunilKalva
23 matches
Mail list logo
