Kafka 0.8.0 in maven

I am using: org.apache.kafka kafka_2.9.2 0.8.0-beta1 I am trying to rebuild iron-count https://github.com/edwardcapriolo/IronCount/tree/iron-ng against kafka 0.8.0 I am having considerable issues getting tests to run correctly. If you run

Re: Kafka 0.8.0 in maven

my github ... > do you see the words " successful completion" printed out towards the > end of the test Run? > > I assuming that you ran mvn:exec java ...etc.. as specified in the Read Me > file.. is that correct? > On Nov 4, 2013 7:52

Re: Kafka 0.8.0 in maven

you see the words " successful completion" printed out towards the > end of the test Run? > > I assuming that you ran mvn:exec java ...etc.. as specified in the Read Me > file.. is that correct? > On Nov 4, 2013 7:52 PM, "

Re: Handling Scala exceptions from ListTopicsCommand in Java

Maybe try catching RuntimeException On Tue, Nov 5, 2013 at 2:24 PM, wrote: > Hi All: > I am trying to programmatically create Topics from a Java client. > > I am using a suggestion from > http://stackoverflow.com/questions/16946778/how-can-we-create-a-topic-in-kafka-from-the-ide-using-api/18480

Re: Handling Scala exceptions from ListTopicsCommand in Java

Actually that will not help. The main probably swallows the exception. On Tue, Nov 5, 2013 at 4:13 PM, Edward Capriolo wrote: > Maybe try catching RuntimeException > > > On Tue, Nov 5, 2013 at 2:24 PM, wrote: > >> Hi All: >> I am trying to programmatically creat

Re: Kafka 0.8.0 in maven

time(KafkaTimer.scala:33) > at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108) > at > > kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:96) > at > kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:88) > at ka

Re: Kafka 0.8.0 in maven

orry about your suffering... I can sympathize ! > > - cb > > > On Wed, Nov 6, 2013 at 7:38 PM, Edward Capriolo >wrote: > > > After about 5 days of relentless head pounding, and twittling about > > everything under the sun, I figured it out. > >

Re: Kafka 0.8.0 in maven

mess > things up with a typo- cb > > > On Wed, Nov 6, 2013 at 9:28 PM, Edward Capriolo >wrote: > > > One thing I noticed about your code. I thought in kafka 0.8.0 topics are > > not created automatically on first message. I do not see anywhere in your > > co

Re: Kafka cluster with lots of topics

Zookeeper will not be the only problem. The first is that each topic is a directory on the file system. Each of those is going to have files inside it. This is going to be fairly overwhelming for the file system. Also I can not speak for the internals but there may be cases where this many topics a

Re: are kafka consumer apps guaranteed to see msgs at least once?

You likely need to use a custom offset solution if you plan on committing every message. With many partitions this puts a large burden on Zookeeper, you end up needing to roll over your zk transaction logs fast as well or rist filling up disk On Thu, Nov 21, 2013 at 6:20 PM, Guozhang Wang wrote:

Attempt to prove Kafka transactions work

Hello all, I used to work in adtech. Adtech was great. CPM for ads is 1-$5 per thousand ad impression. If numbers are 5% off you can blame javascript click trackers. Now, I work in a non addtech industry and they are really, really serious about exactly once. So there is this blog: https://www.

Re: On the number of partitions.

On Saturday, October 26, 2019, M. Manna wrote: > You should also check out Becket Qin’s presentation on producer performance > tuning on YouTube. Both these items should give you all positives and > negatives of having many/less portions. > > Thanks, > > On Sat, 26 Oct 2019 at 09:19, Manasvi Gupt

Re: Attempt to prove Kafka transactions work

Boyang > > On Fri, Oct 25, 2019 at 7:25 PM Edward Capriolo > wrote: > > > Hello all, > > > > I used to work in adtech. Adtech was great. CPM for ads is 1-$5 per > > thousand ad impression. If numbers are 5% off you can blame javascript > > click trackers.

Transactions and offsets topic in multitrack environment

When setting up kakfka in a network with racks or multiple datacenters and manual replica placement is it required that the transaction offset topics match the topology as a user topic would? Are producers or consumers affected by changing these with live traffic. -- Sorry this was sent from mobi

Re: Attempt to prove Kafka transactions work

would of course love if you could share your test results! If you > discover a bug, please report it, so we can fix it. > > > -Matthias > > On 10/28/19 10:06 AM, Edward Capriolo wrote: > > On Sunday, October 27, 2019, Boyang Chen > wrote: > > > >> Hey Edwar

Re: producer transaction and threading model

Spring framework works around this by providing a method like doInTransaction( listofstuff). Behind the scenes it manages a pool of transactional producers with ids like transaction_prefix + id. So each call to dointransaction may initiate a transaction. On Friday, November 15, 2019, Matthias J

Re: Attempt to prove Kafka transactions work

ber 9, 2019, Edward Capriolo wrote: > > On Thu, Oct 31, 2019 at 6:11 AM Matthias J. Sax > wrote: > >> Quite a project to test transactions... >> >> The current system test suite is part of the code base: >> https://github.com/apache/kafka/tree/trunk/tests/kafkat

Re: Attempt to prove Kafka transactions work

Exactly Once semantics. > > In order for the Consumer to reset to the currently committed offsets, you > need to either initiate a Consumer Group Rebalance, or use a combination of > the KafkaConsumer#committed() and KafkaConsumer#seek() methods. > > On Wed, Nov 20, 2019 at 6:

Re: Attempt to prove Kafka transactions work

; independent if the transaction was successful or not. This would be an > incorrect usage of the API. > > > -Matthias > > On 11/20/19 6:16 AM, Edward Capriolo wrote: > > Ok. I'm at a point where I believe the exactly once is in question. > > > > Topic input

Re: Attempt to prove Kafka transactions work

On Wed, Nov 20, 2019 at 6:35 PM Edward Capriolo wrote: > > > On Wednesday, November 20, 2019, Matthias J. Sax > wrote: > >> I am not sure what Spring does, but using Kafka Streams writing the >> output and committing offset would be part of the same transaction. >

Re: Deleting topics in windows - fix estimate or workaround

My development environment is windows. I am actually able to run integration tests launching Kafka and Zookeeper. However I have noticed that the test warn likely about deleting a file that is locked and I tried deleting a topic at one point and saw some of the issues you mentioned. I personally t

Re: Kafka Log4j2.x upgrade plan

All as someone who has been patching all december. There are two aspects to log4j, the log4j API and the log4j jar version. It is possible to use log4j 2.17.1 {core,api} without modifying Kafka.. Kafka itself is internally using the log4j api, but the api is implemented by the patched log4j jars.

Discuss: outage impact of not updating kafka clients to a version that has KAFKA-9893

Hello, The bulk of this thread is to discuss [KAFKA-9893] Configurable TCP connection timeout and improve the initial metadata fetch - ASF JIRA (apache.org) . IMHO t it should be considered a BUG FIX and be potentially backported, but others may dis

Re: Discuss: outage impact of not updating kafka clients to a version that has KAFKA-9893

On Sun, Jan 23, 2022 at 7:51 AM Edward Capriolo wrote: > Hello, > The bulk of this thread is to discuss > [KAFKA-9893] Configurable TCP connection timeout and improve the initial > metadata fetch - ASF JIRA (apache.org) > <https://issues.apache.org/jira/browse/KAFKA-9893>

Re: Log4j 1.2

Explained in another thread log4j api is separate from implementation. Its possible to remove log4j 1.2 jars from classpath and upgrade to log4j 2.17.1 without changing a line of code in kafka. On Monday, January 10, 2022, Tauzell, Dave wrote: > Thanks. Those KIPs show that there is a fair amo

Re: Log4j 1.2

In general you can delete log4j1.jar Replace with log4jcore_2.17.1.jar And log4japi_2.17.1.jar Ed On Monday, January 24, 2022, Men Lim wrote: > Is there a write out of the steps that need to be taken? > > On Mon, Jan 24, 2022 at 10:36 AM Edward Capriolo > wrote: > > >

Re: Kafka Topics

On Thursday, December 30, 2021, Suresh Chidambaram wrote: > Hi Ola, > > I would suggest you can go with single Topic with multiple partitions. Once > the data gets received from the Topic, you can do a DB update kind of a > stuff to store the data , then use the data for analysing. > > Also, the

Re: Kafka Consumer Fairness when fetching events from different partitions.

On Monday, January 31, 2022, Chad Preisler wrote: > Hello, > > I got this from the JavaDocs for KafkaConsumer. > > * If a consumer is assigned multiple partitions to fetch data from, it > will try to consume from all of them at the same time, > * effectively giving these partitions the same pri

Re: Can you use the log4j 2 bridge to replace log4j 1.x in Kafka?

You are confusing the log4j api with the log4j jars. Yes. To your question. You can take log4j api and log4j core 2.17.1 and applications still believe they are using log4j1... On Tuesday, February 1, 2022, Steve Souza wrote: > Where I work we use both Tableau and Microstrategy which in turn use

Re: Java API to monitor Consumer Offset and Lag

We have a nagios check that runs the command line lag check and determines if any partition has a lag greater then X. Its about a 2 hour solutiontocode up right but worth every penny. On Fri, Mar 28, 2014 at 7:45 PM, Otis Gospodnetic < otis.gospodne...@gmail.com> wrote: > Hi Harsh, > > Are you t

Kicking the tires on 0.8.1...tires kicking back

I am trying to convert a few projects to the latest kafka... Is this the latest artifact? org.apache.kafka kafka-perf_2.8.0 0.8.1 I have a piece of code @Test public void test() throws InterruptedException { //super.createTopic("

Re: Kicking the tires on 0.8.1...tires kicking back

quot;; arguments[6] = "--topic"; arguments[7] = name; CreateTopicCommand.main(arguments); try { Thread.sleep(1000); } catch (InterruptedException e) { e.printStackTrace(); } } On Sun, Mar 30, 2014 at 12:30 PM, Edward Capriolo wrote: > I am try

command line clients and request balancing.

After installing Kafka I did the following createTopic --partitions 10 -replication 1 --name mytopic I started producer using bin/kafka-producer-shell I opened two consumers in the same group using bin/kafka-consumer-shell In older versions this test used to result in the messages being sent by

Re: command line clients and request balancing.

d then round-robin to another > partition in the next period. > > Guozhang > > G > > > On Thu, May 1, 2014 at 3:18 PM, Edward Capriolo wrote: > >> After installing Kafka I did the following >> >> createTopic --partitions 10 -replication 1 --name mytopic

Request log is 300MB/hour

At a certain hour I have seen a huge up tick in requests. log4j.appender.requestAppender=org.apache.log4j.DailyRollingFileAppender log4j.appender.requestAppender.DatePattern='.'-MM-dd-HH log4j.appender.requestAppender.File=logs/kafka-request.log log4j.appender.requestAppender.layout=org.apache