Using Kafka Producer inside Oracle DB

Hi, I am trying to post Messages from Oracle to Kafka broker directly using custom java class for that I uploaded the required library in oracle and calling java function using oracle function/procedure. Basic java function is working fine but function in which Kafka is getting initialized and

Broker leaving cluster--even though Kafka broker remains up

Hi Everyone, What causes a broker to leave a cluster even when the broker remains running? Is it loss of sync with Zookeeper? --John

Re: Kafka shutdown gracefully

Don't use `kill -9 PID`. Use `kill -s TERM PID` - sends a signal to the process to end, and will trigger any cleanup routines before exiting. Since the output of the `ps` command used by kafka-server-stop.sh exceeds 4096 characters. It shows "No kafka server to stop" On Thu, Jul 6, 2017 at 3:25 A

Order of ConsumerRecords obtained by records(TopicPartition) in Kafka Consumer API

Hello, I had a question regarding records(TopicPartition partition) method in ConsumerRecords I am trying to use auto.offset.reset=earliest and combine it with manual offset control. I saw this sample code under "Manual Offset" https://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/c

RE: Using Kafka Producer inside Oracle DB

>> java.lang.NoClassDefFound Error You are missing some dependent classes. Two questions: 1. Does the message have more information about what class it couldn't find? 2. What exactly are you putting into your jar file? -Dave -Original Message- From: Rahul R04 [mailto:rahul.kuma...@mph

Re: Kafka Streams 0.10.2.1 client crash - .checkpoint.tmp (No such file or directory)

Hi Damian, Sorry for the delayed reply have been out of office. I'm afraid I cannot check. We have alarms on our auto scaling groups for stream instances to kill them should the CPU utilization be < 1 for 30 mins. On Fri 30 Jun 2017 at 5:05 p.m., Damian Guy wrote: > Hi Ian, > > Can you check i

Unusual(?) behaviour in mirror maker cluster

Hi, We've deployed an EU based mirror maker cluster to consume topics from clusters situated in the US. We have five brokers in the mm cluster, but it seems to be that one particular one consistently has higher throughput than all the others (which stay around the same). We are also having issue

PLEASE - [KAFKA-1194] Log Retention is always throwing IOException

Hi, I have sent numerous emails in the past about the same issue, but no response so far. I was wondering if we will get a patch. I am working on a synchronisation PoC which is reliant on Log cleanup to be successful every day. I have got auto.offset.reset=earliest and the offsets.retention.minut

Kafka partitions are piled up for Consumer

Hi, If we have some high response in backend( let's say database high response), the few of the partitions messages are piled up. Once the backend gets normal still those partitions are not processing any message and it creates a huge lag. Any idea why it happened? We thought once the backend g

Re: Kafka connector throughput reduction upon avro schema change

Bumping this. Has anyone here observed this in their Kafka connect deployments? Thanks, Dave On 5/26/17, 1:44 PM, "Dave Hamilton" wrote: We are currently using the Kafka S3 connector to ship Avro data to S3. We made a change to one of our Avro schemas and have noticed consumer throughput

Re: Kafka partitions are piled up for Consumer

I am observing the same with our .10.2 cluster where consumers are hanged for those partitions where the current offset's data gets deleted due to retention. It looks like a bug to me. Thanks! On Jul 6, 2017 9:23 AM, "Ghosh, Achintya (Contractor)" < achintya_gh...@comcast.com> wrote: > Hi, > > I

Re: Mirroring multiple clusters into one

I'm not sure what the "official" recommendation is. At TiVo, we *do* run all our mirrormakers near the target cluster. It works fine for us, but we're still fairly inexperienced, so I'm not sure how strong of a data point we should be. I think the thought process is, if you are mirroring from a

Consumer offset management in latest Kafka release

Hello Team, I have few questions on consumer offset management done in latest release, if you could spend few minutes and answer the below questions - (please answer based on latest release) 1. Is there any dependency of zookeeper in storing consumer offset ? 2. Can you give some insight on the s

Loading and streaming data from Kafka to BigQuery

Hi guys, I would like to recommend the following post, discussing and comparing techniques for loading data from Kafka to BigQuery. https://medium.com/myheritage-engineering/kafka-to-bigquery-load-a-guide-for-streaming-billions-of-daily-events-cbbf31f4b737 Feedback is welcome. *Ofir Sharony* Ba

RE: Kafka partitions are piled up for Consumer

This is not the case for us. Anyone has any other idea please? How we can put timeout for Kafka thread? Thanks! -Original Message- From: Naanu Bora [mailto:naanub...@gmail.com] Sent: Thursday, July 06, 2017 1:41 PM To: users@kafka.apache.org Subject: Re: Kafka partitions are piled up fo

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Thanks for the KIP Vahid. I think it'd be useful to have these filters. That said, I also agree with Edo. We don't currently rely on the output, but there's been more than one time when debugging an issue that I notice something amiss when I see all the data at once but if it wasn't present in th

Re: Using Kafka Producer inside Oracle DB

Just to add to this, depending upon your use case it may be beneficial to use kafka connect for pulling data out of oracle to publish to kafka. With the JDBC connector you would just need a few configs to stand up kafka connect and start publishing data to kafka, either via a select statement or a

connect in 0.11.0.0 warnings due to class not found exceptions

i just did a test upgrade to kafka 0.11.0.0 and i am seeing lots of ClassNotFoundException in the logs for connect-distributed upon startup, see below. is this expected? kind of curious why its looking for say gson while gson jar is not in libs folder. best, koert [2017-07-06 22:20:41,844] INFO R

updating startup script in Apache Amabari

hi All i'm trying to configure/update Kafka startup script, so Prometheus is able to read metrics from Kafka. I need to add the following to the Broker startup script : -javaagent:/path/to/jmx_exporter.jar=$PORT:$CONFIG Where in Apache Ambari can i modify the Kafka Broker startup script ? Pls

Re: Mirroring multiple clusters into one

James, Thanks for sharing your thoughts and experience. Could you please also confirm whether - you do any encryption for the mirrored data? - you have a many-to-one mirroring similar to what I described? Thanks. --Vahid From: James Cheng To: users@kafka.apache.org Cc: dev Date:

Re: connect in 0.11.0.0 warnings due to class not found exceptions

Hi Koert, these warnings appear to be produced during the class scanning that Connect is performing when it's starting up. Connect is using org.reflections to discover plugins (Connectors, Transformations, Converters) in the various locations that it's configured to search for plugins. (such locat

How to fetch offset for all the partitions of one topic through single KafkaClient function call (0.10.x)?

Hi, I am using Kafka client 0.10.2. Is there a way to fetch latest committed offset for all the partitions in one function call? I am call the KafkaConsumer.commited() to get this for single partition, is there a simple way to batch fetch offsets for all the partitions in single topic in one shot

Error in Kafka startup with the JMX exporter - java.lang.IllegalArgumentException: Collector already registered that provides name: jmx_scrape_duration_seconds

Hi All, I'm getting error in starting up Kafka, with the JMX exporter passed as java agent. Error is as shown below, any ideas ? I'm trying to integrate Prometheus to collect Kafka metrics. [root@nwk2-bdp-kafka-06 kafka]# KAFKA_OPTS="$KAFKA_OPTS -javaagent:/usr/hdp/2.5.3.0-37/prometheus/jmx_prom

Kafka Broker Halting due to "ahead" offsets

Hi All, In our 3-node test cluster running Kafka 0.10.0, we faced this error: FATAL [2017-07-06 07:30:42,962] kafka.server.ReplicaFetcherThread:[Logging$class:fatal:110] - [ReplicaFetcherThread-0-0] - [ReplicaFetcherThread-0-0], Halting because log truncation is not allowed for topic Topic3, Curr

Re: connect in 0.11.0.0 warnings due to class not found exceptions

i did not have log4j.logger.org.reflections=ERROR, because i didnt update my log4j files yet. i will do this now. connect seems to start up fine. i still wonder why its searching for gson. like... where does it get the idea for the start searching for gson? i dont use gson and neither does connec

Re: Mirroring multiple clusters into one

Answers inline below. -James Sent from my iPhone > On Jul 7, 2017, at 1:18 AM, Vahid S Hashemian > wrote: > > James, > > Thanks for sharing your thoughts and experience. > Could you please also confirm whether > - you do any encryption for the mirrored data? Not at the Kafka level. The data

Re: PLEASE - [KAFKA-1194] Log Retention is always throwing IOException

Kafka is not well tested on Windows platform..There are some issues running on Windows. It is recommended to run on Linux machines. On Thu, Jul 6, 2017 at 9:49 PM, M. Manna wrote: > Hi, > > I have sent numerous emails in the past about the same issue, but no > response so far. > > I was wonderin