Hello,
I have a code that access a state store in order to get the latest value of
a specific key. In a case of a regular KTable the store name is the
same store name that was passed together with the aggregator and there is
no problem accessing this store.
In the case of a Windowed KTable, I can'
Hi, I'm new to Kafka, and I would like to use schema registry to manager
the schema of my topic.
The schema I've created:
curl -X POST -i -H "Content-Type: application/vnd.schemaregistry.v1+json"
--data '{ "schema": "{\"type\": \"record\",\"name\": \"Customer\",
\"fields\": [ { \"type\": \"int\",
Hi all,
Group Coordinator can be different for different consumer groups,When a
consumer wants to join a group,how to choose the Group Coordinator?
Thanks,
Yuanjia Li
when a consumer join a group, selection will be triggered, and then
rebalance.
2017-02-15 17:59 GMT+08:00 Yuanjia :
> Hi all,
> Group Coordinator can be different for different consumer groups,When
> a consumer wants to join a group,how to choose the Group Coordinator?
>
> Thanks,
> Yuanjia L
My question is the selection procedure.
Thanks.
From: 陈江枫
Date: 2017-02-15 18:01
To: users
Subject: Re: How to choose one broker as Group Coordinator
when a consumer join a group, selection will be triggered, and then
rebalance.
2017-02-15 17:59 GMT+08:00 Yuanjia :
> Hi all,
> Group Co
Correct me if I am wrong.
Firstly, determine the target partition of __consumer_offsets where the offset
will be stored by calculating:
Math.abs(groupID.hashCode() % 50)
Secondly, find out the leader broker for that partition, and make that broker
as the coordinator.
Hi,
We have made some changes at our side to control the rocksdb.
Firstly we are assigning one partition per thread. Also creating only 4
threads per VM (four cores).
This way only 4 rocks db state stores get created on one VM.
Then we are making sure that VMs state store directory is within the VM
Hi Shimi,
Did you pass in a state store name for the windowed example? E.g., if you are
doing "count" or another aggregate, you can pass in a desired state store name.
Thanks
Eno
> On 15 Feb 2017, at 00:35, Shimi Kiviti wrote:
>
> Hello,
>
> I have a code that access a state store in order to
Our ZK server is secured using SASL.
We have looked at Linkedin Burrow\ KafkaOffsetMonitor, it seems that they does
not support connecting to ZK using SASL.
Does any know of any other Kafka consumer lag \ offset monitoring tool that
support SASL??
Thanks,
Shri
This e-mail and its contents (to
Hi Kafka crew,
I'm rolling out a Kafka Streams application on a pretty large dataset, and
I see some exceptions that worry me, which I haven't seen with smaller data
sets
(Using Kafka streams trunk of a few days ago)
2017-02-15 15:25:14,431 | INFO | StreamThread-42 | StreamTask
| 86 -
Hi Frank,
Very hard to say from the stack trace. The exception isn't the most
informative!
Is there anything obvious in the RocksDB LOG?
Thanks,
Damian
On Wed, 15 Feb 2017 at 06:38 Frank Lyaruu wrote:
> Hi Kafka crew,
>
> I'm rolling out a Kafka Streams application on a pretty large dataset,
A consumer that wishes to join a Consumer Group should issue a Group
Coordinator Request
(https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol#AGuideToTheKafkaProtocol-GroupCoordinatorRequest)
If no Group Coordinator exists at that time, a new one will be elected by
K
No, it sure isn't... The log file doesn't seem to have been touched for
hours. The whole log file seems to be of the same millisecond, but hours
before the stack trace (The stack trace is from 15:25). It seems to have
barfed out of nowhere without logging anything.
2017/02/15-12:38:07.061758 7f1cf
Sometimes this happens because the lock file is not cleared. Did you look
for the .lock file?
Regards
Sab
On 15 Feb 2017 8:59 p.m., "Frank Lyaruu" wrote:
> No, it sure isn't... The log file doesn't seem to have been touched for
> hours. The whole log file seems to be of the same millisecond, bu
Heroku tested this with our usual round of performance benchmarks, and
there seem to be no notable regressions in this RC that we can see (for a
sample on earlier regressions we found using these benchmarks during the
0.10.0.0 release,
https://engineering.heroku.com/blogs/2016-05-27-apache-kafka-01
I still think a clean cluster start should not take > 1 hr for balancing
though. Is this expected or am i doing something different?
I thought this would be a common use case.
Praveen
On Fri, Feb 10, 2017 at 10:26 AM, Onur Karaman <
okara...@linkedin.com.invalid> wrote:
> Pradeep is right.
>
>
I actually found out how to do it by looking at the tests
streams.store() needs to get a WindowStoreType of QueryableStoreType
like in the following
ReadOnlyWindowStore storage = streams.store("STORAGE",
QueryableStoreTypes.windowStore());
On Wed, Feb 15, 2017 at 2:25 PM, Eno Thereska
Hi,
Does Kafka HDFS Connect work with HDFS (SSL). As I see only properties in
security is
hdfs.authentication.kerberos, connect.hdfs.keytab,hdfs.namenode.principal
as these properties are all related to HDFS Kerberos.
As from the configuration and code I see we pass only Kerberos parameters,
not
Hello,
We are using Kafka as data pipeline for log aggregation into ElasticSearch. We
have over 500 apps that will eventually send logs to Kafka topics for
aggregation into ElasticSearch.
Currently, we are using app level topic for sending log messages into Kafka.
Wondering what is the best
Verified with librdkafka v0.9.4-RC1.
2017-02-15 9:18 GMT-08:00 Tom Crayford :
> Heroku tested this with our usual round of performance benchmarks, and
> there seem to be no notable regressions in this RC that we can see (for a
> sample on earlier regressions we found using these benchmarks during
From reading the bylaws it's not entirely clear who closes the vote or how they
decide when to do so.
Given a week has passed and assuming Jay's and Matthias's votes are binding,
we have a result of +4 votes with no other votes cast.
I'll update the KIP with the result shortly :)
> On Feb 14, 20
Oops, sorry, a number of votes were sent only to -dev and not to
-user and so I missed those in the email I just sent. The actual count is more
like +8
> On Feb 15, 2017, at 12:24 PM, Steven Schlansker
> wrote:
>
> From reading the bylaws it's not entirely clear who closes the vote or how
>
+1 (non-binding)
Still looks as good as RC0 did for my streams workload. :-)
Mathieu
On Wed, Feb 15, 2017 at 1:23 PM, Magnus Edenhill wrote:
> Verified with librdkafka v0.9.4-RC1.
>
> 2017-02-15 9:18 GMT-08:00 Tom Crayford :
>
> > Heroku tested this with our usual round of performance benchma
+1 (binding) from me.
For the record, there were 4 binding +1s (Gwen, Guozhang, Jay and myself).
Ismael
On Wed, Feb 15, 2017 at 8:24 PM, Steven Schlansker <
sschlans...@opentable.com> wrote:
> From reading the bylaws it's not entirely clear who closes the vote or how
> they
> decide when to do
In case anyone else runs into this issue:
Turning on TRACE level logs revealed that the config topic we were using
was auto-created to have 12 partitions. As stated in the Kafka Connect User
Guide (http://docs.confluent.io/3.1.2/connect/userguide.html), the internal
topic used to store configs (co
Thanks Ismael.
The KIP voting has been accepted with 4 binding +1, and 5 non-binding +1s.
Guozhang
On Wed, Feb 15, 2017 at 12:47 PM, Ismael Juma wrote:
> +1 (binding) from me.
>
> For the record, there were 4 binding +1s (Gwen, Guozhang, Jay and myself).
>
> Ismael
>
> On Wed, Feb 15, 2017 a
KIP is accepted, discussion now moves to PR.
Thanks
Eno
On Wed, Feb 15, 2017 at 12:28 PM, Steven Schlansker <
sschlans...@opentable.com> wrote:
> Oops, sorry, a number of votes were sent only to -dev and not to
> -user and so I missed those in the email I just sent. The actual count is
> more l
Hi,
according to the feedback, I updated the KIP, and limited its scope to
some extend:
- Instead of changing the creation of KafkaStreams instances, we keep
the current pattern (we might do a follow up KIP on this though).
- We also added a new method #describe() that returns a
TopologyDescript
+1 (non-binding).
Verified source and Scala 2.11 binary artifacts, quick start on source
artifact and Scala 2.11 binary artifacts.
Thanks for managing the release!
Ismael
On Tue, Feb 14, 2017 at 6:39 PM, Ewen Cheslack-Postava
wrote:
> Hello Kafka users, developers and client-developers,
>
> T
+1 (binding).
Verified web docs, protocol docs, javadocs, md5 hash of the artifact.
Guozhang
On Wed, Feb 15, 2017 at 4:06 PM, Ismael Juma wrote:
> +1 (non-binding).
>
> Verified source and Scala 2.11 binary artifacts, quick start on source
> artifact and Scala 2.11 binary artifacts.
>
> Thank
Hi,
Can I use the alter topic command to change the offsets.retention.minutes
setting on __consumer_offsets topic while the broker is running?
Thanks!
-neelesh
Hi, Ewen,
Thanks for running the release. +1. Verified quickstart on 2.10 binary.
Jun
On Tue, Feb 14, 2017 at 10:39 AM, Ewen Cheslack-Postava
wrote:
> Hello Kafka users, developers and client-developers,
>
> This is the third candidate for release of Apache Kafka 0.10.2.0.
>
> This is a minor
On Wed, Feb 15, 2017 at 5:04 PM, Matthias J. Sax
wrote:
> - We also removed method #topologyBuilder() from KStreamBuilder because
> we think #transform() should provide all functionality you need to
> mix-an-match Processor API and DSL. If there is any further concern
> about this, please let us
Hi,all! Our program uses the high level consumer api(the version is 0.8.x).
Sometimes the program will throw an exception in the 42th row in
kafka.utils.IteratorTemplate class，the content is "throw new
IllegalStateException("Expected item but none found.")".
I think it is a race condition pro
Yes, we can change.
On Thu, Feb 16, 2017 at 7:09 AM, Neelesh wrote:
> Hi,
> Can I use the alter topic command to change the offsets.retention.minutes
> setting on __consumer_offsets topic while the broker is running?
>
> Thanks!
> -neelesh
>
Is there any command can be used to get broker.id on one of broker with
command?
I can check zookeeper or check the broker starting log to find broker.id,
but I wander whether there is any command to get the broker.id?
You can just grep it from broker's config/server.properties
On Thu, Feb 16, 2017 at 7:42 AM, kay James wrote:
> Is there any command can be used to get broker.id on one of broker with
> command?
> I can check zookeeper or check the broker starting log to find broker.id,
> but I wander whether th
Hi,
We'd like to be added to the Kafka Powered By list.
Description:
Kafka serves as the backbone of the publish/subscribe service, PubSub, in
YaaS, which is a cloud platform for microservices (
https://devportal.yaas.io/services/us/pubsub/latest/). The PubSub service,
and by extension Kafka, all
38 matches
Mail list logo
