Hi Ratha,
Do you want to give a try to the schema registry and Avro serializers using
it?
http://docs.confluent.io/2.0.1/schema-registry/docs/serializer-formatter.html#serializer
Thanks,
Liquan
On Mon, Apr 18, 2016 at 11:02 PM, Ratha v wrote:
> Hi all;
>
> I try to publish/consume my java obje
Hi, avro Schemas imply a pretty big overhead, if you would include them in
every message. It's good practice to include a schema id with the message...
Then you need a schema repository to lookup the matching schema based on the id.
Have a look at confluent.io. They offer a schema repo among o
Hi ,
In case of Async kafka producer(not using a get on producer.send()) what is
the behavior if the num of messages are grater than batch.size and the we
are still sending the messages and also previous send from kafka is not
successful.
Will the producer block in that condition for any new requ
Hi Everyone,
I am facing some issues with new Kafka consumer API. My consumer polls and
processes records in same thread. I commit explicitly through
commitSync(ConsumerRecord) method. i have seen following issues
1) Sometimes when processing time is too high i see error
"UNKNOWN_MEMBER_ID" on co
Hello,
I went thru QuickStart instructions at:
http://docs.confluent.io/2.1.0-alpha1/streams/quickstart.html
Downloaded confluent-2.1.0-alpha1, started ZK & kafka servers. Continuously
producing to topic : streams-file-input.
However, running the WordCountJob example throws error (pasting the me
Hello,
+1, same problem when I tried it. However, I dove into the code examples so
I can't give you a solution.
2016-04-19 17:20 GMT+02:00 Ramanan, Buvana (Nokia - US) <
buvana.rama...@nokia.com>:
> Hello,
>
> I went thru QuickStart instructions at:
> http://docs.confluent.io/2.1.0-alpha1/stream
Hello,
Thanks again for your reply :)
1) In my example when I send a record from outer table and there is no
matching record from inner table I receive data to the output topic and
vice versa. I am trying it with the topics empties at the first execution.
How is possible?
Why KTable joins does n
Hi,
If I have multiple consumers in a consumer group for load handling for the
same application is there any recommendation if the clientId should all be
unique for each? It's the same application. Each will have it's own
consumer memberId given on the join group so they can always be
distingu
Related to the log compaction question: " it will be log
compacted on the key over time", how do we control the time for log
compaction? For the log compaction implementation, is the storage used to
map a new value for a given key stored in memory or on disk?
On Tue, Apr 19, 2016 at 8:58 AM, Guil
Hello,
I read in the docs that Kafka Streams stores the computed aggregations in a
local embedded key-value store (RocksDB by default), i.e., Kafka Streams
provides so-called state stores. I'm wondering about the relationship
between each state store and its replicated changelog Kafka topic.
If w
As far as I know the cientId is only used for logging, so you could set it
to whatever is most usefull in the logging. You might for example want to
use the ip as the id, so when you get errors you know where to look.
On Tue, Apr 19, 2016 at 6:51 PM Rick Rineholt wrote:
> Hi,
> If I have multipl
Hi,
I enabled imx on kafka brokers by adding
KAFKA_JMX_OPTS= KAFKA_JMX_OPTS="
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=
-Djava.net.preferIPv4Stack=true “
However, when I use ‘kafka.tool
So it just should be used in a way that works best for your own use case.
Thanks Gerard !
On Tue, Apr 19, 2016 at 2:52 PM, Gerard Klijs
wrote:
> As far as I know the cientId is only used for logging, so you could set it
> to whatever is most usefull in the logging. You might for example want to
1) It sounds your should be using KTable.outerjoin(KTable) with your case,
but keep in mind that currently we are still working on exactly-once
semantics, and hence currently the results may be ordering dependent.
We do not support windowing in KTable since itself is an ever-updating
changlog alre
Ok... I don't believe my test results, but I have repeated them.
I found that build 9cfb99 (Mar 22) worked for me -- KafkaProducer didn't
hang/time-out.The next-newer build, 73470b0, did hang and KafkaProducer send()
calls would timeout.
The only differences between the two builds were documentat
Is it true that the aggregation and reduction methods of KStream will emit
a new output message for each incoming message?
I have an application that's copying a Postgres replication stream to a
Kafka topic, and activity tends to be clustered, with many updates to a
given primary key happening in
That is very strange Greg. We read the properties file via the method
provided by the Java standard library. To avoid having the conversation in
two different places, probably better to continue it in the issue you filed.
Ismael
On Tue, Apr 19, 2016 at 9:21 PM, Greg Zoller
wrote:
> Ok... I don'
I have another follow-up question on the compacted kafka topic for RocksDB
replication.
1. From Kafka compaction implementation, looks like all keys from the past
for that topic will be preserved, (the compaction/cleaner will only delete
the records which has same-key occurrences later in the queu
What is the correct way of using SSL between the client and brokers if
client certificates are not used? The broker (0.9.0.0) reports the
following in the log
WARN SSL peer is not authenticated, returning ANONYMOUS instead
as a result of this (I belive) KafkaConsumer.listTopics() returns an empty
Hi,
That warning is harmless. Personally, I think it may be a good idea to
remove as it confuses people in cases such as this.
Do you have multiple brokers? Are the brokers configured to use SSL for
inter-broker communication (security.inter.broker.protocol)? This is
required if the only listener
Thanks..I try to follow that trick..
On 19 April 2016 at 17:10, wrote:
>
>
> Hi, avro Schemas imply a pretty big overhead, if you would include them in
> every message. It's good practice to include a schema id with the
> message... Then you need a schema repository to lookup the matching schema
There is only one broker in this case. There are no errors (besides the
warning below) on either the broker or the client side. It just returns an
empty topic list if plaintext is not configured, even though client is
using SSL in both cases.
marko
> Hi,
>
> That warning is harmless. Personally,
Hi Buvana,
com.fasterxml.jackson.core.JsonGenerator.setCurrentValue is introduced
since Jackson 2.x but not in 1.x:
https://github.com/FasterXML/jackson-core/blob/master/src/main/java/com/fasterxml/jackson/core/JsonGenerator.java
and Kafka Streams depend on version 2.6.3.
It seems you bring in
Hi Guillermo,
1). It will have two rows: {"hello" => 2} and {"world" => 1}.
2). That is correct. Note that changelog records the most recent values for
each key, so if you do not delete the data, the new "hello" => 3 record
would practically make the previous two "hello" => 1 and "hello" => 2
obs
Yes we are aware of this behavior and are working on optimizing it:
https://issues.apache.org/jira/browse/KAFKA-3101
More generally, we are considering to add a "trigger" interface similar to
the Millwheel model where users can customize when they want to emit
outputs to the downstream operators.
Hi Henry,
1) Yes, if your key space is unlimited. But in practice, for KTable streams
where the record key (i.e. the primary key of the "table") is usually a
client-id, service-id, etc, the key space is usually bounded, for example
by the population of the globe, where in this case it should still
Hi
I have a one scenario ass below, I want to know whether its supported
currently. If not is there any work around by using existing kafka features.
I have kafka producer, currently he doesn’t have connection to the broker.
I want to send the messages to kafka broker when the connection is
a
Hi
I have a one scenario ass below, I want to know whether its supported
currently. If not is there any work around by using existing kafka features.
I have kafka producer, currently he doesn’t have connection to the broker.
I want to send the messages to kafka broker when the connection is
a
Hi Yogesh,
No, Kafka does not provide this functionality out of the box, Though you
can easily engineer it by having a localhost Kafka setup.
1. Always write data to the localhost Kafka.
2. When broker connection is available then read data from localhost
Kafka and send it to remote Kafk
Thank You for the reply.
I am running producer in very resource constraint device(IOT hub). I doubt
whether i can accommodate local broker.
On Wed, Apr 20, 2016 at 10:07 AM, Sunny Shah wrote:
> Hi Yogesh,
>
> No, Kafka does not provide this functionality out of the box, Though you
> can easily
Hi Yogesh,
You can even use sqllite/leveldb to buffer the data on client.
Thanks,
Sunny
On Apr 20, 2016 10:31 AM, "Yogesh BG" wrote:
> Thank You for the reply.
>
> I am running producer in very resource constraint device(IOT hub). I doubt
> whether i can accommodate local broker.
>
>
> On Wed,
If you use Go you can use https://github.com/sclasen/event-shuttle which is
nice choice in some cases because footprint, it uses boltdb which is like
leveldb which is like embedded k/v ok
NiFi is cool too https://nifi.apache.org/
So is bruce https://github.com/ifwe/bruce
those are more out of th
In my case, the key space is unbounded. The key would be something like
'ad_id', this id is auto incrementing all the time. I understand the
benefit of using compacted kafka topic for aggregation store, but I don't
see much benefit of using compaction to replicate records in JoinWindow
(there are
For the technique of custom Processor of holding call to context.forward(),
if I hold it for 10 minutes, what does that mean for the consumer
acknowledgement on source node?
I guess if I hold it for 10 minutes, the consumer is not going to ack to
the upstream queue, will that impact the consumer p
34 matches
Mail list logo
