I understand point (1) about when materialization happens. But I cannot
follow your conclusion about how this should influence the DSL because I
don't see a functional difference in "provide a store name in a
overload" vs "call .materialize()" -- both mechanism can do the exact
some thing.
I also
Hi,
yes, all examples have at-least-once semantics because this is the only
"mode" Kafka Streams supports -- you cannot "disable" it. (btw: we are
currently working on exactly-once for Streams that you will be able to
turn off/on).
There is not much documentation about how it work internally, bec
Hi there,
The inconsistency will be resolved, whether with materialize or overloaded
methods.
With the discussion on the DSL & stores I feel we've gone in a slightly
different tangent, which is worth discussing nonetheless. We have entered into
an argument around the scope of the DSL. The DSL
Hi Eno,
I have a really hard time understanding why we can't. From my point of
view everything could be super elegant DSL only + public api for the
PAPI-people as already exist.
The above aproach implementing a .get(K) on KTable is foolisch in my
opinion as it would be to late to know that m
Hi,
I want to read the Kafka Streams JMX metrics using jmxterm, similiar to
this Kafka documentation:
https://cwiki.apache.org/confluence/display/KAFKA/jmxterm+quickstart.
I am using the same version: jmxterm-1.0-alpha-4-uber.jar. I managed to
retrieve metrics from the Kafka Streams application v
So I think there are several important discussion threads that are emerging
here. Let me try to tease them apart:
1. inconsistency in what is materialized and what is not, what is queryable and
what is not. I think we all agree there is some inconsistency there and this
will be addressed with a
Hi Jendrik,
I haven't tried jmxterm. Can you confirm if it is able to access the Kafka
producer/consumer metrics (they exist since Kafka Streams internally uses
Kafka)?
I've personally used jconsole to look at the collected streams metrics, but
that might be limited for your needs.
Thanks
Eno
Hi Eno,
thanks for putting into different points. I want to put a few remarks
inline.
Best Jan
On 30.01.2017 12:19, Eno Thereska wrote:
So I think there are several important discussion threads that are emerging
here. Let me try to tease them apart:
1. inconsistency in what is materialized
Not really, as you can update the schema, and have multiple of them at the
same time. By default each schema has to backwards compatible, so you do
have to exclude the specific topic you use with different schema's. With
every write, the 'id' of the schema used is also written, so when you
deserial
Hi Eno,
I tried accessing the Kafka consumer and producer beans using:
info -d kafka.consumer -b
kafka.consumer:client-id=app-c7117b6f-3af1-473a-a87a-1d981574c071-StreamThread-1-consumer,type=kafka-metrics-count
info -d kafka.producer -b
kafka.producer:client-id=app-c7117b6f-3af1-473a-a87a-1d981
This helps some. W’re planning to write a non-homogeneous set of records to a
single topic (to preserve order). There would be no compatibility between
records of different types. I assume that if I set the schema compatibility
for this subject to “none” this would not be a problem. (can you
i have been playing with kafka connect in standalone and distributed mode.
i like standalone because:
* i get to configure it using a file. this is easy for automated deployment
(chef, puppet, etc.). configuration using a rest api i find inconvenient.
* erors show up in log files instead of having
How about defining an avro union type containing all the schemas you wish
to put on this topic (the schemas themselves could be defined independently
and then bundled into an "uber-schema" at build time)?
That means any messages you put on the topic must match one of the schemas
defined in the uni
Hi,
Is there a way to view message timestamp using console consumer ?
Thanks,
Meghana
Hi,
I think Eno's separation is very clear and helpful. In order to
streamline this discussion, I would suggest we focus back on point (1)
only, as this is the original KIP question.
Even if I started to DSL design discussion somehow, because I thought it
might be helpful to resolve both in a sin
Hi Jendrik,
Could you share with us what is your specified application id and client id?
The reason that "app-c7117b6f-3af1-473a-a87a-1d981574c071" is used as the
client id could be due to that client id was not specified in the configs.
Guozhang
On Mon, Jan 30, 2017 at 4:36 AM, Jendrik Poloc
Hi Meghana,
Please Try : kafka-console-consumer.sh --property print.timestamp=true
- Amrit
On Mon, Jan 30, 2017 at 10:20 PM, Meghana Narasimhan <
mnarasim...@bandwidth.com> wrote:
> Hi,
> Is there a way to view message timestamp using console consumer ?
>
> Thanks,
> Meghana
>
Hi Amrit,
I tried that but received the following warning message,
WARN The configuration 'print.timestamp' was supplied but isn't a known
config. (org.apache.kafka.clients.consumer.ConsumerConfig)
Thanks,
Meghana
On Mon, Jan 30, 2017 at 1:09 PM, Amrit Jangid
wrote:
> Hi Meghana,
>
>
> Please
If it comes to that we may consider it. However will will have a LOT of
different schemas coming through and new ones added frequently.
(Seems we’ve also seen issues that the Schema Registry doesn’t allow references
to anything not in the same “file” for lack of a better term, that would become
I want to do a one:many join between two streams. There should be ~ 1:100
with < 1% having no match.
My topology is relatively simple:
KTable1.join(KTable2)->to("other topic")
\
\---> toStream().print()
In the join it takes both Value1 and Value2 as JSON, converts
Hello Folks,
We have addressed all the comments collected so far, and would like to
propose a voting thread this Wednesday. If you have any further comments on
this KIP, please feel free to continue sending them on this thread before
that.
Guozhang
On Mon, Jan 30, 2017 at 1:10 PM, Jason Gustaf
If you join two KTables, one-to-many join is currently not supported
(only one-to-one, ie, primary key join).
In upcoming 0.10.2 there will be global-KTables that allow something
similar to one-to many joins -- however, only for KStream-GlobalKTable
joins, so not sure if this can help you.
About
Thanks Matthias for your reply.
I'm not trying to stop the application. I'm importing inventory from CSV
files coming from 3rd party sources. The CSVs are snapshots for each
source's inventory. I need to delete all items from that source that
doesn't exist in the latest CSV file.
I was thinking o
On Fri, Jan 27, 2017 at 10:49 AM, Phillip Mann wrote:
> I am looking to product ionize and deploy my Kafka Connect application.
> However, there are two questions I have about the tasks.max setting which
> is required and of high importance but details are vague for what to
> actually set this va
Not sure what special characters you are referring to, but for data in the
key and value fields in Kafka, it handles arbitrary binary data. "Special
characters" aren't special because Kafka doesn't even inspect the data it
is handling: clients tell it the length of the data and then it copies that
On Mon, Jan 30, 2017 at 8:24 AM, Koert Kuipers wrote:
> i have been playing with kafka connect in standalone and distributed mode.
>
> i like standalone because:
> * i get to configure it using a file. this is easy for automated deployment
> (chef, puppet, etc.). configuration using a rest api i
For MySQL you would either want to use Debezium's connector (which can
handle bulk dump + incremental CDC, but requires direct access to the
binlog) or the JDBC connector (does an initial bulk dump + incremental
queries, but has limitations compared to a "true" CDC solution).
Sqoop and the JDBC co
The log compaction functionality uses the key to determine which records to
deduplicate. You can think of it (very roughly) as deleting entries from a
hash map as the value for each key is overwritten. This functionality
doesn't have much of a point unless you include keys in your records.
-Ewen
Note that the documentation that you linked to for upgrades specifically
lists configs that you need to be careful to adjust in your
server.properties.
In fact, the server.properties shipped with Kafka is meant for testing
only. There are some configs in the example server.properties that are not
What are the 26 additional bytes? That sounds like a header that a
decoder/deserializer is handling with the high level consumer. What class
are you using to deserialize the messages with the high level consumer?
-Ewen
On Fri, Jan 27, 2017 at 10:19 AM, Anjani Gupta
wrote:
> I am using kafka_2.1
Stevo,
Agreed that this seems broken if we're just timing out trying to fetch
metadata if we should be able to tell that the topic will never be created.
Clients can't explicitly tell whether auto topic creation is on. Implicit
indication via the error code seems like a good idea. My only concern
31 matches
Mail list logo
