KSQL is lagging

Hi, I have one topic with 300 partitions. I have 4 KSQL instances with 8 threads each on 8 core machine. Topic has lag of around million records. Can I increase number of threads equal to number of partitions so hat lag will be reduced? Or I have to reduce partitions to match total number of

KSQL should not silently overwrite Schemas in Registry with same name

Hi all See existing issue: https://github.com/confluentinc/ksql/issues/2427 <https://github.com/confluentinc/ksql/issues/2427> Does anyone know if there is a work around for this? This problem is driving us crazy. We really don't want to allow KSQL streams to constantly change the

Re: KSQL Enrichment issue with converting decimal to string column

We are unfortunately not in control of the source and destination tables. In the source table the 'quantity' column is numeric. In the destination table 'quantityDest' column is a string. So the question is how can we convert from numeric value to string value using

Re: KSQL Enrichment issue with converting decimal to string column

Why is quantity a STRING? Should it be a numeric? -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Tue, 8 Sep 2020 at 08:52, Rainer Schamm wrote: > Hi all > > we are struggling a bit with a fairly simple stream that is supposed to > map columns from a topic popu

KSQL Enrichment issue with converting decimal to string column

Hi all we are struggling a bit with a fairly simple stream that is supposed to map columns from a topic populated by a Source Jdbc Connector into a format destined for a Sink Jdbc Connector. Our tables look something like this: (These are not the actual column names we are using though) SOURC

Re: KSQL in Production

Hi Carl, this is a good start for sure! https://youtu.be/f3wV8W_zjwE Enjoy, Chris On Mon, May 25, 2020 at 17:23 Carl Graving wrote: > Kafka Collective, > Since KSQL came out there is something I have been watching for in the > form of a blog post, article, or somewhere, but I ha

KSQL in Production

Kafka Collective, Since KSQL came out there is something I have been watching for in the form of a blog post, article, or somewhere, but I have yet to see it. I may have missed it over the last year+, so I figured I would check in and see if anyone has resources. Here are my questions. 1

Re: KSQL to search for data in Kafka Topics

I'm not sure what you mean by "not using stream topology". What does that mean to you that you'd rather avoid? However, you can indeed use KSQL to define streams & tables that process data from a number of topics. However, I think you may have the misimpression that KSQ

KSQL to search for data in Kafka Topics

Hello, I am quite new to KSQL, so apologise for misunderstanding it's concept. I have a list of topics that I want to search data for. I am not using stream process, but plain topics which has data retained for 14 days. All i want to do is search for data in SQL-like way as long as it

How can I create a KSQL-DB materialized view that captures the oldest values for each key?

Hello kafka community, The users_table below does not have any records (its topic has no messages starting offset 0), although source topic input_topic has messages: CREATE STREAM USERS_STREAM( accountid STRING, migrationDate BIGINT, metadata STRING) WITH (KAFKA_TOPIC='input_topic', VA

Re: Cannot terminate query on KSQL

Hi, thank you for your reply, I opened an issue here: https://github.com/confluentinc/ksql/issues/5107 <https://github.com/confluentinc/ksql/issues/5107> > Il giorno 17 apr 2020, alle ore 21:01, Guozhang Wang ha > scritto: > > Hi Federico, > > I'd think yo

Re: Cannot terminate query on KSQL

Hi Federico, I'd think you could re-post this question here: https://github.com/confluentinc/ksql/issues. Cheers, Guozhang On Fri, Apr 17, 2020 at 8:57 AM Federico D'Ambrosio wrote: > So, I tried with another query which hangs and I get this, after its > corresponding T

Re: Cannot terminate query on KSQL

So, I tried with another query which hangs and I get this, after its corresponding TERMINATE command: [2020-04-17 15:25:15,796] INFO stream-client [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_ANAGRAFE_CARTE_CREDITO_INTERNAL_AVRO_166-1207a327-6eed-4c74-9787-95825e1bec43] State transition

Re: Cannot terminate query on KSQL

Looking back at the ksql-stream logs: I find for a query I'm trying to terminate this line: [2020-04-17 09:17:52,502] INFO stream-client [_confluent-ksql-default_query_CSAS_PROD_GTEL_AFA_GOL_PREV_GARANZIE_INTERNAL_AVRO_180-2625ef40-c00a-4cdf-98ea-df239aa2f29d] *Already in the pending shu

Cannot terminate query on KSQL

INATE command from KSQL cli just hangs without any feedback. Ksql log is not much helpful, really just showing the POST request being sent. How should we proceed? Is there anyway to force deletion of a query/stream? What could be causing this issue? Thank you, Federico D'Ambrosio

Re: KSQL UDFs with structs as parameters, structs as output

enge is how to implement the merge operation. As already explained, ideally I would like to implement a UDF which could be used as CREATE STREAM output_stream AS SELECT unnest(merge(before, after)), timestamp FROM avro_stream; To output a flat structure. To sum up my questions: 1. First of all: i

KSQL UDFs with structs as parameters, structs as output

quot;value", "before": {}, "after": {}}) Now, I was using KsqlStruct to enforce types and set values, but then I found this issue (https://github.com/confluentinc/ksql/issues/3413). So, I'm wondering if there's any example of how to use Structs in UDFs, since I cannot see

Re: KSQL - SQL query file

You can use RUN SCRIPT command for this See https://docs.confluent.io/current/ksql/docs/tutorials/examples.html#running-ksql-statements-from-the-command-line On Wed, Jan 15, 2020 at 8:12 PM KhajaAsmath Mohammed < mdkhajaasm...@gmail.com> wrote: > Hi, > > I am looking to run

KSQL - SQL query file

Hi, I am looking to run the ksql query with the queries file in windows. May I know how to acheive this? I cannot run directly on CLI as the query has more than 3000+ columns. I am looking to create stream with it Thanks, Asmath

Re: Ksql with schema registry

Thanks Robin, I was able to do it after passing schema registry url to properties file. Ksql-server.properties Sent from my iPhone > On Oct 7, 2019, at 9:12 AM, Robin Moffatt wrote: > > You can specify VALUE_FORMAT='AVRO' to use Avro serialisation and > the Schema Regis

Re: Ksql with schema registry

You can specify VALUE_FORMAT='AVRO' to use Avro serialisation and the Schema Registry. See docs for more details https://docs.confluent.io/current/ksql/docs/installation/server-config/avro-schema.html -- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On

Ksql with schema registry

Hi, What is the configuration that I need to follow to register schema registry on ksql client installed in my machine that connects to cluster. Creating stream with select should create schema automatically in schema registry . Thanks, Asmath Sent from my iPhone

Using the current date/time in a KSQL query

. (ii) I can imagine a similar stream, reading from the same KTABLE where process_date <= 'today' where today is a KSQL system variable. In this scenario I would not expect to have to start and stop the stream each day, and would be relying on the stream being automatically update

KSQL CLI: cannot execute script with SELECT query (but can execute other scripts)

Hi, one more question on KSQL CLI: using "run script /myScript.ksql" commands I can run scripts with "CREATE STREAM ..." and "CREATE TABLE ..." commands - no problem. However, when I try to run a script with a SELECT statement, I'm getting an error: Scrip

Confluent Center: KSQL query shows less results than when running from KSQL CLI

Hi! I'm running the same query from KSQL CLI and from the Confluent Control Center (KSQL Development). I set 'auto.offset.reset' = 'latest'; to only use latest events. Once I send 20 test events - I get full (correct) results from this query in KSQL CLI - but only parti

Re: KSQL Question

Thanks Liam! Generally speaking, do you advise to deploy KSQL on containers? Or is it better to run it on a different tech like VMWare, Openstack or even bare metal nodes? Thanks! On Wed, May 1, 2019 at 10:05 AM Liam Clarke wrote: > Hi Shalom, > > If you're familiar with Docker

Re: KSQL Question

On Wed, May 1, 2019 at 7:04 PM shalom sagges wrote: > Thanks a lot Vahid! > > I will definitely give it a try. > > Thanks again. :) > > On Tue, Apr 30, 2019 at 6:30 PM Vahid Hashemian > > wrote: > > > Hi Shalom, > > > > This is the Github rep

Re: KSQL Question

Hi Shalom, If you're familiar with Docker the Confluent images are fantastic for experimentation and prototyping. https://docs.confluent.io/current/ksql/docs/installation/install-ksql-with-docker.html On Wed, May 1, 2019 at 7:04 PM Liam Clarke wrote: > > > On Wed, May 1, 2019 at

Re: KSQL Question

Thanks a lot Vahid! I will definitely give it a try. Thanks again. :) On Tue, Apr 30, 2019 at 6:30 PM Vahid Hashemian wrote: > Hi Shalom, > > This is the Github repo for KSQL: https://github.com/confluentinc/ksql > However, in order to get that running you have to download f

Re: KSQL Question

Hi Shalom, This is the Github repo for KSQL: https://github.com/confluentinc/ksql However, in order to get that running you have to download few libraries KSQL depends on. And you'll need Kafka. For the sake of experimentation you are probably better off using the all-in-one Confluent Pla

KSQL Question

Hi All, I'm new to Kafka and wanted to experience with KSQL. However, I can't find a place to download it without downloading the entire Confluent Platform package. Please correct me if I'm wrong, but I understand that KSQL is free, so there must be a place where I can download

Re: KStreams / KSQL processes that span multiple clusters

to C would look similar. Come to think of it, you can probably use Mirror Maker for this. Thanks, -John On Wed, Sep 12, 2018 at 4:38 AM Elliot West wrote: > Hello, > > Apologies if this is a naïve question, but I'd like to understand if and > how KStreams and KSQL can deal with

KStreams / KSQL processes that span multiple clusters

Hello, Apologies if this is a naïve question, but I'd like to understand if and how KStreams and KSQL can deal with topics that reside in more than one cluster. For example, is it possible for a single KS[treams|QL] application to: 1. Source from a topic on cluster A 2. Produce/Consu

Re: Kafka stream or ksql design question

Hello Will, Your question is very high-level and hence I felt less guilty to give you a general answer :) You can read the web docs on on Kafka Streams achieve high throughput via data parallelism here: https://kafka.apache.org/11/documentation/streams/architecture Regarding KSQL, you can

Kafka stream or ksql design question

Hi folks, As far as I know, Kafka Stream is a separate process by reading data from topic, transform, and writing to another topic if needed. In this case, how this process supports high throughout stream as well as load balance in terms of message traffic and computing resource for stream proce

Re: Questions about state stores and KSQL

To understand joins better, you might want to check out: https://www.confluent.io/blog/crossing-streams-joins-apache-kafka/ KSQL uses the same join semantics as Kafka Streams. -Matthias On 7/11/18 8:01 AM, Guozhang Wang wrote: > Hello Jonathan, > > At the very high-level, KSQL stat

Re: Questions about state stores and KSQL

Hello Jonathan, At the very high-level, KSQL statements is compiled into a Kafka Streams topology for execution. And the concept "state stores" are for Kafka Streams, not for KSQL, where inside the topology for those processor nodes that need stateful processing, like Joins, one or

Questions about state stores and KSQL

. Let’s take an example. I have an ‘orders’ stream and a ‘users’ table, and I want to enrich the orders events with the corresponding users information, using the KSQL CLI: CREATE STREAM orders_enriched AS SELECT o.id <http://o.id/>, o.article, o.quantity, o.userId, u.name, u.address, u.emai

Ksql

Has anybody used ksql? What are limitations of this and what it’s best for. Would appreciate if you can share some information on this. Best Regards, Praveen Joshi Sent from my iPhone

Re: source code location for KSQL

I guess Ted's 47 minutes faster than me :P On Fri, Apr 27, 2018 at 5:19 PM, Guozhang Wang wrote: > Hey Henry, > > Here you go https://github.com/confluentinc/ksql > > Guozhang > > On Fri, Apr 27, 2018 at 4:23 PM, Henry Cai > wrote: > >> I think KSQL is

Re: source code location for KSQL

Hey Henry, Here you go https://github.com/confluentinc/ksql Guozhang On Fri, Apr 27, 2018 at 4:23 PM, Henry Cai wrote: > I think KSQL is also open sourced, where is the source code location for > KSQL in github? > > Thanks. > -- -- Guozhang

Re: source code location for KSQL

https://github.com/confluentinc/ksql FYI On Fri, Apr 27, 2018 at 4:23 PM, Henry Cai wrote: > I think KSQL is also open sourced, where is the source code location for > KSQL in github? > > Thanks. >

source code location for KSQL

I think KSQL is also open sourced, where is the source code location for KSQL in github? Thanks.

Re: KSQL tumbling window on a custom timestamp message field

Hi Florin, just want to point out that KSQL is not part of Apache Kafka and you might ask your questions on Confluent Google Group instead: https://groups.google.com/forum/#!forum/confluent-platform To answer your question: by default KSQL uses the record embedded metadata timestamp for tumbling

KSQL tumbling window on a custom timestamp message field

Hello! I've read about the KSQL streaming SQL for Kafka. In my opinion, this is a big step for performing complex event processing on streams. In the provides examples there is a TUMBLING window function. By default, I assume that the windowing is performed on the timestamp generated b

KSQL Question about detecting events in a given timeframe.

stamp) What I want to detect is if these three events happen in 24 hours together; the order isn't essential. >From what I read, using windowing with Hopping (advance of like 3 seconds) would work. I'm just not sure how the query in KSQL would look. -- -Richard L. Burton III @rburton

Re: KSQL table retention policy

hen > similar to the retention policy on the Kafka topics, is there some way to > set the retention on the KSQL tables/streams based on the age of the row or > something? > > > Thanks! > signature.asc Description: OpenPGP digital signature

KSQL table retention policy

Are the tables and streams persisted on the Kafka brokers? If so then similar to the retention policy on the Kafka topics, is there some way to set the retention on the KSQL tables/streams based on the age of the row or something? Thanks!

Re: KSQL with Apache Kafka

s no problem at all. i dont see why it would > be any different with this one. > >> On Tue, Sep 19, 2017 at 1:38 PM, Buntu Dev wrote: >> >> Based on the prerequisites mentioned on Github, Confluent platform seems to >> be required for using KSQL: >> &g

Re: KSQL with Apache Kafka

ent platform seems to > be required for using KSQL: > > https://github.com/confluentinc/ksql/blob/0.1.x/ > docs/quickstart/quickstart-non-docker.md#non-docker-setup-for-ksql > > > Did anyone try KSQL against vanilla Apache Kafka? > > > Thanks! >

KSQL with Apache Kafka

Based on the prerequisites mentioned on Github, Confluent platform seems to be required for using KSQL: https://github.com/confluentinc/ksql/blob/0.1.x/docs/quickstart/quickstart-non-docker.md#non-docker-setup-for-ksql Did anyone try KSQL against vanilla Apache Kafka? Thanks!