When does JDBC upsert execute DELETE?

2022-08-25 Thread Dhavan Vaidya
Hello, I have postgres as source and mysql as sink. The user authenticating with mysql does _not_ have DELETE privileges. In some cases, flink throws error because it is trying to _delete_ records and the user does not have privilege. In most cases (of the same job), upsert is working as expected

Re: Does kafka key is supported in kafka sink table

2022-05-17 Thread Dhavan Vaidya
Hey wang! Perhaps this is what you want: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/kafka/#key-format & https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/kafka/#key-fields ? Note that the fields *have* to be one of the "top" lev

Re: Converting from table to stream, following Avro schema

2022-05-13 Thread Dhavan Vaidya
> was fixed recently and so it's still not released. > > Regards, > Dian > > On Thu, May 12, 2022 at 7:26 PM Dhavan Vaidya > wrote: > >> Hey Dian, >> >> Though my HTTP call's response is indeed JSON, I needed to serialize data >> into A

Re: Join two data stream using DataStream API using PyFlink

2022-05-13 Thread Dhavan Vaidya
Hi Ian, You need to create two streams, and use https://nightlies.apache.org/flink/flink-docs-master/api/python/pyflink.datastream.html#pyflink.datastream.DataStream.connect to "join" them. On Fri, 13 May 2022 at 14:06, lan tran wrote: > Hi team, I have the use case is that I want to join two d

Re: Converting from table to stream, following Avro schema

2022-05-12 Thread Dhavan Vaidya
o me that having `Row` > inside `Row` is causing some issues. > It supports `Row` inside `Row`. Could you share an example which could > reproduce this issue? > > Regards, > Dian > > > On Tue, May 10, 2022 at 9:09 PM Dhavan Vaidya > wrote: > >> Hel

Converting from table to stream, following Avro schema

2022-05-10 Thread Dhavan Vaidya
Hello, I am consuming Kafka messages with Table API connector. I cannot use DataStream API because it does not support Confluent Avro. After consuming the messages, I am converting to DataStream API and using ProcessFunction. The ProcessFunction makes async http calls and emits results with a com

Re: How should I call external HTTP services with PyFlink?

2022-05-05 Thread Dhavan Vaidya
cio operator supported in > Flink, e.g. you are waiting the results of one element before processing > the next element and so it's actually synchronous from this point of view. > > Regards, > Dian > > On Thu, May 5, 2022 at 9:52 PM Dhavan Vaidya > wrote: > >&g

Re: How should I call external HTTP services with PyFlink?

2022-05-05 Thread Dhavan Vaidya
rator is still not supported in PyFlink. > > Regards, > Dian > > On Tue, May 3, 2022 at 3:48 PM Dhavan Vaidya > wrote: > >> Hey Francis! >> >> Thanks for the insights! I am thinking of using Java / Scala for this >> scenario given your input. Introducing a

Re: How should I call external HTTP services with PyFlink?

2022-05-03 Thread Dhavan Vaidya
gt; might be to use statefun for the enrichment stages. We've also changed our > model for enrichment, we're pushing the enrichment data into the pipeline > instead of pulling it, but this won't work in a lot of situations. > > Hope that gives you some ideas. > > On

How should I call external HTTP services with PyFlink?

2022-05-02 Thread Dhavan Vaidya
Hello! I want to make HTTP(S) calls to enrich data streams. The HTTP services are running on our VPC, so the delay is limited, but sometimes these services end up calling third party APIs, and latencies become high. >From documentation ( https://nightlies.apache.org/flink/flink-docs-release-1.14/