Re: Submitting multiple DELETE statements in Flink SQL

Hi Kirill, As far as I know, multiple statements cannot be run by calling an execute() method on each, so you are missing nothing. Try to look at the Statement Set which allows you to submit multiple SQL statements within one Flink app. For example: https://github.com/novakov-alexey/flink-ml-sand

Re: Does Flink serialize events between all operators?

Hi Vadim, Yes, it does serialize objects between operators even if they run within the same Task Manager unless object-reuse configuration is on: https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#pipeline-object-reuse Using immutable data (which is one of the principal o

Re: Unittesting ProcessWindowFunction in Scala

Hi Burak, It seems the variable you have in the unit test has incorrect type or it is not fully compatible: this variable --> stateDesc found : org.apache.flink.api.common.state.ListStateDescriptor[(String, Long)] required: org.apache.flink.api.common.state.StateDescriptor[ _ <: org.

Re: Unittesting async functions with mocked fields

Hi Burak, I recommend not using Mock libraries with Scala as it does not really need that. Just substitute the CaffeineHelper with another implementation for tests. You could create two implementations of the trait CacheHelper[V]: trait CacheHelper[V] extends Serializable { def get(id: String):

Re: Trying to understand watermark in join with FlinkSQL and late events

Hi Guillermo. ORD007 is included due to "LEFT Join" logic. The LEFT JOIN keyword returns all records from the left table, and the matching records from the right table. Watermark table configuration and "FOR SYSTEM_TIME AS OF" do not discard normal LEFT join behavior here. Best regards, Alexey

Re: Flink 1.17 with Scala 2.13 or Scala 3

Hi Patrick, You need to make sure, the pre-packaged Scala runtime is removed from the Flink docker image: > rm flink-dist/lib/flink-scala* This is also mentioned in the Flink blog-post Martijn mentioned earlier. The smallest reproducible example would be required, if you want us to find the ro

Re: Issue with Flink Job when Reading Data from Kafka and Executing SQL Query (q77 TPC-DS)

Hi Vladimir, As I see, your SQL query is reading data from the Kafka topic and pulls all data to the client side. The "*.collect" method is quite network/memory intensive. You probably do want that. If you want to debug/print the ingested data via SQL, I would recommend the "print" connector. htt

Re: dependency error with latest Kafka connector

of the Kafka > connector is meant to be compatible with both Flink 1.17 and 1.18, right? > So the older version should be specified so that the consumer can decide > which Flink version to compile against, otherwise the build tool could > silently update the compile-only dependencies, no? &

Re: dependency error with latest Kafka connector

Hi Günterh, It looks like a problem with the Kafka connector release. https://mvnrepository.com/artifact/org.apache.flink/flink-connector-kafka/3.0.1-1.18 Compile dependencies are still pointing to Flink 1.17. Release person is already contacted about this or will be contacted soon. Best regards

Re: Problem when testing table API in local

Hi, You would need to add the flink-clients module when running in local mode. The *flink-clients* dependency is only necessary to invoke the Flink program locally (for example to run it standalone for testing and debugging). Best regards, Alexey On Fri, Sep 8, 2023 at 3:17 PM Oscar Perez via us

Re: InaccessibleObjectException issue

H Bill, I would recommend you to try that example with JDK 11. Support of JDK 17 is coming in Flink 1.18 https://issues.apache.org/jira/browse/FLINK-15736 Best regards, Alexey On Tue, Aug 29, 2023 at 11:50 AM Xu Bill wrote: > Hello, > > I'm new to Apache Flink. > I followed the instrunctions t

Re: Streaming join performance

Привет Артем! Are your tables backed by Kafka? If - yes, what if you use upsert-kafka connector from Table API , does it help to reduce the number of records in each subsequent join operator? I wrote a blog-p

Re: Question about serialization of java.util classes

Hi Saleh, If you could show us the minimal code example of the issue (event classes), I think someone could help you to solve it. Best regards, Alexey On Mon, Aug 14, 2023 at 9:23 AM wrote: > Hi, > > According to this blog post > https://flink.apache.org/2020/04/15/flink-serialization-tuning-v

Re: Dependency injection framework for flink

I would agree with Ron. If you have a chance to use Scala, then it is much easier to compose Flink process functions (or what have you) into a data stream. Simple Functional Programming power. Coming from a Java background into the Scala ecosystem sometime ago, I was just surprised that proper lan

Re: Application Mode Start Using Restful API

Hi Xiao. I think you need to look at this example: https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/#submitting-a-flink-job Best regards, Alexey On Thu, Jul 6, 2023 at 7:02 PM Xiao Ma wrote: > Hi Alexey, > > Yes, sending a RE

Re: Application Mode Start Using Restful API

Hi Xiao, Do you mean like sending a REST API request to the K8s API Server to start a new Flink job? If - yes, then it is possible today via Flink Kubernetes Operator. Best regards, Alexey On Thu, Jul 6, 2023 at 5:20 PM Xiao Ma wrote: > Hello, > > I am wondering if there is any way to start a

Re: Using HybridSource

Hi Oscar, You could use connected streams and put your file into a special Kafka topic before starting such a job: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/operators/overview/#connect But this may require more work and the event ordering (which is shuffled) in

Re: Average on sliding window

Hi Eugenio, I think it is due to window completion which will be complete once your watermarked field on the event advances 15 days interval since the first received event time. Please also check this default trigger behavior here: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/de

Re: Migrating to Flink 1.17

Hi Patricia, I think one of the think is the memory configuration changes which you will need to take into account and adjust your configuration accordingly: - https://flink.apache.org/2020/04/21/memory-management-improvements-with-apache-flink-1.10/ - https://docs.google.com/spreadsheets/d/1mJaM

Re: How to know when a pipeline ends

Hi Luke, Did you try to use *env.registerJobListener()* to be notified on job execution? https://coderstea.in/post/big-data/flink-job-listener-run-a-task-after-flink-job-is-completed/ Best regards, Alex On Fri, May 12, 2023 at 8:01 PM Luke Xiong wrote: > Hi Weihua and Shammon, > > Thanks for

Re: [Discussion] - Release major Flink version to support JDK 17 (LTS)

If breaking savepoint compatibility will be eventually an option, I would recommend to try to upgrade Flink's Scala even to *2.13* Best regards, Alexey On Fri, Apr 28, 2023 at 10:22 AM Chesnay Schepler wrote: > We don't know yet. I wanted to run some more experiments to see if I cant > get Scal

Re: [Discussion] - Take findify/flink-scala-api under Flink umbrella

rst. >>>> > >>>> > Best, >>>> > D. >>>> > >>>> > On Sun, Apr 16, 2023 at 10:48 AM guenterh.lists < >>>> guenterh.li...@bluewin.ch> >>>> > wrote: >>>> > >>>>

Re: [Discussion] - Take findify/flink-scala-api under Flink umbrella

years, I >> >> predict these wrappers will eventually be abandoned, unless such a >> Scala >> >> library is a part of some bigger community like ASF. >> >> >Also, non-official Scala API will lead people to play safe and choose >> >> Java API only, even if they di

Re: [Discussion] - Take findify/flink-scala-api under Flink umbrella

;>> Java API only, even if they didn't want that at the beginning. >>> >>> Second sentence is my current state. >>> >>> From my point of view it would be very unfortunate if the Flink project >>> would lose the Scala API and thus the integration of con

[Discussion] - Take findify/flink-scala-api under Flink umbrella

Hello Flink PMCs and Flink Scala Users, I would like to propose an idea to take the 3rd party Scala API findify/flink-scala-api project into the Apache Flink organization. *Motivation * The Scala-free Flink idea was finally implemented by the 1.15 rel

Re: Create generic DeserializationSchema (Scala)

Hi Ana, I think you will need to deal with ClassTag to keep all the code generic. I've found such example which should help: https://github.com/amzn/milan/blob/7dfa29b434ced7eef286ea34c5085c10c1b787b6/milan/milan-compilers/milan-flink-compiler/src/main/scala/com/amazon/milan/compiler/flink/serial

Re: Re: Flink SQL support array transform function

Xuekui, I guess you want high-order functions support in Flink SQL like Spark has https://spark.apache.org/docs/latest/api/sql/#transform ? Best regards, Alexey On Wed, Feb 22, 2023 at 10:31 AM Xuekui wrote: > Hi Yuxia and Shammon, > > Thanks for your reply. > > The requirements is dynamic in m

Re: What is the state of Scala wrappers?

gt; > Best regards, > > Martijn > > On Sun, Feb 5, 2023 at 4:39 PM Alexey Novakov via user < > user@flink.apache.org> wrote: > >> Hi Erwan, >> >> I think those 2 projects you mentioned are pretty much the options we >> have at the moment if you want to

Re: What is the state of Scala wrappers?

Hi Erwan, I think those 2 projects you mentioned are pretty much the options we have at the moment if you want to use Scala 2.13 or 3. I believe your contribution to upgrade one of them to Flink 1.16 will be very welcomed. Best regards, Alex On Thu, Feb 2, 2023 at 9:32 AM Erwan Loisant wrote: