Re: serialization issue in streaming job run with scala Future

Hi Debasish, I guess the reason is something unexpectedly involved in serialization due to a reference from inner class (anonymous class or lambda expression). When Flink serializes this inner class instance, it would also serialize all referenced objects, for example, the outer class instance. If

Re: Problem starting taskexecutor daemons in 3 node cluster

Hi Till, Thank you for the reply. I tried to ssh each of the nodes individually with each other and they all can connect to each other. Its just that all the other worker nodes cannot for some reason. connect to the job manager on 150.82.218.218:6123. (Node 1) I got around the problem by setting

Re: Can I do a lookup in FlinkSQL?

The lookup fashion Temporal Join[1] should be a solution for your case and there is an ITCase as an example[2] [1] https://github.com/apache/flink/blob/master/flink-table/flink-table-common/src/main/java/org/apache/flink/table/sources/LookupableTableSource.java [2] https://github.com/apache/flink/

Re: Add Bucket File System Table Sink

Great to hear. Best, Kurt On Tue, Sep 17, 2019 at 11:45 AM Jun Zhang <825875...@qq.com> wrote: > > Hi Kurt: > Thanks. > When I encountered this problem, I found a File System Connector, but its > function is not powerful enough and rich. > I also found that it is built into Flink, there are man

?????? Add Bucket File System Table Sink

Hi Kurt: Thanks. When I encountered this problem, I found a File System Connector, but its function is not powerful enough and rich. I also found that it is built into Flink, there are many unit tests that refer to it, so I dare not easily modify it to enrich its functions

Re: Extending Flink's SQL-Parser

Hi Dominik, I think you can check "parserImpls.ftl" to find out how Flink extends Calcite's original syntax to support features like CREATE TABLE and DROP TABLE, and follow those examples to implement your own syntax. It may also be helpful to check the pom.xml of flink-sql-parser to see how we us

Re: Add Bucket File System Table Sink

Thanks. Let me clarify a bit more about my thinkings. Generally, I would prefer we can concentrate the functionalities about connector, especially some standard & most popular connectors, like kafka, different file system with different formats, etc. We should make these core connectors as powerful

Re: Add Bucket File System Table Sink

Hi Jun, Thanks for bringing this up, in general I'm +1 on this feature. As you might know, there is another ongoing efforts about such kind of table sink, which covered in newly proposed partition support reworking[1]. In this proposal, we also want to introduce a new file system connector, which

Add Bucket File System Table Sink

Hello everyone: I am a user and fan of flink. I also want to join the flink community. I contributed my first PR a few days ago. Can anyone help me to review my code? If there is something wrong, hope I would be grateful if you can give some advice. This PR is mainly in the proce

Re: serialization issue in streaming job run with scala Future

My main question is why serialisation kicks in when I try to execute within a `Future` and not otherwise. regards. On Mon, 16 Sep 2019 at 4:46 PM, Debasish Ghosh wrote: > Yes, they are generated from Avro Schema and implements Serializable .. > > On Mon, Sep 16, 2019 at 4:40 PM Deepak Sharma >

Extending Flink's SQL-Parser

Hey there, I have to extend Flink's SQL-parser such that it accepts and evaluates select-queries with different syntax. Furthermore I use Eclipse Oxygen with Maven plugin and Flink Release 1.8. 0. What I believe to know: For parsing SQL-queries Flink-Table uses Apache Calcite's SQL-parser. Fl

Re: Compound Keys Using Temporal Tables

Hi, No, this is not possible at the moment. You can only pass a single expression as primary key. A work around might be to put the two fields in a nested field (haven't tried if this works) or combine them in a single attribute, for example by casting them to VARCHAR and concating them. Best, Fa

Re: Flink 1.9, MapR secure cluster, high availability

Hi Stephan, sorry for the late answer, didn't have access to cluster. Here is log and stacktrace. Hope this helps, Maxim. - 2019-09-16 18:00:31,804 INFO org.apache.fli

Re: Contribute as freelancer on a flink project from scratch

Hi Deepak, There is a similar thread[1] recently. Could you please checkout it first? Briefly, Flink hasn't maintained a good first issue list so far. But the community is open to answer questions from you and if you provide more information about topics you might be interested in, members who ar

Contribute as freelancer on a flink project from scratch

Hi All I am looking to contribute on a flink project (May be a enterprise use case) Please do let me know if anyone got such opportunities. Thanks Deepak -- Thanks Deepak www.bigdatabig.com www.keosha.net

Joining Pojos

Hi All I'm new to Flink. I am having a lot of success but I'm struggling with Windowed joins over Pojos. In a toy example I am trying to respond to flight delay events and pull some fields from flight details: flightDelaysStream .map(new FlightDelaysMapper()) .join(flightDetailsStream) * .wher

Re: Can I do a lookup in FlinkSQL?

You  can  try to use UDTF -- Original -- From: srikanth flink

Re: Best way to compute the difference between 2 datasets

Hi Ken, Thanks for the suggestion, that idea should also work for implementing a data set difference operation, which is what concerns me here. However, I was also curious about why there is so much performance difference between using sortPartition and sorting in memory by partition, for datasets

Re: serialization issue in streaming job run with scala Future

Yes, they are generated from Avro Schema and implements Serializable .. On Mon, Sep 16, 2019 at 4:40 PM Deepak Sharma wrote: > Does TaxiRide or TaxiRideFare implements Serializable? > > On Mon, Sep 16, 2019 at 3:47 PM Debasish Ghosh > wrote: > >> Hello - >> >> The following piece of code is an

Re: serialization issue in streaming job run with scala Future

Does TaxiRide or TaxiRideFare implements Serializable? On Mon, Sep 16, 2019 at 3:47 PM Debasish Ghosh wrote: > Hello - > > The following piece of code is an example of a connected data streams .. > > val rides: DataStream[TaxiRide] = > readStream(inTaxiRide) > .filter { ride ⇒ ride.getIsSt

serialization issue in streaming job run with scala Future

Hello - The following piece of code is an example of a connected data streams .. val rides: DataStream[TaxiRide] = readStream(inTaxiRide) .filter { ride ⇒ ride.getIsStart().booleanValue } .keyBy("rideId") val fares: DataStream[TaxiFare] = readStream(inTaxiFare) .keyBy("rideId")

Re: [DISCUSS] Drop older versions of Kafka Connectors (0.9, 0.10) for Flink 1.10

Hi, I’m also in favour of at least dropping support for Kafka 0.8. More generally, I think we have to be careful when recommending to use Flink connectors from older versions with a newer Flink versions. The interfaces might be stable but they might use internal code that is not stable, as show

Re: Use of loading flink-conf.yaml in Flink-sql client

Hi Dipanjan, not every configuration options in the flink-conf.yaml are relevant for the SQL client. If you submit to an already existing cluster, then you only need to learn about the address and the port or if it is using high availability where ZooKeeper is running. However, in the general case

Re: [DISCUSS] Drop older versions of Kafka Connectors (0.9, 0.10) for Flink 1.10

I second Stephan here. Moreover, given that the source and sink interfaces are public API, it should still be possible to take the Kafka 0.8/0.9 connector from a previous Flink version and run it with a newer version. Cheers, Till On Mon, Sep 16, 2019 at 10:06 AM Stephan Ewen wrote: > I think t

Re: Flink operators for Kubernetes

Thanks a lot for letting the community know Anand. Great to see that Flink's open source ecosystem is growing. Have you reached out to Robert or Becket in order to include the flinkk8soperator project on Flink's ecosystem website [1]? I've cross posted this thread to the user ML so that Flink user

Re: [DISCUSS] Drop older versions of Kafka Connectors (0.9, 0.10) for Flink 1.10

I think this is very hypothetical, requiring a major version bump to drop an old connector version. What is the actual problem that would arise for users? We have not required a major version to drop some old connectors that in the past, and if we want to continue "mono repo" style in Flink, I don

Re: Flink on yarn use jar on hdfs

And could you please share your github account with me? I am interested to follow you to see how you achieve this feature? Thank you. On 9/16/2019 14:44，Yang Wang wrote： Hi Shengnan, I think you mean to avoid uploading flink-dist jars in submission every time. I ha