Re: Maintain heavy hitters in Flink application

Kostas and Fabian, Thanks for the advice. I guess I will find a workaround to do the state redistribution. I also read about side outputs in this thread, which might be also an option that I will consider. http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Share-state-across-ope

Parallelizing a tumbling group window

Hello, I've inherited some flink application code. We're currently creating a table using a Tumbling SQL query similar to the first example in https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/table/sql. html#group-windows Where each generated SQL query looks something like SELE

Re: aggregate does not allow RichAggregateFunction ?

I understand that. Let me elaborate. The sequence of events is 1. Round robin dispatch to kafka cluster ( it is not partitioned on the key which we may ultimately do and than I will have more questions on how to key y and still keep order, pbly avoid shuffle :) ) . 2. key by a high cardinality k

Exception when using the time attribute in table API

Hi Flink users, I saw this error 12/08/2017 17:31:27 groupBy: (shipmentNumber), window: (TumblingGroupWindow('w$, 'rowtime, 360.millis)), select: (shipmentNumber, SUM(grandTotal) AS EXPR$1) -> to: Row(3/4) switched to FAILED java.lang.NoSuchMethodError: org.apache.flink.api.common.funct

Exception when using the time attribute in table API

Hi Flink users, I saw this error 12/08/2017 17:31:27 groupBy: (shipmentNumber), window: (TumblingGroupWindow('w$, 'rowtime, 360.millis)), select: (shipmentNumber, SUM(grandTotal) AS EXPR$1) -> to: Row(3/4) switched to FAILED java.lang.NoSuchMethodError: org.apache.flink.api.common.funct

Re: REST api: how to upload jar?

Has anyone successfully uploaded to the REST API using command line tools (i.e. curl)? If so, please post an example. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

Sure i’ll Try that. Thanks On Fri, 8 Dec 2017 at 9:18 PM, Stephan Ewen wrote: > I would recommend to add "flink-shaded-hadoop2". That is a bundle of all > Hadoop dependencies used by Flink. > > > On Fri, Dec 8, 2017 at 3:44 PM, Aljoscha Krettek > wrote: > >> I see, thanks for letting us know! >

Re: Flink 1.4.0 RC3 and Avro objects with maps have null values

One thing to be aware of when testing Flink 1.4 is the changes to dependencies and classloading. By default, Flink 1.4 uses now inverted classloading to allow users to use their own copies of dependencies, irrespective of what the underlying classpath is spoiled with. You can for example use a dif

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

I would recommend to add "flink-shaded-hadoop2". That is a bundle of all Hadoop dependencies used by Flink. On Fri, Dec 8, 2017 at 3:44 PM, Aljoscha Krettek wrote: > I see, thanks for letting us know! > > > On 8. Dec 2017, at 15:42, shashank agarwal wrote: > > I had to include two dependencies

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

I see, thanks for letting us know! > On 8. Dec 2017, at 15:42, shashank agarwal wrote: > > I had to include two dependencies. > > hadoop-hdfs (this for HDFS configuration) > hadoop-common (this for Path) > > > ‌ > > On Fri, Dec 8, 2017 at 7:38 PM, Aljoscha Krettek

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

I had to include two dependencies. hadoop-hdfs (this for HDFS configuration) hadoop-common (this for Path) ‌ On Fri, Dec 8, 2017 at 7:38 PM, Aljoscha Krettek wrote: > I think hadoop-hdfs might be sufficient. > > > On 8. Dec 2017, at 14:48, shashank agarwal wrote: > > Can you specifically gui

Re: aggregate does not allow RichAggregateFunction ?

Hi, If you use an AggregatingFunction in this way (i.e. for a window) the ACC should in fact be kept in the state backend. Did you configure the job to use RocksDB? How are the memory problems manifesting? Best, Aljoscha > On 6. Dec 2017, at 14:57, Fabian Hueske wrote: > > Hi Vishal, > > yo

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

I think hadoop-hdfs might be sufficient. > On 8. Dec 2017, at 14:48, shashank agarwal wrote: > > Can you specifically guide which dependencies should I add to extend this : > > https://github.com/apache/flink/blob/release-1.4.0-rc3/flink-connectors/flink-connector-filesystem/src/main/java/org/a

Re: does the flink sink only support bio?

> I have two new questions: > > 1) the async operator must emit some value to the async collector > (even it acts as a sink), right? > I think so, but you should be able to simply return empty collection. > 2) How could I use CheckpointListener with async operator? Could you > give a simple ex

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

Can you specifically guide which dependencies should I add to extend this : https://github.com/apache/flink/blob/release-1.4.0-rc3/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/Bucketer.java is the Hadoop-core is sufficient. ‌ On Fri, Dec 8,

Re: Manual Classloading on the job does not work

Hi, I think the general approach looks good and should be fine. There is even a dedicated end-to-end test checking that this works („ClassLoaderTestProgram“). Have you double checked that all the paths are correct (i.e. the classloading code works in a standalone program with the paths and can

Re: does the flink sink only support bio?

Thank you very much! I have two new questions: 1) the async operator must emit some value to the async collector (even it acts as a sink), right? 2) How could I use CheckpointListener with async operator? Could you give a simple example or doc page? 2017-12-08 18:25 GMT+08:00 Stefan Richter :

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

It's a compilation error. I think I have to include the Hadoop dependencies. ‌ On Fri, Dec 8, 2017 at 6:54 PM, Aljoscha Krettek wrote: > Hi, > > Is this a compilation error or at runtime. But in general, yes you have to > include the Hadoop dependencies if they're not there. > > Best, > Aljos

Re: Can not resolve org.apache.hadoop.fs.Path in 1.4.0

Hi, Is this a compilation error or at runtime. But in general, yes you have to include the Hadoop dependencies if they're not there. Best, Aljoscha > On 8. Dec 2017, at 14:10, shashank agarwal wrote: > > Hello, > > I am trying to test 1.4.0-RC3, Hadoop libraries removed in this version. > A

Re: Problem with runGatherSumApplyIteration

Hi, it would be helpful if you could tell us the Flink version you are using and the full stacktrace. However, this looks like there could be a version conflict, e.g. is your cluster running the same version of Flink that you build your job against? Best, Stefan > Am 08.12.2017 um 10:23 schr

Re: ClassNotFoundException in custom SourceFunction

Hi, Is it possible to go in there with a debugger and see where exactly the code is invoking the ClassLoader? Best, Aljoscha > On 8. Dec 2017, at 14:13, romain.jln wrote: > > Hi, > > FYI, I edited my message on the Nabble archive website because I realised I > sent the wrong stack trace at f

Re: Flink 1.4.0 RC3 and Avro objects with maps have null values

I think this Avro issue might be related: https://issues.apache.org/jira/browse/AVRO-803 > On 7. Dec 2017, at 17:19, Timo Walther wrote: > > Hi Jörn, > > thanks for the little example. Maybe Avro changed the behavior about maps > from the old version we used in 1.3 to the newest version in Fl

Re: ClassNotFoundException in custom SourceFunction

Hi, FYI, I edited my message on the Nabble archive website because I realised I sent the wrong stack trace at first (but I don't know if you've noticed the modification). The first one was actually related to a custom Sink function that sends data to the Eventhub (not sure whether they are related

Can not resolve org.apache.hadoop.fs.Path in 1.4.0

Hello, I am trying to test 1.4.0-RC3, Hadoop libraries removed in this version. Actually, i have created custom Bucketer for the bucketing sink. I am extending org.apache.flink.streaming.connectors.fs.bucketing.Bucketer in the class, i have to use org.apache.hadoop.fs.Path but as hadoop libra

Re: Data loss in Flink Kafka Pipeline

Hi Fabian, Actually I found a JIRA issue for the similar issue : https://issues.apache.org/jira/browse/BEAM-3225 ,This is something similar I am facing too. I have 4 kafka topics as input source. Those are read using GlobalWindow and processingTime triggers. And further joined based on common key

Re: deserilize nested json

Hi Sendoh, it certainly possible to deserialize nested JSON. However, the JsonRowDeserializationSchema doesn't support it yet. You would either have to extend the class or implement a new one. Best, Fabian 2017-12-08 12:33 GMT+01:00 Sendoh : > Hi Flink users, > > Would it be possible to deseri

Re: ClassNotFoundException in custom SourceFunction

Hi, Is the code that is throwing the exception trying to use the Tread Context ClassLoader? If yes, that might explain it because a Thread that you create will not have the correct ClassLoader set. Best, Aljoscha > On 8. Dec 2017, at 12:24, Fabian Hueske wrote: > > Hi, > > thanks a lot for

Re: Data loss in Flink Kafka Pipeline

Hi, Could you maybe post your pipeline code. That way I could have a look. Best, Aljoscha > On 8. Dec 2017, at 12:31, Fabian Hueske wrote: > > Hmm, I see... > I'm running out of ideas. > > You might be right with your assumption about a bug in the Beam Flink runner. > In this case, this wou

deserilize nested json

Hi Flink users, Would it be possible to deserialize nested json by providing TypeInformation? or should we transform nested json to plain json? final TypeInformation typeInformation = Types.ROW( new String[] {"orderNumber", "sales", "country"},

Re: Data loss in Flink Kafka Pipeline

Hmm, I see... I'm running out of ideas. You might be right with your assumption about a bug in the Beam Flink runner. In this case, this would be an issue for the Beam project which hosts the Flink runner. But it might also be an issue on the Flink side. Maybe Aljoscha (in CC), one of the authors

Re: TableException: GenericTypeInfo cannot be converted to Table

Thanks! don't know this works as well. Cheers, Sendoh -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: ClassNotFoundException in custom SourceFunction

Hi, thanks a lot for investigating this problems and the results you shared. This looks like a bug to me. I'm CCing Aljoscha who knows the internals of the DataStream API very well. Which Flink version are you using? Would you mind creating a JIRA issue [1] with all the info you provided so far?

Manual Classloading on the job does not work

Hi I have problems loading resources via the UserCodeClassloader as described in: https://ci.apache.org/projects/flink/flink-docs-release-1.4/monitoring/debugging_classloading.html#manual-classloading-in-the-job I have tried in several scenarios to load properties or wsdl files via a RichMapFunct

Re: TableException: GenericTypeInfo cannot be converted to Table

Yes. Adding .returns(typeInfo) works as well. :-) 2017-12-08 11:29 GMT+01:00 Fabian Hueske : > Hi, > > you give the TypeInformation to your user code but you don't expose it to > the DataStream API (the code of the FlatMapFunction is a black box for the > API). > You're FlatMapFunction should imp

Re: TableException: GenericTypeInfo cannot be converted to Table

Hi, you give the TypeInformation to your user code but you don't expose it to the DataStream API (the code of the FlatMapFunction is a black box for the API). You're FlatMapFunction should implement the ResultTypeQueryable interface and return the TypeInformation. Best, Fabian 2017-12-08 11:19 G

Re: TableException: GenericTypeInfo cannot be converted to Table

Found it. I should use .returns(typeInformation) after the map function. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: ClassNotFoundException in custom SourceFunction

Hi, The stack trace is usually something like : Exception in thread "Thread-49" java.lang.NoClassDefFoundError: com/microsoft/azure/eventhubs/amqp/AmqpErrorCode at com.microsoft.azure.eventhubs.ExceptionUtil.toException(ExceptionUtil.java:30) at com.microsoft.azure.eventhubs.Messa

Re: does the flink sink only support bio?

Hi, Flink currently does not offer async sinks out of the box, but there is no fundamental problem against having them and we will probably offer something is this direction in the future. In the meantime, you can build something like this by replacing the sink with an async io operator that ac

TableException: GenericTypeInfo cannot be converted to Table

Hi Flink users, I found the workarounds to resolve this exception in scala. https://issues.apache.org/jira/browse/FLINK-6500 But I already provide the TypeInformation when deserializing json object, and still see this exception. Is there anything I ignore? The sample code https://gist.github.com

Problem with runGatherSumApplyIteration

Dear All, I got the following error when I use the function "runGatherSumApplyIteration": ... java.lang.NoSuchMethodError: org.apache.flink.api.common.functions.RuntimeContext.hasBroadcastVariable(Ljava/lang/String;)Z I got this problem even when I use the given example from Flink documenta