Wired behavior of DATE_FORMAT UDF in Flink SQL

Hi Experts, I found that DATE_FORMAT(timestamp,format) returns a TIMESTAMP type, it is wired, because normally format result should be a string type. In document it says “Formats timestamp as a string using a specified format string”. But when I run it in Flink SQL, selec

Re: Table API and AVG on dates

Hi Flavio, You are right. Avg on dates is not supported. It requires numeric types. As a workaround, you can transform the datetime into a numeric type using an udf. Best, Hequn On Wed, Oct 31, 2018 at 1:51 AM Flavio Pompermaier wrote: > Hi to all, > I'm using Flink 1.6.1 and it seems that ave

Re: "405 HTTP method POST is not supported by this URL" is returned when POST a REST url on Flink on yarn

Hi Gary, Thank you for your help. Finally I get the host and address of application master using yarn client. Best Henry > 在 2018年9月26日，下午2:26，Gary Yao > 写道： > > Hi Henry, > > The URL below looks like the one from the YARN proxy (note that "proxy

Re: "405 HTTP method POST is not supported by this URL" is returned when POST a REST url on Flink on yarn

Hi Gary, Thank you for your help. Finally I get the host and address of application master using yarn client. Best Henry > 在 2018年9月26日，下午2:26，Gary Yao 写道： > > Hi Henry, > > The URL below looks like the one from the YARN proxy (note that "proxy" > appears in the URL): > >

Re: Savepoint failed with error "Checkpoint expired before completing"

Hi Gagan, I have met with the error the checkpoint timeout too. In my case, it is not due to big checkpoint size, but due to slow sink then cause high backpressure to the upper operator. Then the barrier may take a long time to arrive to sink. Please check if it is the ca

Re: Flink Kafka to BucketingSink to S3 - S3Exception

Hi, thanks for the info Rafi, that seems to be related. I hope Flink version 1.6.2 fixes this. Has anyone encountered this before? I would also like to note that my jar includes a core-site.xml file that uses *s3a*. Is this the recommended configuration to use with BucketingSink? Should the

CsvInputFormat - read header line first

Hi, When we are splitting a csv file into multiple parts we are not sure which part is read first. Is there any way to make sure first part with header is read first ? I need to read header line first to store column name vs index and use this index for processing next records. I could read heade

Re: How to tune Hadoop version in flink shaded jar to Hadoop version actually used?

Hi Hequn & Vino, Finally I rebuild the Flink by change the “hadoop.version” in the pom file. Because Flink use maven shaded plugin to shade the Hadoop dependency, this also means I need to rebuild the hadoop shaded jar each time I upgrade Flink version. Best Henry > 在 2018年10月

Re: Flink CEP Watermark Exception

Following up, we are using Flink 1.5.0 and Flink-CEP 2.11. Thanks, Austin On Tue, Oct 30, 2018 at 3:58 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hi there, > > > We have a streaming application that uses CEP processing but are getting this > error fairly frequently after a che

Flink CEP Watermark Exception

Hi there, We have a streaming application that uses CEP processing but are getting this error fairly frequently after a checkpoint fails, though not sure if it is related. We have implemented both `hashCode` and `equals()` using `Objects.hash(...properties)` and basic equality, respectively. Has

Table API and AVG on dates

Hi to all, I'm using Flink 1.6.1 and it seems that average on dates is not supported..am I right? Is there any effort in implementing it? Best, Flavio

Re: [E] Re: Needed more information about jdbc sink in Flink SQL Client

Timo, It helped, I implemented AppendStream table sink and factory and able to configure JDBC sink. Thank you. Regards, N.Surat On Mon, Oct 29, 2018 at 7:47 PM Timo Walther wrote: > Hi, > > all supported connectors and formats for the SQL Client with YAML can be > found in the connect sectio

[ANNOUNCE] Weekly community update #44

Dear community, this is the weekly community update thread #44. Please post any news and updates you want to share with the community to this thread. # Flink 1.5.5 and 1.6.2 have been released The Flink community is proud to announce the two bugfix release Flink 1.5.5 [1] and Flink 1.6.2 [2] hav

Re: Job fails to restore from checkpoint in Kubernetes with FileNotFoundException

As Vino pointed out, you need to configure a checkpoint directory which is accessible from all TMs. Otherwise you won't be able to recover the state if the task gets scheduled to a different TaskManager. Usually, people use HDFS or S3 for that. Cheers, Till On Tue, Oct 30, 2018 at 9:50 AM vino ya

flink-1.6.2 in standalone-job mode | Cluster initialization failed.

Hi, just tried to launch flink app in flink-1.6.2 and get 2018-10-30 11:07:19,961 ERROR org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Cluster initialization failed. java.lang.AbstractMethodError: org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createResourceManager(Lorg/apa

Savepoint failed with error "Checkpoint expired before completing"

Hi, We have a flink job (flink version 1.6.1) which unions 2 streams to pass through custom KeyedProcessFunction with RocksDB state store which final creates another stream into Kafka. Current size of checkpoint is around ~100GB and checkpoints are saved to s3 with 5 mins interval and incremental c

Re: Unbalanced Kafka consumer consumption

I think my problem is not the same, yours is that you want to consume from partitions with more data faster to avoid consuming first the one with less elements which could advance the event time too fast. Mine is that Kafka only consumes from some partitions even if it seems that it has resources t

Re: Job fails to restore from checkpoint in Kubernetes with FileNotFoundException

Hi John, Is the file system configured by RocksDBStateBackend HDFS?[1] Thanks, vino. [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/state_backends.html#the-rocksdbstatebackend John Stone 于2018年10月30日周二 上午2:54写道： > I am testing Flink in a Kubernetes cluster and am f

Re: Queryable state when key is UUID - getting Kyro Exception

Hi, Jayant Your code looks good to me. And I’ve tried the serialize/deserialize of Kryo on UUID class, it all looks okay. I’m not very sure about this problem. Maybe you can write a very simple demo to try if it works. Jiayi Liao, Best Original Message Sender:Jayant ametawittyam...@gmail.

Re: Unbalanced Kafka consumer consumption

Hi, If I understand your problem correctly, there is a similar JIRA issueFLINK-10348, reported by me. Maybe you can take a look at it. Jiayi Liao,Best Original Message Sender:Gerard garciager...@talaia.io Recipient:fearsome.lucidityfearsome.lucid...@gmail.com Cc:useru...@flink.apache.org Date

Re: FlinkCEP, circular references and checkpointing failures

This is some problem with serializing your events using Kryo. I'm adding Gordon to cc, as he was recently working with serializers. He might give you more insights what is going wrong. Best, Dawid On 25/10/2018 05:41, Shailesh Jain wrote: > Hi Dawid, > > I've upgraded to flink 1.6.1 and rebased