date format in Flink SQL

Hi, I want to convert a string in the format of 1996-8-01 to date and create Table from the dataset of Tuple3 at the end. Since I want to apply SQL queries on the date field of the table, for example, "date_column < 1996-8-01", which java format of date is supported in Flink?

Re: date format in Flink SQL

Hi Soheil, the functions for date/time conversion are pretty limited so far. The full list of supported functions can be found here [1]. If you need more (which is usually the case), it is easy to implement a custom function [2]. We rely on Java's java.sql.Date as a data type. You can use `SE

Re: SQL Client (Streaming + Checkpoint)

Hi Vijay, in general Yun is right, the SQL Client is still in an early prototyping phase. Some configuration features are missing. You can track the progress of this feature here: https://issues.apache.org/jira/browse/FLINK-10265 It should be possible to use the global Flink configuration f

Re: AssertionError: mismatched type $5 TIMESTAMP(3)

Hi Chris, the exception message is a bit misleading. The time attribute (time indicator) type is an internal type and should not be used by users. The following line should solve your issue. Instead of: DataStream> tradesByInstrStream = tableEnv.toRetractStream(tradesByInstr, typeInfo); Yo

Checkpoints failed with NullPointerException

Hi everyone, I am getting NullPointerException when the job is creating checkpoints. My configuration is: Flink 1.7.0 running on AWS EMR, using incremental RockDBStateBackEnd on S3. Sinks are parquet files on S3 and ElasticSearch (I'm not sure whether sinks are relevant to this error). There had b

Re: AssertionError: mismatched type $5 TIMESTAMP(3)

Thanks Timo, I didn't realise supplying Row could automatically apply the correct types. In this case your suggestion doesn't solve the problem though, I still get the exact same error. I assume that's because there isn't a time attribute type on the tradesByInstr table itself, but rather on th

Re: KeyBy is not creating different keyed streams for different keys

Hi Harshith You can replace the GenericDataObject with Tuple3 and keyBy("A", "B") with keyBy(1, 2) then have a try. And you can see the doc[1] for reference also. [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/api_concepts.html#define-keys-using-key-selector-functions Best, Congxi

Re: Checkpoints failed with NullPointerException

I tried to create a savepoint on HDFS, and got the same exception: The program finished with the following exception: org.apache.flink.util.FlinkException: Triggering a savepoint for the job 028e392d02bd229ed08f50a2da5227e2 failed.

Re: Checkpoints failed with NullPointerException

Hi! Thanks for reporting this. This looks like a bug that we fixed in Flink 1.7.1 [1]. Would you be able to try with 1.7.1 and see if the issue is still happening for you? Cheers, Gordon [1] https://issues.apache.org/jira/browse/FLINK-11094 On Tue, Jan 29, 2019, 6:29 PM Averell I tried to cr

Forking a stream with Flink

Hello Flink Community, Let's say I have multiple machines I get data from. I want to process the data of each machine seperately, but in the same way. Is it possible to "fork" a stream by some parameter and then processing the forks indepentently from each other, natively? Or do I need to do that

connecting two streams flink

Team, I have two kafka consumer for same topic and want to join second stream to first after couple of subtasks computation in the first stream then validate the record . KT - C1 ,C2 KT - C1 - Transformation(FlatMap) - Dedup - Validate --ifvalidsave it to DB -C2 - Process

Re: Forking a stream with Flink

Im not sure if i got your question correctly, can you elaborate more on your use case

Re: Forking a stream with Flink

UseCase:- We have kafka consumer to read messages(json ) then it applies to flatmap for transformation based on the rules ( rules are complex ) and convert it to pojo . We want to verify the record(pojo) is valid by checking field by field of that record .if record is invalid due to transformation

Re: connecting two streams flink

If c1 and c2 are listing to the same topic they will consume the same data . so i cant understand this > these two streams one(c2) is fast and other(c1) On Tue, Jan 29, 2019 at 2:44 PM Selvaraj chennappan < selvarajchennap...@gmail.com> wrote: > Team, > > I have two kafka consumer for s

Re: connecting two streams flink

I have pipeline defined . pipeline does source(kafka) ,transformation,dedup and persisting to DB . [image: image.png] Before reaching to DB task lots of transformation is applied on the pipeline Therefore want to validate the record with raw json message which is available in kafka with the tra

Flink Table API Sum method

Hi, How can I use the Flink Table API SUM function? For example something like this: table.agg(sum("feild1"))

Re: Flink Table API Sum method

Hi, It depends if you are using scala or java. In scala: table.agg('field1.sum) In java (this should work in scala as well): table("field1.sum") More examples you can find in the docs[1] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/table/tableApi.html#aggregations On 29

Re: [DISCUSS] Towards a leaner flink-dist

It is not viable for us, as of right now, to release both a lean and fat version of flink-dist. We don't have the required tooling to assemble a correct NOTICE file for that scenario. Besides that his would also go against recent efforts to reduce the total size of a Flink release, as we'd be

Re: Forking a stream with Flink

Hi Selvaraj In your pojo add data member as status or something like that,now set it error in case it is invaild .pass the output of flatmap to split opertor there you can split the stream On Tue, Jan 29, 2019 at 6:39 PM Selvaraj chennappan < selvarajchennap...@gmail.com> wrote: > UseCase:- We h

Filter Date type in Table API

Hi, I want to filter a field of type Date (Java.sql.Date) like the following: filter("f_date <= '1998-10-02'") and filter("f_date <= '1998/10/02'") Expression 'f_date <= 1998/10/02 failed on input check: Comparison is only supported for numeric types and comparable types of same type, got Date

Select feilds in Table API

Hi, I'm trying select some fields: lineitem .select( "l_returnflag," + "l_linestatus," + "l_quantity.sum as sum_qty," + "(l_extendedprice * (l_discount - 1)).sum as sum_disc_price," + "l_extendedprice.sum as s

Re: No resource available error while testing HA

Hi Averell, > Is there any way to avoid this? As if I run this as an AWS EMR job, the job > would be considered failed, while it is actually be restored automatically by > YARN after 10 minutes). You are writing that it takes YARN 10 minutes to restart the application master (AM). However, in my

Re: Forking a stream with Flink

I think there is misunderstanding . I want to compare raw json and transformed record . Hence I need two consumer and merge the stream for comparison. I have pipeline defined . pipeline does source(kafka) ,transformation,dedup and persisting to DB . [image: image.png] Before reaching to DB task l

Re: Flink Yarn Cluster - Jobs Isolation

Run each job individually as described here: https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/yarn_setup.html#run-a-single-flink-job-on-yarn Yes they will run concurrently and be completely isolated from each other. -Jamie On Sun, Jan 27, 2019 at 6:08 AM Eran Twili wr

Connector for IBM MQ

Hi all, I was wondering if anybody has anybody made a connector (Source) to be used with IBM MQ? Also if somebody could point me to any doc on how to write a custom connector, it would be appreciated. We are using Scala 2.12 and Flink 1.7. Kind regards, Jacopo Gobbi Visit our website at http:

Re: Connector for IBM MQ

Yep I did we days back. On Tue, Jan 29, 2019 at 10:13 PM wrote: > Hi all, > > > > I was wondering if anybody has anybody made a connector (Source) to be > used with IBM MQ? > > Also if somebody could point me to any doc on how to write a custom > connector, it would be appreciated. > > > > We ar

Re: Connector for IBM MQ

sorry for typo yep we developed few days back. On Tue, Jan 29, 2019 at 10:27 PM Puneet Kinra < puneet.ki...@customercentria.com> wrote: > Yep I did we days back. > > On Tue, Jan 29, 2019 at 10:13 PM wrote: > >> Hi all, >> >> >> >> I was wondering if anybody has anybody made a connector (Source)

Re: Select feilds in Table API

The problem is that the table "lineitem" does not have a field "l_returnflag". The field in "lineitem" are named [TMP_2, TMP_5, TMP_1, TMP_0, TMP_4, TMP_6, TMP_3]. I guess it depends on how you obtained lineitem. Best, Fabian Am Di., 29. Jan. 2019 um 16:38 Uhr schrieb Soheil Pourbafrani < soheil

Re: Issue setting up Flink in Kubernetes

Hi Tim, There is an end-to-end test in the Flink repository that starts a job cluster in Kubernetes (minikube) [1]. If that does not help you, can you answer the questions below? What docker images are you using? Can you share the kubernetes resource definitions? Can you share the complete logs o

About KafkaConsumer and WM'ing and EventTime charactersitics

In case where one needs t to use kafka event time ( ingestion time ) for watermark generation and timestamp extraction is setting EventTimeCharactersitic as EventTime enough ? Or is this explicit code required ? consumer.assignTimestampsAndWatermarks(new AssignerWithPunctuatedWatermarks() {

Re: About KafkaConsumer and WM'ing and EventTime charactersitics

It seems from https://ci.apache.org/projects/flink/flink-docs-stable/dev/event_time.html that iTimeCharacteristic.IngestionTime should do the trick. Just wanted to confirm that the ingestion time is the event time provided by the kafka producer. On Tue, Jan 29, 2019 at 3:21 PM Vishal Santoshi wr

Re: How to load multiple same-format files with single batch job?

Hi, You can point a file-based input format to a directory and the input format should read all files in that directory. That works as well for TableSources that are internally use file-based input formats. Is that what you are looking for? Best, Fabian Am Mo., 28. Jan. 2019 um 17:22 Uhr schrieb

How to save table with header

Hi, I can save tables in a CSV file like this: TableSink q6Sink = new CsvTableSink(SinkPath, ","); temp.writeToSink(q6Sink); but I want to save the table with the table header as the first line. Is it possible in Flink?

Re: `env.java.opts` not persisting after job canceled or failed and then restarted

Hi Ufuk, I'll answer your question, but first I'll give you an update on how we resolved the issue: * adding `org.apache.hadoop.io.compress.SnappyCodec` to `classloader.parent-first-patterns.additional` in `flink-conf.yaml` (though, putting `org.apache.hadoop.util.NativeCodeLoader` also worked) *

Case When in Flink Table API

How can I use the correct way of *Case When *this example: myTlb.select( "o_orderdate.substring(0,4) as o_year, volume, (when(n_name.like('%BRAZIL%'),volume).else(0)) as case_volume" ) Flink errors on the line (when(n_name.like('%BRAZIL%'),volume).else(0)) as case_volume"

Re: About KafkaConsumer and WM'ing and EventTime charactersitics

Hi Vishal May this doc[1] be helpful for you. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/kafka.html#kafka-consumers-and-timestamp-extractionwatermark-emission Best, Congxian Vishal Santoshi 于2019年1月30日周三 上午4:36写道： > It seems from > https://ci.apache.org/pro

Re: No resource available error while testing HA

Hi Gary, Thanks for the help. Gary Yao-3 wrote > You are writing that it takes YARN 10 minutes to restart the application > master (AM). However, in my experiments the AM container is restarted > within a > few seconds when after killing the process. If in your setup YARN actually > needs 10 minu

Re: `env.java.opts` not persisting after job canceled or failed and then restarted

Hey Aaron, I'm glad to hear that you resolved the issue. I think a docs contribution for this would be very helpful and could update this page: https://github.com/apache/flink/blob/master/docs/monitoring/debugging_classloading.md. If you want to create a separate JIRA ticket for this, ping me wi