Re: dimention table join not work under sql-client fink-1.10.0

2020-03-12 Thread Zhenghua Gao
We don't support 'PROCTIME()' in a temporal table join. Please use a left table's proctime field. [1] [1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/streaming/joins.html#usage-1 *Best Regards,* *Zhenghua Gao* On Fri, Mar 13, 2020 at 11:57 AM wang

Re: Re: dimention table join not work under sql-client fink-1.10.0

2020-03-12 Thread Zhenghua Gao
You are right. The product on alibaba cloud is based on an internal branch. There are much discrepancy between them. *Best Regards,* *Zhenghua Gao* On Fri, Mar 13, 2020 at 1:09 PM wangl...@geekplus.com.cn < wangl...@geekplus.com.cn> wrote: > Thanks, works now. > > Seems the open

Re: How to make two insert-into sqls orderly

2020-03-25 Thread Zhenghua Gao
Hi izual, There is a workaround that you could implement your own sink which write record sink1 and sink2 in turn. *Best Regards,* *Zhenghua Gao* On Wed, Mar 25, 2020 at 10:41 PM Benchao Li wrote: > Hi izual, > > AFAIK, there is no way to to this in pure SQL. > > > > >

Re: Registering UDAF in blink batch app

2020-04-14 Thread Zhenghua Gao
TableEnvironment and use `registerFunction`s. Pls make sure you pass in the correct `isStreamingMode = false` *Best Regards,* *Zhenghua Gao* On Tue, Apr 14, 2020 at 5:58 PM Dmytro Dragan wrote: > Hi All, > > > > Could you please tell how to register custom Aggregation function in bl

Re: JDBC table api questions

2020-04-23 Thread Zhenghua Gao
FLINK-16471 introduce a JDBCCatalog, which implements Catalog interface. Currently we only support PostgresCatalog and listTables(). If you want to get the list of views, you can implement listViews() (currently return an empty list). *Best Regards,* *Zhenghua Gao* On Thu, Apr 23, 2020 at 8:48

Re: How to make two SQLs use the same KafkaTableSource?

2019-08-08 Thread Zhenghua Gao
sqlQuery). *Best Regards,* *Zhenghua Gao* On Fri, Aug 9, 2019 at 12:38 PM Tony Wei wrote: > forgot to send to user mailing list. > > Tony Wei 於 2019年8月9日 週五 下午12:36寫道: > >> Hi Zhenghua, >> >> I didn't get your point. It seems that `isEagerOperationTranslation`

Re: I'm not able to make a stream-stream Time windows JOIN in Flink SQL

2019-08-13 Thread Zhenghua Gao
pache/flink/blob/master/flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/WindowJoinITCase.scala *Best Regards,* *Zhenghua Gao* On Mon, Aug 12, 2019 at 10:49 PM Theo Diefenthal < theo.diefent...@scoop-software.de> wrote: > Hi there, >

Re: I'm not able to make a stream-stream Time windows JOIN in Flink SQL

2019-08-13 Thread Zhenghua Gao
I wrote a demo example for time windowed join which you can pick up [1] [1] https://gist.github.com/docete/8e78ff8b5d0df69f60dda547780101f1 *Best Regards,* *Zhenghua Gao* On Tue, Aug 13, 2019 at 4:13 PM Zhenghua Gao wrote: > You can check the plan after optimize to verify it's a regu

Re: Update tables env after have been register them

2019-08-15 Thread Zhenghua Gao
rce of the coming release 1.9.0[2]. [1] https://issues.apache.org/jira/browse/FLINK-3033 [2] https://github.com/apache/flink/blob/release-1.9/flink-table/flink-table-common/src/main/java/org/apache/flink/table/sources/LookupableTableSource.java *Best Regards,* *Zhenghua Gao* On Thu, Aug 15, 20

Re: processing avro data source using DataSet API and output to parquet

2019-08-16 Thread Zhenghua Gao
-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/sink/filesystem/StreamingFileSink.java [4] https://github.com/apache/flink/blob/master/flink-formats/flink-parquet/src/main/java/org/apache/flink/formats/parquet/ParquetBulkWriter.java *Best Regards,* *Zhenghua Gao* On Fri, Aug 16

Re: processing avro data source using DataSet API and output to parquet

2019-08-19 Thread Zhenghua Gao
,* *Zhenghua Gao* On Fri, Aug 16, 2019 at 11:52 PM Lian Jiang wrote: > Thanks. Which api (dataset or datastream) is recommended for file handling > (no window operation required)? > > We have similar scenario for real-time processing. May it make sense to > use datastream api for both batc

Re: Can I do a lookup in FlinkSQL?

2019-09-16 Thread Zhenghua Gao
/blob/master/flink-table/flink-table-planner-blink/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/AsyncLookupJoinITCase.scala *Best Regards,* *Zhenghua Gao* On Mon, Sep 16, 2019 at 9:23 PM srikanth flink wrote: > Hi there, > > I'm working with streaming in Flin

Re: Joining Pojos

2019-09-17 Thread Zhenghua Gao
POJO is available in KeySelector[1]. Could you provide more information about your problem? Version of Flink? Error messages? [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/api_concepts.html#define-keys-using-key-selector-functions *Best Regards,* *Zhenghua Gao* On Mon, Sep 16

Re: low performance in running queries

2019-10-30 Thread Zhenghua Gao
The reason might be the parallelism of your task is only 1, that's too low. See [1] to specify proper parallelism for your job, and the execution time should be reduced significantly. [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/parallel.html *Best Regards,* *Zhenghu

Re: low performance in running queries

2019-10-30 Thread Zhenghua Gao
I think more runtime information would help figure out where the problem is. 1) how many parallelisms actually working 2) the metrics for each operator 3) the jvm profiling information, etc *Best Regards,* *Zhenghua Gao* On Wed, Oct 30, 2019 at 8:25 PM Habib Mostafaei wrote: > Thanks Gao

Re: low performance in running queries

2019-10-31 Thread Zhenghua Gao
will try to reproduce your scenario and dig the root causes. *Best Regards,* *Zhenghua Gao* On Thu, Oct 31, 2019 at 9:05 PM Habib Mostafaei wrote: > I enclosed all logs from the run and for this run I used parallelism one. > However, for other runs I checked and found that all parallel worke

Re: low performance in running queries

2019-11-03 Thread Zhenghua Gao
ified) or taskmanager.out. It's large (about 4GB in my case) and causes the disk writes high. *Best Regards,* *Zhenghua Gao* On Fri, Nov 1, 2019 at 4:40 PM Habib Mostafaei wrote: > I used streaming WordCount provided by Flink and the file contains text > like "This is some te

Re: Documentation issue maybe

2019-11-06 Thread Zhenghua Gao
https://docs.oracle.com/javase/7/docs/api/java/util/Collections.html#synchronizedList(java.util.List) [2] https://issues.apache.org/jira/browse/FLINK-14650 *Best Regards,* *Zhenghua Gao* On Thu, Nov 7, 2019 at 12:12 AM Romain Gilles wrote: > Hi all, > I think the code example in following se

Re: Does apache flink support stream input from Postgresql ?

2019-11-20 Thread Zhenghua Gao
The jdbc connector can read data from PostgreSQL for Table/SQL users. For pyflink, cc @Hequn *Best Regards,* *Zhenghua Gao* On Wed, Nov 20, 2019 at 7:56 PM Yu Watanabe wrote: > Hello . > > I would like to ask question about possibility of stream read table rows > from PostgresQL u

Re: [DISCUSS] Disable conversion between TIMESTAMP and Long in parameters and results of UDXs

2019-12-01 Thread Zhenghua Gao
ull/10268> [1] https://github.com/apache/flink/pull/10268 [2] https://issues.apache.org/jira/browse/FLINK-14599 *Best Regards,* *Zhenghua Gao* On Sun, Nov 24, 2019 at 8:44 PM Jark Wu wrote: > Hi, > > +1 to disable it in 1.10. I think it's time to disable and correct the >

Re: Table/SQL API to read and parse JSON, Java.

2019-12-03 Thread Zhenghua Gao
the kafka connector jar is missing in your class path *Best Regards,* *Zhenghua Gao* On Mon, Dec 2, 2019 at 2:14 PM srikanth flink wrote: > Hi there, > > I'm following the link > <https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/table/connect.html> &

Re: [DISCUSS] Drop Kafka 0.8/0.9

2019-12-05 Thread Zhenghua Gao
+1 for dropping. *Best Regards,* *Zhenghua Gao* On Thu, Dec 5, 2019 at 11:08 AM Dian Fu wrote: > +1 for dropping them. > > Just FYI: there was a similar discussion few months ago [1]. > > [1] > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/DISCUSS-Drop

Re: [DISCUSS] Set default planner for SQL Client to Blink planner in 1.10 release

2020-01-05 Thread Zhenghua Gao
+1 for making blink planner as the default planner for SQL Client since we have made a huge improvement in 1.10. *Best Regards,* *Zhenghua Gao* On Sun, Jan 5, 2020 at 2:42 PM Benchao Li wrote: > +1 > > We have used blink planner since 1.9.0 release in our production > environ

Re: [ANNOUNCE] Jingsong Lee becomes a Flink committer

2020-02-20 Thread Zhenghua Gao
Congrats Jingsong! *Best Regards,* *Zhenghua Gao* On Fri, Feb 21, 2020 at 11:59 AM godfrey he wrote: > Congrats Jingsong! Well deserved. > > Best, > godfrey > > Jeff Zhang 于2020年2月21日周五 上午11:49写道: > >> Congratulations!Jingsong. You deserve it >> >&g

Re: NoMatchingTableFactoryException when test flink sql with kafka in flink 1.7

2019-01-15 Thread Zhenghua Gao
May be you're generating non-standard JSON record. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Issue with counter metrics for large number of keys

2019-01-16 Thread Zhenghua Gao
So what you want is the counts of every keys ? Why didn't you use count aggregation? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Flink Jdbc streaming source support in 1.7.1 or in future?

2019-01-22 Thread Zhenghua Gao
Actually flink-connectors/flink-jdbc module provided a JDBCInputFormat to read data from a database. u can have a try. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Filter Date type in Table API

2019-01-30 Thread Zhenghua Gao
Just try: filter("f_date <= '1998-10-02'.toDate") -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: ProgramInvocationException when I submit job by 'flink run' after running Flink stand-alone more than 1 month?

2019-02-26 Thread Zhenghua Gao
Seem like there is something wrong with RestServer and the RestClient didn't connect to it. U can check the standalonesession log for investigating causes. btw: The cause of "no cluster was found" is ur pid information was cleaned for some reason. The pid information is stored in ur TMP director