Thank you All.
On Tuesday, 14 March, 2023 at 07:14:05 am IST, yuxia
<luoyu...@alumni.sjtu.edu.cn> wrote:
The plan shows the filters has been pushed down. But remeber, although pused
down, the filesystem table won't accept the filter. So, it'll be still like
scan
all files.
Best regards,
Yuxia
发件人: "Maryam Moafimadani" <maryam.moafimad...@shopify.com>
收件人: "Hang Ruan" <ruanhang1...@gmail.com>
抄送: "yuxia" <luoyu...@alumni.sjtu.edu.cn>, "ravi suryavanshi"
<ravi_suryavan...@yahoo.com>, "Yaroslav Tkachenko" <yaros...@goldsky.com>,
"Shammon FY" <zjur...@gmail.com>, "User" <user@flink.apache.org>
发送时间: 星期一, 2023年 3 月 13日 下午 10:07:57
主题: Re: Are the Table API Connectors production ready?
Hi All,It's exciting to see file filtering in the plan for development. I am
curious whether the following query on a filesystem connector would actually
push down the filter on metadata `file.path`?
Select score, `file.path` from MyUserTable WHERE `file.path` LIKE '%prefix_%'
== Optimized Execution Plan ==
Calc(select=[score, file.path], where=[LIKE(file.path, '%2022070611284%')])
+- TableSourceScan(table=[[default_catalog, default_database, MyUserTable,
filter=[LIKE(file.path, _UTF-16LE'%2022070611284%')]]], fields=[score,
file.path])
Thanks,Maryam
On Mon, Mar 13, 2023 at 8:55 AM Hang Ruan <ruanhang1...@gmail.com> wrote:
Hi, yuxia,I would like to help to complete this task.
Best,Hang
yuxia <luoyu...@alumni.sjtu.edu.cn> 于2023年3月13日周一 09:32写道：
Yeah, you're right. We don't provide filtering files with patterns. And
actually we had already a jira[1] for it.
I was intended to do this in the past, but don't have much time. Anyone who
are insterested can take it over. We're
happy to help review.
[1] https://issues.apache.org/jira/browse/FLINK-17398
Best regards,
Yuxia
发件人: "User" <user@flink.apache.org>
收件人: "Yaroslav Tkachenko" <yaros...@goldsky.com>, "Shammon FY"
<zjur...@gmail.com>
抄送: "User" <user@flink.apache.org>
发送时间: 星期一, 2023年 3 月 13日 上午 12:36:46
主题: Re: Are the Table API Connectors production ready?
Thanks a lot, Yaroslav and Shammon.I want to use the Filesystem Connector. I
tried it works well till it is running. If the job is restarted. It processes
all the files again.
Could not find the move or delete option after collecting the files. Also, I
could not find the filtering using patterns.
Pattern matching is required as different files exist in the same folder.
Regards,Ravi On Friday, 10 March, 2023 at 05:47:27 am IST, Shammon FY
<zjur...@gmail.com> wrote:
Hi Ravi
Agree with Yaroslav and if you find any problems in use, you can create an
issue in jira https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK
. I have used kafka/jdbc/hive in production too, they work well.
Best,Shammon
On Fri, Mar 10, 2023 at 1:42 AM Yaroslav Tkachenko <yaros...@goldsky.com> wrote:
Hi Ravi,
All of them should be production ready. I've personally used half of them in
production.
Do you have any specific concerns?
On Thu, Mar 9, 2023 at 9:39 AM ravi_suryavanshi.yahoo.com via user
<user@flink.apache.org> wrote:
Hi,Can anyone help me here?
Thanks and regards,Ravi
On Monday, 27 February, 2023 at 09:33:18 am IST, ravi_suryavanshi.yahoo.com
via user <user@flink.apache.org> wrote:
Hi Team,
In Flink 1.16.0, we would like to use some of the Table API Connectors for
production. Kindly let me know if the below connectors are production ready or
only for testing purposes.
| Name | Version | Source | Sink |
| Filesystem | | Bounded and Unbounded Scan, Lookup | Streaming Sink, Batch
Sink |
| Elasticsearch | 6.x & 7.x | Not supported | Streaming Sink, Batch Sink |
| Opensearch | 1.x & 2.x | Not supported | Streaming Sink, Batch Sink |
| Apache Kafka | 0.10+ | Unbounded Scan | Streaming Sink, Batch Sink |
| Amazon DynamoDB | | Not supported | Streaming Sink, Batch Sink |
| Amazon Kinesis Data Streams | | Unbounded Scan | Streaming Sink |
| Amazon Kinesis Data Firehose | | Not supported | Streaming Sink |
| JDBC | | Bounded Scan, Lookup | Streaming Sink, Batch Sink |
| Apache HBase | 1.4.x & 2.2.x | Bounded Scan, Lookup | Streaming Sink, Batch
Sink |
| Apache Hive |
Thanks and regards
--
Maryam MoafimadaniSenior Data Developer @Shopify
