Re: Question about Flink-SQL

Hi, Roc > Does Flink-SQL support fetching Mysql meta information automaticly in the > latest version, ? If not, could the you adding this feature ? You can obtain the latest meta information(table schema) by using Flink JdbcCalatag[1], only PostgresCatalog is implemented, user can implemented

Trying to run Flink tests

I forked Flink to work on a PR. When I run `mvn clean package` from a clean branch, Maven says the runtime tests failed but the logs do not appear to have details on the failure. Do I have to do anything to run these? ... [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:

Re: Building Flink on VirtualBox VM failing

Thanks for sharing this, I think the activity of OOM-Killer means high memory pressure (it just kills a process with the highest score of memory consumption). High CPU usage can only be a consequence of it, being constant GC. Currently, tests do not run in parallel, but high memory usage can be ca

Re: HA on AWS EMR

Hello Averell, I don't think ZK data is stored on a master node. And Flink JM data is stored usually on DFS - according to "high-availability.storageDir" [1] In either case, for Flink to be HA, Yarn should also be HA. And I think this is not the case with a single master node. Please consider mu

Re: Building Flink on VirtualBox VM failing

Hey, > Currently, tests do not run in parallel I don't think this is true, at least 100%. In 'top' it's clearly visible that there are multiple JVMs. If not running tests in parallel, what are these doing? In the main pom.xml there's configuration for the plug-in 'maven-surefire-plugin'. I'm not

what's the new version of createTemporaryView?

my code is: tEnv.createTemporaryView("orderA", orderA, "user,product,amount"); I got the hint: createTemporaryView is deprecated Then,what's the newest function to achieve above effect? Thanks for your help

Re: ZooKeeper connection SUSPENDING

Hi Kenzyme, at the moment Flink will stop the execution of jobs when it loses its connection to ZooKeeper for whatever reason. If a ZK rolling update can cause the connection loss to the quorum, then it's what you are seeing. FLINK-10052 wants to add a feature which allows Flink to tolerate a SUSP

Re: ZooKeeper connection SUSPENDING

Hi Till, Thank you for the explanation. Looking forward for this feature to be added in the near future. Best, Kenzyme Original Message On Oct. 20, 2020, 8:46 a.m., Till Rohrmann wrote: > Hi Kenzyme, > > at the moment Flink will stop the execution of jobs when it loses its

Extract column and table lineage from flink sql

I want to build a lineage system for a real-time data warehouse，how can I extract table and column lineage from flink sql?

Support of composite data types in flink-parquet

Hello, as stated at https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/connectors/formats/parquet.html “Attention: Composite data type: Array, Map and Row are not supported”. I could not find a JIRA related to that issue, is there any? If not, could I create one to continue the disc

rename error in flink sql

I'm learning "select"from official document my code is: https://paste.ubuntu.com/p/yVYhZqMNq7/ the error I got is:  total is not found in PojoType

Re: Trying to run Flink tests

Hi Dan, The 'mvn package' command automatically includes 'mvn verify', which triggers the test cases. You can skip the tests with 'mvn package -DskipTests'. You can rely on the ci-tests running on Azure Pipeline, either in your own workspace or in the PR. If it is intended to execute the tests lo

Re: what's the new version of createTemporaryView?

Hi, if you check JavaDocs of the deprecated 'createTemporaryView', it suggests to use another overloaded method: void createTemporaryView(String path, DataStream dataStream, Expression... fields); I suppose it should be then: tEnv.createTemporaryView("orderA", orderA, $("user,product,amount"));

Re: rename error in flink sql

Hi, I am not an SQL expert but I would not expect the original POJO to match the new row with the renamed field. Maybe Timo or Dawid have to add something. Best, Andrey On Tue, Oct 20, 2020 at 4:56 PM 大森林 wrote: > > I'm learning "select"from > official document >

Re: Support of composite data types in flink-parquet

Hi Jon, I have found this ticket [1]. It looks like what you are looking for. Best, Andrey [1] https://issues.apache.org/jira/browse/FLINK-17782 On Tue, Oct 20, 2020 at 4:50 PM Jon Alberdi wrote: > Hello, as stated at > https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/connecto

"stepless" sliding windows?

Hey folks! I have an application that wants to use "stepless" sliding windows, i.e. we produce aggregates on every event. The windows need to be of a fixed size, but to have their start and end times update continuously, and I'd like to trigger on every event. Is this a bad idea? I've googled and

Re: Trying to run Flink tests

Hi Xintong! No changes. I tried -X and no additional log information is logged. -DfailIfNoTests=false does not help. `-DskipTests` works fine. I'm going to go ahead and create a PR and see if it fails. Thanks! - Dan On Tue, Oct 20, 2020 at 8:22 AM Xintong Song wrote: > Hi Dan, > > The 'mvn

Re: Remote Stateful Function Scalability

As a note, I wrote that concepts section before remote functions were implemented. I've made a note to myself to go through and update it. Seth On Sat, Oct 17, 2020 at 9:29 PM Tzu-Li (Gordon) Tai wrote: > Hi Elias, > > On Sun, Oct 18, 2020 at 6:16 AM Elias Levy > wrote: > >> After reading the

Re: "stepless" sliding windows?

whoops.. as usual, posting led me to find some answers myself. Does this make sense given my requirements? Thanks! private class MyWindowAssigner(val windowSize: Time) : WindowAssigner() { private val trigger = CountTrigger.of(1) as Trigger override fun assignWindows( element: Re

Re: Building Flink on VirtualBox VM failing

I think you are right and I like the idea of failing the build fast. However, when trying this approach on my local machine it didn't help: the build didn't crash (probably, because of overcommit). Did you try this approach in your VM? Regards, Roman On Tue, Oct 20, 2020 at 12:12 PM Juha Mynttin

Flink memory usage monitoring

Classification: Internal Hi, Environment - Flink 1.11 on K8s Is there a way to monitor the usage of managed memory, off-heap memory and network memory? -Rajesh ::DISCLAIMER:: The contents of this e-mail and any attachment(s) are confidential and intended for t

Re: Setting JDBC connector options using JdbcCatalog

(Re-sending, so it also goes to the list serve) Leonard, That does seem to work for me, thank you! I was looking in the Table API, so that’s why I didn’t find that option. Is there a way to specify the sql hints using the Table API? I had tried appending them onto a call to .from on the table

Some questions regarding operator IDs

Hi team I'm subscribing 2 topics from Kafka Consumer, joining them and publishing back to a new topic via KafkaProducer (with Exactly Once semantic) As it's highly recommended to set uid for each operator, I'm curious how this works. For example, val topicASource = env .addSource(topicAConsume

NullPointerException when trying to read null array in Postgres using JDBC Connector

I believe I am getting an error because I have a nullable postgres array of text that is set to NULL that I’m reading using the JDBC SQL Connector. Is this something that should be allowed? Looking at the source code line below, it doesn’t look like the case of an array being null would be handl

Job Restart Failure

Hi All, I'm facing an issue in our flink application. This happens in version 1.4.0 and 1.7.2. We have both versions and we are seeing this problem on both. We are running flink on ECS and checkpointing enabled to EFS. When the pipeline restarts due to some node failure or any other reason, it jus

?????? rename error in flink sql

Thanks for your replies. um Could you tell me where I am wrong in this code? I'm weak in Java. The pojo is copied from flink official example: https://paste.ubuntu.com/p/kRKxwBpSVK/ the test code is: https://paste.ubuntu.com/p/yVYhZqMNq7/ Thanks for your help~! -- ?

Re: Setting JDBC connector options using JdbcCatalog

Hi, Dylan > Is there a way to specify the sql hints using the Table API? Unfortunately the sql hints only supported in SQL, there’s no plan to support it in Table API AFAK, I’d suggest use SQL if possible, currently SQL has better feature support than Table API. Best, Leonard

?????? rename error in flink sql

Thanks for your help,I have finally modify it and solve it by luck. but new relevent question occur: https://issues.apache.org/jira/browse/FLINK-19746 Could you please have a look at it? I know JIRA is for bug only,but the email will destroy the table I posted, forgive me for posting in JIRA an