Re: [DISCUSS] Update Scala 2.12.7 to 2.12.15

2022-06-07 Thread Ran Tao
Hi, Martijn. I have asked the scala community, i will reply to you if there is any respond. however, i think you can vote or discuss with 2.12.15. The 2.12.16 just backport some functionalies from higher scala version, such as jvm11, jvm-17 target, the flink may need it. But as Chesnay says, we do

Re: Flink config driven tool ?

2022-06-07 Thread Shengkai Fang
Hi. I am not sure whether Flink SQL satisfies your requirement or not. You can just write the SQL in the file and use the SQL Client to submit it to your cluster. We have a quick start in the Flink CDC and you can make a try[1]. Best, Shengkai [1] https://ververica.github.io/flink-cdc-connector

Re: Cannot cast GoogleHadoopFileSystem to hadoop.fs.FileSystem to list file in Flink 1.15

2022-06-07 Thread 陳昌倬
On Tue, Jun 07, 2022 at 03:26:43PM +0800, czc...@gmail.com wrote: > On Mon, Jun 06, 2022 at 10:42:08AM +0800, Shengkai Fang wrote: > > Hi. In my experience, the step to debug classloading problems are as > > follows: > > Thanks for the help. We get the following log when using > `-verbose:class`:

RE: Apache Flink - Rest API for num of records in/out

2022-06-07 Thread Hailu, Andreas
Hi M, We had a similar requirement – we were able to solve for this by: 1. Supply the operators we’re interested in acquiring metrics for through the various name() methods 2. Use the jobid API [1] and find the operator we’ve named in the “vertices” array [1] https://nightlies.ap

Apache Flink - Rest API for num of records in/out

2022-06-07 Thread M Singh
Hi Folks: I am trying to find if I can get the number of records for an operator using flinks REST API.  I've checked the docs at  https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/ops/rest_api/.    I did see some apis that use vertexid, but could not find how to that info without

Re: Flink config driven tool ?

2022-06-07 Thread sri hari kali charan Tummala
Thanks, I'll check it out. On Tue, Jun 7, 2022, 1:31 PM Austin Cawley-Edwards wrote: > They support Flink as well. Looks like they even support the new Flink k8s > operator.[1] > > Austin > > [1]: > https://seatunnel.apache.org/docs/2.1.1/start/kubernetes#deploying-the-operator > > On Tue, Jun 7

Re: Flink config driven tool ?

2022-06-07 Thread Austin Cawley-Edwards
They support Flink as well. Looks like they even support the new Flink k8s operator.[1] Austin [1]: https://seatunnel.apache.org/docs/2.1.1/start/kubernetes#deploying-the-operator On Tue, Jun 7, 2022 at 3:11 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > thanks but looks lik

Re: Flink config driven tool ?

2022-06-07 Thread sri hari kali charan Tummala
thanks but looks like a spark tool is there something similar in flink? Thanks Sri On Tue, Jun 7, 2022 at 12:07 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hey there, > > No idea if it's any good, but just saw Apache SeaTunnel[1] today which > seems to fit your requirements. > >

Re: Flink config driven tool ?

2022-06-07 Thread Austin Cawley-Edwards
Hey there, No idea if it's any good, but just saw Apache SeaTunnel[1] today which seems to fit your requirements. Best, Austin [1]: https://seatunnel.apache.org/ On Tue, Jun 7, 2022 at 2:19 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi Flink Community, > > can someone p

Flink config driven tool ?

2022-06-07 Thread sri hari kali charan Tummala
Hi Flink Community, can someone point me to a good config-driven flink data movement tool Github repos? Imagine I build my ETL dag connecting source --> transformations --> target just using a config file. below are a few spark examples:- https://github.com/mvrpl/big-shipper https://github.com/Bi

Re: Needed help with skipping savepoint state (unsure how to set --allowNonRestoredState in Docker)

2022-06-07 Thread Chesnay Schepler
You are on the right path with using the --allowNonRestoredState flag; we'll just have to find the right place to put it w.r.t. your setup. Which docker images are you using (flink/statefun/something custom), and how do you submit the job? On 03/06/2022 01:17, Bhavani Balasubramanyam wrote:

Re: Flink source Code Explanation

2022-06-07 Thread Martijn Visser
Hi, The documentation links are working as expected. Best regards, Martijn Op ma 6 jun. 2022 om 04:32 schreef sri hari kali charan Tummala < kali.tumm...@gmail.com>: > I am getting a connection timed out error in firefox and google chrome can > you double-check whether the weblink is working o

Re: Add me to slack

2022-06-07 Thread Martijn Visser
Hi everyone, The link at https://flink.apache.org/community.html#slack is usually your best way to get in; it's updated whenever the 100 sign-up limit has been reached. Best regards, Martijn Op ma 6 jun. 2022 om 07:39 schreef Chengxuan Wang : > Hi Jing, > Could you also send the invite to me?

Re: [DISCUSS] Update Scala 2.12.7 to 2.12.15

2022-06-07 Thread Martijn Visser
Hi all, @Ran When I created this discussion thread, the latest available version of Scala 2.12 was 2.12.15. I don't see 2.12.16 being released yet, only discussed. I did read that they were planning it in the last couple of weeks but haven't seen any progress since. Do you know more about the time

Re: Unable to retrieve savepoint status from non-leader/standby in HA with Flink 1.15

2022-06-07 Thread Chesnay Schepler
I think your analysis is correct; I'll file a ticket. On 03/06/2022 15:28, Nick Birnberg wrote: Hello everyone! Our current setup has us running Flink on Kubernetes in HA mode (Zookeeper) with multiple JobManagers. This appears to be a regression from 1.14. We can use the flink CLI to commu

Re: Add me to slack

2022-06-07 Thread Márton Balassi
Seems you have been added or found the link since then. :-) https://flink.apache.org/community.html#slack On Tue, Jun 7, 2022 at 12:28 AM Sahil Aulakh wrote: > Hi > > Please add me to the slack group. > > Thanks > Sahil Aulakh >

Re: Slow tests on Flink 1.15

2022-06-07 Thread Chesnay Schepler
Can you give us a more complete stacktrace so we can see what call in Flink is waiting for something? Does this happen to all of your tests? Can you provide us with an example that we can try ourselves? If not, can you describe the test structure (e.g., is it using a MiniClusterResource). On

Re: Cannot cast GoogleHadoopFileSystem to hadoop.fs.FileSystem to list file in Flink 1.15

2022-06-07 Thread czchen
On Mon, Jun 06, 2022 at 10:42:08AM +0800, Shengkai Fang wrote: > Hi. In my experience, the step to debug classloading problems are as > follows: Thanks for the help. We get the following log when using `-verbose:class`: [2.074s][info][class,load] org.apache.hadoop.fs.FileSystem source: file:/o