Custom GenericRecord Serializer with Tuple?

Hello! I have a custom Avro GenericRecord serializer that supports reading and writing records without having to pass along the schema with every record by using a centralized registry. I've registered it with the execution environment as: environment.addDefaultKryoSerializer(Ge

RE: ExecutionMode in ExecutionConfig

I can give this a try. Do you know which Flink version does this feature become available in? ah From: zhanghao.c...@outlook.com Sent: Wednesday, September 14, 2022 11:10 AM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Subject: Re: ExecutionMode in ExecutionConfig Could you try

RE: ExecutionMode in ExecutionConfig

4, 2022 1:10 AM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Subject: Re: ExecutionMode in ExecutionConfig https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/datastream/execution_mode/<https://urldefense.proofpoint.com/v2/url?u=https-3A__nightlies.apache.org_

RE: Apache Flink - Rest API for num of records in/out

Hi M, We had a similar requirement – we were able to solve for this by: 1. Supply the operators we’re interested in acquiring metrics for through the various name() methods 2. Use the jobid API [1] and find the operator we’ve named in the “vertices” array [1] https://nightlies.ap

RE: FlinkJobNotFoundException

Hi Matthias, the log file is quite large (21MB) so mailing it over in its entirety may have been a challenge. The file is available here [1], and we’re of course happy to share any relevant parts of it with the mailing list. I think since we’ve shared logs with you before in the past, you weren’

RE: Many S3V4AuthErrorRetryStrategy warn logs while reading/writing from S3

Thanks, Robert. // ah From: Robert Metzger Sent: Wednesday, September 22, 2021 1:49 PM To: Hailu, Andreas [Engineering] Cc: user@flink.apache.org Subject: Re: Many S3V4AuthErrorRetryStrategy warn logs while reading/writing from S3 Hey Andreas, This could be related too https://github.com

Many S3V4AuthErrorRetryStrategy warn logs while reading/writing from S3

Hi, When reading/writing to and from S3 using the flink-fs-s3-hadoop plugin on 1.11.2, we observe a lot of these WARN log statements in the logs: WARN S3V4AuthErrorRetryStrategy - Attempting to re-send the request to s3.amazonaws.com with AWS V4 authentication. To avoid this warning in the fu

RE: 1.9 to 1.11 Managed Memory Migration Questions

Thanks Caizhi, this was very helpful. // ah From: Caizhi Weng Sent: Thursday, August 26, 2021 10:41 PM To: Hailu, Andreas [Engineering] Cc: user@flink.apache.org Subject: Re: 1.9 to 1.11 Managed Memory Migration Questions Hi! I've read the first mail again and discover that the direct m

RE: 1.9 to 1.11 Managed Memory Migration Questions

-direct-or-native // ah From: Caizhi Weng Sent: Wednesday, August 25, 2021 10:47 PM To: Hailu, Andreas [Engineering] Cc: user@flink.apache.org Subject: Re: 1.9 to 1.11 Managed Memory Migration Questions Hi! Why does this ~30% memory reduction happen? I don't know how memory is calculated in

1.9 to 1.11 Managed Memory Migration Questions

Hi folks, We're about half way complete in migrating our YARN batch processing applications from Flink 1.9 to 1.11, and are currently tackling the memory configuration migrations. Our test application's sink failed with the following exception while writing to HDFS: Caused by: java.lang.OutOf

RE: Upgrading from Flink on YARN 1.9 to 1.11

things up. // ah From: David Morávek Sent: Tuesday, August 17, 2021 4:37 AM To: Hailu, Andreas [Engineering] Cc: Ravichandran, Soorya Prasanna [Engineering] ; user@flink.apache.org Subject: Re: Upgrading from Flink on YARN 1.9 to 1.11 Hi Andreas, the problem here is that the command you're

RE: Upgrading from Flink on YARN 1.9 to 1.11

David Morávek Sent: Monday, August 16, 2021 6:28 AM To: Hailu, Andreas [Engineering] Cc: user@flink.apache.org Subject: Re: Upgrading from Flink on YARN 1.9 to 1.11 Hi Andreas, Per-job and session deployment modes should not be affected by this FLIP. Application mode is just a new deployment mo

Upgrading from Flink on YARN 1.9 to 1.11

Hello folks! We're looking to upgrade from 1.9 to 1.11. Our Flink applications run on YARN and each have their own clusters, with each application having multiple jobs submitted. Our current submission command looks like this: $ run -m yarn-cluster --class com.class.name.Here -p 2 -yqu queue-na

RE: Unable to use custom AWS credentials provider - 1.9.2

Hi Arvid, no. We are leveraging it as part of our application code, but not Kinesis – after finding and excluding duplicates of this package in our classpath, we are able to submit a job. Thanks. // ah From: Arvid Heise Sent: Friday, July 30, 2021 1:34 PM To: Hailu, Andreas [Engineering] Cc

RE: Obtain JobManager Web Interface URL

n application ... } These metrics prove immensely valuable as they help us optimize performance, diagnose issues, as well as predict resource requirements for applications. // ah -Original Message- From: Yangze Guo Sent: Sunday, August 1, 2021 10:38 PM To: Hailu, Andreas [Engineeri

RE: Unable to use custom AWS credentials provider - 1.9.2

07.21 17:37, Hailu, Andreas [Engineering] wrote: > Hi team, I’m trying to read and write from and to S3 using a custom AWS > Credential Provider using Flink v1.9.2 on YARN. > > > > I followed the instructions to create a plugins directory in our Flink > distribution location and

RE: Obtain JobManager Web Interface URL

rom: Yangze Guo Sent: Thursday, July 29, 2021 11:17 PM To: Hailu, Andreas [Engineering] Cc: user@flink.apache.org Subject: Re: Obtain JobManager Web Interface URL Hi, Hailu AFAIK, the ClusterClient#getWebInterfaceURL has been available since 1.10. Regarding the JobManager web interface, it will

Obtain JobManager Web Interface URL

Hi team, Is there a method available to obtain the JobManager's REST url? We originally overloaded CliFrontend#executeProgram and nabbed it from the ClusterClient#getWebInterfaceUrl method, but it seems this method's signature has been changed and no longer available as of 1.10.0. Best, Andrea

Unable to use custom AWS credentials provider - 1.9.2

Hi team, I'm trying to read and write from and to S3 using a custom AWS Credential Provider using Flink v1.9.2 on YARN. I followed the instructions to create a plugins directory in our Flink distribution location and copy the FS implementation (I'm using s3-fs-hadoop) package into it. I have al

RE: [1.9.2] Flink SSL on YARN - NoSuchFileException

Monday, 19 April 2021 16:45:25 CEST Hailu, Andreas [Engineering] wrote: > Hi Flink team, > > I'm trying to configure a Flink on YARN with SSL enabled. I've > followed the documentation's instruction [1] to generate a Keystore > and Truststore locally, and added a the pr

RE: [1.9.2] Flink SSL on YARN - NoSuchFileException

://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/security-ssl.html#tips-for-yarn--mesos-deployment // ah From: Arvid Heise Sent: Wednesday, April 21, 2021 1:05 PM To: Hailu, Andreas [Engineering] Cc: user@flink.apache.org Subject: Re: [1.9.2] Flink SSL on YARN - NoSuchFileException Hi Andreas

[1.9.2] Flink SSL on YARN - NoSuchFileException

Hi Flink team, I'm trying to configure a Flink on YARN with SSL enabled. I've followed the documentation's instruction [1] to generate a Keystore and Truststore locally, and added a the properties to my flink-conf.yaml. security.ssl.rest.keystore: /home/user/ssl/deploy-keys/rest.keystore securi

Understanding blocking behavior

Hi folks, I'm trying to get a better understanding of what operations result in blocked partitions. I've got a batch-processing job that reads from 2 sources, and then performs a series of Maps/Filters/CoGroups all with the same parallelism to create a final DataSet to be written to two differen

RE: org.apache.flink.runtime.client.JobSubmissionException: Job has already been submitted

Hi Robert, I appreciate you having a look. I’ll have a closer look and see what I can find. Thanks! // ah From: Robert Metzger Sent: Friday, January 22, 2021 2:41 AM To: Hailu, Andreas [Engineering] Cc: user@flink.apache.org Subject: Re: org.apache.flink.runtime.client.JobSubmissionException

RE: org.apache.flink.runtime.client.JobSubmissionException: Job has already been submitted

this application. Is it possible that it’s busy processing other jobs? // ah From: Robert Metzger Sent: Thursday, January 21, 2021 10:00 AM To: Hailu, Andreas [Engineering] Cc: user@flink.apache.org Subject: Re: org.apache.flink.runtime.client.JobSubmissionException: Job has already been

org.apache.flink.runtime.client.JobSubmissionException: Job has already been submitted

Hello, We're running 1.9.2 on YARN, and are seeing some interesting behavior when submitting jobs in a multi-threaded fashion to an application's Flink cluster. The error we see reported in the client application logs is the following: org.apache.flink.client.program.ProgramInvocationException:

RE: Distribute Parallelism/Tasks within RichOutputFormat?

Thanks Chesnay, Flavio – I believe Flavio’s first recommendation will work well enough. I agree that the second approach may be a bit finicky to use long-term. Cheers. // ah From: Chesnay Schepler Sent: Wednesday, December 23, 2020 4:07 AM To: Flavio Pompermaier ; Hailu, Andreas [Engineering

Distribute Parallelism/Tasks within RichOutputFormat?

Hi folks, I've got a single RichOutputFormat which is comprised of two HadoopOutputFormats, let's call them A and B, each writing to different HDFS directories. If a Record matches a certain condition it's written using A, otherwise it's written with B. Currently, the parallelism that is set at

RE: Runtime Dependency Issues Upgrading to Flink 1.11.2 from 1.9.2

submit jobs. Hopefully any other devs who have similar issues will find this thread useful :) // ah From: Leonard Xu Sent: Friday, October 16, 2020 1:10 AM To: Chesnay Schepler Cc: Hailu, Andreas [Engineering] ; user@flink.apache.org Subject: Re: Runtime Dependency Issues Upgrading to Flink

RE: Runtime Dependency Issues Upgrading to Flink 1.11.2 from 1.9.2

o run just fine, which is interesting. // ah From: Chesnay Schepler Sent: Thursday, October 15, 2020 7:34 AM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Subject: Re: Runtime Dependency Issues Upgrading to Flink 1.11.2 from 1.9.2 I'm not aware of any Flink module bundling this c

Runtime Dependency Issues Upgrading to Flink 1.11.2 from 1.9.2

Hi team! We're trying to upgrade our applications from 1.9.2 to 1.11.2. After re-compiling and updating our runtime dependencies to use 1.11.2, we see this LinkageError: Caused by: java.lang.LinkageError: ClassCastException: attempting to castjar:file:/local/data/scratch/hailua_p2epdlsuat/flink

RE: Blobserver dying mid-application

From: Chesnay Schepler Sent: Thursday, October 1, 2020 1:41 PM To: Hailu, Andreas [Engineering] ; Till Rohrmann Cc: user@flink.apache.org; Nico Kruber Subject: Re: Blobserver dying mid-application All jobs running in a Flink session cluster talk to the same blob server. The time when tasks are submi

RE: Blobserver dying mid-application

thousands of Flink applications running concurrently in our YARN cluster. // ah From: Chesnay Schepler Sent: Thursday, October 1, 2020 5:42 AM To: Till Rohrmann ; Hailu, Andreas [Engineering] Cc: user@flink.apache.org Subject: Re: Blobserver dying mid-application It would also be good to know how

Blobserver dying mid-application

Hello folks, I'm seeing application failures where our Blobserver is refusing connections mid application: 2020-09-30 13:56:06,227 INFO [flink-akka.actor.default-dispatcher-18] org.apache.flink.runtime.taskexecutor.TaskExecutor- Un-registering task and sending final execution state

RE: JobManager refusing connections when running many jobs in parallel?

an increase in our sockets’ max connections, but I would like to know your thoughts. // ah From: Hailu, Andreas [Engineering] Sent: Monday, August 17, 2020 9:51 AM To: 'Robert Metzger' Cc: user@flink.apache.org; Shah, Siddharth [Engineering] Subject: RE: JobManager refusing connec

RE: JobManager refusing connections when running many jobs in parallel?

Interesting – what is the JobManager submission bounded by? Does it only allow a certain number of submissions per second, or is there a number of threads it accepts? // ah From: Robert Metzger Sent: Tuesday, August 11, 2020 4:46 AM To: Hailu, Andreas [Engineering] Cc: user@flink.apache.org

Re: JobManager refusing connections when running many jobs in parallel?

capacity on a port somewhere. This sound correct? Is there a config for us to increase the pool size? From: Robert Metzger Sent: Wednesday, July 29, 2020 1:52:53 AM To: Hailu, Andreas [Engineering] Cc: user@flink.apache.org; Shah, Siddharth [Engineering] Subject: Re

JobManager refusing connections when running many jobs in parallel?

Hi team, We've observed that when we submit a decent number of jobs in parallel from a single Job Master, we encounter job failures due with Connection Refused exceptions. We've seen this behavior start at 30 jobs running in parallel. It's seemingly transient, however, as upon several retries t

RE: History Server Not Showing Any Jobs - File Not Found?

application ID (ala Spark) would ease the burden on the dev + ops side. Perhaps a feature for the future :) // ah From: Chesnay Schepler Sent: Tuesday, June 2, 2020 3:55 AM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Subject: Re: History Server Not Showing Any Jobs - File Not Fo

RE: History Server Not Showing Any Jobs - File Not Found?

ative retention of policy? E.g. only keep the latest X archives in the dir - or is this something we need to manage ourselves? Thanks. // ah From: Hailu, Andreas [Engineering] Sent: Friday, May 29, 2020 8:46 AM To: 'Chesnay Schepler' ; user@flink.apache.org Subject: RE: History Server Not S

RE: History Server Not Showing Any Jobs - File Not Found?

Yes, these are all in the same directory, and we're at 67G right now. I'll try with incrementally smaller directories and let you know what I find. // ah From: Chesnay Schepler Sent: Friday, May 29, 2020 3:11 AM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Subject: R

RE: History Server Not Showing Any Jobs - File Not Found?

May I also ask what version of flink-hadoop you're using and the number of jobs you're storing the history for? As of writing we have roughly 101,000 application history files. I'm curious to know if we're encountering some kind of resource problem. // ah From: Hailu,

RE: History Server Not Showing Any Jobs - File Not Found?

4j12-1.7.15.jar Are the files within /lib. // ah From: Chesnay Schepler Sent: Thursday, May 28, 2020 11:00 AM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Subject: Re: History Server Not Showing Any Jobs - File Not Found? Looks like it is indeed stuck on downloading the archive. I

RE: History Server Not Showing Any Jobs - File Not Found?

ecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) What problems could the flink-shaded-hadoop jar being included introduce? // ah From: Chesnay Schepler Sent: Thursday, May 28, 2020 9:26 AM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Subject:

RE: History Server Not Showing Any Jobs - File Not Found?

Hailu, Andreas [Engineering] ; user@flink.apache.org Subject: Re: History Server Not Showing Any Jobs - File Not Found? yes, exactly; I want to rule out that (somehow) HDFS is the problem. I couldn't reproduce the issue locally myself so far. On 01/05/2020 22:31, Hailu, Andreas wrote: Hi Ch

RE: History Server Not Showing Any Jobs - File Not Found?

ctory? // ah From: Chesnay Schepler Sent: Wednesday, April 29, 2020 8:26 AM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Subject: Re: History Server Not Showing Any Jobs - File Not Found? hmm...let's see if I can reproduce the issue locally. Are the archives from the same version t

RE: History Server Not Showing Any Jobs - File Not Found?

l 27, 2020 10:28 AM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Subject: Re: History Server Not Showing Any Jobs - File Not Found? If historyserver.web.tmpdir is not set then java.io.tmpdir is used, so that should be fine. What are the contents of /local/scratch/flink_historyserver

RE: History Server Not Showing Any Jobs - File Not Found?

:///user/p2epda/lake/delp_qa/flink_hs/ historyserver.web.tmpdir: /local/scratch/flink_historyserver_tmpdir/ Did you have anything else in mind when you said pointing somewhere funny? // ah From: Chesnay Schepler Sent: Monday, April 27, 2020 5:56 AM To: Hailu, Andreas [Engineering] ; user@flin

RE: History Server Not Showing Any Jobs - File Not Found?

I'm having a further look at the code in HistoryServerStaticFileServerHandler - is there an assumption about where overview.json is supposed to be located? // ah From: Hailu, Andreas [Engineering] Sent: Wednesday, April 22, 2020 1:32 PM To: 'Chesnay Schepler' ; Hailu, Andre

RE: History Server Not Showing Any Jobs - File Not Found?

esday, April 22, 2020 2:16 AM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Subject: Re: History Server Not Showing Any Jobs - File Not Found? Which Flink version are you using? Have you checked the history server logs after enabling debug logging? On 21/04/2020 17:16, Hailu, An

History Server Not Showing Any Jobs - File Not Found?

Hi, I'm trying to set up the History Server, but none of my applications are showing up in the Web UI. Looking at the console, I see that all of the calls to /overview return the following 404 response: {"errors":["File not found."]}. I've set up my configuration as follows: JobManager Archive

RE: Flink Conf "yarn.flink-dist-jar" Question

Okay, I’ll continue to watch the JIRAs. Thanks for the update, Till. // ah From: Till Rohrmann Sent: Wednesday, April 15, 2020 10:51 AM To: Hailu, Andreas [Engineering] Cc: Yang Wang ; tison ; user@flink.apache.org Subject: Re: Flink Conf "yarn.flink-dist-jar" Question Hi Andreas,

RE: Flink Conf "yarn.flink-dist-jar" Question

Yang, Tison, Do we know when some solution for 13938 and 14964 will arrive? Do you think it will be in a 1.10.x version? // ah From: Hailu, Andreas [Engineering] Sent: Friday, March 20, 2020 9:19 AM To: 'Yang Wang' Cc: tison ; user@flink.apache.org Subject: RE: Flink Conf "yarn

RE: Flink Conf "yarn.flink-dist-jar" Question

the two tickets. Thank you. // ah From: Yang Wang Sent: Monday, March 16, 2020 9:37 PM To: Hailu, Andreas [Engineering] Cc: tison ; user@flink.apache.org Subject: Re: Flink Conf "yarn.flink-dist-jar" Question Hi Hailu, Sorry for the late response. If the Flink cluster(e.g. Yarn appli

RE: Flink Conf "yarn.flink-dist-jar" Question

: Sunday, March 8, 2020 10:52 PM To: Hailu, Andreas [Engineering] mailto:andreas.ha...@ny.email.gs.com>> Cc: tison mailto:wander4...@gmail.com>>; user@flink.apache.org<mailto:user@flink.apache.org> Subject: Re: Flink Conf "yarn.flink-dist-jar" Question Hi Hailu, tison,

RE: Flink Conf "yarn.flink-dist-jar" Question

general public cache solution is what’s being called for? // ah From: Yang Wang Sent: Sunday, March 8, 2020 10:52 PM To: Hailu, Andreas [Engineering] Cc: tison ; user@flink.apache.org Subject: Re: Flink Conf "yarn.flink-dist-jar" Question Hi Hailu, tison, I created a very similar ticket

RE: Flink Conf "yarn.flink-dist-jar" Question

Hi Tison, thanks for the reply. I’ve replied to the ticket. I’ll be watching it as well. // ah From: tison Sent: Friday, March 6, 2020 1:40 PM To: Hailu, Andreas [Engineering] Cc: user@flink.apache.org Subject: Re: Flink Conf "yarn.flink-dist-jar" Question FLINK-13938 seems a bit

Flink Conf "yarn.flink-dist-jar" Question

Hi, We noticed that every time an application runs, it uploads the flink-dist artifact to the /user//.flink HDFS directory. This causes a user disk space quota issue as we submit thousands of apps to our cluster an hour. We had a similar problem with our Spark applications where it uploaded the

RE: Table API: Joining on Tables of Complex Types

efore the ".returns()" but with this method you override the old type. Regards, Timo On 15.01.20 15:27, Hailu, Andreas wrote: > Dawid, this approach looks promising. I'm able to flatten out my Avro > records into Rows and run simple queries atop of them. I've got a > q

RE: [ANNOUNCE] Apache Flink 1.10.0 released

Congrats all! P.S. I noticed in the release notes that the bullet: [FLINK-14516] The non-credit-based network flow control code was removed, along with the configuration option taskmanager.network.credit.model. Moving forward, Flink will alway

1.9.2 Release Date?

Hi, Do we have any thoughts on a release date for 1.9.2? I've been eyeing FLINK-13184 particularly to help alleviate stress on our RM + Name Node and reduce noise/delays due to sporadic Task Manager timeouts. We submit thousands of jobs per hou

RE: Table API: Joining on Tables of Complex Types

ovided TypeInfos for the Rows? Performance is something that I'm concerned about as I've already introduced a new operation to transform our records to Rows. // ah From: Hailu, Andreas [Engineering] Sent: Wednesday, January 8, 2020 12:08 PM To: 'Dawid Wysakowicz' ; user@flink.apache.or

RE: Table API: Joining on Tables of Complex Types

Very well - I'll give this a try. Thanks, Dawid. // ah From: Dawid Wysakowicz Sent: Wednesday, January 8, 2020 7:21 AM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Cc: Richards, Adam S [Engineering] Subject: Re: Table API: Joining on Tables of Complex Types Hi An

RE: Table API: Joining on Tables of Complex Types

gt;complex object with accessors into a Table of structured types. Or by >structured types, did you mean Row? // ah From: Dawid Wysakowicz Sent: Monday, January 6, 2020 9:32 AM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Cc: Richards, Adam S [Engineering] Subject: Re: Table API: Joi

Table API: Joining on Tables of Complex Types

Hi folks, I'm trying to join two Tables which are composed of complex types, Avro's GenericRecord to be exact. I have to use a custom UDF to extract fields out of the record and I'm having some trouble on how to do joins on them as I need to call this UDF to read what I need. Example below: ba

RE: CoGroup SortMerger performance degradation from 1.6.4 - 1.9.1?

? This is why we were seeing failures in our pipelines which had operators which fed into a CoGroup? // ah From: Zhijiang Sent: Thursday, November 21, 2019 9:48 PM To: Hailu, Andreas [Engineering] ; Piotr Nowojski Cc: user@flink.apache.org Subject: Re: CoGroup SortMerger performance

RE: CoGroup SortMerger performance degradation from 1.6.4 - 1.9.1?

Thanks, Piotr. We’ll rerun our apps today with this and get back to you. // ah From: Piotr Nowojski On Behalf Of Piotr Nowojski Sent: Thursday, November 21, 2019 10:14 AM To: Hailu, Andreas [Engineering] Cc: Zhijiang ; user@flink.apache.org Subject: Re: CoGroup SortMerger performance

RE: CoGroup SortMerger performance degradation from 1.6.4 - 1.9.1?

6.65GB, so it sounds like the problem lies somewhere in the changes around mapped memory. // ah From: Zhijiang Sent: Wednesday, November 20, 2019 11:32 PM To: Hailu, Andreas [Engineering] ; user@flink.apache.org Subject: Re: CoGroup SortMerger performance degradation from 1.6.4 - 1.9.1? Hi Andreas

RE: CoGroup SortMerger performance degradation from 1.6.4 - 1.9.1?

Going through the release notes today - we tried fiddling with the taskmanager.memory.fraction option, going as low as 0.1 with unfortunately no success. It still leads to the container running beyond physical memory limits. // ah From: Hailu, Andreas [Engineering] Sent: Tuesday, November 19

CoGroup SortMerger performance degradation from 1.6.4 - 1.9.1?

Hi, We're in the middle of testing the upgrade of our data processing flows from Flink 1.6.4 to 1.9.1. We're seeing that flows which were running just fine on 1.6.4 now fail on 1.9.1 with the same application resources and input data size. It seems that there have been some changes around how t

RE: Re:RE: Re:Re: File Naming Pattern from HadoopOutputFormat

Very well - thank you both. // ah From: Haibo Sun Sent: Wednesday, July 3, 2019 9:37 PM To: Hailu, Andreas [Tech] Cc: Yitzchak Lieberman ; user@flink.apache.org Subject: Re:RE: Re:Re: File Naming Pattern from HadoopOutputFormat Hi, Andreas I'm glad you have had a solution. If y

RE: Re:Re: File Naming Pattern from HadoopOutputFormat

Haibo Sun Sent: Tuesday, July 2, 2019 5:57 AM To: Yitzchak Lieberman Cc: Hailu, Andreas [Tech] ; user@flink.apache.org Subject: Re:Re: File Naming Pattern from HadoopOutputFormat Hi, Andreas You are right. To meet this requirement, Flink should need to expose a interface to allow custo

File Naming Pattern from HadoopOutputFormat

Hello Flink team, I'm writing Avro and Parquet files to HDFS, and I've would like to include a UUID as a part of the file name. Our files in HDFS currently follow this pattern: tmp-r-1.snappy.parquet tmp-r-2.snappy.parquet ... I'm using a custom output format which extends a RichOutput