I am afraid there is no such functionality available yet. I think though it is a valid request. I think we can use the upcoming FLIP-107 metadata columns for this purpose and expose the file name as metadata column of a filesystem source.
Would you like to create a JIRA issue for it? Best, Dawid [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-107%3A+Handling+of+metadata+in+SQL+connectors On 30/10/2020 13:21, Ruben Laguna wrote: > I've asked this already on [stackoverflow][1] > > Is there anything equivalent to Spark's `f.input_file_name()` ? I > don't see anything that could be used in [system functions][2] > > I have a dataset where they embedded some information in the filenames > (200k files) and I need to extract that as a new column. > > In Spark I could ` > .withColumn("id",f.split(f.reverse(f.split(f.input_file_name(),'/'))[0],'\.')[0])` > but I don't see how can I do the same with Flink. Is it possible? > > I don't see [any JIRA issue about it either][3]. Is it something that > has already been discussed? > > > [1]: > https://stackoverflow.com/questions/64607839/is-there-an-equivalent-to-sparks-f-input-file-name-function-in-apache-flink > [2]: > https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/functions/systemFunctions.html > [3]: > https://issues.apache.org/jira/browse/FLINK-8275?jql=project%20%3D%20FLINK%20AND%20text%20~%20%22filename%22
signature.asc
Description: OpenPGP digital signature
