Hi Rahul, Thanks for explaining. I see. Now there is no way to dynamic control file name in StreamingFileSink.
If the number of organizations is not so huge. Like Sivaprasanna said, you can use "BucketAssigner" to create bucket by your organization ID. The bucket in StreamingFileSink is like Hive/Spark's partition, the information is in directory name. Each organization creates a new directory. Best, Jingsong Lee On Tue, May 19, 2020 at 2:03 AM dhurandar S <dhurandarg...@gmail.com> wrote: > Hi Jingsong, > > We have a system where organizations keep getting added and removed on a > regular basis, As the new organizations get added the data from these > organization starts flowing into the streaming system, we do group by on > Organisation ID which is part of the incoming event, If in the incoming > stream we find any new Organisation Ids that we have not seen before then > we create a new file and start writing data into it. But this is dynamic as > in based on the incoming stream. > > regards, > Rahul > > On Wed, May 13, 2020 at 11:43 PM Jingsong Li <jingsongl...@gmail.com> > wrote: > >> Hi, Dhurandar, >> >> Can you describe your needs? Why do you need to modify file names >> flexibly? What kind of name do you want? >> >> Best, >> Jingsong Lee >> >> On Thu, May 14, 2020 at 2:05 AM dhurandar S <dhurandarg...@gmail.com> >> wrote: >> >>> Yes we looked at it , >>> The problem is the file name gets generated in a dynamic fashion, based >>> on which organization data we are getting we generate the file name from >>> the coming data. >>> >>> Is there any way we can achieve this ?? >>> >>> On Tue, May 12, 2020 at 8:38 PM Yun Gao <yungao...@aliyun.com> wrote: >>> >>>> Hi Dhurandar: >>>> >>>> Currently StreamingFileSink should be able to change the prefix and >>>> suffix of the filename[1], it could be changed to something like >>>> <prefix>-0-0<suffix>. Could this solve your problem ? >>>> >>>> >>>> Best, >>>> Yun >>>> >>>> >>>> >>>> >>>> [1] >>>> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/streamfile_sink.html#part-file-configuration >>>> >>>> >>>> >>>> ------------------------------------------------------------------ >>>> 发件人：dhurandar S<dhurandarg...@gmail.com> >>>> 日 期：2020年05月13日 05:13:04 >>>> 收件人：user<user@flink.apache.org>; <d...@flink.apache.org> >>>> 主 题：changing the output files names in Streamfilesink from part-00 to >>>> something else >>>> >>>> We want to change the name of the file being generated as the output of >>>> our StreamFileSink. >>>> , when files are generated they are named part-00*, is there a way that >>>> we can change the name. >>>> >>>> In Hadoop, we can change RecordWriters and MultipleOutputs. May I >>>> please some help in this regard. This is causing blockers for us and will >>>> force us t move to MR job >>>> >>>> -- >>>> Thank you and regards, >>>> Dhurandar >>>> >>>> >>>> >>> >>> -- >>> Thank you and regards, >>> Dhurandar >>> >>> >> >> -- >> Best, Jingsong Lee >> > > > -- > Thank you and regards, > Dhurandar > > -- Best, Jingsong Lee
