Hi Robert, Thanks for your code. It's really helpful!
However, with the readKeyedState api of state processor, we get dataset for our data instead of datastream and it seems the dataset doesn't support streamfilesink (not addSink method like datastream). If not, I need to transform the dataset to a datastream. I'm not sure it's doable based on https://www.alibabacloud.com/blog/deep-insights-into-flink-sql-flink-advanced-tutorials_596628. If it's doable, then I'll be able to solve our problem with applying streamfilesink to the transformed dataset. Best wishes, Chen-Che Huang On 2021/04/15 19:23:43, Robert Metzger <rmetz...@apache.org> wrote: > Hey Chen-Che Huang, > > I guess the StreamingFileSink is what you are looking for. It is documented > here: > https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/streamfile_sink.html > I drafted a short example (that is not production ready), which does > roughly what you are asking for: > https://gist.github.com/rmetzger/7d5dbdaa118c63f5875c8c9520cc311d > > Hope this helps! > > Best, > Robert > > > On Thu, Apr 15, 2021 at 11:33 AM Chen-Che Huang <acmic...@gmail.com> wrote: > > > Hi all, > > > > We're going to use state processor to make our keyedstate data to be > > written to different files based on the keys. More specifically, we want > > our data to be written to files key1.txt, key2.txt, ..., and keyn.txt where > > the value with the same key is stored in the same file. In each file, the > > data may be stored as follows. As far as I know, I need to implement my own > > Sink (org.apache.flink.streaming.api.functions.sink.RichSinkFunction) to > > meet the requirement. However, I wonder is there a native way to achieve > > this without implementing my own Sink because using official solution is > > usually more efficient and reliable than doing it by myself. Many thanks > > for any comment. > > > > key1.txt > > key1 value11 > > key1 value21 > > key1 value31 > > > > key2.txt > > key2 value21 > > key2 value22 > > key2 value23 > > > > Best wishes, > > Chen-Che Huang > > >
