Re: Proposal to add Google Cloud Storage FileSystem with RecoverableWriter

2021-02-04 Thread Galen Warren
Hi Till -- this is in response to your message before about my proposal to add GCS FileSystem/RecoverableWriter support. I was only subscribed to the dev digest before, and so I didn't actually get an email I could reply to, so sorry for the one-off email. I'm properly subscribed to the dev list no

Re: Proposal to add Google Cloud Storage FileSystem with RecoverableWriter

2021-02-03 Thread Till Rohrmann
Hmm, maybe it was deleted again. I received an email notification yesterday. On Wed, Feb 3, 2021 at 11:57 AM Xintong Song wrote: > @Till, > Did I overlook anything? I don't find Galen's post on the old PR. > > Thank you~ > > Xintong Song > > > > On Wed, Feb 3, 2021 at 6:10 PM Till Rohrmann wrot

Re: Proposal to add Google Cloud Storage FileSystem with RecoverableWriter

2021-02-03 Thread Xintong Song
@Till, Did I overlook anything? I don't find Galen's post on the old PR. Thank you~ Xintong Song On Wed, Feb 3, 2021 at 6:10 PM Till Rohrmann wrote: > @Galen, I've just seen that you posted your ideas on the old Github PR. I > think it would be better to post it on the JIRA ticket [1]. > > [

Re: Proposal to add Google Cloud Storage FileSystem with RecoverableWriter

2021-02-03 Thread Till Rohrmann
@Galen, I've just seen that you posted your ideas on the old Github PR. I think it would be better to post it on the JIRA ticket [1]. [1] https://issues.apache.org/jira/browse/FLINK-11838 Cheers, Till On Tue, Feb 2, 2021 at 12:02 PM Xintong Song wrote: > Hi Galen, > > Thanks for offering the c

Re: Proposal to add Google Cloud Storage FileSystem with RecoverableWriter

2021-02-02 Thread Xintong Song
Hi Galen, Thanks for offering the contribution. As Till has already suggested, please comment on FLINK-11838 your solution proposal. Once we reach consensus on the proposal, I'll assign you to the ticket. Thank you~ Xintong Song On Tue, Feb 2, 2021 at 5:19 PM Till Rohrmann wrote: > Hi Gale

Re: Proposal to add Google Cloud Storage FileSystem with RecoverableWriter

2021-02-02 Thread Till Rohrmann
Hi Galen, I think that adding support for GCS using the StreamingFileSink sounds like a very good idea to me. Looking at FLINK-11838 I believe that this effort has been abandoned. I think that you could take this ticket over if you want. Maybe you could update this ticket with your solution propos

Proposal to add Google Cloud Storage FileSystem with RecoverableWriter

2021-01-30 Thread Galen Warren
Hi -- I'm wondering if you would be interested in a contribution to add a HadoopFileSystem implementation, with associated RecoverableWriter, for Google Cloud Storage. This would be similar to what's already in place for S3, and it would allow writing to GCS using a StreamingFileSink. The implement