Re: [Structured streaming, V2] commit on ContinuousReader

2018-05-03 Thread Joseph Torres
In the master branch, we currently call this method in ContinuousExecution.commit(). Note that the ContinuousReader API is experimental and undergoing active design work. We will definitely include some kind of functionality to back-commit data once it's been processed, but the handle we eventuall

[Structured streaming, V2] commit on ContinuousReader

2018-05-03 Thread Jiří Syrový
Version: 2.3, DataSourceV2, ContinuousReader Hi, We're creating a new data source to fetch data from streaming source that requires commiting received data and we would like to commit data once in a while after it has been retrieved and correctly processed and then fetch more. One option could b