gaogaotiantian commented on PR #54635: URL: https://github.com/apache/spark/pull/54635#issuecomment-4010432824
Okay we need some further discussion on this matter. The problem is streaming. streaming has slightly different logic. Who do we need to confirm about what the expected behavior is for streaming data source? In an ideal world, streaming data source should have the exact same interface as non-streaming one. So `partition` method should be able to return `[None]` and `read` should take it. However, we should confirm that this is expected first. Seems like @siying wrote the test part? Maybe @HeartSaVioR can also shed some light to the matter? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
