westhide commented on PR #1216: URL: https://github.com/apache/datafusion-ballista/pull/1216#issuecomment-2764432837
> Thanks @westhide this PR makes sense, > > It also brings configurable validation option, which I did not really think about. I have few questions, for which I do not have definitive answer, so I'd like to hear your opinion: > > 1. In which cases do we want to enable validation? > 2. If we want to have option to enable/disable it, should that be decision of who ever operates scheduler/executor rather than connected client? Q1: As the `BallistaFlightService` keep listenning on each Executor, it allow client to send a `do_get` request, and without check `FetchPartition` action's `path` is created by shuffle writer, so the client can try to read any file on the executor. In this scene, we can enable validation. Q2: I'm not sure. As currently we just read ipc file created by `ShuffleWriterExec`, it's safe to skip all validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
