andygrove opened a new issue, #2687: URL: https://github.com/apache/datafusion-ballista/issues/2687
PR #2619 adds Python UDF support to the Python client. Schedulers and executors started from Python (`BallistaScheduler` / `BallistaExecutor`) now decode inline Python UDF payloads with datafusion-python's codec, which calls `cloudpickle.loads` on them. That means anyone who can reach the scheduler port can run arbitrary Python on the cluster. This is the same trust model as PySpark, but it isn't written down anywhere yet. We should add a short section to `python/README.md` (and the Python user guide) that covers: - the trust model: only expose a cluster that accepts Python UDFs to trusted clients - Python UDFs only work on clusters started from Python. The stock Rust scheduler and executor binaries and the Docker images can't decode them Follow-up from https://github.com/apache/datafusion-ballista/pull/2619#discussion_r4406551332 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
