[Connect] Install additional python packages after session creation

2025-01-24 Thread Deependra Patel
Hi all, There are ways through the `addArtifacts` API in an existing session but for that we need to have dependencies properly gzipped. In the case of different kernel/OS between client and server, it won't work either I believe. What I am interested in is doing some sort of `pip install https://y

Support for `sparkContext.broadcast` function in Spark Connect?

2024-10-14 Thread Deependra Patel
Hi, I see that Spark Context methods are not supported in Spark Connect. There are many common use cases eg. broadcast machine learning model weights to all executors, so no need to fetch individually. This makes migration of workloads to Spark Connect tougher. I know there are plans to add more &