Thanks for this discussion everyone. It has been very useful in getting an
overall understanding here.
I think in general, consensus is that this change doesn't introduce
behavioral changes, and it's definitely an advantage to reuse the
constructs that Spark provides to us.

Moving on to a different question here - of pushing this through to Spark.
The init containers have been tested over the past two Spark releases by
external users and integration testing - and this would be a fundamental
change to that behavior.
We should work on getting enough test coverage and confidence here.

We can start by getting a PR going perhaps, and start augmenting the
integration testing to ensure that there are no surprises - with/without
credentials, accessing GCS, S3 etc as well.
When we get enough confidence and test coverage, let's merge this in.
Does that sound like a reasonable path forward?



On Wed, Jan 10, 2018 at 2:53 PM, Marcelo Vanzin <van...@cloudera.com> wrote:

> On Wed, Jan 10, 2018 at 2:51 PM, Matt Cheah <mch...@palantir.com> wrote:
> > those sidecars may perform side effects that are undesirable if the main
> Spark application failed because dependencies weren’t available
>
> If the contract is that the Spark driver pod does not have an init
> container, and the driver handles its own dependencies, then by
> definition that situation cannot exist.
>
> --
> Marcelo
>



-- 
Anirudh Ramanathan

Reply via email to