Github user brennonyork commented on the pull request:
https://github.com/apache/spark/pull/3559#issuecomment-66694879
@andrewor14 definitely understand that reasoning. I guess my only question
would be how would we a) answer the bug then and still b) support the
`--properties-file` option? Even if we remove the `--environment-file` CLI
option I'm still confused / not seeing a solution to setting the
`SPARK_CONF_DIR`. If we allowed `SPARK_CONF_DIR` to be set from the CLI then we
would still run into a merge problem with the the
`SPARK_CONF_DIR/spark-defaults.conf` and whatever was set from the
`--properties-file`. I agree though that, regardless, we should minimize
merging of configuration files as that would likely cause more confusion than
help for the end user.
What about this solution... We completely remove the `--properties-file`
CLI option in place of the `--environment-file` option? I thinking that, at the
end of the day, ensuring proper environment settings (i.e. `SPARK_CONF_DIR`,
etc.) are more important than a single properties file, especially when the
`SPARK_CONF_DIR` would point to the configuration file. I'm just throwing that
out, but I'm wondering if any other solution would work without necessitating
file merging. That being said, doesn't Spark already merge defaults into the
given `--properties-file` if it was provided? Now I'm wondering if that
actually **would** be more confusing if we did the same thing with the spark
environment file.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
