[
https://issues.apache.org/jira/browse/SPARK-20435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15980056#comment-15980056
]
Marcelo Vanzin commented on SPARK-20435:
----------------------------------------
bq. Providing passwords that way is supported by Spark
That's not an argument. You can type secrets in any command line and that
doesn't make it OK.
bq. and ps ax works for users with appropriate privileges
"ps ax" works for all users. Try it for yourself.
{noformat}
vanzin@vanzin-t460p:/work/apache/spark-prs$ sudo sleep 100000
{noformat}
{noformat}
vanzin@vanzin-t460p:/tmp$ ps axu | grep sleep
root 32583 0.0 0.0 73264 4524 pts/5 S+ 10:55 0:00 sudo sleep
100000
root 32584 0.0 0.0 7296 760 pts/5 S+ 10:55 0:00 sleep 100000
vanzin 32586 0.0 0.0 14232 1072 pts/2 S+ 10:56 0:00 grep
--color=auto sleep
{noformat}
I'm not saying redacting from logs is useless, but I'm saying that a user that
is providing secrets in the command line is giving up any security, and
redaction won't save him.
> More thorough redaction of sensitive information from logs/UI, more unit tests
> ------------------------------------------------------------------------------
>
> Key: SPARK-20435
> URL: https://issues.apache.org/jira/browse/SPARK-20435
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.0
> Reporter: Mark Grover
>
> SPARK-18535 and SPARK-19720 were works to redact sensitive information (e.g.
> hadoop credential provider password, AWS access/secret keys) from event logs
> + YARN logs + UI and from the console output, respectively.
> While some unit tests were added along with these changes - they asserted
> when a sensitive key was found, that redaction took place for that key. They
> didn't assert globally that when running a full-fledged Spark app (whether or
> YARN or locally), that sensitive information was not present in any of the
> logs or UI. Such a test would also prevent regressions from happening in the
> future if someone unknowingly adds extra logging that publishes out sensitive
> information to disk or UI.
> Consequently, it was found that in some Java configurations, sensitive
> information was still being leaked in the event logs under the
> {{SparkListenerEnvironmentUpdate}} event, like so:
> {code}
> "sun.java.command":"org.apache.spark.deploy.SparkSubmit ... --conf
> spark.executorEnv.HADOOP_CREDSTORE_PASSWORD=secret_password ...
> {code}
> "secret_password" should have been redacted.
> Moreover, previously redaction logic was only checking if the key matched the
> secret regex pattern, it'd redact it's value. That worked for most cases.
> However, in the above case, the key (sun.java.command) doesn't tell much, so
> the value needs to be searched. So the check needs to be expanded to match
> against values as well.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
