Peter Andrew created SPARK-58837:
------------------------------------
Summary: Fine-grained Spark Connect Python loggers
Key: SPARK-58837
URL: https://issues.apache.org/jira/browse/SPARK-58837
Project: Spark
Issue Type: Improvement
Components: Connect, PySpark
Affects Versions: 4.2.0
Reporter: Peter Andrew
The Python Connect client logs everything using the
`pyspark.sql.connect.logging` logger, which is gated on the
`SPARK_CONNECT_LOG_LEVEL` environment variable.
This makes it difficult to control logging in a fine-grained fashion.
For example, when trying to connect to a Connect server that does not respond,
the client will silently retry and backoff, which can be confusing to users.
Enabling logging of gRPC UNAVAILABLE retries would be useful, but is only
possible by setting `SPARK_CONNECT_LOG_LEVEL=DEBUG`, which then produces way
too much log volume.
Instead, Spark could use more fine-grained loggers, so we can log such retries
without logging all the rest of the debugging information.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
