This is a problem because (other than the fact that Parquet uses
java.util.logging) of a bug in Spark in the current master.
ParquetRelation.scala attempts to override the parquet logger but, at least
currently (and if your application simply reads a parquet file before it
does anything else with Parquet), the parquet.Log class hasn't been loaded
yet. Therefore the code in ParquetRelation.enableLogForwarding has no
affect. If you look at the code in parquet.Log there's a static initializer
that needs to be called prior to enableLogForwarding or whatever
enableLogForwarding does gets undone by this static initializer.
Adding: Class.forName("parquet.Log") as the first thing in my main fixed the
problem.
The "fix" would be to force the static initializer to get called in
parquet.Log as part of enableForwardLogging.
Anyone want a PR?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/How-do-I-turn-off-Parquet-logging-in-a-worker-tp18955p18958.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
