Change User Logging

2021-06-29 Thread Joshua Conlin
Hello, I'd like to make some changes in my fork to user logging. I see that the {user}-{host}.out share a log4j config with the main logs, but I believe I need to make some lower level (appender?) changes, specific to only to those user logs. Could someone give me a pointer in the source to wher

Monitoring a Notebook in Spark UI

2020-07-21 Thread Joshua Conlin
Hello, I'm looking for documentation to better understand pyspark/scala notebook execution in Spark. I typically see application runtimes that can be very long, is there always a spark "application" running for a notebook or zeppelin session? Those that are not actually being run in zeppelin typ

[Spark SQL] AnalysisException: cannot resolve '`column_name`' given input columns

2020-04-20 Thread Joshua Conlin
I have the following Spark SQL query: SELECT column_name, * from table_name; I have multiple spark clusters, this query has been running daily on all of the clusters. After a recent zeppelin redeployment, it fails on just one of the clusters with the following exception: AnalysisException: cann