[
https://issues.apache.org/jira/browse/SPARK-59613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samra Pusina updated SPARK-59613:
---------------------------------
Attachment: (was: 404exception.tiff)
> [SHS] /static/sql handler not registered on all pods → intermittent 404s on
> SparkSQL ExecutionPage
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-59613
> URL: https://issues.apache.org/jira/browse/SPARK-59613
> Project: Spark
> Issue Type: Bug
> Components: Bug, Spark Core
> Affects Versions: 4.0.4
> Reporter: Samra Pusina
> Priority: Major
>
> h2. Summary
> The SparkSQL tab on the {{ExecutionPage}} in the Spark History Server (SHS)
> intermittently fails to render its execution graphs. {{GET /static/sql}}
> returns {*}404 intermittently{*}, while core static assets (e.g.
> {{{}/static/jquery-3.5.1.min.js{}}}) consistently return 200. The failure
> only appears when *more than one SHS pod* is running.
> h2. Problem
> * The SparkSQL tab on {{ExecutionPage}} intermittently renders no execution
> graphs.
> * Browser network logs show {{GET /static/sql}} returning *404*
> intermittently.
> * Other static assets served by the core UI (e.g.
> {{{}/static/jquery-3.5.1.min.js{}}}) consistently return {*}200{*}.
> * Manually loading the SQL tab once in the browser appeared to "fix" it —
> but only temporarily/unreliably.
> h2. Root Cause
> {{SQLTab}} registers the {{/static/sql}} Jetty static handler in its
> constructor:
> {code:java}
> parent.addStaticHandler(SQLTab.STATIC_RESOURCE_DIR, "/static/sql") {code}
> {{SQLTab}} is only instantiated {*}per-application, lazily{*}, via
> {{{}SQLHistoryServerPlugin.setupUI{}}}, when an app with SQL/DataFrame
> executions is loaded into the SHS application cache. It is *not* part of the
> core UI wired up at SHS startup (unlike the main {{/static}} handler
> registered by {{{}HistoryServer{}}}).
> Because {{SparkUI}} instances are cached per-application and evicted
> ({{{}ApplicationCache{}}}), the {{/static/sql}} handler's lifetime is tied to
> whichever {{SparkUI}} registered it. When that app is evicted from the cache,
> the handler is torn down with it, so {{/static/sql/*}} requests start 404ing
> — until another SQL-bearing app is loaded and re-registers it. This produces
> the intermittent, cache-eviction-dependent 404 pattern, and matches why
> manually loading the SQL tab once "fixed" it (it triggered
> {{{}SparkUI{}}}/{{{}SQLTab{}}} creation and handler registration for that
> session, working until the next eviction).
> h2. Why this only manifests with more than one SHS pod
> The {{/static/sql}} handler lives in the *in-memory Jetty state of a single
> pod* — the pod whose {{ApplicationCache}} happened to load a SQL-bearing app
> and instantiate {{{}SQLTab{}}}. None of this state is shared across pods:
> * Each SHS pod has its {*}own independent {{ApplicationCache}}{*}, its own
> loaded {{SparkUI}} instances, and therefore its own set of
> dynamically-registered handlers ({{{}/static/sql{}}} among them).
> * The core {{/static}} handler is wired up at {{HistoryServer}} startup, so
> it exists identically on *every* pod — which is why
> {{/static/jquery-3.5.1.min.js}} and the other core assets always return 200
> regardless of routing.
> * {{/static/sql}} only exists on a pod that _currently_ has a SQL app loaded
> and un-evicted.
> With {*}multiple pods behind the Service/load balancer{*}, one browser page
> load fans out into many requests distributed across pods with {*}divergent
> handler state{*}:
> # The {{ExecutionPage}} HTML is served by {*}pod A{*}, which has the app
> loaded and {{/static/sql}} registered.
> # The follow-up {{GET /static/sql/...}} for the SparkSQL graph assets is
> routed to {*}pod B{*}, which never instantiated {{SQLTab}} (or has since
> evicted that {{{}SparkUI{}}}). Pod B has no {{/static/sql}} handler →
> {*}404{*}.
> This is what makes the pattern look random: it depends on which pod each
> individual asset request lands on, not just on eviction timing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
