This is an automated email from the ASF dual-hosted git repository. stigahuang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/impala.git
commit e0a0bb3617936596485e2ae677138bbdbab9192d Author: Surya Hebbar <[email protected]> AuthorDate: Wed Jul 17 20:15:25 2024 +0530 IMPALA-13233: Improve display of instance-level skew in query timeline Representing phases of multiple instances in a single bar with dividers makes it challenging to distinguish between the perimeters of each instance's phases, hindering other instances that have already started with the subsequent phase. Hence, the correlation between the recorded profile metrics and the user's desired representation is less. To improve this representation, the phases of each fragment's instances are now being aggregated and displayed as a histogram on the timeline, making it easier to understand the distribution of start time and end time of different instance's phases. When there are more than 5 instances for a phase, respective timestamps are bucketed into 5 divisions, each spanning 20% of the difference between maximum and minimum timestamp value. Each division's timestamps are aggregated and then maximum value is plotted on the query timeline. The height of each division's phase rectangle is proportional to the number of instances in that division. A division without any instances is not represented. If the number of instances are less than or equal to 5, for each event, instance timestamps are sorted in ascending order before plotting. This results in a descending staircase form of aggregated metrics display, where maximum timestamps within each of the 20% spanning divisions is shown. The precision of SVG elements' position and dimensions have been limited and are set by the user. They will have the same precision as that of the timeticks. The order of rendering phases and iterating through colors for each plan node has been reversed, in order to support this representation. A tooltip containing the following additional details is displayed, when hovering on a plane node's bucketed phase rectangle. - With respect to the hovered on bucketed phase rectangle - No. of instances - Event's Maximum Timestamp (in seconds) - Event's Minimum Timestamp (in seconds) - Event's Average Timestamp (in seconds) Outlines are drawn at the top and bottom of each plan node, according to the timestamp of the instances' final closing phase. The attribute 'stroke-dasharray' has been used to add separators between the perimeters of each instance's phases, instead of an additional SVG line element each time. Redundant calls to 'appendChild' methods and setting fragment's ID after each iteration have been removed. With this approach, the number of SVG elements in the fragment diagram is reduced substantially, resulting in very fast rendering times. Similar to "HASH_JOIN_NODE", the rendering of 2nd and 3rd events ("Waiting for initial build" and "Initial build available") for the node type "NESTED_LOOP_JOIN_NODE" have also been skipped. Replaced equality operators with strict equality operators. Manually tested with various profiles, sizes ranging from 4MB to 430MB. Change-Id: Ied8a5966e9e4111bf7aa25aee11d23881daad7d2 Reviewed-on: http://gerrit.cloudera.org:8080/21593 Reviewed-by: Impala Public Jenkins <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- www/queries.tmpl | 2 +- www/query_backends.tmpl | 2 +- www/query_detail_tabs.tmpl | 2 +- www/query_finstances.tmpl | 2 +- www/query_plan.tmpl | 2 +- www/query_summary.tmpl | 2 +- www/query_timeline.tmpl | 8 +- www/rpcz.tmpl | 2 +- www/scripts/query_timeline/chart_commons.js | 10 +- www/scripts/query_timeline/fragment_diagram.js | 441 +++++++++++++-------- .../query_timeline/fragment_metrics_diagram.js | 21 +- www/scripts/query_timeline/global_dom.js | 2 +- .../query_timeline/host_utilization_diagram.js | 14 +- .../tests/query_timeline/fragment_diagram.test.js | 17 +- 14 files changed, 322 insertions(+), 205 deletions(-) diff --git a/www/queries.tmpl b/www/queries.tmpl index d5085bb6a..124b5fefc 100644 --- a/www/queries.tmpl +++ b/www/queries.tmpl @@ -420,7 +420,7 @@ command line parameter.</p> }; window.onload = () => { - if (localStorage.getItem("imported") == "T") { + if (localStorage.getItem("imported") === "T") { imported_queries_header.scrollIntoView({ behavior : "instant", block : "start" }); localStorage.setItem("imported", "F"); } diff --git a/www/query_backends.tmpl b/www/query_backends.tmpl index 0d8bb07ba..3c8136701 100644 --- a/www/query_backends.tmpl +++ b/www/query_backends.tmpl @@ -77,7 +77,7 @@ $(document).ready(function() { }); function toggleRefresh() { - if (document.getElementById("toggle").checked == true) { + if (document.getElementById("toggle").checked === true) { intervalId = setInterval(refresh, 1000); document.getElementById("refresh_on").textContent = "Auto-refresh on"; } else { diff --git a/www/query_detail_tabs.tmpl b/www/query_detail_tabs.tmpl index 70e8d1cbe..5f00d85c9 100644 --- a/www/query_detail_tabs.tmpl +++ b/www/query_detail_tabs.tmpl @@ -103,7 +103,7 @@ under the License. </ul> <script> -if (typeof index == "undefined") { +if (typeof index === "undefined") { var index = {}; // For inflight (executing or waiting) query diff --git a/www/query_finstances.tmpl b/www/query_finstances.tmpl index 53969a134..75c5464e3 100644 --- a/www/query_finstances.tmpl +++ b/www/query_finstances.tmpl @@ -104,7 +104,7 @@ $(document).ready(function() { }); function toggleRefresh() { - if (document.getElementById("toggle").checked == true) { + if (document.getElementById("toggle").checked === true) { intervalId = setInterval(refresh, 1000); document.getElementById("refresh_on").textContent = "Auto-refresh on"; } else { diff --git a/www/query_plan.tmpl b/www/query_plan.tmpl index 519da0cf6..ef82aa851 100644 --- a/www/query_plan.tmpl +++ b/www/query_plan.tmpl @@ -277,7 +277,7 @@ function refresh() { // Attaches a blob of the current SVG viewport to the associated link export_link.addEventListener('click', function(event) { - if (export_format.value == ".html") { + if (export_format.value === ".html") { var svg_viewport = document.querySelector("svg"); var export_style = document.getElementById("css"); var html_blob = new Blob([`<!DOCTYPE html><body>`, diff --git a/www/query_summary.tmpl b/www/query_summary.tmpl index 9258f1da4..74e60cf27 100644 --- a/www/query_summary.tmpl +++ b/www/query_summary.tmpl @@ -67,7 +67,7 @@ document.getElementById("last-updated").textContent = new Date(); var intervalId = setInterval(refresh, 1000); function toggleRefresh() { - if (document.getElementById("toggle").checked == true) { + if (document.getElementById("toggle").checked === true) { intervalId = setInterval(refresh, 1000); document.getElementById("refresh_on").textContent = "Auto-refresh on"; } else { diff --git a/www/query_timeline.tmpl b/www/query_timeline.tmpl index 216715b7b..7413a6d2b 100644 --- a/www/query_timeline.tmpl +++ b/www/query_timeline.tmpl @@ -187,7 +187,7 @@ function refreshView() { collectFragmentEventsFromProfile(); collectUtilizationFromProfile(); collectFragmentMetricsFromProfile(); - if (last_maxts != maxts) { + if (last_maxts !== maxts) { renderTimingDiagram(); last_maxts = maxts; } @@ -227,11 +227,11 @@ if (window.location.search.includes("imported")) { window.onload = function refreshProfile() { var req = new XMLHttpRequest(); req.onload = function() { - if (req.status == 200) { + if (req.status === 200) { set_profile(JSON.parse(req.responseText)["profile_json"]); refreshView(); if (profile.child_profiles[0].info_strings.find(({key}) => - key === "End Time").value == "") { + key === "End Time").value === "") { setTimeout(refreshProfile, 1000); } } @@ -286,7 +286,7 @@ hor_zoomout.addEventListener('click', function(event) { // Attaches a SVG blob of the complete timeline to the associated link export_link.addEventListener('click', function (event) { - if (export_format.value == ".html") { + if (export_format.value === ".html") { var export_style = document.getElementById("page_export-css"); // Deep clone 'parentNode's as wrappers to SVG components diff --git a/www/rpcz.tmpl b/www/rpcz.tmpl index 82869a9cf..40eced089 100644 --- a/www/rpcz.tmpl +++ b/www/rpcz.tmpl @@ -413,7 +413,7 @@ document.getElementById("last-updated").textContent = new Date(); var intervalId = setInterval(refresh, 1000); function toggleRefresh() { - if (document.getElementById("toggle").checked == true) { + if (document.getElementById("toggle").checked === true) { intervalId = setInterval(refresh, 1000); document.getElementById("refresh_on").textContent = "Auto-refresh on"; } else { diff --git a/www/scripts/query_timeline/chart_commons.js b/www/scripts/query_timeline/chart_commons.js index 5b0151fd6..760dd7718 100644 --- a/www/scripts/query_timeline/chart_commons.js +++ b/www/scripts/query_timeline/chart_commons.js @@ -59,7 +59,7 @@ export function generateTimesamples(timesamples_array, max_samples, extend) { set_maxts(timesamples_array[max_samples.available + 1] * 1e9); } var j = max_samples.available + (extend ? 3 : 2); - for (; j < timesamples_array.length && timesamples_array[j] != null; ++j) { + for (; j < timesamples_array.length && timesamples_array[j] !== null; ++j) { timesamples_array[j] = null; } } @@ -68,7 +68,7 @@ export function mapTimeseriesCounters(time_series_counters, counters) { for (var i = 0; i < counters.length; i++) { var no_change = true; for (var j = 0; j < time_series_counters.length; j++) { - if (time_series_counters[j].counter_name == counters[i][0]) { + if (time_series_counters[j].counter_name === counters[i][0]) { counters[i][2] = j; no_change = false; } @@ -98,13 +98,13 @@ export function aggregateProfileTimeseries(parent_profile, aggregate_array, } export function showTooltip(chart, x) { - if (chart == undefined) return; + if (chart === null) return; chart.tooltip.show({x : chart.internal.findClosestFromTargetsByX( chart.internal.getTargets(), x).x}); } export function hideTooltip(chart) { - if (chart == undefined) return; + if (chart === null) return; chart.tooltip.hide(); } @@ -131,6 +131,6 @@ export function destroyChart(chart, chart_dom_obj) { return null; } -if (typeof process != "undefined" && process.env.NODE_ENV === 'test') { +if (typeof process !== "undefined" && process.env.NODE_ENV === 'test') { exportedForTest = {accumulateTimeseriesValues}; } diff --git a/www/scripts/query_timeline/fragment_diagram.js b/www/scripts/query_timeline/fragment_diagram.js index feec242a3..dd0920ea7 100644 --- a/www/scripts/query_timeline/fragment_diagram.js +++ b/www/scripts/query_timeline/fragment_diagram.js @@ -37,7 +37,9 @@ var rownum; var row_height = 15; var integer_part_estimate = 4; var char_width = 6; +var bucket_size = 5; var fragment_events_parse_successful = false; +var decimals_multiplier = Math.pow(10, decimals); // #phases_header var phases = [ @@ -75,32 +77,41 @@ function removeChildIfExists(parentElement, childElement) { } } -function getSvgRect(fill_color, x, y, width, height, dash, stroke_color) { +function rtd(num) { // round to decimals + return Math.round(num * decimals_multiplier) / decimals_multiplier; +} + +// 'dasharray' and 'stroke_color' are optional parameters +function getSvgRect(fill_color, x, y, width, height, dasharray, stroke_color) { var rect = document.createElementNS("http://www.w3.org/2000/svg";, "rect"); - rect.setAttribute("x", `${x}px`); - rect.setAttribute("y", `${y}px`); - rect.setAttribute("width", `${width}px`); - rect.setAttribute("height", `${height}px`); + rect.setAttribute("x", rtd(x)); + rect.setAttribute("y", rtd(y)); + rect.setAttribute("width", rtd(width)); + rect.setAttribute("height", rtd(height)); rect.setAttribute("fill", fill_color); - rect.setAttribute("stroke", stroke_color); - if (dash) { - rect.setAttribute("stroke-dasharray", "2 2"); + rect.setAttribute("stroke-width", 0.5); + if (stroke_color) { + rect.setAttribute("stroke", stroke_color); + } + if (dasharray) { + rect.setAttribute("stroke-dasharray", dasharray); } return rect; } -function getSvgText(text, fill_color, x, y, height, container_center, max_width = 0) { +// 'max_width' is an optional parameters +function getSvgText(text, fill_color, x, y, height, container_center, max_width) { var text_el = document.createElementNS("http://www.w3.org/2000/svg";, "text"); text_el.appendChild(document.createTextNode(text)); - text_el.setAttribute("x", `${x}px`); - text_el.setAttribute("y", `${y}px`); + text_el.setAttribute("x", x); + text_el.setAttribute("y", y); text_el.style.fontSize = `${height / 1.5}px`; if (container_center) { text_el.setAttribute("dominant-baseline", "middle"); text_el.setAttribute("text-anchor", "middle"); } text_el.setAttribute("fill", fill_color); - if (max_width != 0) { + if (max_width) { text_el.setAttribute("textLength", max_width); text_el.setAttribute("lengthAdjust", "spacingAndGlyphs"); } @@ -109,10 +120,10 @@ function getSvgText(text, fill_color, x, y, height, container_center, max_width function getSvgLine(stroke_color, x1, y1, x2, y2, dash) { var line = document.createElementNS("http://www.w3.org/2000/svg";, "line"); - line.setAttribute("x1", `${x1}px`); - line.setAttribute("y1", `${y1}px`); - line.setAttribute("x2", `${x2}px`); - line.setAttribute("y2", `${y2}px`); + line.setAttribute("x1", rtd(x1)); + line.setAttribute("y1", rtd(y1)); + line.setAttribute("x2", rtd(x2)); + line.setAttribute("y2", rtd(y2)); line.setAttribute("stroke", stroke_color); if (dash) { line.setAttribute("stroke-dasharray", "2 2"); @@ -131,38 +142,153 @@ function getSvgGroup() { return group; } +function markMinMaxAvg(event) { + if (event.no_bar !== undefined) return; + var min = event.ts_list[0]; + var max = min; + var i = 1; + for (; i < event.ts_list.length; i++) { + if (min > event.ts_list[i]) min = event.ts_list[i]; + if (max < event.ts_list[i]) max = event.ts_list[i]; + } + var event_span_t = max - min; + event.parts = Array(bucket_size).fill(null).map( + () => ({count : 0, min : Infinity, max : -Infinity, avg : 0})); + var k; + var ts; + var part; + for (i = 0; i < event.ts_list.length; i++) { + ts = event.ts_list[i]; + k = ((ts - min) * bucket_size / event_span_t) | 0; + if (k >= bucket_size) k = bucket_size - 1; + part = event.parts[k]; + if (ts < part.min) part.min = ts; + if (ts > part.max) part.max = ts; + part.avg += ts; + part.count += 1; + } + for (k = 0; k < bucket_size; k++) { + event.parts[k].avg /= event.parts[k].count; + } +} + +function getPhaseDasharray(dx, dy, l_dx) { + return `0 ${rtd(l_dx)} ${rtd(dx - l_dx + dy)} ${rtd(dx)} ${rtd(dy)} 0`; +} + +function attachBucketedPhaseTooltip(rect, part) { + rect.appendChild(getSvgTitle(`# of Inst. : ${part.count}\n` + + `Min. : ${rtd(part.min / 1e9)}s\n` + + `Max. : ${rtd(part.max / 1e9)}s\n` + + `Avg. : ${rtd(part.avg / 1e9)}s`)); + return rect; +} + function DrawBars(svg, rownum, row_height, events, xoffset, px_per_ns) { - var color_idx = 0; - var last_end = xoffset; + // Structure of each 'events' element - + // { + // no_bar : < Boolean value is set, if event is not to be rendered > + // ts_list : < List of this plan node's event's timestamps from all instances > + // // A total of 5 buckets represented as 'parts' + // parts : [{ // Each object contains statistics of a single timestamps bucket + // count : < Number of timestamps in the current bucket > + // avg : < Average of timestamps in the current bucket > + // min : < Minimum timestamp in the current bucket > + // max : < Maximum timestamp in the current bucket > + // }, < ... 4 other objects of the same format > ] + // } var bar_height = row_height - 2; + var last_e_index = events.length - 1; + var last_ts_index = events[last_e_index].ts_list.length - 1; var plan_node = getSvgGroup(); plan_node.classList.add("plan_node"); - events.forEach(function(ev) { - if (ev.no_bar == undefined) { - var x = last_end; - var y = rownum * row_height; - - var endts = Math.max.apply(null, ev.tslist); - var width = xoffset + endts * px_per_ns - last_end; - last_end = x + width; - - // Block phase outline - plan_node.appendChild(getSvgRect(phases[color_idx].color, x, y, width, bar_height, - false, stroke_fill_colors.black)); - color_idx++; - - // Grey dividers for other instances that finished earlier - ev.tslist.forEach(function(ts) { - var dx = (endts - ts) * px_per_ns; - var ignore_px = 2; // Don't print tiny skews - if (Math.abs(dx) > ignore_px) { - plan_node.appendChild(getSvgLine(stroke_fill_colors.dark_grey, last_end - dx, - y, last_end - dx, y + bar_height, false)); + // coordinates start with (0,0) in the top-left corner + var y = rownum * row_height; // the y-position for the current phase rectangle + var cp_y = y; // copy of y, used to bring y-position back to the top(initial value) + var dx, dy; // dimensions of the current phase rectangle + var bucketed = false; + + if (events[0].ts_list.length > bucket_size) { + events.map(markMinMaxAvg); + bucketed = true; + } else { + events.map((ev) => { + ev.ts_list.sort((a, b) => a - b); + }); + } + + var i; + var color_idx = last_e_index; + for (i = 0; i <= last_e_index; ++i) { + if (events[i].no_bar) --color_idx; + } + + var j; + var dasharray; // current stroke-dasharray + for (var i = last_e_index; i >= 0; --i) { + y = cp_y; // reuse calculation, returing to the initial y-position + if (events[i].no_bar === undefined) { + var l_dx = 0; // previous phase rectangle's width (x-dimension) + if (bucketed) { + // Represent the aggregate distribution of instances for each phase + var part; + for (j = 0; j < events[i].parts.length; j++) { + part = events[i].parts[j]; // mantain a reference for reuse + if (part.count > 0) { + dy = (part.count / events[i].ts_list.length) * bar_height; + } else { + // skip to the next iteration, if the current bucket has no timestamps + continue; + } + dx = part.max * px_per_ns; + // 'stroke-dasharray' is a string of alternative lengths for + // dashes and no dashes. So, the length of dasharray of must be even. + // Width(x-dimension) of previous(upper) phase rectangle is stored to + // calculate boundaries such that only open boundaries are stroked with borders + // using the 'getPhaseDasharray' function + dasharray = getPhaseDasharray(dx, dy, l_dx); + plan_node.appendChild(attachBucketedPhaseTooltip(getSvgRect( + phases[color_idx].color, xoffset, y, dx, dy, dasharray, + stroke_fill_colors.black), part)); + y += dy; + l_dx = dx; } - }); - svg.appendChild(plan_node); + } else { + // Represent phases of each instance seperately, according to the timestamps + dy = bar_height / events[i].ts_list.length; + for (j = 0; j < events[i].ts_list.length; j++) { + dx = events[i].ts_list[j] * px_per_ns; + // Calculations for the 'stroke-dasharray' have been done same as above + dasharray = getPhaseDasharray(dx, dy, l_dx); + plan_node.appendChild(getSvgRect(phases[color_idx].color, xoffset, y, dx, dy, + dasharray, stroke_fill_colors.black)); + y += dy; + l_dx = dx; + } + } + --color_idx; } - }); + } + + y = rownum * row_height; + if (events[last_e_index].no_bar === undefined) { + // Plan node's top and bottom outlines + var top_edge_ts, bottom_edge_ts; + if (bucketed) { + top_edge_ts = events[last_e_index].parts[0].max; + bottom_edge_ts = events[last_e_index].parts[bucket_size - 1].max; + } else { + top_edge_ts = events[last_e_index].ts_list[0]; + bottom_edge_ts = events[last_e_index].ts_list[last_ts_index]; + } + plan_node.appendChild(getSvgLine(stroke_fill_colors.black, xoffset, y, + xoffset + top_edge_ts * px_per_ns, y, false)); + plan_node.appendChild(getSvgLine(stroke_fill_colors.black, xoffset, y + bar_height, + xoffset + bottom_edge_ts * px_per_ns, y + bar_height, + false)); + } + + svg.appendChild(plan_node); } async function renderPhases() { @@ -174,9 +300,9 @@ async function renderPhases() { x = Math.min(x, diagram_width - width); phases_header.appendChild(getSvgRect(stroke_fill_colors.black, x, 0, width, - row_height, false)); + row_height)); phases_header.appendChild(getSvgRect(phases[color_idx++].color, x + 1, 1, - width - 2, row_height - 2, false)); + width - 2, row_height - 2)); phases_header.appendChild(getSvgText(p.label, stroke_fill_colors.black, x + width / 2, (row_height + 4) / 2 , row_height, true, Math.min(p.label.length * char_width, width / 1.5))); @@ -187,7 +313,6 @@ async function renderFragmentDiagram() { clearDOMChildren(fragment_diagram); var px_per_ns = chart_width / maxts; var rownum_l = 0; - var max_indent = 0; var pending_children = 0; var pending_senders = 0; var text_y = row_height - 4; @@ -211,27 +336,19 @@ async function renderFragmentDiagram() { for (var i = 0; i < fragment.nodes.length; ++i) { var node = fragment.nodes[i]; - if (node.events != undefined) { + if (node.events !== undefined) { // Plan node timing row DrawBars(fragment_svg_group, rownum_l, row_height, node.events, name_width, px_per_ns); - fragment_svg_group.id = fragment.name; - fragment_svg_group.addEventListener('click', updateFragmentMetricsChartOnClick); - fragment_diagram.appendChild(fragment_svg_group); - - if (node.type == "HASH_JOIN_NODE") { + if (node.type === "HASH_JOIN_NODE") { fragment_diagram.appendChild(getSvgText("X", stroke_fill_colors.black, - name_width + Math.min.apply(null, node.events[2].tslist) * px_per_ns, + name_width + Math.min.apply(null, node.events[2].ts_list) * px_per_ns, text_y, row_height, false)); fragment_diagram.appendChild(getSvgText("O", stroke_fill_colors.black, - name_width + Math.min.apply(null, node.events[2].tslist) * px_per_ns, + name_width + Math.min.apply(null, node.events[2].ts_list) * px_per_ns, text_y, row_height, false)); } - } else { - fragment_svg_group.id = fragment.name; - fragment_svg_group.addEventListener('click', updateFragmentMetricsChartOnClick); - fragment_diagram.appendChild(fragment_svg_group); } if (node.is_receiver) { @@ -249,21 +366,21 @@ async function renderFragmentDiagram() { fragment_diagram.appendChild(getSvgText(node.name, fragment.color, label_x, text_y, row_height, false)); - if (node.parent_node != undefined) { + if (node.parent_node !== undefined) { var y = row_height * node.parent_node.rendering.rownum; if (node.is_sender) { - var x = name_width + Math.min.apply(null, fevents[3].tslist) * px_per_ns; + var x = name_width + Math.min.apply(null, fevents[3].ts_list) * px_per_ns; // Dotted horizontal connector to received rows fragment_diagram.appendChild(getSvgLine(fragment.color, name_width, y + row_height / 2 - 1, x, y + row_height / 2 - 1, true)); // Dotted rectangle for received rows - var x2 = name_width + Math.max.apply(null, fevents[4].tslist) * px_per_ns; + var x2 = name_width + Math.max.apply(null, fevents[4].ts_list) * px_per_ns; fragment_diagram.appendChild(getSvgRect(stroke_fill_colors.transperent, - x, y + 4, x2 - x, row_height - 10, true, fragment.color)); + x, y + 4, x2 - x, row_height - 10, "2 2", fragment.color)); } - if (node.is_sender && node.parent_node.rendering.rownum != rownum_l - 1) { + if (node.is_sender && node.parent_node.rendering.rownum !== rownum_l - 1) { // DAG edge on right side to distant sender var x = name_width - (pending_senders) * char_width - char_width / 2; fragment_diagram.appendChild(getSvgLine(fragment.color, @@ -297,10 +414,11 @@ async function renderFragmentDiagram() { } } } - + fragment_svg_group.id = fragment.name; + fragment_svg_group.addEventListener('click', updateFragmentMetricsChartOnClick); + fragment_diagram.appendChild(fragment_svg_group); // Visit sender fragments in reverse order to avoid dag edges crossing pending_fragments.reverse().forEach(printFragment); - } }); fragments.forEach(function(fragment) { @@ -318,9 +436,9 @@ async function renderTimeticks() { var timetick_label; for (var i = 1; i <= ntics; ++i) { timeticks_footer.appendChild(getSvgRect(stroke_fill_colors.black, x, y, px_per_tic, - row_height, false)); + row_height)); timeticks_footer.appendChild(getSvgRect(stroke_fill_colors.light_grey, x + 1, - y + 1, px_per_tic - 2, row_height - 2, false)); + y + 1, px_per_tic - 2, row_height - 2)); timetick_label = (i * sec_per_tic).toFixed(decimals); timeticks_footer.appendChild(getSvgText(timetick_label, stroke_fill_colors.black, x + px_per_tic - timetick_label.length * char_width + 2, text_y, row_height, @@ -341,11 +459,11 @@ export function collectFragmentEventsFromProfile() { var execution_profile = profile.child_profiles[2]; var execution_profile_name = execution_profile.profile_name.split(" ").slice(0,2) .join(" "); - console.assert(execution_profile_name == "Execution Profile"); + console.assert(execution_profile_name === "Execution Profile"); execution_profile.child_profiles.forEach(function(fp) { - if (fp.child_profiles != undefined && - fp.child_profiles[0].event_sequences != undefined) { + if (fp.child_profiles !== undefined && + fp.child_profiles[0].event_sequences !== undefined) { var cp = fp.child_profiles[0]; var fevents = fp.child_profiles[0].event_sequences[0].events; @@ -355,15 +473,15 @@ export function collectFragmentEventsFromProfile() { fevents[en].no_bar = true; continue; } - fevents[en].tslist = [ fevents[en].timestamp ]; + fevents[en].ts_list = [ fevents[en].timestamp ]; } for (var instance = 1; instance < fp.child_profiles.length; ++instance) { - if (fp.child_profiles[instance].event_sequences != undefined) { + if (fp.child_profiles[instance].event_sequences !== undefined) { if (fp.child_profiles[instance].event_sequences[0].events.length == fevents.length) { for (var en = 0; en < fevents.length; ++en) { if (fevents[en].no_bar) continue; - fevents[en].tslist.push( + fevents[en].ts_list.push( fp.child_profiles[instance].event_sequences[0].events[en].timestamp); } } else { @@ -371,11 +489,11 @@ export function collectFragmentEventsFromProfile() { while(i < fevents.length && en < fp.child_profiles[instance] .event_sequences[0].events.length) { if (fevents[i].no_bar) { - if (fevents[i].label == fp.child_profiles[instance] + if (fevents[i].label === fp.child_profiles[instance] .event_sequences[0].events[en].label) { en++; } } else if (fp.child_profiles[instance].event_sequences[0] - .events[en].label == fevents[i].label) { - fevents[i].tslist.push(fp.child_profiles[instance].event_sequences[0] + .events[en].label === fevents[i].label) { + fevents[i].ts_list.push(fp.child_profiles[instance].event_sequences[0] .events[en].timestamp); ++en; } @@ -399,92 +517,96 @@ export function collectFragmentEventsFromProfile() { var node_stack = []; var node_name; cp.child_profiles.forEach(function get_plan_nodes(pp, index) { - if (pp.node_metadata != undefined) { - node_path.push(index); - var name_flds = pp.profile_name.split(/[()]/); - var node_type = name_flds[0].trim(); - var node_id = name_flds.length > 1 ? name_flds[1].split(/[=]/)[1] : 0; - node_name = pp.profile_name.replace("_NODE", "").replace("_", " ") - .replace("KrpcDataStreamSender", "SENDER") - .replace("Hash Join Builder", "JOIN BUILD") - .replace("join node_", ""); - if (node_type.indexOf("SCAN_NODE") >= 0) { - var table_name = pp.info_strings.find(({ key }) => key === "Table Name") - .value.split(/[.]/); - node_name = node_name.replace("SCAN", - `SCAN [${table_name[table_name.length - 1]}]`); - } + if (pp.node_metadata === undefined) return; + node_path.push(index); + // pp.profile_name : "AGGREGATION_NODE (id=52)" + var name_flds = pp.profile_name.split(/[()]/); + // name_flds : ["AGGREGATION_NODE ", "id=52", ""] + var node_type = name_flds[0].trim(); + // node_type: "AGGREGATION_NODE" + var node_id = name_flds.length > 1 ? + parseInt(name_flds[1].split(/[=]/)[1]) : 0; + // node_id: 52 + node_name = pp.profile_name.replace("_NODE", "").replace("_", " ") + .replace("KrpcDataStreamSender", "SENDER") + .replace("Hash Join Builder", "JOIN BUILD") + .replace("join node_", ""); + if (node_type.indexOf("SCAN_NODE") >= 0) { + var table_name = pp.info_strings.find(({ key }) => key === "Table Name") + .value.split(/[.]/); + node_name = node_name.replace("SCAN", + `SCAN [${table_name[table_name.length - 1]}]`); + } - var is_receiver = node_type == "EXCHANGE_NODE" || - (node_type == "HASH_JOIN_NODE" && pp.num_children < 3); - - var is_sender = (node_type == "Hash Join Builder" || - node_type == "KrpcDataStreamSender"); - var parent_node; - if (node_type == "PLAN_ROOT_SINK") { - parent_node = undefined; - } else if (pp.node_metadata.data_sink_id != undefined) { - parent_node = receiver_nodes[node_id]; // Exchange sender dst - } else if (pp.node_metadata.join_build_id != undefined) { - parent_node = receiver_nodes[node_id]; // Join sender dst - } else if (node_stack.length > 0) { - parent_node = node_stack[node_stack.length - 1]; - } else if (all_nodes.length) { - parent_node = all_nodes[all_nodes.length - 1]; - } + var is_receiver = node_type === "EXCHANGE_NODE" || + (node_type === "HASH_JOIN_NODE" && pp.num_children < 3); + + var is_sender = (node_type === "Hash Join Builder" || + node_type === "KrpcDataStreamSender"); + var parent_node; + if (node_type === "PLAN_ROOT_SINK") {} + else if (pp.node_metadata.data_sink_id !== undefined) { + parent_node = receiver_nodes[node_id]; // Exchange sender dst + } else if (pp.node_metadata.join_build_id !== undefined) { + parent_node = receiver_nodes[node_id]; // Join sender dst + } else if (node_stack.length > 0) { + parent_node = node_stack[node_stack.length - 1]; + } else if (all_nodes.length) { + parent_node = all_nodes[all_nodes.length - 1]; + } - max_namelen = Math.max(max_namelen, node_name.length + node_stack.length + 1); + max_namelen = Math.max(max_namelen, node_name.length + node_stack.length + 1); - if (pp.event_sequences != undefined) { - var node_events = pp.event_sequences[0].events; + if (pp.event_sequences !== undefined) { + var node_events = pp.event_sequences[0].events; - // Start the instance event list for each event with timestamps - // from this instance - for (var en = 0; en < node_events.length; ++en) { - node_events[en].tslist = [ node_events[en].timestamp ]; - if (node_type == "HASH_JOIN_NODE" && (en == 1 || en == 2)) { - node_events[en].no_bar = true; - } + // Start the instance event list for each event with timestamps + // from this instance + for (var en = 0; en < node_events.length; ++en) { + node_events[en].ts_list = [ node_events[en].timestamp ]; + if ((node_type === "HASH_JOIN_NODE" || + node_type === "NESTED_LOOP_JOIN_NODE") && (en === 1 || en === 2)) { + node_events[en].no_bar = true; } } - var node = { - name: node_name, - type: node_type, - node_id: node_id, - num_children: 0, - child_index: 0, - metadata: pp.node_metadata, - parent_node: parent_node, - events: node_events, - path: node_path.slice(0), - is_receiver: is_receiver, - is_sender: is_sender - } + } + var node = { + name: node_name, + type: node_type, + node_id: node_id, + num_children: 0, + child_index: 0, + metadata: pp.node_metadata, + parent_node: parent_node, + events: node_events, + path: node_path.slice(0), + is_receiver: is_receiver, + is_sender: is_sender + } - if (is_sender) { - node.parent_node.sender_frag_index = fragments.length; - } + if (is_sender) { + node.parent_node.sender_frag_index = fragments.length; + } - if (is_receiver) { - receiver_nodes[node_id] = node; - } + if (is_receiver) { + receiver_nodes[node_id] = node; + } - if (parent_node != undefined) { - node.child_index = parent_node.num_children++; - } + if (parent_node !== undefined) { + node.child_index = parent_node.num_children++; + } - all_nodes.push(node); + all_nodes.push(node); - fragment.nodes.push(node); + fragment.nodes.push(node); - if (pp.child_profiles != undefined) { - node_stack.push(node); - pp.child_profiles.forEach(get_plan_nodes); - node = node_stack.pop(); - } - rownum++; - node_path.pop(); + if (pp.child_profiles !== undefined) { + node_stack.push(node); + pp.child_profiles.forEach(get_plan_nodes); + node = node_stack.pop(); } + rownum++; + node_path.pop(); }); // For each node, retrieve the instance timestamps for the remaining instances @@ -497,22 +619,21 @@ export function collectFragmentEventsFromProfile() { for (var pi = 0; pi < node.path.length; ++pi) { cp = cp.child_profiles[node.path[pi]]; } - console.assert(cp.node_metadata.data_sink_id == undefined || + console.assert(cp.node_metadata.data_sink_id === undefined || cp.profile_name.indexOf(`(dst_id=${node.node_id})`)); - console.assert(cp.node_metadata.plan_node_id == undefined || - cp.node_metadata.plan_node_id == node.node_id); - + console.assert(cp.node_metadata.plan_node_id === undefined || + cp.node_metadata.plan_node_id === node.node_id); // Add instance events to this node - if (cp.event_sequences != undefined) { - if (node.events.length == cp.event_sequences[0].events.length) { + if (cp.event_sequences !== undefined) { + if (node.events.length === cp.event_sequences[0].events.length) { for (var en = 0; en < node.events.length; ++en) { - node.events[en].tslist.push(cp.event_sequences[0].events[en].timestamp); + node.events[en].ts_list.push(cp.event_sequences[0].events[en].timestamp); } } else { var i = 0, en = 0; while(i < node.events.length && en < cp.event_sequences[0].events.length) { - if (node.events[i].label == cp.event_sequences[0].events[en].label) { - node.events[i].tslist.push(cp.event_sequences[0].events[en].timestamp); + if (node.events[i].label === cp.event_sequences[0].events[en].label) { + node.events[i].ts_list.push(cp.event_sequences[0].events[en].timestamp); ++en; ++i; } else { ++i; @@ -581,7 +702,7 @@ fragment_diagram.addEventListener('mousemove', function(e) { if (e.clientX + scrollable_screen.scrollLeft >= name_width && e.clientX + scrollable_screen.scrollLeft <= name_width + chart_width){ removeChildIfExists(fragment_diagram, timestamp_gridline); timestamp_gridline = getSvgLine(stroke_fill_colors.black, e.clientX + scrollable_screen.scrollLeft, 0, e.clientX + scrollable_screen.scrollLeft, - parseInt(fragment_diagram.style.height)); + parseInt(fragment_diagram.style.height), false); fragment_diagram.appendChild(timestamp_gridline); var gridline_time = ((maxts * (e.clientX + scrollable_screen.scrollLeft - name_width) / chart_width) / 1e9); showTooltip(host_utilization_chart, gridline_time); @@ -627,11 +748,13 @@ timeticks_footer.addEventListener('wheel', function(e) { if (e.wheelDelta <= 0) { if (decimals <= 1) return; set_decimals(decimals - 1); + decimals_multiplier /= 10; } else { var rendering_constraint = char_width * ((decimals + 1) + integer_part_estimate) >= chart_width / ntics; if (rendering_constraint) return; set_decimals(decimals + 1); + decimals_multiplier *= 10; } } else { if (e.wheelDelta <= 0 && ntics <= 10) return; @@ -648,6 +771,6 @@ timeticks_footer.addEventListener('wheel', function(e) { plan_order.addEventListener('click', renderFragmentDiagram); -if (typeof process != "undefined" && process.env.NODE_ENV === 'test') { +if (typeof process !== "undefined" && process.env.NODE_ENV === 'test') { exportedForTest = {getSvgRect, getSvgLine, getSvgText, getSvgTitle, getSvgGroup}; } diff --git a/www/scripts/query_timeline/fragment_metrics_diagram.js b/www/scripts/query_timeline/fragment_metrics_diagram.js index b191dfbc7..f329acaf3 100644 --- a/www/scripts/query_timeline/fragment_metrics_diagram.js +++ b/www/scripts/query_timeline/fragment_metrics_diagram.js @@ -27,10 +27,10 @@ import "./global_dom.js"; export var exportedForTest; -export var fragment_id_selections = new Map(); export var fragment_metrics_parse_successful = false; export var fragment_metrics_chart = null; +var fragment_id_selections = new Map(); var fragment_metrics_aggregate; var fragment_metrics_counters = [ ["MemoryUsage", "memory usage", 0], @@ -84,7 +84,7 @@ function initializeFragmentMetricsChart() { y2 : { tick : { - format : function (y2) { return (y2 == Math.floor(y2) ? y2 : ""); } + format : function (y2) { return (y2 === Math.floor(y2) ? y2 : ""); } }, show : true } @@ -115,7 +115,7 @@ function initializeFragmentMetricsChart() { } function dragResizeBar(mousemove_e) { - if (mousemove_e.target.classList[0] == "c3-event-rect") return; + if (mousemove_e.target.classList[0] === "c3-event-rect") return; var next_height = getFragmentMetricsHeight() + (fragment_metrics_resize_bar.offsetTop - mousemove_e.clientY); if (next_height >= diagram_min_height && @@ -163,7 +163,7 @@ export function toogleFragmentMetricsVisibility() { export function collectFragmentMetricsFromProfile() { // do not collect metrics, in case a fragment is not selected - if (fragment_id_selections.size == 0) { + if (fragment_id_selections.size === 0) { fragment_metrics_parse_successful = false; return; } @@ -179,7 +179,7 @@ export function collectFragmentMetricsFromProfile() { // initialize the collection arrays for subsequent collections // arrays are overwritten without further re-allocation to reduce memory usage if(fragment_id_selections.has(fragment_profile.profile_name)) { - if (fragment_metrics_chart == null) { + if (fragment_metrics_chart === null) { ({fragment_metrics_aggregate, sampled_fragment_metrics_timeseries} = initializeFragmentMetrics(fragment_profile, fragment_metrics_counters, max_samples_fragment_metrics, fragment_metrics_timeaxis_name)); @@ -237,7 +237,7 @@ export function collectFragmentMetricsFromProfile() { } export async function resizeFragmentMetricsChart() { - if (fragment_metrics_chart == null) return; + if (fragment_metrics_chart === null) return; var chart_width = diagram_width - margin_chart_end - name_width; fragment_metrics_resize_bar.style.marginLeft = `${name_width + chart_width / 4}px`; fragment_metrics_resize_bar.style.width = `${chart_width / 2}px`; @@ -249,9 +249,10 @@ export async function resizeFragmentMetricsChart() { } export function updateFragmentMetricsChartOnClick(click_event) { + if (click_event.target.tagName !== "rect") return; var selected_fragment_id = click_event.target.parentElement.parentElement.id; fragment_id_selections.delete(selected_fragment_id); - if (fragment_metrics_chart != null) { + if (fragment_metrics_chart !== null) { var unloaded = fragment_metrics_chart.internal.getTargets().every(function (target) { if (target.id.includes(selected_fragment_id)) { var unload_ids = new Array(fragment_metrics_counters.length); @@ -264,7 +265,7 @@ export function updateFragmentMetricsChartOnClick(click_event) { } return true; }); - if (fragment_id_selections.size == 0) { + if (fragment_id_selections.size === 0) { fragment_metrics_chart = destroyChart(fragment_metrics_chart, fragment_metrics_diagram); toogleFragmentMetricsVisibility(); @@ -273,7 +274,7 @@ export function updateFragmentMetricsChartOnClick(click_event) { } if (!unloaded) return; } - if(fragment_id_selections.size == 0) { + if(fragment_id_selections.size === 0) { resetUtilizationHeight(); } fragment_id_selections.set(selected_fragment_id, 1); @@ -306,6 +307,6 @@ fragment_metrics_close_btn.addEventListener('click', closeFragmentMetricsChart); fragment_metrics_close_btn.style.height = `${diagram_controls_height}px`; fragment_metrics_close_btn.style.fontSize = `${diagram_controls_height / 2}px`; -if (typeof process != "undefined" && process.env.NODE_ENV === 'test') { +if (typeof process !== "undefined" && process.env.NODE_ENV === 'test') { exportedForTest = {initializeFragmentMetrics}; } diff --git a/www/scripts/query_timeline/global_dom.js b/www/scripts/query_timeline/global_dom.js index 10e57f734..2f3b065c4 100644 --- a/www/scripts/query_timeline/global_dom.js +++ b/www/scripts/query_timeline/global_dom.js @@ -15,7 +15,7 @@ // specific language governing permissions and limitations // under the License. -if (typeof process != "undefined" && process.env.NODE_ENV === 'test' && +if (typeof process !== "undefined" && process.env.NODE_ENV === 'test' && document.body.innerHTML === "") { const fs = await import('fs'); document.body.innerHTML = fs.readFileSync(`${process.env.IMPALA_HOME}/www/query_timeline.tmpl`, "utf-8") diff --git a/www/scripts/query_timeline/host_utilization_diagram.js b/www/scripts/query_timeline/host_utilization_diagram.js index 2463aee76..1db8a247c 100644 --- a/www/scripts/query_timeline/host_utilization_diagram.js +++ b/www/scripts/query_timeline/host_utilization_diagram.js @@ -141,7 +141,7 @@ function initializeUtilizationChart() { } function dragResizeBar(mousemove_e) { - if (mousemove_e.target.classList[0] == "c3-event-rect") return; + if (mousemove_e.target.classList[0] === "c3-event-rect") return; var next_height = getUtilizationHeight() + (host_utilization_resize_bar.offsetTop - mousemove_e.clientY); if (next_height >= diagram_min_height && window.innerHeight - next_height @@ -154,7 +154,7 @@ function dragResizeBar(mousemove_e) { function initializeUtilizationMetrics(parent_profile, counters_y1, counters_y2, max_samples, timeaxis_name) { - console.assert(parent_profile.profile_name == "Per Node Profiles"); + console.assert(parent_profile.profile_name === "Per Node Profiles"); // user, sys, io and sampled timeticks var cpu_nodes_usage_aggregate = new Array(counters_y1.length); max_samples.available = 0; @@ -204,7 +204,7 @@ export function toogleUtilizationVisibility() { } export async function resizeUtilizationChart() { - if (host_utilization_chart == null) return; + if (host_utilization_chart === null) return; var chart_width = diagram_width - margin_chart_end - name_width; host_utilization_resize_bar.style.marginLeft = `${name_width + chart_width / 4}px`; host_utilization_resize_bar.style.width = `${chart_width / 2}px`; @@ -228,8 +228,8 @@ export function collectUtilizationFromProfile() { // initialize the collection arrays for subsequent collections // arrays are overwritten without further re-allocation to reduce memory usage var per_node_profiles = profile.child_profiles[2].child_profiles[0]; - console.assert(per_node_profiles.profile_name == "Per Node Profiles"); - if (host_utilization_chart == null) { + console.assert(per_node_profiles.profile_name === "Per Node Profiles"); + if (host_utilization_chart === null) { ({cpu_nodes_usage_aggregate, read_write_metrics_aggregate, sampled_utilization_timeseries} = initializeUtilizationMetrics( per_node_profiles, cpu_utilization_counters, read_write_metrics_counters, @@ -237,7 +237,7 @@ export function collectUtilizationFromProfile() { initializeUtilizationChart(); } var impala_server_profile = profile.child_profiles[1]; - console.assert(impala_server_profile.profile_name == "ImpalaServer"); + console.assert(impala_server_profile.profile_name === "ImpalaServer"); // Update the plot, only when number of samples in SummaryStatsCounter is updated if (impala_server_profile.summary_stats_counters[0].num_of_samples == prev_utilization_num_samples) { @@ -329,6 +329,6 @@ host_utilization_close_btn.addEventListener('click', function(e) { host_utilization_close_btn.style.height = `${diagram_controls_height}px`; host_utilization_close_btn.style.fontSize = `${diagram_controls_height / 2}px`; -if (typeof process != "undefined" && process.env.NODE_ENV === 'test') { +if (typeof process !== "undefined" && process.env.NODE_ENV === 'test') { exportedForTest = {initializeUtilizationMetrics}; } diff --git a/www/scripts/tests/query_timeline/fragment_diagram.test.js b/www/scripts/tests/query_timeline/fragment_diagram.test.js index 708b761a5..e54a8bc66 100644 --- a/www/scripts/tests/query_timeline/fragment_diagram.test.js +++ b/www/scripts/tests/query_timeline/fragment_diagram.test.js @@ -25,17 +25,18 @@ describe("Test getSvg*", () => { light_grey : "#F0F0F0", transperent : "rgba(0, 0, 0, 0)" }; test("Test getSvgRect", () => { - expect(getSvgRect(stroke_fill_colors.transperent, 0, 0, 100, 100, true, + expect(getSvgRect(stroke_fill_colors.transperent, 0, 0, 100, 100, "2 2", stroke_fill_colors.black).outerHTML).toBe( - '<rect x="0px" y="0px" width="100px" height="100px"' + '<rect x="0" y="0" width="100" height="100"' + ` fill="${stroke_fill_colors.transperent}"` + + ` stroke-width="0.5"` + ` stroke="${stroke_fill_colors.black}"` + ` stroke-dasharray="2 2"></rect>`); }); test("Test getSvgLine", () => { expect(getSvgLine(stroke_fill_colors.black, 0, 0, 100, 100, true).outerHTML).toBe( - '<line x1="0px" y1="0px" x2="100px" y2="100px"' + '<line x1="0" y1="0" x2="100" y2="100"' + ` stroke="${stroke_fill_colors.black}"` + ' stroke-dasharray="2 2"></line>'); }); @@ -43,15 +44,7 @@ describe("Test getSvg*", () => { test("Test getSvgText", () => { expect(getSvgText("Text", stroke_fill_colors.black, 0, 0, 15, true, 300) .outerHTML).toBe( - '<text x="0px" y="0px" style="font-size: 10px;" dominant-baseline="middle" ' - + `text-anchor="middle" fill="${stroke_fill_colors.black}" textLength="300" ` - + 'lengthAdjust="spacingAndGlyphs">Text</text>'); - }); - - test("Test getSvgText", () => { - expect(getSvgText("Text", stroke_fill_colors.black, 0, 0, 15, true, 300) - .outerHTML).toBe( - '<text x="0px" y="0px" style="font-size: 10px;" dominant-baseline="middle" ' + '<text x="0" y="0" style="font-size: 10px;" dominant-baseline="middle" ' + `text-anchor="middle" fill="${stroke_fill_colors.black}" textLength="300" ` + 'lengthAdjust="spacingAndGlyphs">Text</text>'); });
