That's aligned with what I was thinking. I would caution against in premature-abstraction of course. Since this is all iterable locally, it's easy enough to first target Dot with the initial abstraction, get that working then try to get a 2nd working, allowing the abstraction to evolve to support both, and not be overly coupled to either.
Robert Burke On Tue, Feb 24, 2026 at 10:28 AM Yousuf Ansari <[email protected]> wrote: > Subject: Re: [GSoC 2026] Go SDK Project Idea – Pipeline Visualization in > Prism Runner > > Hi Robert, > > Thank you for the clarification, that perspective helps a lot. > > I agree that the real value is introducing a clean abstraction between the > portable pipeline proto and the rendering layer, rather than coupling > directly to DOT. > > As a next step, I’m planning to explore defining an intermediate graph > representation derived from the portable proto, so that DOT becomes one > pluggable renderer, with room for future formats and Prism integration. > > Does that direction sound reasonable from a package/layout perspective > within the Go SDK? > > Thanks again for the guidance. > > Best, > Yousuf > > On Tue, 24 Feb 2026 at 23:32, Robert Burke <[email protected]> wrote: > >> As a rule, I'm in favour of this project, but outside of code reviews I'm >> not sure how much time I can spend on mentoring beyond that. (But I promise >> I'll look at all PRs tagged with @lostluck.) >> >> While the proprosal around "dot" is largely just a continuation of the >> pre-existing "dot" runner, the important thing is that it gives a basic >> scaffolding of going from something that currently works (Go pipeline to >> some other format for rendering), not the specific outcome (dot graphs). >> It's however a useful starting point for later incorporating the same code >> into Prism. >> >> The ideal case is using the dot runner as the initial proof of concept >> for converting it to use the portable pipeline proto format instead of the >> existing Raw Go SDK structure. Then somehow from there abstract what it >> produces from dot graphs, to some other format (mermaid flow charts, some >> fun HTML thing directly that can be made responsive in the UI with live >> metrics and the like, etc.). >> >> Incorporating it into the prism runner enables Python and Java (or other >> SDKs) to take advantage of it as well, which is where the value of this >> proposal is. >> >> Robert >> >> On Mon, Feb 23, 2026 at 12:18 AM Yousuf Ansari <[email protected]> >> wrote: >> >>> Hi Beam community, >>> >>> My name is Yousuf. I'm a student who has been contributing to Apache >>> projects for a while now - I have 4 merged PRs in Apache Superset and >>> recently opened PR #37673 in Beam, which rewrites the Go SDK dot runner to >>> generate DOT graphs from the portable pipeline proto. >>> >>> The PR notes that the new portable-proto-based approach enables future >>> reuse in Prism and other portable tooling. I'd like to propose a Go SDK >>> GSoC project built around that direction: >>> >>> - Integrating the portable-proto DOT generation into the Prism >>> runner's web UI, so developers can visualize their pipeline while it runs >>> locally >>> - Adding per-transform metrics display on the graph nodes >>> - Potentially surfacing watermark progress per PCollection for >>> streaming pipelines >>> >>> This feels like a natural 350-hour project that directly extends the >>> work already started in #37673 and aligns with the Go SDK roadmap's focus >>> on streaming features and portable runner improvements. >>> >>> I'd love to know: >>> >>> 1. Is there mentor interest in a Go SDK project for GSoC 2026? >>> 2. Does this direction align with the priorities you have in mind? >>> 3. Are there related issues I should pick up to strengthen my >>> proposal before the application window opens? >>> >>> Happy to discuss scope and adjust based on your feedback. >>> >>> Thanks, Yousuf >>> >>
