Do we want to have multiple portable dot runners? Or would it make sense to converge on one?
On Tue, Feb 24, 2026 at 2:47 PM Yousuf Ansari <[email protected]> wrote: > Hi Robert, > > That's a very helpful context, especially the point about choosing when to > collapse composites versus expanding their internals. > > I agree that this is really the core usability question, and that > interactivity is a practical way to avoid hardcoding a single expansion > strategy. I’ll think more carefully about how the Go implementation can > support selective expansion rather than enforcing a fixed view. > > Appreciate the insight. > > On Wed, 25 Feb 2026 at 01:02, Robert Bradshaw via dev <[email protected]> > wrote: > >> I wrote a pipeline proto -> dot graph loop at least a dozen times >> before I finally checked in that dot "runner." The key interesting bit >> is figuring out at what point to render a composite as a single node >> vs. render its internals to be the most useful to the user. >> Interactivity is a good cop-out for that generically unanswerable >> question. >> >> On Tue, Feb 24, 2026 at 11:02 AM Yousuf Ansari >> <[email protected]> wrote: >> > >> > Thank you both for the helpful feedback. >> > >> > Robert - that makes sense about avoiding premature abstraction. I’ll >> focus on getting a clean portable-proto-based DOT implementation working >> first and let the abstraction evolve naturally as additional renderers are >> introduced. >> > >> > Joey - thanks for pointing out the Python implementation. I’ll review >> that to better understand how the proto traversal is structured there and >> see what patterns can inform the Go SDK approach. >> > >> > Appreciate the guidance. >> > >> > Best, >> > Yousuf >> > >> > On Wed, 25 Feb 2026 at 00:11, Joey Tran <[email protected]> >> wrote: >> >> >> >> FWIW, there's already a python-implemented "dot runner" >> >> >> >> >> https://github.com/apache/beam/blob/eac11fed1d45bdfd50c9499b8c4a75f36dfbdfe7/sdks/python/apache_beam/runners/render.py >> >> >> >> It can generate dot graphs from a portable pipeline protos already. >> >> >> >> On Tue, Feb 24, 2026 at 1:36 PM Robert Burke <[email protected]> >> wrote: >> >>> >> >>> That's aligned with what I was thinking. >> >>> >> >>> I would caution against in premature-abstraction of course. Since >> this is all iterable locally, it's easy enough to first target Dot with the >> initial abstraction, get that working then try to get a 2nd working, >> allowing the abstraction to evolve to support both, and not be overly >> coupled to either. >> >>> >> >>> Robert Burke >> >>> >> >>> On Tue, Feb 24, 2026 at 10:28 AM Yousuf Ansari < >> [email protected]> wrote: >> >>>> >> >>>> Subject: Re: [GSoC 2026] Go SDK Project Idea – Pipeline >> Visualization in Prism Runner >> >>>> >> >>>> Hi Robert, >> >>>> >> >>>> Thank you for the clarification, that perspective helps a lot. >> >>>> >> >>>> I agree that the real value is introducing a clean abstraction >> between the portable pipeline proto and the rendering layer, rather than >> coupling directly to DOT. >> >>>> >> >>>> As a next step, I’m planning to explore defining an intermediate >> graph representation derived from the portable proto, so that DOT becomes >> one pluggable renderer, with room for future formats and Prism integration. >> >>>> >> >>>> Does that direction sound reasonable from a package/layout >> perspective within the Go SDK? >> >>>> >> >>>> Thanks again for the guidance. >> >>>> >> >>>> Best, >> >>>> Yousuf >> >>>> >> >>>> On Tue, 24 Feb 2026 at 23:32, Robert Burke <[email protected]> >> wrote: >> >>>>> >> >>>>> As a rule, I'm in favour of this project, but outside of code >> reviews I'm not sure how much time I can spend on mentoring beyond that. >> (But I promise I'll look at all PRs tagged with @lostluck.) >> >>>>> >> >>>>> While the proprosal around "dot" is largely just a continuation of >> the pre-existing "dot" runner, the important thing is that it gives a basic >> scaffolding of going from something that currently works (Go pipeline to >> some other format for rendering), not the specific outcome (dot graphs). >> It's however a useful starting point for later incorporating the same code >> into Prism. >> >>>>> >> >>>>> The ideal case is using the dot runner as the initial proof of >> concept for converting it to use the portable pipeline proto format instead >> of the existing Raw Go SDK structure. Then somehow from there abstract >> what it produces from dot graphs, to some other format (mermaid flow >> charts, some fun HTML thing directly that can be made responsive in the UI >> with live metrics and the like, etc.). >> >>>>> >> >>>>> Incorporating it into the prism runner enables Python and Java (or >> other SDKs) to take advantage of it as well, which is where the value of >> this proposal is. >> >>>>> >> >>>>> Robert >> >>>>> >> >>>>> On Mon, Feb 23, 2026 at 12:18 AM Yousuf Ansari < >> [email protected]> wrote: >> >>>>>> >> >>>>>> Hi Beam community, >> >>>>>> >> >>>>>> My name is Yousuf. I'm a student who has been contributing to >> Apache projects for a while now - I have 4 merged PRs in Apache Superset >> and recently opened PR #37673 in Beam, which rewrites the Go SDK dot runner >> to generate DOT graphs from the portable pipeline proto. >> >>>>>> >> >>>>>> The PR notes that the new portable-proto-based approach enables >> future reuse in Prism and other portable tooling. I'd like to propose a Go >> SDK GSoC project built around that direction: >> >>>>>> >> >>>>>> Integrating the portable-proto DOT generation into the Prism >> runner's web UI, so developers can visualize their pipeline while it runs >> locally >> >>>>>> Adding per-transform metrics display on the graph nodes >> >>>>>> Potentially surfacing watermark progress per PCollection for >> streaming pipelines >> >>>>>> >> >>>>>> This feels like a natural 350-hour project that directly extends >> the work already started in #37673 and aligns with the Go SDK roadmap's >> focus on streaming features and portable runner improvements. >> >>>>>> >> >>>>>> I'd love to know: >> >>>>>> >> >>>>>> Is there mentor interest in a Go SDK project for GSoC 2026? >> >>>>>> Does this direction align with the priorities you have in mind? >> >>>>>> Are there related issues I should pick up to strengthen my >> proposal before the application window opens? >> >>>>>> >> >>>>>> Happy to discuss scope and adjust based on your feedback. >> >>>>>> >> >>>>>> Thanks, Yousuf >> >
