Can you explain a bit more of what you want to achieve here? Do you want to trace how your elements go to the pipeline or do you want to see how every ParDo interacts with external systems?
On Fri, Apr 17, 2020, 17:38 Rion Williams <[email protected]> wrote: > Hi all, > > I'm reaching out today to inquire if Apache Beam has any support or > mechanisms to support some type of distributed tracing similar to something > like Jaeger (https://www.jaegertracing.io/). Jaeger itself has a Java > SDK, however due to the nature of Beam working with transforms that yield > immutable collections, I wasn't sure what the best avenue would be to > correlate various transforms against a particular element would be? > > Coming from a Kafka Streams background, this process was pretty trivial as > I'd simply store my correlation identifier within the message headers and > those would be persisted as the element traveled through Kafka into various > applications and topics. I'm hoping to still leverage some of that in Beam > if at all possible or see what, if any, recommended approaches there are > out there. > > My current approach involves the creation of a "Tracing Context" which > would just be a wrapper for each of my elements that had their own > associated trace with them and instead of just passing around a > PCollection<X> I would use a PCollection<Tracable<X>> that would just be a > wrapper for the tracer and the underlying element so that I could access > the tracer during any element-wise operations in the pipeline. > > Any recommendations or suggestions are more than welcome! I'm very new to > the Beam ecosystem, so I'd love to leverage anything out there that might > help me from reinventing the wheel. > > Thanks much! > > Rion >
