Hi all, I'm reaching out today to inquire if Apache Beam has any support or mechanisms to support some type of distributed tracing similar to something like Jaeger (https://www.jaegertracing.io/). Jaeger itself has a Java SDK, however due to the nature of Beam working with transforms that yield immutable collections, I wasn't sure what the best avenue would be to correlate various transforms against a particular element would be?
Coming from a Kafka Streams background, this process was pretty trivial as I'd simply store my correlation identifier within the message headers and those would be persisted as the element traveled through Kafka into various applications and topics. I'm hoping to still leverage some of that in Beam if at all possible or see what, if any, recommended approaches there are out there. My current approach involves the creation of a "Tracing Context" which would just be a wrapper for each of my elements that had their own associated trace with them and instead of just passing around a PCollection<X> I would use a PCollection<Tracable<X>> that would just be a wrapper for the tracer and the underlying element so that I could access the tracer during any element-wise operations in the pipeline. Any recommendations or suggestions are more than welcome! I'm very new to the Beam ecosystem, so I'd love to leverage anything out there that might help me from reinventing the wheel. Thanks much! Rion
