Hi Alex, As mentioned before, I'm in the process of migrating a pipeline of several Kafka Streams applications over to Apache Beam and I'm hoping to leverage the tracing infrastructure that I had established using Jaeger whenever I can, but specifically to trace an element as it flows through a pipeline or potentially multiple pipelines.
An example might go something like this: - An event is produced from some service and sent to a Kafka Topic (with a tracing id in the headers) - The event enters my pipeline (Beam reads from that topic) and begins applying a series of transforms that evaluate the element itself (e.g. does it have any users associated with it, IP addresses, other interesting information). - When interesting information is encountered on the element (or errors), I'd like to be able to associate them with the trace (e.g. a user was found, this is some information about the user, this is the unique identifier associated with them, or there was an error because the user had a malformed e-mail address) - The traces themselves would be cumulative, so if an event was processed through one pipeline, it would contain all the necessary tracing headers in the message so if another pipeline picked it up from its destination topic (e.g. the destination of the first pipeline), the trace could be continued. I think that being able to pick up interactive systems would be a nice to have (e.g. this record is being sent to Parquet, Mongo, or some other topic), but I'm just trying to focus on being able to add to the trace at the ParDo/element level for now. I hope that helps. Rion On 2020/04/17 16:30:14, Alex Van Boxel <[email protected]> wrote: > Can you explain a bit more of what you want to achieve here? > > Do you want to trace how your elements go to the pipeline or do you want to > see how every ParDo interacts with external systems? > > On Fri, Apr 17, 2020, 17:38 Rion Williams <[email protected]> wrote: > > > Hi all, > > > > I'm reaching out today to inquire if Apache Beam has any support or > > mechanisms to support some type of distributed tracing similar to something > > like Jaeger (https://www.jaegertracing.io/). Jaeger itself has a Java > > SDK, however due to the nature of Beam working with transforms that yield > > immutable collections, I wasn't sure what the best avenue would be to > > correlate various transforms against a particular element would be? > > > > Coming from a Kafka Streams background, this process was pretty trivial as > > I'd simply store my correlation identifier within the message headers and > > those would be persisted as the element traveled through Kafka into various > > applications and topics. I'm hoping to still leverage some of that in Beam > > if at all possible or see what, if any, recommended approaches there are > > out there. > > > > My current approach involves the creation of a "Tracing Context" which > > would just be a wrapper for each of my elements that had their own > > associated trace with them and instead of just passing around a > > PCollection<X> I would use a PCollection<Tracable<X>> that would just be a > > wrapper for the tracer and the underlying element so that I could access > > the tracer during any element-wise operations in the pipeline. > > > > Any recommendations or suggestions are more than welcome! I'm very new to > > the Beam ecosystem, so I'd love to leverage anything out there that might > > help me from reinventing the wheel. > > > > Thanks much! > > > > Rion > > >
