nickstenning commented on PR #61897: URL: https://github.com/apache/airflow/pull/61897#issuecomment-3943154743
> I'm not really fond of creating synthetic spans because you have to [...] generate the `trace_id` and the `span_id`. [...] It was an unconventional way to use OTel [...] I'm not sure I agree, and I think we may conflating the complexity of the existing implementation with a conceptual approach that is relatively straightforward. I linked an example above. It's not that involved at all. The only spans that do anything unconventional are the ones where their start and end are (potentially) handled by different processes. The key steps are: 1. Generate the context up front and store it somewhere. I think storing trace context for a dag run or a task instance in the database is entirely reasonable given the number of different processes which interact with them and the lack of another obvious way to propagate trace context. 2. Extract trace context while handling work and set it as the current context for the SDK. 3. When a task instance or a dag run reaches a terminal state, use the "root" trace context to emit a root span. The only vaguely complicated part of this is forcing the values of the "next" trace ID and span ID, which as in my example can be done straightforwardly with a pair of context keys and a contextmanager helper. If it's possible architecturally to guarantee that one process can supervise an entire dag run or task instance, then maybe this isn't even needed? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
