Hi all, This came from the PR https://github.com/apache/airflow/pull/70059
During AIP-108 testing, users asked whether the UI Code view for mixed-language authoring (a Python file with @task.stub tasks backed by an e.g. Java SDK artifact) could also show the Lang-SDK-side source, e.g. as a second tab next to the Python file. We are leaning toward not supporting this feature as a deliberate design decision *at this moment*. My (personal) main reasoning is the current code view isn’t really that useful in practice to begin with. It’s usable for very simple dags, but in production systems, where the Python file only serves as a top-level entry point that just imports things defined in other modules, those "real" code can’t be shown. This is arguably even worse in Java-land, where you’re REQUIRED to define a public class in a separate file in the first place. In the practical scenario, the only way to display somewhat useful code (instead of just a class that repeats task definitions as the stubs), we’d need to collect all the source files that the JAR containing task definition can reach. This bloats the DagCode table significantly. Even worse, the value of doing this is *still* quite limited. The main reason people want the Java SDK in the first place is to be able to call existing Java functions, potentially deployed by other teams in bytecode form, without a source to display at all. The view is still not very useful if we take a step back. For a real mixed-language case, where you can mix more than one, or even two languages, a cheaper variant showing only the entrypoint file per Lang-SDK, still falls short. But if we display everything everywhere, we’re back to bloating the table. If we intend to improve the situation, I would suggest we fix this for ALL the cases, starting from pure Python ones that import other Python files. The concept of Dag Bundles already paves the way for this, and language SDK support comes naturally after it. However, this is a pretty significant undertaking on its own, and should be its own project with AIPs and stuff. In the meantime, it would be a waste of effort creating an artificially "nice" feature of limited value (if any) in practice for language SDKs. The proposal is therefore * Keep the status quo at this moment: the code view shows only the Python Dag file. Language Java SDK docs state this explicitly. * Merge #70059 to record the decision. * Close #67260 (issue on view Language SDK code) as won't-fix, referencing #70059. Feedback welcome. In particular, if you have a concrete use case where an entrypoint-only view would carry its weight, we'd like to hear it. Otherwise we'll proceed with the ADR by lazy consensus. Also feel free to send me a message if you would like to discuss further on a better-code-view project. It probably should stay out of this thread, however. If you prefer using the mailing list, please open a new thread instead. Best, TP -- Some useful links: - ADR PR: https://github.com/apache/airflow/pull/70059 - Original ask: https://github.com/apache/airflow/issues/67260 - AIP-108: https://cwiki.apache.org/confluence/x/pY4mGQ - AIP-85 (DagImporter): https://cwiki.apache.org/confluence/x/_Q7OEg --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
