Sorry for the delay in following up. I have had several long discussions on this topic with Dmitri and others, which I will try to summarize here:
*TL;DR*: We think the right long-term path forward is to pursue porting SourceKit to Linux, and would like to explore that direction first before trying to develop a compromise which either (a) uses internal and likely-to-break APIs (e.g., dump-ast) or (b) devotes significant engineering resources to a "right solution" for this problem, but which will only solve this problem and not other issues where having an API for use with the Swift language. ## Problem Statement The XCTest API has historically been defined in terms of methods with a particular naming convention `test...` which were in subclasses of XCTestCase. On OS X, these methods can be found via the Objective-C runtime but that does not work on Linux. Our current solution on Linux requires manual specification of all of the test methods, and is a huge maintenance burden for people trying to use XCTest on Linux or maintain cross-platform projects. ## Background This feature works on OS X in two ways: 1. As mentioned, tests are run by dynamic discovery through the Objective-C runtime when the bundle containing the tests is executed. 2. Within the Xcode IDE, tests are "discovered" (for use in UI) through the use of the Xcode indexer. The mechanism at work here, for Swift, is a combination of functionality in the indexing engine (to aggregate and query results) and the raw underlying data provided by SourceKit. We would like any implementation of this feature to share as much code as possible with SourceKit and Xcode's implementation in order for cross-platform projects to behave predictably. It also happens that SwiftPM has several other needs for API-based interactions with the functionality in the Swift compiler. Several examples: * We would like to be able to enforce the strictness of the `Package.swift` manifest file format. This requires APIs to interact with the Swift AST. https://bugs.swift.org/browse/SR-1433 * We would like to support advanced features for dealing with automatic inter-module dependencies. For example, one idea which has been proposed is that if a module attempted to import a module upon which it did not have a dependency, that we would prompt the developer if this dependency should automatically be added to the manifest. Doing this feature well requires APIs to interact with the Swift compiler as it parses source code and reports diagnostics. * We would like to understand the current level of parallelism being used by the Swift compiler, so that llbuild can accurately schedule work. This requires APIs for interacting with an in-flight compilation process. ## Discussion We discussed several avenues of attack on this problem. I will go through them one by one. I am just attempting to summarize the conversation, obviously there is a ton of nuance to each point, and hopefully other people will chime in if I missed something important. ### Language Features One way to view this problem is that XCTest's API (i.e., depending on test naming and dynamic discovery) is a poor fit for Swift today, and that this problem should be tackled in that direction. For example, the Swift compiler itself has a test framework that does not depend on dynamic discovery. One could also imagine language-level features which would solve the arbitrary problem of wanting to find discoverable things; that could take the form of an `@XCTest` attribute, or "generalized attribute" support. The current mission for Swift 3, however, is API parity between Linux and OS X, and so this direction does not lead to any short term solution. For that reason, while many of us ultimately think this is the right long term direction, we need to find another one as well. ### Custom "Supported" XCTest Feature The next option is to pursue a custom, but "supported" feature intended to tackle this problem head on. Some proposals that have come up are, e.g., a new compiler flag which emits the list of test methods. This approach has a couple unfortunate properties: 1. It is non-trivial. We can either design this as an incredibly XCTest specific feature requiring understanding of the backend (compiler directly emits metadata into .o file), or a midway feature (compiler tells us list of test methods, but then we have to manage incremental compilation and the desire to not compile things multiple times more than necessary ourselves). 2. It is not-reusable. The work we do here doesn't help with any of the other ways we want to use the Swift compiler as an API. ### Implement an API-based Interface This approach means porting SourceKit to Linux, and then building this feature on top of that (that itself will be non-trivial, which is something not to be glossed over). Everyone generally agrees we should have some kind of API-based access to the compiler, so this is in line with our long-term direction, but in a way which is actionable now (it does not require design). This approach has its own issues: 1. Porting SourceKit may require a fair amount of work. The current code base is very OS X specific in particular areas (generally around the use of `libxpc`). This will require adding abstractions, developing alternatives, and writing the code and build system changes to get this to happen. 2. Somehwat unrelated, but the compiler itself (`swiftc`) is not yet written in a way that it can be used from SourceKit. If we do port and use SourceKit, it will mean the toolchains increase in size because we effectively will have two copies of many parts of Swift/Clang/LLVM installed. 3. By itself, this does not fully solve the issue. As mentioned in the background, SourceKit itself does not directly manage all parts of XCTest discovery today on OS X, it is a collaboration between SourceKit and the Xcode indexer to do discovery without execution. The expectation, however, is that we can, however, use SourceKit's APIs to implement this feature in a way that is "supported". ## Conclusion The conclusion was that after weighing all of the tradeoffs, it made the most sense to encourage porting of SourceKit to Linux and then using it to build out the Linux test discovery feature. This was most in line with a desirable long-term direction without being blocked on language design. I don't think we are particularly firm in this conclusion. If you feel that the work you have already invested gets us so close that it is worth prioritizing that approach, then please push back. - Daniel > On May 1, 2016, at 8:19 PM, Brian Gesiak <modoca...@gmail.com> wrote: > > I made an attempt at adding `swiftc -frontend -dump-xctest-methods`: > https://github.com/apple/swift/pull/2364 > <https://github.com/apple/swift/pull/2364> > > Feedback appreciated!! > > - Brian Gesiak > > > On Sun, Apr 24, 2016 at 8:59 PM, Brian Gesiak <modoca...@gmail.com > <mailto:modoca...@gmail.com>> wrote: > Thanks to this commit > <https://github.com/apple/swift/commit/ad269b0e1fbc12037ae2c16634b5b451061657c6> > it looks as if IsTestCandidate has been moved out of SourceKit and into > libIndex: > > isTestCandidate(swift::ValueDecl) > <https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L754> > swift::index::FuncDeclIndexSymbol.IsTestCandidate > <https://github.com/apple/swift/blob/41e4e9b6efc745f04df23bd6a803a467c57a66b8/include/swift/Index/IndexSymbol.h#L102> > and where it’s set > <https://github.com/apple/swift/blob/8dad7f780347788f6032ad9e25ce5340aecf4073/lib/Index/Index.cpp#L786>. > I’m looking into adding an option to swiftc to emit XCTest candidate methods. > How does swiftc -frontend -dump-xctest-methods sound? > > - Brian Gesiak > > > On Sun, Apr 17, 2016 at 5:50 PM, Drew Crawford <d...@sealedabstract.com > <mailto:d...@sealedabstract.com>> wrote: > >> On Apr 3, 2016, at 10:05 PM, Dmitri Gribenko <griboz...@gmail.com >> <mailto:griboz...@gmail.com>> wrote: >> >>> Hmm... but then wouldn't that more tightly couple the test discovery tool >>> and the Swift compiler? In an earlier email you said you "like #3 better >>> [...] because that would decouple the test discovery tool from the Swift >>> compiler." I think that part is confusing me. >> >> Sorry -- what I meant is that the compiler remains the point of truth >> about the language and can find the tests. The tools that actually >> generate glue code won't need to parse code, and would be decoupled >> from the compiler. > > I am internally shipping a test framework that discovers tests via an > out-of-tree parser. Teaching swiftc about XCTest syntax is not sufficient to > deprecate my parser, and therefore is not sufficient for the compiler to be > the source of truth for my tests. > >
_______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
