Hi Brian, > On Dec 7, 2015, at 3:54 PM, Brian Gesiak <modoca...@gmail.com> wrote: > > I sent up another attempt for this: > https://github.com/apple/swift-corelibs-xctest/pull/10 > <https://github.com/apple/swift-corelibs-xctest/pull/10> > > Based on the discussion in that pull request, I'm now exploring using lit to > test the XCTest output. > > Rather than asking contributors to install lit via the Python Package Index, > I was considering using the apple/llvm repository's copy. Contributors would > need to have cloned llvm in a specific directory relative to the cloned > swift-corelibs-xctest, like so: > > rootdir/ > llvm/ > swift-corelibs-xctest/ > > This appears to be a convention in the Swift family of repositories, so I > don't anticipate this to cause confusion among contributors. > > If anyone has any input on this approach to lit, or on the direction of these > changes in general, please let me know! All feedback is greatly appreciated. >
This is probably a fine approach - swift-corelibs-foundation, for example, assumes that swift-corelibs-xctest is a sibling directory to it when building. Even if we change our minds later, there’s no doubt that having a starting point is valuable. - Tony > - Brian Gesiak > _____________________________ > From: Mike Ferris <mfer...@apple.com <mailto:mfer...@apple.com>> > Sent: Friday, December 4, 2015 3:42 PM > Subject: Re: [swift-corelibs-dev] [xctest] Who tests the tests? > To: Brian Gesiak <modoca...@gmail.com <mailto:modoca...@gmail.com>> > Cc: Tony Parker <anthony.par...@apple.com <mailto:anthony.par...@apple.com>>, > <swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org>>, Daniel > Dunbar <daniel_dun...@apple.com <mailto:daniel_dun...@apple.com>> > > > Oops. It was pointed out to me that XCTestObserver is deprecated. I get it > confused sometimes with its replacement.. What we’d presumably want to > pursue adding the the corelibs XCTest is the API from XCTestObservation.h > (and associated stuff). > > Your same questions about the bundle stuff will still apply, though. On the > other hand, perhaps a partial implementation that at least allowed > observation starting at the test suites or even just at the test cases would > suffice to start things off. > > > Mike > > On Dec 4, 2015, at 9:08 AM, Brian Gesiak < modoca...@gmail.com > <mailto:modoca...@gmail.com>> wrote: > > Excellent, thanks for the feedback everyone! > > On Thu, Dec 3, 2015 at 10:39 PM, Daniel Dunbar < daniel_dun...@apple.com > <mailto:daniel_dun...@apple.com> > wrote: > It should be possible to use an out-of-process model that still uses XCTest > itself to run the tests. For example, in the package manager we have some > tests which spawn the package manager in order to test the end-to-end > behavior. Ideally we would only do this for a small number of tests that > really need this level of testing, and use unit testing for the rest. > > I'll send a pull request with this approach, since I believe it's the > least invasive to the current API while still getting us regression > tests for the current implementation. Of course I'd be thrilled if > someone beats me to it! :) > > On Dec 3, 2015, at 8:03 PM, Tony Parker < anthony.par...@apple.com > <mailto:anthony.par...@apple.com>> wrote: > The reason I’m asking is that (like Foundation and dispatch), we’re trying to > keep the API surface of this XCTest very similar to the one that ships today > with Xcode. This will help developers who need to integrate their > cross-platform tests into suites that include features that Obj-C XCTest has > that we will probably not add to the Swift one (e.g., UI testing). > > Absolutely agree. Still, once we have regression tests in place, I'd > love to start a conversation about whether we could provide a Obj-C > XCTest-compatible API layer on top of some more flexible architecture. > > On Fri, Dec 4, 2015 at 11:24 AM, Mike Ferris < mfer...@apple.com > <mailto:mfer...@apple.com>> wrote: > One possible direction to consider which would be more consistent with our > goals this year for API-compatibility would be to look into implementing > XCTestObserver. > > Yes! Very exciting. We'll need to consider how methods like > `-[XCTestObservation testBundleWillStart:]` map to SwiftXCTest, > especially considering we don't use NSBundle at all (which I think is > a great thing). I'll be looking forward to participating in more > discussions on this mailing list. > > - Brian Gesiak > > > On Fri, Dec 4, 2015 at 11:24 AM, Mike Ferris < mfer...@apple.com > <mailto:mfer...@apple.com> > wrote: > > One possible direction to consider which would be more consistent with our > goals this year for API-compatibility would be to look into implementing > XCTestObserver. > > Mike > > On Dec 3, 2015, at 8:03 PM, Tony Parker < anthony.par...@apple.com > <mailto:anthony.par...@apple.com>> wrote: > > Hi Brian, > > On Dec 3, 2015, at 3:45 PM, Brian Gesiak < modoca...@gmail.com > <mailto:modoca...@gmail.com>> wrote: > > Hello! This is in reference to > https://github.com/apple/swift-corelibs-xctest/pull/3 > <https://github.com/apple/swift-corelibs-xctest/pull/3>. That pull request > contains a commit that attempts to refactor XCTest such that it is more > "unit-testable”. > > > Cool, thanks for looking into this area. > > To do so, it gives XCTMain an additional parameter: a list of objects > conforming to the Reporter protocol. I think of this as a minimal, corelibs > equivalent to Apple's XCTest's XCTestObserver.h. I say "minimal" because > Reporter only defines Reporter.log(), whereas XCTestObserver has one method > for each kind of test event (started, failed, finished, etc.). > > > Do you think it’d be possible to split out the idea of adding this new API to > XCTest from getting some tests for XCTest itself? > > The reason I’m asking is that (like Foundation and dispatch), we’re trying to > keep the API surface of this XCTest very similar to the one that ships today > with Xcode. This will help developers who need to integrate their > cross-platform tests into suites that include features that Obj-C XCTest has > that we will probably not add to the Swift one (e.g., UI testing). > > We made a concession to language limitations with the XCTMain function, > because there is no way to dynamically discover all of the test cases. I’d > really like to get rid of it in the long term in favor of something else; > maybe a decoration like @testable that we could find automatically. > > - Tony > > These reporters are, for now, storied in a global array. In the future, I'd > like to discuss moving XCTest to a model in which all tests are (optionally) > run in sub-processes, each of which may (optionally) run in parallel. This > global array most certainly won't work for such a change, but for now, I > simply want to have regression tests on the project. It's hard to send pull > requests without knowing everything still works! > > Besides this approach, which modifies XCTest in order to test it, it may be > more prudent to add tests *without* changing XCTest at all. To do so, I could > add tests that run programs that call XCTMain(), then verify what's printed > to stdout. This could be done using a Python script (which would go well with > the build script, also in Python). > > I'd love input on which of these approaches sounds more viable. Other ideas > are also, of course, welcome! > > - Brian Gesiak > > > > _______________________________________________ > swift-corelibs-dev mailing list > swift-corelibs-dev@swift.org <mailto:swift-corelibs-dev@swift.org> > https://lists.swift.org/mailman/listinfo/swift-corelibs-dev > <https://lists.swift.org/mailman/listinfo/swift-corelibs-dev> > >
_______________________________________________ swift-corelibs-dev mailing list swift-corelibs-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-corelibs-dev
