[swift-corelibs-dev] [xctest] Removing outliers from performance tests

[Drew Crawford via swift-corelibs-dev]

Hello folks,

I’m one of the heavy users of XCTest.measureBlock as it exists in Xcode 7.2.  
To give some hard numbers, I have ~50 performance tests in an OSX framework 
project, occupying about 20m wall clock time total.  This occurs on a 
per-commit basis.

The current implementation of measureBlock as it currently exists in 
closed-source Xcode is something like this:

1.  Run 10 trials
2.  Compare the average across those 10 trials to some baseline
3.  Compare the stdev across those 10 trials to some standard value (10% by 
default)

There are really a lot of problems with this algorithm, but maybe the biggest 
one is how it handles outliers.  If you have a test suite running for 20m, 
chances are “something” is going to happen on the build server in that time.  
System background task, software update, gremlins etc.

So what happens lately is exactly *one* of the 10 * 50 = 500 total 
measureBlocks takes a really long time, and it is a different failure each time 
(e.g., it’s not my code, I swear).  A result like this for some test is typical:




The probability of this kind of error grows exponentially with the test suite 
size.  If we assume for an individual measureBlock that it only fails due to 
“chance” .01% of the time, then the overall test suite at N = 500 will only 
pass 60% of the time.  This is very vaguely consistent with what I experience 
at my scale—e.g. a test suite that does not really tell me if my code is broken 
or not.

IMO the problem here is one of experiment design.  From the data in the 
screenshot, this very well might be a real performance regression that should 
be properly investigated.  It is only when I tell you a lot of extra 
information—e.g. that this test will pass fine the next 100 executions and it’s 
part of an enormous test suite where something is bound to fail—that a failure 
due to random chance seems likely.  In other words, running 10 iterations and 
pretending that will find performance regressions is a poor approach.

I’ve done some prototyping on algorithms that use a dynamically sized number of 
trials to find performance regressions.  Apple employees, see rdar://21315474 
for an algorithm for a sliding window for performance tests (that also has 
other benefits, like measuring nanosecond-scale performance).  I am certainly 
willing to contrib that work in the open if there’s consensus it’s a good 
direction.

However, now that this is happening in the open, I’m interested in getting 
others’ thoughts on this problem.  Surely I am not the only serious user of 
performance tests, and maybe people with better statistics backgrounds than I 
have can suggest an appropriate solution.

Drew___
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Re: [swift-corelibs-dev] [xctest] Removing outliers from performance tests

[Drew Crawford via swift-corelibs-dev]

> Unfortunately our corelibs implementation of XCTest isn’t ready yet for 
> performance testing.

That's why I'm here; I'm taking the temperature on implementing it.  I'm at the 
pain level where I need a solution in the next several months, even if the 
solution is to code it up myself.  My tests have failed 10x over this so far 
today.

I think the real question is, if I did implement basic performance testing, and 
I did implement a variable-sized window of runs, would that departure from the 
Old XCTest behavior (which uses 10 runs) disqualify the PR?  It's a basic 
compatibility question about how close we need to follow the Old XCTest 
behavior.

e.g. if XCS wanted to migrate to corelibs-xctest and it used variable #s of 
runs, presumably that would be an undertaking for the XCS team.  But I don't 
know whether those concerns (if they exist) play a role in what this project 
decides to do.

I'm going to do something on this problem eventually, unless someone else 
solves it first.  I'm just trying to work out whether I can do something 
upstream or whether this is a better candidate for an independent effort.







___
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

[swift-corelibs-dev] #if __CORELIBS_FOUNDATION__

[Drew Crawford via swift-corelibs-dev]

Hi folks,

I notice that swift-corelibs-foundation is (apparently) deliberately 
incompatible with Darwin Foundation 
(https://github.com/apple/swift-corelibs-foundation/blame/master/Foundation/NSFileManager.swift#L324
 
)
 at least for the immediate future.

I'm wondering if we can get an #if that would let me write conditional code.

Obviously the "real" answer is to fix the bugs, but (at least on my side) that 
will happen a lot faster if I "can" easily build the same codebase against 
corelibs-foundation and darwin-foundation.


___
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

[swift-corelibs-dev] bridging (SR-138)

[Drew Crawford via swift-corelibs-dev]

I've noticed that e.g. String is not bridged to NSString.  The expected 
workaround seems to be calling .bridge() everywhere.

1.  Is there a plan for bridging e.g. String with NSString?
2.  Would it be appropriate to PR in the meantime e.g.

extension String {
public func cStringUsingEncoding(encoding: UInt) -> UnsafePointer 
{
return self.bridge().cStringUsingEncoding(encoding)
}
}

  3.  The README also says

> We will also drop the 'NS' prefix from all Foundation classes


...is that the resolution?  e.g what is currently class NSString will become 
instead extension String?  Is there a bug open for that?___
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

[swift-corelibs-dev] relationship of CF

[Drew Crawford via swift-corelibs-dev]

I had a question about something I saw in the docs:

> A significant portion of the implementation of Foundation on Apple platforms 
> is provided by another framework called CoreFoundation (a.k.a. CF). CF is 
> written primarily in C and is very portable. Therefore we have chosen to use 
> it for the internal implementation of Swift Foundation where possible. As CF 
> is present on all platforms, we can use it to provide a common implementation 
> everywhere.

(emphasis added)

Is the intent of this paragraph to suggest that most PRs to 
swift-corelibs-foundation should be a C-language implementation to CF with a 
light Swift wrapper?  That goes against my intuition, but it "seems to be" a 
plain reading of the paragraph.

its justification about "all platforms" is also strange–I know CF "kind of" 
builds for Windows, but is anyone actually testing it there?  To make sure we 
aren't breaking it?  Or does "all platforms" mean something else here?

I feel like this paragraph is an opportunity to explain to a patch author how 
to structure their patch between use/maintenance/contributions to the CF layer 
vs the Swift layer.  I feel like it could do a much better job, but I don't 
understand what the design guidance actually is, so I can't fix it.





___
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Re: [swift-corelibs-dev] #if __CORELIBS_FOUNDATION__

[Drew Crawford via swift-corelibs-dev]

> On Jan 4, 2016, at 3:09 PM, Tony Parker  wrote:
> 
> In this particular case, how would you use the #if? Any should be source 
> compatible with AnyObject, since Any is a superset of AnyObject, right?

Well I don't know what you mean by "compatible" but the thing about strongly 
typed languages is we can't do

let a: Any

foo(a: AnyObject) { /* */ }

foo(a)

because that is "type error".  We can of course cast, but if in Darwin a.self 
is already AnyObject, casting again to AnyObject produces a warning, and I try 
to not have any warnings in my code.

This is kind of a side quest–there are a lot of ways to work around these 
problems–but the obviously straightforward one was to use #if to conditionally 
cast, and that wasn't available.

More broadly I find myself using the #if that I've created very often–my 
workflow seems to be to create an extension on an NSClass in-file that adds 
missing functionality and then PR it to corelibs-foundation afterwards–and 
conditional compilation really helps with that workflow.___
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev