[swift-corelibs-dev] Language or XCTest improvements to allow testing assert/precondition failures

[David Hart via swift-corelibs-dev]

Currently, it is impossible for XCTest to unit test assert and precondition 
failures because they kill the process. And those are important to unit test: 
how would you test NSArray's objectAtIndex bound conditions? Currently, the 
standard library tests those with StdlibUnittest, a small piece of code which 
runs those tests in a forked process.

I want to start a proposal but I'm not sure if it should come as a XCTest 
improvement which spawns a process or if the language should implement assert 
and precondition as a special kind of throw?

David
___
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Re: [swift-corelibs-dev] [swift-evolution] Language or XCTest improvements to allow testing assert/precondition failures

[Marc Knaup via swift-corelibs-dev]

+1 from me. I'm really annoyed by the fact I can't reach 100% code coverage
due to this :)

Being able to test them is very valuable. Take Swift's promise to be safe
for example. If you can't test the behavior in cases where an array index
was used which is out of bounds then you can't test even the promise to be
safe.

On Sun, Dec 13, 2015 at 11:03 AM, David Hart via swift-evolution <
swift-evolut...@swift.org> wrote:

> Currently, it is impossible for XCTest to unit test assert and
> precondition failures because they kill the process. And those are
> important to unit test: how would you test NSArray's objectAtIndex bound
> conditions? Currently, the standard library tests those with
> StdlibUnittest, a small piece of code which runs those tests in a forked
> process.
>
> I want to start a proposal but I'm not sure if it should come as a XCTest
> improvement which spawns a process or if the language should implement
> assert and precondition as a special kind of throw?
>
> David
> ___
> swift-evolution mailing list
> swift-evolut...@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution
>
___
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Re: [swift-corelibs-dev] [xctest] Removing outliers from performance tests

[Mike Ferris via swift-corelibs-dev]

I think that there’s a lot of room for improvement in how we measure and 
analyze perf test results. And I like some of your ideas of adaptive numbers of 
runs and what to do with outlier results, etc… that you presented below.

We would very much like to not diverge the API of the CoreLibs XCTest and 
Xcode’s.

There may be a bit more room for some variation in the behavior. So talking 
about doing something where the number of runs varies dynamically not via API 
but through smarter execution mechanisms, or doing better statistical analysis 
such as removing outlier results, this would introduce some change in the 
results that might be reported between the two implementations, but it would 
not give rise to people writing tests that would not work cross-platform.

Further, there’s always room for discussion of taking such ideas, and even API 
additions and supporting them in Xcode’s XCTest as well. But that sort of 
discussion will have to include the practicalities of schedule and resource on 
the Xcode side (and, to some unavoidable extent, there will be aspects to that 
discussion that will not be as transparent since we’ll be weighing that work 
against other work for the testing team that cannot be discussed as freely.)

This year, our primary goal for the core libraries is to broaden the 
implementation of the existing APIs in the OS X versions of the frameworks. For 
XCTest, we would also love to come up with some better answer for test 
discovery and potentially elicit help from the community in achieving that. 
Beyond that, the Xcode team’s bandwidth for incorporating other things that 
would necessaitate change to the Xcode XCTest is going to be limited.

Mike


> On Dec 12, 2015, at 6:10 PM, Drew Crawford via swift-corelibs-dev 
>  wrote:
> 
>> Unfortunately our corelibs implementation of XCTest isn’t ready yet for 
>> performance testing.
> 
> That's why I'm here; I'm taking the temperature on implementing it.  I'm at 
> the pain level where I need a solution in the next several months, even if 
> the solution is to code it up myself.  My tests have failed 10x over this so 
> far today.
> 
> I think the real question is, if I did implement basic performance testing, 
> and I did implement a variable-sized window of runs, would that departure 
> from the Old XCTest behavior (which uses 10 runs) disqualify the PR?  It's a 
> basic compatibility question about how close we need to follow the Old XCTest 
> behavior.
> 
> e.g. if XCS wanted to migrate to corelibs-xctest and it used variable #s of 
> runs, presumably that would be an undertaking for the XCS team.  But I don't 
> know whether those concerns (if they exist) play a role in what this project 
> decides to do.
> 
> I'm going to do something on this problem eventually, unless someone else 
> solves it first.  I'm just trying to work out whether I can do something 
> upstream or whether this is a better candidate for an independent effort.
> 
> 
> 
> 
> 
> 
> 
> ___
> swift-corelibs-dev mailing list
> swift-corelibs-dev@swift.org
> https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

___
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Re: [swift-corelibs-dev] Unresolved enumeration symbols

[Joseph Bell via swift-corelibs-dev]

Tony, all:

I went back to basics and used the following to build:

./swift/utils/build-script -R --xctest --foundation -t

Unfortunately I get the same result - missing symbols for enumerations in
libFoundation.so.  Once again I'm building against the master branch for
all repositories.

I'm going to build the debug configuration now and see - I would be curious
to know if anyone can reproduce this, I'd be happy to write up a Dockerfile
or something to at least have someone confirm the reproducibility.

Joe


On Sat, Dec 12, 2015 at 10:27 PM, Joseph Bell  wrote:

> Installing Ubuntu 15.10 on a VM, bringing in all of the prerequisites, I
> get the same result.  The build completes successfully but libFoundation.so
> is not including the enumerations in NSDateFormatter.
>
> public enum NSDateFormatterStyle : UInt {
> case NoStyle
> case ShortStyle
> case MediumStyle
> case LongStyle
> case FullStyle
> }
>
> For grins, I searched through libFoundation.so with nm -D looking for
> another enum, say one of the NSNumber format styles:
>
> $ nm -D libFoundation.so |grep CurrencyStyle
> $ nm -D libFoundation.so |grep PercentStyle
>
> No dice, Chicago.
>
> I can't say all enumerations are missing but these definitely are.
>
> I've also posted what I use to prep and build at
> https://github.com/iachievedit/package-swift; hoping there isn't an
> embarrassing oversight.
>
> Joe
>
>
> On Sat, Dec 12, 2015 at 8:00 PM, Joseph Bell  wrote:
>
>> Information about the system I'm building with, will note it doesn't have
>> anything else "special" i.e., no 4.x kernel or non-standard libc.  It is
>> really interesting, everything compiles okay, no errors thrown, etc.  I'm
>> going to try a build on a 15.10 VM as well to see if I can reproduce.
>>
>> uname -a
>> Linux darthvader 3.19.0-39-generic #44~14.04.1-Ubuntu SMP Wed Dec 2
>> 10:00:35 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
>>
>> lsb_release -a
>> No LSB modules are available.
>> Distributor ID: Ubuntu
>> Description: Ubuntu 14.04.3 LTS
>> Release: 14.04
>> Codename: trusty
>>
>> clang --version
>> Ubuntu clang version 3.6.0-2ubuntu1~trusty1 (tags/RELEASE_360/final)
>> (based on LLVM 3.6.0)
>> Target: x86_64-pc-linux-gnu
>> Thread model: posix
>>
>> cat /proc/cpuinfo
>> processor   : 0
>> vendor_id   : AuthenticAMD
>> cpu family  : 21
>> model   : 1
>> model name  : AMD FX(tm)-8120 Eight-Core Processor
>> stepping: 2
>> microcode   : 0x6000626
>> cpu MHz : 1400.000
>> cache size  : 2048 KB
>> physical id : 0
>> siblings: 8
>> core id : 0
>> cpu cores   : 4
>> apicid  : 16
>> initial apicid  : 0
>> fpu : yes
>> fpu_exception   : yes
>> cpuid level : 13
>> wp  : yes
>> flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> mca cmov pat pse3
>> 6 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp
>> lm constant_tsc
>>  rep_good nopl nonstop_tsc extd_apicid aperfmperf pni pclmulqdq monitor
>> ssse3 cx16 sse4_
>> 1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy
>> abm sse4a misali
>> gnsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 nodeid_msr topoext
>> perfctr_core per
>> fctr_nb arat cpb hw_pstate npt lbrv svm_lock nrip_save tsc_scale
>> vmcb_clean flushbyasid
>> decodeassists pausefilter pfthreshold vmmcall
>> bugs: fxsave_leak
>> bogomips: 6228.24
>> TLB size: 1536 4K pages
>> clflush size: 64
>> cache_alignment : 64
>> address sizes   : 48 bits physical, 48 bits virtual
>> power management: ts ttp tm 100mhzsteps hwpstate cpb
>>
>>
>>
>> On Sat, Dec 12, 2015 at 7:24 PM, Joe Bell  wrote:
>>
>>> Tony, yes, I'm building on an Ubuntu 14.04 system.  I can provide any
>>> uname or kernel information if you need.
>>>
>>> Joe
>>>
>>> On Dec 12, 2015, at 7:17 PM, Tony Parker 
>>> wrote:
>>>
>>> Hi Joe,
>>>
>>> I see you used the preset for linux, but I just wanted to double-check
>>> that you are indeed building on Linux.
>>>
>>> - Tony
>>>
>>> On Dec 12, 2015, at 3:48 PM, Joseph Bell via swift-corelibs-dev <
>>> swift-corelibs-dev@swift.org> wrote:
>>>
>>> Howdy.
>>>
>>> I'm using the basic build instructions (checking out swift, llvm, clang,
>>> etc.) and can successfully build a packaged release with
>>>
>>> ./swift/utils/build-script --preset=buildbot_linux_1404
>>> install_destdir=${INSTALL_DIR} i
>>> nstallable_package=${PACKAGE}
>>>
>>> However, the resulting build output for Foundation is missing
>>> enumeration types, and I noticed it in particular in the NSDateFormatter
>>> class:
>>>
>>> ➜  Foundation git:(master) ✗ nm NSDateFormatter.swift.o|grep Medium
>>>
>>>
>>> Of course this results in the symbols not being included in
>>> libFoundation.so
>>> ➜  Foundation git:(master) ✗ nm -D libFoundation.so |grep Medium
>>>
>>> Looking at my build from source I am missing about 200K of symbols that
>>> the SNAPSHOT build is including:
>>>
>>>   Foundation

Re: [swift-corelibs-dev] [swift-evolution] Language or XCTest improvements to allow testing assert/precondition failures

[Daniel Dunbar via swift-corelibs-dev]

> On Dec 13, 2015, at 2:03 AM, David Hart via swift-evolution 
>  wrote:
> 
> Currently, it is impossible for XCTest to unit test assert and precondition 
> failures because they kill the process. And those are important to unit test: 
> how would you test NSArray's objectAtIndex bound conditions? Currently, the 
> standard library tests those with StdlibUnittest, a small piece of code which 
> runs those tests in a forked process.
> 
> I want to start a proposal but I'm not sure if it should come as a XCTest 
> improvement which spawns a process or if the language should implement assert 
> and precondition as a special kind of throw?

One way to start a proposal is to try and work through what the implications of 
each approach are, decide which is best, and then write that up in a proposal 
explaining why you chose the approach you did.

If you are interested in working on this, I would definitely also survey how 
other frameworks do this. Googletest, for example, supports this via a "death 
tests" mechanism, which would be good to understand and compare to:
  
https://github.com/google/googletest/blob/master/googletest/docs/AdvancedGuide.md#death-tests

I suspect that spawning a process is the best approach, because it can cover 
more cases than the language built in features, and doesn't require language 
support, but I would be interested in seeing an in depth study of what this 
would mean for the framework and the APIs.

 - Daniel

> 
> David
> ___
> swift-evolution mailing list
> swift-evolut...@swift.org
> https://lists.swift.org/mailman/listinfo/swift-evolution

___
swift-corelibs-dev mailing list
swift-corelibs-dev@swift.org
https://lists.swift.org/mailman/listinfo/swift-corelibs-dev

Re: [swift-corelibs-dev] Unresolved enumeration symbols

[Tony Parker via swift-corelibs-dev]

I’m curious why it’s these particular enums are different. If you try to build 
a sample app that actually uses these enum values, does it fail to link?

- Tony

> On Dec 13, 2015, at 9:19 AM, Joseph Bell  wrote:
> 
> Tony, all:
> 
> I went back to basics and used the following to build:
> 
> ./swift/utils/build-script -R --xctest --foundation -t
> 
> Unfortunately I get the same result - missing symbols for enumerations in 
> libFoundation.so.  Once again I'm building against the master branch for all 
> repositories.
> 
> I'm going to build the debug configuration now and see - I would be curious 
> to know if anyone can reproduce this, I'd be happy to write up a Dockerfile 
> or something to at least have someone confirm the reproducibility.
> 
> Joe
> 
> 
> On Sat, Dec 12, 2015 at 10:27 PM, Joseph Bell  > wrote:
> Installing Ubuntu 15.10 on a VM, bringing in all of the prerequisites, I get 
> the same result.  The build completes successfully but libFoundation.so is 
> not including the enumerations in NSDateFormatter.
> 
> public enum NSDateFormatterStyle : UInt {
> case NoStyle
> case ShortStyle
> case MediumStyle
> case LongStyle
> case FullStyle
> }
> For grins, I searched through libFoundation.so with nm -D looking for another 
> enum, say one of the NSNumber format styles:
> 
> $ nm -D libFoundation.so |grep CurrencyStyle
> $ nm -D libFoundation.so |grep PercentStyle
> 
> No dice, Chicago.
> 
> I can't say all enumerations are missing but these definitely are.
> 
> I've also posted what I use to prep and build at 
> https://github.com/iachievedit/package-swift 
> ; hoping there isn't an 
> embarrassing oversight.
> 
> Joe
> 
> 
> On Sat, Dec 12, 2015 at 8:00 PM, Joseph Bell  > wrote:
> Information about the system I'm building with, will note it doesn't have 
> anything else "special" i.e., no 4.x kernel or non-standard libc.  It is 
> really interesting, everything compiles okay, no errors thrown, etc.  I'm 
> going to try a build on a 15.10 VM as well to see if I can reproduce. 
> 
> uname -a
> Linux darthvader 3.19.0-39-generic #44~14.04.1-Ubuntu SMP Wed Dec 2 10:00:35 
> UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
> 
> lsb_release -a
> No LSB modules are available.
> Distributor ID:   Ubuntu
> Description:  Ubuntu 14.04.3 LTS
> Release:  14.04
> Codename: trusty
> 
> clang --version
> Ubuntu clang version 3.6.0-2ubuntu1~trusty1 (tags/RELEASE_360/final) (based 
> on LLVM 3.6.0)
> Target: x86_64-pc-linux-gnu
> Thread model: posix
> 
> cat /proc/cpuinfo
> processor   : 0
> vendor_id   : AuthenticAMD
> cpu family  : 21
> model   : 1
> model name  : AMD FX(tm)-8120 Eight-Core Processor
> stepping: 2
> microcode   : 0x6000626
> cpu MHz : 1400.000
> cache size  : 2048 KB
> physical id : 0
> siblings: 8
> core id : 0
> cpu cores   : 4
> apicid  : 16
> initial apicid  : 0
> fpu : yes
> fpu_exception   : yes
> cpuid level : 13
> wp  : yes
> flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca 
> cmov pat pse3
> 6 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm 
> constant_tsc
>  rep_good nopl nonstop_tsc extd_apicid aperfmperf pni pclmulqdq monitor ssse3 
> cx16 sse4_
> 1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm 
> sse4a misali
> gnsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 nodeid_msr topoext 
> perfctr_core per
> fctr_nb arat cpb hw_pstate npt lbrv svm_lock nrip_save tsc_scale vmcb_clean 
> flushbyasid
> decodeassists pausefilter pfthreshold vmmcall
> bugs: fxsave_leak
> bogomips: 6228.24
> TLB size: 1536 4K pages
> clflush size: 64
> cache_alignment : 64
> address sizes   : 48 bits physical, 48 bits virtual
> power management: ts ttp tm 100mhzsteps hwpstate cpb
> 
> 
> 
> On Sat, Dec 12, 2015 at 7:24 PM, Joe Bell  > wrote:
> Tony, yes, I'm building on an Ubuntu 14.04 system.  I can provide any uname 
> or kernel information if you need.
> 
> Joe
> 
> On Dec 12, 2015, at 7:17 PM, Tony Parker  > wrote:
> 
>> Hi Joe,
>> 
>> I see you used the preset for linux, but I just wanted to double-check that 
>> you are indeed building on Linux.
>> 
>> - Tony
>> 
>>> On Dec 12, 2015, at 3:48 PM, Joseph Bell via swift-corelibs-dev 
>>> mailto:swift-corelibs-dev@swift.org>> wrote:
>>> 
>>> Howdy.
>>> 
>>> I'm using the basic build instructions (checking out swift, llvm, clang, 
>>> etc.) and can successfully build a packaged release with 
>>> 
>>> ./swift/utils/build-script --preset=buildbot_linux_1404 
>>> install_destdir=${INSTALL_DIR} i
>>> nstallable_package=${PACKAGE}
>>> 
>>> However, the resulting build output for Foundation is missing enumeration 
>>> types, and I noticed it in particular in the NSDat