On 17.04.2013 03:25, Jim Harris wrote:
On Tue, Apr 16, 2013 at 3:04 PM, Alexander Motin <m...@freebsd.org <mailto:m...@freebsd.org>> wrote: Hi. Recently I've got 6-core/12-thread system on Sandy Bridge-E Core i7-3930K CPU and was unpleasantly surprised to see that TSCs are not synchronized there. While all 11 APs were synchronized, BSP was far behind them. Since it is single-socket system, I don't know any good reason for such behavior except some BIOS bug. But I've recalled that somewhere was some discussions about possible TSC synchronization. I've implemented patch below that allows to adjust TSC values of BSPs to AP's one on boot using CPU MSRs, hoping that they should not diverge after that: http://people.freebsd.org/~__mav/tsc_adj2.patch <http://people.freebsd.org/~mav/tsc_adj2.patch> I don't know very much about all different TSC hardware to predict when it is safe to enable the functionality, but at least on my system being enabled via loader tunable it seems working well. Comments? You may be remembering this thread on r238755 last year: http://lists.freebsd.org/pipermail/svn-src-head/2012-July/038992.html This was a bug fix in the TSC synchronization test code though, not anything for trying to adjust out-of-sync TSCs.
I remember that thread, but I think I've seen somebody told somewhere that it could be interesting to implement some MI mechanism. Never mind.
The Intel SDM (volume 3, section 17.13 of March 2013 revision) says earlier models can only write to lower 32 bits of IA32_TIME_STAMP_COUNTER, but these models also should not have invariant TSC so they would never even get to your new routine. So your patch seems OK for Intel CPUs, at least as a tunable that is disabled by default.
Thanks.
My only concern would be why TSC on the BSP started out-of-sync on your system. Theoretically, BIOS could adjust TSCs in SMM to try to hide SMI code execution from the OS, which could then make them out-of-sync again. Not sure if that's what's happening here, but might be worth a test putting the TSC test code on a periodic timer to see if they ever get out of sync again.
I did one more interesting observation: on every reboot drift between BSP and APs is growing proportionally to the previous system power-on time. On first boot it is -3878361036 (just above one second), after reboot some minutes later it is -1123454492776 (about 6 minutes), after another reboot it is -1853033521804 (about 10 minutes).
Unless my adjustment code would be active, I would guess that AP's TSC is running linearly while BSP's for some reason reset to zero on every reboot. But since I am synchronizing them on each boot, the only possibility for it I see is that there is some other timer(s) / counter(s) not affected by MSR writes that ticks linearly and reloading AP's TSC, but for some reason not reloading BSP's.
-- Alexander Motin _______________________________________________ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
