Thanks. Made the go code similar to python using CopyBuffer with a block
size of 65536.
buf := make([]byte, 65536)
if _, err := io.CopyBuffer(hash, file, buf); err != nil {
fmt.Println(err)
}
Didn't make too much of a difference, was slightly faster.
What got it to the same place was running ComputeHash in the same goroutine
as the Walk function vs its own go routine for each file
+ ComputeHash(path, info, queue, wg)
- go ComputeHash(path, info, queue, wg)
*2.88s user 0.98s system 23% cpu 16.086 total, memory usage ~ 7MB*
Here's the before and after pprof webs:
BEFORE with 'go ComputeHash(...):
<https://lh3.googleusercontent.com/-aRKwq1P9_ec/WALXpkl_yxI/AAAAAAAADFY/WXn0PDcOw_Mk909yNp9Hh1tWUl0PlSVJACLcB/s1600/prof.cpu-with-go_compute_hash.png>
AFTER with 'ComputeHash(...):
<https://lh3.googleusercontent.com/-8LnYMr_UhOg/WALXsbuxkjI/AAAAAAAADFc/EAk7vOvl2zMJZARfcz2JpgXmZc_3YfFKwCLcB/s1600/prof.cpu-no-go.png>
Since disk read are SOO much slower, computing the hash for each file in
its own goroutine caused a huge slowdown..
btw this is on a RAID10, with SSD:
Old code SSD: 3.31s user 17.87s system 244% cpu 8.667 total
New code SDD:* 2.88s user 0.84s system 69% cpu 5.369 total*
Shows you can throw hardware at a problem BUT the old code locks up my
system momentarily..
On Saturday, October 15, 2016 at 3:27:38 PM UTC-4, Kevin Malachowski wrote:
>
> Sorry, I meant that calling Write on the hash type might be slower if it's
> called more often.
>
> (I'm on mobile right now. When I get back to a keyboard I'll try to come
> up with an example)
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
