Sri G,
How does this time compare to my “Dup” program? I can’t test for you…since it is your filesystem…but I thought I had it going about as fast as possible a few years ago when I wrote that one. https://github.com/MichaelTJones/dup Michael From: <golang-nuts@googlegroups.com> on behalf of Sri G <sriakhil.gogin...@gmail.com> Date: Saturday, October 15, 2016 at 6:46 PM To: golang-nuts <golang-nuts@googlegroups.com> Subject: [go-nuts] Re: Duplicate File Checker Performance Thanks. Made the go code similar to python using CopyBuffer with a block size of 65536. buf := make([]byte, 65536) if _, err := io.CopyBuffer(hash, file, buf); err != nil { fmt.Println(err) } Didn't make too much of a difference, was slightly faster. What got it to the same place was running ComputeHash in the same goroutine as the Walk function vs its own go routine for each file + ComputeHash(path, info, queue, wg) - go ComputeHash(path, info, queue, wg) 2.88s user 0.98s system 23% cpu 16.086 total, memory usage ~ 7MB Here's the before and after pprof webs: BEFORE with 'go ComputeHash(...): AFTER with 'ComputeHash(...): Since disk read are SOO much slower, computing the hash for each file in its own goroutine caused a huge slowdown.. btw this is on a RAID10, with SSD: Old code SSD: 3.31s user 17.87s system 244% cpu 8.667 total New code SDD: 2.88s user 0.84s system 69% cpu 5.369 total Shows you can throw hardware at a problem BUT the old code locks up my system momentarily.. On Saturday, October 15, 2016 at 3:27:38 PM UTC-4, Kevin Malachowski wrote: Sorry, I meant that calling Write on the hash type might be slower if it's called more often. (I'm on mobile right now. When I get back to a keyboard I'll try to come up with an example) -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
