Re: [PATCH] sort: parallel external sort implementation

2010/3/4 Pádraig Brady > On 04/03/10 06:01, Joey Degges wrote: > >> Hello, >> >> Matt (cc'd, not on list) and I have developed a patch that parallelizes >> sort >> using pthreads. The threading approach taken here is to break up input >> files &

Re: [PATCH] sort: parallel external sort implementation

2010/3/4 Pádraig Brady > On 04/03/10 09:55, Chen Guo wrote: > >> Hi Padraig, >> Actually we're in the same class. They were assigned external sort, my >> group >> worked on internal sort. Our patch submission's imminent, looks like you >> guys >> are gonna be busy with code reviews :-) >> > >

[PATCH] sort: parallel external sort implementation

luntary context switches: 256 File system inputs: 1566224 From 5403741c2d5bc814c2f722d8e33537469ab2d064 Mon Sep 17 00:00:00 2001 From: Joey Degges Date: Tue, 2 Mar 2010 23:59:28 -0800 Subject: [PATCH] sort: parallel external sort implementation - input files are broken up into groups based

Re: Taking advantage of L1 and L2 cache in sort

2010/3/2 Pádraig Brady > Currently when sorting we take advantage of the RAM vs disk > speed bump by using a large mem buffer dependent on the size of RAM. > However we don't take advantage of the cache layer in the > memory hierarchy which has an increasing importance in modern > systems given t

Re: [PATCH] sort: use posix_fadvise to announce access patterns on files opened for reading

2010/3/2 Pádraig Brady > On 02/03/10 11:42, Erik Auerswald wrote: > >> Hi, >> >> On Mon, Mar 01, 2010 at 05:33:38PM -0800, Joey Degges wrote: >> >>> Were you sure to remount your devices to clear the cache before running >>> these tests? While test

Re: [PATCH] sort: use posix_fadvise to announce access patterns on files opened for reading

2010/3/2 Pádraig Brady > On 02/03/10 06:32, Joey Degges wrote: > >> 2010/3/1 Joey Degges >> >> Thanks for testing, these results are interesting. I will run some tests >>> on >>> various systems tonight and report back. Maybe my earlier re

Re: [PATCH] sort: use posix_fadvise to announce access patterns on files opened for reading

2010/3/1 Joey Degges > Thanks for testing, these results are interesting. I will run some tests on > various systems tonight and report back. Maybe my earlier results were > caused by peculiar hardware. > > Results differ once again, this time I've tested on a different syste

Re: [PATCH] sort: use posix_fadvise to announce access patterns on files opened for reading

2010/3/1 Pádraig Brady > On 01/03/10 09:36, Pádraig Brady wrote: > >> On 28/02/10 00:46, Pádraig Brady wrote: >> >>> On 27/02/10 22:16, Joey Degges wrote: >>> >>>> 2010/2/27 Pádraig Brady mailto:p...@draigbrady.com>> >>>> >

Re: [PATCH] sort: use posix_fadvise to announce access patterns on files opened for reading

atch containing both WILLNEED and DONTNEED is included below. To accommodate different advice flags I added some defines for the different flags. Would you recommend doing this any other way? Thanks, Joey >From bd934f42675739d064776bae4e631b0b2f02f7ab Mon Sep 17 00:00:00 2001 From: Joey Degg

Re: [PATCH] sort: use posix_fadvise to announce access patterns on files opened for reading

2010/2/26 Pádraig Brady > Thanks for the patch. > The speed up of 1s was what % as a matter of interest? > I presume the drive was the bottleneck and the CPU was not at 100%? > Thank you for the comments. Averaging a few runs shows 2.92% improvement. The average CPU usage came out to be 61%. See

[PATCH] sort: use posix_fadvise to announce access patterns on files opened for reading

s the variance was much lower and I saw a consistent speed up of 1 second. Can you offer any suggestions/comments? Thanks, Joey >From 34a93042301f0838d6086a4ca1aa251c89cf1ba0 Mon Sep 17 00:00:00 2001 From: Joey Degges Date: Thu, 25 Feb 2010 22:54:50 -0800 Subject: [PATCH] sort: use posix_fa

Re: [PATCH] sort.c: Fix minor memory leak, "files" is never free'd

On Tue, Feb 16, 2010 at 11:23 AM, Jim Meyering wrote: > Joey Degges wrote: > > On Tue, Feb 16, 2010 at 5:05 AM, Eric Blake wrote: > > > >> According to Joey Degges on 2/16/2010 2:02 AM: > >> > Hello, > >> > > >> > At sort.c:3271 

Re: [PATCH] sort.c: Fix minor memory leak, "files" is never free'd

On Tue, Feb 16, 2010 at 5:05 AM, Eric Blake wrote: > According to Joey Degges on 2/16/2010 2:02 AM: > > Hello, > > > > At sort.c:3271 'files' is allocated but it is not free'd before main > exits: > > files = xnmalloc (argc, sizeof *files); >

[PATCH] sort.c: Fix minor memory leak, monthtab is never free'd if HAVE_NL_LANGINFO is defined and we're not in the "C" locale

nd of main(). >From 75920d3d8e9632be1f7317d461b62ad14f3a12f0 Mon Sep 17 00:00:00 2001 From: Joey Degges Date: Mon, 15 Feb 2010 23:35:43 -0800 Subject: [PATCH 3/3] Fix minor memory leak: monthtab is never free'd if HAVE_NL_LANGINFO is defined and we're not in the "C" locale. -

[PATCH] sort.c: Fix minor memory leak, 'temp_dirs' is never free'd

Hello, At either sort.c 3488 or 3647 temp_dirs is alloc'd through add_temp_dir(). In any case temp_dirs is going to alloc'd. This patch frees temp_dirs at the bottom of main(). >From b65f65a84dde8447636c7b3efa597b5ac933 Mon Sep 17 00:00:00 2001 From: Joey Degges Date: Mon, 15 F

[PATCH] sort.c: Fix minor memory leak, "files" is never free'd

Hello, At sort.c:3271 'files' is allocated but it is not free'd before main exits: files = xnmalloc (argc, sizeof *files); This patch frees files at the end of main(). >From 31ec4fb66f8b7fd15e2ef297b3de81f2b91da2f2 Mon Sep 17 00:00:00 2001 From: Joey Degges Date: Mon, 15