New problem:
I'm following all the advice I summarized into the OP of this thread, and
testing on a test system. (A laptop). And it's just not working. I am
jumping into the dedup performance abyss far, far eariler than predicted...
My test system is a laptop with 1.5G ram, c_min =150M, c_max =1.2G
I have just a single sata 7.2krpm hard drive, no SSD.
Before I start, I have 1G free ram (according to top.)
According to everything we've been talking about, I expect roughly 1G
divided by 376 bytes = 2855696 (2.8M) blocks in my pool before I start
running out of ram to hold the DDT and performance degrades.
I create a pool. Enable dedup. Set recordsize=512
I write a program that will very quickly generate unique non-dedupable data:
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char *argv[]) {
int i;
int numblocks=atoi(argv[1]);
// Note: Expect one command-line argument integer.
FILE *outfile;
outfile=fopen("junk.file","w");
for (i=0; i<numblocks ; i++)
fprintf(outfile,"%512d",i);
fflush(outfile);
fclose(outfile);
}
Disable dedup. Run with a small numblocks. For example: time
~/datagenerator 100
Enable dedup and repeat.
They both complete instantly.
Repeat with a higher numblocks... 1000, 10000, 100000...
Repeat until you find the point where performance with dedup is
significantly different from performance without dedup.
See below. Right around 400,000 blocks, dedup is suddenly an order of
magnitude slower than without dedup.
Times to create the file:
numblocks dedup=off dedup=verify DDTsize Filesize
100000 2.5sec 1.2sec 36 MB 49 MB
110000 1.4sec 1.3sec 39 MB 54 MB
120000 1.4sec 1.5sec 43 MB 59 MB
130000 1.5sec 1.8sec 47 MB 63 MB
140000 1.6sec 1.6sec 50 MB 68 MB
150000 4.8sec 7.0sec 54 MB 73 MB
160000 4.8sec 7.6sec 57 MB 78 MB
170000 2.1sec 2.1sec 61 MB 83 MB
180000 5.2sec 5.6sec 65 MB 88 MB
190000 6.0sec 10.1sec 68 MB 93 MB
200000 4.7sec 2.6sec 72 MB 98 MB
210000 6.8sec 6.7sec 75 MB 103 MB
220000 6.2sec 18.0sec 79 MB 107 MB
230000 6.5sec 16.7sec 82 MB 112 MB
240000 8.8sec 10.4sec 86 MB 117 MB
250000 8.2sec 17.0sec 90 MB 122 MB
260000 8.4sec 17.5sec 93 MB 127 MB
270000 6.8sec 19.2sec 97 MB 132 MB
280000 13.1sec 16.5sec 100 MB 137 MB
290000 9.4sec 73.1sec 104 MB 142 MB
300000 8.5sec 7.7sec 108 MB 146 MB
310000 8.5sec 7.7sec 111 MB 151 MB
320000 8.6sec 11.9sec 115 MB 156 MB
330000 9.3sec 33.5sec 118 MB 161 MB
340000 8.3sec 54.3sec 122 MB 166 MB
350000 8.3sec 50.0sec 126 MB 171 MB
360000 9.3sec 109.0sec 129 MB 176 MB
370000 9.5sec 12.5sec 133 MB 181 MB
380000 10.1sec 28.6sec 136 MB 186 MB
390000 10.2sec 14.6sec 140 MB 190 MB
400000 10.7sec 136.7sec 143 MB 195 MB
410000 11.4sec 116.6sec 147 MB 200 MB
420000 11.5sec 220.9sec 151 MB 205 MB
430000 11.7sec 151.3sec 154 MB 210 MB
440000 12.7sec 144.7sec 158 MB 215 MB
450000 12.0sec 202.1sec 161 MB 220 MB
460000 13.9sec 134.7sec 165 MB 225 MB
470000 12.2sec 127.6sec 169 MB 229 MB
480000 13.1sec 122.7sec 172 MB 234 MB
490000 13.1sec 106.3sec 176 MB 239 MB
500000 15.8sec 174.6sec 179 MB 244 MB
550000 14.2sec 216.6sec 197 MB 269 MB
600000 15.6sec 294.2sec 215 MB 293 MB
650000 16.7sec 332.8sec 233 MB 317 MB
700000 19.0sec 269.6sec 251 MB 342 MB
750000 20.1sec 472.0sec 269 MB 366 MB
800000 21.0sec 465.6sec 287 MB 391 MB
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
