On Jun 16, 2011, at 3:36 PM, Sven C. Merckens wrote:
> Hi roy, Hi Dan,
>
> many thanks for Your responses.
>
> I am using napp-it to control the OpenSolaris-Systems
> The napp-it-interface shows a dedup factor of 1.18x on System 1 and 1.16x on
> System 2.
You're better off disabling dedup for this workload. If the dedup ratio was
more like
10, 342, or some number > 2 dedup can be worthwhile. IMHO, dedup ratios < 2 are
not good candidates for dedup.
You can look at existing, non-deduped data to get an estimate of the potential
dedup
savings using:
zdb -S poolname
> Dedup is on "always" (not only at the start), also compression is activated:
> System 1 = compression on (lzib?)
> System 2 = compression on (gzip-6)
> compression rates:
> System 1 = 1.10x
> System 2 = 1.48x
>
> compression and dedup were some of the primary reasons to choose zfs in this
> situation.
Compression is a win for a large number of cases. Dedup, not so much.
>
> We tried more RAM (48GB) at the beginning to check if this would do anything
> on performance. But it did not, but we had only about 3-4TB of data on the
> storage at that time (performance was good). So I will order some RAM-modules
> and double the RAM to 48GB.
>
> The RAM usage is about 21GB, 3GB free (on both systems after a while). At the
> start (after a few hours of usage and only 3-4TB of data) the usage was
> identical. I read on some places, that zfs will use all memory, leaving only
> 1GB left (so I thought the RAM wasn't used completely). The swap isn't used
> by the system,
>
> Now the systems are idle and the RAM-usage is very little:
> top:
>
> System 1
> Memory: 24G phys mem, 21G free mem, 12G total swap, 12G free swap
>
> System 2
> Memory: 24G phys mem, 21G free mem, 12G total swap, 12G free swap
>
>
> Starting to read about 12GB from System 2 and RAM-Usage goes up to
> performance is at about 65-70MB/s via GigaBit (iSCSI).
> Memory: 24G phys mem, 3096M free mem, 12G total swap, 12G free swap
>
> ok, I understand, more RAM will be no fault.. ;)
>
> On System 1 there is no such massive change in RAM usage while copying files
> to and from the volume.
> But the Performance is only about 20MB/s via GigaBit (iSCSI).
This is likely not a dedup issue. More likely, Nagle is biting you or there is a
serialization that is not immediately obvious. I have also seen questionable
network configurations cause strange slowdowns for iSCSI.
-- richard
>
> So RAM can´t be the issue on System 1 (which has more data stored).
> This system ist also equipped with a 240GB SSD used for L2ARC at the second
> LSI-controller inside the server-enclosure.
>
>
> Roy:
> But is the L2ARC also important while writing to the device? Because the
> storeges are used most of the time only for writing data on it, the
> Read-Cache (as I thought) isn´t a performance-factor... Please correct me, if
> my thoughts are wrong...
>
> But this is only a "small cost"-addition, to add also a 120GB/240GB OCZ
> Vertex 2 SSD to System 2 (≈ 150/260 Euro). I will give it a try.
>
> Would it be better to add the SSD to the LSI-Controller (put it in the
> JBOD-Storage) or put it in the server enclosure itself and connect it to the
> internal SATA-Controller?
>
>
> Do You have any tips for the settings of the dataset?
> These are the settings
>
>
> PROPERTY System 1 System 2
> used 34.4T 19.4T
> available 10.7T 40.0T
> referenced 34.4T 19.4T
> compressratio 1.10x 1.43x
> mounted yes yes
> quota none none
> reservation none none
> recordsize 128K 128k
> mountpoint / /
> sharenfs off off
> checksum on on
> compression on gzip
> atime off off
> devices on on
> exec on on
> setuid on on
> readonly off off
> zoned off off
> snapdir hidden hidden
> aclinherit passthrough passthrough
> canmount on on
> xattr on on
> copies 1 1
> version 5 5
> utf8only off off
> normalization none none
> casesensitivity insensitive
> insensitive
> vscan off off
> nbmand off off
> sharesmb off off
> refquota none none
> refreservation none none
> primarycache metadata all
> secondarycache all all
> usedbysnapshots 0 0
> usedbydataset 34.4T 19.4T
> usedbychildren 0 0
> usedbyrefreservation 0 0
> logbias latency latency
> dedup on on
> mlslabel none none
> sync standard standard
>
>
> Best regards
>
>
> Sven C.
> merckensscATmac.com
>
> On Wed, Jun 15, 2011 at 07:19:05PM +0200, Roy Sigurd Karlsbakk wrote:
>>
>> Dedup is known to require a LOT of memory and/or L2ARC, and 24GB isn't
>> really much with 34TBs of data.
>
> The fact that your second system lacks the l2arc cache device is absolutely
> your prime suspect.
>
>
> Am 15.06.2011 um 19:19 schrieb Roy Sigurd Karlsbakk:
>
>>> System1 (inhouse)
>>> ----------------
>>> SuperMicro-enclosure 2U SC825TQ
>>> Mainboard: X8DTH-IF
>>> 1 x 1 x Quad-Core Intel Xeon E5620 processor, 2.4GHz, 12MB L3 Cache
>>> 24GB of RAM (3 x 8GB)
>>> 1 x LSISAS9211-8I (for the internal 8 drive-carriers)
>>> 1 x LSISAS9200-8E (for the JBOD)
>>>
>>> attached is a
>>> 1 x SC847E16-RJBOD1 with two backplanes (each backplane is connected
>>> to one port of the LSI-controller).
>>
>> You mention further down that you had dedup turned on for some time. Could
>> you do a "zpool list" and report the dedup amount?
>>
>> Dedup is known to require a LOT of memory and/or L2ARC, and 24GB isn't
>> really much with 34TBs of data.
>>
>> Vennlige hilsener / Best regards
>>
>> roy
>> --
>> Roy Sigurd Karlsbakk
>> (+47) 97542685
>> r...@karlsbakk.net
>> http://blogg.karlsbakk.net/
>> --
>> I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det
>> er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av
>> idiomer med fremmed opprinnelse. I de fleste tilfeller eksisterer adekvate
>> og relevante synonymer på norsk.
>
> _______________________________________________
> zfs-discuss mailing list
> zfs-discuss@opensolaris.org
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
