Re: [zfs-discuss] zfs compression with Oracle - anyone implemented?

Ed, See my answers inline: "I don't think your question is clear. What do you mean "on oracle backed by storage luns?"" We'll be using luns from a storage array vs ZFS controller disks. The luns are mapped the db server and from there initialize under ZFS. " Do you mean "on oracle hardware?"

Re: [zfs-discuss] zfs compression with Oracle - anyone implemented?

> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss- > boun...@opensolaris.org] On Behalf Of Brad > > Hi! I'd been scouring the forums and web for admins/users who deployed > zfs with compression enabled on Oracle backed by storage array luns. > Any problems with cpu/memory overhead?

[zfs-discuss] zfs compression with Oracle - anyone implemented?

Hi! I'd been scouring the forums and web for admins/users who deployed zfs with compression enabled on Oracle backed by storage array luns. Any problems with cpu/memory overhead? -- This message posted from opensolaris.org ___ zfs-discuss mailing list

Re: [zfs-discuss] ZFS compression

>> I've read a small amount about compression, enough to find that it'll effect >> performance (not a problem for me) and that once you enable compression it >> only effects new files written to the file system. > > Yes, that's true. Compression on defaults to lzjb which is fast; but gzip-9 >

Re: [zfs-discuss] ZFS compression

On 2010-Jul-25 21:12:08 +0800, Ben wrote: >I've read a small amount about compression, enough to find that it'll effect >performance (not a problem for me) and that once you enable compression it >only effects new files written to the file system. >Is this still true of b134? And if it is, ho

Re: [zfs-discuss] ZFS compression

Thanks Alex, I've set compression on and have transferred data from the OpenSolaris machine to my Mac, deleted any snapshots and am now transferring them back. It seems to be working, but there's lots to transfer! I didn't know that MacZFS was still going, it's great to hear that people are sti

Re: [zfs-discuss] ZFS compression

On 25 Jul 2010, at 14:12, Ben wrote: > I've read a small amount about compression, enough to find that it'll effect > performance (not a problem for me) and that once you enable compression it > only effects new files written to the file system. Yes, that's true. Compression on defaults to l

[zfs-discuss] ZFS compression

Hi all, I'm running out of space on my OpenSolaris file server and can't afford to buy any new storage for a short while. Seeing at the machine has a dual core CPU at 2.2GHz and 4GB ram, I was thinking compression might be the way to go... I've read a small amount about compression, enough to

Re: [zfs-discuss] ZFS compression and deduplication on root pool on SSD

One of the great privileges of using OpenSolaris is the helpfulness and deep knowledge of the community. Thanks for the suggestions. I cleared out /var/pkg/downloads, and got back a couple of gigabytes. I've enabled compress and dedup. Is there a simple set of commands that I could send that fi

Re: [zfs-discuss] ZFS compression and deduplication on root pool on SSD

On Sun, Feb 28, 2010 at 07:36:30PM -0800, Bill Sommerfeld wrote: > To avoid this in the future, set PKG_CACHEDIR in your environment to > point at a filesystem which isn't cloned by beadm -- something outside > rpool/ROOT, for instance. +1 - I've just used a dataset mounted at /var/pkg/downloa

Re: [zfs-discuss] ZFS compression and deduplication on root pool on SSD

On 02/28/10 15:58, valrh...@gmail.com wrote: Also, I don't have the numbers to prove this, but it seems to me > that the actual size of rpool/ROOT has grown substantially since I > did a clean install of build 129a (I'm now at build133). WIthout > compression, either, that was around 24 GB, but

Re: [zfs-discuss] ZFS compression and deduplication on root pool on SSD

On Sun, 28 Feb 2010, valrh...@gmail.com wrote: backup server, I should be able to compress by about a factor of 1.5x. If I enable both on the rpool filesystem, then clone the boot environment, that should enable it on the new BE (which would be a child of rpool/ROOT), right? If by 'clone' yo

[zfs-discuss] ZFS compression and deduplication on root pool on SSD

I am running my root pool on a 60 GB SLC SSD (OCZ Agility EX). At present, my rpool/ROOT has no compression, and no deduplication. I was wondering about whether it would be a good idea, from a performance and data integrity standpoint, to use one, the other, or both, on the root pool. My current

Re: [zfs-discuss] ZFS compression on Clearcase

On 4 Feb 2010, at 16:35, Bob Friesenhahn wrote: > On Thu, 4 Feb 2010, Darren J Moffat wrote: >>> Thanks - IBM basically haven't test clearcase with ZFS compression therefore, they don't support currently. Future may change, as such my customer cannot use compression. I have asked IBM for road

Re: [zfs-discuss] ZFS compression on Clearcase

On 04/02/2010 12:42, Darren J Moffat wrote: On 04/02/2010 12:13, Roshan Perera wrote: Hi Darren, Thanks - IBM basically haven't test clearcase with ZFS compression therefore, they don't support currently. Future may change, as such my customer cannot use compression. I have asked IBM for roadma

Re: [zfs-discuss] ZFS compression on Clearcase

On 4 Feb 2010, at 16:35, Bob Friesenhahn wrote: > On Thu, 4 Feb 2010, Darren J Moffat wrote: >>> Thanks - IBM basically haven't test clearcase with ZFS compression >>> therefore, they don't support currently. Future may change, as such my >>> customer cannot use compression. I have asked IBM for

Re: [zfs-discuss] ZFS compression on Clearcase

On Thu, 4 Feb 2010, Darren J Moffat wrote: Thanks - IBM basically haven't test clearcase with ZFS compression therefore, they don't support currently. Future may change, as such my customer cannot use compression. I have asked IBM for roadmap info to find whether/when it will be supported.

Re: [zfs-discuss] ZFS compression on Clearcase

Subject: Re: [zfs-discuss] ZFS compression on Clearcase To: Roshan Perera Cc: zfs-discuss@opensolaris.org > On 04/02/2010 12:13, Roshan Perera wrote: > >Hi Darren, > > > >Thanks - IBM basically haven't test clearcase with ZFS compression > therefore, they don't su

Re: [zfs-discuss] ZFS compression on Clearcase

On 04/02/2010 12:13, Roshan Perera wrote: Hi Darren, Thanks - IBM basically haven't test clearcase with ZFS compression therefore, they don't support currently. Future may change, as such my customer cannot use compression. I have asked IBM for roadmap info to find whether/when it will be sup

Re: [zfs-discuss] ZFS compression on Clearcase

- Original Message - From: Darren J Moffat Date: Thursday, February 4, 2010 11:59 am Subject: Re: [zfs-discuss] ZFS compression on Clearcase To: Roshan Perera Cc: zfs-discuss@opensolaris.org > On 04/02/2010 11:54, Roshan Perera wrote: > >Anyone in the group using ZFS compression on cl

Re: [zfs-discuss] ZFS compression on Clearcase

On 04/02/2010 11:54, Roshan Perera wrote: Anyone in the group using ZFS compression on clearcase vobs? If so any issues, gotchas? There shouldn't be any issues and I'd be very surprised if there was. IBM support informs that ZFS compression is not supported. Any views on this? Need more da

[zfs-discuss] ZFS compression on Clearcase

Hi All, Anyone in the group using ZFS compression on clearcase vobs? If so any issues, gotchas? IBM support informs that ZFS compression is not supported. Any views on this? Rgds Roshan ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http:/

Re: [zfs-discuss] zfs compression algorithm : jpeg ??

On Fri, 4 Sep 2009, Louis-Frédéric Feuillette wrote: JPEG2000 uses arithmetic encoding to do the final compression step. Arithmetic encoding has a higher compression rate (in general) than gzip-9, lzbj or others. There is an opensource implementation of jpeg2000 called jasper[1]. Jasper is the

Re: [zfs-discuss] zfs compression algorithm : jpeg ??

On Fri, Sep 04, 2009 at 01:41:15PM -0700, Richard Elling wrote: > On Sep 4, 2009, at 12:23 PM, Len Zaifman wrote: > >We have groups generating terabytes a day of image data from lab > >instruments and saving them to an X4500. > > Wouldn't it be easier to compress at the application, or between

Re: [zfs-discuss] zfs compression algorithm : jpeg ??

On Fri, 2009-09-04 at 13:41 -0700, Richard Elling wrote: > On Sep 4, 2009, at 12:23 PM, Len Zaifman wrote: > > > We have groups generating terabytes a day of image data from lab > > instruments and saving them to an X4500. > > Wouldn't it be easier to compress at the application, or between th

Re: [zfs-discuss] zfs compression algorithm : jpeg ??

On Sep 4, 2009, at 12:23 PM, Len Zaifman wrote: We have groups generating terabytes a day of image data from lab instruments and saving them to an X4500. Wouldn't it be easier to compress at the application, or between the application and the archiving file system? We have tried lzbj : com

[zfs-discuss] zfs compression algorithm : jpeg ??

We have groups generating terabytes a day of image data from lab instruments and saving them to an X4500. We have tried lzbj : compressratio = 1.13 in 11 hours , 1.3 TB -> 1.1 TB gzip -9 : compress ratio = 1.68 in > 37 hours, 1.3 TB -> .75 TB The filesystem performance was

Re: [zfs-discuss] zfs compression - btrfs compression

[EMAIL PROTECTED] wrote: > On Mon, Nov 03, 2008 at 12:33:52PM -0600, Bob Friesenhahn wrote: >> On Mon, 3 Nov 2008, Robert Milkowski wrote: >>> Now, the good filter could be to use MAGIC numbers within files or >>> approach btrfs come up with, or maybe even both combined. >> You are suggesting that

Re: [zfs-discuss] zfs compression - btrfs compression

On Mon, Nov 03, 2008 at 12:33:52PM -0600, Bob Friesenhahn wrote: > On Mon, 3 Nov 2008, Robert Milkowski wrote: > > Now, the good filter could be to use MAGIC numbers within files or > > approach btrfs come up with, or maybe even both combined. > > You are suggesting that ZFS should detect a GIF or

Re: [zfs-discuss] zfs compression - btrfs compression

Bob Friesenhahn wrote: > On Mon, 3 Nov 2008, Robert Milkowski wrote: >> Maybe that's a good one - so if couple of blocks do not compress then >> flag it in file metadata and do not try to compress any blocks within >> the file anymore. Of course for some files it will be suboptimal so >> maybe a da

Re: [zfs-discuss] zfs compression - btrfs compression

On Mon, 3 Nov 2008, Robert Milkowski wrote: > Now, the good filter could be to use MAGIC numbers within files or > approach btrfs come up with, or maybe even both combined. You are suggesting that ZFS should detect a GIF or JPEG image stored in a database BLOB. That is pretty fancy functionality

Re: [zfs-discuss] zfs compression - btrfs compression

On Mon, 3 Nov 2008, Robert Milkowski wrote: > > Maybe that's a good one - so if couple of blocks do not compress then > flag it in file metadata and do not try to compress any blocks within > the file anymore. Of course for some files it will be suboptimal so > maybe a dataset option? This is inte

Re: [zfs-discuss] zfs compression - btrfs compression

Hello Darren, Monday, November 3, 2008, 12:44:29 PM, you wrote: DJM> Robert Milkowski wrote: >> Hello zfs-discuss, >> >> http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable-standalone.git;a=commit;h=eecfe5255c533fefd38072a04e4afb56c40d9719 >> "If compression for a given set of pages f

Re: [zfs-discuss] zfs compression - btrfs compression

Robert Milkowski wrote: > Hello zfs-discuss, > > http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable-standalone.git;a=commit;h=eecfe5255c533fefd38072a04e4afb56c40d9719 > "If compression for a given set of pages fails to make them smaller, the > file is flagged to avoid future compressio

[zfs-discuss] zfs compression - btrfs compression

Hello zfs-discuss, http://git.kernel.org/?p=linux/kernel/git/mason/btrfs-unstable-standalone.git;a=commit;h=eecfe5255c533fefd38072a04e4afb56c40d9719 "If compression for a given set of pages fails to make them smaller, the file is flagged to avoid future compression attempts later." Maybe that's a

Re: [zfs-discuss] ZFS compression or lack thereof...

On Sun, Sep 7, 2008 at 15:08, Bob Friesenhahn <[EMAIL PROTECTED]> wrote: > I should mention that if copying the file causes it to be nicely > compressed, then you can use this to your advantage. Your log-file > rotator can copy the file and delete the original rather than just > renaming it. You

Re: [zfs-discuss] ZFS compression or lack thereof...

On Sun, 7 Sep 2008, Bob Friesenhahn wrote: > If the problem is due to the "trickle factor" then you should see that > if you copy a large log file that the filesystem now shows that some > data is compressed. I should mention that if copying the file causes it to be nicely compressed, then you ca

Re: [zfs-discuss] ZFS compression or lack thereof...

On Sun, 7 Sep 2008, Kenny wrote: > > So I enable the lzjb compression feature and start poring in syslog > files. However the problem appears that the files are not > compressing. A zfs get all command shows compression is "on". My > compressratio reports 1.00 and the files are the same size

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

Adam Leventhal wrote: > This is a great idea. I'd like to add a couple of suggestions: > > It might be interesting to focus on compression algorithms which are > optimized for particular workloads and data types, an Oracle database for > example. NB. Oracle 11g has builtin compression. In genera

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

pedantic comment below... dave johnson wrote: > "Richard Elling" <[EMAIL PROTECTED]> wrote: >> Dave Johnson wrote: >>> "roland" <[EMAIL PROTECTED] > wrote: >>> > >>> > there is also no filesystem based approach in >>> compressing/decompressing a whole filesystem. >>>

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

> Wouldn't ZFS's being an integrated filesystem make it > easier for it to > identify the file types vs. a standard block device > with a filesystem > overlaid upon it? > > I read in another post that with compression enabled, > ZFS attempts to > compress the data and stores it compressed if it

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

"Richard Elling" <[EMAIL PROTECTED]> wrote: > Dave Johnson wrote: >> "roland" <[EMAIL PROTECTED] > wrote: >> > >> > there is also no filesystem based approach in >> compressing/decompressing a whole filesystem. >> one could kludge this by setting the compression parame

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

On Mon, Jul 09, 2007 at 03:42:03PM -0700, Darren Dunham wrote: > > Wouldn't ZFS's being an integrated filesystem make it easier for it to > > identify the file types vs. a standard block device with a filesystem > > overlaid upon it? > > I'm not sure. I would think that most applications are go

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

On Mon, Jul 09, 2007 at 05:27:44PM -0500, Haudy Kazemi wrote: > Wouldn't ZFS's being an integrated filesystem make it easier for it to > identify the file types vs. a standard block device with a filesystem > overlaid upon it? How? The API to ZFS that most everything uses is the POSIX API. Ther

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

> Wouldn't ZFS's being an integrated filesystem make it easier for it to > identify the file types vs. a standard block device with a filesystem > overlaid upon it? I'm not sure. I would think that most applications are going to use the POSIX layer where there's no separate API for filetypes.

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

On Jul 9 2007, Domingos Soares wrote: >Hi, > >> It might be interesting to focus on compression algorithms which are >> optimized for particular workloads and data types, an Oracle database for >> example. > >Yes, I agree. That is what I meant when I said "The study might be >extended to the analy

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

Hi, > why not starting with lzo first - it`s already in zfs-fuse on linux and it > looks, that it`s just "in between lzjb and gzip" in terms of performance and > compression ratio. > there needs yet to be demonstrated that it behaves similar on solaris. Good question and I'm afraid I don't have

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

Hi, > It might be interesting to focus on compression algorithms which are > optimized for particular workloads and data types, an Oracle database for > example. Yes, I agree. That is what I meant when I said "The study might be extended to the analysis of data in specific applications (e.g. web

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

Dave Johnson wrote: > "roland" <[EMAIL PROTECTED] > wrote: > > > > there is also no filesystem based approach in > compressing/decompressing a whole filesystem. you can have 499gb of data > on a 500gb partition - and if you need some more space you would think > turni

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

"roland" <[EMAIL PROTECTED]> wrote: > > there is also no filesystem based approach in compressing/decompressing a > > whole filesystem. you can have 499gb of data on a 500gb partition - and if > > you need some more space you would think turning on compression on that fs > > would solve your pro

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

> zfs tries to compress a datablock and if that isn`t compressible enough, it > doesn`t store it compressed. That feature was pointed out to me off-list, and it makes great sense. I had not heard about that before this thread. This message posted from opensolaris.org ___

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

>One thing ZFS is missing is the ability to select which files to compress. yes. there is also no filesystem based approach in compressing/decompressing a whole filesystem. you can have 499gb of data on a 500gb partition - and if you need some more space you would think turning on compression on

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

Awesome initiative. One thing ZFS is missing is the ability to select which files to compress. Even a simple heuristic like "don't compress mp3,avi,zip,tar files" would yield a tremendous change in which data is compressed on consumer computers. I don't know if such a heuristic is planned o

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

nice idea! :) >We plan to start with the development of a fast implementation of a Burrows >Wheeler Transform based algorithm (BWT). why not starting with lzo first - it`s already in zfs-fuse on linux and it looks, that it`s just "in between lzjb and gzip" in terms of performance and compressi

Re: [zfs-discuss] ZFS Compression algorithms - Project Proposal

This is a great idea. I'd like to add a couple of suggestions: It might be interesting to focus on compression algorithms which are optimized for particular workloads and data types, an Oracle database for example. It might be worthwhile to have some sort of adaptive compression whereby ZFS could

[zfs-discuss] ZFS Compression algorithms - Project Proposal

Bellow, follows a proposal for a new opensolaris project. Of course, this is open to change since I just wrote down some ideas I had months ago, while researching the topic as a graduate student in Computer Science, and since I'm not an opensolaris/ZFS expert at all. I would really appreciate any s

Re: [zfs-discuss] zfs compression - scale to multiple cpu ?

On 16 June, 2007 - George sent me these 1,1K bytes: > Where can you find the timeframe on that Tomas? http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6460622 /Tomas -- Tomas Ögren, [EMAIL PROTECTED], http://www.acc.umu.se/~stric/ |- Student at Computing Science, University of Umeå `-

Re: [zfs-discuss] zfs compression - scale to multiple cpu ?

Where can you find the timeframe on that Tomas? Thanks! On 6/16/07, Tomas Ögren <[EMAIL PROTECTED]> wrote: :n 16 June, 2007 - roland sent me these 0,5K bytes: > hi ! > > i think i have read somewhere that zfs gzip compression doesn`t scale > well since the in-kernel compression isn`t done mult

Re: [zfs-discuss] zfs compression - scale to multiple cpu ?

:n 16 June, 2007 - roland sent me these 0,5K bytes: > hi ! > > i think i have read somewhere that zfs gzip compression doesn`t scale > well since the in-kernel compression isn`t done multi-threaded. > > is this true - and if so - will this be fixed ? > > what about default lzjb compression - is

[zfs-discuss] zfs compression - scale to multiple cpu ?

hi ! i think i have read somewhere that zfs gzip compression doesn`t scale well since the in-kernel compression isn`t done multi-threaded. is this true - and if so - will this be fixed ? what about default lzjb compression - is it different regarding this "issue" ? thanks roland This messa

Re: [zfs-discuss] ZFS compression / ARC interaction

Mark Maybee wrote: Yup, your assumption is correct. We currently do compression below the ARC. We have contemplated caching data in compressed form, but have not really explored the idea fully yet. The same applies to encryption, the current play is to encrypt just after where we currently c

Re: [zfs-discuss] ZFS compression / ARC interaction

On 12/7/06, Andrew Miller <[EMAIL PROTECTED]> wrote: Quick question about the interaction of ZFS filesystem compression and the filesystem cache. We have an Opensolaris (actually Nexenta alpha-6) box running RRD collection. These files seem to be quite compressible. A test filesystem conta

Re: [zfs-discuss] ZFS compression / ARC interaction

On 12/8/06, Mark Maybee <[EMAIL PROTECTED]> wrote: Yup, your assumption is correct. We currently do compression below the ARC. We have contemplated caching data in compressed form, but have not really explored the idea fully yet. Hmm... interesting idea. That will incur CPU to do a decompres

Re: [zfs-discuss] ZFS compression / ARC interaction

Andrew Miller wrote: Quick question about the interaction of ZFS filesystem compression and the filesystem cache. We have an Opensolaris (actually Nexenta alpha-6) box running RRD collection. These files seem to be quite compressible. A test filesystem containing about 3,000 of these files sho

[zfs-discuss] ZFS compression / ARC interaction

Quick question about the interaction of ZFS filesystem compression and the filesystem cache. We have an Opensolaris (actually Nexenta alpha-6) box running RRD collection. These files seem to be quite compressible. A test filesystem containing about 3,000 of these files shows a compressratio

Re: [zfs-discuss] ZFS compression

On Tue, Aug 22, 2006 at 10:09:19AM -0700, Rich Teer wrote: > On Tue, 22 Aug 2006, Matthew Ahrens wrote: > > > gzip. We plan to implement a broader range of compression algorithms in > > the future. > > Cool. Presumably, the algorithm used will be a user-settable property? That's correct, you w

Re: [zfs-discuss] ZFS compression

On Tue, 22 Aug 2006, Matthew Ahrens wrote: > gzip. We plan to implement a broader range of compression algorithms in > the future. Cool. Presumably, the algorithm used will be a user-settable property? -- Rich Teer, SCNA, SCSA, OpenSolaris CAB member President, Rite Online Inc. Voice: +1 (2

RE: [zfs-discuss] ZFS compression

>-Original Message- >From: [EMAIL PROTECTED] [mailto:zfs-discuss->[EMAIL PROTECTED] On Behalf Of roland >can someone tell, how effective is zfs compression and space-efficiency >(regarding small files) ? >linux-kernel source tree % ls -l linux-2.6.17.tar.gz -rw--- 1 nliu staff

Re: [zfs-discuss] ZFS compression

On Tue, Aug 22, 2006 at 08:43:32AM -0700, roland wrote: > can someone tell, how effective is zfs compression and > space-efficiency (regarding small files) ? > > since compression works at the block level, i assume compression may > not come into effect as some may expect. (maybe i`m wrong here)

[zfs-discuss] ZFS compression

Hello ! I searched the net and the forum for this, but couldn`t find anything about this. can someone tell, how effective is zfs compression and space-efficiency (regarding small files) ? since compression works at the block level, i assume compression may not come into effect as some may exp

Re: [zfs-discuss] ZFS compression best-practice?

On Thu, Jul 27, 2006 at 03:54:02PM -0400, Christine Tran wrote: > - What is the compression algorithm used? It is based on the Lempel-Ziv algorithm. > - Is there a ZFS feature that will output the real uncompressed size of > the data? The scenario is if they had to move a compressed ZFS > file

[zfs-discuss] ZFS compression best-practice?

Hi there, Customer has some questions regarding ZFS compression. - What is the compression algorithm used? - How much space saving, ballpark? (I said it depends on the data but customer still wants to know if we have stats.) - Is there a ZFS feature that will output the real uncompressed siz

[zfs-discuss] ZFS compression API (Was Re: [osol-discuss] Re: Re: where to start?)

[EMAIL PROTECTED] wrote: Robert Milkowski wrote: But only if compression is turned on for a filesystem. Of course, and the default is off. However I think it would be good to have an API so application can decide what to compress and what not. I agree that an API would be good. However I do