Re: [zfs-discuss] ZFS as a Gateway for a stroage network
This is basically the setup I suggested a year or so ago. While the theory is sound, the major problem with it is that iSCSI and ZFS are not a great combination when a device (in your case server) goes down. If you create a pool of several iSCSI devices, when any one fails, the entire pool will lock up for 3 minutes while it waits for iSCSI to timeout. Provided you have redundancy it will work fine after this. And in terms of building a fail over cluster, yes this is also pretty easy to do, and something I tested myself. My notes on this are pretty old now, but drop me an e-mail on googlemail.com if you'd like a copy of them. I got a cluster working and failing over fine for CIFS and NFS clients with next to no prior experience of Solaris. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Hybrid Pools - Since when?
Hi, Does anyone know since when hybrid pools are available in ZFS? Are there ZFS "versions"? XVM Server is based on Nevada b93, and I need to know if it supports hybrid pools. TIA, Rafael. -- => Rafael Friedlander => Sun Microsystems => OEM Specialist => +972 544 971 564 ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS home server SATA disk setup
That card does work well, and uses the same chipset as the Thumper. I've found that there are some issues with hot swap, but other than that it works fine. I've got one in a live ZFS server right now. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS as a Gateway for a stroage network
That is very interesting. What kind of hardware did you use? Do you have any statistics about throughput and I/O behavior? Maybe you could provide the detailed architecture. Unfortunately I did not find your e-mail address in your user profile for direct contact. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS as a Gateway for a stroage network
Bob, thank you very much for your detailed answer. Indeed, Resilvering could be a very difficult situation in such a big storage pool. I could solve this issue by building a pool for each backend node. But then I run into the same problems I have at this moment: The disk space of my server is heavily varying. A fixed size pool could not provide enough space or it provides much more space than is needed in one moment. One big storage pool is very interesting because it does not run out of space than several dedicated storage pools. With a JOBOD I will run into the same problems which I have at this moment if the number of server is growing, won’t I? Is there some other way you would recommend in order to solve this problem (one big storage pool; several backup nodes)? -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS as a Gateway for a stroage network
Dave, what kind of hardware did you used? I am scared about the bandwith and I/O throughput of the zfs gateway. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS as a Gateway for a stroage network
On Sun, 14 Dec 2008, Dak wrote: > dedicated storage pools. With a JOBOD I will run into the same > problems which I have at this moment if the number of server is > growing, won't I? Is there some other way you would recommend in > order to solve this problem (one big storage pool; several backup > nodes)? I don't understand your concern regarding JBOD. JBOD provides a way that your storage pool size can be increased by installing more drives, or adding another JBOD storage array. ZFS is very good at growing its storage size by adding more disks. With this approach you can maximize usable storage capacity by using a space efficient strategy like raidz or raidz2. Using backend servers with a complex disk-based OS (e.g. Solaris) is surely more failure prone than using devices which requires only "simple" firmware to boot and provide service. The iSCSI protocol over ethernet adds more latency and offers less throughput than the SAS or fiber channel that a JBOD array will use. If you are truely expecting your backend servers to be "backup nodes" then I think you are stuck with using simple mirroring on the head-end so that all of the data is available on each backup node. As someone pointed out to me, this approach achieves "maximum disk utilization" by consuming twice as much disk space. Bob == Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Hybrid Pools - Since when?
Rafael Friedlander wrote: > Hi, > > Does anyone know since when hybrid pools are available in ZFS? Are there > ZFS "versions"? > > XVM Server is based on Nevada b93, and I need to know if it supports > hybrid pools. > Hybrid pool slogs (ZIL) were introduced in Nevada builds 68 and 69, and is also in latest Solaris 10. I'm not sure when "cache" (L2ARC) devices were introduced. There used to be a web page with a list of all the zpool versions, but it's not where it was, and I can't find it now. -- Andrew ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Hybrid Pools - Since when?
On 14 Dec 2008, at 16:58, Andrew Gabriel wrote: > Rafael Friedlander wrote: >> Hi, >> >> Does anyone know since when hybrid pools are available in ZFS? Are >> there >> ZFS "versions"? >> >> XVM Server is based on Nevada b93, and I need to know if it supports >> hybrid pools. >> > > Hybrid pool slogs (ZIL) were introduced in Nevada builds 68 and 69, > and > is also in latest Solaris 10. > > I'm not sure when "cache" (L2ARC) devices were introduced. > There used to be a web page with a list of all the zpool versions, but > it's not where it was, and I can't find it now. Run "zpool upgrade -v" for a list of the versions known to that version of zpool. According to that, ZIL came in zfs version 7, and L2ARC came in zfs version 10. It also reports: --- For more information on a particular version, including supported releases, see: http://www.opensolaris.org/os/community/zfs/version/N Where 'N' is the version number. --- Cheers, Chris ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Hybrid Pools - Since when?
Andrew Gabriel wrote: > Rafael Friedlander wrote: > >> Hi, >> >> Does anyone know since when hybrid pools are available in ZFS? Are there >> ZFS "versions"? >> >> XVM Server is based on Nevada b93, and I need to know if it supports >> hybrid pools. >> >> > > Hybrid pool slogs (ZIL) were introduced in Nevada builds 68 and 69, and > is also in latest Solaris 10. > b68, June 2007. > I'm not sure when "cache" (L2ARC) devices were introduced. > There used to be a web page with a list of all the zpool versions, but > it's not where it was, and I can't find it now. > The flag days list is a good one to bookmark. http://opensolaris.org/os/community/on/flag-days/all/ L2ARC arrived in NV at the same time as ZFS boot, b79, November 2007. It was not back-ported to Solaris 10u6. -- richard ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS as a Gateway for a stroage network
I don't have any statistics myself, but if you search the forum for iscsi, there was a long thread a few months back with some performance figures. I didn't really do that much testing myself, once I hit the iscsi bug there wasn't any point doing much more. There has been some work on that recently though, and somebody here posted steps on how to compile the iscsi initiator in order to manually reduce the timeout which I plan to test as soon as I get enough free time at work. And no, you won't find my e-mail address in my profile, I try not to publish it to cut down on spam, but if you send a mail to my username at googlemail.com it'll get through. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpol mirror creation after non-mirrored zpool is setup
Jeff Bonwick wrote: > On Sat, Dec 13, 2008 at 04:44:10PM -0800, Mark Dornfeld wrote: >> I have installed Solaris 10 on a ZFS filesystem that is not mirrored. Since >> I have an identical disk in the machine, I'd like to add that disk to the >> existing pool as a mirror. Can this be done, and if so, how do I do it? > > Yes: > > # zpool attach > And if you want to be able to boot off of the newly attached replica you might want to install a boot block on it. See http://docs.sun.com/app/docs/doc/816-5166/installboot-1m # installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk \ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] zpol mirror creation after non-mirrored zpool is setup
Thanks everyone for your answers. I am ashamed to say that I have become lax in reading documentation due to the high quality of the forums. Of course, my question was easily answered both by reading docs and quick responses. I rarely see anyone telling a questioner to RTFM anymore since the forums are such an efficient format. I guess the new directive should be RTFF. On 14-Dec-08, at 2:09 PM, Bob Netherton wrote: Jeff Bonwick wrote: On Sat, Dec 13, 2008 at 04:44:10PM -0800, Mark Dornfeld wrote: I have installed Solaris 10 on a ZFS filesystem that is not mirrored. Since I have an identical disk in the machine, I'd like to add that disk to the existing pool as a mirror. Can this be done, and if so, how do I do it? Yes: # zpool attach And if you want to be able to boot off of the newly attached replica you might want to install a boot block on it. See http://docs.sun.com/app/docs/doc/816-5166/installboot-1m # installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk \ Mark T. Dornfeld, Cyantic Systems Corporation 203-2800 Skymark Avenue Mississauga, Ontario, L4W 5A6 CANADA Cell: (416) 616-6682 Main: (416) 621-6161 Fax: (416) 622-4130 Email: m...@cyantic.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS as a Gateway for a stroage network
You might want to look into the products from a company called DataCore Software, http://datacore.com/products/prod_home.asp. I've used them and they are great stuff. They make very high performing iSCSI and FC storage controllers out of leveraging commodity hardware, like the one comment of JBOD arrays earlier in this discussion. If you were to look at things like the Storage Performance Council, http://www.storageperformance.org/results/benchmark_results_spc1, or the VMTN, http://communities.vmware.com/thread/73745, you'll see they beat all the popular storage arrays on the market. Since they are block based storage virtualization devices, they work just fine with ZFS, UFS, any Open Systems FS / O.S. Their high availability is true H/A with two stacks of disk and automatic failover and failback, very cool stuff. Ross wrote: > I don't have any statistics myself, but if you search the forum for iscsi, > there was a long thread a few months back with some performance figures. I > didn't really do that much testing myself, once I hit the iscsi bug there > wasn't any point doing much more. > > There has been some work on that recently though, and somebody here posted > steps on how to compile the iscsi initiator in order to manually reduce the > timeout which I plan to test as soon as I get enough free time at work. > > And no, you won't find my e-mail address in my profile, I try not to publish > it to cut down on spam, but if you send a mail to my username at > googlemail.com it'll get through. > ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] UFS over zvol major performance hit
Hi, I have been doing some basic performance tests, and I am getting a big hit when I run UFS over a zvol, instead of directly using zfs. Any hints or explanations is very welcome. Here's the scenario. The machine has 30G RAM, and two IDE disks attached. The disks have 2 fdisk partitons (c4d0p2, c3d0p2) that are mirrored and form a zpool. When using filebench with 20G files writing directly on the zfs filesystem, I get the following results: RandomWrite-8k: 0.8M/s SingleStreamWriteDirect1m: 50M/s MultiStreamWrite1m: 51M/s MultiStreamWriteDirect1m: 50M/s Pretty consistent and lovely. The 50M/s rate sounds pretty reasonable, while the random 0.8M/s is a bit too low ? All in all, things look ok to me though here The second step, is to create a 100G zvol, format it with UFS, then bench that under same conditions. Note that this zvol lives on the exact same zpool used previously. I get the following: RandomWrite-8k: 0.9M/s SingleStreamWriteDirect1m: 5.8M/s (??) MultiStreamWrite1m: 33M/s MultiStreamWriteDirect1m: 11M/s Obviously, there's a major hit. Can someone please shed some light as to why this is happening ? If more info is required, I'd be happy to test some more ... This is all running on osol 2008.11 release. Note: I know ZFS autodisables disk-caches when running on partitions (is that slices, or fdisk partitions?!) Could this be causing what I'm seeing ? Thanks for the help Regards ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] UFS over zvol major performance hit
On Mon, 15 Dec 2008, Ahmed Kamal wrote: > > RandomWrite-8k: 0.9M/s > SingleStreamWriteDirect1m: 5.8M/s (??) > MultiStreamWrite1m: 33M/s > MultiStreamWriteDirect1m: 11M/s > > Obviously, there's a major hit. Can someone please shed some light as to why > this is happening ? If more info is required, I'd be happy to test some more > ... This is all running on osol 2008.11 release. What blocksize did you specify when creating the zvol? Perhaps UFS will perform best if the zvol blocksize is similar to the UFS blocksize. For example, try testing with a zvol blocksize of 8k. Bob == Bob Friesenhahn bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer,http://www.GraphicsMagick.org/ ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Need Help Invalidating Uberblock
I have a ZFS pool that has been corrupted. The pool contains a single device which was actually a file on UFS. The machine was accidentally halted and now the pool is corrupt. There are (of course) no backups and I've been asked to recover the pool. The system panics when trying to do anything with the pool. root@:/$ zpool status panic[cpu1]/thread=fe8000758c80: assertion failed: dmu_read(os, smo->smo_object, offset, size, entry_map) == 0 (0x5 == 0x0), file: ../../common/fs/zfs/space_map.c, line: 319 I've booted single user, moved /etc/zfs/zpool.cache out of the way, and now have access to the pool from the command line. However zdb fails with a similar assertion. r...@kestrel:/opt$ zdb -U -bcv zones Traversing all blocks to verify checksums and verify nothing leaked ... Assertion failed: dmu_read(os, smo->smo_object, offset, size, entry_map) == 0 (0x5 == 0x0), file ../../../uts/common/fs/zfs/space_map.c, line 319 Abort (core dumped) I've read Victor's suggestion to invalidate the active uberblock, forcing ZFS to use an older uberblock and thereby recovering the pool. However I don't know how to figure the offset to the uberblock. I have the following information from zdb. r...@kestrel:/opt$ zdb -U -uuuv zones Uberblock magic = 00bab10c version = 4 txg = 1504158 guid_sum = 10365405068077835008 timestamp = 1229142108 UTC = Sat Dec 13 15:21:48 2008 rootbp = [L0 DMU objset] 400L/200P DVA[0]=<0:52e3edc00:200> DVA[1]=<0:6f9c1d600:200> DVA[2]=<0:16e280400:200> fletcher4 lzjb LE contiguous birth=1504158 fill=172 cksum=b0a5275f3:474e0ed6469:e993ed9bee4d:205661fa1d4016 I've also checked the labels. r...@kestrel:/opt$ zdb -U -lv zpool.zones LABEL 0 version=4 name='zones' state=0 txg=4 pool_guid=17407806223688303760 top_guid=11404342918099082864 guid=11404342918099082864 vdev_tree type='file' id=0 guid=11404342918099082864 path='/opt/zpool.zones' metaslab_array=14 metaslab_shift=28 ashift=9 asize=42944954368 LABEL 1 version=4 name='zones' state=0 txg=4 pool_guid=17407806223688303760 top_guid=11404342918099082864 guid=11404342918099082864 vdev_tree type='file' id=0 guid=11404342918099082864 path='/opt/zpool.zones' metaslab_array=14 metaslab_shift=28 ashift=9 asize=42944954368 LABEL 2 version=4 name='zones' state=0 txg=4 pool_guid=17407806223688303760 top_guid=11404342918099082864 guid=11404342918099082864 vdev_tree type='file' id=0 guid=11404342918099082864 path='/opt/zpool.zones' metaslab_array=14 metaslab_shift=28 ashift=9 asize=42944954368 LABEL 3 version=4 name='zones' state=0 txg=4 pool_guid=17407806223688303760 top_guid=11404342918099082864 guid=11404342918099082864 vdev_tree type='file' id=0 guid=11404342918099082864 path='/opt/zpool.zones' metaslab_array=14 metaslab_shift=28 ashift=9 asize=42944954368 I'm hoping somebody here can give me direction on how to figure the active uberblock offset, and the dd parameters I'd need to intentionally corrupt the uberblock and force an earlier uberblock into service. The pool is currently on Solaris 05/08 however I'll transfer the pool to OpenSolaris if necessary. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] zfs source documentation
Is there a documentation available for zfs source code? -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
