Thanks again for the quick reply. As you said, I am wondering about apples and oranges primarily. The same EBS volume backed setup on Linux beat the OmniOS setup my more than 2x. That is more weird to me than normal. This I tested with simple RAID0 (40Gb * 5 EBS devices) on both the operating systems.
Also, I had followed the LevelDB tuning docs and set 50% of 8GB ram for LevelDB, and rest was supposed to be for os. But the ZFS ARC cache seems to have a mind of its own. This is unlike how LInux efficiently manages a disk cache automatically. Again quoting your words, ZFS do needs ram, but I find a lack of proper documentation on the ARC cache atleast so that I could tune it specifically for AWS. Finally, I would want my RAID setup to reduce the latency rather than to be fault tolerant, since I hope Riak will be available. Also if RAIDZ is going to have trade-offs of its own, I would not want that to add up to the unreliability of standard EBS volumes. As summarized, I would repeat my tests focusing on maybe PIOPS because price is also a concern. Ultimately, I just wish to avoid any common pitfalls or dead-ends. Regards, Hari John Kuriakose. On Wed, Jan 22, 2014 at 3:26 AM, Jason Campbell <xia...@xiaclo.net> wrote: > First off, make sure you are comparing apples to apples. I am assuming > the "default" RAID is RAIDZ on ZFS, so make sure the LVM raid is using > RAID5 for comparable performance. > > Generally, I wouldn't be using RAIDZ on AWS at all. The primary issue > with RAID5/RAIDZ is that the IO speeds are not only limited to that of a > single EBS volume, but that of the slowest EBS volume. That speed can vary > with which host you get assigned to, time of day, contention from other > customers, and seemingly the phase of the moon. I don't think anyone would > be surprised if you ran the same test again and got completely different > results. Provisioned IOPS disks will help even out extremely slow disks, > but you can still get quite a bit of variance. > > I would suggest moving to mirrored disks (RAID1) in both ZFS and Ubuntu. > I'm not sure about LVM, but ZFS will use the mirror to even out reads > (writes are harder) which should fix some of the high latency, even on > normal EBS volumes. I would suggest 4 disks in a RAID 10 configuration > (striping 2 mirrored pairs). Even better would be a 4-way mirror in ZFS if > price isn't much of a concern. This will limit you to the capacity of a > single EBS volume, but reads will use the fastest disk of the 4 disks, > instead of the slowest. It also has a nice side effect of being extremely > fault tolerant. > > The other thing to keep in mind is that ZFS is extremely RAM hungry and > read performance drops considerably when it is RAM starved, so I would > ensure that Riak doesn't use the last 1 - 1.5 GB of RAM so ZFS can use it > for caches. > > So in summary: > - Test, test and test again on different instances, this will help even > out EBS issues. > - Use mirrors, not RAID5/RAIDZ > - Use dedicated IOPS (at least for testing purposes) > - Ensure ZFS has some RAM to play with > > Hope this helps, > Jason Campbell > > ----- Original Message ----- > From: "Jared Morrow" <ja...@basho.com> > To: ejh...@gmail.com > Cc: "riak-users" <riak-users@lists.basho.com> > Sent: Wednesday, 22 January, 2014 7:51:19 AM > Subject: Re: Performance Tuning in OmniOS > > > > Oh I think OmniOS is far from hopeless. The problem you are having is the > same problem you'd have if you were on ubuntu and you made a LVM raid on > vanilla EBS. EBS is the problem when it comes to predictable write / read > speed. People still use it, but not without careful thought and > consideration. You can try using provisioned IOPS for EBS, which the > m1.large supports, or ask in risk-users what other AWS users have setup. I > know we have a lot of customers and OSS users running on AWS, so they are > far more knowledgeable about real-world performance than I am. > > > Good luck, > Jared > > > > > > > > > On Tue, Jan 21, 2014 at 12:05 PM, Hari John Kuriakose < ejh...@gmail.com> > wrote: > > > > > I am using the default raid itself. > > Well, if this is the case, I will run the tests again with a different > setup as you said, and get back as soon as possible. I would just like to > believe that OmniOS is not too hopeless. > > Thank you. > > > On Jan 21, 2014 11:17 PM, "Jared Morrow" < ja...@basho.com > wrote: > > > > What type of RAID did you chose for your spool of 5 volumes? If you chose > the default of raidz, you will not be getting much of a performance boost > over vanilla EBS, just a big integrity boost. Also, unless you are using > provisioned IOPS for EBS, you are starting from an extremely slow > base-case, so adding ZFS on top might not help matters much. > > > If speed is the concern, as a test I'm willing to bet if you do another > test run against the two instance storage disks on that m1.large, you will > probably beat those 5 EBS volumes pretty easily. > > > -Jared > > > > On Tue, Jan 21, 2014 at 9:22 AM, Hari John Kuriakose < ejh...@gmail.com > > wrote: > > > > Hello, > > > I am using standard EBS devices, with a zpool in an instance comprising of > five 40GB volumes. > Each of the Riak instance is of m1.large type. > > > I have made the following changes in zfs properties: > > > > # My reason: the default sst block size for leveldb is 4k. > zfs set recordsize=4k tank/riak > # My reason: by default, leveldb verifies checksums automatically. > zfs set checksum=off tank/riak > zfs set atime=off tank/riak > zfs set snapdir=visible tank/riak > > > And I did the following with help from Basho AWS tuning docs: > > > projadd -c "riak" -K "process.max-file-descriptor=(basic,65536,deny)" > user.riak > > bash -c "echo 'set rlim_fd_max=65536' >> /etc/system" > > > bash -c "echo 'set rlim_fd_cur=65536' >> /etc/system" > ndd -set /dev/tcp tcp_conn_req_max_q0 40000 > > > ndd -set /dev/tcp tcp_conn_req_max_q 4000 > ndd -set /dev/tcp tcp_tstamp_always 0 > ndd -set /dev/tcp tcp_sack_permitted 2 > ndd -set /dev/tcp tcp_wscale_always 1 > ndd -set /dev/tcp tcp_time_wait_interval 60000 > ndd -set /dev/tcp tcp_keepalive_interval 120000 > ndd -set /dev/tcp tcp_xmit_hiwat 2097152 > ndd -set /dev/tcp tcp_recv_hiwat 2097152 > ndd -set /dev/tcp tcp_max_buf 8388608 > > > Thanks again. > > > > > > On Tue, Jan 21, 2014 at 9:12 PM, Hector Castro < hec...@basho.com > wrote: > > > Hello, > > Can you please clarify what type of disk you are using within AWS? > EBS, EBS with PIOPS, instance storage? In addition, maybe some details > on volume sizes and instance types. > > These details may help someone attempting to answer your question. > > -- > Hector > > > > > On Tue, Jan 21, 2014 at 8:11 AM, Hari John Kuriakose < ejh...@gmail.com > > wrote: > > > > I am running LevelDB on ZFS in Solaris (OmniOS specifically) in Amazon > AWS. > > The iops is very very low. There is no significant progress with tuning > too. > > > > Why I chose ZFS is that since LevelDB requires the node to be stopped > before > > taking a backup, I needed a filesystem with snapshot ability. And the > most > > favourable Amazon community AMI seemed to be using OmniOS (fork of > Solaris). > > Everything is fine, except the performance. > > > > I did all the AWS tuning proposed by Basho but still Basho Bench gave > twice > > iops on Ubuntu as compared to OmniOS, under same conditions. Also, I am > > using riak-js client library, and its a 5 node Riak cluster with 8GB ram > > each. > > > > Could not yet figure out what is really causing the congestion in OmniOS. > > Any pointers will be really helpful. > > > > Thanks and regards, > > Hari John Kuriakose. > > > > > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
