Hi Ryan I'm running the cluster on 4 physical servers with many cores each so CPU is not a limitation. Regarding the storage, we use Sun Openstorage which essential means, that we are writing to a flashdisk. Besides, I tried to use memorybased filesystem for storage with non significant changes in performance (50% better or so), so I doubt, that IO is my challenge.
But I have noted the ZFS tuning recommendations - that will come in handy when we go in larger roll out. But - David Dawson pointed me in the right direction - my simple converter script did a "fetch record, write record, repeat" lifecycle and the bare roundtrip time was a killer. I have not extended my little script to become multithreaded (each write to the cluster happens in a dedicated thread) and that very simple enhancement now gave me more then 200 writes/sec, so I believe that the bottleneck is in my script and not the RIAK cluster right now. I have run out of time for today, but I will continue my experiments tomorrow and finetune my script more (read: make it more parallel and perhaps distribute it to more hosts). Best regards, Karsten On May 10, 2010, at 23:46 , Ryan Tilder wrote: > A couple of quick questions for you Karsten that should help us get an idea > of what kind of issues you might be having. > > How many physical hosts are you running the four OpenSolaris virtuals on? If > they're all running on the same host and you don't have a pretty substantial > RAID array backing their local storage, you're just going to get I/O > contention between the virtuals, slowing down writes. > > There are some ZFS tuning parameters we've found that can improve write > throughput. Since you're using dets there's one in particular that will be > helpful. You can run this command as root on each OpenSolaris virtual: > > zfs atime=off <pool> > > The fact that you can essentially double your performance by running another > client in parallel does make me wonder whether or not it might be a mild > performance issue with your invocation of the ripple client. Do you see a > linear increase in write performance as you increase the number of parallel > writers? > > --Ryan > > On Mon, May 10, 2010 at 8:36 AM, Karsten Thygesen <kar...@netic.dk> wrote: > Hi > > I'm doing a small proof-of-concept and the goal is to store about 250.000.000 > records in a Riak cluster. Today, we have the data in MySQL, but we strive > for better performance and we might even expect up to 5 times as mush data > during the next couple of years. The data is denormalized and "document" like > so they are an easy match for NoSQL paradigm. > > For the small POC, I have built a 4 node cluster with 4 dedicated virtual > servers running Opensolaris on top of VMWare but with quite fast storage > below. In fron of the cluster I have a loadbalancer which will distribute > reuests evenly among the nodes. > > Each node is running riak-0.10 with almost deafult configuration. I have > added "-smp enabled" to vm.args and each node is otherwise using default > configuration (except for name of cause). This also implies N=2 and dest for > storage backend. > > I have written a small ruby script which uses riak-client from Ripple (latest > version) as well as curd for http connections and it quite simple takes each > record from the database and stores is in riak. Each record is around > 500-1000 bytes large and entirely structured text/data. I store them as JSON > objects. > > The script can easily read more than 15.000 records/second, process them and > print them to the screen, so I doubt the script is the bottleneck. > > When I try to write them to the riak cluster via the loadbalancer, I can only > write around 50-60 records/second and while writing, the beam process is only > using around 10% cpu and no major IO activity is going on. > > I have tried to move the data directory to /tmp (memory filesystem) and with > this setup, I can get around 90 write/sec (yes - only for testing - I can not > live with memoryfilesystem in production with this dataset). > > I have also noticed, that the performance I get is almost equivalent nomatter > if I write through the loadbalancer or I just select a node and sends all my > writes to that one. > > I have also tried a "multithreaded" approach where I simply run two of my > datamover scripts in parallel, and that way, I can get around 110 > writes/second. > > With the current performance, it will take me more than a month to move my > data from mysql to Riak, so I need a multitude of better performance. > > Do you have any suggestions for how to get better performance? I was hoping > for towards 1000 writes/second so feel free to speculate - perhaps I should > just add quite a bunch of more servers? > > Best regards, > Karsten > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
