Re: Scaling/Parallel patterns for a View executing complex database transaction

2015-06-16 Thread mesulphur
Exactly! We did prototype with Mongo and Riak to see if there were significant performance gains and the answer was no. Well tuned Postgres performed as well and as others. Add to this the complexities and risks associated with introducing a new component in your stack as against something stab

Re: Scaling/Parallel patterns for a View executing complex database transaction

2015-06-14 Thread Javier Guerra Giraldez
On Sun, Jun 14, 2015 at 3:13 PM, Peter of the Norse wrote: > Almost all of them are faster at key/value returns than PostgreSQL. i seriously doubt that. most "fast" key-value databases are only fast if you either: keep all data in RAM, forget about durability, or distribute the load on several

Re: Scaling/Parallel patterns for a View executing complex database transaction

2015-06-14 Thread Peter of the Norse
On May 14, 2015, at 10:30 PM, Me Sulphur wrote: > > * It is a large key-value pair table (approaching ~ 1 bn rows) with an MD5 > key and JSON for value. The look ups depend on the business logic but are > necessary. Nevertheless, there are no more than 10-12 queries executed by > combining ke

Re: Scaling/Parallel patterns for a View executing complex database transaction

2015-05-15 Thread mesulphur
Hi Stephen, We did try with Redis (instead of memcached) but the memory usage Redis is 2.5x-3x the volume of data, the economics of having entire data in memory then breaks down. Also because of the large degree of variation/random access patterns, there is little benefit in using an LRU cach

Re: Scaling/Parallel patterns for a View executing complex database transaction

2015-05-14 Thread Stephen J. Butler
If it's just key-value storage, adding a memcached layer sounds like a good thing to investigate. Do the tuples frequently change? On Thu, May 14, 2015 at 11:30 PM, Me Sulphur wrote: > Hi Russ, > > Thanks! While there are some pointers that we can pick up from your answer, > let me add some more

Re: Scaling/Parallel patterns for a View executing complex database transaction

2015-05-14 Thread Me Sulphur
Hi Russ, Thanks! While there are some pointers that we can pick up from your answer, let me add some more details. * It is a large key-value pair table (approaching ~ 1 bn rows) with an MD5 key and JSON for value. The look ups depend on the business logic but are necessary. Nevertheless, ther

Re: Scaling/Parallel patterns for a View executing complex database transaction

2015-05-14 Thread Russell Keith-Magee
On Thu, May 14, 2015 at 6:03 PM, Me Sulphur wrote: > Stack: Django 1.7 + Postgres 9.3 + Linux (No caching) > > Our application has a view which is called/executed very frequently. The > view receives, parses and responds a JSON. > > In between request and response, there are about 3-5 inserts and

Scaling/Parallel patterns for a View executing complex database transaction

2015-05-14 Thread Me Sulphur
Stack: Django 1.7 + Postgres 9.3 + Linux (No caching) Our application has a view which is called/executed very frequently. The view receives, parses and responds a JSON. In between request and response, there are about 3-5 inserts and around 1200-5000 look ups depending upon some if..else busines

Re: Database Transaction Migration

2013-01-24 Thread Sandeep kaur
> I haven't used South for complex data migrations, but I think it's > most useful just after (or simultaneously to) doing the schema > of course, if both old and new applications are written with Django > and handle data via the Django ORM, then you can do things like: > > > import oldapp.models

Database Transaction Migration

2013-01-24 Thread Sandeep kaur
-- Forwarded message -- From: Javier Guerra Giraldez Date: Wed, Jan 23, 2013 at 3:34 AM Subject: Re: Database Transaction To: django-users@googlegroups.com On Tue, Jan 22, 2013 at 11:54 AM, Sandeep kaur wrote: > Yes, I also thought south will be helpful but when we need to

Re: Database Transaction

2013-01-22 Thread Javier Guerra Giraldez
On Tue, Jan 22, 2013 at 11:54 AM, Sandeep kaur wrote: > Yes, I also thought south will be helpful but when we need to do > complete migration like old table is without foreign keys but new > table has foreign keys etc. then I could not understand how to use > south. Can you provide me some tutoria

Re: Database Transaction

2013-01-22 Thread Sandeep kaur
On Tue, Jan 22, 2013 at 7:32 PM, Javier Guerra Giraldez wrote: > On Mon, Jan 21, 2013 at 10:03 PM, Jian Chang wrote: >> you can't do the transaction between two different databases. > > i think the OP is talking about translation and not transaction. as > in translating the data from an old repr

Re: Database Transaction

2013-01-22 Thread Javier Guerra Giraldez
On Mon, Jan 21, 2013 at 10:03 PM, Jian Chang wrote: > you can't do the transaction between two different databases. i think the OP is talking about translation and not transaction. as in translating the data from an old representation to the new one. a better word might be migration. of course

Re: Database Transaction

2013-01-21 Thread Jian Chang
you mean two different databases? you can't do the transaction between two different databases. Anyway you can control the transactions by yourself. like: commit all or rollback all if all actions are ok or not, 2013/1/21 Sandeep kaur > Hello Django users, > I have created a django application w

Database Transaction

2013-01-21 Thread Sandeep kaur
Hello Django users, I have created a django application whose database is totally different(different foreign keys, fields, tables etc.) from that of old application. But now when I have created the application I require the data from old application too. How can I do the data transaction among the