Re: Handling increasingly-intensive processes

2014-12-16 Thread Sam Raker
Now that someone's said it, "just store tweets" seems like such a "duh" move. Thanks! -sam On Monday, December 15, 2014 6:35:13 AM UTC-5, Thomas Heller wrote: > > Hey, > > without knowing much about your application/business needs its hard to > speculate what might be good for you. The root of

Re: Handling increasingly-intensive processes

2014-12-15 Thread Thomas Heller
Hey, without knowing much about your application/business needs its hard to speculate what might be good for you. The root of your problem might be CouchDB since it was never meant for "Big Data" and since we are talking tweets I generally think "a lot". I'm not sure how your map value looks bu

Re: Handling increasingly-intensive processes

2014-12-14 Thread Ashton Kemerling
Sam, It sounds like you need to either find a caching strategy that works for your application's needs, or you'll need to adjust how your data is stored (model or data store). Without knowing more about your performance and business needs I can't really speculate with any confidence. -- Ashton

Re: Handling increasingly-intensive processes

2014-12-14 Thread Ashton Kemerling
Apologies on the email flood, my email client decided to do the most useless of all possible actions. On Sun, Dec 14, 2014 at 11:53 PM, Ashton Kemerling wrote: Honestly, it sounds like you'll either need to move the indexing into memor On Sun, Dec 14, 2014 at 8:54 PM, Sam Raker wrote: I'm

Handling increasingly-intensive processes

2014-12-14 Thread Sam Raker
I'm (still) pulling tweets from twitter, processing them, and storing them in CouchDB with hashtags as doc ids, such that if a tweet contains 3 hashtags, that tweet will be indexed under each of those 3 hashtags. My application hits CouchDB for the relevant document and uses Cheshire to convert