After looking at this again, I think the real problem is that fact that I'm using map reduce. The Riak java client does not support simple search but only search map reduce. At this point, I think the only good way to really use Riak search from Java is to use a solr client and point it to the Riak Solr interface. Any thoughts?
On Thu, Nov 21, 2013 at 2:08 PM, Roger Diller <ro...@flexrentalsolutions.com > wrote: > Ok, I'm not exactly sure what will be most helpful to you, but I am > attaching an example of a Key/Value. > > What you see in the file, we would have thousands of those. Probably would > get to at least a million or so KV's like that. > > The bucket is search enabled, so all those JSON fields are going to be > indexed. > > However, we are primarily search filtering on the systemId, indexId, & > fullText fields in the value. > > We currently have a 6 node cluster with each server having approx 16 cores > & in most cases at least 10 GB (20 GB free on most servers probably) free > memory with all the process running (including Riak). Computing power > should not be an issue period! > > Let me know what use that would be useful to provide. > > > > On Wed, Nov 20, 2013 at 12:46 PM, Todd Tyree <t...@basho.com> wrote: > >> Hi Roger, >> >> Sorry, meant to reply to the mailing list, but accidentally replied >> directly to you. >> >> Before I can say whether or not secondary indexes are suitable, I need to >> know more about your data, access patterns and query patterns. >> >> Can you share this information with me here? What kind of data is being >> searched and how frequently is it updated? >> >> Best, >> Todd >> >> >> On Wed, Nov 20, 2013 at 5:05 PM, Roger Diller < >> ro...@flexrentalsolutions.com> wrote: >> >>> Do you have any other suggestions on how we can find data in real time >>> from a bucket (without moving to 2.0)? What about secondary indexes? >>> >>> >>> On Wed, Nov 20, 2013 at 11:59 AM, Todd Tyree <t...@basho.com> wrote: >>> >>>> Hi Roger, >>>> >>>> You are essentially correct in that map reduce was never designed as a >>>> realtime query tool. >>>> >>>> However, we do have a solution in a technology preview release stage >>>> that may solve this problem for you: Yokozuna [0]. It is a tight >>>> integration of Riak and Solr and brings the best of both technologies >>>> together. >>>> >>>> It is currently scheduled for release as part of Riak 2.0, but you can >>>> clone the repo and build it now if you would like. Just be aware it is >>>> still undergoing development and the API may be subject to change before >>>> the final release. >>>> >>>> [0] https://github.com/basho/yokozuna >>>> >>>> Best, >>>> Todd >>>> >>> >> >> On Wed, Nov 20, 2013 at 4:45 PM, Roger Diller < >> ro...@flexrentalsolutions.com> wrote: >> >>> I could dig up all our nitty gritty Riak details but I don't think that >>> will help really. >>> >>> The point I think is this: Using search map reduce is not a viable way >>> to do real time search queries. Especially ones that may have 2000+ plus >>> results each. Couple that with search requests coming in every few seconds >>> from 300+ customer app instances and you literally bring Riak to it's >>> knees. >>> >>> Not that Riak is the problem really, it's just we are using it in a way >>> it was not designed for. In essence, we are using Riak as a search engine >>> for our application data. Correct me if I'm wrong but Riak is more for >>> storing large amounts of KV data, but not really for finding that data in a >>> search sense. >>> >>> Am I missing something here? Is there a viable way for doing real time >>> search queries on a bucket with 1 million keys? >>> >>> >>> On Mon, Nov 18, 2013 at 5:29 PM, Alexander Sicular >>> <sicul...@gmail.com>wrote: >>> >>>> More info please... >>>> >>>> Version >>>> Current config >>>> Hardware >>>> Data size >>>> Search Schema >>>> Etc. >>>> >>>> But I would probably say that your search is returning too many keys to >>>> your mr. More inline. >>>> >>>> @siculars >>>> http://siculars.posthaven.com >>>> >>>> Sent from my iRotaryPhone >>>> >>>> On Nov 18, 2013, at 13:59, Roger Diller <ro...@flexrentalsolutions.com> >>>> wrote: >>>> >>>> Using the Riak Java client, I am executing a search map reduce like >>>> this: >>>> >>>> MapReduceResult result = riakClient.mapReduce(SEARCH_BUCKET, >>>> search).execute(); >>>> >>>> >>>> ^is this part a typo. Cause otherwise it looks like you do a s>mr, set >>>> the search and then another s>mr. >>>> >>>> >>>> String search = "systemId:" + systemName + " AND indexId:" + indexId; >>>> >>>> MapReduceResult result = riakClient.mapReduce(SEARCH_BUCKET, >>>> search).execute(); >>>> >>>> This worked fine when the bucket contained a few thousand keys. Now >>>> that we have far more data stored in the bucket (at least 250K keys), it's >>>> throwing this generic error: >>>> >>>> com.basho.riak.client.RiakException: java.io.IOException: >>>> {"error":"map_reduce_error"} >>>> >>>> We've also noticed that storing new key/values in the bucket has slowed >>>> WAY down. >>>> >>>> Any idea what's going on? >>>> >>>> >>>> Your data set is incorrectly sized to your production config. >>>> >>>> Are there limitations to Search Map Reduce? >>>> >>>> >>>> Certainly >>>> >>>> Are there configuration options that need changed? >>>> >>>> >>>> Possibly >>>> >>>> Any help would be greatly appreciated. >>>> >>>> >>>> -- >>>> Roger Diller >>>> Flex Rental Solutions, LLC >>>> Email: ro...@flexrentalsolutions.com >>>> Skype: rogerdiller >>>> Time Zone: Eastern Time >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>> >>> >>> -- >>> Roger Diller >>> Flex Rental Solutions, LLC >>> Email: ro...@flexrentalsolutions.com >>> Skype: rogerdiller >>> Time Zone: Eastern Time >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> >> >> -- >> *Todd Tyree* >> Client Services Engineer Basho <http://www.basho.com/> >> >> mobile: +44(0)7861 220 182 >> web: www.basho.com >> github: tatyree <http://github.com/tatyree> >> > > > > -- > Roger Diller > Flex Rental Solutions, LLC > Email: ro...@flexrentalsolutions.com > Skype: rogerdiller > Time Zone: Eastern Time > -- Roger Diller Flex Rental Solutions, LLC Email: ro...@flexrentalsolutions.com Skype: rogerdiller Time Zone: Eastern Time
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
