*Jason, Mike*: first, thank you. > *In order to target the nodes that would supposedly hold the data of interest you need to know the keys you are looking for. If you know the keys why are you querying in the first place? Just do getAll(keys).*
Two reasons... 1. I want to apply some "additional filtering" that can only be handled elegantly by a OQL query predicate after a subset of the data has been identified/targeted (using keys). I have example of this somewhere (doh) after working with a customer on this exact UC 2. I don't want the entire object (i.e. row); I only need a specific "projection" of the (object) data. This is particularly important if I have very large and complex object graph and I am streaming data across the wire (client/server). > *The trouble happens when you are NOT hitting your indices. * Yes, good point. > *If you do a query that requires a full table scan, then every row in the database table needs to be examined, and to examine it, it has to be in memory at least briefly.* Of course. *Denis*- > *The disk entries that are mentioned by John were located in memory before and were overflowed on disk at some point of time. It means that if you start your cluster from scratch and want to run OQL queries over the indexed data then you have to preload all the data from the persistence.* I don't specifically recall how much persistent data Geode reloads on restart (Geode is a shared-nothing architecture though so each data node has it's own persistence; additionally primaries must come online before secondaries are accessible). The question is how much data gets reloaded on restart. It would seem silly if the disk store contained more data then would fit in memory and reload everything knowing some of the data would be OVERFLOW on preload when it would not all fit. Geode will reload the Index though, which is stored as well. I let the experts answer this one. On Fri, Aug 19, 2016 at 2:04 PM, Denis Magda <magda7...@gmail.com> wrote: > Hi John, Jason, > > If to expand more on this > > > *If an index can be used, the index look up is executed and entries added > to the result set. If any of the entries that match the predicates is > actually on disk, those values will need to be loaded to memory before > being returned as a result.* > > The disk entries that are mentioned by John were located in memory before > and were overflowed on disk at some point of time. It means that if you > start your cluster from scratch and want to run OQL queries over the > indexed data then you have to preload all the data from the persistence. > Yes, some of the data may be overflowed back to disk during the preloading > but you'll have your indexes in a valid state. > > Correct me if I'm still missing something. > > -- > Denis > > > On Fri, Aug 19, 2016 at 1:52 PM, Jason Huynh <jhu...@pivotal.io> wrote: > >> Hi John, >> >> I think you were referring to Mike's explanation of: >> "If, however, you ever resort to hitting the disk-based data for a query >> it is going to have to read every record that isn't in memory from disk >> which is going to be extremely slow. I personally would never use Geode >> that way." >> >> When stating: >> "Additionally, assuming the Indexes were defined properly based on the >> predicates in the queries (most often) used, that it would target the data >> on disk matching the predicate and load only the data required (no data >> store, RDBMS or otherwise, especially disk-bound stores, should have to >> load the entire table/Region/Map/whatever to access the data matching the >> predicate; that's absurd, OOMEs galore)." >> >> Let me try to clear things up slightly...hopefully not causing more >> confusion... >> If an index can be used, the index look up is executed and entries added >> to the result set. If any of the entries that match the predicates is >> actually on disk, those values will need to be loaded to memory before >> being returned as a result. >> I think what Mike was saying was that if an index is not used, then the >> query itself would execute across the entire region, which means loading >> every entry into memory. We would need to inspect each entry to see if >> fulfill the criteria. >> >> -Jason >> >> >> >> On Fri, Aug 19, 2016 at 1:39 PM John Blum <jb...@pivotal.io> wrote: >> >>> Hi All- >>> >>> DISCLAIMER: I am no expert in querying and index >>> architecture/implementation; mostly a consumer. >>> >>> Perhaps *Anil* or *Jason* can shed more light on the subject, but for >>> my own understanding/sanity, it would seem we could do better than this, >>> meaning... >>> >>> I would think any UC partially depends on the organization of your data >>> in the grid as well. If you used a PARTITION data management policy >>> [1], for instance, then, of course, your data would be distributed and >>> partitioned across all the data nodes in the grid (cluster) holding the >>> data (i.e. data nodes that have declared the same PARTITION Region). >>> It should then be possible to make this more optimal by have a redundancy >>> level of 1 or more (depending on the frequency of transactions and data >>> changes) to parallelize the data access. >>> >>> Not only does having more nodes mean better (or more optimal) >>> organization, but more memory. Still, given a very large data set, clearly >>> some of the data will need to OVERFLOW (to disk). >>> >>> But, by combining the Function Execution service with querying (on >>> PARTITIONED data) [2], you could target the nodes that would supposedly >>> hold the data of interests, and execute the queries there. >>> >>> Additionally, assuming the Indexes were defined properly based on the >>> predicates in the queries (most often) used, that it would target the data >>> on disk matching the predicate and load only the data required (no data >>> store, RDBMS or otherwise, especially disk-bound stores, should have to >>> load the entire table/Region/Map/whatever to access the data matching the >>> predicate; that's absurd, OOMEs galore). >>> >>> TMK, Geode keeps Indexes in memory (even loads them on startup) and >>> updates them (either sync/async depending on your configuration) as the >>> data changes. You would assume the data would not be changing in the >>> OVERFLOW, disk-based data set. If the data did change, then wouldn't you >>> also assume that that data would then have to be in-memory (I think so). >>> >>> Please let me know if I am way of basis here, but I would think Geode >>> gives you enough options that particular UCs could be made, with nominal >>> effort, more optimal. >>> >>> Additional references... >>> >>> * Query Partitioned Regions [3] >>> * Working with Indexes [4], and then... >>> * Tips and Guidelines on Using Indexes [5], but also important... >>> * Using Indexes with Overflow Regions [6] >>> >>> Hope this helps. >>> >>> Cheers! >>> -John >>> >>> >>> [1] http://geode.docs.pivotal.io/docs/developing/region_opti >>> ons/region_types.html >>> [2] http://geode.docs.pivotal.io/docs/developing/querying_ba >>> sics/performance_considerations.html >>> [3] http://geode.docs.pivotal.io/docs/developing/querying_ba >>> sics/querying_partitioned_regions.html >>> [4] http://geode.docs.pivotal.io/docs/developing/query_index >>> /query_index.html >>> [5] http://geode.docs.pivotal.io/docs/developing/query_index >>> /indexing_guidelines.html >>> [6] http://geode.docs.pivotal.io/docs/developing/query_index >>> /indexes_with_overflow_regions.html >>> >>> >>> On Fri, Aug 19, 2016 at 12:55 PM, Denis Magda <magda7...@gmail.com> >>> wrote: >>> >>>> Thanks, now I see. >>>> >>>> This works the same way as in Ignite then. If you set up an eviction >>>> policy in Ignite the data may be evicted to swap at some point of time and >>>> if a query is executed right after that the it may swap in the data back to >>>> memory. However the indexes must always be in memory. >>>> >>>> -- >>>> Denis >>>> >>>> >>>> On Fri, Aug 19, 2016 at 12:43 PM, Michael Stolz <mst...@pivotal.io> >>>> wrote: >>>> >>>>> There is a notion of data aging out in Geode. We call it overflow to >>>>> disk. >>>>> >>>>> The idea is that as data gets old you can have the records in memory >>>>> expire, and that expiry can be to disk. That's the cold data. >>>>> >>>>> You may have built an index while you were initially loading the data, >>>>> and if your predicates only hit the indexes you will still get really fast >>>>> queries if the result sets aren't large. >>>>> >>>>> If, however, you ever resort to hitting the disk-based data for a >>>>> query it is going to have to read every record that isn't in memory from >>>>> disk which is going to be extremely slow. I personally would never use >>>>> Geode that way. >>>>> >>>>> >>>>> -- >>>>> Mike Stolz >>>>> Principal Engineer, GemFire Product Manager >>>>> Mobile: 631-835-4771 >>>>> >>>>> On Fri, Aug 19, 2016 at 3:35 PM, Denis Magda <magda7...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi Mike, >>>>>> >>>>>> Thanks a lot for the explanation! It makes perfect sense to me. >>>>>> >>>>>> I just thought that you were able to do something with indexes in a >>>>>> such way that there is no need to preload everything from disk into >>>>>> memory >>>>>> when a query is executed over cold data. >>>>>> >>>>>> Then what does "execution over cold data" mean? I'm referring to the >>>>>> following sentence from the main page: >>>>>> >>>>>> *Object Query Language allows distributed query execution on hot and >>>>>> cold data, with SQL-like capabilities, including joins.* >>>>>> >>>>>> -- >>>>>> Denis >>>>>> >>>>>> >>>>>> On Fri, Aug 19, 2016 at 12:27 PM, Michael Stolz <mst...@pivotal.io> >>>>>> wrote: >>>>>> >>>>>>> Here's the thing... >>>>>>> >>>>>>> On any In-memory data grid, if you run a query before the data has >>>>>>> been loaded into memory, it is going to cause the exact same amount of >>>>>>> disk >>>>>>> i/o to do the query as it will take to load everything into memory. >>>>>>> >>>>>>> And the system will still have to go ahead and load everything into >>>>>>> memory anyway so you're going to end up doing all that disk i/o TWICE. >>>>>>> >>>>>>> Geode DOES have a nice feature for key based access though. We >>>>>>> actually store the keys in a separate file from the data and we can load >>>>>>> that file very quickly. Then if you go after the data for one of those >>>>>>> keys >>>>>>> we can lazily load it from disk on demand if it hasn't yet been loaded >>>>>>> into >>>>>>> memory. >>>>>>> >>>>>>> The Lucene integration work that is going on in Geode might also >>>>>>> make it possible to load the indexes first and lazily load the data >>>>>>> based >>>>>>> on queries against the indexes. >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Mike Stolz >>>>>>> Principal Engineer, GemFire Product Manager >>>>>>> Mobile: 631-835-4771 >>>>>>> >>>>>>> On Fri, Aug 19, 2016 at 2:59 PM, Denis Magda <magda7...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hello Geode community, >>>>>>>> >>>>>>>> I've been investigating possibilities of Geode Persistence for a >>>>>>>> while and still can't get it clear whether I need to have all my data >>>>>>>> in >>>>>>>> memory if I want to execute OQL queries or OQL engine works over the >>>>>>>> persistence as well. >>>>>>>> >>>>>>>> My use case is the following. During the cluster startup I don't >>>>>>>> want to wait while all the data has been pre-loaded from the >>>>>>>> persistence to >>>>>>>> RAM and want to execute OQL queries right away. Is it feasible to >>>>>>>> implement >>>>>>>> with Geode? Please provide me with the links where I can read more >>>>>>>> about >>>>>>>> this. >>>>>>>> >>>>>>>> Regards, >>>>>>>> Denis >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Удачи, >>>>>> Денис Магда >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Удачи, >>>> Денис Магда >>>> >>> >>> >>> >>> -- >>> -John >>> 503-504-8657 >>> john.blum10101 (skype) >>> >> > > > -- > Удачи, > Денис Магда > -- -John 503-504-8657 john.blum10101 (skype)
