Ignite is horizontally scalable. It can use as much memory as you have. 1Tb isn’t that much; I know of people with over 20Tb of data in memory. If you don’t want to keep everything in memory, you can use native persistence and keep less used data only on disk.
> On 19 Jul 2023, at 11:52, Arunima Barik <arunimabari...@gmail.com> wrote: > > My data is around 1TB huge > > I don't think so much can be loaded into memory > Plus my data is in parquet files. So loading it to spark and writing to > Ignite is very time consuming. > > Any thoughts on this please. > > Regards > > On Wed, 19 Jul, 2023, 2:14 pm Stephen Darlington, > <stephen.darling...@gridgain.com <mailto:stephen.darling...@gridgain.com>> > wrote: >> A more common approach would be that Ignite has all your data and Spark has >> a subset. Ignite SQL is generally faster than Spark SQL, since it can use >> indexes, etc. But it’s not magic; it can’t query data it doesn’t know about. >> >>> On 19 Jul 2023, at 04:40, Arunima Barik <arunimabari...@gmail.com >>> <mailto:arunimabari...@gmail.com>> wrote: >>> >>> How does write through work? I mean if I add a row in Ignite dataframe, how >>> does it reflect to spark? >>> >>> I have 50 rows in Ignite and all 100 rows in Spark. >>> If I perform a union all, wont the performance degrade? >>> I mean if will get slower than just querying spark >>> >>> On Tue, 18 Jul, 2023, 10:43 pm Stephen Darlington, >>> <stephen.darling...@gridgain.com <mailto:stephen.darling...@gridgain.com>> >>> wrote: >>>> Write through works regardless of how you insert data into Ignite. >>>> >>>> I’m not clear what you mean by federated query. Are the records in Spark a >>>> subset of those in the cache? >>>> >>>> Assuming not, create a data frame with a SQL query against Ignite. Create >>>> a data frame with a SQL query against your Spark data frame. Union >>>> together. >>>> >>>> > On 13 Jul 2023, at 08:27, Arunima Barik <arunimabari...@gmail.com >>>> > <mailto:arunimabari...@gmail.com>> wrote: >>>> > >>>> > Hello Team >>>> > >>>> > I want to build a read write through Ignite cache over Spark >>>> > >>>> > If I have 50 rows in the cache and entire 100 row data in spark then how >>>> > can I use federated queries? >>>> > >>>> > Also, how to perform write through using Sql queries? >>>> > >>>> > Regards >>>> > Arunima >>>> >>
