Agreed But the servers do not have so much of capacity. Hence only in memory data grid works for us.
On Wed, 19 Jul, 2023, 6:35 pm Stephen Darlington, < stephen.darling...@gridgain.com> wrote: > Ignite is horizontally scalable. It can use as much memory as you have. > 1Tb isn’t *that* much; I know of people with over 20Tb of data in memory. > If you don’t want to keep everything in memory, you can use native > persistence and keep less used data only on disk. > > On 19 Jul 2023, at 11:52, Arunima Barik <arunimabari...@gmail.com> wrote: > > My data is around 1TB huge > > I don't think so much can be loaded into memory > Plus my data is in parquet files. So loading it to spark and writing to > Ignite is very time consuming. > > Any thoughts on this please. > > Regards > > On Wed, 19 Jul, 2023, 2:14 pm Stephen Darlington, < > stephen.darling...@gridgain.com> wrote: > >> A more common approach would be that Ignite has all your data and Spark >> has a subset. Ignite SQL is generally faster than Spark SQL, since it can >> use indexes, etc. But it’s not magic; it can’t query data it doesn’t know >> about. >> >> On 19 Jul 2023, at 04:40, Arunima Barik <arunimabari...@gmail.com> wrote: >> >> How does write through work? I mean if I add a row in Ignite dataframe, >> how does it reflect to spark? >> >> I have 50 rows in Ignite and all 100 rows in Spark. >> If I perform a union all, wont the performance degrade? >> I mean if will get slower than just querying spark >> >> On Tue, 18 Jul, 2023, 10:43 pm Stephen Darlington, < >> stephen.darling...@gridgain.com> wrote: >> >>> Write through works regardless of how you insert data into Ignite. >>> >>> I’m not clear what you mean by federated query. Are the records in Spark >>> a subset of those in the cache? >>> >>> Assuming not, create a data frame with a SQL query against Ignite. >>> Create a data frame with a SQL query against your Spark data frame. Union >>> together. >>> >>> > On 13 Jul 2023, at 08:27, Arunima Barik <arunimabari...@gmail.com> >>> wrote: >>> > >>> > Hello Team >>> > >>> > I want to build a read write through Ignite cache over Spark >>> > >>> > If I have 50 rows in the cache and entire 100 row data in spark then >>> how can I use federated queries? >>> > >>> > Also, how to perform write through using Sql queries? >>> > >>> > Regards >>> > Arunima >>> >>> >> >
