My data is around 1TB huge I don't think so much can be loaded into memory Plus my data is in parquet files. So loading it to spark and writing to Ignite is very time consuming.
Any thoughts on this please. Regards On Wed, 19 Jul, 2023, 2:14 pm Stephen Darlington, < stephen.darling...@gridgain.com> wrote: > A more common approach would be that Ignite has all your data and Spark > has a subset. Ignite SQL is generally faster than Spark SQL, since it can > use indexes, etc. But it’s not magic; it can’t query data it doesn’t know > about. > > On 19 Jul 2023, at 04:40, Arunima Barik <arunimabari...@gmail.com> wrote: > > How does write through work? I mean if I add a row in Ignite dataframe, > how does it reflect to spark? > > I have 50 rows in Ignite and all 100 rows in Spark. > If I perform a union all, wont the performance degrade? > I mean if will get slower than just querying spark > > On Tue, 18 Jul, 2023, 10:43 pm Stephen Darlington, < > stephen.darling...@gridgain.com> wrote: > >> Write through works regardless of how you insert data into Ignite. >> >> I’m not clear what you mean by federated query. Are the records in Spark >> a subset of those in the cache? >> >> Assuming not, create a data frame with a SQL query against Ignite. Create >> a data frame with a SQL query against your Spark data frame. Union together. >> >> > On 13 Jul 2023, at 08:27, Arunima Barik <arunimabari...@gmail.com> >> wrote: >> > >> > Hello Team >> > >> > I want to build a read write through Ignite cache over Spark >> > >> > If I have 50 rows in the cache and entire 100 row data in spark then >> how can I use federated queries? >> > >> > Also, how to perform write through using Sql queries? >> > >> > Regards >> > Arunima >> >> >
