It is important to consider potential impacts on Spark tables stored in the Hive metastore during an "upgrade". Depending on the upgrade path, the Hive metastore schema or SerDes behavior might change, requiring adjustments in the Sparkark code or configurations. I mentioned the need to test the Spark applications thoroughly after a Hive upgrade, which will necessitates liaising with Hive group as your are relying on their metdadata
Mich Talebzadeh, Technologist | Architect | Data Engineer | Generative AI | FinCrime London United Kingdom view my Linkedin profile https://en.everybodywiki.com/Mich_Talebzadeh Disclaimer: The information provided is correct to the best of my knowledge but of course cannot be guaranteed . It is essential to note that, as with any advice, quote "one test result is worth one-thousand expert opinions (Werner Von Braun)". On Wed, 1 May 2024 at 04:30, Wenchen Fan <cloud0...@gmail.com> wrote: > > Yes, Spark has a shim layer to support all Hive versions. It shouldn't be an > issue as many users create native Spark data source tables already today, by > explicitly putting the `USING` clause in the CREATE TABLE statement. > > On Wed, May 1, 2024 at 12:56 AM Mich Talebzadeh <mich.talebza...@gmail.com> > wrote: >> >> @Wenchen Fan Got your explanation, thanks! >> >> My understanding is that even if we create Spark tables using Spark's >> native data sources, by default, the metadata about these tables will >> be stored in the Hive metastore. As a consequence, a Hive upgrade can >> potentially affect Spark tables. For example, depending on the >> severity of the changes, the Hive metastore schema might change, which >> could require Spark code to be updated to handle these changes in how >> table metadata is represented. Is this assertion correct? >> >> Thanks >> >> Mich Talebzadeh, >> >> Technologist | Architect | Data Engineer | Generative AI | FinCrime >> >> London >> United Kingdom >> >> >> view my Linkedin profile >> >> >> https://en.everybodywiki.com/Mich_Talebzadeh >> >> >> >> Disclaimer: The information provided is correct to the best of my >> knowledge but of course cannot be guaranteed . It is essential to note >> that, as with any advice, quote "one test result is worth one-thousand >> expert opinions (Werner Von Braun)". --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
