Re: How useful are tools for Hive data modeling

2020-11-11 Thread Austin Hackett
Hi Mich Hive also has non-validated primary key, foreign key etc constraints. Whilst I’m not too familiar with the modelling tools you mention, perhaps they’re able to use these for generating SQL etc? ORC files have indexes (min, max, bloom filters) - not particularly relevant to the data mod

Re: How useful are tools for Hive data modeling

2020-11-11 Thread Austin Hackett
r any loss, > damage or destruction of data or any other property which may arise from > relying on this email's technical content is explicitly disclaimed. The > author will in no case be liable for any monetary damages arising from such > loss, damage or destruction. >

Re: Is Insert Overwrite table partition on s3 is an atomic operation ?

2021-01-11 Thread Austin Hackett
Hi Mark It’s my understanding that when you do an INSERT OVERWRITE into a partition, Hive will take out an exclusive lock on the partition and a shared lock on the table itself. This blocks are read and write operations on the partition, and allows reads against the other partitions to proceed.

Hive 3.1.3 Hadoop Compatability

2023-12-21 Thread Austin Hackett
Hi List I was hoping that someone might be able to clarify which Hadoop versions Hive 3.1.3 is compatible with? https://hive.apache.org/general/downloads/ says that Hive release 3.1.3 works with Hadoop 3.x.y which is straightforward enough. However, I notice the 4.0.0 releases only work with H

Re: Hive 3.1.3 Hadoop Compatability

2023-12-21 Thread Austin Hackett
; Hive 3.1.3 & 4.0.0-alpha-1 works with Hadoop-3.1.0 > > HIve 4.0.0-alpha-2 & 4.0.0-beta-1 works with Hadoop-3.3.1 > > The upcoming Hive 4.0 GA release would be compatible with Hadoop-3.3.6 > > -Ayush > > On Thu, 21 Dec 2023 at 17:39, Austin Hackett wrote: >>

Re: Hive 3.1.3 Hadoop Compatability

2023-12-22 Thread Austin Hackett
particular hive release. > > -Ayush > >> On Fri, 22 Dec 2023 at 03:03, Austin Hackett wrote: >> >> Hi Ayush >> >> Many thanks for your response. >> >> I’d really appreciate a clarification if that’s OK? >> >> Does this just mea

"org.apache.thrift.transport.TTransportException: Invalid status -128" errors when SASL is enabled

2024-01-08 Thread Austin Hackett
Hi List I'm having an issue where Hive Metastore operations (e.g. show databases) are failing with "org.apache.thrift.transport.TTransportException: Invalid status -128" errors when I enable SASL. I am a bit stuck on how to go about troubleshooting this further, and any pointers would be gre

Re: "org.apache.thrift.transport.TTransportException: Invalid status -128" errors when SASL is enabled

2024-01-11 Thread Austin Hackett
For the benefit of anyone who comes across this error in future, it was solved by adding hive.metastore.sasl.enabled and hive.metastore.kerberos.principal to hive-site.xml on the client side, e.g. $SPARK_HOME/conf > On 8 Jan 2024, at 16:18, Austin Hackett wrote: > > Hi List >