Intermittent ArrayIndexOutOfBoundsException on Hive Merge

2020-07-14 Thread Bernard Quizon
Hi. I'm using Hive 3.1.0 (Tez Execution Engine) and I'm running into intermittent errors when doing Hive Merge. Just to clarify, the Hive Merge query probably succeeds 60% of the time using the same source and destination table for the Hive Merge query. By the way, both the source and destinatio

Re: Intermittent ArrayIndexOutOfBoundsException on Hive Merge

2020-07-14 Thread Bernard Quizon
Hi. I see that this piece of code is the source of the error: final int maxSize = (vectorizedTestingReducerBatchSize > 0 ? Math.min(vectorizedTestingReducerBatchSize, batch.getMaxSize()) : batch.getMaxSize()); Preconditions.checkState(maxSize > 0); int rowIdx = 0; int batchByt

RE: Intermittent ArrayIndexOutOfBoundsException on Hive Merge

2020-07-14 Thread Aaron Grubb
This is just a suggestion but I recently ran into an issue with vectorized query execution and a map column type, specifically when inserting into an HBase table with a map to column family setup. Try using “set hive.vectorized.execution.enabled=false;” Thanks, Aaron From: Bernard Quizon Sen

Re: Hive metastore

2020-07-14 Thread Eric Pogash
Ping on this, does anyone know of a health endpoint? Eric On Wed, Jul 8, 2020 at 3:04 PM Eric Pogash wrote: > Hello, > > I'm looking to establish a readiness and liveness probe in kubernetes > where we are hosting a hive standalone metastore. Is there an http health > endpoint available for th

Re: Hive metastore

2020-07-14 Thread Sungwoo Park
Hello, We use just TCP readiness/liveness probes checking the Metastore listener port (specified by hive.metastore.port or metastore.thrift.port). I don't know if an HTTP endpoint is available for Metastore. readinessProbe: tcpSocket: port: 9083 initialDela

Re: Hive metastore

2020-07-14 Thread Kwanghee Park
First of all, I examined that why you want to check the health of Hive Meta Store Process 먼저 왜 하이브 메타스토어 프로세스의 헬스체크를 하려고 하는지 생각해봤음 As far I know Hive Meta Store(HMS) is stateless architecture(it writes and reads meta information from database like mysql) 내가 알기론 하이브메타스토어는 Stateless 구조야 (HMS는 MySQL과

Re: Hive metastore

2020-07-14 Thread Kwanghee Park
OMG, it's my mistake! I apologize to all who read my ugly email. First of all, I examined that why you want to check the health of Hive MetaStore Process As far as I know Hive MetaStore(HMS) is stateless architecture (it writes and reads meta information from DB like MySQL). Almost all guys wil