Re: Hive on Tez Counters

2014-08-25 Thread Siddharth Seth
gleitner < > ghagleit...@hortonworks.com> wrote: > > > Hive logs the same counters regardless of whether you run with Tez or MR. > > We've removed some counters in hive 0.13 (HIVE-4518) - the specific one > > you're looking for might be in that list. > >

Re: Hive on Tez Counters

2014-08-21 Thread Siddharth Seth
I'll let Hive folks answer the questions about the Hive counters. In terms of the CPU counter - that was a bug in Tez-0.4.0, which has been fixed in 0.5.0. COMMITTED_HEAP_BYTES just represents the memory available to the JVM (Runtime.getRuntime().totalMemory()). This will only vary if the VM is s

Re: [VOTE] Stricter commit guidelines

2018-05-15 Thread Siddharth Seth
+1 On Mon, May 14, 2018 at 10:44 PM, Jesus Camacho Rodriguez < jcama...@apache.org> wrote: > After work has been done to ignore most of the tests that were failing > consistently/intermittently [1], I wanted to start this vote to gather > support from the community to be stricter wrt committing p

Re: [DISCUSS] Unsustainable situation with ptests

2018-05-15 Thread Siddharth Seth
Very nice. There was an effort to get fast and green builds back in 2016. There wasn't any strict "must be a green build" before commit at the time though. Instead jiras were filed and the expectation was that they'd be cited / new ones created pre commit(looking at the jiras now - this was likely

Re: Invitation for Hive committers to become ORC committers

2017-01-04 Thread Siddharth Seth
Hi Owen, I'd be interested as well, if not too late. Thanks, Sid On Wed, Jan 4, 2017 at 10:34 AM, Owen O'Malley wrote: > Ferd, I've added you. > > Suneel, I'm sorry, but the offer is limited to current Hive committers. > http://people.apache.org/phonebook.html?unix=hive > > .. Owen > > On Mon, J

Re: [VOTE] Drop support for Java7 in master branch

2017-02-28 Thread Siddharth Seth
+1 On Mon, Feb 27, 2017 at 8:54 PM, Thejas Nair wrote: > There was a [DISCUSS] thread on the topic of moving to jdk8 for unit tests > [1], and many people also expressed the opinion that we should drop JDK 7 > support in Hive. Public updates by Oracle was stopped on Apr 2015 [2]. > > This vote t

Re: Green unit test results

2017-03-13 Thread Siddharth Seth
There's far too many flaky tests to enforce a clean test run prior to commit. However, there also needs to be a very conscious effort to fix flaky tests - otherwise, we're not going to reach the state where this can be enforced. I'm also not sure whether this is just flaky tests, or whether there a

Re: [Announce] New committer: Vineet Garg

2017-05-09 Thread Siddharth Seth
Congratulations Vineet. On Tue, May 9, 2017 at 1:24 PM, Ashutosh Chauhan wrote: > The Project Management Committee (PMC) for Apache Hive has invited Vineet > Garg to become a committer and we are pleased to announce that he has > accepted. > > Welcome, Vineet! > > Thanks, > Ashutosh >

Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

2017-05-16 Thread Siddharth Seth
/llap/daemon/impl/TaskExecutorTestHelpers.java 27c426c ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java 868eec7 Diff: https://reviews.apache.org/r/59325/diff/1/ Testing --- Thanks, Siddharth Seth

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

2017-05-17 Thread Siddharth Seth
-- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59325/#review175184 --- On May 17, 2017, 12:30 a.m., Siddharth Seth wrote: > > -

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

2017-05-17 Thread Siddharth Seth
- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59325/#review175192 --- On May 17, 2017, 12:30 a.m., Siddharth Seth wrote: > > -

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

2017-05-17 Thread Siddharth Seth
- Thanks, Siddharth Seth

Re: Review Request 59325: Cleanup of structures required when LLAP access from external clients completes

2017-05-17 Thread Siddharth Seth
me out and > > be cleaned? > > > > also what if readLock blocks all cleanup attempts? > > Siddharth Seth wrote: > In terms of the callable - was trying to keep it simple. Can try making > use of a single callable. > > Why would a readLock b

Re: [Announce] New PMC members

2017-05-30 Thread Siddharth Seth
Congratulations! On Tue, May 30, 2017 at 8:55 AM, Sergio Pena wrote: > Thanks all for your comments, and congratulations to all new PMC members > too !!! > > - Sergio > > On Sun, May 28, 2017 at 8:43 PM, Rajesh Balamohan > wrote: > > > Congratulations to new PMC members! > > > > ~Rajesh.B > > >

Re: Review Request 37333: HIVE-11515 Still some possible race condition in DynamicPartitionPruner

2015-08-24 Thread Siddharth Seth
/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicPartitionPruner.java (line 472) <https://reviews.apache.org/r/37333/#comment151511> Should be an error (IllegalStateException), instead of just logging this. - Siddharth Seth On Aug. 11, 2015, 12:49 a.m., Navis Ryu

Re: unit tests in patches

2015-09-22 Thread Siddharth Seth
Can a 'Target Version' field be added to jiras ? That would help to get rid of the confusion caused by Fix Version being used to represent branches a jira does go into. On Mon, Sep 21, 2015 at 12:55 PM, Ashutosh Chauhan wrote: > Hi everyone, > > Generally, its a good idea to add unit tests in p

Re: Review Request 38783: HIVE-11969 start Tez session in background when starting CLI

2015-09-30 Thread Siddharth Seth
I believe the behaviour will be similar to what it is rightnow for invocation of open(). i.e. a RuntimeException will end up being thrown in case of most errors. Wondering if a SessionNotRunning exception from waitTillReady() needs to be handled differently in the async model. Separate

Re: [ANNOUNCE] New Hive Committer - Siddharth Seth

2015-10-22 Thread Siddharth Seth
com> wrote: > > > >> Congratulations !! > >> > >> -Original Message- > >> From: Chetna C [mailto:chetna@gmail.com] > >> Sent: Thursday, October 22, 2015 8:58 AM > >> To: dev@hive.apache.org > >> Cc: Siddharth Seth > >&g

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-23 Thread Siddharth Seth
Haven't looked at the details of the ZKSecretManager - but it looks like the Tokens issued by any of the LLAP instances can be used by an application to communicate with all other instances. Also, are the tokens the same for different applications ? - Siddharth Seth

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-24 Thread Siddharth Seth
review comments in a bit. llap-server/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java (line 100) <https://reviews.apache.org/r/40315/#comment167199> Precondition check for security==true and token!=null - Siddharth Seth On Nov. 24, 2015, 11:11 p.m.,

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-24 Thread Siddharth Seth
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/40315/#review107877 --- - Siddharth Seth On Nov. 24, 2015, 11:11 p.m., Sergey Shelukhin

Re: Review Request 40315: HIVE-12341 LLAP security

2015-11-24 Thread Siddharth Seth
> On Nov. 24, 2015, 3:56 a.m., Siddharth Seth wrote: > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 2361 > > <https://reviews.apache.org/r/40315/diff/2/?file=1127510#file1127510line2361> > > > > Don't think the default v

Re: Review Request 40315: HIVE-12341 LLAP security

2015-12-08 Thread Siddharth Seth
hive/llap/security/LlapSecurityHelper.java (line 66) <https://reviews.apache.org/r/40315/#comment168911> New jira to make this configurable - Siddharth Seth On Dec. 5, 2015, 9:40 p.m., Sergey Shelukhin wrote: > > --- > This is an automat

Re: Review Request 40315: HIVE-12341 LLAP security

2015-12-08 Thread Siddharth Seth
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/40315/#review109413 --- Ship it! Ship It! - Siddharth Seth On Dec. 5, 2015, 9:40 p.m

Re: Review Request 40315: HIVE-12341 LLAP security

2015-12-08 Thread Siddharth Seth
creating new jiras. The compiled proto file seems to be missing from the RB patch. Assuming that'll be part of the commit. - Siddharth Seth On Dec. 5, 2015, 9:40 p.m., Sergey Shelukhin wrote: > > --- > This is an automatical

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

2015-12-18 Thread Siddharth Seth
rify that though - without setting up explicit properties for tests only. - Siddharth Seth On Dec. 18, 2015, 10:10 p.m., Sergey Shelukhin wrote: > > --- > This is an automatically

Re: Hive 2.0 branch - starting next week, please ask RM before committing (in preparation for the release)

2016-01-15 Thread Siddharth Seth
I'd like to get the changes under HIVE-12446 committed. The Tez 0.8.2 release is in progress and looks like it'll be available by early next week. 0.8.2 has some bug fixes, and support for features which allow better locality for LLAP in Hive 2.0. Thanks - Sid On Thu, Jan 14, 2016 at 11:15 AM, Se

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

2016-01-20 Thread Siddharth Seth
Thread uncaughtExceptionHandler / Callable with a ThreadPoolExecutor ? ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java (line 206) <https://reviews.apache.org/r/41377/#comment176450> Is this statement still valid ? If not, please delete or modify accordingly. - Siddharth

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

2016-01-20 Thread Siddharth Seth
Exception propagation - looks good to me. - Siddharth Seth On Jan. 4, 2016, 11:12 p.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache

Re: Review Request 41377: HIVE-12528 don't start HS2 Tez sessions in a single thread

2016-01-22 Thread Siddharth Seth
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/41377/#review115890 --- Ship it! Ship It! - Siddharth Seth On Jan. 21, 2016, 12:27 a.m

Target version not showing up

2016-02-16 Thread Siddharth Seth
Have created an INFRA ticket for this - https://issues.apache.org/jira/browse/INFRA-11276.

Re: Hive 2.0 + tez setting up

2016-02-22 Thread Siddharth Seth
As part of using Tez-0.8, you will need to update the tez tar which resides on HDFS. The path can be found out by looking at tez.lib.uris in tez-site.xml. You could replace the file, or upload a new one and modify tez-site to point to this. At the moment, the client is using 0.8, while the AM is us

Failing tests, precommit test runtimes

2016-03-28 Thread Siddharth Seth
There's several tests which are failing consistently. Should we disable these and create blocker jiras to fix and enable thees for the next release. The list of test is at the end of the mail. Also, each pre-commit run seems to be taking over 3 hours. Is this normal ? Having test results come back

Re: Failing tests, precommit test runtimes

2016-03-29 Thread Siddharth Seth
sh is looking I believe, I havent had a chance to take a look myself. > > Szehon > > On Mon, Mar 28, 2016 at 2:28 PM, Siddharth Seth wrote: > > > There's several tests which are failing consistently. Should we disable > > these and create blocker jiras to fix and enable

Re: Failing tests, precommit test runtimes

2016-03-31 Thread Siddharth Seth
it. Since bug > seems to be in spark runtime, it may take a little while to fix it. > Increased run time for QA till then would be waste of resources. > > Thanks, > Ashutosh > > On Tue, Mar 29, 2016 at 9:06 AM, Siddharth Seth wrote: > > > Thanks Szehon. Should we dis

Re: [discuss] jdk8 support

2016-04-11 Thread Siddharth Seth
Option 3 sounds good. I'd ideally like to make JDK8 the minimum requirement soon as well. On Mon, Apr 11, 2016 at 4:59 PM, Szehon Ho wrote: > Sounds like a good plan, +1 > > On Mon, Apr 11, 2016 at 4:31 PM, Mohit Sabharwal > wrote: > > > Thanks, Ashutosh. Makes sense to keep the source and targ

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

2016-04-22 Thread Siddharth Seth
mment193941> Can we restrict this configuration parameter to be used within HS2 only ? Any other client (CLI) would have it off by default - and fetch the token from LLAP itself. me - Siddharth Seth On April 22, 2016, 6:47 p.m.

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

2016-04-22 Thread Siddharth Seth
> On April 23, 2016, 12:38 a.m., Siddharth Seth wrote: > > llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapSecurityHelper.java, > > line 193 > > <https://reviews.apache.org/r/46579/diff/1/?file=1357677#file1357677line193> > > > > Can we r

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

2016-04-25 Thread Siddharth Seth
make a change before commit - "hs2" as a constant somewhere rather than referring the string literal separately in 2 places, and maybe call it 'hiveserver' in case there's ever an hs3 / hs2 goes away. - Siddharth Seth On April 23, 2016, 1:57 a.m

Re: Review Request 46597: HIVE-13445 LLAP: token should encode application and cluster ids

2016-04-25 Thread Siddharth Seth
ports multiple instances, which will otherwise end up with different UUIDs. - Siddharth Seth On April 23, 2016, 2:08 a.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To reply

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

2016-04-30 Thread Siddharth Seth
- create a pool of UGI instances - 1 per executor, and share those. The allowCache, static lock, compare principal seems unnecessary. - Siddharth Seth On April 27, 2016, 11:01 p.m., Sergey Shelukhin wrote: > > --- > This

Re: Review Request 46597: HIVE-13445 LLAP: token should encode application and cluster ids

2016-04-30 Thread Siddharth Seth
oop/hive/llap/daemon/MiniLlapCluster.java (line 196) <https://reviews.apache.org/r/46597/#comment195150> This can send in a real cluster name, instead of leaving it null ? - Siddharth Seth On April 26, 2016, 12:58 a.m., Sergey Shelukhin wrote: > > --

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

2016-05-03 Thread Siddharth Seth
/LlapSecurityHelper.java (line 95) <https://reviews.apache.org/r/46754/#comment195512> Wondering if this ends up being a little too brittle. HADOOP-13081 itself may break this code. - Siddharth Seth On May 2, 2016, 9:44 p.m., Sergey Shelukhin

Re: Review Request 46597: HIVE-13445 LLAP: token should encode application and cluster ids

2016-05-03 Thread Siddharth Seth
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46597/#review131512 --- Ship it! Looks good. - Siddharth Seth On May 2, 2016, 10:01

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

2016-05-16 Thread Siddharth Seth
time. Looks like not sharing the SecretManager could be expensive given the KDC login each time ? - Siddharth Seth On May 5, 2016, 9:55 p.m., Sergey Shelukhin wrote: > > --- > This is an automatically generat

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

2016-05-17 Thread Siddharth Seth
a performance perspective, as well as to make sure the threads started by the SecretManager are shutdown. - Siddharth Seth On May 17, 2016, 12:56 a.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail.

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

2016-05-17 Thread Siddharth Seth
- > > (Updated May 17, 2016, 9:58 p.m.) > > > Review request for hive, Gunther Hagleitner, Siddharth Seth, and Vikram Dixit > Kumaraswamy. > > > Repository: hive-git > > > Description > --- > > see JIRA > > > Diffs

Re: Review Request 46579: HIVE-13449 LLAP: HS2 should get the token directly, rather than from LLAP

2016-05-17 Thread Siddharth Seth
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46579/#review133646 --- Ship it! Ship It! - Siddharth Seth On May 17, 2016, 9:58

Re: Review Request 47546: HIVE-13448 LLAP: check ZK acls for ZKSM and fail if they are too permissive

2016-05-19 Thread Siddharth Seth
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/47546/#review134040 --- Ship it! Ship It! - Siddharth Seth On May 18, 2016, 7:07

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

2016-05-19 Thread Siddharth Seth
The one created earlier to generate tokens should be passed in. The KeySigner could be an interface instead, and SecretManager (extends ZKDelegationTokenSecretManager) can implement this. ACL checks etc are already setup there. There's no requirement to have two independent c

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

2016-05-23 Thread Siddharth Seth
> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote: > > llap-common/src/java/org/apache/hadoop/hive/llap/security/SecretManager.java, > > line 134 > > <https://reviews.apache.org/r/46956/diff/2/?file=1387295#file1387295line134> > > > > Can a second

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

2016-05-23 Thread Siddharth Seth
che.org/r/46956/#comment199239> This should explicitly check for "false". Any other value should either default to true or thrown an error. Err on the side of being secure. - Siddharth Seth On May 21, 2016, 12:07 a.m.,

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

2016-05-23 Thread Siddharth Seth
issue. - Siddharth Seth On May 20, 2016, 1:49 a.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

2016-05-23 Thread Siddharth Seth
> On May 20, 2016, 2:21 a.m., Siddharth Seth wrote: > > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/LlapProtocolServerImpl.java, > > line 290 > > <https://reviews.apache.org/r/46956/diff/2/?file=1387298#file1387298line290> > > > > Wha

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

2016-05-24 Thread Siddharth Seth
wner" end up getting treated as false. ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java <https://reviews.apache.org/r/46956/#comment199458> This parameter can be removed from HiveConf now that it's not used. - Siddharth Seth On M

Re: Review Request 46956: HIVE-13444 LLAP: add HMAC signatures to LLAP; verify them on LLAP side

2016-05-27 Thread Siddharth Seth
on the test before committing. - Siddharth Seth On May 27, 2016, 12:39 a.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

2016-05-31 Thread Siddharth Seth
NullPointerException. Hopefully this will be caught by some tests. Would be good to add a couple of tests here as well. - Siddharth Seth On May 27, 2016, 9:32 p.m., Sergey Shelukhin wrote: > >

Re: Review Request 46754: HIVE-13391 add an option to LLAP to use keytab to authenticate to read data

2016-05-31 Thread Siddharth Seth
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/46754/#review135722 --- Ship it! Ship It! - Siddharth Seth On May 28, 2016, 2:04

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

2016-06-01 Thread Siddharth Seth
> On May 31, 2016, 11:09 p.m., Siddharth Seth wrote: > > llap-ext-client/src/java/org/apache/hadoop/hive/llap/LlapBaseInputFormat.java, > > line 350 > > <https://reviews.apache.org/r/47563/diff/3/?file=1399778#file1399778line350> > > > > Is a signatu

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

2016-06-01 Thread Siddharth Seth
> On May 31, 2016, 11:09 p.m., Siddharth Seth wrote: > > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java, line > > 381 > > <https://reviews.apache.org/r/47563/diff/3/?file=1399784#file1399784line381> > > > > Not sure why the use

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

2016-06-01 Thread Siddharth Seth
> Review request for hive, Jason Dere and Siddharth Seth. > > > Repository: hive-git > > > Description > --- > > see JIRA. Please ignore the first iteration, RB doesn't allow base patches on > submit > > > Diffs > - > > com

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

2016-06-01 Thread Siddharth Seth
s like it's being checked in createSignedVertexSpec. The comment still seems incorrect though, since TaskSpec is never sent any longer. Assuming the serialized spec without a signature works in LLAP (i.e. when the daemon gets the VertexSpecBinary without a signature - it handles it correc

Re: Review Request 47563: HIVE-13675 LLAP: add HMAC signatures to LLAPIF splits

2016-06-01 Thread Siddharth Seth
e the unnecessary TaskSpec creation, and potentially skipping the unnecessary VertexSpec being sent each time as a response from GenericUDTFGetSplits. - Siddharth Seth On June 2, 2016, midnight, Sergey Shelukhin wrote: > > --- &

Re: [discuss] jdk8 support

2016-06-03 Thread Siddharth Seth
gt; > >> > > > > >> On Tue, Apr 19, 2016 at 10:43 AM, Mohit Sabharwal < > > mo...@cloudera.com > > > > > > > > >> wrote: > > > > >> > > > > >> > Created HIVE-13547 to track switching 2x tests to JDK8. > > &

Re: Review Request 48186: HIVE-13443 LLAP: signing for the second state of submit (the event)

2016-06-03 Thread Siddharth Seth
unfortunately has a really large Hive configuration) ql/src/java/org/apache/tez/dag/api/TaskSpecBuilder.java (line 66) <https://reviews.apache.org/r/48186/#comment201101> Please elaborate ? (This is not a public Tez API). Should this function just be removed. Used only once -

Re: Review Request 48186: HIVE-13443 LLAP: signing for the second state of submit (the event)

2016-06-03 Thread Siddharth Seth
- > > (Updated June 2, 2016, 8:55 p.m.) > > > Review request for hive and Siddharth Seth. > > > Repository: hive-git > > > Description > --- > > see jira > > > Diffs > - > > llap-client/src/java/org/apache/hadoop/hive/

Re: Review Request 48502: HIVE-13731 LLAP: return LLAP token with the splits

2016-06-10 Thread Siddharth Seth
stop sending this repeated information. Think there's a jira open to avoid sending the VertexSpec per split - that can include sending the token only once. - Siddharth Seth On June 9, 2016, 7:07 p.m., Sergey Shelukhin wrote: > > ---

Review Request 48818: See jira

2016-06-16 Thread Siddharth Seth
/TestLlapTaskCommunicator.java 1901328 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java bdf254b Diff: https://reviews.apache.org/r/48818/diff/ Testing --- Thanks, Siddharth Seth

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

2016-06-16 Thread Siddharth Seth
/ Testing --- Thanks, Siddharth Seth

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

2016-06-16 Thread Siddharth Seth
o do. It can be re-introduced if required ? (The Hook interface still exists in TezProcessor). - Siddharth --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/48818/#review138141 --

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

2016-06-16 Thread Siddharth Seth
/hadoop/hive/llap/tezplugins/TestLlapTaskCommunicator.java 1901328 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java bdf254b Diff: https://reviews.apache.org/r/48818/diff/ Testing --- Thanks, Siddharth Seth

Re: Review Request 48818: HIVE-14023. LLAP: Make the Hive query id available in ContainerRunner

2016-06-16 Thread Siddharth Seth
/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java bdf254b Diff: https://reviews.apache.org/r/48818/diff/ Testing --- Thanks, Siddharth Seth

Re: Review Request 48886: HIVE-14052: Cleanup of structures required when LLAP access from external clients completes

2016-06-30 Thread Siddharth Seth
/r/48886/#comment205692> Does it make sense to return a FragmentCompletionHandler rather than a Closeable ? ql/src/test/org/apache/hadoop/hive/llap/TestLlapOutputFormat.java (line 52) <https://reviews.apache.org/r/48886/#comment205693> Needs tests. O

Re: Review Request 51397: HIVE-14536 Unit test code cleanup

2016-08-25 Thread Siddharth Seth
have a smaller window for conflicts. There's several jiras open to fix either testing infrastructure, or individual tests, or get things working in the IDE. I know Zoltan was driving towards getting tests working via the IDE - I don't know how this affects that effort. - Siddharth Se

Re: Review Request 51397: HIVE-14536 Unit test code cleanup

2016-08-26 Thread Siddharth Seth
> On Aug. 26, 2016, 2:01 a.m., Siddharth Seth wrote: > > Not a lot here related to the patch itself in this comment. > > Looks like everyone has gotten interested in fixing Hive tests at about the > > same time - and there's a good reason for that :) > > I

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

2016-08-30 Thread Siddharth Seth
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestHostAffinitySplitLocationProvider.java (line 94) <https://reviews.apache.org/r/51312/#comment214490> This isn't part of the patch uploaded to jira? - Siddharth Seth On Aug. 23, 2016, 1:41 a.m., Sergey Shelukhin wrote: >

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

2016-08-30 Thread Siddharth Seth
/ (private static final UUID uniq = UUID.randomUUID();) - Siddharth Seth On Aug. 23, 2016, 1:41 a.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://revi

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

2016-09-01 Thread Siddharth Seth
> On Aug. 30, 2016, 10:35 p.m., Siddharth Seth wrote: > > llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java, > > line 538 > > <https://reviews.apache.org/r/51312/diff/1/?file=1481607#file1481607line538> > > > &g

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

2016-09-01 Thread Siddharth Seth
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51312/#review147548 --- Will look at the updated patch tomorrow. - Siddharth Seth On

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

2016-09-01 Thread Siddharth Seth
> On Aug. 30, 2016, 10:36 p.m., Siddharth Seth wrote: > > Unrelated to this patch - Any idea why 'uniq' is static/ (private static > > final UUID uniq = UUID.randomUUID();) > > Sergey Shelukhin wrote: > it's daemon identity, so it's global per

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

2016-09-01 Thread Siddharth Seth
> On Aug. 30, 2016, 10:35 p.m., Siddharth Seth wrote: > > llap-client/src/java/org/apache/hadoop/hive/llap/registry/impl/LlapZookeeperRegistryImpl.java, > > line 554 > > <https://reviews.apache.org/r/51312/diff/1/?file=1481607#file1481607line554> > > > >

Re: Review Request 51312: HIVE-14589 add consistent node replacement to LLAP for splits

2016-09-01 Thread Siddharth Seth
need resolving. Follow up jiras. +1, assuming it's been tested locally for regresions and the new functionality. - Siddharth Seth On Sept. 1, 2016, 6:35 p.m., Sergey Shelukhin wrote: > > --- > This is an automatically generated e-ma

ptest result structure change (batched unit tests)

2016-09-16 Thread Siddharth Seth
HIVE-14540 changes ptest to batch unit tests together (instead of invoking mvn test on individual test classes). As a result, output for multiple tests is a common directory - similar to what happens for batched q tests. To identify the relevant directory, lookup the consoleOuput for the run. e.g

[DISCUSS] Pre-commit tests before commits

2016-10-13 Thread Siddharth Seth
There's been a lot of work to make the test runs faster, as well as more reliable via HIVE-14547, HIVE-13503, and several other jiras. Test runtimes are around the 1 hour mark, and going down. There were a few green pre-commit runs (after years?). At the same time, there's still some flaky tests.

Re: [DISCUSS] Pre-commit tests before commits

2016-10-14 Thread Siddharth Seth
tables to init script as this will > increase > >> the initialization time. > >> > >> 7) If the test case does not require explain plan then avoid it as most > >> failures are explain diffs. > >> > >> 8) If the test case requires explain and

Re: [DISCUSS] Pre-commit tests before commits

2016-10-24 Thread Siddharth Seth
he list, or modified directly. Thanks, Sid On Mon, Oct 17, 2016 at 12:07 PM, Sergey Shelukhin wrote: > +1 > > On 16/10/13, 23:10, "Siddharth Seth" wrote: > > >There's been a lot of work to make the test runs faster, as well as more > >reliable via HIVE-1

Re: [DISCUSS] Pre-commit tests before commits

2016-10-25 Thread Siddharth Seth
Guide: Adding new unit tests ><https://cwiki.apache.org/confluence/display/Hive/ > DeveloperGuide#DeveloperGuide-Addingnewunittests> > (outdated >instructions) > > -- Lefty > > > On Mon, Oct 24, 2016 at 4:49 PM, Siddharth Seth wrote: > > > There

Re: build bot on master

2016-10-28 Thread Siddharth Seth
Restarted the ptest server. That normally fixes this error. On Fri, Oct 28, 2016 at 10:33 AM, Eugene Koifman wrote: > Hi, > all test runs are failing with the error below. Does anyone know how to > fix it? > > Thanks, > Eugene > > from https://builds.apache.org/view/H-L/view/Hive/job/ > PreCom

Re: [DISCUSS] Yetus pre-commit tests

2016-11-10 Thread Siddharth Seth
Peter In terms of the modules - ignoring the time taken - I would vote for asflicense, author, findbugs, javac, maybe javadoc, wrhitespace. Not sure what checkstyle does, and some form of test4tests is already covered in ptest. This will at least help preventing new issues. Fixing the existing set

Number of retained PreCommit runs

2016-11-10 Thread Siddharth Seth
Looks like we retain the last 10 runs at max. With tests running in less than an hour, very often, we end up with situations where test reports are not available? I suspect the 10 run limit is due to the size of the logs generated by each run - close to 6GB (on the ptest server). I think it will be

[jira] [Created] (HIVE-8597) SMB join small table side should use the same set of serialized payloads across tasks

2014-10-24 Thread Siddharth Seth (JIRA)
Siddharth Seth created HIVE-8597: Summary: SMB join small table side should use the same set of serialized payloads across tasks Key: HIVE-8597 URL: https://issues.apache.org/jira/browse/HIVE-8597

[jira] [Updated] (HIVE-8597) SMB join small table side should use the same set of serialized payloads across tasks

2014-10-24 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-8597: - Attachment: HIVE-8597.1.patch Patch to create one set of serialized splits for each bucket, and re

[jira] [Updated] (HIVE-8597) SMB join small table side should use the same set of serialized payloads across tasks

2014-10-24 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-8597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-8597: - Status: Patch Available (was: Open) > SMB join small table side should use the same set

[jira] [Commented] (HIVE-7976) Merge tez branch into trunk (tez 0.5.0)

2014-09-05 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123337#comment-14123337 ] Siddharth Seth commented on HIVE-7976: -- +1. The Tez changes look good. > Me

[jira] [Created] (HIVE-20763) Add google cloud storage (gs) to the exim uri schema whitelist

2018-10-17 Thread Siddharth Seth (JIRA)
Siddharth Seth created HIVE-20763: - Summary: Add google cloud storage (gs) to the exim uri schema whitelist Key: HIVE-20763 URL: https://issues.apache.org/jira/browse/HIVE-20763 Project: Hive

[jira] [Created] (HIVE-7427) Changes for EdgeConfigurations

2014-07-15 Thread Siddharth Seth (JIRA)
Siddharth Seth created HIVE-7427: Summary: Changes for EdgeConfigurations Key: HIVE-7427 URL: https://issues.apache.org/jira/browse/HIVE-7427 Project: Hive Issue Type: Sub-task

[jira] [Updated] (HIVE-7427) Changes for EdgeConfigurations

2014-07-15 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-7427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-7427: - Attachment: HIVE-7427.1.patch > Changes for EdgeConfigurati

[jira] [Updated] (HIVE-6988) Hive changes for tez-0.5.x compatibility

2014-07-15 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-6988: - Status: Open (was: Patch Available) > Hive changes for tez-0.5.x compatibil

[jira] [Updated] (HIVE-6988) Hive changes for tez-0.5.x compatibility

2014-07-15 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-6988: - Attachment: HIVE-6988.4.patch Updated merged patch to include changes from HIVE-7427. > H

[jira] [Updated] (HIVE-6988) Hive changes for tez-0.5.x compatibility

2014-07-15 Thread Siddharth Seth (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-6988: - Status: Patch Available (was: Open) > Hive changes for tez-0.5.x compatibil

  1   2   3   4   5   >