Re: RFC: Major HCatalog refactoring

2013-09-03 Thread Eugene Koifman
The plan was 2 releases. On Tue, Sep 3, 2013 at 12:14 PM, Edward Capriolo wrote: > You may have already said this but remind me again. If we go with this > approach, how long until we retired the duplicated code and insist end > users use the new name? 1 release? > > A similar debate is likely

Re: RFC: Major HCatalog refactoring

2013-09-03 Thread Eugene Koifman
We explored the idea you suggest and given the number of APIs (and their transitive closure) it would would be very difficult and the result would be fragile. So unfortunately that is not possible. For example, oldpackage.A has a method foo() that returns oldpackage.B. You could create newpacka

Re: RFC: Major HCatalog refactoring

2013-09-03 Thread Edward Capriolo
There is already a few things in hcat that I would argue are things we do not need to test. Example: testAMQListener I would say it is a bit out of scope to have a component like this. Could we mock the message queue, do we need these dependencies in project? I can not even fathom how long the t

Re: RFC: Major HCatalog refactoring

2013-09-03 Thread Edward Capriolo
You may have already said this but remind me again. If we go with this approach, how long until we retired the duplicated code and insist end users use the new name? 1 release? A similar debate is likely why the hive classes are still packaged as org.apache.hadoop.hive, rather then org.apache.hive

Re: RFC: Major HCatalog refactoring

2013-09-03 Thread Edward Capriolo
I understand. Can we do something like this? oldpackage.HCatologLoader extends newpackage.HCatlogloader { } If we do something like this we don't need to test both classes, it is safe to assume they both do the same thing. I understand that we do not want users to have to specify a new class na

Re: RFC: Major HCatalog refactoring

2013-09-03 Thread Eugene Koifman
Edward, "If a testing framework is truly testing all code paths twice, there is not much of a win there from a unit/integration tests standpoint. If the unit tests created more coverage of the code that would be an obvious win. I have not looked at your patch but from your description it sounds li

Re: RFC: Major HCatalog refactoring

2013-09-03 Thread Edward Capriolo
I would say a main goal of unit and integration testing is to try all code paths. If a testing framework is truly testing all code paths twice, there is not much of a win there from a unit/integration tests standpoint. If the unit tests created more coverage of the code that would be an obvious win

Re: RFC: Major HCatalog refactoring

2013-09-03 Thread Thejas Nair
One thing to note is that the 0.11 interfaces are going to be deprecated and will be taken away in a later release. When the interface is taken away, the additional unit tests will also go away. On Tue, Sep 3, 2013 at 9:57 AM, Eugene Koifman wrote: > Current (sequential) run of all hive/hcat unit

Re: RFC: Major HCatalog refactoring

2013-09-03 Thread Eugene Koifman
Current (sequential) run of all hive/hcat unit tests takes 10-15 hours. Is another 20-30 minutes that significant? I'm generally wary of unit tests that are not run continuously and automatically. It delays the detection of problems and then what was probably an obvious fix at the time the chang

Re: RFC: Major HCatalog refactoring

2013-09-03 Thread Brock Noland
OK that should be fine. Though I would echo Edwards sentiment about adding so much test time. Do these tests have to run each time? Does it make sense to have an test target such as test-all-hcatalog and then have then run them periodically manually, especially before releases? On Mon, Sep 2, 201

Re: RFC: Major HCatalog refactoring

2013-09-02 Thread Eugene Koifman
These will be new (I.e. 0.11 version) test classes which will be in the old org.apache.hcatalog package. How does that affect the new framework? On Saturday, August 31, 2013, Brock Noland wrote: > Will these be new Java class files or new test methods to existing > classes? I am just curious as

Re: RFC: Major HCatalog refactoring

2013-08-31 Thread Edward Capriolo
>> not quite double but close (on my Mac that means it will go up from 35 >> minutes to 55-60) so in greater scheme of things it should be negligible Can't we make the classes extend each other, and just test them once. Or test them once before the patch and only include half the tests in the fin

Re: RFC: Major HCatalog refactoring

2013-08-31 Thread Brock Noland
Will these be new Java class files or new test methods to existing classes? I am just curious as to how this will play into the distributed testing framework. On Sat, Aug 31, 2013 at 10:19 AM, Eugene Koifman wrote: > not quite double but close (on my Mac that means it will go up from 35 > minut

Re: RFC: Major HCatalog refactoring

2013-08-31 Thread Eugene Koifman
not quite double but close (on my Mac that means it will go up from 35 minutes to 55-60) so in greater scheme of things it should be negligible On Sat, Aug 31, 2013 at 7:35 AM, Edward Capriolo wrote: > By coverage do you mean to say that: > > > Thus, the published HCatalog JARs will contain bo

Re: RFC: Major HCatalog refactoring

2013-08-31 Thread Edward Capriolo
By coverage do you mean to say that: > Thus, the published HCatalog JARs will contain both packages and the unit > tests will cover both versions of the API. We are going to double the time of unit tests for this module? On Fri, Aug 30, 2013 at 8:41 PM, Eugene Koifman wrote: > This will change

Re: RFC: Major HCatalog refactoring

2013-08-30 Thread Eugene Koifman
This will change every file under hcatalog so it has to happen before the branching. Most likely at the beginning of next week. Thanks On Wed, Aug 28, 2013 at 5:24 PM, Eugene Koifman wrote: > Hi, > > > Here is the plan for refactoring HCatalog as was agreed to when it was > merged into Hive du

RFC: Major HCatalog refactoring

2013-08-28 Thread Eugene Koifman
Hi, Here is the plan for refactoring HCatalog as was agreed to when it was merged into Hive during. HIVE-4869 is the umbrella bug for this work. The changes are complex and touch every single file under hcatalog. Please comment. When HCatalog project was merged into Hive on 0.11 several integ