I understand.
Can we do something like this?
oldpackage.HCatologLoader extends newpackage.HCatlogloader { }
If we do something like this we don't need to test both classes, it is safe
to assume they both do the same thing.
I understand that we do not want users to have to specify a new class name,
but 15 minutes of unit tests around a re-name is overkill.
On Tue, Sep 3, 2013 at 2:13 PM, Eugene Koifman <ekoif...@hortonworks.com>wrote:
> Edward,
>
> "If a testing framework is truly testing all code paths twice, there
> is not much of a win there from a unit/integration tests standpoint. If the
> unit tests created more coverage of the code that would be an obvious win.
> I have not looked at your patch but from your description it sounds like we
> are attempting to test a rename that does not sound like a win to me."
>
> Actually this is not what we are testing. The package name change (as well
> as any changes made in 0.12) will be tested by current tests (which will
> also change package name).
>
> The goal of bringing 0.11 version of the source (and corresponding tests)
> into 0.12 is to ensure that users who use HCatalog from scripts/MR jobs,
> etc (e.g. a Pig script: A = LOAD 'tablename' USING
> org.apache.hcatalog.pig.HCatLoader();) will not have to update all the
> their scripts/programs when upgrading to 0.12. Having 0.11 tests in 0.12
> branch ensures that this compatibility layer continues to work while HIve
> 0.12 and later versions are evolving.
>
>
>
>
>
> On Tue, Sep 3, 2013 at 10:22 AM, Edward Capriolo <edlinuxg...@gmail.com
> >wrote:
>
> > I would say a main goal of unit and integration testing is to try all
> code
> > paths. If a testing framework is truly testing all code paths twice,
> there
> > is not much of a win there from a unit/integration tests standpoint. If
> the
> > unit tests created more coverage of the code that would be an obvious
> win.
> > I have not looked at your patch but from your description it sounds like
> we
> > are attempting to test a rename that does not sound like a win to me.
> >
> > If the current hcatalog tests run in 15 minutes, you make a change and
> then
> > the run is 30 minutes. 15 minutes is a nice long coffee break, 30 minutes
> > is a TV show :)
> >
> > As for the overall hive build taking 10-15 hours. I know that :) I used
> to
> > run them, by hand, on my laptop, because no one would share their build
> > farm with me. I have heard that Hive consumes the vast majority of the
> > resources of apache's build farm! I think we need to be good citizens at
> > apache and attempt to make this better, not worse.
> >
> > Now that we have pre-commit builds we can work at a reasonable pace. Now
> > that we have this nice pre-commit farm, I do not want to create a
> precedent
> > that now we can go "nuts", and start down the same slippery slope.
> >
> >
> >
> >
> > On Tue, Sep 3, 2013 at 12:57 PM, Eugene Koifman <
> ekoif...@hortonworks.com
> > >wrote:
> >
> > > Current (sequential) run of all hive/hcat unit tests takes 10-15 hours.
> > Is
> > > another 20-30 minutes that significant?
> > >
> > > I'm generally wary of unit tests that are not run continuously and
> > > automatically. It delays the detection of problems and then what was
> > > probably an obvious fix at the time the change was made becomes a long
> > > debugging session (often by someone other than whose change broke
> > things).
> > > I think this is especially true given how many people are contributing
> > to
> > > hive.
> > >
> > >
> > >
> > > On Tue, Sep 3, 2013 at 7:25 AM, Brock Noland <br...@cloudera.com>
> wrote:
> > >
> > > > OK that should be fine. Though I would echo Edwards sentiment about
> > > > adding so much test time. Do these tests have to run each time? Does
> > > > it make sense to have an test target such as test-all-hcatalog and
> > > > then have then run them periodically manually, especially before
> > > > releases?
> > > >
> > > > On Mon, Sep 2, 2013 at 10:36 AM, Eugene Koifman
> > > > <ekoif...@hortonworks.com> wrote:
> > > > > These will be new (I.e. 0.11 version) test classes which will be in
> > the
> > > > old
> > > > > org.apache.hcatalog package. How does that affect the new
> framework?
> > > > >
> > > > > On Saturday, August 31, 2013, Brock Noland wrote:
> > > > >
> > > > >> Will these be new Java class files or new test methods to existing
> > > > >> classes? I am just curious as to how this will play into the
> > > > >> distributed testing framework.
> > > > >>
> > > > >> On Sat, Aug 31, 2013 at 10:19 AM, Eugene Koifman
> > > > >> <ekoif...@hortonworks.com> wrote:
> > > > >> > not quite double but close (on my Mac that means it will go up
> > from
> > > > 35
> > > > >> > minutes to 55-60) so in greater scheme of things it should be
> > > > negligible
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > On Sat, Aug 31, 2013 at 7:35 AM, Edward Capriolo <
> > > > edlinuxg...@gmail.com
> > > > >> >wrote:
> > > > >> >
> > > > >> >> By coverage do you mean to say that:
> > > > >> >>
> > > > >> >> > Thus, the published HCatalog JARs will contain both packages
> > and
> > > > the
> > > > >> unit
> > > > >> >> > tests will cover both versions of the API.
> > > > >> >>
> > > > >> >> We are going to double the time of unit tests for this module?
> > > > >> >>
> > > > >> >>
> > > > >> >> On Fri, Aug 30, 2013 at 8:41 PM, Eugene Koifman <
> > > > >> ekoif...@hortonworks.com
> > > > >> >> >wrote:
> > > > >> >>
> > > > >> >> > This will change every file under hcatalog so it has to
> happen
> > > > before
> > > > >> the
> > > > >> >> > branching. Most likely at the beginning of next week.
> > > > >> >> >
> > > > >> >> > Thanks
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Wed, Aug 28, 2013 at 5:24 PM, Eugene Koifman <
> > > > >> >> ekoif...@hortonworks.com
> > > > >> >> > >wrote:
> > > > >> >> >
> > > > >> >> > > Hi,
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > Here is the plan for refactoring HCatalog as was agreed to
> > when
> > > > it
> > > > >> was
> > > > >> >> > > merged into Hive during. HIVE-4869 is the umbrella bug for
> > > this
> > > > >> work.
> > > > >> >> > The
> > > > >> >> > > changes are complex and touch every single file under
> > hcatalog.
> > > > >> Please
> > > > >> >> > > comment.
> > > > >> >> > >
> > > > >> >> > > When HCatalog project was merged into Hive on 0.11 several
> > > > >> integration
> > > > >> >> > > items did not make the 0.11 deadline. It was agreed to
> > finish
> > > > them
> > > > >> in
> > > > >> >> > 0.12
> > > > >> >> > > release. Specifically:
> > > > >> >> > >
> > > > >> >> > > 1. HIVE-4895 - change package name from org.apache.hcatalog
> > to
> > > > >> >> > > org.apache.hive.hcatalog
> > > > >> >> > >
> > > > >> >> > > 2. HIVE-4896 - create binary backwards compatibility layer
> > for
> > > > hcat
> > > > >> >> users
> > > > >> >> > > upgrading from 0.11 to 0.12
> > > > >> >> > >
> > > > >> >> > > For item 1, we’ll just move every file under
> > > org.apache.hcatalog
> > > > to
> > > > >> >> > > org.apache.hive.hcatalog and update all “package” and
> > “import”
> > > > >> >> statement
> > > > >> >> > as
> > > > >> >> > > well as all hcat/webhcat scripts. This will include all
> > JUnit
> > > > >> tests.
> > > > >> >> > >
> > > > >> >> > > Item 2 will ensure that if a user has a M/R program or Pig
> > > > script,
> > > > >> etc.
> > > > >> >> > > that uses HCatalog public API, their programs will continue
> > to
> > > > work
> > > > >> w/o
> > > > >> >> > > change with hive 0.12.
> > > > >> >> > >
> > > > >> >> > > The proposal is to make the changes that have as little
> > impact
> > > on
> > > > >> the
> > > > >> >> > > build system, in part to make upcoming ‘mavenization’ of
> hive
> > > > >> easier,
> > > > >> >> in
> > > > >> >> > > part to make the changes more manageable.
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > The list of public interfaces (and their transitive
> closure)
> > > for
> > > > >> which
> > > > >> >> > > backwards compat will be provided.
> > > > >> >> > >
> > > > >> >> > > 1.
> > > > >> >> > >
> > > > >> >> > > HCatLoader
> > > > >> >> > > 2.
> > > > >> >> > >
> > > > >> >> > > HCatStorer
> > > > >> >> > > 3.
> > > > >> >> > >
> > > > >> >> > > HCatInputFormat
> > > > >> >> > > 4.
> > > > >> >> > >
> > > > >> >> > > HCatOutputFormat
> > > > >> >> > > 5.
> > > > >> >> > >
> > > > >> >> > > HCatReader
> > > > >> >> > > 6.
> > > > >> >> > >
> > > > >> >> > > HCatWriter
> > > > >> >> > > 7.
> > > > >> >> > >
> > > > >> >> > > HCatRecord
> > > > >> >> > > 8.
> > > > >> >> > >
> > > > >> >> > > HCatSchema
> > > > >> >> > >
> > > > >> >> > >
> > > > >> >> > > To achieve this, 0.11 version of these classes will be
> added
> > in
> > > > >> >> > > org.apache.hcatalog package (after item 1 is done). Each
> of
> > > > these
> > > > >> >> > classes
> > > > >> --
> > > > >> Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
> > > > >>
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > >
> > > >
> > > >
> > > > --
> > > > Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
