Hi Team, If everyone agrees, tomorrow I would like to start the release process for 4.0.0-alpha-1.
Is there any outstanding blocker jira that you know of? Thanks, Peter > On 2022. Mar 9., at 17:01, Stamatis Zampetakis <zabe...@gmail.com> wrote: > > I just logged HIVE-26022 [1] which seems to be another potential blocker > for 4.0.0-alpha-1. > > Best, > Stamatis > > [1] https://issues.apache.org/jira/browse/HIVE-26022 > > On Thu, Mar 3, 2022 at 3:54 PM Peter Vary <pv...@cloudera.com> wrote: > >> Hi Team, >> >> Here is our status: >> We collected the blocker tickets and marked them with fixVersion >> 4.0.0-alpha-1: >> >> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20%3D%20HIVE%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%204.0.0-alpha-1 >> <https://issues.apache.org/jira/issues/?filter=-1&jql=project%20=%20HIVE%20AND%20resolution%20=%20Unresolved%20AND%20fixVersion%20=%204.0.0-alpha-1> >> >> - HIVE-26002 - Create db scripts for 4.0.0-alpha-1 >> - HIVE-25994 - Analyze table runs into ClassNotFoundException-s in >> case binary distribution is used >> - HIVE-25935 - Cleanup IMetaStoreClient#getPartitionsByNames APIs >> >> Please create a jira and mark it with fixVersion 4.0.0-alpha-1, if you >> happen to know of any other blockers. >> >> We plan to fix these jiras, and then release the following artifacts >> together: >> >> - Storage API - 4.0.0-alpha-1 >> - Standalone Metastore - 4.0.0-alpha-1 >> - Hive - 4.0.0-alpha-1 >> >> >> Thanks, >> Peter >> >> >> On 2022. Mar 2., at 11:50, Peter Vary <pv...@cloudera.com> wrote: >> >> Will continue this discussion on the #hive ASF slack. If you are >> interested, please join. >> We will do updates here time-to-time, so the ones who are not using slack >> can participate that way. >> >> On 2022. Mar 2., at 11:11, Peter Vary <pv...@cloudera.com> wrote: >> >> Good idea Zoltan, joined the channel. >> I would like to scope reasonably small, so I agree with focusing on >> 4.0.0-alpha-1 >> >> On 2022. Mar 2., at 11:01, Zoltan Haindrich <k...@rxd.hu> wrote: >> >> Hey, >> >> regarding 4.0.0 / 4.0.0-alpha-1 target/fix versions in the jira: >> * I think we should change all already resolved tickets with fix version >> 4.0.0 to have fix version 4.0.0-alpha-1 >> ** this could be postponed until we are actually releasing the thing as I >> think everyone committing to the master is entering 4.0.0 as fix version >> without much aftertought...this could probably change after we get the >> first release out. >> * regarding the the existing tickets with fix version/target version 4.0.0 >> - I think that would be a bit too much (>200 tickets) >> ** some numbers: >> *** 239 tickets open now >> *** 224 was not updated in the last 90 days >> *** 216 was not changed in the last 180 days >> *** 178 was not updated in the last 360 days >> ** as a matter of fact I think many of these tickets shouldn't even have a >> target or fix version - and most of them should be unassigned...I don't >> want to get lost in this right now...I think for now we should keep the >> scope small and only care with 4.0.0-alpha-1 tickets >> >> https://issues.apache.org/jira/issues/? >> jql=project%20%3D%20hive%20and%20resolutiondate%20%20is%20empty%20and%20(fixVersion%20%20in%20(%274.0.0%27)%20or%20cf%5B12310320%5D%20%20in%20(%274.0.0%27)) >> >> I think for faster communication regarding these things we could also >> utilize the #hive channel on the ASF slack - what do you guys think? >> >> cheers, >> Zoltan >> >> On 3/2/22 9:51 AM, Stamatis Zampetakis wrote: >> >> Agree with Peter, creating JIRAs is the way to go. >> Putting the appropriate priority (e.g., BLOCKER) and version (4.0.0 or >> 4.0.0-alpha-1) when creating the JIRA should be enough to keep us on track. >> I am mentioning both 4.0.0 and 4.0.0-alpha-1 because eventually I think we >> are gonna move everything with target 4.0.0 to 4.0.0-alpha-1. >> On Wed, Mar 2, 2022 at 9:37 AM Peter Vary <pv...@cloudera.com.invalid> >> wrote: >> >> Hi Team, >> >> Could we create tickets for the issues? >> I think it would be good to collect the issues/potential blockers in the >> jira instead of having a complicated mail thread. >> >> If we set the target version to 4.0.0-alpha-1, then we can easily use the >> following filter to see the status of the tasks: >> >> >> https://issues.apache.org/jira/issues/?jql=project%3D%22HIVE%22%20AND%20%22Target%20Version%2Fs%22%3D%224.0.0-alpha-1%22 >> < >> >> https://issues.apache.org/jira/issues/?jql=project=%22HIVE%22%20AND%20%22Target%20Version/s%22=%224.0.0-alpha-1%22 >> >> >> >> @Stamatis: Sadly I have missed your letter/jira and created my own with >> the fix for building from the src package: >> https://issues.apache.org/jira/browse/HIVE-25997 < >> https://issues.apache.org/jira/browse/HIVE-25997> >> If you have time, I would like to ask you to review. >> >> If anyone knows of any blocker I would like to ask them to create a jira >> for that too. >> >> Thanks, >> Peter >> >> >> On 2022. Mar 2., at 7:04, Sungwoo Park <c...@pl.postech.ac.kr> wrote: >> >> Hello Alessandro, >> >> For the latest commit, loading ORC tables fails (with the log message >> >> shown below). Let me try to find a commit that introduces this bug and >> create a JIRA ticket. >> >> >> --- Sungwoo >> >> 2022-03-02 05:41:56,578 ERROR [Thread-73] exec.StatsTask: Failed to run >> >> stats task >> >> java.io.IOException: org.apache.hadoop.mapred.InvalidInputException: >> >> Input path does not exist: >> >> hdfs://blue0:8020/tmp/hive/gitlab-runner/a236e1b4-b354-4343-b900-3d71b1bc7504/hive_2022-03-02_05-40-50_966_446574755576325031-1/-mr-10000/.hive-staging_hive_2022-03-02_05-40-50_966_446574755576325031-1/-ext-10001 >> >> at >> >> >> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:622) >> >> at >> >> >> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.constructColumnStatsFromPackedRows(ColStatsProcessor.java:105) >> >> at >> >> >> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:200) >> >> at >> >> >> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:93) >> >> at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:107) >> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212) >> at >> >> >> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) >> >> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:83) >> Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path >> >> does not exist: >> >> hdfs://blue0:8020/tmp/hive/gitlab-runner/a236e1b4-b354-4343-b900-3d71b1bc7504/hive_2022-03-02_05-40-50_966_446574755576325031-1/-mr-10000/.hive-staging_hive_2022-03-02_05-40-50_966_446574755576325031-1/-ext-10001 >> >> at >> >> >> org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:294) >> >> at >> >> >> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:236) >> >> at >> >> >> org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:45) >> >> at >> >> >> org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:322) >> >> at >> >> >> org.apache.hadoop.hive.ql.exec.FetchOperator.generateWrappedSplits(FetchOperator.java:435) >> >> at >> >> >> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:402) >> >> at >> >> >> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:306) >> >> at >> >> >> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:560) >> >> ... 7 more >> >> On Tue, 1 Mar 2022, Alessandro Solimando wrote: >> >> Hi Sungwoo, >> last time I tried to run TPCDS-based benchmark I stumbled upon a similar >> situation, finally I found that statistics were not computed, so CBO was >> not kicking in, and the automatic retry goes with CBO off which was >> >> failing >> >> for something like 10 queries (subqueries cannot be decorrelated, but >> >> also >> >> some runtime errors). >> >> Making sure that (column) statistics were correctly computed fixed the >> problem. >> >> Can you check if this is the case for you? >> >> HTH, >> Alessandro >> >> On Tue, 1 Mar 2022 at 15:28, POSTECH CT <c...@pl.postech.ac.kr> wrote: >> >> Hello Hive team, >> >> I wonder if anyone in the Hive team has tried the TPC-DS benchmark on >> the master branch recently. We occasionally run TPC-DS system tests >> using the master branch, and the tests don't succeed completely. Here >> is how our TPC-DS tests proceed. >> >> 1. Compile and run Hive on Tez (not Hive-LLAP) >> 2. Load ORC tables from 1TB TPC-DS raw text data, and compute >> >> statistics >> >> 3. Run 99 TPC-DS queries which were slightly modified to return >> varying number of rows (rather than 100 rows) >> 4. Compare the results against the previous results >> >> The previous results were obtained and cross-checked by running Hive >> 3.1.2 and SparkSQL 2.3/3.2, so we are faily confident about their >> correctness. >> >> For the latest commit in the master branch, step 2 fails. For earlier >> commits (for example, commits in February 2021), step 3 fails where >> several queries either fail or return wrong results. >> >> We can compile and report the test results in this mailing list, but >> would like to know if similar results have been reproduced by the Hive >> team, in order to make sure that we did not make errors in our tests. >> >> If it is okay to open a JIRA ticket that only reports failures in the >> TPC-DS test, we could also perform git bi-sect to locate the commit >> that begin to generate wrong results. >> >> --- Sungwoo Park >> >> On Tue, 1 Mar 2022, Zoltan Haindrich wrote: >> >> Hey, >> >> Great to hear that we are on the same side regarding these things :) >> >> For around a week now - we have nightly builds for the master branch: >> http://ci.hive.apache.org/job/hive-nightly/12/ >> >> I think we have 1 blocker issue: >> https://issues.apache.org/jira/browse/HIVE-25665 >> >> I know about one more thing I would rather get fixed before we release >> >> it: >> >> https://issues.apache.org/jira/browse/HIVE-25994 >> The best would be to introduce smoke tests (HIVE-22302) to ensure that >> something like this will not happen in the future - but we should >> >> probably >> >> start moving forward. >> >> I think we could call the first iteration of this as "4.0.0-alpha-1" >> >> :) >> >> >> I've added 4.0.0-alpha-1 as a version - and added the above two ticket >> >> to it. >> >> >> >> >> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HIVE%20AND%20fixVersion%20%3D%204.0.0-alpha-1 >> >> >> Are there any more things you guys know which would be needed? >> >> cheers, >> Zoltan >> >> >> On 2/22/22 12:18 PM, Peter Vary wrote: >> >> I would vote for 4.0.0-alpha-1 or similar for all of the components. >> >> When we have more stable releases I would keep the 4.x.x schema, >> >> since >> >> everyone is familiar with it, and I do not see a really good reason >> >> to >> >> change it. >> >> Thanks, >> Peter >> >> >> On 2022. Feb 10., at 3:34, Szehon Ho <szehon.apa...@gmail.com> >> >> wrote: >> >> >> +1 that would be awesome to see Hive master released after so long. >> >> Either 4.0 or 4.0.0-alpha-1 makes sense to me, not sure how we would >> >> pick >> >> any 3.x or calendar date (which could tend to slip and be more >> confusing?). >> >> Thanks in any case to get the ball rolling. >> Szehon >> >> On Wed, Feb 9, 2022 at 4:55 AM Zoltan Haindrich <k...@rxd.hu> >> >> wrote: >> >> >> Hey, >> >> Thank you guys for chiming in; versioning is for sure something we >> >> should >> >> get to some common ground. >> Its a triple problem right now; I think we have the following >> >> things: >> >> * storage-api >> ** we have "2.7.3-SNAPSHOT" in the repo >> *** >> >> >> >> https://github.com/apache/hive/blob/0d1cffffc7c5005fe47759298fb35a1c67edc93f/storage-api/pom.xml#L27 >> >> ** meanwhile we already have 2.8.1 released to maven central >> *** >> >> https://mvnrepository.com/artifact/org.apache.hive/hive-storage-api >> >> * standalone-metastore >> ** 4.0.0-SNAPSHOT in the repo >> ** last release is 3.1.2 >> * hive >> ** 4.0.0-SNAPSHOT in the repo >> ** last release is 3.1.2 >> >> Regarding the actual version number I'm not entirely sure where we >> >> should >> >> start the numbering - that's why I was referring to it as Hive-X >> >> in my >> >> first letter. >> >> I think the key point here would be to start shipping releases >> >> regularily >> >> and not the actual version number we will use - I'll kinda open to >> >> any >> >> versioning scheme which >> reflects that this is a newer release than 3.1.2. >> >> I could imagine the following ones: >> (A) start with something less expected; but keep 3 in the prefix to >> reflect that this is not yet 4.0 >> I can imagine the following numbers: >> 3.900.0, 3.901.0, ... >> 3.9.0, 3.9.1, ... >> (B) start 4.0.0 >> 4.0.0, 4.1.0, ... >> (C) jump to some calendar based version number like 2022.2.9 >> trunk based development has pros and cons...making a move like >> >> this >> >> irreversibly pledges trunk based development; and makes release >> >> branches >> >> hard to introduce >> (X) somewhat orthogonal is to (also) use some suffixes >> 4.0.0-alpha1, 4.0.0-alpha2, 4.0.0-beta1 >> this is probably the most tempting to use - but this versioning >> schema with a non-changing MINOR and PATCH number will >> also suggest that the actual software is fully compatible - and >> >> only >> >> bugs are being fixed - which will not be true... >> >> I really like the idea to suffix these releases with alpha or beta >> >> - >> >> which >> will communicate our level commitment that these are not 100% >> >> production >> >> ready artifacts. >> >> I think we could fix HIVE-25665; and probably experiment with >> 4.0.0-alpha1 >> for start... >> >> This also means there should *not* be a branch-4 after releasing >> >> Hive >> >> 4.0 >> >> and let that diverge (and becomes the next, super-ignored >> >> branch-3), >> >> correct; no need to keep a branch we don't maintain...but in any >> >> case >> >> I >> >> think we can postpone this decision until there will be something >> >> to >> >> release... :) >> >> cheers, >> Zoltan >> >> >> >> On 2/9/22 10:23 AM, L?szl? Bodor wrote: >> >> Hi All! >> >> A purely technical question: what will the SNAPSHOT version become >> >> after >> >> releasing Hive 4.0.0? I think this is important, as it defines and >> >> reflects >> >> the future release plans. >> >> Currently, it's 4.0.0-SNAPSHOT, I guess it's since Hive 3.0 + >> >> branch-3. >> >> Hive is an evolving and super-active project: if we want to make >> >> regular >> >> releases, we should simply release Hive 4.0 and bump pom to >> >> 4.1.0-SNAPSHOT, >> >> which clearly says that we can release Hive 4.1 anytime we want, >> >> without >> >> being frustrated about "whether we included enough cool stuff to >> >> release >> >> 5.0". >> >> This also means there should *not* be a branch-4 after releasing >> >> Hive >> >> 4.0 >> and let that diverge (and becomes the next, super-ignored >> >> branch-3), >> >> only >> when we end up bringing a minor backward-incompatible thing that >> >> needs a >> >> 4.0.x, and when it happens, we'll create *branch-4.0 *on demand. >> >> For >> >> me, >> >> a >> >> branch called *branch-4.0* doesn't imply either I can expect cool >> >> releases >> >> in the future from there or the branch is maintained and tries to >> >> be >> >> in >> >> sync with the *master*. >> >> Regards, >> Laszlo Bodor >> >> Alessandro Solimando <alessandro.solima...@gmail.com> ezt ?rta >> >> (id?pont: >> >> 2022. febr. 8., K, 16:42): >> >> Hello everyone, >> thank you for starting this discussion. >> >> I agree that releasing the master branch regularly and >> >> sufficiently >> >> often >> >> is welcome and vital for the health of the community. >> >> It would be great to hear from others too, especially PMC members >> >> and >> >> committers, but even simple contributors/followers as myself. >> >> Best regards, >> Alessandro >> >> On Wed, 2 Feb 2022 at 12:22, Stamatis Zampetakis < >> >> zabe...@gmail.com >> >> >> wrote: >> >> Hello, >> >> Thanks for starting the discussion Zoltan. >> >> I strongly believe that it is important to have regular and >> >> often >> >> releases >> >> otherwise people will create and maintain separate Hive forks. >> The latter is not good for the project and the community may >> >> lose >> >> valuable >> >> members because of it. >> >> Going forward I fully agree that there is no point bringing up >> >> strong >> >> blockers for the next release. For sure there are many backward >> incompatible changes and possibly unstable features but unless >> >> we >> >> get >> >> a >> release out it will be difficult to determine what is broken and >> >> what >> >> needs >> >> to be fixed. >> >> Due to the big number of changes that are going to appear in the >> >> next >> >> version I would suggest using the terms Hive X-alpha, Hive >> >> X-beta >> >> for >> >> the >> >> first few releases. This will make it clear to the end users >> >> that >> >> they >> >> need >> >> to be careful when upgrading from an older version and it will >> >> give us >> >> a >> >> bit more time and freedom to treat issues that the users will >> >> likely >> >> discover. >> >> The only real blocker that we may want to treat is HIVE-25665 >> >> [1] >> >> but >> >> we >> >> can continue the discussion under that ticket and re-evaluate if >> >> necessary, >> >> >> Best, >> Stamatis >> >> [1] https://issues.apache.org/jira/browse/HIVE-25665 >> >> >> On Tue, Feb 1, 2022 at 5:03 PM Zoltan Haindrich <k...@rxd.hu> >> >> wrote: >> >> >> Hey All, >> >> We didn't made a release for a long time now; (3.1.2 was >> >> released >> >> on >> >> 26 >> >> August 2019) - and I think because we didn't made that many >> >> branch-3 >> >> releases; not too many fixes >> were ported there - which made that release branch kinda erode >> >> away. >> >> >> We have a lot of new features/changes in the current master. >> I think instead of aiming for big feature-packed releases we >> >> should >> >> aim >> >> for making a regular release every few months - we should make >> regular >> releases which people could >> install and use. >> After all releasing Hive after more than 2 years would be big >> >> step >> >> forward >> >> in itself alone - we have so many improvements that I can't >> >> even >> >> count... >> >> >> But I may know not every aspects of the project / states of >> >> some >> >> internal >> >> features - so I would like to ask you: >> What would be the bare minimum requirements before we could >> >> release >> >> the >> >> current master as Hive X? >> >> There are many nice-to-have-s like: >> * hadoop upgrade >> * jdk11 >> * remove HoS or MR >> * ? >> but I don't think these are blockers...we can make any of these >> >> in >> >> the >> next release if we start making them... >> >> cheers, >> Zoltan >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>
