+1 for Option 1: Make IMetaStoreClient pluggable
Thanks Sungwoo,
Regarding performance testing, am I correct to assume that the "original" Hive
table is an external one?
Since Iceberg supports deletes, it might be worth comparing it against Hive
ACID. We could generate 10-20% of the updates and measure the read performance
overhead.
Additi
Hi,
I'm a bit hesitant switching to Iceberg as the default atm. I lean more toward
setting the default table format at the database level instead.
Hive Iceberg currently lacks automatic table maintenance, comprehensive support
for partition-level statistics, and various partition-aware optimiza
Hi Anmol,
Short answer it's not. The 3.x line reached end-of-life on October 8, 2024, so
you'll need to upgrade your entire stack to 4.x
Regards,
Denys
Thanks Butao!
We'll support you all the way.
ATM we should probably finalize the scope for the release (create new label
hive-4.1.0-must) and focus of getting those items prioritised.
There is also ongoing discussion on JDK-8 and JDK-17. Would be great if we
manage to add support for JDK-17 i
Note: Legacy Managed tables should be converted to External:
The following pseudo-code changes a managed table, if it is not transactional,
to external. The data and metadata is dropped when the table is dropped.
ALTER TABLE ... SET
TBLPROPERTIES('EXTERNAL'='TRUE','external.table.purge'='true')
Hi Valentine,
Hive has been enhanced to include a MANAGEDLOCATION clause to specify the
location of managed tables as shown in the following syntax:
CREATE (DATABASE|SCHEMA) [IF NOT EXISTS] database_name
[LOCATION external_table_path]
[MANAGEDLOCATION managed_table_directory_path]
[WITH D
Hi All,
It's time we start prepping for the next release.
Since Hive 4.0 was cut, a number of bug-fixes, features, and performance
improvements landed in the master.
We are looking for a volunteer (Committer / PMC) to fill the role of the
next Release Manager.
Thanks,
Denys
Hi Gabor,
It's a bit odd to get the following feedback from the Impala folks:
"I'd like to understand the motivation why this whole replication of code
happened between Iceberg and Hive."
when you know exactly why.
FYI, we've raised our concerns multiple times to the iceberg community, for
ex
Hi Peter,
Thanks for bringing it up!
I think that option 1 is the only viable solution here (remove the hive-runtime
from the iceberg repo). Main reason: lack of reviewers for things other than
Spark.
Note: need to double check, but I am pretty sure there is no difference between
Hive `iceb
Congratulations, Shohei!
HIVE-28519 still has open review comments, so I've merged the revert to unblock
other contributors.
Apologies for any inconvenience this may have caused
eberg,
enhancing performance and fixing multiple bugs.
Dmitriy, welcome, thank you for your contributions, and we look
forward to your further interactions with the community!
Please review the guidelines for new committers [1] and take
additional actions as needed.
Denys Kuzmenko (on behalf o
https://ci.hive.apache.org/job/hive-precommit/job/master/2323/testReport/
is the first build we got -60k tests.
Seems related to HIVE-28417: Bump Log4j2 to 2.24.1 to facilitate compilation of
GraalVM Native Image
We should be more careful with CVEs and dependency upgrades, they don't bring
much
Hi @lisoda
Thanks for bringing up the Ranger issue. It would be very helpful if you could
share the JIRA with the issue details. Downstream we are using a version
somewhat close to Hive-4.0 and it is fully integrated with Ranger.
Another integration issue that I am familiar is HIVE-28211, that
Thank you, Zhihua for driving this release, and a big shout-out to the Apache
Hive team and contributors for making it possible.
Keep up the great work! Next stop is JDK-17 :)
+1 (Binding)
Since the licensing concerns were addressed I would propose to return to the
original thread. I don't think it's worth the effort to create and validate a
new RC.
WDYT?
My take on this is that we do not need a new RC since the code hasn't changed,
`impl` files were removed just from packaged sources.
Maybe we could start a new vote, but there is no need for a new RC. We don't
need to repeat tests on binaries, just build from sources and do the checks.
And by
Could we regenerate just the `apache-hive-4.0.1-src.tar.gz` and update the
signature + checksum files?
Hive 3.1 is certainly the most widely adopted release line, however, we do not
see any activity on this line, even in terms of providing security fixes and
CVE backports. Hive 3.1.3, the most recent release of the 3.x, was released on
2022-06-17.
We don't want to scatter our resources so we foc
+1 (Binding)
Thanks Zhihua for driving the release!
1. Verified the checksums and signatures: ✓ OK
2. Initialized the schema for MySQL: ✓ OK
3. Validated schema upgrade from 3.1.0 to 4.0.1: ✓ OK
4. Deployed the binaries and started HS2 and HMS with Hadoop 3.3.6 and Tez
0.10.4: ✓ OK
5. Executed I
./schematool -dbType mysql -initSchema
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/Users/denyskuzmenko/Downloads/apache-hive-4.0.0-bin/lib/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/Users/deny
hi Avinash,
try below configs in HMS hive-site.xml:
metastore.server.thrift.transport.mode=http
metastore.client.thrift.transport.mode=http
metastore.authentication.jwt.jwks.url=...
metastore.authentication=JWT
metastore.client.auth.mode=JWT
Regards,
Denys
I think that could be helpful if we could consolidate existing Tez
documentation (wiki pages) and migrate into the Tez site space.
+1 on simple getting started, as it's the first place new users would check
Also few additional resource might be added into the user guides:
1. https://blog.cloude
Just a few known issues in master:
https://issues.apache.org/jira/issues/?jql=labels%20%3D%20hive-4.1.0-must%20and%20resolution%20%3D%20Unresolved
To get a better view of things, we need to execute TPC-DS workload against
branch-4.0.1 and master.
Thanks, Laszlo!
+ 1 on tracking every incoming report using JIRA. Currently, it's unclear
what's pending or what's the status of a particular CVE.
Hi Okumin,
If you have a list, I can update the wiki or grant you the edit permissions.
Hi Stamatis,
`It's been already 4 months since the release of Hive 4.0.0`. That's true,
Zhihua can give a more detailed view of things, however, the main reason is
that it took a while to resolve the identified bugs.
ATM the only blocking item is TEZ-4557, which might not be available until
Se
Congrats Stephen!
On Thu, Jul 18, 2024 at 7:01 AM Akshat m wrote:
> Congratulations Stephen !
>
> Regards
> Akshat
>
> On Thu, Jul 18, 2024 at 8:07 AM kokila narayanan <
> kokilanarayana...@gmail.com> wrote:
>
>> Congratulations Stephen !!
>>
>> Regards,
>> Kokila N
>>
>> On Thu, 18 Jul, 2024, 0
jackson-databind:2.9.4 comes from Calcite avatica:1.12.0 shaded jar:
https://mvnrepository.com/artifact/org.apache.calcite.avatica/avatica/1.12.0
that jar has also reported vulnerability :CVE-2022-36364, we should try to
upgrade it.
Another one is htrace-core:3.1.0-incubating from accumulo-core:
Hi Sreek,
Oh, thanks! Ideally docker image should be build from Hive-4.0 branch artifacts
via the GH action. Let me check, I just hope it wasn't manually uploaded
Hi,
Hive-4.0 use jackson-databind version 2.16.1. I don't see any CVEs reported in
maven central for that artifact:
https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind/2.16.1
com.fasterxml.jackson
jackson-bom
2.16.1
pom
import
Are you using MR execution mode? it was deprecated, see warning
"Hive-on-MR is deprecated in Hive 2 and may not be available in the future
versions. Consider using a different execution engine"
On 2024/06/19 04:59:22 张三 wrote:
> -- Forwarded message -
> 发件人： 张三
> Date: 2024年6月19日
+1 (binding)
I agree that shaded hive-exec should be the proper way to go, however, ATM it's
a show-stopper for many downstream projects to upgrade.
Also based on the mail threads, they clearly understand the risks of using an
unshaded jar but still insist on keeping it.
If we'd like to improve the project
Just found out that Amoro project is also using hive-exec:jar:core
+1 to restore
Would we fix the problem by relocating just guava and joda-time?
Here is how it's done in Impala:
https://github.com/apache/impala/blob/master/java/shaded-deps/hive-exec/pom.xml#L70-L77
d happy to help as much as I can :)
>
> Best,
> Stamatis
>
> On Thu, Apr 18, 2024 at 11:53 AM Denys Kuzmenko wrote:
> >
> > Hi Stamatis,
> >
> > That is the standard practice to create minor version release for bugfixes.
> > Many upstream projec
t; >> would be 4.1.0 and that it would be cut from master. I would like to
> >> understand how 4.0.1 is different and if it is, what is the
> >> contribution pattern that contributors and committers should follow?
> >> If the idea is to maintain and commit in t
Hi Ravi,
hive-exec is the uber jar, the same thing is done in Hive-3.x. We are
considering restoring the hive-core jar that only has classes from ql
On 2024/04/16 06:24:20 Mergu Ravi wrote:
> Hi Hive Team,
>
> I'm currently using the Hive 4.0.0 libraries and facing an issue with the
> hive-exec
We might need it sooner as identified some critical issues in the recent code:
1. HIVE-28166: Truncate on Iceberg table disregards the branch name and
operates on a main;
2. HIVE-28190: Materialized view rebuild lock heart-beating is broken;
Looks great, thanks Ayush!
Hi Henri,
The idea to provide a simple, open standard interface to enable integration of
3rd party compute engines and tools is great, however, there are concerns about
the suggested design.
-1 on the current implementation (+1 more Jetty server on HMS instance)
org.apache.iceberg.rest.HMSCata
The Apache Hive team is proud to announce the release of Apache Hive
version 4.0.0.
The Apache Hive (TM) data warehouse software facilitates querying and
managing large datasets residing in distributed storage. Built on top
of Apache Hadoop (TM), it provides, among others:
* Tools to enable easy
Thanks to everyone who verified the release candidate and gave their
comments and votes.
Voting has closed and the results are as follows:
4 binding +1s:
* Ayush Saxena
* Denys Kuzmenko
* Marta Kuczora
* Krisztian Kasa
1 +0
* Stamatis Zampetakis
7 non-binding +1s:
* Butao Zhang
+1 (binding)
* Executed functional test suite against Postgres DB
Hi Everyone,
We would like to thank everyone who has contributed to the project and request
the Hive PMC members to review and vote on this new release candidate.
Apache Hive 4.0.0 RC-0 artifacts are available here:*
https://people.apache.org/~dkuzmenko/apache-hive-4.0.0-rc0/
The checksums are
Hi Pravin,
TPC-DS tracking Jira: HIVE-27929.
The functional test run is completed, waiting for the TPC-DS execution. There
were some glitches with the infra last week, expecting results in the next 2-3
days. After that, we should be ready with RC.
Regards,
Denys
---
> > | From | Attila Turoczy |
> > | Date | 11/23/2023 22:29 |
> > | To | |
> > | Subject | Re: Release of Hive 4 and TPC-DS benchmark |
> > Excellent new Denys! Hive 4 is here! Can't wait :)
> >
> > -Attila
> >
> >
> > On Thu,
Hi Ravi,
ATM we have 3 items to resolve before the release [1].
1. HIVE-27924 will be merged in a couple of days;
2. HIVE-27775 is under review;
3. HIVE-27858 is under investigation, however, we might exclude it from 4.0
release;
In parallel, we are running TPC-DS benchmarks.
Optimistically,
+1
+1 (binding)
+1 for marking Hive 1.x EOL
Assuming no volunteers willing to take ownership of branch-2 maintenance, +1 to
declare it EOL as well.
Regards,
Denys
Hi All,
Hive 4.0.0 release branch cut is happening today.
Best,
Denys
Update:
1. Query 2, 71: Resolved in HIVE-27006 [1];
2. Query 97: Under review in HIVE-27269 [2];
Thanks, Seonggon for providing a reproduce qfile.
3. Query 14: Reported in HIVE-24167 [3];
set hive.optimize.cte.materialize.threshold to -1 by default in Hive 4 and fix
it in the next versions
4.
Congrats, very well deserved!
Hi Sungwoo,
Thank you for bringing this up again.
Based on HIVE-26654, it looks like we have 3 PR pending review:
1. HIVE-26986 - Query 71
2. HIVE-27006 - Query 2
3. HIVE-27269 - Query 97 (is that ready to be reviewed?)
We'll prioritize those.
For query 14, as you suggested, we might set
`hive
Instead of adding the dependencies, can't we add the possibility to include
jdbc jars in the classpath?
something like this:
org.apache.maven.plugins
maven-failsafe-plugin
integration-test
verify
${itest.jdbc.jars}
+1
* Verified signatures and checksum;
* Checked binary content and successfully built from the source;
* Skimmed through the release notes;
* Initialized backend DB schema and launched HMS & HS2 locally;
* Conducted basic checks via beeline:
- Created a few ACID & Iceberg tables and loaded da
I’ll second the above. We already have a number of flaky tests that would go
undetected and just increase the build time.
Nice, tnx Ayush!
great, this is the current list of release managers:
4.0.0 Stamatis Zampetakis
4.1.0 Denys Kuzmenko
4.2.0 Sai Hemanth Gantasala
Should we keep the same RM order and just shift the releases or find a
volunteer for the 4.0.0-beta release, WDYT?
Hi folks,
The master branch has many new features, bug fixes, and performance
improvements since alpha-2. However, we still have several correctness bugs
[HIVE-26654] and performance issues that should be eliminated before the GA.
Could we consider doing a beta release to keep at least a 6-mon
Thanks, Sungwoo for running the TPC-DS benchmark. Do we know if the same level
of performance degradation was present in 4.0.0-alpha1?
All: please use the `hive-4.0.0-must` label in a ticket if you think it's a
show-stopper for the release.
Hi, I can take the following one: 4.1.0
Sorry for being late to the party. I think what Kirti proposes would be good
for the project and end-users. As mentioned, we could start with 2-3 releases
per year, and once we improve on the process and automation (CI/CD) we could
reevaluate.
Denys Kuzmenko created HIVE-27160:
-
Summary: Iceberg: Optimise delete (entire) data from table
Key: HIVE-27160
URL: https://issues.apache.org/jira/browse/HIVE-27160
Project: Hive
Issue Type
Denys Kuzmenko created HIVE-27155:
-
Summary: Iceberg: Vectorize virtual columns
Key: HIVE-27155
URL: https://issues.apache.org/jira/browse/HIVE-27155
Project: Hive
Issue Type: Task
Denys Kuzmenko created HIVE-26995:
-
Summary: Iceberg: Enhance time travel syntax with expressions
Key: HIVE-26995
URL: https://issues.apache.org/jira/browse/HIVE-26995
Project: Hive
Issue
Denys Kuzmenko created HIVE-26873:
-
Summary: Whiltelist iceberg configs for sql std authorization
Key: HIVE-26873
URL: https://issues.apache.org/jira/browse/HIVE-26873
Project: Hive
Issue
Denys Kuzmenko created HIVE-26811:
-
Summary: SharedWorkOptimizer: take the union of virtual columns in
mergeable TableScan
Key: HIVE-26811
URL: https://issues.apache.org/jira/browse/HIVE-26811
Hi Cameron,
That's fine, however, I usually contribute to the original PR.
You could download github cli and checkout the branch via the following
command:
`gh pr checkout `, in your case it was 3534.
Unfortunately, Epic is not on the list of supported issue types for the Hive
project. I woul
Hi Cameron, Rohan,
Are you still planning to contribute or need some help/support in terms of code
review/etc.
I've checked Rohan's PR, it looks good, but requires a rebase.
Please let us know how we could help.
Kind regards,
Denys
The Apache Hive team is proud to announce the release of Apache Hive
version 4.0.0-alpha-2
The Apache Hive (TM) data warehouse software facilitates querying and
managing large datasets residing in distributed storage. Built on top
of Apache Hadoop (TM), it provides, among others:
* Tools to enabl
Hi Stamatis,
Thanks for bringing it up! +1 to implement the same process
> > >> $ gpg --verify apache-hive-4.0.0-alpha-2-bin.tar.gz.asc
> > >> apache-hive-4.0.0-alpha-2-bin.tar.gz
> > >> gpg: Signature made Mon 7 Nov 19:04:05 2022 CET
> > >> gpg:using RSA key
> 50606DE1BDBD5CF862A595A907C5682DAFC73125
> &g
UPD: Voting will conclude in 1 week (Monday 14th).
On Mon, Nov 7, 2022 at 7:57 PM Denys Kuzmenko
wrote:
> Hi team,
>
> Let's give it 1 more chance.
>
> Apache Hive 4.0.0-alpha-2 Release Candidate 1 is available
> here:https://people.apache.org/~dkuzmenko/re
Hi team,
Let's give it 1 more chance.
Apache Hive 4.0.0-alpha-2 Release Candidate 1 is available
here:https://people.apache.org/~dkuzmenko/release-4.0.0-alpha-2-rc1/
Change log:
- Updated copyright notices (year of distribution was taken from
`project.build.outputTimestamp` defined in apache r
Denys Kuzmenko created HIVE-26704:
-
Summary: Cleaner shouldn't be blocked by global min open txnId
Key: HIVE-26704
URL: https://issues.apache.org/jira/browse/HIVE-26704
Project: Hive
Hi team,
Thank you for taking time to verify this RC!
Unfortunately, we didn't get enough votes to go ahead with the release.
Closing this vote as unsuccessful.
Kind regards,
Denys
sn't updated in alpha-1 is most likely an
> oversight.
>
> Having said that the final decision is up to the release manager.
>
> Best,
> Stamatis
>
> On Fri, Oct 28, 2022 at 1:57 PM Denys Kuzmenko
> wrote:
>
> > Hi Stamatis,
> >
> > My bad, sorr
4.0.0-alpha-1 [1] and you migrated to Hive 3.x, Then how did you face
> > this problem? Some patched version of Hive, or some missing FixVersion in
> > the Jira?
> >
> > Good to mention the version post which the change in prefix happens in
> the
> > wiki as w
ct 28, 2022 at 10:32 AM Ayush Saxena wrote:
>
> > +1 (non-binding)
> > * Built from source.
> > * Verified Checksums.
> > * Verified Signatures
> > * Ran some basic unit tests.
> > * Ran some basic ACID & Iceberg related queries with Tez.
> > * Skim
Hi Chris,
Please try now
On Fri, Oct 28, 2022 at 12:43 PM Stamatis Zampetakis
wrote:
> Hive PMC members can provide edit rights to the wiki.
>
> @Naveen, Dennys, Adam: Can someone please give write privileges to Chris?
>
> Best,
> Stamatis
>
> On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth wrot
t; > Hi everyone,
> > > >
> > > > unfortunately my vote is -1 (although non-binding) due to a classpath
> > > error
> > > > which prevents queries involving Tez to complete (all the details at
> > the
> > > > end of the email, apologies f
; Alessandro
>
> On Thu, 27 Oct 2022 at 15:22, Denys Kuzmenko .invalid>
> wrote:
>
> > Hi Adam,
> >
> > Thanks for pointing that out! Upstream release guide is outdated. Once I
> > receive the edit rights, I'll amend the instructions.
> > Updated t
g was missing
> during build, can you please rebuild?
>
> Thanks,
> Adam
>
> On 2022/10/25 11:20:23 Denys Kuzmenko wrote:
> > Hi team,
> >
> >
> > Apache Hive 4.0.0-alpha-2 Release Candidate 0 is available
> > here:https://people.apache.org/~dk
Hi team,
Apache Hive 4.0.0-alpha-2 Release Candidate 0 is available
here:https://people.apache.org/~dkuzmenko/release-4.0.0-alpha-2-rc0/
The checksums are these:
- 7d4c54ecfe2b04cabc283a84defcc1e8a02eed0e13baba2a2c91ae882b6bfaf7
apache-hive-4.0.0-alpha-2-bin.tar.gz
- 8c4639915e9bf649f4a55cd9adb
ra to avoid issues with building Release
> > Notes for our next 4.0.0 release
> >
> > A simple INFRA ticket or anyone with Hive Jira Admin rights should be
> able
> > to do so.
> >
> > [1]
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%2
Hi Team,
Branching for Hive *4.0.0-alpha-2* was done today:
https://github.com/apache/hive/tree/branch-4.0.0-alpha-2
The next development version is *4.0.0-SNAPSHOT*
*alpha-2 *branch would be open for commits until Monday. If you would like
to include something major in it, but need more time, p
Denys Kuzmenko created HIVE-26645:
-
Summary: Preparing for 4.0.0-GA development
Key: HIVE-26645
URL: https://issues.apache.org/jira/browse/HIVE-26645
Project: Hive
Issue Type: Task
Denys Kuzmenko created HIVE-26487:
-
Summary: Auto-compaction may hang or skip some of the eligible
items
Key: HIVE-26487
URL: https://issues.apache.org/jira/browse/HIVE-26487
Project: Hive
Denys Kuzmenko created HIVE-26433:
-
Summary: HivePrivilegeObject's objectName is NULL when
JdbcStorageHandler is used with METASTORE type
Key: HIVE-26433
URL: https://issues.apache.org/jira/browse/HIVE-
Denys Kuzmenko created HIVE-26416:
-
Summary: AcidUtils$MetaDataFile.isRawFormatFile() throws
InvalidProtocolBufferException for non-ORC file
Key: HIVE-26416
URL: https://issues.apache.org/jira/browse/HIVE-26416
Denys Kuzmenko created HIVE-26293:
-
Summary: Migrate remaining exclusive DDL operations to EXCL_WRITE
lock & bug fixes
Key: HIVE-26293
URL: https://issues.apache.org/jira/browse/HIVE-26293
Pro
Denys Kuzmenko created HIVE-26252:
-
Summary: Missing locks in case of MERGE with multiple branches
Key: HIVE-26252
URL: https://issues.apache.org/jira/browse/HIVE-26252
Project: Hive
Issue
Denys Kuzmenko created HIVE-26165:
-
Summary: Remove READ locks for ACID tables with SoftDelete enabled
Key: HIVE-26165
URL: https://issues.apache.org/jira/browse/HIVE-26165
Project: Hive
Denys Kuzmenko created HIVE-26149:
-
Summary: Non blocking DROP DATABASE implementation
Key: HIVE-26149
URL: https://issues.apache.org/jira/browse/HIVE-26149
Project: Hive
Issue Type: Task
1 - 100 of 361 matches
Mail list logo
