[jira] [Created] (HADOOP-19593) S3A: tests against third party stores failing after latest update & conditional

2025-06-20 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19593: --- Summary: S3A: tests against third party stores failing after latest update & conditional Key: HADOOP-19593 URL: https://issues.apache.org/jira/browse/HADOOP-1

[jira] [Created] (HADOOP-19592) S3A: S3 Express bucket failure of conditional overwrite of multiparts is 200 + error, not 412

2025-06-19 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19592: --- Summary: S3A: S3 Express bucket failure of conditional overwrite of multiparts is 200 + error, not 412 Key: HADOOP-19592 URL: https://issues.apache.org/jira/browse/HADOOP

[jira] [Resolved] (HADOOP-19583) JDK8: RawLocalFileSystem calls ByteBuffer.flip

2025-06-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19583. - Resolution: Won't Fix too many other code snippets do this; need a large PR, or

[jira] [Resolved] (HADOOP-19353) Über-jira: S3A Hadoop 3.4.2 features

2025-06-16 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19353. - Resolution: Fixed > Über-jira: S3A Hadoop 3.4.2 featu

[jira] [Reopened] (HADOOP-19353) Über-jira: S3A Hadoop 3.4.2 features

2025-06-16 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-19353: - > Über-jira: S3A Hadoop 3.4.2 featu

[jira] [Created] (HADOOP-19586) Über-jira: S3A Hadoop 3.4.3 features

2025-06-16 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19586: --- Summary: Über-jira: S3A Hadoop 3.4.3 features Key: HADOOP-19586 URL: https://issues.apache.org/jira/browse/HADOOP-19586 Project: Hadoop Common Issue

[jira] [Resolved] (HADOOP-19353) Über-jira: S3A Hadoop 3.4.2 features

2025-06-16 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19353. - Fix Version/s: 3.4.2 Resolution: Fixed > Über-jira: S3A Hadoop 3.4.2 featu

Re: [VOTE] Release Apache Hadoop 3.4.2

2025-06-11 Thread Steve Loughran
this is my PR for parquet buffers https://github.com/apache/hadoop/pull/7732 doesn't do any complex work, just * adds a capability to declare buffers may be sliced * adds a switch to turn verification off for file:// * adds tests to show that slicing may happen * docs For Parquet I'm fixing the t

[jira] [Resolved] (HADOOP-19295) S3A: fs.s3a.connection.request.timeout too low for large uploads over slow links

2025-06-11 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19295. - Fix Version/s: 3.4.2 Resolution: Fixed > S3A: fs.s3a.connection.request.time

[jira] [Reopened] (HADOOP-18296) Memory fragmentation in ChecksumFileSystem Vectored IO implementation.

2025-06-09 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-18296: - Assignee: Steve Loughran > Memory fragmentation in ChecksumFileSystem Vectored

Re: [VOTE] Release Apache Hadoop 3.4.2

2025-06-06 Thread Steve Loughran
regarding parquet, the jira there is : https://github.com/apache/parquet-java/issues/3237 and while the change in that lib is "release the buffers", it hits a problem with coalesced reads as there the releasing doesn't work. my fix will be: disable coalescing, keep the code (for now), and add a p

Re: [VOTE] Release Apache Hadoop 3.4.2

2025-06-04 Thread Steve Loughran
the overloaded ByteByffer.flip() method, which is absent from very old JDK releases. If you encounter the error " java.lang.NoSuchMethodError: java.nio.ByteBuffer.flip()Ljava/nio/ByteBuffer;" you need to upgrade to a more recent version of java 8 or later. On Wed, 4 Jun 2025 at 16:45, Steve Lo

[jira] [Created] (HADOOP-19583) JDK8: RawLocalFileSystem calls ByteBuffer.flip

2025-06-04 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19583: --- Summary: JDK8: RawLocalFileSystem calls ByteBuffer.flip Key: HADOOP-19583 URL: https://issues.apache.org/jira/browse/HADOOP-19583 Project: Hadoop Common

Re: [VOTE] Release Apache Hadoop 3.4.2

2025-06-04 Thread Steve Loughran
a.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) this can actually be fixed trivially, casting to java.io.Buffer first: ((Byffer)buff).flip()) I'll do this On Tue, 3 Jun 2025 at 16:04, Steve Loughran wrote: > >

Re: [VOTE] Release Apache Hadoop 3.4.2

2025-06-03 Thread Steve Loughran
t's indicative of a memory leak. Surprised and Annoyed I hadn't spotted this before...the tests went in last year. On Tue, 3 Jun 2025 at 10:55, Steve Loughran wrote: > > Ahmar, no need for prs there, it's a "commit then review" repo, as in > other people can

Re: [VOTE] Release Apache Hadoop 3.4.2

2025-06-03 Thread Steve Loughran
t; https://github.com/apache/hadoop-release-support/pull/4 > > > Masatake - Yes, sounds good, will update the documentation for the new RC > if it is created. > > > Mukund - I realised too late that I messed up the commit for that one. > Will figure out how to fix.. > >

[jira] [Reopened] (HADOOP-19516) S3A: SDK reads content twice during PUT to S3 Express store.

2025-06-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-19516: - Assignee: (was: Steve Loughran) > S3A: SDK reads content twice during PUT to

Re: [VOTE] Release Apache Hadoop 3.4.2

2025-06-02 Thread Steve Loughran
can you put up the relevant changes to the hadoop-release-support module; I'd like to use it as part of my validation and I'm assuming you have that src/releases/release-info-3.4.2.properties file On Wed, 28 May 2025 at 13:25, Ahmar Suhail wrote: > Hey all, > > The first release candidate for Ha

[jira] [Created] (HADOOP-19581) Update PR template to ask about AI controbution

2025-05-29 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19581: --- Summary: Update PR template to ask about AI controbution Key: HADOOP-19581 URL: https://issues.apache.org/jira/browse/HADOOP-19581 Project: Hadoop Common

[jira] [Resolved] (HADOOP-19567) S3A: error stack traces printed on analytics stream factory close

2025-05-26 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19567. - Resolution: Fixed > S3A: error stack traces printed on analytics stream factory cl

[jira] [Resolved] (HADOOP-19259) upgrade to jackson 2.14.3

2025-05-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19259. - Fix Version/s: 3.5.0 Assignee: PJ Fanning Resolution: Fixed > upgr

[jira] [Created] (HADOOP-19573) S3A: ITestS3AConfiguration.testDirectoryAllocatorDefval() faiilng

2025-05-20 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19573: --- Summary: S3A: ITestS3AConfiguration.testDirectoryAllocatorDefval() faiilng Key: HADOOP-19573 URL: https://issues.apache.org/jira/browse/HADOOP-19573 Project

[jira] [Resolved] (HADOOP-19554) LocalDirAllocator still doesn't always recover from directory tree deletion

2025-05-19 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19554. - Fix Version/s: 3.4.3 3.5.0 Resolution: Fixed

[jira] [Created] (HADOOP-19569) S3A: NPE in S3AFileSystem.lambda$abortMultipartUpload fs closed and multipart upload in progress

2025-05-14 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19569: --- Summary: S3A: NPE in S3AFileSystem.lambda$abortMultipartUpload fs closed and multipart upload in progress Key: HADOOP-19569 URL: https://issues.apache.org/jira/browse

[jira] [Created] (HADOOP-19567) error stack traces printed on analytics stream factory close

2025-05-13 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19567: --- Summary: error stack traces printed on analytics stream factory close Key: HADOOP-19567 URL: https://issues.apache.org/jira/browse/HADOOP-19567 Project: Hadoop

[jira] [Resolved] (HADOOP-19557) S3A: S3ABlockOutputStream to never log/reject hflush(): calls

2025-05-01 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19557. - Fix Version/s: 3.5.0 3.4.2 Resolution: Fixed >

[jira] [Created] (HADOOP-19557) S3A: S3ABlockOutputStream to never log/reject hflush(): calls

2025-04-29 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19557: --- Summary: S3A: S3ABlockOutputStream to never log/reject hflush(): calls Key: HADOOP-19557 URL: https://issues.apache.org/jira/browse/HADOOP-19557 Project

[jira] [Created] (HADOOP-19554) LocalDirAllocator doesn't always recover from directory tree deletion

2025-04-25 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19554: --- Summary: LocalDirAllocator doesn't always recover from directory tree deletion Key: HADOOP-19554 URL: https://issues.apache.org/jira/browse/HADOOP-

[jira] [Resolved] (HADOOP-19485) S3A: Upgrade AWS V2 SDK to 2.29.52

2025-04-25 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19485. - Fix Version/s: 3.5.0 3.4.2 Resolution: Fixed > S3A: Upgr

[jira] [Resolved] (HADOOP-19527) S3A: testVectoredReadAfterNormalRead() failing with 412 response from S3

2025-04-25 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19527. - Fix Version/s: 3.5.0 3.4.2 Resolution: Fixed >

[jira] [Resolved] (HADOOP-19523) Upgrade to hadoop-thirdparty 1.4.0

2025-04-25 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19523. - Fix Version/s: 3.4.2 Resolution: Fixed > Upgrade to hadoop-thirdparty 1.

Re: [DISCUSS] Optimize the invoke retry of async repsonder when some nameservices are slow

2025-04-23 Thread Steve Loughran
think you meant to send this to hdfs-dev, not the -subscribe list. On Wed, 16 Apr 2025 at 14:09, Zhanghaobo wrote: > Hello, everyone, so sorry to bother you and I would like to discuss how > to optimize the invoke retry logic of async repsonder when some > nameservices are slow. > Currently,

Re: HADOOP-19343 feature branch: GCS FileSystem

2025-04-23 Thread Steve Loughran
I like it, even if gcs want to be more agile at first 1. can help testing/debugging against a broadly used store 2. stops us accidentally breaking things (who would do that 🙂) 3. makes it easier to do changes across the modules Because Chris is involved, I'm not worried about this being

upgrai

2025-04-16 Thread Steve Loughran
FYI, I'm backporting HADOOP-19315, Bump avro from 1.9.2 to 1.11.4 to branch-3.4 https://github.com/apache/hadoop/pull/7615# it means there's binary incompatibility with downstream apps using avro 1.9.2 *but the avro bug is so critical they need to stop doing that ASAP* Our release notes will ha

Upgrading Avro on branch-3.4 to one that doesn't do RCEs

2025-04-15 Thread Steve Loughran
I've now got enough knowledge of how vulnerable avro (and to a very much lower extent parquet) is to RCEs from malicious files which trigger classloading. We need to fix this on branch-3.4, even though it means that the release is somewhat incompatible with older versions. https://issues.apache.o

[jira] [Resolved] (HADOOP-19256) S3A: Support S3 Conditional Writes

2025-04-11 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19256. - Fix Version/s: 3.5.0 3.4.2 Resolution: Fixed > S3A: Supp

[jira] [Resolved] (HADOOP-19455) HADOOP-19455. S3A: Enable logging of SDK client metrics

2025-04-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19455. - Fix Version/s: 3.5.0 Resolution: Fixed > HADOOP-19455. S3A: Enable logging

[jira] [Created] (HADOOP-19523) Upgrade to hadoop-thirdparty 1.4.0

2025-04-05 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19523: --- Summary: Upgrade to hadoop-thirdparty 1.4.0 Key: HADOOP-19523 URL: https://issues.apache.org/jira/browse/HADOOP-19523 Project: Hadoop Common Issue

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

2025-04-05 Thread Steve Loughran
gelogs look good. > > > > > > On Fri, Mar 21, 2025 at 4:35 PM Chris Nauroth > wrote: > > > > > > > +1 > > > > > > > > * Verified signatures. > > > > * LICENSE-binary has the expected dependencies and versions. > > >

[jira] [Created] (HADOOP-19527) S3A: testVectoredReadAfterNormalRead() failing with 412 response from S3

2025-04-05 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19527: --- Summary: S3A: testVectoredReadAfterNormalRead() failing with 412 response from S3 Key: HADOOP-19527 URL: https://issues.apache.org/jira/browse/HADOOP-19527

[jira] [Created] (HADOOP-19512) S3A: analytics acclerator vector IO test skip logic incomplete

2025-04-05 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19512: --- Summary: S3A: analytics acclerator vector IO test skip logic incomplete Key: HADOOP-19512 URL: https://issues.apache.org/jira/browse/HADOOP-19512 Project

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

2025-04-04 Thread Steve Loughran
t; wrote: > > > >> I just did > >> gpg --verify .tar.gz.asc .tar.gz > >> > >> Ok after importing hadoop KEYS > >> <https://downloads.apache.org/hadoop/common/KEYS>, this is the output > >> which > >> looks good to me: > >>

[jira] [Resolved] (HADOOP-19492) S3A: some tests failing on third-party stores

2025-04-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19492. - Fix Version/s: 3.5.0 3.4.2 Resolution: Duplicate > S3A: s

[jira] [Resolved] (HADOOP-19512) S3A: Test failures during AWS SDK upgrade

2025-04-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19512. - Fix Version/s: 3.5.0 3.4.2 Assignee: Steve Loughran (was

[jira] [Created] (HADOOP-19524) bulk delete must fully qualify paths before validating them

2025-03-31 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19524: --- Summary: bulk delete must fully qualify paths before validating them Key: HADOOP-19524 URL: https://issues.apache.org/jira/browse/HADOOP-19524 Project: Hadoop

[jira] [Resolved] (HADOOP-19080) S3A to support writing to object lock buckets

2025-03-31 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19080. - Fix Version/s: 3.5.0 Resolution: Fixed > S3A to support writing to object l

[ANNOUNCE] Apache Hadoop Thirdparty 1.4.0 release

2025-03-27 Thread Steve Loughran
On behalf of the Apache Hadoop Project Management Committee, I am pleased to announce the release of Apache Hadoop Thirdparty 1.4.0. For the list of changes, see https://issues.apache.org/jira/browse/HADOOP-19483 Many thanks to everyone who helped in this release. steve

[jira] [Resolved] (HADOOP-19483) Release Hadoop Third-Party 1.4.0

2025-03-27 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19483. - Resolution: Fixed > Release Hadoop Third-Party 1.

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

2025-03-27 Thread Steve Loughran
The time is up, the votes are in and they are +1 (binding) Ayush Saxena (binding) Chris Nauroth (binding) Steve Loughran (binding) Wei-Chiu Chuang (binding) Xiaoqiao He (binding) = 0: none -1: none Accordingly, the RC is the release; I will do the next steps in the process ASAP Steve On Tue

[jira] [Created] (HADOOP-19516) S3A: SDK reads content twice during PUT to S3 Express store.

2025-03-26 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19516: --- Summary: S3A: SDK reads content twice during PUT to S3 Express store. Key: HADOOP-19516 URL: https://issues.apache.org/jira/browse/HADOOP-19516 Project: Hadoop

[jira] [Created] (HADOOP-19514) SecretManager logs at info in bin/hadoop calls

2025-03-25 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19514: --- Summary: SecretManager logs at info in bin/hadoop calls Key: HADOOP-19514 URL: https://issues.apache.org/jira/browse/HADOOP-19514 Project: Hadoop Common

[jira] [Created] (HADOOP-19511) S3A: AWS4SignerType failing

2025-03-24 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19511: --- Summary: S3A: AWS4SignerType failing Key: HADOOP-19511 URL: https://issues.apache.org/jira/browse/HADOOP-19511 Project: Hadoop Common Issue Type: Task

[VOTE] Release Apache Hadoop Thirdparty 1.4.0

2025-03-21 Thread Steve Loughran
Apache Hadoop-Thirdparty 1.4.0 I have built a release candidate (RC1) for Hadoop-Thirdparty 1.4.0. The RC is available at: https://dist.apache.org/repos/dist/dev/hadoop/thirdparty-1.4.0-RC1/ The git tag is release-1.4.0-RC1, and commit a1eda71bbfa630138b97ecac1a977c18d05f0003 The maven artifact

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

2025-03-17 Thread Steve Loughran
GPG key expired? > > gpg: Signature made Fri Mar 14 08:56:00 2025 PDT > gpg:using RSA key 38237EE425050285077DB57AD22CF846DBB162A0 > gpg: Good signature from "Steve Loughran (ASF code sign key - 2018) < > ste...@apache.org>" [expired] > gpg:

Re: [VOTE] Release Apache Hadoop Thirdparty 1.4.0

2025-03-17 Thread Steve Loughran
Mar 2025 at 21:46, Wei-Chiu Chuang wrote: > Looks like your GPG key expired? > > gpg: Signature made Fri Mar 14 08:56:00 2025 PDT > gpg:using RSA key 38237EE425050285077DB57AD22CF846DBB162A0 > gpg: Good signature from "Steve Loughran (ASF code sign key - 2018) &

[jira] [Resolved] (HADOOP-18544) S3A: add option to disable probe for dir marker recreation on delete/rename.

2025-03-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18544. - Resolution: Won't Fix Harshit tried this and too many things broke. pity >

[VOTE] Release Apache Hadoop Thirdparty 1.4.0

2025-03-14 Thread Steve Loughran
I have built a release candidate (RC0) for Hadoop-Thirdparty 1.4.0. The RC is available at: https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.4.0-RC0/ The git tag is release-1.4.0-RC0, and commit 5595f1357eaad02e1cfb660bcce7fd34515197ff The maven artifacts are staged at https://r

Re: Dropping support for builds on Centos 7

2025-03-14 Thread Steve Loughran
I think we really need to push ahead on a 3.5 release for this stuff and for java 17. I'm doing the new thirdparty jar once I've got the docker building *and signing* it...I'm having to do the build in EC2 as it doesn't seem to work on my arm64 laptop right now. Then ahmar can start the 3.4.2 rel

[jira] [Resolved] (HADOOP-19300) upgrade hadoop thirdparty avro to 1.11.4

2025-03-13 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19300. - Fix Version/s: thirdparty-1.4.0 Resolution: Fixed > upgrade hadoop thirdpa

[jira] [Created] (HADOOP-19492) S3A: ITestS3AEndpointRegion failing on third-party stores

2025-03-11 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19492: --- Summary: S3A: ITestS3AEndpointRegion failing on third-party stores Key: HADOOP-19492 URL: https://issues.apache.org/jira/browse/HADOOP-19492 Project: Hadoop

[jira] [Resolved] (HADOOP-19496) Release hadoop-thirdparty 1.4.0

2025-03-11 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19496. - Resolution: Duplicate > Release hadoop-thirdparty 1.

[jira] [Created] (HADOOP-19496) Release hadoop-thirdparty 1.4.0

2025-03-11 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19496: --- Summary: Release hadoop-thirdparty 1.4.0 Key: HADOOP-19496 URL: https://issues.apache.org/jira/browse/HADOOP-19496 Project: Hadoop Common Issue Type

[jira] [Created] (HADOOP-19490) S3A: AWS SDK 2.30+ incompatible with third party stores

2025-03-10 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19490: --- Summary: S3A: AWS SDK 2.30+ incompatible with third party stores Key: HADOOP-19490 URL: https://issues.apache.org/jira/browse/HADOOP-19490 Project: Hadoop

[jira] [Resolved] (HADOOP-19086) Update commons-logging to 1.3.0

2025-03-07 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19086?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19086. - Fix Version/s: 3.5.0 3.4.2 Resolution: Fixed > Upd

Re: hadoop thirdparty 1.4.0

2025-03-07 Thread Steve Loughran
t version of the plugin. Let's see what happens. Maybe we will need to do dependency checks on java11 and releases on java8 Whatever the outcome, I will put up the RC tonight. Hadoop does build with it... I've not tested it though > On Wed, 5 Mar 2025 at 20:41, Steve Loughran > wr

[jira] [Resolved] (HADOOP-18579) Warn when no region is configured

2025-03-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18579. - Resolution: Won't Fix > Warn when no region is co

[jira] [Resolved] (HADOOP-16945) log accepted/rejected fs.s3a.authoritative.path paths @ debug

2025-03-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16945. - Resolution: Won't Fix > log accepted/rejected fs.s3a.authoritative.path paths

[jira] [Created] (HADOOP-19484) S3A: disable strong integrity checksum feature from recent AWS SDKs

2025-03-05 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19484: --- Summary: S3A: disable strong integrity checksum feature from recent AWS SDKs Key: HADOOP-19484 URL: https://issues.apache.org/jira/browse/HADOOP-19484 Project

hadoop thirdparty 1.4.0

2025-03-05 Thread Steve Loughran
I'm going to make a new third party jar. Is there anything we need to do in it other than the avro update? https://github.com/apache/hadoop-thirdparty/pull/43 1. I don't want to rush to do a parquet or guava update unless there are CVEs. we had problems with parquet and java 8 last time.

[jira] [Created] (HADOOP-19483) Release Hadoop Third-Party 1.4.0

2025-03-05 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19483: --- Summary: Release Hadoop Third-Party 1.4.0 Key: HADOOP-19483 URL: https://issues.apache.org/jira/browse/HADOOP-19483 Project: Hadoop Common Issue Type

[jira] [Created] (HADOOP-19482) NPE in dependency-check of hadoop-thirdparty

2025-03-05 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19482: --- Summary: NPE in dependency-check of hadoop-thirdparty Key: HADOOP-19482 URL: https://issues.apache.org/jira/browse/HADOOP-19482 Project: Hadoop Common

[jira] [Resolved] (HADOOP-19303) VectorIO API to support releasing buffers on failure

2025-03-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19303. - Fix Version/s: 3.5.0 3.4.2 Resolution: Fixed > VectorIO

[jira] [Created] (HADOOP-19478) S3A: pull out new configuration load/probes under S3AStore

2025-03-03 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19478: --- Summary: S3A: pull out new configuration load/probes under S3AStore Key: HADOOP-19478 URL: https://issues.apache.org/jira/browse/HADOOP-19478 Project: Hadoop

Re: [DISCUSS] Hadoop 3.4.2 release

2025-02-28 Thread Steve Loughran
If anyone can help with this, it'd be great. Mukund couldn't get the arm docker image to do a build On Thu, 27 Feb 2025 at 09:38, Cheng Pan wrote: > Thanks Ahmar for preparing a new release. > > The aarch64 binary distribution tarball is absent in 3.4.1, better to > restore in this release. > >

[jira] [Resolved] (HADOOP-19464) S3A: Restore Compatibility with EMRFS FileSystem

2025-02-25 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19464. - Fix Version/s: 3.5.0 Resolution: Fixed > S3A: Restore Compatibility with EM

[jira] [Created] (HADOOP-19470) S3A: region resolution within AWS infra always goes to us-east-2

2025-02-24 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19470: --- Summary: S3A: region resolution within AWS infra always goes to us-east-2 Key: HADOOP-19470 URL: https://issues.apache.org/jira/browse/HADOOP-19470 Project

[jira] [Created] (HADOOP-19468) S3A: use blockchain datasets for parquet data

2025-02-21 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19468: --- Summary: S3A: use blockchain datasets for parquet data Key: HADOOP-19468 URL: https://issues.apache.org/jira/browse/HADOOP-19468 Project: Hadoop Common

[jira] [Resolved] (HADOOP-19144) S3A prefetching to support Vector IO

2025-02-20 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19144. - Resolution: Won't Fix > S3A prefetching to support V

[jira] [Resolved] (HADOOP-19354) S3A: InputStreams to be created by factory under S3AStore

2025-02-20 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19354. - Fix Version/s: 3.5.0 Resolution: Fixed > S3A: InputStreams to be created

[jira] [Created] (HADOOP-19455) S3A: support metrics publishers

2025-02-17 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19455: --- Summary: S3A: support metrics publishers Key: HADOOP-19455 URL: https://issues.apache.org/jira/browse/HADOOP-19455 Project: Hadoop Common Issue Type

[jira] [Resolved] (HADOOP-19393) ABFS: Returning FileAlreadyExists Exception for UnauthorizedBlobOverwrite Rename Errors

2025-02-17 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19393. - Target Version/s: 3.5.0, 3.4.2 Resolution: Fixed > ABFS: Return

Re: [DISCUSS] Integration of Volcano Engine TOS in Hadoop.

2025-02-13 Thread Steve Loughran
t; responsible to review and test about this module. > Beside that, just suggest following the Hadoop guidelines[1] to develop > the new features. > > @Steve Loughran @Shilun Fan > leave > some comments including some concerns in JIRA, would you mind giving more > suggestions fo

[jira] [Reopened] (HADOOP-19451) S3A: AAL - Add support for stream leak detection

2025-02-07 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-19451: - > S3A: AAL - Add support for stream leak detect

[jira] [Resolved] (HADOOP-19451) S3A: AAL - Add support for stream leak detection

2025-02-07 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19451. - Fix Version/s: 3.5.0 Assignee: Steve Loughran Resolution: Duplicate

[jira] [Created] (HADOOP-19442) add openFile() option to pass down footer length

2025-01-30 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19442: --- Summary: add openFile() option to pass down footer length Key: HADOOP-19442 URL: https://issues.apache.org/jira/browse/HADOOP-19442 Project: Hadoop Common

[jira] [Created] (HADOOP-19405) hadoop-aws tests have stopped running

2025-01-28 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19405: --- Summary: hadoop-aws tests have stopped running Key: HADOOP-19405 URL: https://issues.apache.org/jira/browse/HADOOP-19405 Project: Hadoop Common Issue

Fwd: Error while running notifications feature from refs/heads/trunk:.asf.yaml in hadoop!

2025-01-27 Thread Steve Loughran
Not sure why I got this message directly (as did common-commit), or on a file which hasn't changed for years. maybe it as some transient On Mon, 27 Jan 2025 at 11:38, Apache Infrastructure wrote: > > An error occurred while running notifications feature in .asf.yaml!: > Expecting value: line 1

[jira] [Created] (HADOOP-19388) S3A: Validate bulk delete through Iceberg HadoopFileIO

2025-01-15 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19388: --- Summary: S3A: Validate bulk delete through Iceberg HadoopFileIO Key: HADOOP-19388 URL: https://issues.apache.org/jira/browse/HADOOP-19388 Project: Hadoop Common

[jira] [Resolved] (HADOOP-19351) S3A: Add config option to skip test with performance mode

2025-01-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19351. - Fix Version/s: 3.5.0 Resolution: Fixed > S3A: Add config option to skip t

[jira] [Resolved] (HADOOP-13846) S3A to implement rename(final Path src, final Path dst, final Rename... options)

2025-01-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-13846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-13846. - Release Note: I've played with this in the path and am just going to give up. r

[jira] [Resolved] (HADOOP-18650) improve s3a committer stats collected

2025-01-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18650. - Resolution: Won't Fix Don't think this is worth the effort > improve

[jira] [Resolved] (HADOOP-15944) S3AInputStream logging to make it easier to debug file leakage

2025-01-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-15944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-15944. - Fix Version/s: 3.4.2 Resolution: Duplicate > S3AInputStream logging to make

[jira] [Resolved] (HADOOP-16154) ITestS3A select tests fail if user kinited in

2025-01-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16154. - Fix Version/s: 3.4.1 Resolution: Won't Fix > ITestS3A select tests fail

[jira] [Resolved] (HADOOP-18486) hadoop-aws tests to take a configurable subdir in the test bucket

2025-01-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18486. - Resolution: Duplicate > hadoop-aws tests to take a configurable subdir in the t

[jira] [Resolved] (HADOOP-19229) Vector IO on cloud storage: what is a good minimum seek size?

2025-01-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19229. - Fix Version/s: 3.4.2 Release Note: The thresholds at which adjacent vector IO

[jira] [Reopened] (HADOOP-19229) Vector IO on cloud storage: what is a good minimum seek size?

2025-01-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-19229: - > Vector IO on cloud storage: what is a good minimum seek s

[jira] [Resolved] (HADOOP-16480) S3 Select Exceptions are not being converted to IOEs

2025-01-15 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16480. - Resolution: Won't Fix > S3 Select Exceptions are not being converted

[jira] [Resolved] (HADOOP-18862) [JDK17] MiniYarnClusters don't launch in hadoop-aws integration tests

2025-01-14 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-18862. - Fix Version/s: 3.4.2 Resolution: Not A Problem no longer happens > [JD

[jira] [Created] (HADOOP-19385) S3A: add a file-format-parsing module for testing format parsing

2025-01-10 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-19385: --- Summary: S3A: add a file-format-parsing module for testing format parsing Key: HADOOP-19385 URL: https://issues.apache.org/jira/browse/HADOOP-19385 Project

[jira] [Resolved] (HADOOP-19369) S3A Analytics-Accelerator: Move input stream creation to S3AStore

2025-01-10 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-19369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-19369. - Resolution: Duplicate > S3A Analytics-Accelerator: Move input stream creation

  1   2   3   4   5   6   7   8   9   10   >