[
https://issues.apache.org/jira/browse/HADOOP-19133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17833082#comment-17833082
]
Attila Doroszlai commented on HADOOP-19133:
-------------------------------------------
{{Configuration#get()}} returns {{null}} for properties specified via CLI.
Other contract tests work because they use {{test.fs.s3a.name}} indirectly, via
{{fs.contract.test.fs.s3a}}. {{Configuration#substituteVars}} applies the
system property in that case.
The workaround for me is to set both values in the XML to another property, and
set that via CLI:
{code}
<property>
<name>fs.contract.test.fs.s3a</name>
<value>${custom.fs.s3a.name}</value>
</property>
<property>
<name>test.fs.s3a.name</name>
<value>${custom.fs.s3a.name}</value>
</property>
{code}
> "No test bucket" error in ITestS3AContractVectoredRead if provided via CLI
> property
> -----------------------------------------------------------------------------------
>
> Key: HADOOP-19133
> URL: https://issues.apache.org/jira/browse/HADOOP-19133
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3, test
> Affects Versions: 3.4.0, 3.3.6
> Reporter: Attila Doroszlai
> Priority: Minor
>
> ITestS3AContractVectoredRead fails with {{NullPointerException: No test
> bucket}} if test bucket is defined as {{-Dtest.fs.s3a.name=...}} via CLI ,
> not in {{auth-keys.xml}}. The same setup works for other S3A contract tests.
> Tested on 3.3.6.
> {code:title=src/test/resources/auth-keys.xml}
> <configuration>
> <property>
> <name>fs.s3a.endpoint</name>
> <value>${test.fs.s3a.endpoint}</value>
> </property>
> <property>
> <name>fs.contract.test.fs.s3a</name>
> <value>${test.fs.s3a.name}</value>
> </property>
> </configuration>
> {code}
> {code}
> export AWS_ACCESS_KEY_ID='<redacted>'
> export AWS_SECRET_KEY='<redacted>'
> mvn -Dtest=ITestS3AContractVectoredRead -Dtest.fs.s3a.name="s3a://mybucket"
> -Dtest.fs.s3a.endpoint="http://localhost:9878/"; clean test
> {code}
> {code:title=test results}
> Tests run: 46, Failures: 0, Errors: 8, Skipped: 0, Time elapsed: 7.879 s <<<
> FAILURE! - in org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead
> testMinSeekAndMaxSizeDefaultValues[Buffer type :
> direct](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead) Time
> elapsed: 1.95 s <<< ERROR!
> java.lang.NullPointerException: No test bucket
> at org.apache.hadoop.util.Preconditions.checkNotNull(Preconditions.java:88)
> at
> org.apache.hadoop.fs.s3a.S3ATestUtils.getTestBucketName(S3ATestUtils.java:714)
> at
> org.apache.hadoop.fs.s3a.S3ATestUtils.removeBaseAndBucketOverrides(S3ATestUtils.java:775)
> at
> org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead.testMinSeekAndMaxSizeDefaultValues(ITestS3AContractVectoredRead.java:104)
> ...
> testMinSeekAndMaxSizeConfigsPropagation[Buffer type :
> direct](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead) Time
> elapsed: 0.176 s <<< ERROR!
> testMultiVectoredReadStatsCollection[Buffer type :
> direct](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead) Time
> elapsed: 0.179 s <<< ERROR!
> testNormalReadVsVectoredReadStatsCollection[Buffer type :
> direct](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead) Time
> elapsed: 0.155 s <<< ERROR!
> testMinSeekAndMaxSizeDefaultValues[Buffer type :
> array](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead) Time
> elapsed: 0.116 s <<< ERROR!
> testMinSeekAndMaxSizeConfigsPropagation[Buffer type :
> array](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead) Time
> elapsed: 0.102 s <<< ERROR!
> testMultiVectoredReadStatsCollection[Buffer type :
> array](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead) Time
> elapsed: 0.105 s <<< ERROR!
> testNormalReadVsVectoredReadStatsCollection[Buffer type :
> array](org.apache.hadoop.fs.contract.s3a.ITestS3AContractVectoredRead) Time
> elapsed: 0.107 s <<< ERROR!
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
