[
https://issues.apache.org/jira/browse/HADOOP-19129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anuj Modi updated HADOOP-19129:
-------------------------------
Description:
Test Script used by ABFS to validate changes has following two issues:
# When there are a lot of test failures or when error message of any failing
test becomes very large, the regex used today to filter test results does not
work as expected and fails to report all the failing tests.
To resolve this, we have come up with new regex that will only target one line
test names for reporting them into aggregated test results.
# While running the test suite for different combinations of Auth type and
account type, we add the combination specific configs first and then include
the account specific configs in core-site.xml file. This will override the
combination specific configs like auth type if the same config is present in
account specific config file. To avoid this, we will first include the account
specific configs and then add the combination specific configs.
Due to above bug in test script, some test failures in ABFS were not getting
our attention. This PR also targets to resolve them. Following are the tests
fixed:
Following tests are failing on OSS trunk:
# ITestAbfsClient.testListPathWithValueGreaterThanServerMaximum(): Fails
Intermittently only for HNS enabled accounts. Test wants to assert that
client.list() does not return more objects than what is configured in
maxListResults. Assertions should be that number of objects returned should be
less than expected as server might end up returning even lesser due to
partition splits along with a continuation token.
# ITestGetNameSpaceEnabled.testGetIsNamespaceEnabledWhenConfigIsTrue(): Fail
when "fs.azure.test.namespace.enabled" config is missing. Ignore the test if
config is missing.
# ITestGetNameSpaceEnabled.testGetIsNamespaceEnabledWhenConfigIsFalse(): Fail
when "fs.azure.test.namespace.enabled" config is missing. Ignore the test if
config is missing.
# ITestGetNameSpaceEnabled.testNonXNSAccount(): Fail when
"fs.azure.test.namespace.enabled" config is missing. Ignore the test if config
is missing.
# ITestAbfsStreamStatistics.testAbfsStreamOps: Fails when
"fs.azure.test.appendblob.enabled" is set to true. Test wanted to assert that
number of read operations can be more in case of append blobs as compared to
normal blob becuase of automatic flush. It could be same as that of normal blob
as well.
# ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:
Fails for FNS Account only when following config is present:
fs.azure.account.hns.enabled". Failure is because test wants to assert that
when driver does not know if the account is HNS enabled or not it makes a
server call and fails. But above config is letting driver know the account type
and skipping the head call. Remove these configs from the test specific
configurations and not from the account settings file.
# ITestAbfsTerasort.test_120_terasort: Fails with OAuth on HNS account.
Failure is because of identity mismatch. OAuth uses service principle OID as
owner of the resources whereas Shared Key uses local system identities. Fix is
to set configs that will allow overwrite of OID to localidentity. This will
require a new config to be set by user that specify which OID has to be
substituted. OAuth by default uses Superuser Identity, so same needs to be
configured to be overwritten as well.
*New test config: "fs.azure.account.oauth2.client.service.principal.object.id"*
# ITestExponentialRetryPolicy.testThrottlingIntercept: Fails with SharedKey
only. Test was using a dummy account to create a new instance of
AbfsConfiguration and for that dummy account, SharedKey was not configured. Fix
is to Add non-account specific SharedKey in accountconfigs.
# ITestAzureBlobFileSystemAuthorization: Fails when contract related configs
are not present with NPE. Fix is to Ignore if required configs are not present.
# ITestAzureBlobFileSystemAppend.testCloseOfDataBlockOnAppendComplete(): Fails
when append blob is enabled. Append Blobs does not require explicit flush and
hence flush/close related asserts do not work for append blobs.
# ITestAzureBlobFileSystemLease:testTwoCreate(): Fail when
"fs.azure.test.namespace.enabled" config is missing. Fix is to Ignore the test
if config is missing.
was:
Test Script used by ABFS to validate changes has following two issues:
# When there are a lot of test failures or when error message of any failing
test becomes very large, the regex used today to filter test results does not
work as expected and fails to report all the failing tests.
To resolve this, we have come up with new regex that will only target one line
test names for reporting them into aggregated test results.
# While running the test suite for different combinations of Auth type and
account type, we add the combination specific configs first and then include
the account specific configs in core-site.xml file. This will override the
combination specific configs like auth type if the same config is present in
account specific config file. To avoid this, we will first include the account
specific configs and then add the combination specific configs.
Due to above bug in test script, some test failures in ABFS were not getting
our attention. This PR also targets to resolve them. Following are the tests
fixed:
Following tests are failing on OSS trunk:
# ITestAbfsClient.testListPathWithValueGreaterThanServerMaximum(): Fails
Intermittently only for HNS enabled accounts. Test wants to assert that
client.list() does not return more objects than what is configured in
maxListResults. Assertions should be that number of objects returned should be
less than expected as server might end up returning even lesser due to
partition splits along with a continuation token.
# ITestGetNameSpaceEnabled.testGetIsNamespaceEnabledWhenConfigIsTrue(): Fail
when "fs.azure.test.namespace.enabled" config is missing. Ignore the test if
config is missing.
# ITestGetNameSpaceEnabled.testGetIsNamespaceEnabledWhenConfigIsFalse(): Fail
when "fs.azure.test.namespace.enabled" config is missing. Ignore the test if
config is missing.
# ITestGetNameSpaceEnabled.testNonXNSAccount(): Fail when
"fs.azure.test.namespace.enabled" config is missing. Ignore the test if config
is missing.
# ITestAbfsStreamStatistics.testAbfsStreamOps: Fails when
"fs.azure.test.appendblob.enabled" is set to true. Test wanted to assert that
number of read operations can be more in case of append blobs as compared to
normal blob becuase of automatic flush. It could be same as that of normal blob
as well.
# ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:
Fails for FNS Account only when following config is present:
fs.azure.account.hns.enabled". Failure is because test wants to assert that
when driver does not know if the account is HNS enabled or not it makes a
server call and fails. But above config is letting driver know the account type
and skipping the head call. Remove these configs from the test specific
configurations and not from the account settings file.
# ITestAbfsTerasort.test_120_terasort: Fails with OAuth on HNS account.
Failure is because of identity mismatch. OAuth uses service principle OID as
owner of the resources whereas Shared Key uses local system identities. Fix is
to set configs that will allow overwrite of OID to localidentity. This will
require a new config to be set by user that specify which OID has to be
substituted. OAuth by default uses Superuser Identity, so same needs to be
configured to be overwritten as well.
*New test config: "fs.azure.account.oauth2.client.service.principal.object.id"*
# ITestExponentialRetryPolicy.testThrottlingIntercept: Fails with SharedKey
only. Test was using a dummy account to create a new instance of
AbfsConfiguration and for that dummy account, SharedKey was not configured. Fix
is to Add non-account specific SharedKey in accountconfigs.
# ITestAzureBlobFileSystemAuthorization: Fails when contract related configs
are not present with NPE. Fix is to Ignore if required configs are not present.
# ITestAzureBlobFileSystemAppend.testCloseOfDataBlockOnAppendComplete(): Fails
when append blob is enabled. Append Blobs does not require explicit flush and
hence flush/close related asserts do not work for append blobs.
# ITestAzureBlobFileSystemLease:testTwoCreate(): Fail when
"fs.azure.test.namespace.enabled" config is missing. Fix is to Ignore the test
if config is missing.
> ABFS: Fixing Test Script Bug and Some Known test Failures in ABFS Test Suite
> ----------------------------------------------------------------------------
>
> Key: HADOOP-19129
> URL: https://issues.apache.org/jira/browse/HADOOP-19129
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/azure
> Affects Versions: 3.4.0, 3.4.1
> Reporter: Anuj Modi
> Assignee: Anuj Modi
> Priority: Major
>
> Test Script used by ABFS to validate changes has following two issues:
> # When there are a lot of test failures or when error message of any failing
> test becomes very large, the regex used today to filter test results does not
> work as expected and fails to report all the failing tests.
> To resolve this, we have come up with new regex that will only target one
> line test names for reporting them into aggregated test results.
> # While running the test suite for different combinations of Auth type and
> account type, we add the combination specific configs first and then include
> the account specific configs in core-site.xml file. This will override the
> combination specific configs like auth type if the same config is present in
> account specific config file. To avoid this, we will first include the
> account specific configs and then add the combination specific configs.
> Due to above bug in test script, some test failures in ABFS were not getting
> our attention. This PR also targets to resolve them. Following are the tests
> fixed:
> Following tests are failing on OSS trunk:
> # ITestAbfsClient.testListPathWithValueGreaterThanServerMaximum(): Fails
> Intermittently only for HNS enabled accounts. Test wants to assert that
> client.list() does not return more objects than what is configured in
> maxListResults. Assertions should be that number of objects returned should
> be less than expected as server might end up returning even lesser due to
> partition splits along with a continuation token.
> # ITestGetNameSpaceEnabled.testGetIsNamespaceEnabledWhenConfigIsTrue(): Fail
> when "fs.azure.test.namespace.enabled" config is missing. Ignore the test if
> config is missing.
> # ITestGetNameSpaceEnabled.testGetIsNamespaceEnabledWhenConfigIsFalse():
> Fail when "fs.azure.test.namespace.enabled" config is missing. Ignore the
> test if config is missing.
> # ITestGetNameSpaceEnabled.testNonXNSAccount(): Fail when
> "fs.azure.test.namespace.enabled" config is missing. Ignore the test if
> config is missing.
> # ITestAbfsStreamStatistics.testAbfsStreamOps: Fails when
> "fs.azure.test.appendblob.enabled" is set to true. Test wanted to assert that
> number of read operations can be more in case of append blobs as compared to
> normal blob becuase of automatic flush. It could be same as that of normal
> blob as well.
> # ITestAzureBlobFileSystemCheckAccess.testCheckAccessForAccountWithoutNS:
> Fails for FNS Account only when following config is present:
> fs.azure.account.hns.enabled". Failure is because test wants to assert that
> when driver does not know if the account is HNS enabled or not it makes a
> server call and fails. But above config is letting driver know the account
> type and skipping the head call. Remove these configs from the test specific
> configurations and not from the account settings file.
> # ITestAbfsTerasort.test_120_terasort: Fails with OAuth on HNS account.
> Failure is because of identity mismatch. OAuth uses service principle OID as
> owner of the resources whereas Shared Key uses local system identities. Fix
> is to set configs that will allow overwrite of OID to localidentity. This
> will require a new config to be set by user that specify which OID has to be
> substituted. OAuth by default uses Superuser Identity, so same needs to be
> configured to be overwritten as well.
> *New test config:
> "fs.azure.account.oauth2.client.service.principal.object.id"*
> # ITestExponentialRetryPolicy.testThrottlingIntercept: Fails with SharedKey
> only. Test was using a dummy account to create a new instance of
> AbfsConfiguration and for that dummy account, SharedKey was not configured.
> Fix is to Add non-account specific SharedKey in accountconfigs.
> # ITestAzureBlobFileSystemAuthorization: Fails when contract related configs
> are not present with NPE. Fix is to Ignore if required configs are not
> present.
> # ITestAzureBlobFileSystemAppend.testCloseOfDataBlockOnAppendComplete():
> Fails when append blob is enabled. Append Blobs does not require explicit
> flush and hence flush/close related asserts do not work for append blobs.
> # ITestAzureBlobFileSystemLease:testTwoCreate(): Fail when
> "fs.azure.test.namespace.enabled" config is missing. Fix is to Ignore the
> test if config is missing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
