sumitagrawl commented on code in PR #7431:
URL: https://github.com/apache/ozone/pull/7431#discussion_r1846920335
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java:
##########
@@ -685,28 +685,30 @@ public FileStatus[] listStatus(Path f) throws IOException
{
LinkedList<FileStatus> statuses = new LinkedList<>();
List<FileStatus> tmpStatusList;
String startKey = "";
-
+ int entriesAdded;
do {
tmpStatusList =
adapter.listStatus(pathToKey(f), false, startKey, numEntries, uri,
workingDir, getUsername(), true)
.stream()
.map(this::convertFileStatus)
.collect(Collectors.toList());
-
+ entriesAdded = 0;
if (!tmpStatusList.isEmpty()) {
if (startKey.isEmpty() || !statuses.getLast().getPath().toString()
.equals(tmpStatusList.get(0).getPath().toString())) {
statuses.addAll(tmpStatusList);
+ entriesAdded += tmpStatusList.size();
} else {
statuses.addAll(tmpStatusList.subList(1, tmpStatusList.size()));
+ entriesAdded += tmpStatusList.size() - 1;
}
startKey = pathToKey(statuses.getLast().getPath());
}
// listStatus returns entries numEntries in size if available.
// Any lesser number of entries indicate that the required entries have
// exhausted.
- } while (tmpStatusList.size() == numEntries);
+ } while (entriesAdded > 0);
Review Comment:
we just need check "tmpStatusList.size() > 0" to verify if any further
elements are available, as list size can be controlled by server also.
entriesAdded check is not required.
##########
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java:
##########
@@ -938,25 +938,27 @@ private List<FileStatusAdapter> listStatusAdapter(Path f,
boolean lite) throws I
LinkedList<FileStatusAdapter> statuses = new LinkedList<>();
List<FileStatusAdapter> tmpStatusList;
String startPath = "";
-
+ int entriesAdded;
do {
tmpStatusList =
adapter.listStatus(pathToKey(f), false, startPath,
numEntries, uri, workingDir, getUsername(), lite);
-
+ entriesAdded = 0;
if (!tmpStatusList.isEmpty()) {
if (startPath.isEmpty() || !statuses.getLast().getPath().toString()
.equals(tmpStatusList.get(0).getPath().toString())) {
statuses.addAll(tmpStatusList);
+ entriesAdded += tmpStatusList.size();
} else {
statuses.addAll(tmpStatusList.subList(1, tmpStatusList.size()));
+ entriesAdded += tmpStatusList.size() - 1;
}
startPath = pathToKey(statuses.getLast().getPath());
}
// listStatus returns entries numEntries in size if available.
// Any lesser number of entries indicate that the required entries have
// exhausted.
- } while (tmpStatusList.size() == numEntries);
+ } while (entriesAdded > 0);
Review Comment:
Need have only check, tmpStatusList.size() > 0 to continue loop to get next
set, it all records are retrieved and no more available.
##########
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/OzoneBucket.java:
##########
@@ -1808,7 +1808,7 @@ private boolean getChildrenKeys(String keyPrefix, String
startKey,
// 1. Get immediate children of keyPrefix, starting with startKey
List<OzoneFileStatusLight> statuses = proxy.listStatusLight(volumeName,
name, keyPrefix, false, startKey, listCacheSize, true);
- boolean reachedLimitCacheSize = statuses.size() == listCacheSize;
+ boolean hasMoreEntries = !statuses.isEmpty();
Review Comment:
can remove reachLimitCacheSize usages, as this has no impact, and in loop,
where checked, it will be always true.
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/protocolPB/OzoneManagerRequestHandler.java:
##########
@@ -181,9 +183,16 @@ public class OzoneManagerRequestHandler implements
RequestHandler {
LoggerFactory.getLogger(OzoneManagerRequestHandler.class);
private final OzoneManager impl;
private FaultInjector injector;
+ private long maxKeyListSize;
+
public OzoneManagerRequestHandler(OzoneManager om) {
this.impl = om;
+ this.maxKeyListSize =
om.getConfiguration().getLong(OZONE_OM_SERVER_LIST_MAX_SIZE,
+ OZONE_OM_SERVER_LIST_MAX_SIZE_DEFAULT);
+ if (this.maxKeyListSize <= 0) {
+ this.maxKeyListSize = Integer.MAX_VALUE;
Review Comment:
needs to set to OZONE_OM_SERVER_LIST_MAX_SIZE_DEFAULT when list size is
invalid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
