Jackie-Jiang commented on code in PR #14250:
URL: https://github.com/apache/pinot/pull/14250#discussion_r2211773234
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -919,15 +919,18 @@ private void deleteSegmentsInternal(String
tableNameWithType, List<String> segme
@GET
@Path("segments/{tableName}/metadata")
+ @Encoded
Review Comment:
I think we usually put this annotation in front of the argument. Do we want
to annotate the entire method?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/util/TableMetadataReader.java:
##########
@@ -127,50 +130,104 @@ private TableReloadJsonResponse
processSegmentMetadataReloadResponse(
/**
* This api takes in list of segments for which we need the metadata.
+ * This calls the server to get the metadata for all segments instead of
making a call per segment.
*/
- public JsonNode getSegmentsMetadata(String tableNameWithType, List<String>
columns, Set<String> segmentsToInclude,
+ public JsonNode getSegmentsMetadata(String tableNameWithType, List<String>
columns, Set<String> segments,
Review Comment:
(minor) Annotate both `columns` and `segments` as `@Nullable`
##########
pinot-controller/src/main/java/org/apache/pinot/controller/util/TableMetadataReader.java:
##########
@@ -127,50 +130,104 @@ private TableReloadJsonResponse
processSegmentMetadataReloadResponse(
/**
* This api takes in list of segments for which we need the metadata.
+ * This calls the server to get the metadata for all segments instead of
making a call per segment.
*/
- public JsonNode getSegmentsMetadata(String tableNameWithType, List<String>
columns, Set<String> segmentsToInclude,
+ public JsonNode getSegmentsMetadata(String tableNameWithType, List<String>
columns, Set<String> segments,
int timeoutMs)
throws InvalidConfigException, IOException {
- return getSegmentsMetadataInternal(tableNameWithType, columns,
segmentsToInclude, timeoutMs);
+ return getSegmentsMetadataInternal(tableNameWithType, columns, segments,
timeoutMs);
}
- private JsonNode getSegmentsMetadataInternal(String tableNameWithType,
List<String> columns,
- Set<String> segmentsToInclude, int timeoutMs)
+ /**
+ * Common helper used by both the new (server-level) and legacy
(segment-level) endpoints.
+ */
+ private JsonNode fetchAndAggregateMetadata(List<String> urls,
+ BiMap<String, String> endpoints,
+ boolean perSegmentJson,
+ String tableNameWithType,
+ int timeoutMs)
throws InvalidConfigException, IOException {
- final Map<String, List<String>> serverToSegmentsMap =
- _pinotHelixResourceManager.getServerToSegmentsMap(tableNameWithType);
- BiMap<String, String> endpoints =
-
_pinotHelixResourceManager.getDataInstanceAdminEndpoints(serverToSegmentsMap.keySet());
- ServerSegmentMetadataReader serverSegmentMetadataReader =
- new ServerSegmentMetadataReader(_executor, _connectionManager);
+ CompletionServiceHelper cs = new CompletionServiceHelper(_executor,
_connectionManager, endpoints);
+ CompletionServiceHelper.CompletionServiceResponse resp =
+ cs.doMultiGetRequest(urls, tableNameWithType, perSegmentJson,
timeoutMs);
+ // all requests will fail if new server endpoint is not available
+ if (resp._failedResponseCount == urls.size()) {
+ throw new InvalidConfigException("All requests to server instances
failed.");
Review Comment:
This shouldn't be `InvalidConfigException`. Probably a `RuntimeException`
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -1156,14 +1159,17 @@ public List<Map<TableType, List<String>>>
getSelectedSegments(
* This is a helper method to get the metadata for all segments for a given
table name.
* @param tableNameWithType name of the table along with its type
* @param columns name of the columns
+ * @param segments name of the segments to include in metadata
* @return Map<String, String> metadata of the table segments -> map of
segment name to its metadata
*/
- private JsonNode getSegmentsMetadataFromServer(String tableNameWithType,
List<String> columns)
+ private JsonNode getSegmentsMetadataFromServer(String tableNameWithType,
List<String> columns,
+ Set<String> segments)
Review Comment:
(minor) Annotate both `columns` and `segments` as `@Nullable`
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -925,8 +925,10 @@ private void deleteSegmentsInternal(String
tableNameWithType, List<String> segme
public String getServerMetadata(
@ApiParam(value = "Name of the table", required = true)
@PathParam("tableName") String tableName,
@ApiParam(value = "OFFLINE|REALTIME") @QueryParam("type") String
tableTypeStr,
+ @ApiParam(value = "Segments name", allowMultiple = true)
@QueryParam("segments")
Review Comment:
(minor) Suggest changing the value to `"Segments to include"`
##########
pinot-controller/src/main/java/org/apache/pinot/controller/util/TableMetadataReader.java:
##########
@@ -127,50 +130,104 @@ private TableReloadJsonResponse
processSegmentMetadataReloadResponse(
/**
* This api takes in list of segments for which we need the metadata.
+ * This calls the server to get the metadata for all segments instead of
making a call per segment.
*/
- public JsonNode getSegmentsMetadata(String tableNameWithType, List<String>
columns, Set<String> segmentsToInclude,
+ public JsonNode getSegmentsMetadata(String tableNameWithType, List<String>
columns, Set<String> segments,
int timeoutMs)
throws InvalidConfigException, IOException {
- return getSegmentsMetadataInternal(tableNameWithType, columns,
segmentsToInclude, timeoutMs);
+ return getSegmentsMetadataInternal(tableNameWithType, columns, segments,
timeoutMs);
}
- private JsonNode getSegmentsMetadataInternal(String tableNameWithType,
List<String> columns,
- Set<String> segmentsToInclude, int timeoutMs)
+ /**
+ * Common helper used by both the new (server-level) and legacy
(segment-level) endpoints.
+ */
+ private JsonNode fetchAndAggregateMetadata(List<String> urls,
+ BiMap<String, String> endpoints,
Review Comment:
(minor, format) We don't usually put parameters into multiple lines. Same
for other methods
##########
pinot-controller/src/main/java/org/apache/pinot/controller/util/TableMetadataReader.java:
##########
@@ -127,50 +130,104 @@ private TableReloadJsonResponse
processSegmentMetadataReloadResponse(
/**
* This api takes in list of segments for which we need the metadata.
+ * This calls the server to get the metadata for all segments instead of
making a call per segment.
*/
- public JsonNode getSegmentsMetadata(String tableNameWithType, List<String>
columns, Set<String> segmentsToInclude,
+ public JsonNode getSegmentsMetadata(String tableNameWithType, List<String>
columns, Set<String> segments,
int timeoutMs)
throws InvalidConfigException, IOException {
- return getSegmentsMetadataInternal(tableNameWithType, columns,
segmentsToInclude, timeoutMs);
+ return getSegmentsMetadataInternal(tableNameWithType, columns, segments,
timeoutMs);
}
- private JsonNode getSegmentsMetadataInternal(String tableNameWithType,
List<String> columns,
- Set<String> segmentsToInclude, int timeoutMs)
+ /**
+ * Common helper used by both the new (server-level) and legacy
(segment-level) endpoints.
Review Comment:
(minor) Remove extra space
```suggestion
* Common helper used by both the new (server-level) and legacy
(segment-level) endpoints.
```
##########
pinot-controller/src/main/java/org/apache/pinot/controller/util/TableMetadataReader.java:
##########
@@ -127,50 +130,104 @@ private TableReloadJsonResponse
processSegmentMetadataReloadResponse(
/**
* This api takes in list of segments for which we need the metadata.
+ * This calls the server to get the metadata for all segments instead of
making a call per segment.
*/
- public JsonNode getSegmentsMetadata(String tableNameWithType, List<String>
columns, Set<String> segmentsToInclude,
+ public JsonNode getSegmentsMetadata(String tableNameWithType, List<String>
columns, Set<String> segments,
int timeoutMs)
throws InvalidConfigException, IOException {
- return getSegmentsMetadataInternal(tableNameWithType, columns,
segmentsToInclude, timeoutMs);
+ return getSegmentsMetadataInternal(tableNameWithType, columns, segments,
timeoutMs);
}
- private JsonNode getSegmentsMetadataInternal(String tableNameWithType,
List<String> columns,
- Set<String> segmentsToInclude, int timeoutMs)
+ /**
+ * Common helper used by both the new (server-level) and legacy
(segment-level) endpoints.
+ */
+ private JsonNode fetchAndAggregateMetadata(List<String> urls,
+ BiMap<String, String> endpoints,
+ boolean perSegmentJson,
+ String tableNameWithType,
+ int timeoutMs)
throws InvalidConfigException, IOException {
- final Map<String, List<String>> serverToSegmentsMap =
- _pinotHelixResourceManager.getServerToSegmentsMap(tableNameWithType);
- BiMap<String, String> endpoints =
-
_pinotHelixResourceManager.getDataInstanceAdminEndpoints(serverToSegmentsMap.keySet());
- ServerSegmentMetadataReader serverSegmentMetadataReader =
- new ServerSegmentMetadataReader(_executor, _connectionManager);
+ CompletionServiceHelper cs = new CompletionServiceHelper(_executor,
_connectionManager, endpoints);
+ CompletionServiceHelper.CompletionServiceResponse resp =
+ cs.doMultiGetRequest(urls, tableNameWithType, perSegmentJson,
timeoutMs);
+ // all requests will fail if new server endpoint is not available
+ if (resp._failedResponseCount == urls.size()) {
Review Comment:
This is not robust. What if some servers succeed and others fail? E.g. some
servers are running new code and some servers are running old code?
We should use the table level API only when all servers responded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
