shounakmk219 commented on code in PR #13296:
URL: https://github.com/apache/pinot/pull/13296#discussion_r1634199499
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSchemaRestletResource.java:
##########
@@ -119,6 +121,28 @@ public List<String> listSchemaNames(@Context HttpHeaders
headers) {
return
_pinotHelixResourceManager.getSchemaNames(headers.getHeaderString(DATABASE));
}
+ @GET
+ @Produces(MediaType.APPLICATION_JSON)
+ @Path("/schemas/info")
+ @Authorize(targetType = TargetType.CLUSTER, action =
Actions.Cluster.GET_SCHEMA_INFO)
+ @ApiOperation(value = "List all schemas info with count of field specs",
notes = "Lists all schemas with field "
+ + "count details")
+ public List<SchemaInfo> getSchemaInfo(@Context HttpHeaders headers) {
+ List<SchemaInfo> schemasInfo = new ArrayList<>();
+ List<String> schemaList =
_pinotHelixResourceManager.getSchemaNames(headers.getHeaderString(DATABASE));
Review Comment:
Use tableCache here instead of hitting zk for all the get ops. Also I see
you are already computing the SchemaInfo in tableCache so try using that itself.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -230,25 +231,54 @@ public List<Map<TableType, List<String>>> getSegments(
@Path("segments/{tableName}/servers")
@Authorize(targetType = TargetType.TABLE, paramName = "tableName", action =
Actions.Table.GET_SERVER_MAP)
@Produces(MediaType.APPLICATION_JSON)
- @ApiOperation(value = "Get a map from server to segments hosted by the
server",
- notes = "Get a map from server to segments hosted by the server")
+ @ApiOperation(value = "Get a map from server to segments hosted by the
server", notes = "Get a map from server to "
+ + "segments hosted by the server")
public List<Map<String, Object>> getServerToSegmentsMap(
@ApiParam(value = "Name of the table", required = true)
@PathParam("tableName") String tableName,
- @ApiParam(value = "OFFLINE|REALTIME") @QueryParam("type") String
tableTypeStr, @Context HttpHeaders headers) {
+ @ApiParam(value = "OFFLINE|REALTIME") @QueryParam("type") String
tableTypeStr,
+ @QueryParam("detailed") @DefaultValue("true") boolean detailed, @Context
HttpHeaders headers) {
tableName = DatabaseUtils.translateTableName(tableName, headers);
- List<String> tableNamesWithType = ResourceUtils
- .getExistingTableNamesWithType(_pinotHelixResourceManager, tableName,
Constants.validateTableType(tableTypeStr),
- LOGGER);
+ List<String> tableNamesWithType =
ResourceUtils.getExistingTableNamesWithType(_pinotHelixResourceManager,
tableName,
+ Constants.validateTableType(tableTypeStr), LOGGER);
List<Map<String, Object>> resultList = new
ArrayList<>(tableNamesWithType.size());
for (String tableNameWithType : tableNamesWithType) {
Map<String, Object> resultForTable = new LinkedHashMap<>();
resultForTable.put("tableName", tableNameWithType);
- resultForTable.put("serverToSegmentsMap",
_pinotHelixResourceManager.getServerToSegmentsMap(tableNameWithType));
+ if (detailed) {
+ resultForTable.put("serverToSegmentsMap",
_pinotHelixResourceManager.getServerToSegmentsMap(tableNameWithType));
+ } else {
+ resultForTable.put("serverToSegmentsCountMap",
+
_pinotHelixResourceManager.getServerToSegmentsCountMap(tableNameWithType));
Review Comment:
Good to have `serverToSegmentsCountMap ` info in both detailed and
non-detailed response. To avoid double computations just use the response from
`_pinotHelixResourceManager.getServerToSegmentsMap` itself to build the count
map and skip populating `serverToSegmentsMap` when `detailed` is `false`.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/SegmentStatusInfo.java:
##########
@@ -0,0 +1,46 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.api.resources;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * This class gives the details of a particular segment and it's status
+ *
+ */
+public class SegmentStatusInfo {
+ public String getSegmentName() {
+ return _segmentName;
+ }
+
+ @JsonProperty("schemaName")
+ String _segmentName;
+
+ public String getSegmentStatus() {
+ return _segmentStatus;
+ }
+
+ @JsonProperty("schemaStatus")
+ String _segmentStatus;
Review Comment:
Does it make sense to have a defined enum for all possible status?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -230,25 +231,54 @@ public List<Map<TableType, List<String>>> getSegments(
@Path("segments/{tableName}/servers")
@Authorize(targetType = TargetType.TABLE, paramName = "tableName", action =
Actions.Table.GET_SERVER_MAP)
@Produces(MediaType.APPLICATION_JSON)
- @ApiOperation(value = "Get a map from server to segments hosted by the
server",
- notes = "Get a map from server to segments hosted by the server")
+ @ApiOperation(value = "Get a map from server to segments hosted by the
server", notes = "Get a map from server to "
+ + "segments hosted by the server")
public List<Map<String, Object>> getServerToSegmentsMap(
@ApiParam(value = "Name of the table", required = true)
@PathParam("tableName") String tableName,
- @ApiParam(value = "OFFLINE|REALTIME") @QueryParam("type") String
tableTypeStr, @Context HttpHeaders headers) {
+ @ApiParam(value = "OFFLINE|REALTIME") @QueryParam("type") String
tableTypeStr,
+ @QueryParam("detailed") @DefaultValue("true") boolean detailed, @Context
HttpHeaders headers) {
tableName = DatabaseUtils.translateTableName(tableName, headers);
- List<String> tableNamesWithType = ResourceUtils
- .getExistingTableNamesWithType(_pinotHelixResourceManager, tableName,
Constants.validateTableType(tableTypeStr),
- LOGGER);
+ List<String> tableNamesWithType =
ResourceUtils.getExistingTableNamesWithType(_pinotHelixResourceManager,
tableName,
+ Constants.validateTableType(tableTypeStr), LOGGER);
List<Map<String, Object>> resultList = new
ArrayList<>(tableNamesWithType.size());
for (String tableNameWithType : tableNamesWithType) {
Map<String, Object> resultForTable = new LinkedHashMap<>();
resultForTable.put("tableName", tableNameWithType);
- resultForTable.put("serverToSegmentsMap",
_pinotHelixResourceManager.getServerToSegmentsMap(tableNameWithType));
+ if (detailed) {
+ resultForTable.put("serverToSegmentsMap",
_pinotHelixResourceManager.getServerToSegmentsMap(tableNameWithType));
+ } else {
+ resultForTable.put("serverToSegmentsCountMap",
+
_pinotHelixResourceManager.getServerToSegmentsCountMap(tableNameWithType));
+ }
resultList.add(resultForTable);
}
return resultList;
}
+ @GET
+ @Produces(MediaType.APPLICATION_JSON)
+ @Path("/segments/{tableName}/info")
Review Comment:
Can we move this endpoint to `TableViews` resource and maybe change the path
to `/tables/{tableName}/segmentsStatus`?
This will also help you get around all the refactors in `TableViews` and
`PinotHelixResourceManager`
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/SegmentStatusInfo.java:
##########
@@ -0,0 +1,46 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.api.resources;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * This class gives the details of a particular segment and it's status
+ *
+ */
+public class SegmentStatusInfo {
Review Comment:
nit: move all the field declarations before the methods.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -2798,6 +2801,26 @@ public Map<String, List<String>>
getServerToSegmentsMap(String tableNameWithType
return serverToSegmentsMap;
}
+ /**
+ * Returns a map from server instance to count of segments it serves for the
given table. Ignore OFFLINE segments from
+ * the ideal state because they are not supposed to be served.
+ */
+ public Map<String, Integer> getServerToSegmentsCountMap(String
tableNameWithType) {
Review Comment:
We can skip this as we will be reusing `getServerToSegmentsMap` itself
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -230,25 +231,54 @@ public List<Map<TableType, List<String>>> getSegments(
@Path("segments/{tableName}/servers")
@Authorize(targetType = TargetType.TABLE, paramName = "tableName", action =
Actions.Table.GET_SERVER_MAP)
@Produces(MediaType.APPLICATION_JSON)
- @ApiOperation(value = "Get a map from server to segments hosted by the
server",
- notes = "Get a map from server to segments hosted by the server")
+ @ApiOperation(value = "Get a map from server to segments hosted by the
server", notes = "Get a map from server to "
+ + "segments hosted by the server")
public List<Map<String, Object>> getServerToSegmentsMap(
@ApiParam(value = "Name of the table", required = true)
@PathParam("tableName") String tableName,
- @ApiParam(value = "OFFLINE|REALTIME") @QueryParam("type") String
tableTypeStr, @Context HttpHeaders headers) {
+ @ApiParam(value = "OFFLINE|REALTIME") @QueryParam("type") String
tableTypeStr,
+ @QueryParam("detailed") @DefaultValue("true") boolean detailed, @Context
HttpHeaders headers) {
tableName = DatabaseUtils.translateTableName(tableName, headers);
- List<String> tableNamesWithType = ResourceUtils
- .getExistingTableNamesWithType(_pinotHelixResourceManager, tableName,
Constants.validateTableType(tableTypeStr),
- LOGGER);
+ List<String> tableNamesWithType =
ResourceUtils.getExistingTableNamesWithType(_pinotHelixResourceManager,
tableName,
+ Constants.validateTableType(tableTypeStr), LOGGER);
List<Map<String, Object>> resultList = new
ArrayList<>(tableNamesWithType.size());
for (String tableNameWithType : tableNamesWithType) {
Map<String, Object> resultForTable = new LinkedHashMap<>();
resultForTable.put("tableName", tableNameWithType);
- resultForTable.put("serverToSegmentsMap",
_pinotHelixResourceManager.getServerToSegmentsMap(tableNameWithType));
+ if (detailed) {
+ resultForTable.put("serverToSegmentsMap",
_pinotHelixResourceManager.getServerToSegmentsMap(tableNameWithType));
+ } else {
+ resultForTable.put("serverToSegmentsCountMap",
+
_pinotHelixResourceManager.getServerToSegmentsCountMap(tableNameWithType));
+ }
resultList.add(resultForTable);
}
return resultList;
}
+ @GET
+ @Produces(MediaType.APPLICATION_JSON)
+ @Path("/segments/{tableName}/info")
+ @Authorize(targetType = TargetType.TABLE, paramName = "tableName", action =
Actions.Table.GET_SEGMENT_STATUS)
+ @ApiOperation(value = "Get segment names to segment status map", notes =
"Get segment statuses of each segment")
+ public String getSegmentsStatusDetails(
+ @ApiParam(value = "Name of the table", required = true)
@PathParam("tableName") String tableName,
+ @ApiParam(value = "realtime|offline", required = false)
@QueryParam("tableType") String tableTypeStr,
+ @DefaultValue("0") @QueryParam("offset") int offset,
Review Comment:
let's skip the pagination part for now unless we have an efficient way to
serve it. Right now its doing same amount of computation irrespective of the
demanded page and effectively it will just increase the computational load on
controller as UI will send separate requests for all pages.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
