rodolphogarrido commented on issue #11407:
URL: https://github.com/apache/pinot/issues/11407#issuecomment-1761283894
@KKcorps after double checking, it seems that the first segment isn't
actually beeing commited (sorry for the mistake). Pinot is able to consume some
record (since I can query a few ingested records), but when it tries to commit
the first segment it fails and than consumption also stops.
I started testing pinot with the version 0.12.1 (didn't try older version).
I've tried the new version (1.0.0) and the same issue happens.
Trying with version 1.0.0, the server logs show:
```
2023-10-13 07:17:26 2023/10/13 10:17:26.493 ERROR
[LLRealtimeSegmentDataManager_events_upsert_full__1__0__20231013T1014Z]
[events_upsert_full__1__0__20231013T1014Z] Holding after response from
Controller:
{"status":"NOT_SENT","streamPartitionMsgOffset":null,"isSplitCommitType":false,"buildTimeSec":-1,"offset":-1}
2023-10-13 07:17:29 2023/10/13 10:17:29.438 ERROR
[ServerSegmentCompletionProtocolHandler]
[events_upsert_partial__1__0__20231013T1014Z] Could not send request
http://pinot-controller-0:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=2&instance=Server_pinot-server-0_8098&offset=-1&name=events_upsert_partial__1__0__20231013T1014Z&rowCount=2&memoryUsedBytes=1140
2023-10-13 07:17:29
org.apache.pinot.common.exception.HttpErrorStatusException: Got error status
code: 403 (Forbidden) with reason: "Permission is denied for READ
'/segmentConsumed'" while sending request:
http://pinot-controller-0:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=2&instance=Server_pinot-server-0_8098&offset=-1&name=events_upsert_partial__1__0__20231013T1014Z&rowCount=2&memoryUsedBytes=1140
to controller: pinot-controller-0, version: Unknown
2023-10-13 07:17:29 at
org.apache.pinot.common.utils.http.HttpClient.wrapAndThrowHttpException(HttpClient.java:448)
~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
2023-10-13 07:17:29 at
org.apache.pinot.common.utils.FileUploadDownloadClient.sendSegmentCompletionProtocolRequest(FileUploadDownloadClient.java:1129)
~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
2023-10-13 07:17:29 at
org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.sendRequest(ServerSegmentCompletionProtocolHandler.java:221)
[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
2023-10-13 07:17:29 at
org.apache.pinot.server.realtime.ServerSegmentCompletionProtocolHandler.segmentConsumed(ServerSegmentCompletionProtocolHandler.java:188)
[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
2023-10-13 07:17:29 at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.postSegmentConsumedMsg(LLRealtimeSegmentDataManager.java:1152)
[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
2023-10-13 07:17:29 at
org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:700)
[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
2023-10-13 07:17:29 at java.lang.Thread.run(Thread.java:829) [?:?]
2023-10-13 07:17:29 2023/10/13 10:17:29.440 ERROR
[LLRealtimeSegmentDataManager_events_upsert_partial__1__0__20231013T1014Z]
[events_upsert_partial__1__0__20231013T1014Z] Holding after response from
Controller:
{"status":"NOT_SENT","streamPartitionMsgOffset":null,"isSplitCommitType":false,"buildTimeSec":-1,"offset":-1}
```
If I try the same request from the error, using the admin secret:
```
curl -i -X GET -H 'Content-Type: application/json' -u "admin:verysecret"
localhost:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=2&instance=Server_pinot-server-0_8098&offset=-1&name=events_upsert_partial__1__0__20231013T1014Z&rowCount=2&memoryUsedBytes=1140
```
The output is:
```
{"status":"COMMIT","isSplitCommitType":true,"buildTimeSec":126,"streamPartitionMsgOffset":"2","controllerVipUrl":"http://pinot-controller-0:9000","offset":2}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
