The GitHub Actions job "Run Integration Tests" on rocketmq.git/zhimin/tiered-store-fixes has succeeded. Run started by GitHub user lizhimins (triggered by lizhimins).
Head commit for run: ceb05e1ed7965da393e23172b5cf6c705c2c6f36 / lizhimins <[email protected]> [ISSUE #11168] Fix tiered storage commit failure reporting and related hazards Found while analysing 24h of production tiered-store logs in cn-hangzhou (302M lines, of which 16,902 ERROR). A transient upload failure was reported as "get file size error after commit" although no size lookup ever took place. TieredStoreException defaults its position to -1, which collides with GET_FILE_SIZE_ERROR, so a provider-reported failure and a failed lookup shared one branch and one message. handleCommitException now performs no remote lookup at all: it reconciles from the length the provider reported on the exception, and otherwise leaves the input stream in place for the next commitAsync, which already did that lookup on the dispatcher thread. This matters because the handler is registered via exceptionally with no executor, so it runs on a netty IO thread for a network provider. GET_FILE_SIZE_ERROR keeps its single meaning, and a failed lookup can only be reported by the code that performs it. The same method logged three differently worded messages with three different field sets, one of which described a success. Both commit failure paths now emit one line with a shared field set and a result discriminator: REMOTE_LANDED, RETRY_AFTER_REWIND and RETRY_AFTER_RECONCILE from handleCommitException, SIZE_LOOKUP_FAILED from commitAsync, which previously called the same quantity "buffer" instead of "content" and omitted expect and remote. The commit position is logged as of the append rather than after correctPosition overwrote it, so commit, content and expect stay mutually consistent. FlatAppendFile#destroyExpiredFile deleted the remote object before unregistering its metadata. A crash in between leaves a metadata row pointing at a deleted object, reloaded on every restart, after which every read of that segment fails with NoSuchKey. Reverse the order so the worst case becomes an orphaned object. FileSegment#readAsync shortened a read to the committed length and logged it at DEBUG. The truncated buffer surfaces later as a splitMessageBuffer failure whose own message carries no topic, queueId or offset, so the cause was invisible in production. Raise it to WARN. MessageStoreFetcherImpl compared fetcherCache.estimatedSize(), a count of entries, against memoryMaxSize * 0.8, a number of bytes, so cacheBusy was always false. Compare the weighted size, which is what maximumWeight bounds. Report URL: https://github.com/apache/rocketmq/actions/runs/34970147065 With regards, GitHub Actions via GitBox
