morningman opened a new pull request, #68151:
URL: https://github.com/apache/doris/pull/68151
Cherry-picked from #67783, #68034, #68033, #68057, #68094, #68087
Batch pick of every merged PR carrying the `incremental-computation` label
that `branch-incremental-computation` does not have yet (no
`incremental-computation-picked` label), in the order they landed on master
(`git log --first-parent`). One commit per PR, each created with `git
cherry-pick -x` so the message ends with `(cherry picked from commit <master
sha>)`. Follows the same convention as #67830, #68017 and #68073.
| # | Master commit | PR | Title |
|---|---|---|---|
| 1 | e545b13e089 | #67783 | [fix](policy) Enforce row policies on MOW time
travel |
| 2 | f7a0842f968 | #68034 | [fix](binlog) Fix missing DELETE events in row
binlog |
| 3 | e85575e3ad6 | #68033 | [fix](binlog) Preserve row binlog compaction
policy |
| 4 | 21160d78b42 | #68057 | [fix](binlog) Decouple row binlog compaction
from CCR binlog config |
| 5 | 04aa5a11584 | #68094 | [fix](binlog) Persist row binlog config updates
in cloud mode |
| 6 | da8feed859d | #68087 | [fix](binlog) Report streams with missing base
tables as stale |
Not included on purpose:
- The 20 labelled PRs that already carry `incremental-computation-picked`
(#62606 in the fork point, #67508 via #67712, the nine of #67830, the six of
#68017, the two of #68073, and #68050 whose content this branch got directly
through #68012).
- #68012 carries the label but is a PR against this branch itself (merged as
`6f7c87fa892`); nothing to pick.
- #67820 is still open on master; this branch already carries its content
via #67861.
### Prerequisite check
For every pick I listed the master commits between the fork point
(`efedf10c7e3`) and the pick that touch the same files and are not on this
branch, and checked whether the pick's behavior depends on them.
- **#67783** declares no related PR. It builds on the MOW time-travel
rewrite (`BindRelation.buildMowTimeTravelUnion`, #67480) which is already here.
Two unlabelled master commits overlap: #66770 (authorization plugin SPI:
reworks the data-mask / row-filter API of `LogicalCheckPolicy` and its test)
and #67811 (removes the `isPlayNereidsDump()` check in `CheckPolicy`). Neither
is a functional prerequisite: the fix consists of `getPolicyTable()` unwrapping
`OlapTableWrapper`, `CheckPolicy` collecting the whole filter chain below the
policy mask, and `BindRelation` putting a `LogicalCheckPolicy` above each union
branch — none of that uses the #66770 API. Not picked; adapted instead (see
below).
- **#68034**, **#68033**, **#68087** (#68087 relates to #67173, which is
before the fork point): no unpicked master commit touches any of their files;
the picks applied cleanly and are byte-identical to master.
- **#68057**: only `be/test/cloud/cloud_compaction_test.cpp` overlaps with
the unrelated #67972 (refresh tablet meta of continuously ingested tablets);
auto-merged, the pick only adds new `TEST_F` blocks. Main-code hunks are
byte-identical to master.
- **#68094**: overlaps with #68090 / #67972 / #66598 (`cloud_tablet.cpp`),
#67295 / #67618 / #68090 (`base_tablet.{h,cpp}`), #66598 / #67637
(`meta_service.cpp`), #66598 (`cloud.proto`, `cloud_tablet_test.cpp`) and
#67761 / #66598 / #67637 (`meta_service_test.cpp`). All auto-merged. The fix —
`BaseTablet::binlog_config()` under the meta lock, `BinlogConfig::operator==`,
`CloudTablet::sync_meta()` refreshing the binlog config, `update_tablet` in the
meta service accepting `binlog_config`, and `CloudSchemaChangeHandler` routing
ROW-binlog property updates through it — does not use anything those commits
add. Its hunks are byte-identical to master except one trailing context line in
`sync_meta()` (`last_sync_tablet_meta_time_s` comes from #67972).
Only #67783 needed adaptation, recorded in its commit message:
- `LogicalCheckPolicy.java`: the conflicting context was master's
`parsePolicyExpression()` helper (#66770), which does not exist here; the new
`getPolicyTable()` is inserted in the same place without it.
- `CheckRowPolicyTest.java`: the data-mask mock is written against this
branch's per-column `AccessControllerManager.evalDataMaskPolicy(..., column) ->
Optional<DataMaskPolicy>` instead of master's `evalDataMaskPolicies(...,
Set<String>) -> Map<String, DataMaskSpec>`. The masks are the same (concat for
the random-distribution table; the non-movable `k2` mask plus identity masks
for the hidden reconstruction columns of the MOW table). The master-only `Or`
import (#66770) is not carried; `Collections` / `Locale` imports were added
because they arrive with #66770 on master.
### Drift check against master
Each pick's `+`/`-` lines are identical to the master commit's, except for
the #67783 adaptation above. After the six picks, the touched files still
differ from master at `da8feed859d` in: `cloud_tablet.cpp`,
`base_tablet.{h,cpp}`, `cloud_compaction_test.cpp`, `cloud_tablet_test.cpp`,
`meta_service.cpp`, `meta_service_test.cpp`, `cloud.proto`, `CheckPolicy.java`,
`LogicalCheckPolicy.java`, `CheckRowPolicyTest.java`. Replaying the nine
unpicked master commits listed above (#67637, #67618, #66770, #66598, #67972,
#67811, #67761, #67295, #68090) in a temporary index and removing this branch's
own #67861 (`GetTsoRecoveryTransactions` in `cloud.proto` /
`meta_service_test.cpp`) brings every file to zero diff against master, except
the two `LogicalCheckPolicy.java` / `CheckRowPolicyTest.java` hunks of #66770
that overlap the adaptation. Nothing else is left over.
### Verification
- FE: `run-fe-ut.sh --run` on this branch (regenerates thrift/protobuf,
compiles fe-core main + test) with the test classes touched by the picks plus
`DeleteFromCommandTest` from #68034's checklist: 5 classes, 58 tests, 0
failures, 0 errors, BUILD SUCCESS — `CheckRowPolicyTest` 9 (the three tests
#67783 adds included), `ExplainTableStreamPlanTest` 24,
`CloudSchemaChangeHandlerTest` 18 (the tests of #68033 and #68094 included),
`DropTableStreamTest` 5, `DeleteFromCommandTest` 2.
- FE checkstyle on fe-core: 0 violations.
- BE: `-fsyntax-only` with the flags of the Release build
(`compile_commands.json`, regenerated `gen_cpp` headers incl. the new
`TabletMetaInfoPB.binlog_config`) passes for `cloud/cloud_tablet.cpp`,
`cloud/cloud_storage_engine.cpp`, `storage/olap_server.cpp`,
`storage/tablet/tablet.cpp`, `storage/tablet/base_tablet.cpp`, and with
`-DBE_TEST -fno-access-control` for `test/cloud/cloud_compaction_test.cpp`,
`test/cloud/cloud_tablet_test.cpp`,
`test/storage/compaction/compaction_task_test.cpp`,
`test/storage/tablet/tablet_test.cpp`.
- Meta service: the two-line `update_tablet` hunk of #68094 only uses the
generated `TabletMetaInfoPB::has_binlog_config()/binlog_config()` and
`TabletMetaCloudPB::mutable_binlog_config()` accessors, all present in the
regenerated headers.
- The two new groovy suites
(`row_binlog_p0/test_row_binlog_mow_light_delete`,
`time_travel_p0/test_mow_time_travel_row_policy`) parse cleanly (groovy parser
check).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
