Aias00 opened a new issue, #7832: URL: https://github.com/apache/shenyu/issues/7832
## Goal and motivation Reduce the bytes and object-processing work associated with large Admin-to-gateway configuration synchronization by introducing an explicit, compact **gateway wire projection**, starting with Discovery upstream data. Keep Admin's internal domain/event objects intact. Reuse existing transport envelopes rather than introducing a generic intermediate event framework or duplicating every configuration model immediately. Reviewed revision: Apache ShenYu `2.7.2-SNAPSHOT`, master `04f7c2e8b7d282f14510157a9ab0bfaab538ecac`. This proposal concerns configuration propagation, initial synchronization and reconnect snapshots. It does not claim that database management fields are on the normal gateway request hot path, or that smaller JSON alone establishes a particular request-latency improvement. ## Current behavior and evidence The same common DTO families are used across Admin event publication, listener caches, Admin-to-Admin event forwarding and gateway-facing serialization. For example: - [WebsocketDataChangedListener.sendByNamespace():148–166](https://github.com/apache/shenyu/blob/04f7c2e8b7d282f14510157a9ab0bfaab538ecac/shenyu-admin/src/main/java/org/apache/shenyu/admin/listener/websocket/WebsocketDataChangedListener.java#L148-L166) groups the existing DTOs by namespace and serializes them directly inside WebsocketData. - [DiscoveryUpstreamData](https://github.com/apache/shenyu/blob/04f7c2e8b7d282f14510157a9ab0bfaab538ecac/shenyu-common/src/main/java/org/apache/shenyu/common/dto/DiscoveryUpstreamData.java#L25-L78) contains both runtime upstream configuration and persistence/binding fields. - ClusterDataChangedEventForwarder serializes event.getSource() for the other Admin. That is an internal/control-plane contract, not automatically the right place to trim a gateway payload. - HTTP polling and registry-backed sync channels have their own serialization/cache/key paths and must be audited explicitly; a WebSocket-only change should not be presented as optimizing every channel. The existing DTOs are already partly runtime-oriented. Most RuleData/SelectorData matching and plugin fields are meaningful. The proposal is targeted field projection, not the assumption that most configuration data is unused. ## Field-level findings Repository consumer searches identified the following **candidates for standard gateway wire payloads**, not fields proven removable from every external/custom consumer: | Object | Candidate fields | Reason / qualification | | --- | --- | --- | | DiscoverySyncData | discoveryHandlerId | Admin binding identifier; no standard gateway read found | | DiscoveryUpstreamData | id, dateUpdated, discoveryHandlerId | Persistence/update metadata; no standard gateway read found | | DiscoveryUpstreamData | per-upstream namespaceId | Repeats namespace information; standard handlers consume the enclosing DiscoverySyncData context. Preserve and validate the outer namespace; audit extensions before removal. | | SelectorData | beforeConditionList | No current built-in gateway read found; updates may carry a previous condition list. Do not confuse this with the required current conditionList. | | RuleData | beforeConditionDataList | Same qualification; current conditionDataList remains required. | The first implementation should focus on Discovery. Treat the old-condition fields as a follow-up after old gateway, cache-invalidation and extension compatibility are verified. ### Fields that must not be dropped indiscriminately - **dateCreated**: [DivideUpstreamDataHandler:52–63](https://github.com/apache/shenyu/blob/04f7c2e8b7d282f14510157a9ab0bfaab538ecac/shenyu-plugin/shenyu-plugin-proxy/shenyu-plugin-divide/src/main/java/org/apache/shenyu/plugin/divide/handler/DivideUpstreamDataHandler.java#L52-L63) converts it into the runtime upstream timestamp. Load-balancer warmup uses that timestamp. WebSocket/gRPC upstream handlers also consume it. Omission can reset it to the current time and change behavior. - **protocol, url, status, weight, props**: required upstream configuration. props can contain warmup, health-check, gray and other plugin metadata; do not trim its opaque contents without a plugin-specific schema/contract. - **DiscoverySyncData.selectorName**: TCP routing and discovery deletion identity use it; it is not merely a UI label. - **SelectorData.pluginId**: LogCollectConfigUtils.isSampled uses it for global logging-sampler fallback. - **RuleData.name**: [McpServerPluginDataHandler:149–153](https://github.com/apache/shenyu/blob/04f7c2e8b7d282f14510157a9ab0bfaab538ecac/shenyu-plugin/shenyu-plugin-mcp-server/src/main/java/org/apache/shenyu/plugin/mcp/server/handler/McpServerPluginDataHandler.java#L149-L153) uses it as a default tool name; it is not universally diagnostic-only. - **Current matching fields, handles and flags**: id/parent identity, pluginName, enabled, sort, matchMode, matchRestful, condition data, continued and plugin handles are used by matching, ordering and plugin handlers. - **Namespace and snapshot/event semantics**: keep the identity required for isolation, grouping, deletion and authoritative replacement. The existing WebsocketData namespaceId/fullSnapshot fields must retain their meaning. MetaData and AppAuthData were also inspected; no safe generic field-removal set was established for them. Their apparent extra fields participate in RPC/metadata or authentication behavior. ### Null versus empty GsonUtils' normal serializer already omits null fields. Null omission is not a new saving to claim. It still serializes empty arrays. SelectorDO/RuleDO snapshot conversion supplies empty before-condition lists in some paths, so omitting a redundant empty field could save repeated names/array syntax, but null/empty/missing semantics and older consumers must first be tested. ## Size experiment: useful evidence, not a production promise A standalone Java probe used the current built DTOs, GsonUtils and WebsocketData to serialize 10,000 synthetic upstreams in a Discovery snapshot, then removed only the Discovery candidates in the table above. Runtime fields including dateCreated were retained. Fixture: - one divide selector and one namespace; - 10,000 distinct upstream addresses; - 19-digit row IDs, a repeated handler ID and a 36-character namespace; - created/updated timestamps, protocol=http://, status=0, weight=100; - props="{}" to make the fixture reproducible. Exact generation for i=0..9999: id=`2111700000000000000+i`; handler ID=`2111700000000000000`; namespace=`649330b6-c2d7-4edc-be8e-8a54df9eb385`; dateCreated=`Timestamp(1792300000000+i*1000)`; dateUpdated=`Timestamp(1792301000000+i*1000)`; url=`10.0.(i/250).(i%250+1):8080`. The outer Discovery item used selectorId=`2111700000000000001`, selectorName=`/example`, pluginName=`divide`, and the same handler ID/namespace. The WebsocketData constructor used DISCOVER_UPSTREAM/MYSELF and its existing defaults. Preserve the current GsonUtils timestamp encoding when reproducing the byte counts. | Representation | Original | Candidate projection | | --- | ---: | ---: | | UTF-8 JSON bytes | 2,793,466 | 1,203,423 | | Additional standalone gzip comparison | 129,926 | 59,014 | The JSON reduction is approximately 57% **for this fixture**. This does not mean all deployments save 57%, that the current sync channel already uses gzip, or that end-to-end latency improves by 57%. Large handles/props and compression can change the distribution substantially. The experiment checked serialized size, not complete mixed-version runtime compatibility. Minimal fixture/reduction recipe: ```java // Populate 10,000 DiscoveryUpstreamData entries with all fields described above. // Wrap them in DiscoverySyncData and WebsocketData("DISCOVER_UPSTREAM", "MYSELF", ...). String original = GsonUtils.getInstance().toJson(frame); JsonObject compact = JsonParser.parseString(original).getAsJsonObject(); for (JsonElement item : compact.getAsJsonArray("data")) { JsonObject discovery = item.getAsJsonObject(); discovery.remove("discoveryHandlerId"); for (JsonElement entry : discovery.getAsJsonArray("upstreamDataList")) { JsonObject upstream = entry.getAsJsonObject(); for (String field : List.of("id", "dateUpdated", "discoveryHandlerId", "namespaceId")) { upstream.remove(field); } // dateCreated and all runtime routing fields remain present. } } // Compare original.getBytes(UTF_8).length with compact.toString().getBytes(UTF_8).length. ``` JSON-tree removal was only the measurement technique. It is **not** the proposed production implementation. ## Proposed design boundary ```text Admin entities / existing DataChangedEvent / internal DTOs | gateway-sync projection | compact, explicitly defined gateway payload DTO | existing transport envelope -> one serialization -> gateway adaptation ``` 1. Keep the original Admin event and shared DTO contracts for internal caches, persistence-related operations and Admin-to-Admin forwarding. 2. Introduce a small, explicit payload DTO only where it differs meaningfully, for example DiscoveryUpstreamSyncDTO and its enclosing gateway view. Names/package placement are proposals; use an existing shared contract module rather than adding a new runtime module/dependency by default. 3. Project from typed source objects directly into that DTO at the gateway-facing boundary and serialize once. Avoid production object -> JSON -> delete fields -> JSON round trips. 4. Reuse WebsocketData for WebSocket framing, and preserve the corresponding existing envelope/key contracts on other channels. Do not build a second generic event bus/envelope merely to remove a few fields. 5. Keep plugin-specific handle/props opaque initially. Do not make projection decisions solely from the fact that one test deployment enables only divide. 6. Do not globally mark existing shared fields transient/ignored: this can silently affect other serialization consumers. ### Event-specific payloads - UPDATE/CREATE need the complete intended runtime item unless a separate patch protocol is explicitly designed; a missing field must not silently mean clear/reset. - DELETE can be more compact, but the key is group/plugin specific, not universally just id. [DiscoveryUpstreamKey](https://github.com/apache/shenyu/blob/04f7c2e8b7d282f14510157a9ab0bfaab538ecac/shenyu-sync-data-center/shenyu-sync-data-api/src/main/java/org/apache/shenyu/sync/data/api/DiscoveryUpstreamKey.java#L22-L38) already defines pluginName, selectorId and optional selectorName; preserve namespace context and audit deletion handlers. - MYSELF/REFRESH must preserve full-group/namespace replacement semantics, including authoritative empty arrays. Omitting redundant fields is different from dropping the empty data array that clears configuration. ## Compatibility and rollout An internal common-DTO getter search does not prove an external plugin, WASM guest, older Bootstrap, or registry consumer ignores a field. Define supported protocol/capability negotiation or a backward-compatible opt-in before enabling compact output in a mixed fleet. Older consumers must retain a supported full-payload path unless compatibility is demonstrated. Audit WebSocket, HTTP polling, registry-backed sync, initial snapshots, incremental changes, failover/reconnect and plugin-store/WASM consumers separately. Preserve identity/path construction and consider representation-specific serialization, hashes/digests, deduplication and cache keys: changing field presence must not cause cross-version cache confusion or repeated refresh storms. ## Acceptance criteria - [ ] Publish a field-to-consumer inventory for the first projected group and distinguish unused built-in fields from extension/legacy contract requirements. - [ ] Retain Admin internal event/cache/cluster-forward behavior; prove the gateway projection does not mutate the shared source objects. - [ ] Implement the minimal typed projection and keep one production serialization step; preserve each affected transport envelope. - [ ] Verify old/new Admin and gateway combinations, custom/WASM consumers where supported, and legacy fallback/capability behavior. - [ ] Test UPDATE/CREATE/DELETE/MYSELF/REFRESH, null/empty/missing fields, authoritative empty snapshots and namespace isolation. - [ ] Test divide/gRPC warmup timestamps, TCP selector naming, logging sampler fallback, MCP default tool names and relevant plugin handlers before declaring fields removable. - [ ] Keep deletion/replacement correctness separate and intact, including #7815 and #7791; smaller messages must not retain deleted routes. - [ ] Compare 1k/10k+ realistic payload shapes, with small and large handles/props. Measure JSON and compressed bytes, serialization/deserialization CPU, allocation/GC, synchronization/apply time and business-request p99 during sync. - [ ] Record gateway/Admin revisions, JVM/configuration, commands and repeat-run variation. Keep only demonstrated benefits; do not extrapolate the synthetic size ratio into a universal latency claim. ## Related work and scope separation - #7650: performance baseline/tracker. - #7645: batching and invalidation while applying received namespace snapshots. This proposal reduces/defines the transmitted representation before application; the two changes should be benchmarked independently. - #7815 / #7791: synchronization/cache correctness prerequisites, not payload-size optimizations. No implementation changes have been made for this proposal. Consumer tracing and the standalone serialization-size experiment are the current evidence. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
