This is an automated email from the ASF dual-hosted git repository. chaokunyang pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/fory-site.git
commit 0f109758fb215a0a95324117ab1e2f702884c9da Author: chaokunyang <[email protected]> AuthorDate: Sat Jul 4 18:12:45 2026 +0000 🔄 synced local 'docs/guide/' with remote 'docs/guide/' --- docs/guide/cpp/configuration.md | 49 +++++++++++++++++++++++------- docs/guide/csharp/basic-serialization.md | 5 +++ docs/guide/csharp/configuration.md | 41 +++++++++++++++++++------ docs/guide/csharp/references.md | 4 +++ docs/guide/dart/configuration.md | 41 +++++++++++++++++++------ docs/guide/go/configuration.md | 40 ++++++++++++++++++------ docs/guide/go/custom-serializers.md | 2 +- docs/guide/go/index.md | 4 +-- docs/guide/go/native-serialization.md | 4 +-- docs/guide/java/configuration.md | 9 ++++++ docs/guide/javascript/configuration.md | 52 ++++++++++++++++++++++++-------- docs/guide/kotlin/configuration.md | 19 ++++++++++++ docs/guide/python/configuration.md | 43 +++++++++++++++++--------- docs/guide/rust/configuration.md | 44 +++++++++++++++++++++------ docs/guide/scala/configuration.md | 19 ++++++++++++ docs/guide/swift/configuration.md | 18 +++++++++-- 16 files changed, 311 insertions(+), 83 deletions(-) diff --git a/docs/guide/cpp/configuration.md b/docs/guide/cpp/configuration.md index d617450041..d72f82b330 100644 --- a/docs/guide/cpp/configuration.md +++ b/docs/guide/cpp/configuration.md @@ -96,6 +96,30 @@ When enabled, avoids duplicating shared objects and handles cycles. **Default:** `true` +### max_graph_memory_bytes(int64_t) + +Set an approximate graph-memory gate for one root deserialization. + +```cpp +auto fory = Fory::builder() + .max_graph_memory_bytes(64 * 1024 * 1024) + .build(); +``` + +The default limit is a fixed `128 MiB` for byte-array, `Buffer`, and stream +roots. Positive values override the default. Explicit non-positive values are +rejected when the runtime is created. + +This budget is an approximate lower-bound estimate for materialized graph +owners, mainly collections, maps, arrays, structs, and objects. It is not an +exact process heap limit; actual process memory can be higher. Dedicated +string, binary, primitive scalar, and primitive dense-array leaf values are +skipped by this budget and continue to rely on byte-availability checks: if the +unread input does not contain enough bytes, Fory will not read or create that +leaf value. + +**Default:** `128 MiB` + ### max_dyn_depth(uint32_t) Set maximum allowed nesting depth for dynamically-typed objects. @@ -200,17 +224,18 @@ auto fory = Fory::builder().build_thread_safe(); // Returns ThreadSafeFory ## Configuration Summary -| Option | Description | Default | -| ------------------------------------------------ | ------------------------------------------------- | ------- | -| `xlang(bool)` | Use xlang mode | `true` | -| `compatible(bool)` | Enable schema evolution | `true` | -| `track_ref(bool)` | Enable reference tracking | `true` | -| `max_dyn_depth(uint32_t)` | Maximum nesting depth for dynamic types | `5` | -| `max_type_fields(uint32_t)` | Max fields in one received struct metadata body | `512` | -| `max_type_meta_bytes(uint32_t)` | Max encoded bytes in one received metadata body | `4096` | -| `max_schema_versions_per_type(uint32_t)` | Max remote metadata versions for one logical type | `10` | -| `max_average_schema_versions_per_type(uint32_t)` | Average remote metadata versions across types | `3` | -| `check_struct_version(bool)` | Enable struct version checking | `false` | +| Option | Description | Default | +| ------------------------------------------------ | ------------------------------------------------- | --------- | +| `xlang(bool)` | Use xlang mode | `true` | +| `compatible(bool)` | Enable schema evolution | `true` | +| `track_ref(bool)` | Enable reference tracking | `true` | +| `max_graph_memory_bytes(int64_t)` | Approximate graph-memory gate per root read | `128 MiB` | +| `max_dyn_depth(uint32_t)` | Maximum nesting depth for dynamic types | `5` | +| `max_type_fields(uint32_t)` | Max fields in one received struct metadata body | `512` | +| `max_type_meta_bytes(uint32_t)` | Max encoded bytes in one received metadata body | `4096` | +| `max_schema_versions_per_type(uint32_t)` | Max remote metadata versions for one logical type | `10` | +| `max_average_schema_versions_per_type(uint32_t)` | Average remote metadata versions across types | `3` | +| `check_struct_version(bool)` | Enable struct version checking | `false` | ## Security @@ -218,6 +243,8 @@ Security-related configuration: - Register all structs and polymorphic implementations before deserializing untrusted payloads. - Use `check_struct_version(true)` with `compatible(false)` for intentional same-schema payloads. +- Keep `max_graph_memory_bytes(...)` at the fixed `128 MiB` default for most inputs, or set a + positive value for a trusted workload that needs a different collection/map/struct gate. - Keep `max_dyn_depth(...)` as low as your model permits to reject unexpectedly deep polymorphic graphs. - Keep the remote schema metadata limits at their defaults unless the data is not malicious and a diff --git a/docs/guide/csharp/basic-serialization.md b/docs/guide/csharp/basic-serialization.md index d112990ccd..5c88675bc8 100644 --- a/docs/guide/csharp/basic-serialization.md +++ b/docs/guide/csharp/basic-serialization.md @@ -104,6 +104,11 @@ byte[] payload = fory.Serialize<object?>(value); object? decoded = fory.Deserialize<object?>(payload); ``` +Dynamic maps normally decode as `Dictionary<object, object?>` when they have no +null key. If the payload uses reference tracking for the dynamic map itself, C# +returns `NullableKeyDictionary<object, object?>` so nested references and null +keys point to the decoded map owner. + ## Buffer Writer API Serialize directly into `IBufferWriter<byte>` targets. diff --git a/docs/guide/csharp/configuration.md b/docs/guide/csharp/configuration.md index e7c0c24d42..b8891270d9 100644 --- a/docs/guide/csharp/configuration.md +++ b/docs/guide/csharp/configuration.md @@ -35,16 +35,17 @@ ThreadSafeFory threadSafe = Fory.Builder().BuildThreadSafe(); `Fory.Builder().Build()` uses: -| Option | Default | Description | -| --------------------------------- | ------- | ------------------------------------------------- | -| `TrackRef` | `false` | Reference tracking disabled | -| `Compatible` | `true` | Compatible schema-evolution metadata enabled | -| `CheckStructVersion` | `false` | Struct schema hash checks disabled | -| `MaxDepth` | `20` | Max dynamic nesting depth | -| `MaxTypeFields` | `512` | Max fields in one received struct metadata body | -| `MaxTypeMetaBytes` | `4096` | Max encoded bytes in one received metadata body | -| `MaxSchemaVersionsPerType` | `10` | Max remote metadata versions for one logical type | -| `MaxAverageSchemaVersionsPerType` | `3` | Average remote metadata versions across types | +| Option | Default | Description | +| --------------------------------- | ----------- | ------------------------------------------------- | +| `TrackRef` | `false` | Reference tracking disabled | +| `Compatible` | `true` | Compatible schema-evolution metadata enabled | +| `CheckStructVersion` | `false` | Struct schema hash checks disabled | +| `MaxDepth` | `20` | Max dynamic nesting depth | +| `MaxGraphMemoryBytes` | `134217728` | Approximate graph-memory gate per root read | +| `MaxTypeFields` | `512` | Max fields in one received struct metadata body | +| `MaxTypeMetaBytes` | `4096` | Max encoded bytes in one received metadata body | +| `MaxSchemaVersionsPerType` | `10` | Max remote metadata versions for one logical type | +| `MaxAverageSchemaVersionsPerType` | `3` | Average remote metadata versions across types | ## Builder Options @@ -96,6 +97,24 @@ Fory fory = Fory.Builder() `value` must be greater than `0`. +### `MaxGraphMemoryBytes(long value)` + +Sets an approximate graph-memory gate for one root deserialization. The estimate mainly covers +materialized collections, maps, arrays, structs, and objects. It skips leaf values such as strings, +binary data, primitive scalars, and dense primitive arrays, so actual process memory can be higher +than this value. + +```csharp +Fory fory = Fory.Builder() + .MaxGraphMemoryBytes(64L * 1024 * 1024) + .Build(); +``` + +The default limit is a fixed `128 MiB` for all root input forms. A positive value overrides the +default. Explicit non-positive values are rejected when the runtime is created. Skipped leaf values +are still gated by remaining input bytes: if the unread input does not contain enough bytes, Fory +will not read or create that leaf value. + ### `MaxTypeFields(int value)` Sets the maximum fields accepted in one received remote struct metadata body. @@ -173,6 +192,8 @@ Security-related configuration: - Register only the expected types before deserializing untrusted payloads. - Use `CheckStructVersion(true)` with `Compatible(false)` for intentional same-schema payloads. - Set `MaxDepth(...)` to reject unexpectedly deep dynamic object graphs. +- Set `MaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input bytes. - Keep the remote schema metadata limits at their defaults unless the data is not malicious and a trusted peer sends larger metadata or many schema versions. - Prefer generated or registered concrete models over broad dynamic fields for untrusted input. diff --git a/docs/guide/csharp/references.md b/docs/guide/csharp/references.md index 772b2535ca..6b6b2363c0 100644 --- a/docs/guide/csharp/references.md +++ b/docs/guide/csharp/references.md @@ -65,6 +65,10 @@ System.Diagnostics.Debug.Assert(object.ReferenceEquals(decoded, decoded.Next)); `TrackRef(false)` can be faster for tree-like, acyclic data where reference identity does not matter. +C# union wrappers are immutable and are created after their case payload is read. +Reference cycles from a union case payload back to the containing union are not +supported; Fory rejects unresolved refs instead of returning a partial union. + ## Related Topics - [Configuration](configuration.md) diff --git a/docs/guide/dart/configuration.md b/docs/guide/dart/configuration.md index 6a4c640f6a..238700a6d5 100644 --- a/docs/guide/dart/configuration.md +++ b/docs/guide/dart/configuration.md @@ -38,6 +38,7 @@ final fory = Fory( maxTypeMetaBytes: 4096, maxSchemaVersionsPerType: 10, maxAverageSchemaVersionsPerType: 3, + maxGraphMemoryBytes: 64 * 1024 * 1024, ); ``` @@ -107,17 +108,37 @@ final fory = Fory( - `maxAverageSchemaVersionsPerType` limits the average across accepted remote types. The effective global floor is `8192` schemas. +### `maxGraphMemoryBytes` + +Sets an approximate graph-memory gate for one root deserialization. The estimate mainly covers +materialized lists, sets, maps, arrays, structs, and objects. It skips leaf values such as strings, +binary data, primitive scalars, and dense typed-array payloads, so actual process memory can be +higher than this value. Leaf values remain protected by byte-availability checks: if the unread +input does not contain enough bytes, Fory will not read or create that leaf value. + +The default is a fixed `128 MiB` and is not derived from input size. + +Set a positive value when a trusted workload legitimately needs a larger or smaller +collection/map/struct gate: + +```dart +final fory = Fory(maxGraphMemoryBytes: 256 * 1024 * 1024); +``` + +Explicit non-positive values are rejected when the runtime is created. + ## Defaults -| Option | Default | -| --------------------------------- | ------- | -| `compatible` | `true` | -| `checkStructVersion` | `false` | -| `maxDepth` | 256 | -| `maxTypeFields` | 512 | -| `maxTypeMetaBytes` | 4096 | -| `maxSchemaVersionsPerType` | 10 | -| `maxAverageSchemaVersionsPerType` | 3 | +| Option | Default | +| --------------------------------- | --------- | +| `compatible` | `true` | +| `checkStructVersion` | `false` | +| `maxDepth` | 256 | +| `maxTypeFields` | 512 | +| `maxTypeMetaBytes` | 4096 | +| `maxSchemaVersionsPerType` | 10 | +| `maxAverageSchemaVersionsPerType` | 3 | +| `maxGraphMemoryBytes` | 134217728 | ## Xlang Notes @@ -134,6 +155,8 @@ Security-related configuration: - Register only the expected generated models before deserializing untrusted payloads. - Use `checkStructVersion: true` with `compatible: false` for intentional same-schema payloads. - Set `maxDepth` to reject unexpectedly deep payload shapes. +- Keep `maxGraphMemoryBytes` at the default for most inputs, or set an explicit positive byte gate + for known trusted collection/map/struct-heavy payloads. - Keep the remote schema metadata limits at their defaults unless the data is not malicious and a trusted peer sends larger metadata or many schema versions. - Prefer generated schemas and explicit field metadata over broad dynamic fields for untrusted input. diff --git a/docs/guide/go/configuration.md b/docs/guide/go/configuration.md index 20d9012aee..2ddad20800 100644 --- a/docs/guide/go/configuration.md +++ b/docs/guide/go/configuration.md @@ -33,16 +33,17 @@ f := fory.New(fory.WithXlang(true)) Default settings: -| Option | Default | Description | -| ------------------------------- | ------- | ------------------------------------------------- | -| TrackRef | false | Reference tracking disabled | -| MaxDepth | 20 | Maximum nesting depth | -| IsXlang | true | Xlang mode enabled | -| Compatible | true | Compatible schema-evolution metadata enabled | -| MaxTypeFields | 512 | Max fields in one received struct metadata body | -| MaxTypeMetaBytes | 4096 | Max encoded bytes in one received metadata body | -| MaxSchemaVersionsPerType | 10 | Max remote metadata versions for one logical type | -| MaxAverageSchemaVersionsPerType | 3 | Average remote metadata versions across types | +| Option | Default | Description | +| ------------------------------- | --------- | ------------------------------------------------- | +| TrackRef | false | Reference tracking disabled | +| MaxDepth | 20 | Maximum nesting depth | +| IsXlang | true | Xlang mode enabled | +| Compatible | true | Compatible schema-evolution metadata enabled | +| MaxGraphMemoryBytes | 134217728 | Approximate graph-memory gate per root read | +| MaxTypeFields | 512 | Max fields in one received struct metadata body | +| MaxTypeMetaBytes | 4096 | Max encoded bytes in one received metadata body | +| MaxSchemaVersionsPerType | 10 | Max remote metadata versions for one logical type | +| MaxAverageSchemaVersionsPerType | 3 | Average remote metadata versions across types | ### With Options @@ -51,6 +52,7 @@ f := fory.New( fory.WithXlang(true), fory.WithTrackRef(true), fory.WithMaxDepth(10), + fory.WithMaxGraphMemoryBytes(128 * 1024 * 1024), fory.WithMaxTypeFields(512), fory.WithMaxTypeMetaBytes(4096), fory.WithMaxSchemaVersionsPerType(10), @@ -127,6 +129,24 @@ f := fory.New(fory.WithMaxDepth(30)) - Protects against deeply nested, recursive structures or malicious data - Serialization fails with error when exceeded +### WithMaxGraphMemoryBytes + +Set an approximate graph-memory gate for one root deserialization: + +```go +f := fory.New(fory.WithMaxGraphMemoryBytes(256 * 1024 * 1024)) +``` + +The estimate mainly covers materialized slices, maps, sets, arrays, structs, and objects. It skips +leaf values such as strings, binary data, primitive scalars, and dense primitive arrays, so actual +process memory can be higher than this value. + +The default limit is a fixed `128 MiB` for all root input forms. A positive value overrides the +default. Explicit non-positive values are rejected when the runtime is created. Graph memory +reservation complements byte-availability checks; it does not replace them. Skipped leaf values are +still gated by remaining input bytes: if the unread input does not contain enough bytes, Fory will +not read or create that leaf value. + ### WithMaxTypeFields Set the maximum fields accepted in one received remote struct metadata body: diff --git a/docs/guide/go/custom-serializers.md b/docs/guide/go/custom-serializers.md index 9c0461f0fc..3f9a74503e 100644 --- a/docs/guide/go/custom-serializers.md +++ b/docs/guide/go/custom-serializers.md @@ -25,7 +25,7 @@ Custom serializers allow you to define exactly how a type is serialized and dese - **Special encoding**: Types that need a specific binary format - **Third-party types**: Types from external libraries that Fory doesn't handle automatically -- **Optimization**: When you can serialize more efficiently than the default reflection-based approach +- **Optimization**: When you can serialize more efficiently than the default fast serializer path - **Cross-language compatibility**: When you need precise control over the binary format for interoperability ## ExtensionSerializer Interface diff --git a/docs/guide/go/index.md b/docs/guide/go/index.md index 1a57278b6d..df5f5e5bc6 100644 --- a/docs/guide/go/index.md +++ b/docs/guide/go/index.md @@ -19,14 +19,14 @@ license: | limitations under the License. --- -Apache Fory Go is a high-performance serialization library for Go. It supports xlang mode for cross-language payloads and native mode for Go-only payloads, with reflection-based object graph serialization, circular references, polymorphism, and schema-aware struct handling. +Apache Fory Go is a high-performance serialization library for Go. It supports xlang mode for cross-language payloads and native mode for Go-only payloads, with fast object graph serialization, circular references, polymorphism, and schema-aware struct handling. ## Why Fory Go? - **High Performance**: Fast serialization and optimized binary protocols - **Xlang**: Seamless data exchange with Java, Python, C++, Rust, JavaScript/TypeScript, C#, Swift, Dart, Scala, and Kotlin -- **Automatic Serialization**: Serialize Go structs directly with reflection +- **Automatic Serialization**: Serialize Go structs with fast serializers - **Reference Tracking**: Built-in support for circular references and shared objects - **Type Safety**: Strong typing with schema-aware serializers - **Schema Evolution**: Compatible mode for forward/backward compatibility diff --git a/docs/guide/go/native-serialization.md b/docs/guide/go/native-serialization.md index 1fce23d30f..32680064e6 100644 --- a/docs/guide/go/native-serialization.md +++ b/docs/guide/go/native-serialization.md @@ -38,7 +38,7 @@ Use native serialization when: the writer. - You want compatible schema evolution for Go-only rolling deployments without committing to a cross-language type mapping. -- You are using Go reflection-based serializers for structs that never leave Go. +- You are using Go fast serializers for structs that never leave Go. ## Create a Native-Mode Fory Instance @@ -126,7 +126,7 @@ Native serialization keeps Go data in Go-native form: - Pointers and nil values, including nil slices and maps. - Interfaces and dynamic values when registered serializers can resolve their concrete types. - Time values such as `time.Time` and `time.Duration`. -- Reflection-based serializers. +- Fast serializers. Use [Supported Types](supported-types.md) for the full type surface and xlang mapping details. diff --git a/docs/guide/java/configuration.md b/docs/guide/java/configuration.md index 7b3ce60bf6..d3a56e5e97 100644 --- a/docs/guide/java/configuration.md +++ b/docs/guide/java/configuration.md @@ -38,6 +38,7 @@ This page documents all configuration options available through `ForyBuilder`. | `registerGuavaTypes` | Whether to pre-register Guava types such as `RegularImmutableMap`/`RegularImmutableList`. These types are not public API, but seem pretty stable. [...] | `requireClassRegistration` | Disabling may allow unknown classes to be deserialized, potentially causing security risks. [...] | `maxDepth` | Set max depth for deserialization, when depth exceeds, an exception will be thrown. This can be used to refuse deserialization DDOS attack. [...] +| `maxGraphMemoryBytes` | Approximate graph-memory gate for one root deserialization. It mainly covers materialized collections, maps, arrays, structs, and objects; leaf values are gated by remaining input bytes. [...] | `maxTypeFields` | Maximum fields accepted in one received remote struct metadata body. [...] | `maxTypeMetaBytes` | Maximum encoded body bytes accepted for one received TypeDef or TypeMeta body, excluding the 8-byte header and any extended-size varint. [...] | `maxSchemaVersionsPerType` | Maximum accepted remote metadata versions for one logical type. [...] @@ -90,6 +91,7 @@ Keep class registration enabled for production and any untrusted payload source: Fory fory = Fory.builder() .requireClassRegistration(true) .withMaxDepth(50) + .withMaxGraphMemoryBytes(128L * 1024 * 1024) .build(); ``` @@ -97,6 +99,13 @@ Security-related options: - `requireClassRegistration(true)` restricts deserialization to registered classes. - `withMaxDepth(...)` rejects unexpectedly deep object graphs. +- `withMaxGraphMemoryBytes(...)` sets an approximate gate for materialized graph memory during one + root deserialization. The estimate mainly covers collections, maps, arrays, structs, and objects; + it skips leaf values such as strings, binary data, primitive scalars, and dense primitive arrays. + Actual process memory can be higher than this limit. Leaf values remain protected by + byte-availability checks: if the unread input does not contain enough bytes, Fory will not read or + create that leaf value. The default is a fixed `128 MiB`; set a positive byte limit when trusted + workloads need a larger or smaller gate. - `withMaxTypeFields(...)` and `withMaxTypeMetaBytes(...)` bound the field count and encoded body size of one received remote metadata body. - `withMaxSchemaVersionsPerType(...)` and diff --git a/docs/guide/javascript/configuration.md b/docs/guide/javascript/configuration.md index 058bccf4b3..f3051a1b53 100644 --- a/docs/guide/javascript/configuration.md +++ b/docs/guide/javascript/configuration.md @@ -43,6 +43,7 @@ const fory = new Fory({ ref: true, compatible: true, maxDepth: 100, + maxGraphMemoryBytes: 128 * 1024 * 1024, maxTypeFields: 512, maxTypeMetaBytes: 4096, maxSchemaVersionsPerType: 10, @@ -51,18 +52,19 @@ const fory = new Fory({ }); ``` -| Option | Default | Description | -| --------------------------------- | ------- | ------------------------------------------------------------------------------------- | -| `ref` | `false` | Enable reference tracking for shared or circular object graphs | -| `compatible` | `true` | Allow field additions/removals without breaking existing messages | -| `maxDepth` | `50` | Maximum nesting depth. Must be `>= 2`. Increase for deeply nested structures | -| `maxTypeFields` | `512` | Maximum fields accepted in one received remote struct metadata body | -| `maxTypeMetaBytes` | `4096` | Maximum encoded body bytes accepted for one received TypeMeta body | -| `maxSchemaVersionsPerType` | `10` | Maximum accepted remote metadata versions for one logical type | -| `maxAverageSchemaVersionsPerType` | `3` | Average accepted remote metadata versions across accepted remote types | -| `useSliceString` | `false` | Optional string-reading optimization for Node.js. Leave at default unless benchmarked | -| `hps` | unset | Optional fast string helper from `@apache-fory/hps` (Node.js 20+) | -| `hooks.afterCodeGenerated` | unset | Callback to inspect the generated serializer code, useful for debugging | +| Option | Default | Description | +| --------------------------------- | --------- | ------------------------------------------------------------------------------------- | +| `ref` | `false` | Enable reference tracking for shared or circular object graphs | +| `compatible` | `true` | Allow field additions/removals without breaking existing messages | +| `maxDepth` | `50` | Maximum nesting depth. Must be `>= 2`. Increase for deeply nested structures | +| `maxGraphMemoryBytes` | `128 MiB` | Approximate graph-memory gate accepted during one root deserialization | +| `maxTypeFields` | `512` | Maximum fields accepted in one received remote struct metadata body | +| `maxTypeMetaBytes` | `4096` | Maximum encoded body bytes accepted for one received TypeMeta body | +| `maxSchemaVersionsPerType` | `10` | Maximum accepted remote metadata versions for one logical type | +| `maxAverageSchemaVersionsPerType` | `3` | Average accepted remote metadata versions across accepted remote types | +| `useSliceString` | `false` | Optional string-reading optimization for Node.js. Leave at default unless benchmarked | +| `hps` | unset | Optional fast string helper from `@apache-fory/hps` (Node.js 20+) | +| `hooks.afterCodeGenerated` | unset | Callback to inspect the generated serializer code, useful for debugging | ## Reference Tracking @@ -92,6 +94,29 @@ to that struct. For cross-language payloads, set `compatible: false` only after verifying that every language uses the same schema, or when native types are generated from Fory schema IDL. See [Schema Evolution](schema-evolution.md). +## Graph Memory Budget + +`maxGraphMemoryBytes` sets an approximate graph-memory gate for one root deserialization. The +estimate mainly covers materialized arrays, sets, maps, structs, and objects. It skips leaf values +such as strings, binary data, primitive scalars, and dense primitive arrays, so actual JavaScript +heap use can be higher than this value. Leaf values remain protected by byte-availability checks: if +the unread input does not contain enough bytes, Fory will not read or create that leaf value. The +default is a fixed `128 MiB` and is not derived from input size. + +Use a positive byte value to set an explicit lower or higher limit: + +```ts +const fory = new Fory({ + maxGraphMemoryBytes: 32 * 1024 * 1024, +}); +``` + +Explicit non-positive values are rejected when the runtime is created. + +String, binary, and dedicated dense primitive array payloads keep their normal +byte-size checks and do not consume this graph budget. Raise the limit only for +trusted workloads that legitimately contain very compact object graphs. + ## Optional HPS String Path `@apache-fory/hps` provides an optional Node.js string fast path: @@ -110,6 +135,9 @@ Security-related configuration: - Register only the expected schemas before deserializing untrusted payloads. - Set `maxDepth` for the maximum nesting depth your service accepts. +- Set `maxGraphMemoryBytes` as an approximate gate for collection, map, array, struct, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input + bytes. - Keep `maxTypeFields` and `maxTypeMetaBytes` at their defaults unless the data is not malicious and a trusted peer sends larger remote metadata. - Keep `maxSchemaVersionsPerType` and diff --git a/docs/guide/kotlin/configuration.md b/docs/guide/kotlin/configuration.md index 061f14a7f8..069de266d2 100644 --- a/docs/guide/kotlin/configuration.md +++ b/docs/guide/kotlin/configuration.md @@ -116,6 +116,21 @@ val fory = ForyKotlin.builder().withXlang(false) .build() ``` +### Graph Memory Budget + +Kotlin uses the Java `withMaxGraphMemoryBytes(...)` option. It sets an approximate graph-memory gate +for one root deserialization, mainly for materialized collections, maps, arrays, structs, and +objects. It skips leaf values such as strings, binary data, primitive scalars, and dense primitive +arrays, so actual process memory can be higher than this value. Leaf values are still gated by +remaining input bytes: if the unread input does not contain enough bytes, Fory will not read or +create that leaf value. + +```kotlin +val fory = ForyKotlin.builder() + .withMaxGraphMemoryBytes(128L * 1024 * 1024) + .build() +``` + ## Compatible Mode Compatible mode is enabled by default through the Java builder in both xlang and native mode. Keep @@ -135,6 +150,7 @@ and any untrusted payload source: val fory = ForyKotlin.builder() .requireClassRegistration(true) .withMaxDepth(50) + .withMaxGraphMemoryBytes(128L * 1024 * 1024) .withMaxTypeFields(512) .withMaxTypeMetaBytes(4096) .build() @@ -144,6 +160,9 @@ Security-related configuration: - Keep `requireClassRegistration(true)` and register application classes or generated modules. - Use `withMaxDepth(...)` to reject unexpectedly deep object graphs. +- Use `withMaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input + bytes. - Keep `withMaxTypeFields(...)`, `withMaxTypeMetaBytes(...)`, and the remote schema-version limits at their defaults unless the data is not malicious and a trusted peer sends larger metadata or many schema versions. diff --git a/docs/guide/python/configuration.md b/docs/guide/python/configuration.md index fdd6459fea..e150f34a5a 100644 --- a/docs/guide/python/configuration.md +++ b/docs/guide/python/configuration.md @@ -40,6 +40,7 @@ class Fory: max_type_meta_bytes: int = 4096, max_schema_versions_per_type: int = 10, max_average_schema_versions_per_type: int = 3, + max_graph_memory_bytes: int = 128 * 1024 * 1024, policy: DeserializationPolicy = None, field_nullable: bool = False, meta_compressor=None, @@ -59,21 +60,22 @@ class ThreadSafeFory: ## Parameters -| Parameter | Type | Default | Description | -| -------------------------------------- | ------------------------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `xlang` | `bool` | `True` | Use xlang mode. Set `False` for Python native mode. | -| `ref` | `bool` | `False` | Enable reference tracking for shared/circular references. Disable for better performance if your data has no shared references. | -| `strict` | `bool` | `True` | Require type registration for security. Keep this enabled for production unless a policy owns trust decisions. | -| `compatible` | `bool \| None` | `None` | Schema evolution mode. `None` enables compatible mode in both xlang and native mode. Set `False` only when every reader and writer uses the same schema. | -| `max_depth` | `int` | `50` | Maximum deserialization depth for security, preventing stack overflow attacks. | -| `max_type_fields` | `int` | `512` | Maximum fields accepted in one received remote struct metadata body. | -| `max_type_meta_bytes` | `int` | `4096` | Maximum encoded body bytes accepted for one received TypeDef body, excluding the 8-byte header and any extended-size varint. | -| `max_schema_versions_per_type` | `int` | `10` | Maximum accepted remote metadata versions for one logical type. | -| `max_average_schema_versions_per_type` | `int` | `3` | Average accepted remote metadata versions across accepted remote types. The effective global floor is `8192` schemas. | -| `policy` | `DeserializationPolicy \| None` | `None` | Deserialization policy used for security checks. Strongly recommended when `strict=False`. | -| `field_nullable` | `bool` | `False` | Treat dataclass fields as nullable by default. | -| `meta_compressor` | `Any` | `None` | Optional metadata compressor used for compatible-mode metadata encoding. | -| `fory_factory` | `Callable \| None` | `None` | `ThreadSafeFory` factory hook. When set, `ThreadSafeFory` creates instances via this callback; otherwise it forwards `**kwargs` to `Fory` construction. | +| Parameter | Type | Default | Description | +| -------------------------------------- | ------------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `xlang` | `bool` | `True` | Use xlang mode. Set `False` for Python native mode. | +| `ref` | `bool` | `False` | Enable reference tracking for shared/circular references. Disable for better performance if your data has no shared references. | +| `strict` | `bool` | `True` | Require type registration for security. Keep this enabled for production unless a policy owns trust decisions. | +| `compatible` | `bool \| None` | `None` | Schema evolution mode. `None` enables compatible mode in both xlang and native mode. Set `False` only when every reader and writer uses the same schema. | +| `max_depth` | `int` | `50` | Maximum deserialization depth for security, preventing stack overflow attacks. | +| `max_type_fields` | `int` | `512` | Maximum fields accepted in one received remote struct metadata body. | +| `max_type_meta_bytes` | `int` | `4096` | Maximum encoded body bytes accepted for one received TypeDef body, excluding the 8-byte header and any extended-size varint. | +| `max_schema_versions_per_type` | `int` | `10` | Maximum accepted remote metadata versions for one logical type. | +| `max_average_schema_versions_per_type` | `int` | `3` | Average accepted remote metadata versions across accepted remote types. The effective global floor is `8192` schemas. | +| `max_graph_memory_bytes` | `int` | `134217728` | Approximate graph-memory gate for one root deserialization. Explicit non-positive values are rejected. | +| `policy` | `DeserializationPolicy \| None` | `None` | Deserialization policy used for security checks. Strongly recommended when `strict=False`. | +| `field_nullable` | `bool` | `False` | Treat dataclass fields as nullable by default. | +| `meta_compressor` | `Any` | `None` | Optional metadata compressor used for compatible-mode metadata encoding. | +| `fory_factory` | `Callable \| None` | `None` | `ThreadSafeFory` factory hook. When set, `ThreadSafeFory` creates instances via this callback; otherwise it forwards `**kwargs` to `Fory` construction. | ## Key Methods @@ -197,6 +199,7 @@ fory = pyfory.Fory( max_type_meta_bytes=4096, max_schema_versions_per_type=10, max_average_schema_versions_per_type=3, + max_graph_memory_bytes=128 * 1024 * 1024, ) fory.register(UserModel, name="example.User") @@ -222,6 +225,14 @@ Received remote metadata is also limited: - `max_type_meta_bytes` limits the encoded body bytes accepted for one received TypeDef body. - `max_schema_versions_per_type` limits accepted remote metadata versions for one logical type. - `max_average_schema_versions_per_type` limits the average across accepted remote types. +- `max_graph_memory_bytes` sets an approximate gate for materialized graph memory during one root + deserialization. The estimate mainly covers lists, tuples, sets, dicts, object arrays, structs, + and Python objects. It skips leaf values such as strings, binary data, primitive scalars, and + dense primitive arrays, so actual process memory can be higher than this value. Leaf values remain + protected by byte-availability checks: if the unread input does not contain enough bytes, Fory + will not read or create that leaf value. The default is a fixed `128 MiB` for all root input + forms. Set a positive byte value for trusted payloads that legitimately need a larger or smaller + gate. These limits do not change `strict`, `policy`, dynamic loading, unknown-class handling, or schema-evolution semantics. @@ -278,6 +289,8 @@ unchanged. - Register all expected application types before deserialization. - Use `DeserializationPolicy` when `strict=False` is necessary. - Keep `max_depth` low enough to reject unexpectedly deep payloads. +- Keep `max_graph_memory_bytes` at the fixed `128 MiB` default for most inputs, or set a positive + explicit gate for trusted workloads with different legitimate collection/map/struct sizes. - Do not treat xlang/native mode choice as a security control. ## Related Topics diff --git a/docs/guide/rust/configuration.md b/docs/guide/rust/configuration.md index 58bd070567..c851c56479 100644 --- a/docs/guide/rust/configuration.md +++ b/docs/guide/rust/configuration.md @@ -110,6 +110,24 @@ let fory = Fory::builder() - `max_average_schema_versions_per_type` defaults to `3` and limits the average across accepted remote types. The effective global floor is `8192` schemas. +### Graph Memory Budget + +`max_graph_memory_bytes(...)` sets an approximate graph-memory gate for one root read. The estimate +mainly covers materialized collections, maps, arrays, structs, and objects; it is not an exact +process heap limit. It skips leaf values such as strings, binary data, primitive scalars, and dense +primitive arrays, so actual process memory can be higher than this value. Leaf values remain +protected by byte-availability checks: if the unread input does not contain enough bytes, Fory will +not read or create that leaf value. The default is a fixed `128 MiB` for all root input forms. Set a +positive byte value when trusted payloads need a larger or smaller gate: + +```rust +let fory = Fory::builder() + .max_graph_memory_bytes(256 * 1024 * 1024) + .build(); +``` + +Zero is rejected when the runtime is created. + ### Explicit Xlang Examples Set `.xlang(true)` explicitly for xlang serialization examples: @@ -135,6 +153,11 @@ let fory = Fory::builder().xlang(false).compatible(false).build(); // Custom depth limit let fory = Fory::builder().max_dyn_depth(10).build(); +// Custom graph memory budget +let fory = Fory::builder() + .max_graph_memory_bytes(256 * 1024 * 1024) + .build(); + // Combined configuration let fory = Fory::builder() .xlang(false) @@ -144,15 +167,16 @@ let fory = Fory::builder() ## Configuration Summary -| Option | Description | Default | -| --------------------------------------------- | ------------------------------------------------- | ------- | -| `compatible(bool)` | Enable schema evolution | `true` | -| `xlang(bool)` | Use xlang mode | `true` | -| `max_dyn_depth(u32)` | Maximum nesting depth for dynamic types | `5` | -| `max_type_fields(usize)` | Max fields in one received struct metadata body | `512` | -| `max_type_meta_bytes(usize)` | Max encoded bytes in one received metadata body | `4096` | -| `max_schema_versions_per_type(usize)` | Max remote metadata versions for one logical type | `10` | -| `max_average_schema_versions_per_type(usize)` | Average remote metadata versions across types | `3` | +| Option | Description | Default | +| --------------------------------------------- | ------------------------------------------------- | --------- | +| `compatible(bool)` | Enable schema evolution | `true` | +| `xlang(bool)` | Use xlang mode | `true` | +| `max_dyn_depth(u32)` | Maximum nesting depth for dynamic types | `5` | +| `max_graph_memory_bytes(usize)` | Approximate graph-memory gate per root read | `128 MiB` | +| `max_type_fields(usize)` | Max fields in one received struct metadata body | `512` | +| `max_type_meta_bytes(usize)` | Max encoded bytes in one received metadata body | `4096` | +| `max_schema_versions_per_type(usize)` | Max remote metadata versions for one logical type | `10` | +| `max_average_schema_versions_per_type(usize)` | Average remote metadata versions across types | `3` | ## Compatible Mode @@ -169,6 +193,8 @@ Security-related configuration: - Register application structs and trait-object implementations before deserializing untrusted payloads. - Use `max_dyn_depth(...)` to reject unexpectedly deep dynamic object graphs. +- Keep `max_graph_memory_bytes(...)` at the fixed `128 MiB` default for most inputs, or set a + positive byte gate for trusted workloads with different legitimate collection/map/struct sizes. - Keep the remote schema metadata limits at their defaults unless the data is not malicious and a trusted peer sends larger metadata or many schema versions. - Prefer concrete typed fields over `dyn Any` or broad trait-object fields for untrusted input. diff --git a/docs/guide/scala/configuration.md b/docs/guide/scala/configuration.md index 397657d873..260c8d5a33 100644 --- a/docs/guide/scala/configuration.md +++ b/docs/guide/scala/configuration.md @@ -140,6 +140,21 @@ val fory = ForyScala.builder().withXlang(false) .build() ``` +### Graph Memory Budget + +Scala uses the Java `withMaxGraphMemoryBytes(...)` option. It sets an approximate graph-memory gate +for one root deserialization, mainly for materialized collections, maps, arrays, structs, and +objects. It skips leaf values such as strings, binary data, primitive scalars, and dense primitive +arrays, so actual process memory can be higher than this value. Leaf values are still gated by +remaining input bytes: if the unread input does not contain enough bytes, Fory will not read or +create that leaf value. + +```scala +val fory = ForyScala.builder() + .withMaxGraphMemoryBytes(128L * 1024 * 1024) + .build() +``` + ## Xlang Mode For Scala xlang or schema IDL generated code, use the default xlang mode and @@ -179,6 +194,7 @@ and any untrusted payload source: val fory = ForyScala.builder() .requireClassRegistration(true) .withMaxDepth(50) + .withMaxGraphMemoryBytes(128L * 1024 * 1024) .withMaxTypeFields(512) .withMaxTypeMetaBytes(4096) .build() @@ -188,6 +204,9 @@ Security-related configuration: - Keep `requireClassRegistration(true)` and register application classes or generated modules. - Use `withMaxDepth(...)` to reject unexpectedly deep object graphs. +- Use `withMaxGraphMemoryBytes(...)` as an approximate gate for collection, map, array, struct, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input + bytes. - Keep `withMaxTypeFields(...)`, `withMaxTypeMetaBytes(...)`, and the remote schema-version limits at their defaults unless the data is not malicious and a trusted peer sends larger metadata or many schema versions. diff --git a/docs/guide/swift/configuration.md b/docs/guide/swift/configuration.md index e3a0478817..44c56ed822 100644 --- a/docs/guide/swift/configuration.md +++ b/docs/guide/swift/configuration.md @@ -31,6 +31,7 @@ public struct Config { public let compatible: Bool public let checkClassVersion: Bool public let maxDepth: Int + public let maxGraphMemoryBytes: Int64 public let maxTypeFields: Int public let maxTypeMetaBytes: Int public let maxSchemaVersionsPerType: Int @@ -90,8 +91,17 @@ let fory = Fory(compatible: false, checkClassVersion: true) ### Size and Depth Limits -`maxDepth` bounds decoded payload nesting depth. Compatible-mode remote metadata -is also limited: +`maxDepth` bounds decoded payload nesting depth. + +`maxGraphMemoryBytes` sets an approximate graph-memory gate for one root deserialization. The +estimate mainly covers materialized arrays, dictionaries, sets, structs, classes, and objects. It +skips leaf values such as strings, binary data, primitive scalars, and dense primitive arrays, so +actual process memory can be higher than this value. Leaf values remain protected by +byte-availability checks: if the unread input does not contain enough bytes, Fory will not read or +create that leaf value. The default limit is a fixed `128 MiB` for all root input forms. A positive +value overrides the default. Explicit non-positive values are rejected when the runtime is created. + +Compatible-mode remote metadata is also limited: - `maxTypeFields` defaults to `512` and limits fields in one received struct metadata body. - `maxTypeMetaBytes` defaults to `4096` and limits encoded body bytes in one received TypeMeta body, @@ -104,6 +114,7 @@ is also limited: ```swift let fory = Fory( maxDepth: 5, + maxGraphMemoryBytes: 128 * 1024 * 1024, maxTypeFields: 512, maxTypeMetaBytes: 4096, maxSchemaVersionsPerType: 10, @@ -140,5 +151,8 @@ Security-related configuration: - Register only the expected generated models before deserializing untrusted payloads. - Use `checkClassVersion` with `compatible: false` for intentional same-schema payloads. - Set `maxDepth` for the largest nesting depth your service accepts. +- Set `maxGraphMemoryBytes` as an approximate gate for collection, map, array, struct, class, and + object-heavy payloads. It is not an exact heap cap; leaf values are gated by remaining input + bytes. - Keep the remote schema metadata limits at their defaults unless the data is not malicious and a trusted peer sends larger metadata or many schema versions. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
