dosubot[bot] commented on issue #3057: URL: https://github.com/apache/hugegraph/issues/3057#issuecomment-4657126889
<!-- Answer --> You're right that the task result mechanism isn't suitable for your use case. While `task.result_size_limit` is technically a `Long` type, it has a **hard upper limit of 1 GB** enforced by a range validator [[1]](https://github.com/apache/hugegraph/blob/9126c80e414c073b60fc074361b211cfb409e0eb/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/config/CoreOptions.java#L292-L298) — so even adjusting it won't get you to 2GB+. Additionally, task results are stored and retrieved as a single complete compressed blob with no support for chunked/paged retrieval [[2]](https://github.com/apache/hugegraph/blob/9126c80e414c073b60fc074361b211cfb409e0eb/hugegraph-server/hugegraph-core/src/main/java/org/apache/hugegraph/task/HugeTaskResult.java#L72-L89). For exporting data exceeding 2GB, the async task result approach fundamentally doesn't fit. Instead, consider these alternatives: **1. `hugegraph-tools` dump/backup commands** (recommended for bulk export): - Designed specifically for large datasets with shard-based parallel processing [[3]](https://github.com/apache/hugegraph-toolchain/blob/bc49e6683fcc8f00e32d290cd558a0320c1850a3/hugegraph-tools/src/main/java/org/apache/hugegraph/manager/BackupManager.java#L163-L175) - Supports configurable `--split-size`, `--thread-num`, and output to local or HDFS [[4]](https://github.com/apache/hugegraph-toolchain/blob/bc49e6683fcc8f00e32d290cd558a0320c1850a3/hugegraph-tools/src/main/java/org/apache/hugegraph/cmd/SubCommands.java#L245-L267) - Example: `hugegraph-tools dump --split-size 1048576 --thread-num 4 --directory ./export/` **2. Scan API with pagination** (for programmatic/custom export): - Use `/graphs/{graph}/graph/vertices/scan` with `start`, `end`, and `page` parameters [[5]](https://github.com/apache/hugegraph/blob/9126c80e414c073b60fc074361b211cfb409e0eb/hugegraph-server/hugegraph-api/src/main/java/org/apache/hugegraph/api/traversers/VerticesAPI.java#L103-L130) - Split the keyspace into shards via `/graphs/{graph}/graph/vertices/shards?split_size=...` - Iterate through shards to export incrementally without hitting memory limits **3. Gremlin pagination** (for filtered queries): - Use `.range(start, end)` in your Gremlin query to paginate results client-side These approaches let you stream/paginate results rather than materializing everything into a single task result, which is the core problem with the current approach. <!-- Dosu Comment Footer --> *To reply, just mention [@dosu](https://go.dosu.dev/dosubot?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=reply-with-mention&utm_term=apache%2Fhugegraph).* --- Share context across your team and agents. Try [Dosu](https://dosu.dev?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-tagline&utm_term=apache%2Fhugegraph). [](https://app.dosu.dev/response-feedback?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-feedback&utm_term=apache%2Fhugegraph&message_id=172eb7b7-0f41-4179-9fd9-9687ab4f50bc) [](https://github.dosu.com/apache/hugegraph?utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-learn-repo&utm_term=apache%2Fhugegraph) [](https://app.dosu.dev/signup?referrer=openSource&source=github-footer&utm_source=github&utm_medium=bot-comment&utm_campaign=github-comment-footer-20260415&utm_content=knowledge-infrastructure-add-team&utm_term=apache%2Fhugegraph) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
