imbajin commented on issue #3057:
URL: https://github.com/apache/hugegraph/issues/3057#issuecomment-4668834701
For HugeGraph 1.5.0, there is no built-in API to paginate the result payload
of a single task. The task API supports paging when listing task metadata, but
`GET /tasks/{id}` still loads and returns the task result as one complete value.
In your case, the result is produced by a complex Gremlin algorithm and
finally becomes something like `Map<String, String>`. The existing vertex/edge
`page` or shard-scan APIs cannot directly paginate that final computed map,
because those APIs are storage-level cursors, while your result is an
algorithm-level result set.
I don't recommend disabling `task.result_size_limit` or increasing the task
DB executor threads as the main solution:
- Disabling the size check only removes one guard. The server still needs to
save, load, decompress, serialize, and return a huge single payload through
`/tasks/{id}`.
- A 2GB+ single task result can still block TaskStore, consume a large
amount of memory, and break frontend rendering.
- The error `Can't operate a tx in other threads` is expected if the task
transaction is used from different DB worker threads. The task transaction is
thread-bound, so increasing task DB threads is not a safe workaround.
If you want to modify the source code yourself, the simplest practical
direction is **not** to add pagination over the existing task result string.
Instead, add a small chunked-output helper for Gremlin jobs, then make the task
return only a small manifest.
A minimal design could be:
```text
Gremlin algorithm
-> compute partial entries
-> write result chunks incrementally
-> update task progress
-> return a small manifest from the task result
```
For example, add methods to `GremlinJob.GremlinJobProxy` such as:
```java
gremlinJob.writeResultChunk(String name, String content)
gremlinJob.appendResultLine(String name, String line)
gremlinJob.resultManifest()
```
Then the Gremlin script can write output while it is computing, instead of
returning one huge `Map<String, String>`:
```groovy
def chunk = 0
def count = 0
def buffer = new StringBuilder()
resultMap.each { key, value ->
buffer.append(toJsonLine(key, value)).append('\n')
count++
if (count % 10000 == 0) {
gremlinJob.writeResultChunk("chunk-${chunk}.jsonl",
buffer.toString())
buffer.setLength(0)
chunk++
gremlinJob.updateProgress(...)
}
}
if (buffer.length() > 0) {
gremlinJob.writeResultChunk("chunk-${chunk}.jsonl", buffer.toString())
}
return gremlinJob.resultManifest()
```
The final task result should be small, for example:
```json
{
"type": "chunked-result",
"format": "jsonl",
"chunk_count": 245,
"total_count": 12345678,
"result_uri": "/path/or/url/to/result/manifest.json"
}
```
This keeps `/tasks/{id}` lightweight and makes repeated result reads
possible by reading chunk files or a dedicated result-chunk API.
For a quick single-node RocksDB deployment, writing chunks to a configured
local directory is probably the smallest change. For production or distributed
deployment, the same idea should use object storage, HDFS, or a dedicated
result-chunk table/API, with cleanup and path/security checks.
So the recommended source-level change is:
1. Keep task result small.
2. Add a Gremlin-accessible chunk writer.
3. Store large output outside the task result.
4. Return only manifest/progress from the task.
This is much safer than making `/tasks/{id}` return a 2GB+ payload or trying
to paginate a giant string after it has already been materialized.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
