contrueCT commented on issue #3059:
URL: https://github.com/apache/hugegraph/issues/3059#issuecomment-4721709741
Since this is on HugeGraph 1.5.0 and `force=true` still does not work, I
think this is closely related to #3057, but there are two separate things to
handle:
1. Recover the already stuck task.
2. Prevent future Gremlin tasks from writing huge results into the task
result field again.
For the already stuck task, the normal task delete path may still be too
heavy. In 1.5.0 local scheduler, a large result can be stored on the `~task`
vertex as `~result`, and the normal query/delete path may still construct a
`HugeTask` through `HugeTask.fromVertex()`. If the `~result` payload is very
large, even `DELETE /tasks/{id}?force=true` can hit the same memory pressure or
be interrupted.
So for recovery, I think a temporary maintenance cleanup path is needed. A
practical recovery flow could be:
```text
start HugeGraph in a temporary maintenance/recovery mode
-> skip automatic task restoration on startup
-> delete the problematic task metadata/result by task id
without calling HugeTask.fromVertex()
-> restart HugeGraph normally
```
The cleanup code should remove the in-memory task entry first, then query
the task vertex by id and delete the vertex directly. It should not deserialize
the task, decompress the `~result` payload, or return the full task result. For
local scheduler, deleting the `~task` vertex should also remove its `~result`
property. If separated task-result storage has been used, the corresponding
`~taskresult` record should be cleaned as well.
This should be done in a maintenance window and tested on a backup first.
Deleting the whole RocksDB graph directory should only be the last resort
because it removes graph data, not just the bad task.
For future large Gremlin results, the workaround from #3057 is the better
direction: don't return a huge `Map<String, String>` or huge JSON string as the
task result. Add a Gremlin-accessible chunk writer, write results incrementally
during computation, and make the task return only a small manifest.
For example, add helper methods such as:
```java
gremlinJob.writeResultChunk(String name, String content)
gremlinJob.appendResultLine(String name, String line)
gremlinJob.resultManifest()
```
Then the Gremlin script can flush partial output while it is computing,
instead of building one huge result in memory and returning it through
`/tasks/{id}`. The final task result should stay small, for example:
```json
{
"type": "chunked-result",
"format": "jsonl",
"chunk_count": 245,
"total_count": 12345678,
"result_uri": "..."
}
```
One important detail: chunking should happen during the computation. If the
script still builds the whole `Map<String, String>` first and only writes
chunks at the end, it may still cause the same memory pressure.
Separately, the memory-not-reclaimed part should be diagnosed by comparing
JVM heap usage with process RSS. If heap is not high but RSS is high, RocksDB
native memory, OS page cache, allocator fragmentation, and RocksDB buffer/cache
settings are more relevant. If heap is high, task results, caches, or
restored/running tasks are more likely.
Also, 1.5.0 is not the latest release. Upgrading to the latest HugeGraph
release is recommended as a better baseline for existing fixes, but this
large-single-task-result pattern should still be changed, because `/tasks/{id}`
is not suitable as a bulk result download API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
