pithecuse527 opened a new issue, #13836:
URL: https://github.com/apache/gravitino/issues/13836
### What would you like to be improved?
Java GVFS client asks the server for the real path of every file it touches.
Each open, getFileStatus, create, ... becomes one GET
.../filesets/{fs}/location?sub_path=....
```
Spark executors (hundreds) Gravitino server
open(dt=1/a.parquet) ── getFileLocation ──▶ ┐
getFileStatus(dt=1/a.parquet) ── getFileLocation ──▶ │ one request
open(dt=1/b.parquet) ── getFileLocation ──▶ │ per file op
... ┘
```
During a TPC-DS SF1000 `catalog_sales` scan with over a thousand partitions,
individual sub-paths were requested multiple times, resulting in total REST
calls scaling roughly with partitions multiplied by files per partition and
calls per file.
The metadata cache
([#7572](https://github.com/apache/gravitino/issues/7572)) does not help,
because every file has a different sub path.
I've checked the design document from issue
[[4021](https://github.com/apache/gravitino/issues/4021#issuecomment-2217361842)](https://github.com/apache/gravitino/issues/4021#issuecomment-2217361842)
and it shows that path resolution originally done on the GVFS client, but it
was moved to the server side to support auditing and avoiding duplicated path
logic in every client language.
However... because path resolution happens on the server for every single
file operation, i've observed that Spark jobs slow down significantly as these
requests pile up and it could also impact other Spark jobs using gravitino
fileset.
### How should we improve?
### 1. Resolve locally in GVFS (rollback).
- GVFS already has the storage locations from `loadFileset()`
- `getFileLocation` has the same authorization check, so nothing is skipped
- Since the [Rust FUSE client is
deprecated](https://github.com/apache/gravitino/issues/12592), only Java and
Python need to be considered
### 2. Keep the audit by reporting resolved locations in batches (tentative)
- some rough idea...
```
GVFS (executor) Gravitino server
open / create → resolve locally → storage
│ record (subPath, locationName, op)
▼
in-memory queue ── every 1s, and on close() ──▶ POST
.../filesets/{fs}/location-reports
└─ replays each entry
via getFileLocation
→ same
GetFileLocationEvent
```
- with new config (e.g. fs.gravitino.fileLocation.localResolve.enable=false
by default), so nothing changes unless an operator turns it on
### Trade-offs
- Reporting is best effort. Entries can be lost on a crash or a long outage;
bounded retries cover network errors and 5xx
- we could introduce retry logic?
- It is not fail-closed like the old call
- The queue needs a few config values... (firing seconds, queue size...)
let me know your thoughts
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
