Davide Polato created TIKA-4815:
-----------------------------------
Summary: tika-grpc: all parse requests share one single-threaded
PipesClient
Key: TIKA-4815
URL: https://issues.apache.org/jira/browse/TIKA-4815
Project: Tika
Issue Type: Bug
Components: tika-pipes
Affects Versions: 4.0.0
Reporter: Davide Polato
Attachments: ConcurrencyLocalDiagnostic.java
The gRPC server creates exactly one PipesClient and every parse handler thread
uses it. PipesClient's own javadoc says it is single-threaded, and nothing in
TikaGrpcServerImpl synchronizes access to it. The management RPCs are not
involved - this is about fetchAndParse and its streaming variants, which all
funnel through the same client.
Attached is a small repro: one sequential warmup call so the forked worker is
up and serving, then 4 concurrent fetchAndParse calls at an in-process server
built without directExecutor (the production server uses a thread pool too).
Measured on current master:
DIAG-CONC WARMUP status=PARSE_SUCCESS
DIAG-CONC OK ... status=UNSPECIFIED_CRASH
DIAG-CONC OK ... status=FAILED_TO_INITIALIZE
DIAG-CONC OK ... status=UNSPECIFIED_CRASH
DIAG-CONC OK ... status=UNSPECIFIED_CRASH
SUMMARY concurrency=4 answered=4 transportFailed=0 elapsedMs=61358
Every call got a normal gRPC OK answer - transport-level success masking
application-level failure. The statuses say the worker died or never came up
for the request, and four tiny HTML files took a minute. A crawler pointed at
this records every page as unparseable while the server looks perfectly healthy.
Why no test caught it: TikaGrpcServerTest sends its requests sequentially and
builds servers with directExecutor(), so nothing ever overlaps.
Effectively today's server has an undeclared limit of one in-flight parse. This
is also what blocks cancellation (see TIKA-4795): you can't interrupt one
request's parse without disturbing the worker everyone else is on.
Measured on current master (4.0.0-SNAPSHOT). The 3.x server has the same shape
but I haven't run the probe there.
tika-pipes already has the right mechanism: PipesParser keeps a queue of
clients sized by numClients, admission with a timeout, release in finally.
tika-grpc bypasses it and builds a bare client. Moving the server onto a pool
looks like most of the fix, with two open questions: what should pool
exhaustion map to on the wire (I'd lean gRPC RESOURCE_EXHAUSTED -
infrastructure outcomes as status, parse outcomes in the reply - but that's
this project's call), and what marks a client as unreusable before pooling -
PipesClient.process() rethrows InterruptedException without closing the
connection, and PipesParser's finally re-offers the client unconditionally.
I can work on this - the repro converts to a regression test directly. Drop the
attachment into tika-grpc/src/test/java/org/apache/tika/pipes/grpc/ and run
with -Dtest=ConcurrencyLocalDiagnostic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
