krickert opened a new pull request, #3013: URL: https://github.com/apache/tika/pull/3013
# TIKA-4771 PR draft **Title:** TIKA-4771: composable Document design sketch (discussion only, no code) **Base:** `apache/tika:main` ← **Head:** `ai-pipestream:TIKA-4771-composable-document` · **Draft PR** --- This is the API design for [`tika-grpc/TIKA-4771-composable-document.md`](https://github.com/ai-pipestream/tika/blob/TIKA-4771-composable-document/tika-grpc/TIKA-4771-composable-document.md). It's the API for external parser results into the typed `Document`, as a starting point for discussion on [TIKA-4771](https://issues.apache.org/jira/browse/TIKA-4771). @dpol1's did this and I provided a few rounds of feedback - I like it. It's simple, flexible, and non-invasive. So I'm opening a PR for @dpol1 since he did it on my branch. ## tldr Shows a composer (a StormCrawler bolt today, a coordinator service later) sends the same bytes to Tika `ParseBytes` and to any external parsers. Doing a claim-check pattern. Each external parser returns an `ExtensionResult` envelope: `producer_id`, status, a `google.protobuf.Any` payload with a producer-owned schema. The composer merges everything into one `Document` via `repeated ExtensionResult extensions = 40`, under fixed rules: core/extension ownership is disjoint, input bytes are hash-checked against `origin.sha256`, duplicates resolve deterministically, and every planned slot is accounted for (`OK | EMPTY | FAILED | SKIPPED`). Simple, can bring in multiple suppliers, and minimal grpc surface. ## How this relates to the ticket as filed TIKA-4771 proposes Tika brokering registered external parsers the way fetchers and emitters register today. Since it's grpc and a parser, this allows for parsers to return strongly typed results that are unknown to tika pipes. ParseStream mode, payload mappings, and a schema registry are all called out as later rounds of work. ## Dependencies - **#2961** (TIKA-4766) provides the `Document` contract this extends field 40. - [TIKA-4795](https://issues.apache.org/jira/browse/TIKA-4795) provides the exact-byte path and parity oracle the first demo would use (branch exists, no PR yet). It's depending on #2961 which is why this is a draft. Strong feelback encouraged. Thanks! <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> Thanks for your contribution to [Apache Tika](https://tika.apache.org/)! Your help is appreciated! Before opening the pull request, please verify that * there is an open issue on the [Tika issue tracker](https://issues.apache.org/jira/projects/TIKA) which describes the problem or the improvement. We cannot accept pull requests without an issue because the change wouldn't be listed in the release notes. * the issue ID (`TIKA-XXXX`) - is referenced in the title of the pull request - and placed in front of your commit messages surrounded by square brackets (`[TIKA-XXXX] Issue or pull request title`) * commits are squashed into a single one (or few commits for larger changes) * Tika is successfully built and unit tests pass by running `./mvnw clean test` * there should be no conflicts when merging the pull request branch into the *recent* `main` branch. If there are conflicts, please try to rebase the pull request branch on top of a freshly pulled `main` branch * if you add new module that downstream users will depend upon add it to relevant group in `tika-bom/pom.xml`. We will be able to faster integrate your pull request if these conditions are met. If you have any questions how to fix your problem or about using Tika in general, please sign up for the [Tika mailing list](http://tika.apache.org/mail-lists.html). Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
