[
https://issues.apache.org/jira/browse/TIKA-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18095885#comment-18095885
]
ASF GitHub Bot commented on TIKA-4766:
--------------------------------------
krickert commented on PR #2921:
URL: https://github.com/apache/tika/pull/2921#issuecomment-4958257264
@nddipiazza @tballison -
I figured I'd explain why and what I'm trying to do. I view tika is a
primary input plane and think the gRPC server is where I'd want it to
integrate.
So a Tika `Document` shape first, including any revisions you think it needs
before it becomes a contract. I would use this as my input.
The OpenNLP gRPC server is deliberately following OpenNLP 3.0 as it
develops. As features are merged to OpenNLP `main`, I add the corresponding
capabilities to the server. My goal is for the gRPC server to be fully
compliant with the OpenNLP 3.0 API when I release it and have the ability to
extend it (you can see the OpenVINO and TEI extensions already and they're
working prototypes).
If Tika adopts this `Document` contract, I will make it a first-class input
to that server. The remaining integration details include flattening the block
tree, preserving block-to-text offsets, mapping annotations back to source
blocks, and deciding which operations should stream. Those can be worked out
after the contract is settled. I can also contribute to the fetching, parsing,
and performance work here - which I'd love to do.
I want to avoid defining and implementing a second document input format
before we settle this boundary. If Tika later chooses a different shape,
OpenNLP would have an unused input contract and both projects would carry
unnecessary compatibility surface.
So I'm curious: **Is this `Document` direction an acceptable shared boundary
for Tika output and downstream consumers such as OpenNLP?**
If not, which parts need to change before it can serve that role?
I'm in no rush to get the answer - but I've never had an opportunity
explicit about the vision and goal. By agreeing to a model, it can allow both
projects to launch with an amazing open source story to tell.
The current OpenNLP gRPC work is here:
https://github.com/apache/opennlp-sandbox/tree/OPENNLP-1833-grpc-expansion
Once we agree on a shape, I can make the OpenNLP server conform to it as the
rest of the 3.0 features land.
This is not going to happen right away - we're still going to tease more
releases so I'd expect both servers to go through many iterations. What I'm
trying to do on our side is to have a generic enough output shape so we can
survive a lot of new features for point releases. With Tika - I want to help
integrate and transform a lot of parsing surfaces and connectors (as you saw
with the markdown parser).
But it all starts with a document shape - then there's no rush to add on
more features.
Thoughts?
> Replace tika-grpc fields map with a typed Document parse contract
> -----------------------------------------------------------------
>
> Key: TIKA-4766
> URL: https://issues.apache.org/jira/browse/TIKA-4766
> Project: Tika
> Issue Type: New Feature
> Components: tika-pipes
> Affects Versions: 4.0.0
> Reporter: Kristian Rickert
> Priority: Major
> Labels: grpc, pipes, protobuf
>
> Replace the flat {{FetchAndParseReply.fields}} map ({{map<string,string>}})
> with a typed parse result.
> Approach (reshaped from the original {{ParseResponse}} design after review in
> [PR #2916|https://github.com/apache/tika/pull/2916]): a single small
> {{Document}} proto (~200 lines) rather than per-format metadata messages.
> * *Content*: a structured markdown block tree -- headings, paragraphs, lists,
> tables, code blocks, inline runs (CommonMark + GFM) -- plus the rendered
> markdown string {{ToMarkdownContentHandler}} already produces (TIKA-4730).
> * *Typed common metadata*: title, authors, keywords, languages,
> created/modified as {{Timestamp}}s, page/word/character counts, dimensions,
> rights.
> * *Lossless tagged tail*: every remaining metadata key,
> multivalue-preserving, typed only where Tika's own {{Property}} declares a
> type, string otherwise -- never guessed.
> * *Embedded documents* recurse as fully typed child {{Document}}s.
> * Format specifics live in per-parser {{DocumentTransformer}} code
> ({{tika-grpc-mapper}}), never in the wire contract, so metadata churn never
> forces a client rebuild.
> Modules: {{tika-grpc-api}} (proto + generated messages + bundled
> {{FileDescriptorSet}}), {{tika-grpc-mapper}}, {{tika-grpc}} integration.
> Breaking change for clients reading {{fields}} (field number reserved).
> PR: [https://github.com/apache/tika/pull/2921] (supersedes
> [#2916|https://github.com/apache/tika/pull/2916])
> Follow-ups will be tracked in separate issues: pluggable external parsers
> (opaque {{Any}} extension results from registered gRPC services), and a
> Markdown input parser.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
