[
https://issues.apache.org/jira/browse/TIKA-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18098232#comment-18098232
]
ASF GitHub Bot commented on TIKA-4766:
--------------------------------------
krickert commented on PR #2961:
URL: https://github.com/apache/tika/pull/2961#issuecomment-5048633623
> From a quick read, this looks great. I'm struggling with how to migrate
from what we have to a more structured approach given what I'm hoping is a
4.0.0 release fairly soon.
>
> I personally don't want to hold 4.0.0 for this, but I do think this is
important.
>
> Apologies if you've done this in previous iterations, but would it be
possible to keep grpc as is and then work on a `tika.v2` service that covers
this work and the potential follow on work. We can then release `tika.v2` when
this is finalized enough.
>
> Is that too complicated?
>
> @krickert and @nddipiazza wdyt?
The v2 idea is spot on. The strange lesson I had by transitioning from a
java kingdom to a serialized schema IDL is that the models should never break.
I used to purposefully break my APIs to force customers to write the data how
I wanted them to. Now I see the light.
Protobufs provide an "out" for most of it, and the ultimate out you get is
when you break the contract and just add a "v2" so your "v1" shapes can still
serialize. Then it'll be up to you to map it. Otherwise data structures are
always additive and you deprecate or retire old fields.
So your suggestion is exactly what the "v{n}" package name is for. We can
release a new point version of the tika engine separately from the model
version since they play well together.
And I don't mind as many iterations as we need. The API shape is something
we get stuck with, and the Tika team is the one that defines the shape after
parsing.
There are two schools of data structuring that come out of parsing: data
shape (the physical look of it like bounding boxes) and then the data surface
(NLP analysis, tokens, the chunks plus body, and governed metadata). I've found
in parser-land that it's best to separate these two concerns. Docling falls
more into the first category (layout-first, NLP second) while tika seems to
follow the second (NLP first, layout second). Both sorta flirt with each other
as far as data goes, but that tends to be where each parser aims to be.
There are ways to more accurately combine them, but don't ever try to pull
an Icarus and fly too close to a unified single model. If you try, you'll end
up just angering both camps to use it. The idea is that you share metadata or
link between the two data views.
A lot of theory there, but hope this makes sense..
BTW - if you want instant docling-looking data using diskless OCR, [this is
a grpc service I slapped
together](https://github.com/ai-pipestream/gRParse.git) that's faster than
docling (nvidia/IBM gfx compatible) but returns the same data and [this is a
grpc service](https://github.com/ai-pipestream/grPOIc) that wraps poi into
grpc.
> Replace tika-grpc fields map with a typed Document parse contract
> -----------------------------------------------------------------
>
> Key: TIKA-4766
> URL: https://issues.apache.org/jira/browse/TIKA-4766
> Project: Tika
> Issue Type: New Feature
> Components: tika-pipes
> Affects Versions: 4.0.0
> Reporter: Kristian Rickert
> Priority: Major
> Labels: grpc, pipes, protobuf
>
> Replace the flat {{FetchAndParseReply.fields}} map ({{map<string,string>}})
> with a typed parse result.
> Approach (reshaped from the original {{ParseResponse}} design after review in
> [PR #2916|https://github.com/apache/tika/pull/2916]): a single small
> {{Document}} proto (~200 lines) rather than per-format metadata messages.
> * *Content*: a structured markdown block tree -- headings, paragraphs, lists,
> tables, code blocks, inline runs (CommonMark + GFM) -- plus the rendered
> markdown string {{ToMarkdownContentHandler}} already produces (TIKA-4730).
> * *Typed common metadata*: title, authors, keywords, languages,
> created/modified as {{Timestamp}}s, page/word/character counts, dimensions,
> rights.
> * *Lossless tagged tail*: every remaining metadata key,
> multivalue-preserving, typed only where Tika's own {{Property}} declares a
> type, string otherwise -- never guessed.
> * *Embedded documents* recurse as fully typed child {{Document}}s.
> * Format specifics live in per-parser {{DocumentTransformer}} code
> ({{tika-grpc-mapper}}), never in the wire contract, so metadata churn never
> forces a client rebuild.
> Modules: {{tika-grpc-api}} (proto + generated messages + bundled
> {{FileDescriptorSet}}), {{tika-grpc-mapper}}, {{tika-grpc}} integration.
> Breaking change for clients reading {{fields}} (field number reserved).
> PR: [https://github.com/apache/tika/pull/2921] (supersedes
> [#2916|https://github.com/apache/tika/pull/2916])
> Follow-ups will be tracked in separate issues: pluggable external parsers
> (opaque {{Any}} extension results from registered gRPC services), and a
> Markdown input parser.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
