krickert commented on PR #2921: URL: https://github.com/apache/tika/pull/2921#issuecomment-4958257264
@nddipiazza @tballison - I figured I'd explain why and what I'm trying to do. I view tika is a primary input plane and think the gRPC server is where I'd want it to integrate. So a Tika `Document` shape first, including any revisions you think it needs before it becomes a contract. I would use this as my input. The OpenNLP gRPC server is deliberately following OpenNLP 3.0 as it develops. As features are merged to OpenNLP `main`, I add the corresponding capabilities to the server. My goal is for the gRPC server to be fully compliant with the OpenNLP 3.0 API when I release it and have the ability to extend it (you can see the OpenVINO and TEI extensions already and they're working prototypes). If Tika adopts this `Document` contract, I will make it a first-class input to that server. The remaining integration details include flattening the block tree, preserving block-to-text offsets, mapping annotations back to source blocks, and deciding which operations should stream. Those can be worked out after the contract is settled. I can also contribute to the fetching, parsing, and performance work here - which I'd love to do. I want to avoid defining and implementing a second document input format before we settle this boundary. If Tika later chooses a different shape, OpenNLP would have an unused input contract and both projects would carry unnecessary compatibility surface. So I'm curious: **Is this `Document` direction an acceptable shared boundary for Tika output and downstream consumers such as OpenNLP?** If not, which parts need to change before it can serve that role? I'm in no rush to get the answer - but I've never had an opportunity explicit about the vision and goal. By agreeing to a model, it can allow both projects to launch with an amazing open source story to tell. The current OpenNLP gRPC work is here: https://github.com/apache/opennlp-sandbox/tree/OPENNLP-1833-grpc-expansion Once we agree on a shape, I can make the OpenNLP server conform to it as the rest of the 3.0 features land. This is not going to happen right away - we're still going to tease more releases so I'd expect both servers to go through many iterations. What I'm trying to do on our side is to have a generic enough output shape so we can survive a lot of new features for point releases. With Tika - I want to help integrate and transform a lot of parsing surfaces and connectors (as you saw with the markdown parser). But it all starts with a document shape - then there's no rush to add on more features. Thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
