In our 1.27 branch this new pseudo test prints out the output below
it. I'm looking into why this content is slightly longer than the
write limit...
@Test
public void testJsonWriteLimitEmbedded() throws Exception {
for (int i = 500; i < 10000; i += 500) {
Response response = WebClient.create(endPoint + META_PATH +
"/text").accept("application/json")
.header("writeLimit",
Integer.toString(i)).put(ClassLoader.getSystemResourceAsStream(TEST_RECURSIVE_DOC));
List<Metadata> metadata = JsonMetadataList.fromJson(
new InputStreamReader(((InputStream)
response.getEntity()), StandardCharsets.UTF_8));
int len = 0;
int j = 0;
for (Metadata m : metadata) {
j++;
len +=
m.get(AbstractRecursiveParserWrapperHandler.TIKA_CONTENT).length();
}
System.out.println("write limit: "+i + " : actual len: "+ len);
}
}
write limit: 500 : actual len: 998
write limit: 1000 : actual len: 1498
write limit: 1500 : actual len: 1998
write limit: 2000 : actual len: 2498
write limit: 2500 : actual len: 2998
write limit: 3000 : actual len: 3498
write limit: 3500 : actual len: 3998
write limit: 4000 : actual len: 4498
write limit: 4500 : actual len: 4998
write limit: 5000 : actual len: 5498
write limit: 5500 : actual len: 5998
write limit: 6000 : actual len: 6498
write limit: 6500 : actual len: 6998
write limit: 7000 : actual len: 7498
write limit: 7500 : actual len: 7998
write limit: 8000 : actual len: 8498
write limit: 8500 : actual len: 8672
write limit: 9000 : actual len: 8672
write limit: 9500 : actual len: 8672
On Mon, Apr 26, 2021 at 2:06 PM Tim Allison <[email protected]> wrote:
>
> Yikes and sorry... Your understanding is what I intended to happen and added
> a unit test to confirm. Tika should return the partial contents up to the
> write limit. Can you give me more details about how you're calling tika?
> /rmeta endpoint with text or xhtml or body? What is the writelimit you're
> choosing? Can you reproduce on a unit test file in our repo? Thank you for
> alerting us to this!
>
> Best,
>
> Tim
>
>
> On Mon, Apr 26, 2021 at 12:39 PM <[email protected]> wrote:
>>
>> Hi Tim,
>>
>>
>>
>> I tried the fix you did on Tika 1.26 for the issue TIKA-3325 I reported. As
>> a reminder it was concerning the limit of extracted content and the goal was
>> to limit across all content in a container file. It works well but not as I
>> expected , indeed if the limit is reached, Tika just returns no content at
>> all.
>>
>>
>>
>> On my side, I was thinking it would return content up to the limit and that
>> the content exceeding the limit would be truncated to fits the limit. Would
>> it be possible to have another header option implementing such behavior or
>> is it too complicated ?
>>
>>
>>
>> Regards,
>>
>> Julien
>>
>>
>>
