[
https://issues.apache.org/jira/browse/TIKA-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880594#comment-17880594
]
Tim Allison commented on TIKA-4305:
-----------------------------------
Thank you. Y, as I mentioned above, I effectively conflated UCS2=UTF-16 and
UCS4=UTF32 and declared victory.
Are there significant differences in bytes in the UCS2 vs UTF-16 or for UCS4 vs
UTF32 in the file you submitted? The UniversalCharsetDetector linked above does
mention {{X-ISO-10646-UCS-4-3412 / X-ISO-10646-UCS-4-2143}} but it did not
detect UCS4 on the test file.
Unfortunately, as with UTF-7, the answer is basically the same: unless you can
get the upstream projects to detect these charsets or unless you can find
another detector with a friendly license, there's not much we can do.
> Tika producing empty output for UCS encoded txt files; parses UTF-7 files as
> UTF-8
> ----------------------------------------------------------------------------------
>
> Key: TIKA-4305
> URL: https://issues.apache.org/jira/browse/TIKA-4305
> Project: Tika
> Issue Type: Bug
> Components: tika-app, tika-core
> Affects Versions: 2.9.2
> Environment: Ubuntu 22.04 LTS
> Reporter: Manish S N
> Priority: Minor
> Attachments: multilingual_test_new_UCS-2.txt,
> multilingual_test_new_UCS-4.txt, multilingual_test_new_UTF-7.txt,
> multilingual_test_new_UTF-8.txt, pom.xml, tika_UTF-7_output.txt
>
>
> Tika producing empty string as output for UCS-2 and UCS-4 encoded txt files.
> No logs or errors just an empty string.
> Other formats are okay except UTF-7 files are parsed as UTF-8 which wreaks
> havoc with non ascii characters.
> how do i know that?: I opened an the UTF-7 file as UTF-8 encoded using open
> dialog of gedit and found the outputs similar
>
> I am attaching all four encoded files along with tika's output from parsing
> the UTF-7 for reference
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
