[
https://issues.apache.org/jira/browse/TIKA-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17924076#comment-17924076
]
Subbu edited comment on TIKA-4370 at 2/5/25 1:55 PM:
-----------------------------------------------------
Appreciate your responses here.
_Perhaps, if the file is application/octet-stream, run the charset detector on
the bytes and if the charset is detected as Shift-JIS then return text/pain._
1) If you don't prefer such logic in tika code base, are we open here to make a
change such that application passes character encoding in metadata (using pre
ran TXTParser) in their classpath and TextDetector, if it detects
application/octet-stream and metadata says Shift_JIS, it can return fallback as
text/plain?
Can you help me understand what problems may arise due to this?
2) Even if i do that (either in tika-core / or in my app logic) I think the
best I can do is without file name, if returned type is
application/octet-stream, I can char encoding detection and confirm it's
text/plain. But for other textual files, won't the problem exist? Like csv
files always with Shift-JIS. Or do you think problem is specific to text/plain
and all other SJIS files should work (without file name)?
[~tallison]
was (Author: JIRAUSER307746):
Appreciate your responses here.
_Perhaps, if the file is application/octet-stream, run the charset detector on
the bytes and if the charset is detected as Shift-JIS then return text/pain._
1) Do you see that I can add this logic in TextDetector and create a PR? But I
believe I won't have tika-parsers dependency in tika-core. Or are you
suggesting I do this in my application layer?
If you prefer the latter, are we open here to make a change such that
application passes character encoding in metadata (using pre ran TXTParser) in
their classpath and TextDetector, if it detects application/octet-stream and
metadata says Shift_JIS, it can return fallback as text/plain?
2) Even if i do that (either in tika-core / or in my app logic) I think the
best I can do is without file name, if returned type is
application/octet-stream, I can char encoding detection and confirm it's
text/plain. But for other textual files, won't the problem exist? Like csv
files always with Shift-JIS. Or do you think problem is specific to text/plain
and all other SJIS files should work (without file name)?
[~tallison]
> SJIS Encoded Files Can't be Detected
> ------------------------------------
>
> Key: TIKA-4370
> URL: https://issues.apache.org/jira/browse/TIKA-4370
> Project: Tika
> Issue Type: Bug
> Reporter: Subbu
> Priority: Major
>
> When character encoding of file is SJIS, without file name in the metadata,
> most files content-type detected as application/octet-stream. Is there zero
> support for SJIS?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
