[
https://issues.apache.org/jira/browse/TIKA-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866059#comment-17866059
]
Tilman Hausherr commented on TIKA-4278:
---------------------------------------
Many files are detected as csv that are not, e.g. govdocs1/040/040251.txt
govdocs1/242/242970.txt, govdocs1/001/001605.txt: now has a ":" as separator
although it's obvious that it's a ",". Maybe because of TIME_HH:MM:SS?!
govdocs1/346/346152.txt is considered to be pipe-separated, despite that it's a
text file, although it's a table. IMHO it shouldn't "detect" something that
isn't in the first line. This would also solve the problem with
govdocs1/040/040251.txt .
govdocs1/113/113291.txt: claims that it contains "컴컴" but it doesn't. I assume
this is a different change than mine because my changes aren't related to the
encoding.
I'll rerun the tests with a change that returns 0 confidence in CSVSniffer when
the delimiter is not in row zero.
> TextAndCSVParser doesn't detect semicolon separated file
> --------------------------------------------------------
>
> Key: TIKA-4278
> URL: https://issues.apache.org/jira/browse/TIKA-4278
> Project: Tika
> Issue Type: Bug
> Components: parser
> Affects Versions: 2.9.2
> Reporter: Tilman Hausherr
> Assignee: Tilman Hausherr
> Priority: Major
> Labels: csv, csvparser
> Fix For: 3.0.0, 2.9.3
>
> Attachments: reports_csv_2.9.2_vs_2.9.3.tar.xz
>
>
> I ran the code from the attached SO issue and yes it doesn't detect semicolon
> separated files. The reason is this line in {{TextAndCSVParser.java}}:
> {code:java}
> private static final char[] DEFAULT_DELIMITERS = new char[]\{',', '\t'};
> {code}
> This is later used by {{CSVSniffer}}. For some reason the other delimiters
> (pipe, colon and semicolon) aren't in that array, although they are in
> {{CHAR_TO_STRING_DELIMITER_MAP}}. I modified {{DEFAULT_DELIMITERS}} and now
> it works for semicolon.
> Can I change this by adding the missing delimiters or was there a reason that
> I missed? Proposed change would change CSVSniffer so that delimiters is a set
> and then pass {{CHAR_TO_STRING_DELIMITER_MAP.keySet()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
