[
https://issues.apache.org/jira/browse/ARROW-7085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16971401#comment-16971401
]
Artem Alekseev edited comment on ARROW-7085 at 11/11/19 10:57 AM:
------------------------------------------------------------------
[~apitrou] We need to encode strings with custom encoder and avoid the overhead
of parsing column as a string and converting it to integers.
I think there are 2 options of desirable API:
* add an option to define custom Converter for ExtensionType (for now it's
impossible to use extension types in CSV reader)
* add an option to define custom ColumnBuilder for a specific column
I think such API can be useful, but for my problem, I found that arrow
dictionary encoding can do the job. But I'm not sure if there is a way to e.g.
check if a specified string is in the dictionary, or how to get string index.
Does Arrow provide such API?
was (Author: fexolm):
[~apitrou] We need to encode strings with custom encoder and avoid the overhead
of parsing column as a string and converting it to integers.
I think there are 2 options of desirable API:
* add an option to define custom Converter for ExtensionType (for now it's
impossible to use extension types in CSV reader)
* add an option to define custom ColumnBuilder for a specific column
> [C++][CSV] Add support for Extention type in csv reader
> -------------------------------------------------------
>
> Key: ARROW-7085
> URL: https://issues.apache.org/jira/browse/ARROW-7085
> Project: Apache Arrow
> Issue Type: Improvement
> Components: C++
> Reporter: Artem Alekseev
> Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
