[
https://issues.apache.org/jira/browse/ARROW-17913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17613226#comment-17613226
]
Weston Pace commented on ARROW-17913:
-------------------------------------
I'm not sure {{ReadRangeCache}} is the answer here (it is already being used).
I don't think the problem is that we are issuing few vs. many {{Read}} calls
(these are pretty cheap in a memory mapped file). I think the problem is that
reading a subset of fields is causing a copy (which is a problem whether you
are using memory mapped I/O or not).
The code was roughly (very liberal pseudo-code here)...
{noformat}
Buffer body = read_everything();
std::vector<Array> arrays = allocate_arrays();
for (arr in arrays) {
for (buffer_index in expected_buffers_for_array(arr)) {
arr.buffers[buffer_index] = get_range_for_arr_buffer(arr, buffer_index);
}
}
{noformat}
In order to keep that latter part unchanged I changed the partial-fields case
to...
{noformat}
Buffer body = AllocateBigBuffer();
for (auto buffer_range in needed_ranges) {
Buffer little_body = read_partial(buffer_range);
body.set(appropriate_offset).to(little_body);
}
std::vector<Array> arrays = allocate_arrays();
for (arr in arrays) {
for (buffer_index in expected_buffers_for_array(arr)) {
arr.buffers[buffer_index] = get_range_for_arr_buffer(arr, buffer_index);
}
}
{noformat}
That triggers a memcpy (I don't recall realizing this at the time). A better
algorithm would be to bite the bullet and change the lower half to get...
{noformat}
std::vector<Buffer> buffers;
for (auto buffer_range in needed_ranges) {
Buffer little_body = read_partial(buffer_range);
buffers.push_back(little_body);
}
std::vector<Array> arrays = allocate_arrays();
for (arr in arrays) {
for (buffer_index in expected_buffers_for_array(arr)) {
arr.buffers[buffer_index] = buffers[buffer_index + some_offset];
}
}
{noformat}
> feather.read_table 150x slower when reading columns in newer versions
> ---------------------------------------------------------------------
>
> Key: ARROW-17913
> URL: https://issues.apache.org/jira/browse/ARROW-17913
> Project: Apache Arrow
> Issue Type: Bug
> Affects Versions: 7.0.0, 8.0.0, 9.0.0
> Environment: python 3.9, ubuntu 20.04
> Reporter: Håkon Magne Holmen
> Priority: Major
> Labels: feather, performance
>
> h3. Description
> Performance when reading columns using {{feather.read_table}} on Arrow
> 7.0.0-9.0.0 is drastically slower than it was in 6.0.0.
> Profiling the code below shows that the bottleneck is somewhere in the
> {{read_names}} function of {{pyarrow._feather.FeatherReader}}.
> h5. Example
> Setup code:
> {code}
> import pandas as pd
> from pyarrow import feather
> rows, cols = (1_000_000, 10)
> data = {f'c{c}': range(rows) for c in range(cols)}
> df = pd.DataFrame(data=data)
> feather.write_feather(df, 'test.feather', compression="uncompressed"){code}
> Benchmarks Arrow 9.0.0:
> {code}
> %timeit feather.read_table('test.feather', memory_map=True)
> %timeit feather.read_table('test.feather', columns=list(df.columns),
> memory_map=True)
> > 178 µs ± 1.23 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
> 33.8 ms ± 964 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
> {code}
> Benchmarks Arrow 6.0.0:
> {code}
> %timeit feather.read_table('test.feather', memory_map=True)
> %timeit feather.read_table('test.feather', columns=list(df.columns),
> memory_map=True)
> > 173 µs ± 2.12 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
> 224 µs ± 12.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
