[
https://issues.apache.org/jira/browse/ARROW-17913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17614191#comment-17614191
]
Weston Pace commented on ARROW-17913:
-------------------------------------
I think we could add a ReadRanges method to the filesystem and then move the
read range cache into the filesystem. It would probably simplify the readers a
little bit. However, the burden is still on the readers to figure out all the
ranges they need first and then issue the reads second. This is probably the
hardest part of using a read range cache.
Alternatively, we could do something at the filesystem layer like "if a read is
in progress" (or enough reads that the filesystem is saturated) then add the
request to a queue and coalesce within the queue. This is very similar to
Linux's ["plugging"|https://lwn.net/Articles/438256/].
> feather.read_table 150x slower when reading columns in newer versions
> ---------------------------------------------------------------------
>
> Key: ARROW-17913
> URL: https://issues.apache.org/jira/browse/ARROW-17913
> Project: Apache Arrow
> Issue Type: Bug
> Affects Versions: 7.0.0, 8.0.0, 9.0.0
> Environment: python 3.9, ubuntu 20.04
> Reporter: Håkon Magne Holmen
> Priority: Major
> Labels: feather, performance
>
> h3. Description
> Performance when reading columns using {{feather.read_table}} on Arrow
> 7.0.0-9.0.0 is drastically slower than it was in 6.0.0.
> Profiling the code below shows that the bottleneck is somewhere in the
> {{read_names}} function of {{pyarrow._feather.FeatherReader}}.
> h5. Example
> Setup code:
> {code}
> import pandas as pd
> from pyarrow import feather
> rows, cols = (1_000_000, 10)
> data = {f'c{c}': range(rows) for c in range(cols)}
> df = pd.DataFrame(data=data)
> feather.write_feather(df, 'test.feather', compression="uncompressed"){code}
> Benchmarks Arrow 9.0.0:
> {code}
> %timeit feather.read_table('test.feather', memory_map=True)
> %timeit feather.read_table('test.feather', columns=list(df.columns),
> memory_map=True)
> > 178 µs ± 1.23 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
> 33.8 ms ± 964 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
> {code}
> Benchmarks Arrow 6.0.0:
> {code}
> %timeit feather.read_table('test.feather', memory_map=True)
> %timeit feather.read_table('test.feather', columns=list(df.columns),
> memory_map=True)
> > 173 µs ± 2.12 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
> 224 µs ± 12.1 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
