[
https://issues.apache.org/jira/browse/IGNITE-8359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Lang updated IGNITE-8359:
------------------------------
Description:
I am testing the use of Ignite's native persistence to store a data set long
term. This is on a 2.5 nightly build. To do this I am using Ignite's data
streamers to stream in 6,000,000 entries into cache A, and 12,000,000 entries
into cache B to simulate the upper limit for 2 years worth of data.
The test ran smoothly on my personal machine which has a SSD running Windows,
but ran into tremendous issues on a development test machine which is a Linux
VM using a HDD. I realize when looking at Ignite documentation that it
specifically excludes HDD's as something to base a persistent store on, but
perhaps my experience could yield improvements for SSD performance too.
The root issue is that cache updates over time become severely bottlenecked by
reading SQL index pages from disk in order to update the index. If I had to
guess this would be related to BPlusTree.findInsertionPoint() and it having to
load pages from disk if they've been evicted.
I used a 2.5 nightly build because 2.3 and 2.4 have the same issue where this
whole process was further bottlenecked by a lock behind held by Ignite while it
read the page from disk in PageMemoryImpl.acquirePage(). 2.5 fixed this.
The performance issue was much more severe in the previously mentioned cache B,
which contains user comments on entries in cache A. The key for each comment
entry is a Java class containing the creation timestamp and the string key of
the owning entry in cache A. This owning entry key is indexed so comments can
be queried by their owner. In this test case there were two comments in cache B
for every entry in cache A.
I found that even 25% of the way through streaming data into cache B, it would
take anywhere from 15 to 35 seconds to insert a batch of 2000 comments. This
slowed streaming to a crawl and ensures that streaming would need to continue
overnight to have any hope of finishing.
This also brings up concerns about data rebalancing which will have the same
performance penalty and similarly take a day at least to rebalance both caches.
I am worried about the dependency on a large amount of disk reads being done to
update the index, even though it is considerably faster with an SSD than
without. I've also not been able to test whether performance for an SSD will be
different when running in a VM, which is another worry.
was:
I am testing the use of Ignite's native persistence to store a data set long
term. This is on a 2.5 nightly build. To do this I am using Ignite's data
streamers to stream in 6,000,000 entries into cache A, and 12,000,000 entries
into cache B to simulate the upper limit for 2 years worth of data.
The test ran smoothly on my personal machine which has a SSD running Windows,
but ran into tremendous issues on a development test machine which is a Linux
VM using a HDD. I realize when looking at Ignite documentation that it
specifically excludes HDD's as something to base a persistent store on, but
perhaps my experience could yield improvements for SSD performance too.
The root issue is that cache updates over time become severely bottlenecked by
reading SQL index pages from disk in order to update the index. If I had to
guess this would be related to BPlusTree.findInsertionPoint() and it having to
load pages from disk if they've been evicted.
I used a 2.5 nightly build because 2.3 and 2.4 have the same issue where this
whole process was further bottlenecked by a lock behind held by Ignite while it
read the page from disk in PageMemoryImpl.acquirePage(). 2.5 fixed this.
The performance issue was much more severe in the previously mentioned cache B,
which contains user comments on entries in cache A. The key for each comment
entry is a Java class containing the creation timestamp and the string key of
the owning entry in cache A. This owning entry key is indexed so comments can
be queried by their owner. In this test case there were two comments in cache B
for every entry in cache A.
I found that even 25% of the way through streaming data into cache B, it would
take anywhere from 15 to 35 seconds to insert a batch of 2000 comments. This
slowed streaming to a crawl and ensures that streaming would need to continue
overnight to have any hope of finishing.
This also brings up concerns about data rebalancing which will have the same
performance penalty and similarly take a day at least to rebalance both caches.
I am worried about the dependency on a large amount of disk reads being done to
update the index, even though it is considerably faster with an SSD than
without.
> Severe performance degradation with persistence and data streaming on HDD
> -------------------------------------------------------------------------
>
> Key: IGNITE-8359
> URL: https://issues.apache.org/jira/browse/IGNITE-8359
> Project: Ignite
> Issue Type: Bug
> Components: cache, persistence, sql, streaming
> Affects Versions: 2.4, 2.5
> Environment: Linux CentOS 7 VM using Ignite DirectIO plugin with HDD.
> Reporter: Joel Lang
> Priority: Major
>
> I am testing the use of Ignite's native persistence to store a data set long
> term. This is on a 2.5 nightly build. To do this I am using Ignite's data
> streamers to stream in 6,000,000 entries into cache A, and 12,000,000 entries
> into cache B to simulate the upper limit for 2 years worth of data.
> The test ran smoothly on my personal machine which has a SSD running Windows,
> but ran into tremendous issues on a development test machine which is a Linux
> VM using a HDD. I realize when looking at Ignite documentation that it
> specifically excludes HDD's as something to base a persistent store on, but
> perhaps my experience could yield improvements for SSD performance too.
> The root issue is that cache updates over time become severely bottlenecked
> by reading SQL index pages from disk in order to update the index. If I had
> to guess this would be related to BPlusTree.findInsertionPoint() and it
> having to load pages from disk if they've been evicted.
> I used a 2.5 nightly build because 2.3 and 2.4 have the same issue where this
> whole process was further bottlenecked by a lock behind held by Ignite while
> it read the page from disk in PageMemoryImpl.acquirePage(). 2.5 fixed this.
> The performance issue was much more severe in the previously mentioned cache
> B, which contains user comments on entries in cache A. The key for each
> comment entry is a Java class containing the creation timestamp and the
> string key of the owning entry in cache A. This owning entry key is indexed
> so comments can be queried by their owner. In this test case there were two
> comments in cache B for every entry in cache A.
> I found that even 25% of the way through streaming data into cache B, it
> would take anywhere from 15 to 35 seconds to insert a batch of 2000 comments.
> This slowed streaming to a crawl and ensures that streaming would need to
> continue overnight to have any hope of finishing.
> This also brings up concerns about data rebalancing which will have the same
> performance penalty and similarly take a day at least to rebalance both
> caches.
> I am worried about the dependency on a large amount of disk reads being done
> to update the index, even though it is considerably faster with an SSD than
> without. I've also not been able to test whether performance for an SSD will
> be different when running in a VM, which is another worry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
