Hi Vinokurov,
Thanks for your reply.
I try to write batches by 100 entries.
And I got a worse result.
The writing speed is down to 12.09 KB per second.
Below is my code which I try to use putAll and writeAll to rewrite.
Did I make some mistakes?
Main function:
Ignite ignite = Ignition.start("PATH/example-cache.xml");
IgniteCache<String, String> igniteCache =
ignite.getOrCreateCache("testCache ");
for(int i = 0; i < 100; i++)
{
parameterMap.put(Integer.toString(i), "writeAll_val");
}
while(true)
{
igniteCache.putAll(parameterMap);
}
Write all to PostgreSQL through JDBC:
@Override
public void writeAll(Collection<Cache.Entry<? extends String, ? extends
String>> entries) throws CacheWriterException {
Iterator<Cache.Entry<? extends String, ? extends String>> it =
entries.iterator();
Map<String, Object> parameterMap = new HashMap<>();
int count = 1;
while (it.hasNext()) {
Cache.Entry<? extends String, ? extends String> entry = it.next();
String valCount = "val";
valCount += Integer.toString(count);
parameterMap.put(valCount, entry.getValue());
count++;
it.remove();
}
String sqlString = "INSERT INTO test_writeall(val) VALUES "
+
"(:val1),(:val2),(:val3),(:val4),(:val5),(:val6),(:val7),(:val8),(:val9),(:val10),"
+
"(:val11),(:val12),(:val13),(:val14),(:val15),(:val16),(:val17),(:val18),(:val19),(:val20),"
+
"(:val21),(:val22),(:val23),(:val24),(:val25),(:val26),(:val27),(:val28),(:val29),(:val30),"
+
"(:val31),(:val32),(:val33),(:val34),(:val35),(:val36),(:val37),(:val38),(:val39),(:val40),"
+
"(:val41),(:val42),(:val43),(:val44),(:val45),(:val46),(:val47),(:val48),(:val49),(:val50),"
+
"(:val51),(:val52),(:val53),(:val54),(:val55),(:val56),(:val57),(:val58),(:val59),(:val60),"
+
"(:val61),(:val62),(:val63),(:val64),(:val65),(:val66),(:val67),(:val68),(:val69),(:val70),"
+
"(:val71),(:val72),(:val73),(:val74),(:val75),(:val76),(:val77),(:val78),(:val79),(:val80),"
+
"(:val81),(:val82),(:val83),(:val84),(:val85),(:val86),(:val87),(:val88),(:val89),(:val90),"
+
"(:val91),(:val92),(:val93),(:val94),(:val95),(:val96),(:val97),(:val98),(:val99),(:val100);";
jdbcTemplate.update(sqlString, parameterMap);
}
From: Vinokurov Pavel [mailto:[email protected]]
Sent: Wednesday, March 14, 2018 5:42 PM
To: [email protected]
Subject: Re: Performance of Ignite integrating with PostgreSQL
Hi,
You could try to use igniteCache.putAll for write batches by 1000 entries.
Use following script in PostgresDBStore#writeAll method to put data into the
database:
String sqlString = "INSERT INTO test(val) VALUES (:val1)(:val2)(:val3);";
2018-03-14 11:58 GMT+03:00
<[email protected]<mailto:[email protected]>>:
Hi,
I try to use Ignite to integrate with PostgreSQL.
And I use “atop” to monitor the data write to PostgreSQL.
Then observed that the writing speed is 1 MB per second.
This performance is not really good. Below is my configuration and code. Please
help me to improve it.
Thanks.
There is my cache configuration:
<bean
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value= "testCache"/>
<property name="cacheMode"
value="PARTITIONED"/>
<property name="atomicityMode" value="
ATOMIC"/>
<property name="atomicWriteOrderMode"
value="PRIMARY"/>
<property name="readThrough" value="true"/>
<property name="writeThrough"
value="true"/>
<property name="writeBehindEnabled" value="true"/>
<property name="writeBehindFlushThreadCount"
value="64"/>
<property name="writeBehindBatchSize"
value="131072" />
<property name="writeBehindFlushSize"
value="131072" />
<property name="offHeapMaxMemory" value="0" />
<property name="cacheStoreFactory">
<bean
class="javax.cache.configuration.FactoryBuilder$SingletonFactory">
<constructor-arg>
<bean
class="com.blu.imdg.jdbc.PostgresDBStore">
</bean>
</constructor-arg>
</bean>
</property>
<property name="backups" value="0"/>
<property name="indexedTypes">
<list>
<value>java.lang.String</value>
<value>java.lang.String</value>
</list>
</property>
</bean>
Main function:
Ignite ignite = Ignition.start("PATH/example-cache.xml");
IgniteCache<String, String> igniteCache =
ignite.getOrCreateCache("testCache ");
int seqint = 0;
while(true)
{
igniteCache.put(Integer.toString(seqint),
"valueString");
seqint++;
}
Write behind to PostgreSQL through JDBC:
@Override
public void write(Cache.Entry<? extends String, ? extends String> entry) throws
CacheWriterException {
Map<String, Object> parameterMap = new HashMap<>();
parameterMap.put(“val”, entry.getValue());
String sqlString = "INSERT INTO test(val) VALUES (:val);";
jdbcTemplate.update(sqlString, parameterMap);
}
--
本信件可能包含工研院機密資訊，非指定之收件者，請勿使用或揭露本信件內容，並請銷毀此信件。 This email may contain
confidential information. Please do not use or disclose it in any way and
delete it if you are not the intended recipient.
--
Regards
Pavel Vinokurov
--
本信件可能包含工研院機密資訊，非指定之收件者，請勿使用或揭露本信件內容，並請銷毀此信件。 This email may contain
confidential information. Please do not use or disclose it in any way and
delete it if you are not the intended recipient.
