Hi Vinokurov,
I changed my code
>> IgniteCache<String, String> igniteCache = ignite.getOrCreateCache("testCache
>> ");
To
IgniteCache<String, String> igniteCache = ignite.cache("testCache");
And update to 2.4.0 version.
But the writing speed is still about 100 KB per second.
Below is jdbc connection initialization:
@Autowired
public NamedParameterJdbcTemplate jdbcTemplate;
@Override
public void start() throws IgniteException {
ConfigurableApplicationContext context = new
ClassPathXmlApplicationContext("postgres-context.xml");
this.jdbcTemplate = context.getBean(NamedParameterJdbcTemplate.class);
}
The PostgreSQL configuration, “postgres-context.xml” :
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xmlns:context="http://www.springframework.org/schema/context";
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
http://www.springframework.org/schema/context
http://www.springframework.org/schema/context/spring-context.xsd";>
<context:component-scan base-package="com.blu.imdg.jdbc"/>
<context:property-placeholder location="classpath:jdbc.properties"/>
<bean id="dataSource"
class="org.springframework.jdbc.datasource.DriverManagerDataSource">
<property name="driverClassName" value="${jdbc.driver}"/>
<property name="url" value="${jdbc.url}"/>
<property name="username" value="${jdbc.username}"/>
<property name="password" value="${jdbc.password}"/>
</bean>
<bean id="jdbcTemplate"
class="org.springframework.jdbc.core.namedparam.NamedParameterJdbcTemplate">
<constructor-arg ref="dataSource"/>
</bean>
</beans>
Thanks.
From: Vinokurov Pavel [mailto:[email protected]]
Sent: Thursday, March 22, 2018 1:50 PM
To: [email protected]
Subject: Re: Performance of Ignite integrating with PostgreSQL
Also it makes sense to use new 2.4 version.
2018-03-22 8:37 GMT+03:00 Vinokurov Pavel
<[email protected]<mailto:[email protected]>>:
>> IgniteCache<String, String> igniteCache = ignite.getOrCreateCache("testCache
>> ");
please, change to ignite.cache("testCache") to be sure the we use
configuration from the file.
2018-03-22 8:19 GMT+03:00 Vinokurov Pavel
<[email protected]<mailto:[email protected]>>:
You already showed the cache configuration, but could you show jdbc connection
initialization
2018-03-22 7:59 GMT+03:00 Vinokurov Pavel
<[email protected]<mailto:[email protected]>>:
Hi,
Could you please show the "PATH/example-cache.xml" file.
2018-03-21 9:40 GMT+03:00
<[email protected]<mailto:[email protected]>>:
Hi Vinokurov,
Thanks for your reply.
I try to write batches by 100 entries.
And I got a worse result.
The writing speed is down to 12.09 KB per second.
Below is my code which I try to use putAll and writeAll to rewrite.
Did I make some mistakes?
Main function:
Ignite ignite = Ignition.start("PATH/example-cache.xml");
IgniteCache<String, String> igniteCache =
ignite.getOrCreateCache("testCache ");
for(int i = 0; i < 100; i++)
{
parameterMap.put(Integer.toString(i), "writeAll_val");
}
while(true)
{
igniteCache.putAll(parameterMap);
}
Write all to PostgreSQL through JDBC:
@Override
public void writeAll(Collection<Cache.Entry<? extends String, ? extends
String>> entries) throws CacheWriterException {
Iterator<Cache.Entry<? extends String, ? extends String>> it =
entries.iterator();
Map<String, Object> parameterMap = new HashMap<>();
int count = 1;
while (it.hasNext()) {
Cache.Entry<? extends String, ? extends String> entry = it.next();
String valCount = "val";
valCount += Integer.toString(count);
parameterMap.put(valCount, entry.getValue());
count++;
it.remove();
}
String sqlString = "INSERT INTO test_writeall(val) VALUES "
+
"(:val1),(:val2),(:val3),(:val4),(:val5),(:val6),(:val7),(:val8),(:val9),(:val10),"
+
"(:val11),(:val12),(:val13),(:val14),(:val15),(:val16),(:val17),(:val18),(:val19),(:val20),"
+
"(:val21),(:val22),(:val23),(:val24),(:val25),(:val26),(:val27),(:val28),(:val29),(:val30),"
+
"(:val31),(:val32),(:val33),(:val34),(:val35),(:val36),(:val37),(:val38),(:val39),(:val40),"
+
"(:val41),(:val42),(:val43),(:val44),(:val45),(:val46),(:val47),(:val48),(:val49),(:val50),"
+
"(:val51),(:val52),(:val53),(:val54),(:val55),(:val56),(:val57),(:val58),(:val59),(:val60),"
+
"(:val61),(:val62),(:val63),(:val64),(:val65),(:val66),(:val67),(:val68),(:val69),(:val70),"
+
"(:val71),(:val72),(:val73),(:val74),(:val75),(:val76),(:val77),(:val78),(:val79),(:val80),"
+
"(:val81),(:val82),(:val83),(:val84),(:val85),(:val86),(:val87),(:val88),(:val89),(:val90),"
+
"(:val91),(:val92),(:val93),(:val94),(:val95),(:val96),(:val97),(:val98),(:val99),(:val100);";
jdbcTemplate.update(sqlString, parameterMap);
}
From: Vinokurov Pavel
[mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, March 14, 2018 5:42 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Performance of Ignite integrating with PostgreSQL
Hi,
You could try to use igniteCache.putAll for write batches by 1000 entries.
Use following script in PostgresDBStore#writeAll method to put data into the
database:
String sqlString = "INSERT INTO test(val) VALUES (:val1)(:val2)(:val3);";
2018-03-14 11:58 GMT+03:00
<[email protected]<mailto:[email protected]>>:
Hi,
I try to use Ignite to integrate with PostgreSQL.
And I use “atop” to monitor the data write to PostgreSQL.
Then observed that the writing speed is 1 MB per second.
This performance is not really good. Below is my configuration and code. Please
help me to improve it.
Thanks.
There is my cache configuration:
<bean
class="org.apache.ignite.configuration.CacheConfiguration">
<property name="name" value= "testCache"/>
<property name="cacheMode"
value="PARTITIONED"/>
<property name="atomicityMode" value="
ATOMIC"/>
<property name="atomicWriteOrderMode"
value="PRIMARY"/>
<property name="readThrough" value="true"/>
<property name="writeThrough"
value="true"/>
<property name="writeBehindEnabled" value="true"/>
<property name="writeBehindFlushThreadCount"
value="64"/>
<property name="writeBehindBatchSize"
value="131072" />
<property name="writeBehindFlushSize"
value="131072" />
<property name="offHeapMaxMemory" value="0" />
<property name="cacheStoreFactory">
<bean
class="javax.cache.configuration.FactoryBuilder$SingletonFactory">
<constructor-arg>
<bean
class="com.blu.imdg.jdbc.PostgresDBStore">
</bean>
</constructor-arg>
</bean>
</property>
<property name="backups" value="0"/>
<property name="indexedTypes">
<list>
<value>java.lang.String</value>
<value>java.lang.String</value>
</list>
</property>
</bean>
Main function:
Ignite ignite = Ignition.start("PATH/example-cache.xml");
IgniteCache<String, String> igniteCache =
ignite.getOrCreateCache("testCache ");
int seqint = 0;
while(true)
{
igniteCache.put(Integer.toString(seqint),
"valueString");
seqint++;
}
Write behind to PostgreSQL through JDBC:
@Override
public void write(Cache.Entry<? extends String, ? extends String> entry) throws
CacheWriterException {
Map<String, Object> parameterMap = new HashMap<>();
parameterMap.put(“val”, entry.getValue());
String sqlString = "INSERT INTO test(val) VALUES (:val);";
jdbcTemplate.update(sqlString, parameterMap);
}
--
本信件可能包含工研院機密資訊，非指定之收件者，請勿使用或揭露本信件內容，並請銷毀此信件。 This email may contain
confidential information. Please do not use or disclose it in any way and
delete it if you are not the intended recipient.
--
Regards
Pavel Vinokurov
--
本信件可能包含工研院機密資訊，非指定之收件者，請勿使用或揭露本信件內容，並請銷毀此信件。 This email may contain
confidential information. Please do not use or disclose it in any way and
delete it if you are not the intended recipient.
--
Regards
Pavel Vinokurov
--
Regards
Pavel Vinokurov
--
Regards
Pavel Vinokurov
--
Regards
Pavel Vinokurov
--
本信件可能包含工研院機密資訊，非指定之收件者，請勿使用或揭露本信件內容，並請銷毀此信件。 This email may contain
confidential information. Please do not use or disclose it in any way and
delete it if you are not the intended recipient.
