IgniteDataStreamer causing inconsistent partition update counters

Hi, I posted this question on Stack Overflow a few days ago and hoping I can get some eyes on it: https://stackoverflow.com/questions/79600594/ignitedatastreamer-resulting-in-partition-states-validation-has-failed-partition. Essentially, if there's a partition map exchange running while I s

IgniteDataStreamer order guarantees

The javadoc of the IgniteDataStreamer explicitly states that data order is not guaranteed. I’m hoping there’s an exception. If I set allowOverwrite to true, will two writes (from the same thread) on the same key be processed in order? Or do I need to write a custom StreamReceiver that checks

Re: Cache Preload from database using IgniteDatastreamer

It’s worth mentioning, that best performance is usually achieved when adding data to the same IgniteDataStreamer instance from multiple threads. Denis On 12 Oct 2019, 06:46 +0300, ravichandra , wrote: > Hi Om Thacker, > > I am trying to implement / follow the same approach for

Re: Cache Preload from database using IgniteDatastreamer

Hi Om Thacker, I am trying to implement / follow the same approach for my POC to understand how that can be applied to a real time scenario's. I googled but couldn't find a complete implementation in spring boot. Is it possible to share the source code location, it would help me a lot. Thanks, R

Re: DataLoad using IgniteDataStreamer

I Will look into it and get back. Thanks Saikat. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: DataLoad using IgniteDataStreamer

have any questions. Regards, Saikat On Thu, Oct 10, 2019 at 4:10 PM ravichandra < ravichandra.polemre...@gmail.com> wrote: > Hi, > > I am working on a POC to preload the data into database with > IgniteDataStreamer using client node > and do rest of the functionalities usin

DataLoad using IgniteDataStreamer

Hi, I am working on a POC to preload the data into database with IgniteDataStreamer using client node and do rest of the functionalities using server node. Is there any sample code for reference? I am using spring boot, extending IgniteRepository for repository layer Instead of using config.xml

Re: Questions on IgniteDataStreamer

gger.error(message); > ObjectMapper mapper = new ObjectMapper(); > JsonNode rootNode; > try { > rootNode = mapper.readTree(message); > Users user = new Use

Re: Questions on IgniteDataStreamer

user = new Users(); IgniteDataStreamer stmr = ignite.dataStreamer(IgniteProperties.USERS_CACHE.getName()); // stmr.allowOverwrite(true); /* * stmr.receiver(new StreamT

Re: Cache Preload from database using IgniteDatastreamer

Thanks for the reply..I was just wanted to confirm that the way I am using datastreamer to load the data in cache is correct or not. Regarding gc overhead- I was running it in my local machine with limited heap.so that's no the exact problem.please ignore that part. -- Sent from: http://apach

Re: Cache Preload from database using IgniteDatastreamer

There is no so called 'silver bullet' for data streamer parameters that will fit in all or most cases, you should try tuning individual streamer parameters to find out how it affects performance. Also it would be useful to understand is there anything that could be done against GC pressure (tuning

Cache Preload from database using IgniteDatastreamer

I have searched a lot to find out how to use IgniteDataStreamer to preload cache from database, but all i couldn't find any. So I tried to do it myslef. Following is my approach: 1. A spring boot application having java based server configuration for cache - Ignite server application 2. A s

Re: Populating tables via IgniteDataStreamer

mTransformerExample.java >> >> >> There’s nothing really “special” about caches/tables created using DDL >> rather than one of the other methods. >> >> Regards, >> Stephen >> >> On 19 Feb 2019, at 03:49, Clay Teahouse wrote: >> >> Hello

Re: Populating tables via IgniteDataStreamer

eb 2019, at 03:49, Clay Teahouse wrote: > > Hello All, > > Can someone share an example of utilizing IgniteDataStreamer API to > populate a table created via a DDL? > > thank you, > Clay > > > >

Re: Populating tables via IgniteDataStreamer

example of utilizing IgniteDataStreamer API to populate > a table created via a DDL? > > thank you, > Clay >

Re: Populating tables via IgniteDataStreamer

Hi Clay, With JDBC, I have used the following successfully: dbConnection = HikariCPDatasource.getConnection(); dbConnection.setSchema("public"); *dbConnection.createStatement().execute("ALTER TABLE tablename NOLOGGING");* *dbConnection.createStatement().execute("SET STREAMING ON");* f

Populating tables via IgniteDataStreamer

Hello All, Can someone share an example of utilizing IgniteDataStreamer API to populate a table created via a DDL? thank you, Clay

Re: How to benchmark IgniteDataStreamer?

Hello! You will need to benchmark batches, of course. IgniteDataStreamer has flush() which is synchronous. Regards, -- Ilya Kasnacheev ср, 19 дек. 2018 г. в 09:33, ashishb008 : > Hello, > > IgniteDataStreamer is async, so how will you find the exact time? > > > > &

Re: How to benchmark IgniteDataStreamer?

Hello, IgniteDataStreamer is async, so how will you find the exact time? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: How to benchmark IgniteDataStreamer?

в 08:10, ashishb008 : > We can get the total number of gets/puts & average time for gets/puts on > Caches. But in my case I do not use get/put on cache, I use > IgniteDataStreamer to load data in cache. I need to know time taken by > IgniteDataStreamer. > > Thanks > >

How to benchmark IgniteDataStreamer?

We can get the total number of gets/puts & average time for gets/puts on Caches. But in my case I do not use get/put on cache, I use IgniteDataStreamer to load data in cache. I need to know time taken by IgniteDataStreamer. Thanks -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: KafkaStreamer and IgniteDataStreamer dynamic cache name

Hello, If you have some way to determine required cache name for update from your data model, then you can implement custom StreamVisitor for the DataStreamer, code example is on the same page. -- Sent from: http://apache-i

KafkaStreamer and IgniteDataStreamer dynamic cache name

Seems KafkaStreamer and IgniteDataStreamer has one to one mapping. that's to say a KafkaStreamer can only persistent to one cache. right now my case is a bit complicated. KafkaStreamerGenericEntity*> kafkaStreamer = new KafkaStreamer<>(); during persist phrase, I need to save a

答复: data loss using IgniteDataStreamer API

Thank you, i found that autoflus is what i need. 发件人: Ilya Kasnacheev 发送时间: 2018年8月16日 21:22:10 收件人: user@ignite.apache.org 主题: Re: data loss using IgniteDataStreamer API Hello! For starters, I don't see that you do stmr.close() anywhere. Data Streamer is

答复: data loss using IgniteDataStreamer API

Thank you, i found that autoflus is what i need. 发件人: Ilya Kasnacheev 发送时间: 2018年8月16日 21:22:10 收件人: user@ignite.apache.org 主题: Re: data loss using IgniteDataStreamer API Hello! For starters, I don't see that you do stmr.close() anywhere. Data Streamer is

Re: data loss using IgniteDataStreamer API

Hello! For starters, I don't see that you do stmr.close() anywhere. Data Streamer is only guaranteed to write all data to cache after it is closed properly. Regards, -- Ilya Kasnacheev 2018-08-16 16:19 GMT+03:00 Huang Meilong : > Hi all, > > > I'm use IgniteData

data loss using IgniteDataStreamer API

Hi all, I'm use IgniteDataStreamer API to ingest 1,000,000 record to a sql table cache, but only 996,626 records, 0: jdbc:ignite:thin://127.0.0.1/> select count(*) from APMMETRIC; ++ |COUNT(*)| ++

Re: Metrics for IgniteDataStreamer

Hi, It's possible to set only one DataStreamer to the KafkaStreamer, So, I'd recommend using one KafkaStreamer per cache. Evgenii 2018-07-05 20:03 GMT+03:00 vbm : > HI Evangii, > > Thanks for the reply. I have some more question regarding ignite data > streamer. > > Below is our scenario: > We

Re: Metrics for IgniteDataStreamer

HI Evangii, Thanks for the reply. I have some more question regarding ignite data streamer. Below is our scenario: We have many Kafka topics and now we want to use ignite data streamer to pull data to ignite cache and data is such that each kafka topic correspond to a cache in ignite. >From what

Re: Metrics for IgniteDataStreamer

Well, then you can just check the cache size after the certain period for each method and compare it. I'm not sure that the metric for the thing you want will make any sense - if you will have any pauses in ingestion, this metric won't be informative at all. At the same time, absolutely the same in

Re: Metrics for IgniteDataStreamer

Hi Evgenii, To compare the 2 ingestion methods (DataStreamer and KafkaConnect), we wanted to know what are the key parameters that needs to be monitored. For example: How fast the data is being put in cache. Regards, Vishwas -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Metrics for IgniteDataStreamer

Hi, What kind of metrics will help you? Why just checking of the cache size won't be enough? Regards, Evgenii 2018-07-03 19:53 GMT+03:00 vbm : > Hi > > As part of our POC we wanted to compare the ingestion in to ignite using > Kafka Connect and Ignite Data Streamer. > > For comparison, what are

Re: Questions on IgniteDataStreamer

Hi, Can anyone provide some info on this ? Regards, Vishwas -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Metrics for IgniteDataStreamer

Hi As part of our POC we wanted to compare the ingestion in to ignite using Kafka Connect and Ignite Data Streamer. For comparison, what are the metrics that we can monitor to measure the performance. Regards, Vishwas -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteDataStreamer

Thank you - going to trying this in a bit. -gene -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteDataStreamer

Thank you Thomas, I did not even know the star "*" makes a CacheConfiguration bean registered as a template. I am not sure Ignite has it documented anywhere.

RE: IgniteDataStreamer

uot;cache_name" your cache will be called "SQL_PUBLIC_MYCACHE" ./t -Original Message- From: gene [mailto:gene.foj...@gm.com] Sent: tirsdag 16. januar 2018 04.31 To: user@ignite.apache.org Subject: Re: IgniteDataStreamer I saw that method. But the 2.3 for CREATE TABLE indicat

Re: IgniteDataStreamer

I saw that method. But the 2.3 for CREATE TABLE indicates you can passing by WITH for the template. WITH - accepts additional parameters not defined by ANSI-99 SQL: TEMPLATE= - case-sensitive​ name of a cache template registered in Ignite to use as a configuration for the distributed cache that i

Re: IgniteDataStreamer

I do not think there is a way to add a cache template via XML configuration file. You have to write code ignite.addCacheConfiguration(cacheTemplate) where cacheTemplate is an instance of CacheConfiguration that you reference from CREATE TABLE ... WITH "template=NAME" by NAME.

Re: IgniteDataStreamer

thank you. -g -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteDataStreamer

Thank you. I've moved past this issue now and have been able to create Table/Cache's using the DDL and WITH parameters. The only issue I'm facing now is how to register a cache template that I can call using the WITH Template= -g -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: IgniteDataStreamer

gt; You can access a cache created with DDL (CREATE TABLE) from Java API > including IgniteDataStreamer API by prepending "SQL_PUBLIC_" to the > capitalised table name. For example, if you created a table as "create table > person ... " then you can ac

Re: IgniteDataStreamer

You can access a cache created with DDL (CREATE TABLE) from Java API including IgniteDataStreamer API by prepending "SQL_PUBLIC_" to the capitalised table name. For example, if you created a table as "create table person ... " then you can access if from Java API as

IgniteDataStreamer

Is it possible to utilize IgniteDataStreamer with cache created from DDL? If yes, a short example please.. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: deploy is not working by IgniteDataStreamer

r for both objects. And there are no any information about it in the documentation and examples( -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/deploy-is-not-working-by-IgniteDataStreamer-tp13804p14125.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: deploy is not working by IgniteDataStreamer

u, Jun 15, 2017 at 10:40 AM, nash1k [via Apache Ignite Users] <[hidden > email] <http:///user/SendEmail.jtp?type=node&node=13963&i=0>> wrote: > >> Hi! >> I tried to use IgniteDataStreamer with kafka integration (extends >> StreamAdapter), but had a lit

Re: deploy is not working by IgniteDataStreamer

tried to use IgniteDataStreamer with kafka integration (extends > StreamAdapter), but had a little strange situation. > final IgniteDataStreamer streamer = > ignite.dataStreamer(callCache().getName()); > streamer.autoFlushFrequency(5000); > streamer.keepBinary(false); > streamer.perNodeBufferSize

deploy is not working by IgniteDataStreamer

Hi! I tried to use IgniteDataStreamer with kafka integration (extends StreamAdapter), but had a little strange situation. final IgniteDataStreamer streamer = ignite.dataStreamer(callCache().getName()); streamer.autoFlushFrequency(5000); streamer.keepBinary(false); streamer.perNodeBufferSize(5120

Re: IgniteDataStreamer with Continuous Query

n(c); } Presently, your continuous query is automatically closed right after the initial result is received. This is triggered by try-with-resource block. -- Denis -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IgniteDataStreamer-with-Continuous-Qu

Re: IgniteDataStreamer with Continuous Query

fifo.FifoEvictionPolicy"> > > > > > > > > > > > > > > > > > > > Ignite Data Streamer Code via StreamTransformer > > CacheConfiguration ipCacheConfiguratio

Re: IgniteDataStreamer with Continuous Query

he ipCache = ignite.getOrCreateCache(ipCacheConfiguration); Random RAND = new Random(); try (IgniteDataStreamer igniteDataStreamer = ignite.dataStreamer(ipCache.getName())){ igniteDataStreamer.allowOverwrite(true); igniteDataStreamer.receiver(new StreamT

Re: IgniteDataStreamer with Continuous Query

g/apache/ignite/examples/streaming/ > StreamTransformerExample.java > > [2] > https://github.com/apache/ignite/blob/master/examples/ > src/main/java/org/apache/ignite/examples/datagrid/ > CacheContinuousQueryExample.java > > > > -- > View this message in context: http:/

Re: IgniteDataStreamer with Continuous Query

/examples/src/main/java/org/apache/ignite/examples/datagrid/CacheContinuousQueryExample.java -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IgniteDataStreamer-with-Continuous-Query-tp9779p9781.html Sent from the Apache Ignite Users mailing list archive at

IgniteDataStreamer with Continuous Query

.nabble.com/IgniteDataStreamer-with-Continuous-Query-tp9779.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Inject the data through IgniteDataStreamer

rough-IgniteDataStreamer-tp8505p8564.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Inject the data through IgniteDataStreamer

message in context: http://apache-ignite-users.70518.x6.nabble.com/Inject-the-data-through-IgniteDataStreamer-tp8505p8534.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Inject the data through IgniteDataStreamer

ugh IgniteDataStreamer(use flow calculation > through apache ignite StreamTransformer),it turns out > error:org.apache.ignite.binary.BinaryObjectException: Duplicate field ID: > profileSQL. This exception means that within one class there are two fields that have the same hash code (taking field name

Re: Does IgniteDataStreamer support traffic control

sage in context: http://apache-ignite-users. > 70518.x6.nabble.com/Does-IgniteDataStreamer-support- > traffic-control-tp7210p7234.html > Sent from the Apache Ignite Users mailing list archive at Nabble.com. > -- Vladislav Pyatkov

Re: Does IgniteDataStreamer support traffic control

-IgniteDataStreamer-support-traffic-control-tp7210p7234.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Does IgniteDataStreamer support traffic control

it works. On Mon, Aug 22, 2016 at 6:12 PM, ght230 wrote: > Sometimes the requests to put the data to IgniteDataStreamer are very > frequently, this will cause the Ignite cluster instability. > > I want to know does IgniteDataStreamer support traffic control or anything > like back

Does IgniteDataStreamer support traffic control

Sometimes the requests to put the data to IgniteDataStreamer are very frequently, this will cause the Ignite cluster instability. I want to know does IgniteDataStreamer support traffic control or anything like backpress(supported by storm)? -- View this message in context: http://apache

Re: IgniteDataStreamer can't wirte data to ignite cluster

Continue discussion here http://apache-ignite-users.70518.x6.nabble.com/IgniteDataStreamer-can-t-wirte-data-to-ignite-cluster-tt7012.html -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/IgniteDataStreamer-can-t-wirte-data-to-ignite-cluster-tp7003p7018.html Sent

Re: IgniteDataStreamer can't wirte data to ignite cluster

> > When there is little data in the ignite cluster, I can use > IgniteDataStreamer to write data into the ignite cluster. But if there have > been more than 200G data in the ignite cluster, I can't use > IgniteDataStreamer to write more data into the cluster. The memory and d

IgniteDataStreamer can't wirte data to ignite cluster

IgniteDataStreamer to write data into the ignite cluster. But if there have been more than 200G data in the ignite cluster, I can't use IgniteDataStreamer to write more data into the cluster. The memory and disk space are both enough. and the CPU load average is 2.09. The client log is : [

IgniteDataStreamer can't wirte data to ignite cluster

IgniteDataStreamer to write data into the ignite cluster. But if there have been more than 200G data in the ignite cluster, I can't use IgniteDataStreamer to write more data into the cluster. The memory and disk space are both enough. and the CPU load average is 2.09. The client log is : [17:04:0

Re: IgniteDataStreamer lose data in multithreading?

context: http://apache-ignite-users.70518.x6.nabble.com/IgniteDataStreamer-lose-data-in-multithreading-tp6949p6992.html Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Re: Ignite : IgniteDataStreamer question about units/valid ranges for perNodeBufferSize, autoFlushFrequency

Hello, I do not see any restrictions other than reasonable. perNodeBufferSize only positive value autoFlushFrequency only positive value and if auto flush disabled then 0 On Mon, Jun 6, 2016 at 11:33 AM, M Singh wrote: > Is there any valid range for these attributes ? > > > On Monday, June 6,

Re: Ignite : IgniteDataStreamer question about units/valid ranges for perNodeBufferSize, autoFlushFrequency

Is there any valid range for these attributes ? On Monday, June 6, 2016 1:31 AM, M Singh wrote: Thanks Vladislav for the clarification. On Monday, June 6, 2016 12:45 AM, Vladislav Pyatkov wrote: Hello, 1) perNodeBufferSize - is the number of entries in the buffer.2)  autoFlus

Re: Ignite : IgniteDataStreamer question about units/valid ranges for perNodeBufferSize, autoFlushFrequency

Thanks Vladislav for the clarification. On Monday, June 6, 2016 12:45 AM, Vladislav Pyatkov wrote: Hello, 1) perNodeBufferSize - is the number of entries in the buffer.2)  autoFlushFrequency - in milliseconds On Sun, Jun 5, 2016 at 8:14 PM, M Singh wrote: Hi: I was looking at the java

Re: Ignite : IgniteDataStreamer question about units/valid ranges for perNodeBufferSize, autoFlushFrequency

Hello, 1) perNodeBufferSize - is the number of entries in the buffer. 2) autoFlushFrequency - in milliseconds On Sun, Jun 5, 2016 at 8:14 PM, M Singh wrote: > Hi: > > I was looking at the javadoc for some of the methods in this interface and > am not sure of units as well as the ranges for thes

Ignite : IgniteDataStreamer question about units/valid ranges for perNodeBufferSize, autoFlushFrequency

Hi: I was looking at the javadoc for some of the methods in this interface and am not sure of units as well as the ranges for these allowed values.  The impl class just checks for positive argument.   If anyone has any pointers, please let me know.  Thanks     /**     * Gets size of per node key