Re: apache-ignite compatibility with armhf(32-bit arm linux)

2020-07-24 Thread Stanislav Lukyanov
It should work fine. I ran Ignite on various platforms non-Intel platforms, including arm. There were some issues in the past but modern versions work well. But you do have to keep in mind that release testing for Ignite is done on Intel platforms. Also arm can bring some surprises in terms of perf

[Webinar] Networking in Apache Ignite - Talk at Sep 9 2020

2020-09-09 Thread Stanislav Lukyanov
Hi Igniters, I'll be giving a webinar titled Networking in Apache Ignite. We'll look at Apache Ignite's networking components - Discovery and Communication - to see how they work together to implement various networking functions. The webinar will be held at 10 AM PT / 1 PM ET / 5 PM GMT. Detail

Re: Ignite timeouts and trouble interpreting the logs

2020-11-12 Thread Stanislav Lukyanov
Hi, This looks weird but with the right logs we should figure it out. One thing that I don't like about these settings is the asymmetry of the server's and client's timeouts. The server will use clientFailureDetectionTimeout=30s when talking to the client. The client will use failureDetectionti

Re: High availability of local listeners for ContinuousQuery or Events

2020-11-13 Thread Stanislav Lukyanov
Hi, It's true that currently you need to implement something for Continuous Queries failover in your application code. Continuous Queries have setInitialQuery API to help with that. How it's supposed to work is: you define an initial query (e.g. an SQL query) which fetches the data that's alrea

Re: Native persistence and upgrading

2020-12-14 Thread Stanislav Lukyanov
Yes, you can upgrade from an older version to a newer one and keep the data, it will just work. You don't really need snapshots for that, although I assume snapshots would also work. > On 10 Dec 2020, at 16:20, xero wrote: > > Hi Dimitry and community, > Is this still true? My intention is to

RE: Event listeners on servers only

2019-03-26 Thread Stanislav Lukyanov
The options I see 1. Register a local listener on each node; you can call localListen() from a broadcast() job or when the node starts. 2. Deploy a cluster-singleton service that calls remoteListen() in its initialize(). I guess the first one will perform better. Stan From: maros.urbanec Sen

Re: Ignite Client getting OOM, GridAffinityProcessor grows in size

2019-03-27 Thread Stanislav Lukyanov
The memory leak looks very much like https://issues.apache.org/jira/browse/IGNITE-7918. Can you check on 2.7? Stan -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: GridGain perfomance

2019-03-27 Thread Stanislav Lukyanov
First, this is a mailing list for Apache Ignite, although the results would be more or less equal as GridGian is based on Ignite. Second, the question is too broad. You shouldn’t really think about running on 1 core as Ignite is for scaling to many cores and machines. The performance will vary g

Re: Ignite DataStreamer Memory Problems

2019-04-21 Thread Stanislav Lukyanov
I've put a full answer on SO - https://stackoverflow.com/questions/55752357/possible-memory-leak-in-ignite-datastreamer/55786023#55786023 . In short, so far it doesn't look like a memory leak to me - just a misconfiguration. There is a memory pool in JVM for direct memory buffers which is by defau

Re: Ignite DataStreamer Memory Problems

2019-04-23 Thread Stanislav Lukyanov
Can you share your full configuration (Ignite config and JVM options) and the server logs of Ignite? Which version of Ignite you use? Can you confirm that on this version and configuration simply disabling Ignite persistence removes the problem? If yes, can you try running with walMode=NONE? It w

Re: SQL delete command is slow and can cause OOM

2019-05-08 Thread Stanislav Lukyanov
Hi, You have to also fetch values to do a "compare-and-delete". Before deleting each entry you check if it has been concurrently modified. If it was then it's possible that the entry doesn't match your WHERE anymore. So yes, for now deleting a large number of entries is heap-intensive. It should

Re: Ignite backup/restore Cache-wise

2019-08-22 Thread Stanislav Lukyanov
GridGain Snapshots allow you to take a backup on a live, working cluster. If you can allow to stop the cluster activity while snapshot is being taken you can: - Deactive the cluster (e.g. control.sh --deactivate) - Copy the persistence files (you would need work/binary_meta, work/marshaller, work/d

Re: One of Ignite pod keeps crashing and not joining the cluster

2019-08-22 Thread Stanislav Lukyanov
Hi, Please share - Ignite version you're running - Exact steps and events (a node was restarted, a client joined, etc) - Logs of all three servers Thanks, Stan On Mon, Aug 19, 2019 at 3:27 PM radha jai wrote: > Hi , > Ignite being deployed on the kubernetes, there were 3 replicas of ignite >

Re: ZooKeeper Discovery - Handling large number of znodes and their cleanup

2019-08-23 Thread Stanislav Lukyanov
Hi Abhishek, What's your Ignite version? Anything else to note about the cluster? E.g. frequent topology changes (clients or servers joining and leaving, caches starting and stopping)? What was the topology version when this happened? Regarding the GC. Try adding -XX:+PrintGCApplicationStoppedTim

Re: Question on submitted post

2019-08-23 Thread Stanislav Lukyanov
Hi, It looks like the issue is that you're ending up sending an instance of your gRPC server inside your service. This approach is generally incorrect. What you should do is - not pass gRPC to the service instance - add an init() method implementation to your service - in your init() start your gR

Re: Cache Miss Using Thin Client

2019-08-23 Thread Stanislav Lukyanov
Hi, I'm thinking this could be related to differences in the binary marshaller configuration. Are you using Java thin client? What version? What is the cache key type? Are you setting a BinaryConfiguration explicitly on the client or server? Thanks, Stan On Fri, Aug 23, 2019 at 3:38 PM wrote:

Re: Support for latest version of MongoDB in Ignite web console

2019-08-26 Thread Stanislav Lukyanov
Hi, I believe support for MongoDB 4.x is already implemented in https://issues.apache.org/jira/browse/IGNITE-10847. Also, I believe Ignite doesn't require a specific version of MongoDB. Have you tried to install the latest 3.4.x version? Thanks, Stan On Sun, Aug 25, 2019 at 7:04 PM Ashfaq Ahamed

Re: HTTP/2 implementations do not robustly handle abnormal traffic and resource exhaustion

2019-08-26 Thread Stanislav Lukyanov
Hi, AFAICS this is not about the *protocol*, this is about *implementations* of the protocol. I've followed the links and found this matrix of vulnerable technologies: https://vuls.cert.org/confluence/pages/viewpage.action?pageId=56393752 >From this matrix, Ignite uses only Node.js in WebConsole,

Re: Ignite WAL and WAL archive size estimation

2019-08-26 Thread Stanislav Lukyanov
Hi, In normal circumstances checkpoint is triggered on timeout, e.g. every 3 minutes (controlled by checkpointFrequency). So, the size of the checkpoint is the amount of data written/updated in a 3-minute interval. The best way to estimate it in your system is to enable data storage metrics (DataS

Re: Question on submitted post

2019-08-26 Thread Stanislav Lukyanov
f these custom classes into the > libs folder in ignite but that has also not helped. > > On Fri, Aug 23, 2019 at 6:56 PM Stanislav Lukyanov > wrote: > >> Hi, >> >> It looks like the issue is that you're ending up sending an instance of >> your gRPC serve

Re: Drop index do not release memory used ?

2019-11-01 Thread Stanislav Lukyanov
Hi, What version do you use? There was an issue with recycling pages between data and indexes which has been fixed in 2.7 https://issues.apache.org/jira/browse/IGNITE-4958. In AI 2.7 and later this should be working fine. Stan On Sat, Oct 26, 2019 at 5:22 PM yann Blazart wrote: > Yes the data

Re: Throttling getAll

2019-11-01 Thread Stanislav Lukyanov
The right answer to this is probably not to use getAll in such cases. If you want to load data in batches then you should either split the keys yourself or use Query APIs, like ScanQuery or SqlQuery. Stan On Mon, Oct 28, 2019 at 10:36 PM Abhishek Gupta (BLOOMBERG/ 919 3RD A) < agupta...@bloomberg

Re: Query execution in ignite

2019-11-01 Thread Stanislav Lukyanov
This is not exactly correct. When you do an SQL query with only PARTITIONED tables, or with a mix of PARTITIONED and REPLICATED, the data will be taken from the primary partions of PARTITIONED tables and *all* partitions of REPLICATED tables. When you do an SQL query with only REPLICATED tables, th

Re: Ignite cache partition size

2019-11-01 Thread Stanislav Lukyanov
It's best to have the number of partitions being a power of two, so better to go with 32768 I think. There are big clusters with hundreds of nodes out there, and they do use large partition numbers sometimes - as large as 16k or 32k. Note that it will bring some overhead on the metadata being stor

Re: How does Apache Ignite distribute???

2019-11-05 Thread Stanislav Lukyanov
I believe that the correct answer to your question - don't do that. The strength of distributed systems is that you have a number of identical pieces which you can scale out virtually with no limits. If your cluster is heterogenous - i.e. all the nodes are different in size, amount of data and po

Re: Cluster in AWS can not have more than 100 nodes?

2019-11-05 Thread Stanislav Lukyanov
Each node is supposed to add its own IP and port to the S3 bucket when it starts. That said, I wouldn't check the cluster state based on the contents of the bucket alone. Check your logs for errors. Try using some tools (e.g. check out Web Console - either the one in Ignite https://apacheignite-too

Re: Ignite singel instance

2019-11-05 Thread Stanislav Lukyanov
First, 1700 TPS given your transaction structure is 17 simple operations per second, which is quite substantial - especially if you're doing that from a single thread / single ODBC client. Second, note that TRANSACTIONAL_SNAPSHOT is in beta and is not ready for production use. There are no cla

Re: ignite-rest-http in classpath error

2019-11-05 Thread Stanislav Lukyanov
Hi, Web Console requires ignite-rest-http module to be enabled. It is not enabled by default in Ignite binaries nor Docker image. The steps that you've taken are done while the container is running - so, AFTER the Ignite process has started. That's why copying the module has no effect. Try settin

Re: How to insert data?

2019-11-05 Thread Stanislav Lukyanov
There are multiple ways to configure a cache to use SQL. The easiest is to use @QuerySqlField annotation. Check out this doc https://www.gridgain.com/docs/8.7.6/developers-guide/SQL/sql-api#querysqlfield-annotation . On Tue, Nov 5, 2019 at 5:52 PM BorisBelozerov wrote: > I have 3 nodes, and I co

Re: Error while adding the node the baseline topology

2019-11-05 Thread Stanislav Lukyanov
This message actually looks worrisome: 2019-10-22 10:31:42,441][WARN ][data-streamer-stripe-3-#52][PageMemoryImpl] Parking thread=data-streamer-stripe-3-#52 for timeout (ms)=771038 It means that Ignite's throttling algorithm has decided to put a thread to sleep for 771 seconds. Can you share

Re: How sql works for near cache

2019-12-10 Thread Stanislav Lukyanov
Not out of the box but you could use SQL or ScanQuery for that. With SQL: SELECT _key FROM mycache (given that your cache is SQL-enabled). With ScanQuery: cache.query(new ScanQuery(), Cache.Entry::getKey) (may need to fix type errors to compile this) Stan On Wed, Dec 4, 2019 at 2:36 AM

Re: Client (weblogic) attempting to rejoin cluster causes shutdown.

2019-12-10 Thread Stanislav Lukyanov
Ok, there is a lot to dig through here but let me try with establishing simple things first. 1. If two nodes (client or server) have the same cache specified in the configuration, the configs must be identical. 2. If one node has a cache configuration, it will be shared between all nodes automatica

Re: Local node terminated after segmentation

2019-12-11 Thread Stanislav Lukyanov
In Ignite a node can go into "segmented" state in two cases really: 1. A node was unavailable (sleeping. hanging in full GC, etc) for a long time 2. Cluster detected a possible split-brain situation and marked the node as "segmented". Yes, split-brain protection (in GridGain implementation and in

Re: PutAll Behavior Single vs Multiple Servers

2019-12-11 Thread Stanislav Lukyanov
This is a very common pitfall with distributed systems - comparing 1 node vs 3 nodes. In short, this is not correct to compare them. When you write to one node each write does the following: 1) client sends the request to the server 2) server updates data 3) server sends the response to the client

[Meetup] The Role and Specifics of Networking - Talk at June 11 2020 for Bay Area IMC Meetup

2020-06-10 Thread Stanislav Lukyanov
Hi Igniters, Tomorrow I'll be talking at an online meetup of the Bay Area In-Memory Computing community. The subject is The Role and Specifics of Networking in Distributed Systems. We'll use Apache Ignite's protocols as an example - experienced Ignite users will guess that we'll be looking at D

RE: High cpu on ignite server nodes

2018-07-02 Thread Stanislav Lukyanov
I don't really have much to say but to try reducing/balancing on heap cache size. If you have a lot of objects on heap, you need to have a large heap, obviously. If you need to constantly add/remove on-heap objects, you'll have a lot of work for GC. Perhaps you can review your architecture to avoi

RE: OptimizedMarshaller instead of BinaryMarshaller is used forScanQuery

2018-07-09 Thread Stanislav Lukyanov
Hi Calvin, It should work the same for all queries. Ideally OptimizedMarshaller shouldn’t even be used (except for JDK classes). How did you check which marshaller is used in each case? Can you share the code of your POJO? Or perhaps a runnable reproducer? Can you also share the logs/check th

RE: OptimizedMarshaller instead of BinaryMarshaller is usedforScanQuery

2018-07-10 Thread Stanislav Lukyanov
[?:1.8.0_152]     at org.apache.ignite.internal.processors.cache.query.GridCacheQueryResponseEntry.readExternal(GridCacheQueryResponseEntry.java:90) ~[ignite-core-2.3.0-clsa.20180130.59.jar:2.3.0-clsa.20180130.59] Thanks, Calvin From: Stanislav Lukyanov [mailto:stanlukya...@gmail.com] Sent: Monday, July 0

RE: Physical colocation of Ignite nodes in different JVM's

2018-07-10 Thread Stanislav Lukyanov
Hi Gregory, > So I would need a function that returns a node id from a Cache Key then a > function returning a node of the cluster give its id I don’t quiet get how it getting a node for a key fits here (you’d need to know some key then), but ignite.affinity().mapNodeToKey() does this. How abou

RE: Does ignite support Read-consistent queries?

2018-07-10 Thread Stanislav Lukyanov
Ignite transactions support this with REPEATABLE_READ isolation level. More info here: https://apacheignite.readme.io/docs/transactions Stan From: Prasad Bhalerao Sent: 9 июля 2018 г. 14:50 To: user@ignite.apache.org Subject: Does ignite support Read-consistent queries? Read-consistent queries:

RE: Deployment of Ignite/Application upgrades to production in 100+ nodecluster

2018-07-16 Thread Stanislav Lukyanov
The functionality you’re looking for is generally called Rolling Upgrade. Ignite doesn’t support clusters with mixed versions out of the box. There are third-party solutions on top of Ignite, such as GridGain, that do have that. Thanks, Stan From: KR Kumar Sent: 16 июля 2018 г. 12:44 To: user@ig

RE: Docker Healthcheck command recommendation

2018-07-18 Thread Stanislav Lukyanov
I’d look into calling control.sh or ignitevisorcmd.sh and parsing their output. E.g. check that control.sh --cache can connect to the local node and return one of your caches. However, this check is not purely for the local node, as the command will connect to the cluster as a whole. A more loca

RE: Call to deployClusterSingleton blocked.

2018-07-19 Thread Stanislav Lukyanov
Most likely it’s either https://issues.apache.org/jira/browse/IGNITE-8023 or https://issues.apache.org/jira/browse/IGNITE-7753 Until these issues are fixed, avoid starting/restarting nodes while cluster activation is in progress. Thanks, Stan From: Calvin KL Wong, CLSA Sent: 19 июля 2018 г. 5:

RE: Insert Query in Gridgrain WebConsole

2018-07-19 Thread Stanislav Lukyanov
If you’re using GridGain, it would be better to contact their customer support. Also, check out this docs page about queries in WebConsole: https://apacheignite-tools.readme.io/docs/queries-execution Stan From: UmeshPandey Sent: 19 июля 2018 г. 12:54 To: user@ignite.apache.org Subject: Insert Qu

RE: How to increase memory of Gridgrain database.

2018-07-19 Thread Stanislav Lukyanov
If you’re using GridGain, it would be better to contact their customer support. Stan From: UmeshPandey Sent: 19 июля 2018 г. 12:59 To: user@ignite.apache.org Subject: How to increase memory of Gridgrain database. Can anyone tells, how to increase a memory of Gridgrain database? I tried below sni

RE: Insert Query in Gridgrain WebConsole

2018-07-19 Thread Stanislav Lukyanov
You can write INSERT the same way as you write SELECT. Syntax for INSERT is described here: https://apacheignite-sql.readme.io/docs/insert Thanks, Stan From: UmeshPandey Sent: 19 июля 2018 г. 13:43 To: user@ignite.apache.org Subject: RE: Insert Query in Gridgrain WebConsole In this blog insert

RE: Implementing custom affinity logic

2018-07-23 Thread Stanislav Lukyanov
Hi, I think all you need is to store (a+b)/c in a separate field of the key and annotate it with @AffinityKeyMapped: class Key { private long a, b, c; @AffinityKeyMapped private long affKey; public Key(long a, long b, long c) { this.a = a;

RE: Ignite threads in wait/park; cache stops responding

2018-07-23 Thread Stanislav Lukyanov
Hi, I’d suggest try upgrading from 2.2 to 2.6. One thing that stands out in your configs is expiryPolicy settings. expiryPolicy property was deprecated, you need to use expiryPolicyFactory instead. I happen to have recently shared a sample of setting it in an XML config on SO: https://stackoverf

RE: Random-LRU-2 ?

2018-07-24 Thread Stanislav Lukyanov
Random-LRY selects the minimum of 5 timestamps (5 pages, 1 timestamp for each page). Random-2-LRU selects the minimum of 10 timestamps (5 pages, 2 timestamps for each page). My advice is not to go that deep. Random-2-LRU I protected from the “one-hit wonder” and has a very tiny overhead compar

RE: odbc - conversion failed because data value overflowed

2018-07-24 Thread Stanislav Lukyanov
Are you trying to use MS SQL tools to connect to Ignite? I don’t think that’s possible – at least, Ignite doesn’t claim to support that. Stan From: wt Sent: 24 июля 2018 г. 16:01 To: user@ignite.apache.org Subject: odbc - conversion failed because data value overflowed I have a SQL server instan

RE: odbc - conversion failed because data value overflowed

2018-07-24 Thread Stanislav Lukyanov
I don’t know how that MSSQL studio works, but it might be dependent on T-SQL or some system views/tables specific for MSSQL. DBeaver doesn’t use any DB-specific features and is known to work with Ignite. Take a look at this page: https://apacheignite-sql.readme.io/docs/sql-tooling Stan From: wt

RE: ***UNCHECKED*** Executing SQL on cache using affinnity key

2018-07-25 Thread Stanislav Lukyanov
What do you mean by “execute select query on cache using affinity key” and what is the problem you’re trying to solve? Stan From: Prasad Bhalerao Sent: 25 июля 2018 г. 10:03 To: user@ignite.apache.org Subject: ***UNCHECKED*** Executing SQL on cache using affinnity key Hi, Is there any way to ex

RE: ***UNCHECKED*** Executing SQL on cache using affinnity key

2018-07-25 Thread Stanislav Lukyanov
the sql using a affinity key so that it gets executed only on a node which owns that data? Thanks, Prasad On Wed, Jul 25, 2018 at 3:01 PM Stanislav Lukyanov wrote: What do you mean by “execute select query on cache using affinity key” and what is the problem you’re trying to solve?   Stan

RE: What is the difference between TcpCommunicationSpi &TcpTcpDiscoverySpi ?

2018-07-25 Thread Stanislav Lukyanov
Please don’t send messages like “?”. Mailing list is not a chatroom, messages need to be complete and meaningful. Reproducer is a code snippet or project that can be run standalone to reproduce the problem. The behavior you see isn’t expected nor known, so we need to see it before we can commen

RE: Ignite.NET + Apache Cassandra integration for SQL cache

2018-07-30 Thread Stanislav Lukyanov
Hi, You don’t need to use DDL here. It’s better to - Define Key class to be like class Key { [QuerySqlField] String CacheKey; [AffinityKeyMapped] long AffinityKey; } - set your Key and UserData to be indexed cacheCfg.QueryEntities = { new QueryEntity(typeof(Key), typeo

RE: in a four node ignite cluster,use atomicLong and process is stuck incountdown latch

2018-08-02 Thread Stanislav Lukyanov
Hi, I guess this is the problem: https://issues.apache.org/jira/browse/IGNITE-8987 Stan From: zhouxy1123 Sent: 2 августа 2018 г. 9:57 To: user@ignite.apache.org Subject: in a four node ignite cluster,use atomicLong and process is stuck incountdown latch hi , in a four node ignite cluster,use a

RE: Ignite client and server on yarn with cache read through

2018-08-14 Thread Stanislav Lukyanov
Seems to be answered in a nearby thread: http://apache-ignite-users.70518.x6.nabble.com/Ignite-yarn-cache-store-class-not-found-tt23016.html Stan From: debashissinha Sent: 28 июля 2018 г. 13:01 To: user@ignite.apache.org Subject: Ignite client and server on yarn with cache read through Hi , I h

Re: Node with BaselineTopology cannot join mixed clusterrunningincompatibility mode

2018-08-14 Thread Stanislav Lukyanov
FYI the issue https://issues.apache.org/jira/browse/IGNITE-8774 is now fixed, the fix will be available in Ignite 2.7. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Data index loss problem

2018-08-24 Thread Stanislav Lukyanov
What's your version? Do you use native persistence? Stan -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Possible issue with Web Console

2018-08-24 Thread Stanislav Lukyanov
This is the commit https://github.com/apache/ignite/commit/bab61f1. Fixed in 2.7. Stan -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

RE: Eviction Policy on Dirty data

2018-09-09 Thread Stanislav Lukyanov
With writeThrough an entry in the cache will never be "dirty" in that sense - cache store will update the backing DB at the same time the cache update happens. From: monstereo Sent: 14 августа 2018 г. 22:39 To: user@ignite.apache.org Subject: Re: Eviction Policy on Dirty data yes, using cachest

RE: IGNITE-8386 question (composite pKeys)

2018-09-17 Thread Stanislav Lukyanov
Hi, The thing is that the PK index is currently created roughly as CREATE INDEX T(_key) and not CREATE INDEX T(customer_id, date). You can’t use the _key column in the WHERE clause directly, so the query optimizer can’t use the index. After the IGNITE-8386 is fixed the index will be cre

RE: Failed to get page IO instance (page content is corrupted) after onenode failed when trying to reboot.

2018-09-17 Thread Stanislav Lukyanov
I see a node in the topology flacking up and down every minute in the restart1.log: TcpDiscoveryNode [id=d6e52510-3380-4258-8a8e-798640b1786c, addrs=[10.29.42.49, 127.0.0.1], sockAddrs=[/10.29.42.49:47500, /127.0.0.1:47500], discPort=47500, order=596, intOrder=302, lastExchangeTime=153715439345

RE: BufferUnderflowException on GridRedisProtocolParser

2018-10-09 Thread Stanislav Lukyanov
It definitely does. IGNITE-7153 seems to be applicable for objects greater than 8 kb. Is that your case? If so then I guess that has to be the same issue. Stan From: Michael Fong Sent: 9 октября 2018 г. 15:54 To: user@ignite.apache.org Subject: BufferUnderflowException on GridRedisProtocolParser

RE: The problem after the ignite upgrade

2018-10-09 Thread Stanislav Lukyanov
Hi, Clients should be able to connect to the cluster normally. Please share logs if you’re still seeing this. Stan From: hulitao198758 Sent: 27 июля 2018 г. 14:32 To: user@ignite.apache.org Subject: Re: The problem after the ignite upgrade I opened the persistent storage cluster, version 2.3 a

RE: Effect of WriteSynchronizationMode on write operations inside atransaction in Apache Ignite

2018-10-09 Thread Stanislav Lukyanov
Hi, Generally, your initial thoughts on the expected latency are correct – PRIMARY_SYNC allows you not to wait for the writes to complete on the backups. However, some of the operations still have to be completed on all nodes (e.g. acquiring key locks), so increasing the number of backups does a

RE: Inconsistent data.

2018-10-09 Thread Stanislav Lukyanov
Hi, No, the behavior you’re describing is not expected. Moreover, there is a bunch of tests in Ignite that make sure the reads are consistent during the rebalance, so this shouldn’t be possible. Can you create a small reproducer project and share it? Thanks, Stan From: Shrikant Haridas Sonone

RE: Is ID generator split brain compliant?

2018-10-09 Thread Stanislav Lukyanov
Hi, If the issue is that the node is starting with a new persistent storage each time there could some issue with file system. The algorithm to choose a persistent folder is 1) If IgniteConfiguration.consistentId is specified, use the folder of that name 2) Otherwise, check if there are any exis

RE: Ignite + Spark: json4s versions are incompatible

2018-10-09 Thread Stanislav Lukyanov
Hi, AFAICS Ignite doesn’t even use json4s itself. I assume it’s only in the dependencies and binary distribution for Spark to work. So, if Spark actually needs 3.2.X you can try using that. You can remove/replace the Ignite’s json4s jar with the required one, or use your-favorite-build-system’s

RE: Ignite Query Slow

2018-10-09 Thread Stanislav Lukyanov
The call setIndexedTypes(Long.class, Person.class) will search for all `@QuerySqlField` fields in Person and create indexes for all of them with `index=true`. For example, if your class looks like this class Person { @QuerySqlField(index = true) private String name;

RE: Ignite sporadically gets java.lang.VerifyError

2018-10-09 Thread Stanislav Lukyanov
Hi, Looks like a JVM bug. The message means that bytecode generated for a lambda in the Ignite code is structurally incorrect, but the bytecode is generated in parts by the javac and JVM, so the issue must be there. I suggest you upgrade to the latest JDK 8 (currently 8u181) and see if it helps

RE: Configuration does not guarantee strict consistency betweenCacheStore and Ignite data storage upon restarts.

2018-10-09 Thread Stanislav Lukyanov
Well, exactly what it says – Ignite doesn’t guarantee consistency between CacheStore and Native Persistence. Because of this you may end up with different data in the 3rd party DB and Ignite’s persistence, so using such configuration is not advised. See this page https://apacheignite.readme.io/d

RE: Partition Loss Policy options

2018-10-09 Thread Stanislav Lukyanov
Hi, I’ve tried your test and it works as expected, with some partitions lost and the final size being ~850 (~150 less than on the start). Am I missing something? Thanks, Stan From: Roman Novichenok Sent: 2 октября 2018 г. 22:21 To: user@ignite.apache.org Subject: Re: Partition Loss Policy optio

RE: Spark Ignite Data Load failing On Large Cache

2018-10-09 Thread Stanislav Lukyanov
Hi, Please share configurations and full logs from all nodes. Stan From: ApacheUser Sent: 8 октября 2018 г. 17:49 To: user@ignite.apache.org Subject: Spark Ignite Data Load failing On Large Cache Hi,I am testing large Ignite Cache of 900GB, on 4 node VM(96GB RAM, 8CPU and 500GB SAN Storage) Sp

RE: Ignite as Hibernate L2 Cache

2018-10-09 Thread Stanislav Lukyanov
Hi, Can’t say much about Java EE usage exactly but overall the idea is 1) Start an Ignite node named “foo”, in the same JVM as the application, before Hibernate is initialized 2) Specify `org.apache.ignite.hibernate.ignite_instance_name=foo` in the Hibernate session properties 3) Do NOT specify

RE: Ignite on Spring Boot 2.0

2018-10-09 Thread Stanislav Lukyanov
Ignite 2.0 is FAR from being recent :) Try Ignite 2.6. If it doesn’t work, share configuration, logs and code that starts Ignite. Thanks, Stan From: ignite_user2016 Sent: 9 октября 2018 г. 22:00 To: user@ignite.apache.org Subject: Re: Ignite on Spring Boot 2.0 Small update - I also upgraded i

RE: Role of H2 datbase in Apache Ignite

2018-10-09 Thread Stanislav Lukyanov
In short, Ignite replaces H2’s storage level with its own. For example, Ignite implements H2’s Index interface with its own off-heap data structures underneath. When Ignite executes an SQL query, it will ask H2 to process it, then H2 will callback to Ignite’s implementations of H2’s interfaces (

RE: Partition Loss Policy options

2018-10-10 Thread Stanislav Lukyanov
.  Simpler implementation could just raise exceptions on queries when policy is ..._SAFE and some partitions are unavailable. Thanks again, Roman On Tue, Oct 9, 2018 at 2:54 PM Stanislav Lukyanov wrote: Hi,   I’ve tried your test and it works as expected, with some partitions lost and the final

RE: Ignite on Spring Boot 2.0

2018-10-10 Thread Stanislav Lukyanov
Looks like you’re not actually starting Ignite there. You need to either - Provide Ignite configuration path via SpringcacheManager.configurationPath - Provide Ignite configuration bean via SpringcacheManager.configuration - Start Ignite manually in the same JVM prior to the SB app initialization I

RE: BufferUnderflowException on GridRedisProtocolParser

2018-10-11 Thread Stanislav Lukyanov
Well, you need to wait for the IGNITE-7153 fix then. Or contribute it! :) I checked the code, and it seems to be a relatively easy fix. One needs to alter the GridRedisProtocolParser to use ParserState in the way GridTcpRestParser::parseMemcachePacker does. Stan From: Michael Fong Sent: 11 октя

RE: Working example for HDFS-Ignite data eviction

2018-10-11 Thread Stanislav Lukyanov
It looks like your use case is having Ignite as a cache for HDFS, as described here https://ignite.apache.org/use-cases/hadoop/hdfs-cache.html. Try using this guide https://apacheignite-fs.readme.io/docs/secondary-file-system. Stan From: Divya Darshan DD Sent: 26 сентября 2018 г. 9:19 To: user@

RE: Load data into particular Ignite cache from HDFS

2018-10-11 Thread Stanislav Lukyanov
I think this describes what you want: https://apacheignite-fs.readme.io/docs/secondary-file-system Stan From: Divya Darshan DD Sent: 26 сентября 2018 г. 13:51 To: user@ignite.apache.org Subject: Load data into particular Ignite cache from HDFS Can you tell me a method to load files from HDFS in

RE: Using annotations to remap fields when deserializing BinaryObjects

2018-10-11 Thread Stanislav Lukyanov
Hi, Nope, it doesn’t work like that. Names of fields in the Java class are always the same as the names of the fields in BinaryObject. Frankly, I don’t really see a strong use case for the customization you’re asking for. If it is just to support different naming conventions in different places

RE: 2 questions about eviction and transactions

2018-10-11 Thread Stanislav Lukyanov
Hi, // Sidenote: better not to ask two unrelated questions in a single email. It complicates things if the threads grow. Roughly speaking, REPLICATED cache is the same as PARTITIONED with an infinite number of backups. The behavior is supposed to always be the same. Some components cut corners

RE: client probe cache metadata

2018-10-11 Thread Stanislav Lukyanov
No, seems there are no methods for that. I assume it could be added to the metadata response. But why do you need it? Stan From: wt Sent: 11 октября 2018 г. 15:12 To: user@ignite.apache.org Subject: client probe cache metadata Hi The rest service has a meta method that returns fields and index

RE: Can I use limit and offset these two h2 features for pagination

2018-10-11 Thread Stanislav Lukyanov
`limit` and `offset` should work, with usual semantics. Thanks, Stan From: kcheng.mvp Sent: 11 октября 2018 г. 18:59 To: user@ignite.apache.org Subject: Can I use limit and offset these two h2 features for pagination I know in h2 standalone mode, we can use *limit* and *offset* features(functio

RE: Configuration does not guarantee strict consistencybetweenCacheStore and Ignite data storage upon restarts.

2018-10-11 Thread Stanislav Lukyanov
Refer to this: https://apacheignite.readme.io/docs/baseline-topology#section-usage-scenarios Stan From: the_palakkaran Sent: 9 октября 2018 г. 22:59 To: user@ignite.apache.org Subject: Re: Configuration does not guarantee strict consistencybetweenCacheStore and Ignite data storage upon restarts

RE: Can I use limit and offset these two h2 features for pagination

2018-10-11 Thread Stanislav Lukyanov
Yep, `order by` is usually needed for `limit`, otherwise you’ll get random rows of the dataset as by default there is no ordering. If I’m not mistaken, in Ignite this options work on both map and reduce steps. E.g. `limit 100` will first take the 100 rows from each node, then combine them in a t

RE: Determine Node Health?

2018-10-11 Thread Stanislav Lukyanov
You’re creating a new cache on each heath check call and never destroy them – of course, that leads to a memory leak; it’s also awful for the performance. Don’t create a new cache each time. If you really want to check that cache operations work, use the same one every time. Thanks, Stan Fr

RE: Query 3x slower with index

2018-10-11 Thread Stanislav Lukyanov
Hi, It is a rather lengthy thread and I can’t dive into details right now, but AFAICS the issue now is making affinity key index to work with a secondary index. The important things to understand is 1) Ignite will only use one index per table 2) In case of a composite index, it will apply the co

RE: Ignite on Spring Boot 2.0

2018-10-11 Thread Stanislav Lukyanov
Uhm, don’t have a tested example but it seems pretty trivial. It would be something like @Bean public SpringCacheManager SpringCacheManager() { IgniteConfiguration cfg = new IgniteConfiguration(); cfg.setIgniteInstanceName("WebGrid"); // set more Ignite parameters if

RE: Role of H2 datbase in Apache Ignite

2018-10-12 Thread Stanislav Lukyanov
tree, then merge them. Stan From: eugene miretsky Sent: 11 октября 2018 г. 20:58 To: user@ignite.apache.org Subject: Re: Role of H2 datbase in Apache Ignite Thanks!  So does it mean that CacheConfiguration.queryParallelism is really an H2 settings?  On Tue, Oct 9, 2018 at 4:27 PM Stanislav

RE: Query 3x slower with index

2018-10-12 Thread Stanislav Lukyanov
Yes, sure. From: Dave Harvey Sent: 11 октября 2018 г. 23:59 To: user@ignite.apache.org Subject: Re: Query 3x slower with index "Ignite will only use one index per table" I assume you mean "Ignite will only use one index per table per query"? On Thu, Oct 11, 2018 at 1:55 P

RE: data streamer - failed to update keys (GUID)

2018-10-12 Thread Stanislav Lukyanov
Yes, there is a direct support for UUID. If you don’t know where the error is coming from, please share the code and the logs. Stan From: wt Sent: 12 октября 2018 г. 13:00 To: user@ignite.apache.org Subject: data streamer - failed to update keys (GUID) hi I just wanted to check something. I ha

RE: data streamer - failed to update keys (GUID)

2018-10-12 Thread Stanislav Lukyanov
There is an error “Failed to update index, incorrect key class”. Any chance you’ve changed an integer field to a string one, or something like that? Changing field types is generally not supported. Stan From: wt Sent: 12 октября 2018 г. 14:06 To: user@ignite.apache.org Subject: RE: data streamer

RE: Remote Listen Events are available in C#.Net

2018-10-12 Thread Stanislav Lukyanov
Nope. Here is the JIRA to add that: https://issues.apache.org/jira/browse/IGNITE-1683. Seems like the functionality was broken at some point, and because of that it was removed from Ignite. Someone needs to address that and bring the API back to C#. Stan From: Hemasundara Rao Sent: 12 октября 20

RE: Spark Dataframe write is hanging

2018-10-12 Thread Stanislav Lukyanov
Hi, Were you able to find the root cause of this? If yes, what was it? The error indicates a network connection issue, so I guess the solution should be about the network configuration. Stan From: eugene miretsky Sent: 21 августа 2018 г. 23:35 To: user@ignite.apache.org Subject: Spark Dataframe

RE: unable to create indexing on dataframe

2018-10-12 Thread Stanislav Lukyanov
Hi, If you want help, you have to explain what is your problem. Sharing code is great, but people also need to understand what exactly you’re trying to do and what to look for. Also, your code seems to contain private classes (com.inn.*) so no one will be able to run it anyway. Stan From: shru

RE: Thin client vs client node performance in Spark

2018-10-12 Thread Stanislav Lukyanov
On your questions > 1) How does one increase write throuput without increasing number of clients > (the server nodes are underutilized at the moment) Actually, adding more clients is the supposed way of increasing throughput if servers have capacity. > 2) We have use cases where we many have man

RE: .net Datetime loads without error but table empty

2018-10-15 Thread Stanislav Lukyanov
A guess: the value is being saved, but due to an issue with name or type matching in the QueryEntity SQL engine doesn’t return it. Look for the problem in the cache config (queryEntities property), pay attention to the names, etc. Stan From: wt Sent: 15 октября 2018 г. 12:47 To: user@ignite.ap

  1   2   3   >