Re: Affinity Keys in Ignite .NET for joining more than two caches

2025-02-03 Thread Pavel Tupitsyn
Hi, in this case you'll have to add the Employee.DepartmentId column and use department id as an affinity key in all caches. On Mon, Feb 3, 2025 at 9:13 AM wrote: > Hi Pavel, > > > > I have lets say three below caches and want to join using Affinity > key for

Affinity Keys in Ignite .NET for joining more than two caches

2025-02-02 Thread satyajit.mandal.barclays.com via user
Hi Pavel, I have lets say three below caches and want to join using Affinity key for these three caches so that data is collocated . So how affinity key needs to be built for Employee Cache and Organization Cache so that Employee Cache, Oganization Cache and

Ignite 2.10 - affinity key backups mismatch while adding new node

2024-06-12 Thread Charlin S
). IgniteCheckException: Affinity key backups mismatch [cachename=TestModel, localaffinityKeyBackups=0, remoteaffinityKeyBackups=2 ] fix cache configuration or set system property -DIGNITE_SKIP_CONFIGURATION_CONSISTENCY_CHECK=true Server node configuration as below in all 3 servers(IP address masked) http

Re: Ignite Affinity - sql join Question

2023-06-14 Thread MJ
n the equality operation of affinity keys. Left side: PERSON; right side: ORGANIZATION Thanks ---Original--- From: "Jiang Jacky"https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/sql/SqlQueriesExample.java Get Outlook for iOS

Re: Ignite Affinity - sql join Question

2023-06-13 Thread Jiang Jacky
6733...@qq.com> Sent: Tuesday, June 13, 2023 1:24:48 AM To: user Subject: Re: Ignite Affinity - sql join Question No luck. With the change its throwing exception BinaryObjectImpl cannot be cast to java.lang.String when it tried to put a new entey into "companies" cache. B

Re: Ignite Affinity - sql join Question

2023-06-12 Thread MJ
marked on sql join's both left and right sides? Thanks, MJ ---Original--- From: "Jiang Jacky"https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation and added with some additional annotation changes.   public class AffinityCollocationExample2 {    

Re: Ignite Affinity - sql join Question

2023-06-12 Thread Jiang Jacky
Hello You have to set your company configuration with affinity key CacheConfiguration, Company> companyCfg = newCacheConfiguration<>("companies"); Get Outlook for iOS<https://aka.ms/o0ukef> From: MJ <6733...@qq.com> Sent: Monda

Ignite Affinity - sql join Question

2023-06-12 Thread MJ
] org.apache.ignite.internal.processors.query.h2.QueryParser - For join two partitioned tables join condition should contain the equality operation of affinity keys. Left side: PERSON; right side: COMPANY   Ignite version: 2.13.0 see below test code, which is extracted from https://ignite.apache.org/docs/latest/data-modeling

Re: How to check that affinity key works?

2023-01-12 Thread Jeremy McMillan
Using distributed join will set a baseline for correct result set. If you get different results without distributed join, then you know there's a mismatch between join conditions and affinity. If you get the same results, then data is distributed well for this join. On Thu, Jan 12, 2023,

Re: How to check that affinity key works?

2023-01-12 Thread Айсина Роза Мунеровна
Hi Jeremy! Thank you for your reply. Am I right that if JOIN is collocated then absence of affinity key will lead to incorrect results as data will not be fetched from other nodes? So the correct way to check influence of affinity key is to enable distributed JOIN? On 10 Jan 2023, at 10:48

Re: How to check that affinity key works?

2023-01-10 Thread Jeremy McMillan
If you are only doing colocated joins, then there will be no runtime overhead incurred by collecting distributed rows (colocated joins mean do not try to join data that might be distributed across nodes), so there might not be much difference in runtimes. The difference between different affinity

How to check that affinity key works?

2023-01-10 Thread Айсина Роза Мунеровна
” as affinity key so that all data required for join will be located on the same node. (Column “product_id” is always part of the primary key) But! After this we made experiments: - put affinity key to other part of primary key (for example, if primary key is "(product_id, store_id)", the

Re: How to handle missed indexes and affinity keys?

2022-12-20 Thread Zhenya Stanilovsky via user
>with persistence mode ? > > > Hi Maksim! > The problem is that simple SELECT query runs in ~20min - this index does not >work. > More over, other (not corrupted) tables with affinity key == primary key have >index by concrete column, not _KEY, and have specified affinity key

Re: How to handle missed indexes and affinity keys?

2022-12-20 Thread Айсина Роза Мунеровна
this index does not work. More over, other (not corrupted) tables with affinity key == primary key have index by concrete column, not _KEY, and have specified affinity key - see my first message with example. We have hypothesis that somehow these corrupted caches were created by Key-Value API

Re: How to handle missed indexes and affinity keys?

2022-12-19 Thread Zhenya Stanilovsky via user
Hi Roza, when did you observe such a problem after restart ? and your caches with persistence mode ?   >Hi Maksim! >  >The problem is that simple SELECT query runs in ~20min - this index does not >work. >  >More over, other (not corrupted) tables with affinity key == primar

Re: How to handle missed indexes and affinity keys?

2022-12-19 Thread Maksim Timonin
it. >> The more important question - is there any way to rebuild index and add affinity key back? Affinity can be set only during cache creation. Please check the docs if you create a cache before table: https://ignite.apache.org/docs/2.14.0/data-modeling/affinity-collocation#configuring-affinity-k

Re: How to handle missed indexes and affinity keys?

2022-12-19 Thread Айсина Роза Мунеровна
Hi Maksim! The problem is that simple SELECT query runs in ~20min - this index does not work. More over, other (not corrupted) tables with affinity key == primary key have index by concrete column, not _KEY, and have specified affinity key - see my first message with example. We have

Re: How to handle missed indexes and affinity keys?

2022-12-16 Thread Maksim Timonin
Hi Roza, In this ddl primary key (product_sku) equals the affinity key (product_sku). In such cases Ignite skips creating an additional index because _key_PK index already covers primary key. Thanks, Maksim On Fri, Dec 16, 2022 at 2:06 PM Айсина Роза Мунеровна < roza.ays...@sbermarket.ru>

Re: How to handle missed indexes and affinity keys?

2022-12-16 Thread Айсина Роза Мунеровна
persistence and all caches either replicated or partitioned with affinity key. All caches are created via DDL with CREATE TABLE IF NOT EXISTS statements in one regular job (once per day). The problem is that we hit Query execution is too long warning. After some debug we found out that some ta

Re: How to handle missed indexes and affinity keys?

2022-12-16 Thread Stephen Darlington
; So we have 5 node cluster with persistence and all caches either replicated > or partitioned with affinity key. > All caches are created via DDL with CREATE TABLE IF NOT EXISTS statements in > one regular job (once per day). > > The problem is that we hit Query execution is too

How to handle missed indexes and affinity keys?

2022-12-16 Thread Айсина Роза Мунеровна
Hola! We've discovered some strange behaviour in Ignite cluster and now we are trying to understand how to recover from this state. So we have 5 node cluster with persistence and all caches either replicated or partitioned with affinity key. All caches are created via DDL with CREATE TAB

Re: Node crashed with error "Getting affinity for too old topology version that is already out of history"

2022-04-08 Thread Вячеслав Коптилин
> > > Can you please advise if there are anything that we can check on the > below? Thanks. > > > > Regards, > > Marcus > > > > *From:* Lo, Marcus [ICG-IT] > *Sent:* Wednesday, March 30, 2022 11:55 AM > *To:* user > *Subject:* Node crashed with er

RE: Node crashed with error "Getting affinity for too old topology version that is already out of history"

2022-04-06 Thread Lo, Marcus
Hi Ignite team, Can you please advise if there are anything that we can check on the below? Thanks. Regards, Marcus From: Lo, Marcus [ICG-IT] Sent: Wednesday, March 30, 2022 11:55 AM To: user Subject: Node crashed with error "Getting affinity for too old topology version that is alread

Re: Affinity function with thin clients

2021-11-23 Thread R. Subhash Babu
Thnk you Stephen and Pavel. Seems like I have to go with a thick client for my use case. On Mon, Nov 22, 2021, 8:35 PM Pavel Tupitsyn wrote: > You can use define one or more cache templates [1] with a custom affinity > function (or any other setting) on the server side, > then crea

Re: Affinity function with thin clients

2021-11-22 Thread Pavel Tupitsyn
You can use define one or more cache templates [1] with a custom affinity function (or any other setting) on the server side, then create caches using the template from the thin client. [1] https://ignite.apache.org/docs/latest/configuring-caches/configuration-overview#cache-templates On Mon

Re: Affinity function with thin clients

2021-11-22 Thread Stephen Darlington
I think you’ll either have to configure that on the server side or in a thick-client. > On 22 Nov 2021, at 14:42, R. Subhash Babu wrote: > > Hi Igniters, > > I am trying to create a cache using a thin client (ClientCacheConfiguration) > and trying to see how I can >

Affinity function with thin clients

2021-11-22 Thread R. Subhash Babu
Hi Igniters, I am trying to create a cache using a thin client (ClientCacheConfiguration) and trying to see how I can 1) provide affinity function 2) restrict the number of partitions of the cache Any possible approach please? Thanks Subhash

Re: Best practices on how to approach data centre aware affinity

2021-08-16 Thread Stephen Darlington
work, but it could degrade your throughput and you need > to be careful about split-brain and other networking issues. > > Regards, > Stephen > >> On 5 Aug 2021, at 15:24, Courtney Robinson > <mailto:courtney.robin...@hypi.io>> wrote: >> >> Hi Alex, >

Re: Best practices on how to approach data centre aware affinity

2021-08-16 Thread Courtney Robinson
Hi Alex, > Thanks for the reply. I'm glad I asked before the team went any further. > So we can achieve this with the built in affinity function and the backup > filter. The real complexity is going to be in migrating our existing caches. > > So to clarify the steps involved he

Re: Best practices on how to approach data centre aware affinity

2021-08-16 Thread Stephen Darlington
. Regards, Stephen > On 5 Aug 2021, at 15:24, Courtney Robinson wrote: > > Hi Alex, > Thanks for the reply. I'm glad I asked before the team went any further. > So we can achieve this with the built in affinity function and the backup > filter. The real complexity is going

Re: Best practices on how to approach data centre aware affinity

2021-08-05 Thread Courtney Robinson
Hi Alex, Thanks for the reply. I'm glad I asked before the team went any further. So we can achieve this with the built in affinity function and the backup filter. The real complexity is going to be in migrating our existing caches. So to clarify the steps involved here are 1. because I

Re: Best practices on how to approach data centre aware affinity

2021-08-05 Thread Alex Plehanov
Hello, You can create your own cache templates with the affinity function you require (currently you use a predefined "partitioned" template, which only sets cache mode to "PARTITIONED"). See [1] for more information about cache templates. > Is this the right approach >

Best practices on how to approach data centre aware affinity

2021-08-04 Thread Courtney Robinson
our case, most tables use a template that looks like this: partitioned,backups=2,data_region=hypi,cache_group=hypi,write_synchronization_mode=primary_sync,affinity_key=instance_id,atomicity=ATOMIC,cache_name=Person,key_type=PersonKey,value_type=PersonValue I'm aware of affinity

Re: Designing Affinity Key for more locality

2021-04-29 Thread William.L
I am using a user centric modeling approach where most of the computations would be on a per-user basis (joins) before aggregation. The idea is to put the data (across different tables/caches) for the same user in the same partition/server. That's the reason why I chose user-id as the affinit

Re: Designing Affinity Key for more locality

2021-04-28 Thread Ilya Kasnacheev
Hello! SQL query planner will not understand the locality if you use surrogate value as affinity key. Maybe you need to define your own affinity function (extends RendezvousAffinityFunction) which will map keys to partitions. I'm not sure that it will help query planner though. Re

Re: Designing Affinity Key for more locality

2021-04-26 Thread Pavel Tupitsyn
Hi William, Can you describe the use case and domain model in more detail? 1. AffinityKey is used to colocate some data with other data. What do you achieve with user-id being the affinity key? 2. If you'd like to put all users for a given tenant/group to the same node for effic

Re: Designing Affinity Key for more locality

2021-04-26 Thread William.L
Came across this statement in the Data Partitioning documents: "The affinity function determines the mapping between keys and partitions. Each partition is identified by a number from a limited set (0 to 1023 by default)." Looks like there is no point for adding another layer of mappin

Designing Affinity Key for more locality

2021-04-26 Thread William.L
Hi, I am currently using user-id as the affinity key and because it is a uuid/string, it will distribute across all partitions in the cluster. However, for my scenario, I am writing and reading users that are for the same tenant/group so it seems to me that it is better to design with more read

Re: Generated affinity key from multiple fields and SQL partition pruning

2021-04-21 Thread Ilya Kasnacheev
Hello! I suggest generating a surrogate affinity key in this case, such as @AffinityKeyMapped String affKey = key1 + "-" + key2; You can also file a feature request against Apache Ignite JIRA, but given the lack of community interest towards indexing nested objects, it is unlikely

Generated affinity key from multiple fields and SQL partition pruning

2021-04-14 Thread michal.holic
x27;t know how and from which fields to generate the affinity key... and there's probably no way to configure it. Could we get this feature, please? Regar

Re: [2.10]Binary type has different affinity key fields

2021-04-02 Thread Ilya Kasnacheev
or > appears. > We don't want to lose all the data. It's acceptable to rebuild only one > table. > > -- 原始邮件 -- > *发件人:* "user" ; > *发送时间:* 2021年4月1日(星期四) 晚上11:24 > *收件人:* "user"; > *主题:* Re: [2.10]Binary ty

[2.10]Binary type has different affinity key fields

2021-04-01 Thread 38797715
Hello Ilya, Our approach is not to empty binary_meta directory, because if you empty this directory, the data of all tables may be lost. We use the cat *.bin file to find the specific bin file of a table, and then delete it on all nodes. After this operation, the following error appears. We

Re: [2.10]Binary type has different affinity key fields

2021-04-01 Thread Ilya Kasnacheev
> ~[ignite-core-2.10.0.jar!/:2.10.0] > at > org.apache.ignite.internal.processors.cache.GridCacheProcessor.onKernalStart(GridCacheProcessor.java:677) > ~[ignite-core-2.10.0.jar!/:2.10.0] > at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1407) > ~[ignite-core-2.

[2.10]Binary type has different affinity key fields

2021-03-31 Thread 38797715
ryObjectException: Binary type has different affinity key fields [typeName=WorkspaceKey, affKeyFieldName1=ID, affKeyFieldName2=null] at org.apache.ignite.internal.binary.BinaryUtils.mergeMetadata(BinaryUtils.java:999) ~[ignite-core-2.10.0.jar!/:2.10.0] at org.apache.ignite.internal.bi

Re: PME and affinity guarantees

2021-02-01 Thread Вячеслав Коптилин
Hello Ivan, Well, EVT_CACHE_REBALANCE_PART_LOADED is triggered when the corresponding partition is fully rebalanced and moved into the "OWNING" state, and so, invoking Affinity#isPrimaryOrBackup(ignite.cluster().localNode(), key) on this node should return "true". to b

PME and affinity guarantees

2021-01-28 Thread Ivan.Fedorenkov
Dear Igniters, Could someone please clarify if it is guaranteed that the Affinity (GridCacheAffinityImpl) will have the most up-to-date information about partitions distribution and the following scenario is impossible: 1. We have a registered listener of EVT_CACHE_REBALANCE_PART_LOADED

Re: Slow cache loading with an object key using affinity key

2021-01-24 Thread vtchernyi
Hi,Please take a look at my post [1]. It is not exactly about you question, but it works in production for row count you mentionRegards,Vladimir[1] https://www.gridgain.com/resources/blog/how-fast-load-large-datasets-apache-ignite-using-key-value-api22:39, 22 января 2021 г., gvaidya :Hi Alex,The da

Re: Slow cache loading with an object key using affinity key

2021-01-22 Thread gvaidya
Hi Alex, The data distribution of itemid is very even. For the ~537K records data set there are exactly 12 records per itemid for all values of itemid. I have reviewed all the documentation in the links you provided. I also switched to jdk8 (i was using jdk15 earlier), set my jvm options per the

Re: Slow cache loading with an object key using affinity key

2021-01-21 Thread akorensh
. more here: https://ignite.apache.org/docs/latest/data-modeling/data-partitioning https://ignite.apache.org/docs/latest/data-modeling/affinity-collocation Affinity mapping takes up more memory as various intermediate objects are created. Make sure you've allocated sufficient memory Ca

Re: Slow cache loading with an object key using affinity key

2021-01-21 Thread gvaidya
where the table is imported as a partitioned cache with no (or default) affinity defined. Test2 is where the same table is imported as a partitioned cache with an affinity key for colocating data based on a column that is not part of PK. In case of Test1, I am able to import/load cache with ~537K

Re: Slow cache loading with an object key using affinity key

2021-01-21 Thread akorensh
Hi, Can you give more context regarding your data load. How big is your cluster?* Are the load times comparable if you use only one node for both use cases? * If you remove the affinity column in the second use case, does the load speed up in a cluster env? If you lower the amount

Slow cache loading with an object key using affinity key

2021-01-20 Thread gvaidya
define an affinity key... assuming it defaulted to Id PK column of the table. The loadCache for my full data ran in ~16 minutes on a single node on a moderate sized 2CPU 32Gb Win machine. I want to use another integer column as the affinity key for colocation purposes. However, the loadCache process

Re: Failed to wait for affinity ready future for topology version: AffinityTopologyVersion

2020-12-11 Thread andrei
/ restore -/SnapshotDiscoveryMessage/ * Global WAL enable / disable -/WalStateAbstractMessage/ * Late affinity assignment -/CacheAffinityChangeMessage/ BR, Andrei 12/10/2020 3:54 AM, 38797715 пишет: Hi Andrei, Another question is, what is the starting point of PME? PME starts when a new node join

Re: Failed to wait for affinity ready future for topology version: AffinityTopologyVersion

2020-12-09 Thread 38797715
this true? log are as follows: [19:15:34,959][SEVERE][query-#44319][GridMapQueryExecutor] Failed to execute local query. class org.apache.ignite.IgniteException: Failed to wait for affinity ready future for topology version: AffinityTopologyVe

Re: Failed to wait for affinity ready future for topology version: AffinityTopologyVersion

2020-12-09 Thread 38797715
affinity ready future for topology version: AffinityTopologyVersion [topVer=57, minorTopVer=0]     at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.awaitTopologyVersion(GridAffinityAssignmentCache.java:909)     at

Re: Failed to wait for affinity ready future for topology version: AffinityTopologyVersion

2020-12-09 Thread andrei
org.apache.ignite.IgniteException: Failed to wait for affinity ready future for topology version: AffinityTopologyVersion [topVer=57, minorTopVer=0]     at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.awaitTopologyVersion(GridAffinityAssignmentCache.java:909)     at

Failed to wait for affinity ready future for topology version: AffinityTopologyVersion

2020-12-09 Thread 38797715
affinity ready future for topology version: AffinityTopologyVersion [topVer=57, minorTopVer=0]     at org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.awaitTopologyVersion(GridAffinityAssignmentCache.java:909)     at

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

2020-10-15 Thread Ilya Kasnacheev
Hello! Why not, if it fixes the issue for you. Then you can try 2.9.0 once it is out. Unfortunately, a small snippet of server log is not sufficient, complete log is needed. I can already spot some communication problems, though. Regards, -- Ilya Kasnacheev чт, 15 окт. 2020 г. в 09:50, swara

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

2020-10-14 Thread swara
Hello We are using this type of cache design from last 3 years with 2.5.6 version. we never got any issue. When we upgraded to 2.8.1 we are getting this type of errors. Should we downgrade to 2.8.0 or 2.5.6. Please suggest? Thank You Swara -- Sent from: http://apache-ignite-users.70518.x6.nab

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

2020-10-14 Thread swara
Hello Same error we got again today. Ignite.log- java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history [locNode=TcpDiscoveryNode [id=57473290-f181-49e5-9f5e-81188abc012f, consistentId=57473290-f181-49e5-9f5e-81188abc012f, addrs=ArrayList

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

2020-10-13 Thread Ilya Kasnacheev
Hello! How often do you create or drop tables? Minor version of 255 suggests a lot of operations. Can you share logs for your nodes prior to that error? Regards, -- Ilya Kasnacheev пн, 12 окт. 2020 г. в 16:04, swara : > one per table, whenever the table definition/data changes > hourly/daily

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

2020-10-12 Thread swara
one per table, whenever the table definition/data changes hourly/daily/weekly. please suggest if there is any other way to do joins without using cache? Thank You Swara -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

2020-10-12 Thread Ilya Kasnacheev
Hello! How many caches do you have? How often are they created? Regards, -- Ilya Kasnacheev пн, 12 окт. 2020 г. в 13:35, swara : > We are creating caches for each table and joining tables. > > Thank You > Swara > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

2020-10-12 Thread swara
We are creating caches for each table and joining tables. Thank You Swara -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

2020-10-12 Thread Ilya Kasnacheev
9 - [EVENT FAILURE Anonymous:null@unknown -> > /DefaultName/com.inmemory.IgniteInMemoryDAO] Exception : > java.lang.IllegalStateException: Getting affinity for too old topology > version that is already out of history [locNode=TcpDiscoveryNode > [id=cfb6e618-4a12-43db-b14a-68efdea05a8c,

java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history

2020-10-11 Thread swara
Hi Getting this error run time. ERROR IgniteInMemoryDAO:119 - [EVENT FAILURE Anonymous:null@unknown -> /DefaultName/com.inmemory.IgniteInMemoryDAO] Exception : java.lang.IllegalStateException: Getting affinity for too old topology version that is already out of history [locNode=TcpDiscoveryN

Re: Basic Affinity Question

2020-05-14 Thread Stephen Darlington
No, it needs to be able to determine which node the data should be on without reference to anything else. This means that you’d need company-id in your address table. > On 13 May 2020, at 21:23, narges saleh wrote: > >  > My other question is whether affinity is transitiv

Re: Basic Affinity Question

2020-05-13 Thread narges saleh
My other question is whether affinity is transitive. Company (primary key: company-id) Employee(primary key: employee-id + company-id) <- company id = affinity key Address(Primary key: address-id + employee -id) <- employee id = affinity key Would the records related company, employ

Re: Basic Affinity Question

2020-05-13 Thread narges saleh
Thanks. If I try AffinityRun with collection of caches and the affinity key, I'd get NPE with cache.localpeek using the primary key of the dependent cache/table but the cache query itself succeeds. Is there a working example? On Wed, May 13, 2020 at 8:10 AM Stephen Darlington < stephen

Re: Basic Affinity Question

2020-05-13 Thread Stephen Darlington
in a config > file, > Employee (partitioned) > employee-id > company-id > name > primary key(employee-id, company-id), affinity(company-id) > > Company (partitioned) > company-id. > name > primary key(company-id) > > If I do AffinityRun or mapkeyston

Basic Affinity Question

2020-05-13 Thread narges saleh
Hi All, If I have these two caches/tables defined via query entities, in a config file, Employee (partitioned) employee-id company-id name primary key(employee-id, company-id), affinity(company-id) Company (partitioned) company-id. name primary key(company-id) If I do AffinityRun

Re: Query Entity and Affinity Key

2020-03-17 Thread Evgenii Zhuravlev
>>> >>> >>> >>> >>> On Mon, Mar 16, 2020 at 3:22 PM Evgenii Zhuravlev < >>> e.zhuravlev...@gmail.com> wrote: >>> >>>> Hi, >

Re: Query Entity and Affinity Key

2020-03-16 Thread narges saleh
t; EmployeeId >> >> >> >> >> >> On Mon, Mar 16, 2020 at 3:22 PM Evgenii Zhuravlev < >> e.zhuravlev...@gmail.com&

Re: Query Entity and Affinity Key

2020-03-16 Thread Evgenii Zhuravlev
e cache per table(query entity) and configure separate >> CacheKeyConfigurations for each of the caches. >> >> Evgenii >> >> пн, 16 мар. 2020 г. в 11:42, narges saleh : >> >>> Hi All, >>> >>> I have a question that might be trivial. >>

Re: Query Entity and Affinity Key

2020-03-16 Thread narges saleh
y entity for each in the XML configuration file and >> define the affinity key via CacheKeyConfiguration (again in the xml file), >> considering that here affinity key is defined at IgniteConfiguration level, >> how do I specify which cache keys (i.e., query entity key fields) will u

Re: Query Entity and Affinity Key

2020-03-16 Thread Evgenii Zhuravlev
onfiguration file and > define the affinity key via CacheKeyConfiguration (again in the xml file), > considering that here affinity key is defined at IgniteConfiguration level, > how do I specify which cache keys (i.e., query entity key fields) will use > the affinity key and which on

Query Entity and Affinity Key

2020-03-16 Thread narges saleh
Hi All, I have a question that might be trivial. If I define my query entity for each in the XML configuration file and define the affinity key via CacheKeyConfiguration (again in the xml file), considering that here affinity key is defined at IgniteConfiguration level, how do I specify which

Re: Affinity key & data rebalancing

2020-01-29 Thread ashishb888
As expected. Thank you Ilya for response. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Affinity key & data rebalancing

2020-01-22 Thread Ilya Kasnacheev
Hello! If you run out of data region, you will get IgniteOutOfMemoryException with probable data loss. You should make sure to not run out of data region. Regards, -- Ilya Kasnacheev ср, 22 янв. 2020 г. в 15:20, ashishb888 : > We have defined an affinity key e.g. someId for a cache. T

Affinity key & data rebalancing

2020-01-22 Thread ashishb888
We have defined an affinity key e.g. someId for a cache. This affinity key will be linked to a partition of the cache. Now this partition is held by 1 node. In case the data for the partition grows more than that node limit, then what will happen? BR, Ashish -- Sent from: http://apache-ignite

Re: Good metrics to assess affinity quality

2019-12-27 Thread steve.hostettler
Hello, my objective is to come up with a good affinity to optimise data distribution.To reduce local misses by improving co-location. My data structure is however legacy and quite complex and I do not have a clear candidate. Therefore, I would like to test several affinities and to measure what

Re: Good metrics to assess affinity quality

2019-12-26 Thread akurbanov
Hello Steve, Could you please provide a bit of clarification, what is exactly you are testing when you are talking about different affinities? What is exactly your target, data distribution or affinity function itself? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Good metrics to assess affinity quality

2019-12-25 Thread steve.hostettler
Hello Igniters, first let me wish you happy holidays. I am working to testing different affinities and I was wondering what would be good metrics to assess affinity quality. I have a near cache to deal with less than perfect affinities. I assume network traffic is a good one but is there

Re: Does affinity work with cache mode of LOCAL?

2019-11-27 Thread camer314
be heavy on read/write To try and improve the initial cache population from file(which can be millions of rows) I distribute a job to the cluster that each reads a piece of the file to get some sort of upload parallelization. I am using affinity keys so that the calculations only have to process

Re: Does affinity work with cache mode of LOCAL?

2019-11-27 Thread akurbanov
this key in while true — you will receive the values that were put locally no matter what was executed on other node. The concept of affinity is completely unapplicable in this case: you don't need a function that maps keys to nodes as LOCAL cache exists only on a single node. Could you plea

Re: Does affinity work with cache mode of LOCAL?

2019-11-27 Thread camer314
Actually the code is adding data to each nodes individual LOCAL cache, its just the affinity is not working as expected, all affinity jobs are run on the node which is invoking them (in my case the client node) rather than where the actual data exists. -- Sent from: http://apache-ignite-users

Re: Does affinity work with cache mode of LOCAL?

2019-11-27 Thread camer314
, when I send the affinity compute I notice those compute functions are actually sent to my client application and not the server nodes as happens in PARTITIONED mode. But my client app is ClientMode=TRUE so it has no data. If i change it to ClientMode=FALSE so that it participates in the data load

Does affinity work with cache mode of LOCAL?

2019-11-27 Thread camer314
I have been using affinity keys with a PARTITIONED cache and then use those keys to send computations to the nodes that have the data which all works as expected. I wanted to test LOCAL mode for performance but I found no calculations are now sent to the nodes. Is that expected behavior? How

Re: Affinity key for ignite tables

2019-10-03 Thread Denis Magda
Only one-to-one relation is possible for affinity collocation. Your Sales can be either collocated with Product or Customer. More details are here: https://www.gridgain.com/docs/8.7.6/developers-guide/data-modeling/affinity-collocation - Denis On Thu, Oct 3, 2019 at 7:31 AM Anton Kurbanov

Re: Affinity key for ignite tables

2019-10-03 Thread Anton Kurbanov
Hello, Ignite does map the entries to nodes in the following way: entry key > affinity key > partition > node You can control the first part of this mapping by defining an affinity key for the type. All the cache entries with same affinity key will be stored at the same node (aff

Affinity key for ignite tables

2019-10-03 Thread Muhammed Favas
HI, I have read through the collocating data in ignite node and it is possible to give affinity key in ignite tables using "AFFINITY_KEY" keywords. All the examples I have gone through was mention with simple two tables and it has one to one relationship. I still confused on how th

Re: possible ignite bug around binary marshaller and data affinity

2019-09-25 Thread Ilya Kasnacheev
Hello! I have trouble contemplating your reproducer, but I think it's related to the fact you are using QueryEntity. QueryEntity means that your annotations are not used. In this case you should use CacheKeyConfiguration to configure affinity for such caches: https://ignite.apache.org/rel

Re: Affinity collocation and node rebalancing

2019-08-08 Thread Denis Magda
Thanks for raising the questions! We'll update the documentation accordingly clarifying some tricky areas and scenarios. - Denis On Thu, Aug 1, 2019 at 1:33 PM lhendra wrote: > Denis, again this is really great - definitely something that I will take > into consideration during the schema desi

Re: Affinity collocation and node rebalancing

2019-08-01 Thread lhendra
Denis, again this is really great - definitely something that I will take into consideration during the schema design. Really appreciate your thoughtful replies and explanations! Lily -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Affinity collocation and node rebalancing

2019-08-01 Thread Denis Magda
st two cities of the size of NY and you have 2 nodes that consume too much space) you can come up with an alternate collocation scheme when for Residents living in NY you use a complex affinity key like (city_id, district_id) and with that help the district_id will also be involved in the data dis

Re: Affinity collocation and node rebalancing

2019-08-01 Thread lhendra
Denis, thank you for the detailed explanation One follow-up question, if you don't mind. Is there a limit to how many tables you can chain in the affinity collocation? We might have a use case where we want have 30 tables chained together in affinity collocation (not too many columns

Re: Affinity collocation and node rebalancing

2019-07-31 Thread Denis Magda
be moved to the new node. If to go in details, Ignite rebalances partitions. Once your cluster topology changes, the affinity function recalculates partitions distributions to make it even cluster-wide. As a result, some of the partitions will stay on existing nodes while the others will be moved

Affinity collocation and node rebalancing

2019-07-30 Thread lhendra
Hi, I'm totally new to Ignite and have been reading lots of documentation. I'm trying to find info on how node rebalancing is handled in relation to affinity collocation. There are 2 use cases that I want to confirm: (1) Create a new table with an affinity key to an existing table. (

Re: Affinity on non-key field

2019-07-30 Thread Stephen Darlington
No, and it’s not really possible. The problem is with something like IgniteCache.get(), the only information it has to find the value is the key. In that sense you’d be able to co-locate the data when you saved it to the cache, but you’d have no efficient way to find it again afterwards. Regard

Affinity on non-key field

2019-07-30 Thread Yohan Fernando
If you have a cache where you have the key as ObjectKey and value as ObjectValue, and the classes are like the following, is it possible in Ignite to have data co-location based on a non-key attribute? In this case on the private int c field in ObjectValue? I believe @AffinityKeyMapped annotatio

  1   2   3   4   5   6   >