[ https://issues.apache.org/jira/browse/IGNITE-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Roman Puchkovskiy updated IGNITE-21753: --------------------------------------- Description: To enable Metastorage compaction (that is, removal of overwritten revisions that are not needed to anyone any more), there is IGNITE-19417, but it might take some time to define the corresponding LWM's rules. As a crutch, we might implement a simplified procedure for some keys. For instance, a big percentage of garbage is created by a constantly overwritten key related to lease tracking. If a key is never scanned (either distributively, via range()/prefix(), or locally, via xxxLocally() returning a Cursor) and only its latest version is always accessed (this seems to be true for the above mentioned lease-related key), we can just do compactKey() for such a key after any update involving it. We might also need to reimplement getAndPut() to read before doing a put as the current implementation requires 2 latest versions to be stored (because it makes a put first and only then reads using the 'before the put' revision). The crutch might be implemented by whitelisting all such 'special' keys in the KV storage backing the Metastorage to make it opaque for the API, or an API might be temporarily changed. Anyway, a detailed design is needed. was: To enable Metastorage compaction (that is, removal of overwritten revisions that are not needed to anyone any more), there is IGNITE-19417, but it might take some time to define the corresponding LWM's rules. As a crutch, we might implement a simplified procedure for some keys. For instance, a big percentage of garbage is created by a constantly overwritten key related to lease tracking. If a key is never scanned (either distributively, via range()/prefix(), or locally, via xxxLocally() returning a Cursor) and only its latest version is always accessed (this seems to be true for the above mentioned lease-related key), we can just do compactKey() for such a key after any update involving it. We might also need to reimplement getAndPut() to read before doing a put as the current implementation requires 2 latest versions to be stored (because it makes a put first and only then reads using the 'before the put' revision). The crutch might be implemented by whitelisting all such 'special' keys in the KV storage backing the Metastorage to make it opaque for the API, or an API might be temporarily changed. > Implement a hacky self-compaction for selected Metastorage keys > --------------------------------------------------------------- > > Key: IGNITE-21753 > URL: https://issues.apache.org/jira/browse/IGNITE-21753 > Project: Ignite > Issue Type: Improvement > Reporter: Roman Puchkovskiy > Priority: Major > Labels: ignite-3 > > To enable Metastorage compaction (that is, removal of overwritten revisions > that are not needed to anyone any more), there is IGNITE-19417, but it might > take some time to define the corresponding LWM's rules. > As a crutch, we might implement a simplified procedure for some keys. For > instance, a big percentage of garbage is created by a constantly overwritten > key related to lease tracking. > If a key is never scanned (either distributively, via range()/prefix(), or > locally, via xxxLocally() returning a Cursor) and only its latest version is > always accessed (this seems to be true for the above mentioned lease-related > key), we can just do compactKey() for such a key after any update involving > it. > We might also need to reimplement getAndPut() to read before doing a put as > the current implementation requires 2 latest versions to be stored (because > it makes a put first and only then reads using the 'before the put' revision). > The crutch might be implemented by whitelisting all such 'special' keys in > the KV storage backing the Metastorage to make it opaque for the API, or an > API might be temporarily changed. > Anyway, a detailed design is needed. -- This message was sent by Atlassian Jira (v8.20.10#820010)