Hi,
Do you have JIRA tickets for this change? I did not see this in the
conversation.
Andrey
7/8/2021 4:51 PM, Alexei Scherbakov пишет:
I assume DurableBackgroundTask will no longer use references to root ids.
The proposed solution looks good to me.
чт, 8 июл. 2021 г. в 10:28, Ivan Bessonov <bessonov...@gmail.com>:
Hi guys,
I see that the original message and its clarification is hard to consume.
The problem is that indexes are identified by their names. So, the
situation
that Kirill described is a valid one:
- you have index X for columns (a, b);
- you drop it;
- you create index X for columns (a, c).
This is a realistic scenario. If all of that happened during a single
checkpoint
interval and node failed.
Real issue here is the fact that we restore metastorage separately from
everything else.
This means that DurableBackgroundTask will start deleting X (a, c) if we're
not careful.
It all depends on the implementation. I'd suggest Alexeys solution with a
few tweaks:
- I'm not convinced that we need to log indexes creation, this is
excessive;
- how do you drop index for the metatree:
-- pick a unique name;
-- acquire checkpoint read lock;
-- create DurableBackgroundTask that will delete index with that unique
name;
-- create logical WAL record "drop index X";
-- "rename" link to the index in meta tree;
-- release checkpoint read lock;
-- start DurableBackgroundTask.
Recovery for "drop index X" should also create another
DurableBackgroundTask,
this way I think we'll be able to avoid any possible issues. If
DurableBackgroundTask
has a name of an unexisting index then that's fine, task will be completed
as
successful, that's a normal flow.
I hope everything's clear here, I'm ready to clarify more details about my
idea. Thank you!
ср, 7 июл. 2021 г. в 17:20, ткаленко кирилл <tkalkir...@yandex.ru>:
I'll try to explain:
We create indexes through InlineIndexFactory#createIndex, where at the
beginning we create the root page using QueryIndexDefinition#treeName, so
if we create the same index 2 times, then we will refer to the same root.
In order to make the deletion of the index independent, I propose to
replace the name of the root pages with an arbitrary name (other than
QueryIndexDefinition#treeName, for example UUID.toString()), in order to
avoid the problems of a node crash, I propose to fix this in a logical
record.
After we change the name of the root page to delete the index, we can
easily delete it and rebuild it (if the user does it) in parallel.
If we consider your proposal, then the creation and deletion of the index
will go in parallel and they will look at the same roots.
You can see how the renaming will take place here:
https://github.com/apache/ignite/pull/9223
07.07.2021, 10:28, "Alexei Scherbakov" <alexey.scherbak...@gmail.com>:
вт, 6 июл. 2021 г. в 15:57, ткаленко кирилл <tkalkir...@yandex.ru>:
>> Can you clarify what it means to rename root index trees ?
Replacing
org.apache.ignite.internal.processors.cache.persistence.IndexStorageImpl.IndexItem,
changing IndexItem#idxName, but keeping fIndexItem#pageId.
Changing to what ? Some temporary name ? Can you give a detailed step
by
step description of the algorithm ?
Suggested solution is not suitable for the situation: add index ->
drop
index -> add an index. We can start deleting the last added index.
How can we do that, give me an example ?
From my understanding, the suggested solution should work ok for any
number
of create/drop sequences.
06.07.2021, 14:00, "Alexei Scherbakov" <alexey.scherbak...@gmail.com
:
> Can you clarify what it means to rename root index trees ?
>
> The simple solution which immediately comes to me is
>
> 1) write logical record on index creation - on reading it create an
index
> during logical recovery
> 2) write logical record on index deletion - on reading it delete an
index
> during logical recovery and start background clearing task with
real
root
> pages.
>
> Will it work for you ?
>
> вт, 6 июл. 2021 г. в 12:27, ткаленко кирилл <tkalkir...@yandex.ru
:
>
>> Hello everyone!
>>
>> Currently, dropping indexes consists of the following steps (based
on
>> SchemaAbstractDiscoveryMessage's):
>>
>> Step 1: Removing the index from the SQL engine and starting
>> DurableBackgroundCleanupIndexTreeTask, which removes the index
trees
in the
>> background;
>> Step 1.1: DurableBackgroundCleanupIndexTreeTask is added to the
>> metaStorage and removed after successful completion at the next
checkpoint.
>>
>> Step 2: Removing the index from the cache configuration and
persist
it.
>>
>> Problems:
>>
>> 1)We add and immediately delete the index, a checkpoint does not
happen
>> and the node crashes, after restarting
>> DurableBackgroundCleanupIndexTreeTask will not be able to complete
and
will
>> periodically restart due to the fact that it saves
>> DurableBackgroundCleanupIndexTreeTask#rootPages (root pages of
index
trees)
>> that have not appeared;
>>
>> 2)After adding a DurableBackgroundCleanupIndexTreeTask node
crashes,
after
>> restarting the node, the task will clean the index trees and there
will be
>> errors when using the index;
>>
>> 3)etc.
>>
>> Suggested solution:
>>
>> Rename the root index trees and write about this with a logical
entry
in
>> the WAL and do this at the first start of
>> DurableBackgroundCleanupIndexTreeTask.
>> Thus, if we find the renamed root pages in task 1, we can clear
the
index
>> trees to the end, otherwise the task can be completed.
>> Also, if we find that rename pages are present, and the step 2 has
not
>> been completed, then we can start rebuilding the indexes.
>>
>> WDYT?
>
> --
>
> Best regards,
> Alexei Scherbakov
--
Best regards,
Alexei Scherbakov
--
Sincerely yours,
Ivan Bessonov
