[
https://issues.apache.org/jira/browse/SOLR-7187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815771#comment-17815771
]
Michael Gibney commented on SOLR-7187:
--------------------------------------
This is still an issue; I encountered it while trying to make sense of "ulog"
vs. "tlog" for SOLR-16962. The issue is that shared storage has to be scoped by
collection and and core, which is done
[here|https://github.com/apache/solr/blob/aac01255dba559df6324ac9e294afe05169c0597/solr/modules/hdfs/src/java/org/apache/solr/hdfs/HdfsDirectoryFactory.java#L468-L472]
in {{HdfsDirectoryFactory.getDataHome(CoreDescriptor)}}. So the data home for
a given core implicitly creates this extra hierarchy, with no references for
cleanup.
Idk about back-compat, but one way to handle this would be to construct the
path as a single path element, i.e. {{[collection]-[core_nodeN]-data}}. (This
highlights the fact that the "data" component is superfluous -- we could
currently simply make the {{core_nodeN}} dir be the data dir, since dataDir is
the only thing it's used for). This might pose a practical problem in terms of
having a potentially large number of "dataDir" entries in a single top-level
hdfs-home directory. If we're stuck with creating hierarchy (either of the
{{collection/core_nodeN}} type, or some other approach (because of the "number
of directories" issue), then tbh I think we might do best to just take the
naive approach of conditionally trying to delete parent directories. Could
probably build some safeguards in that would make it safer, if still not
elegant; but I don't really see other good options.
I'm not planning to do anything about this atm, but I wanted to drop these
thoughts in here in case they're helpful. (I was investigating this hoping that
I could find a good way to clean up hdfs ulog/tlog paths outside of the data
dir, but it looks like that would be a challenge for similar reasons. In fact,
configuring a custom ulog/tlog path within hdfs makes no sense whatsoever imo,
and I plan to propose disallowing it in SOLR-16962).
> SolrCloud does not fully clean collection after delete
> ------------------------------------------------------
>
> Key: SOLR-7187
> URL: https://issues.apache.org/jira/browse/SOLR-7187
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.10.3
> Reporter: Mike Drob
> Priority: Major
> Attachments: log.out.gz
>
>
> When I attempt to delete a collection using
> {{/admin/collections?action=DELETE&name=collection1}} if I go into HDFS I
> will still see remnants from the collection. No files, but empty directories
> stick around.
> {noformat}
> [root@solr1 ~]# sudo -u hdfs hdfs dfs -ls -R /solr/collection1
> drwxr-xr-x - solr solr 0 2015-03-03 15:42
> /solr/collection1/core_node1
> drwxr-xr-x - solr solr 0 2015-03-03 15:42
> /solr/collection1/core_node2
> drwxr-xr-x - solr solr 0 2015-03-03 15:42
> /solr/collection1/core_node3
> drwxr-xr-x - solr solr 0 2015-03-03 15:42
> /solr/collection1/core_node4
> drwxr-xr-x - solr solr 0 2015-03-03 15:42
> /solr/collection1/core_node5
> drwxr-xr-x - solr solr 0 2015-03-03 15:42
> /solr/collection1/core_node6
> {noformat}
> (Edit: I had the wrong log portion here originally)
> In the logs, after deleting all the data, I see:
> {noformat}
> 2015-03-03 16:15:14,762 INFO org.apache.solr.servlet.SolrDispatchFilter:
> [admin] webapp=null path=/admin/cores
> params={deleteInstanceDir=true&action=UNLOAD&core=collection1_shard5_replica1&wt=javabin&qt=/admin/cores&deleteDataDir=true&version=2}
> status=0 QTime=362
> 2015-03-03 16:15:14,787 INFO org.apache.solr.common.cloud.ZkStateReader: A
> cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> 2015-03-03 16:15:14,854 INFO org.apache.solr.cloud.DistributedQueue:
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> 2015-03-03 16:15:14,879 INFO org.apache.solr.cloud.DistributedQueue:
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> 2015-03-03 16:15:14,896 INFO org.apache.solr.cloud.DistributedQueue:
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> 2015-03-03 16:15:14,920 INFO org.apache.solr.cloud.DistributedQueue:
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> 2015-03-03 16:15:15,151 INFO org.apache.solr.cloud.DistributedQueue:
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> 2015-03-03 16:15:15,170 INFO org.apache.solr.cloud.DistributedQueue:
> LatchChildWatcher fired on path: /overseer/queue state: SyncConnected type
> NodeChildrenChanged
> 2015-03-03 16:15:15,279 INFO org.apache.solr.common.cloud.ZkStateReader: A
> cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged
> path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
> 2015-03-03 16:15:15,546 INFO org.apache.solr.cloud.DistributedQueue:
> LatchChildWatcher fired on path:
> /overseer/collection-queue-work/qnr-0000000016 state: SyncConnected type
> NodeDataChanged
> 2015-03-03 16:15:15,562 INFO org.apache.solr.cloud.DistributedQueue:
> LatchChildWatcher fired on path: /overseer/collection-queue-work state:
> SyncConnected type NodeChildrenChanged
> 2015-03-03 16:15:15,562 INFO
> org.apache.solr.cloud.OverseerCollectionProcessor: Overseer Collection
> Processor: Message id:/overseer/collection-queue-work/qn-0000000016 complete,
> response:{success={solr1.example.com:8983_solr={responseHeader={status=0,QTime=207}},solr1.example.com:8983_solr={responseHeader={status=0,QTime=243}},solr1.example.com:8983_solr={responseHeader={status=0,QTime=243}},solr1.example.com:8983_solr={responseHeader={status=0,QTime=342}},solr1.example.com:8983_solr={responseHeader={status=0,QTime=346}},solr1.example.com:8983_solr={responseHeader={status=0,QTime=362}}}}
> {noformat}
> This might be related to SOLR-5023, but I'm not sure.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org
For additional commands, e-mail: issues-h...@solr.apache.org
