Hi, On Tue, Sep 09, 2008 at 12:04:27AM +0300, Sergiu Ivanov wrote:
> On Sun Sep 8, there was a discussion on #hurd between antrik and me > (scolobb), in which several important points were spoken out. I will > try to enumerate these points below: Actually, many of the listed things were not firm decisions, but rough ideas requireing further consideration. As these points are all quite intertwined, rather than commenting them individually, I will bring up a more fundamental question, which has been hovering in my mind during the later part of that discussion, and which I have considered further since: A lot of these design considerations hinge on the question how to handle recursion, i.e. what behaviour is desired if we have a dynamic translator, and do further magic lookups. When doing magic lookups on a dynamic translator exporting a directory tree, it's rather obvious that we want further magic lookups to work: Say we use Flavio's unzip translator to look into an OpenOffice document, we might want to use an XML translator on the individual files in there. Insert any number of other examples. The trickier question is what to do about lookups a dynamic translator does on the underlying filesystem. (This was actually my main motivation for suggesting that shadow translators would do lookups on the nsmux filesystem -- this way, it should work without any special handling.) Logically I think we do want that as well: If we use nsmux, it should work just as if we used a filesystem that introduces namespace-based translator selection direcly in the standard lookup mechanism, i.e. we really want *all* lookups to have magic handling. OTOH, I can't think of any specific case of a translator doing lookups on the underlying filesystem, using directly or indirectly user-provided (and thus potentially magic) file names. I'm not saying use cases do not exist; but they are probably rare, so I'm not sure how much we should care... Though definitely I feel some uneasiness about just ignoring the matter. BTW, I'm not quite sure now that it's really useful to have dynamic translators continue working on shadow nodes when nsmux goes away. nsmux is effectively serving as the main filesystem on which the dynamic translators are invoked; and in other situations, translators do not have any protection against the underlying filesystem going away either... If they don't need the underlying filesystem, they will be fine with being orphaned; otherwise, they produce failures. Considering that external shadow translators do not turn out as simple as I had first hoped either, I'm not sure anymore whether it is a useful idea at all. I haven't abandoned the idea alltogether yet, but probably we shouldn't spend to much thought on it right now... There is another point that popped up during these considerations: What if a client for some reason does multiple lookups on the same magic filename? With the current approach, a new dynamic translator is created every time. While this should pose no semantic problems, if all translators are well designed (don't have any temporary state not reflected in shared storage), it could be quite inefficient in extreme cases. I wonder thus whether it wouldn't be advisable to share dynamic translators resulting from equivalent lookups. But this would again pose problems with badly designed translators, that behave differently with a single instance shared between clients than individual instances for each client... OTOH, I already found other situations in the past where we might have a great many instances of the same translator invoked in parallel or successively -- so I guess this is really an issue that needs to be addressed on a more generic level sooner or later... In any case, I think this is an issue worth putting on a ToDo list, but we probably shouldn't bother about it much right now. As for the remaining points: > * The nodes with translator stacks on them should not go away until > there are clients using these translator stacks. > > * No specific node caching should be done in nsmux, but rather the > nodes should be created on demand and destroyed, when they are not > used already (the destruction should happen precisely when the port to > the topmost translator in the stack will receive a no_senders > notification) > > * In the current implementation of nsmux, the shadow nodes are > maintained by nsmux, therefore all the translator stacks should be > shut down *before* nsmux goes away when requested, lest the > translators should become orphaned. > > * A netfs node cannot go away when all references to it are gone, but > there are senders using the port to the topmost translator in the > translator stack sitting on this node. To make this possible, there > should be a way to keep this node alive, probably in a more explicit > fashion, than a node cache might have done it. [...] > * A shadow node may go away in case the real filesystem node it is > mirroring goes away. I don't really understand all of these points: some of them seem to contradict what I actually said, so I'm not sure there hasn't been some serious misunderstanding in the original discussion... Let me restate the requirements in my own words: The dynamic translators are created upon client request, and should thus never go away, unless they are not needed by the client anymore. The only exception is the underlying node going away for a translator that actually needs the underlying node, in which case it's best for the translator also to go away immediately, so the clients immediately get the information, just as if they had been using the underlying node directly. While we can make sure these policies are met by any translators developed as part of the project, we have no control of external translators we launch of course -- they might stay around longer than necessary... However, all normal translators will just go away after a couple of minutes, if there are no outstanding client connections. The delay of several minutes is actually not necessary for dynamic translators, as presently they are never reused, so once the original client drops the reference, we *know* the translator won't be used anymore. It might be useful in this case to tell the translator explicitely to go away, so it won't waste resources sitting around -- but I don't think this is very important, so the ability or non-ability to do so probably should not affect any other design decisions... All in all, as a first approximation, I think we should *never* explicitely destroy dynamic translators -- anything else is just optimization. > * In the current implementation of nsmux the filter will ask for the > underlying port to be opened with O_NOTRANS. However, the port the > filter will get will point to a netfs node, not to the real filesystem > node. It means, that when the filter will try to trace the translator > stack, it will find itself only (in the case of 'file,,-u', for > instance). It will not see the static translator stack on the real > filesystem. (This problem is, probably, only specific to the current > implementation of nsmux). Not sure what you mean saying that this problem is specific to the current implementation... It is obviously wrong -- it must (and can) be fixed, no matter what the implementation. > Right now the following questions have occurred to me: > > * Why does a shadow translator have to be invisible on the translator > stack? I don't think a filter will ever want to ignore a shadow > translator, since it is pretty useless. And nobody, save for a filter, > will possibly ever require the underlying node opened in O_NOTRANS > fashion. I'm not sure about the possibility of cases where it is actually necessary for the filter (or any other translator using this mechanism) not to be aware of the fact that it sits on a shadow node... But that is not really the point. It's not as much about hiding the shadow translator, as about showing the original translator stack *instead* -- which obviously requires special handling in the parent filesystem (nsmux). > * Suppose nsmux is asked for 'file,,x'. It will set the shadow > translator on a copy of its root node and will tell the shadow > translator to mirror the file 'file'. Then it will set translator 'x' > on top of the shadow translator. Do I understand everything right? Yes, that *was* the idea... -antrik-
