Pier, Mikal: I agree, I think the juices are flowing. See below
Tom ----- Original Message ----- From: "Mika Goeckel" <[EMAIL PROTECTED]> To: "Tomcat Developers List" <[EMAIL PROTECTED]> Sent: Tuesday, November 13, 2001 1:37 AM Subject: Re: Tomcat: Distributed Session Management revisited | Pier, Tom, | | cool, the discussion is starting to become interesting. :-) | | comments below: | | ----- Original Message ----- | From: "Pier Fumagalli" <[EMAIL PROTECTED]> | To: "Tomcat Developers List" <[EMAIL PROTECTED]> | Sent: Tuesday, November 13, 2001 3:04 AM | Subject: Re: Tomcat: Distributed Session Management revisited | | | > On 13/11/2001 12:54 am, "Tom Drake" <[EMAIL PROTECTED]> wrote: | > | > > Mika: | > > | > > Thanks for the reply. Here's some more thoughts on this subject. | > > | > > The primary problem that I see with the collaborative method | > > (e.g. extending the multicast solution) is | > > that all sessions will have to be sent to all cluster nodes. The | > > number session updates that have to travel 'on the wire' is in | > > relation to the number of nodes in the cluster. | > | > Linear growth, that's the best we can aim for... | > | > > Further more, when a new tomcat is brought on-line, it must | > > somehow retrieve a copy of all active sessions from somewhere. | > > There is nothing in place for this currently. Using multicast | > > is problematic. If a multicast request is made then all other nodes | > > would respond with all sessions. So, some other approach would | > > need to be taken which would result in two protocols being used | > > to make this feature work. This seems too complicated. | > | > Not "that" complicated. Most of the work on elective processes has been | done | > already in the scope of other projects, so, we would only need to adapt it | > to our scope... | | I agree with Pier, in my view that's separating the "application layer" | (content) from the transportation control layer (where, how). | Point taken, however, I strongly believe in keeping things simple. I'd not want to introduce extra communication channels unless there there is a REALLY good reason to do so. | > | > > --------------------------------------- | > > Consider this scenario: | > > | > > A user establishes a session on node 1 (of a 10 node cluster), | > > Tomcat would create a new session and transmit it to the | > > multicast port, which would then transmit 10 copies of this | > > session (1 to each cluster node). | > > Now suppose that the next request from this user is sent to | > > node 2, which causes an update to the session to occur. Again | > > 11 copies of the Session are transferred. | > > [...] | > > NOTE: remember this is UDP traffic. The more packets that | > > fly around, the greater the likely-hood of dropping packets. | > > Dropped packets in this case means that some tomcat | > > instances may have stale (or no) data for a given session. | > | > Indeed... Quite huge... | | Yes, multicast udp should only be used to autoconfigure the cluster (who's | there, who's taking sessions etc.), which should also be configurable for | non-multicast-environments. In that case lists of adresses are used to | select who's the next to take over. In fact, if all node's are holding | information about the peers, we don't need to have long lists. An upcoming | node would need only one already configured node to ask the cluster's spread | via TCP. It's join could be communicated via daisy-chain. (one message per | member is linear). | This is certainly a reasonable approach. However, we could use the SNMP approach to auto-discovery instead of multicast. This would simplify configuration (e.g. no need for multicast port) and achieve the goal using a fairly standard way. Having said that. I'm not lobbying for this approach, simply tossing into the ether. One last comment about the 'anything-but simple network management protocol'. Limited SNMP mib implementation support within tomcat / session server may be helpful in those environments that depend heavily on SNMP management (read telco's). Again, I really don't want to take a left turn here, it just made me think about SNMP. | > | > > ------------------------------------------ | > > With a centralized session manager the following traffic would | > > occur instead: | > > | > > node1 sends new session to server manager | > > node 2 requests the given (session id) session from the server manager | > > manager sends a copy of the session to node 2 | > > node 2 updates the session and sends it back to the manager. | > > manager sends the 'invalidateSession(sessionId)' method in each of | nodes. | > > (note: invalidateSession only contains the value of 'SessionId' + any | > > additional | > > RMI overhead. This is far smaller than a complete Session object) | > > | > > The number of session copies sent as the result of an update is 2. | > > This number does not depend or vary based on the number of nodes. | > > | > > Now, let's add to the story. Let's say that Tomcat is smart enough to | cache | > > Session objects in it's memory space. Once Tomcat gets its hands on a | > > 'Session' | > > it keeps it until it becomes 'too old' or an | 'invalidateSession(sessionId)' | > > message is | > > received from the remote Session Manager. This could cut down the the | number | > > of transfers of Session data from 2 to somewhere between 1 and 2. | > | > Yes, but in this case, we don't have redundancy of sessions... So, if the | > Tomcat which has the session dies, the whole session dies with him... | > See my previous email on this. This doesn't eliminate redundancy of sessions. All session servers keep an up-to-date copy of all sessions. | > > ----------------------------------------------------- | > > On Redundant Session Managers. | > > | > > There are a couple ways to achieve this. One way is to place two Session | > > Managers in the network. One of them is the 'active' one, the other one | could | > > simply register itself as a client of the 'active' server. As a client, | it can | > > obtain copies of all new and changed sessions from the active server. If | for | > > some reason the active server needs to be brought down, it will send a | message | > > to all of it's clients (including the 'dormant' session manager) | indicating | > > that it's shutting down. The clients could, on receipt of this message, | > > connect to the 'next' session server (in their pre-configured list of | > > servers). The clients could simply carry on with the new server. | > | > Indeed... | > | > > If the active server simply goes off the air for some mysterious reason. | The | > > clients would get a RemoteException the next time they tried to talk to | the | > > server. This would be their clue to 'cut-over' to the other server (as | > > described above). | > | > But how would they know where the sessions ended up???? All session managers keep a copy of all sessions. So, it doesn't matter which server a client talks to. | > | > > Last point. Sending Session delta's instead of the entire Session: | > > | > > This should be doable. The main thing that we care about are Session | > > attributes which are changed by the application. It's up to the | > > web-application to replace these values into the Session if their | contents | > > change. This is enough for us to be able to track which attributes have | > > actually changed. | > | > This can actually be done if we consider every operation on a session | > (adding/replacing/removing an attribute) and atomic operation.... | > | > Let's see if I can complicate things a little bit :) (Love doing that). | > | > Let's imagine to have a pool of session managers (SA, SB, SC...) and a | pool | > of servlet containers (T1, T2, T3...). | > | > The first thing we want to do is bring up our session managers. Once we | > start them SA, SB, SC and SD are available to accept sessions. | > | > Then we start our servlet containers T1, T2, T3 and T4. When a request | comes | > in in any of the servlet containers, the servlet container simply | broadcasts | > a message saying "who can hold a session for me?"? All four managers will | > reply to that request, and the servlet manager can "order" them. For | > example, if we want a redundancy level of 2, the container might choose SB | > as the "primary" session manager, and SA as the "replica 1" session | manager; | > if we want a redundancy level of 3, the container might choose SD as | > "primary", SA as "replica 1" and SB as "replica 2". | > | > The information about "who is primary" and "who is replica X" is stored | > within the session manager itself. | > | > When one of the servlet containers needs to read or write from a session, | he | > will broadcast (again) the message "who holds this session?", of course, | all | > session managers holding (primary or replica) that session, will reply | with | > their "status" (primary, or replica #), and then the servlet container | will | > persist the data in -first of all- the primary session manager, -then- in | > all the replicas, and at the end return control to the caller (the thread | > which called "setAttribute/getAttribute"). | > | > What happens if one of the session managers goes down? That simply the | > servlet container will notice that something is going wrong, because if | > configured with a "replica factor" of 3, he gets only 2 responses to | "who's | > holding this session?", we know for sure that one of the replicas (or the | > master) has gone down, so, simply, we can "elect" one of the replicas as | > "primary" (if the primary has gone down), and/or broadcast a message | saying | > "who can be replica for this session?"... The session is then persisted in | | > all three places (the two old ones, plus the new one), and the thing goes | > on... | > | > What does it gives us? A lot of flexibility in terms that only a little | data | > is broadcasted (messages such as "who can hold this session?" or "who has | > this session?" or "who can be replica for this session"), so we avoid | > problems with UDP, then we have a sub-linear growth in a way that the | > traffic over the network is only (N*(sessiondata+overhead)) where N is the | > replica factor, and the administrator is free to trade his own data safety | > (more replicas, more traffic, more redundancy), with speed (less replicas, | > less traffic, less redundancy)... | > | > We don't have a single point of failure (whohoooo!), we don't need to | > replicate sessions with linear growth on N (where N is the number of | session | > managers), and we get load balancing of sessions for free... | > | > The only problem is that we need to use multicast, but that shouldn't be a | > big issue... | > | | Yeah, ok, consider as follows: | | We don't differenciate between servers beeing SessionManager or | ServletContainer, instead we act as hermaphrodite. When a new session is | about to be created | (FancyDistributedFailOverSaveSessionManager.createSession()), it asks "Who's | holding that stuff". If he doesn't get an answer, he empowers himself as the | primary SessionManager and elects n secondary nodes. (Ahhh yes, as well when | a container calls findSession()) | Only the primary SC is allowed to change the session, secondaries only | listen to changes. No lock distribution, no transaction problem so far. If we become more granular, such that individual session attribute changes are communicated independantly (rather than a complete copy of the session) our need for locking may diminish. However, I'm still not convinced about the need for locking to begin with. The only 'problem' I see with Tomcat acting as the distributed session manager is that it makes each tomcat do a fair amount of extra work (e.g. sending invalidate session / update session attribute messages to all other nodes). With (a pool of) separate session manager(s), all that extra work is done by another process, possibly running on another CPU somewhere. Tomcat is free to simply communicate session changes with one other party (the session manager it is connected to). Having said that, Tomcat could certainly host the session server functionality, at some expense. Multiple session managers could be set up in a daisy chain as you mentioned earlier. In this way each session manager and each session client (tomcat) only needs to be preconfigured with a single session managers location (unless we use some type of auto-discovery mechanism). The session managers can share their knowledge of each other such that all servers should be able to know where all other servers are. In addition, this information can be made available to each of the clients. In this way, you could dynamically add a new fail-over server to the network, only telling it about one other server. When it comes up, it would introduce itself to the other server, obtain a list of all other servers and announce itself to them too. It would also need to obtain copies of all 'current' sessions from another server. As each session server receives this notification of a new server, it could simply tell all of it's clients of the arrival of a new server to the network. | If createsession() get's an answer, we need to decide what to do. On one | hand, we could get the primary status over from the original owner (how to | do that savely is another question). The original owner could become a | secondary or just drop the session at all. On the other hand, we could pass | over the processing to the other node. Why transfer all that data to us, if | there is a guy who has all that stuff. | The latter would restrict us somewhat in load-balancing, but would reduce | the traffic enormously and get around locks and transaction problems. Is | there a way to do that? I.e. tell mod_webapp/mod_jk "Use that node instead"? | Or just proxying to the other node (which would result in passing over the | response twice in our network). | If the createSession() only get's answers from secondaries, we suddenly | realise that we lost a beloved member of our group. Then we get the session | from a secondary and make ourself the new numbero uno. (Or pass over to a | secondary to become the new leader of the pack). Again no locking problems. | But we may be in the middle of a transaction. I can't see a way yet, how to | prevent that, as we can't know when a node falls over. I think taking over | is better because we have to replicate to another node anyway, so why not | ourself? | | Mika | :wq | | | | -- | To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]> | For additional commands, e-mail: <mailto:[EMAIL PROTECTED]> | | | -- To unsubscribe, e-mail: <mailto:[EMAIL PROTECTED]> For additional commands, e-mail: <mailto:[EMAIL PROTECTED]>
