Bob Proulx wrote: > My plan is that tomorrow, Thursday, late morning I am going to start > walking through the systems and moving them to the new IP addresses. > This will be at least somewhat disruptive as in some cases four > systems must all be up and online simultaneously in order to function.
Today was a busy day! The systems except for frontend0 all have IPv6 addresses now. frontend-dev was previously migrated. nfs1 is migrated. internal0 now allows connections from both IPv4 and IPv6. download0 has been mostly migrated but is still using the old IPv4 connection to internal0. I have been rolling the systems from the hand generated iptables over to Shorewall as I have been going because of the protection Shorewall provides with "safe-restart" and other helpfulness. > Hopefully things will be concluded by early afternoon. Hopefully. Hahaha! It's always easy to be optistic before really getting into it. :-) It's just a very tedious and time consuming process to walk through everything. Progress is being made. We will get there. It just takes a little time. Two big snafus caught me. The worst being that we have a fragility I didn't realize existed. For vcs0 and download0 ssh access is through ssh keys through a custom ssh key configuration. AuthorizedKeysCommand /root/bin/sv_get_authorized_keys Where sv_get_authorized_keys contacts internal0 for the account database. A fragility is that if for whatever reason it cannot communicate with internal0 the old script installed on download0 would hang and block all login access including root login access. Turns out I had rewritten that script a couple of years ago and it was installed on vcs0 but not yet installed on download0. And needless to say during the process it wasn't able to connect to internal0 at which time I could no longer log into the system. Oops. I had to get the FSF admins to mount the image and neuter that file so I could log in again until I could figure it out. This needs improvement so that when the connection to the database system is impossible that root can still log into the system. That's currently not happening. Plus it does not work using a raw IPv6 address and requires a name. That is less than optimal too. Then perhaps more serious of a problem was that libnss-mysql is used for the account database. I rolled it from the IPv4 addresses to the new IPv6 addresses so that then I could move the IPv4 ones. This seemed to work. Initially. But for some reason does not work long term. It very quickly complains that connections to internal0 are refused due to too many database connections. internal0 is configured for 250 connections and that has always been sufficient previously. The root, root cause is not yet known. It is weird that using the IPv6 address instead of the IPv4 would cause a too many connection error but seemingly that was the behavior. I switched it back to IPv4 so that we could all go home and get some dinner since it was getting late by that time. I'll note that editing files when there are hundreds of wall messages thrown to logins complaining from libnss about too many connections is interesting. (Emacs tramp remote file editing works around that problem. FTW!) There was also something funky with the download0 http /release mirror redirector. It also failed the regression test when the database connection was IPv6 but magically came back to working order when returning to IPv4. The big systems left are vcs0 and the backend NFS storage on old vcs and olddownload. As with everything it is the connection pair of systems that need to move together. Lots still left to do for tomorrow. Bob
