I’ve created Phabricator task T242632 to track this. Please coordinate there as well for anyone with information and time.
Brooke Storm Senior SRE Wikimedia Cloud Services bst...@wikimedia.org <mailto:bst...@wikimedia.org> IRC: bstorm_ > On Jan 13, 2020, at 9:10 AM, Brooke Storm <bst...@wikimedia.org> wrote: > > I suspect this will affect containers that run Debian Buster packages. I see > php7.3 and python3.7. > I’d suggest not even restarting web services on those runtimes until we have > it fixed. For anyone who has done so, we are working on it. Any logs could > be helpful. > > Brooke Storm > Senior SRE > Wikimedia Cloud Services > bst...@wikimedia.org <mailto:bst...@wikimedia.org> > IRC: bstorm_ > >> On Jan 13, 2020, at 9:06 AM, Brooke Storm <bst...@wikimedia.org >> <mailto:bst...@wikimedia.org>> wrote: >> >> I see a problem with at least one container image (which has nothing to do >> with the new cluster, I can see it on the old cluster as well). It looks >> like I’m going to be trying to fix that now. >> (Magnus, this is probably what you are seeing as well). >> >> Brooke Storm >> Senior SRE >> Wikimedia Cloud Services >> bst...@wikimedia.org <mailto:bst...@wikimedia.org> >> IRC: bstorm_ >> >>> On Jan 13, 2020, at 8:47 AM, Magnus Manske via Cloud >>> <cloud@lists.wikimedia.org <mailto:cloud@lists.wikimedia.org>> wrote: >>> >>> I switched a few "big ones" successfully, but ran into one that doesn't >>> work: >>> >>> glamtools https://tools.wmflabs.org/glamtools/ >>> <https://tools.wmflabs.org/glamtools/> is 503 but `webservice status` says >>> "Your webservice of type php7.3 is running". >>> >>> On Thu, Jan 9, 2020 at 9:58 PM Bryan Davis <bd...@wikimedia.org >>> <mailto:bd...@wikimedia.org>> wrote: >>> I am happy to announce that a new and improved Kubernetes cluster is >>> now available for use by beta testers on an opt-in basis. A page has >>> been created on Wikitech [0] outlining the self-service migration >>> process. >>> >>> Timeline: >>> * 2020-01-09: 2020 Kubernetes cluster available for beta testers on an >>> opt-in basis >>> * 2020-01-23: 2020 Kubernetes cluster general availability for >>> migration on an opt-in basis >>> * 2020-02-10: Automatic migration of remaining workloads from 2016 >>> cluster to 2020 cluster by Toolforge admins >>> >>> This new cluster has been a work in progress for more than a year >>> within the Wikimedia Cloud Services team, and a top priority project >>> for the past six months. About 35 tools, including >>> https://tools.wmflabs.org/admin/ <https://tools.wmflabs.org/admin/>, are >>> currently running on what we are >>> calling the "2020 Kubernetes cluster". This new cluster is running >>> Kubernetes v1.15.6 and Docker 19.03.4. It is also using a newer >>> authentication and authorization method (RBAC), a new ingress routing >>> service, and a different method of integrating with the Developer >>> account LDAP service. We have built a new tool [1] which makes the >>> state of the Kubernetes cluster more transparent and on par with the >>> information that we already expose for the grid engine cluster [2]. >>> >>> With a significant number of tools managed by Toolforge administrators >>> already migrated to the new cluster, we are fairly confident that the >>> basic features used by most Kubernetes tools are covered. It is likely >>> that a few outlying issues remain to be found as more tools move, but >>> we have confidence that we can address them quickly. This has led us >>> to propose a fairly short period of voluntary beta testing, followed >>> by a short general availability opt-in migration period, and finally a >>> complete migration of all remaining tools which will be done by the >>> Toolforge administration team for anyone who has not migrated their >>> self. >>> >>> Please help with beta testing if you have some time and are willing to >>> get help on irc, Phabricator, and the cloud@lists.wikimedia.org >>> <mailto:cloud@lists.wikimedia.org> >>> mailing list for early adopter issues you may encounter. >>> >>> I want to publicly praise Brooke Storm and Arturo Borrero González for >>> the hours that they have put into reading docs, building proof of >>> concept clusters, and improving automation and processes to make the >>> 2020 Kubernetes cluster possible. The Toolforge community can look >>> forward to more frequent and less disruptive software upgrades in this >>> cluster as a direct result of this work. We have some other feature >>> improvements in planning now that I think you will all be excited to >>> see and use later this year! >>> >>> [0]: >>> https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration >>> <https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration> >>> [1]: https://tools.wmflabs.org/k8s-status/ >>> <https://tools.wmflabs.org/k8s-status/> >>> [2]: https://tools.wmflabs.org/sge-status/ >>> <https://tools.wmflabs.org/sge-status/> >>> >>> Bryan (on behalf of the Toolforge admins and the Cloud Services team) >>> -- >>> Bryan Davis Technical Engagement Wikimedia Foundation >>> Principal Software Engineer Boise, ID USA >>> [[m:User:BDavis_(WMF)]] irc: bd808 >>> >>> _______________________________________________ >>> Wikimedia Cloud Services announce mailing list >>> cloud-annou...@lists.wikimedia.org >>> <mailto:cloud-annou...@lists.wikimedia.org> (formerly >>> labs-annou...@lists.wikimedia.org >>> <mailto:labs-annou...@lists.wikimedia.org>) >>> https://lists.wikimedia.org/mailman/listinfo/cloud-announce >>> <https://lists.wikimedia.org/mailman/listinfo/cloud-announce> >>> _______________________________________________ >>> Wikimedia Cloud Services mailing list >>> Cloud@lists.wikimedia.org <mailto:Cloud@lists.wikimedia.org> (formerly >>> lab...@lists.wikimedia.org <mailto:lab...@lists.wikimedia.org>) >>> https://lists.wikimedia.org/mailman/listinfo/cloud >>> <https://lists.wikimedia.org/mailman/listinfo/cloud>_______________________________________________ >>> Wikimedia Cloud Services mailing list >>> Cloud@lists.wikimedia.org <mailto:Cloud@lists.wikimedia.org> (formerly >>> lab...@lists.wikimedia.org <mailto:lab...@lists.wikimedia.org>) >>> https://lists.wikimedia.org/mailman/listinfo/cloud >>> <https://lists.wikimedia.org/mailman/listinfo/cloud> >
signature.asc
Description: Message signed with OpenPGP
_______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
