Per that ticket, I no longer think there is any issue with the images for python at least. There was an issue with some nodes (that is being/mostly fixed). I’ll take a look at glamtools
Brooke Storm Senior SRE Wikimedia Cloud Services bst...@wikimedia.org <mailto:bst...@wikimedia.org> IRC: bstorm_ > On Jan 13, 2020, at 9:14 AM, Brooke Storm <bst...@wikimedia.org> wrote: > > I’ve created Phabricator task T242632 to track this. Please coordinate there > as well for anyone with information and time. > > Brooke Storm > Senior SRE > Wikimedia Cloud Services > bst...@wikimedia.org <mailto:bst...@wikimedia.org> > IRC: bstorm_ > >> On Jan 13, 2020, at 9:10 AM, Brooke Storm <bst...@wikimedia.org >> <mailto:bst...@wikimedia.org>> wrote: >> >> I suspect this will affect containers that run Debian Buster packages. I >> see php7.3 and python3.7. >> I’d suggest not even restarting web services on those runtimes until we have >> it fixed. For anyone who has done so, we are working on it. Any logs could >> be helpful. >> >> Brooke Storm >> Senior SRE >> Wikimedia Cloud Services >> bst...@wikimedia.org <mailto:bst...@wikimedia.org> >> IRC: bstorm_ >> >>> On Jan 13, 2020, at 9:06 AM, Brooke Storm <bst...@wikimedia.org >>> <mailto:bst...@wikimedia.org>> wrote: >>> >>> I see a problem with at least one container image (which has nothing to do >>> with the new cluster, I can see it on the old cluster as well). It looks >>> like I’m going to be trying to fix that now. >>> (Magnus, this is probably what you are seeing as well). >>> >>> Brooke Storm >>> Senior SRE >>> Wikimedia Cloud Services >>> bst...@wikimedia.org <mailto:bst...@wikimedia.org> >>> IRC: bstorm_ >>> >>>> On Jan 13, 2020, at 8:47 AM, Magnus Manske via Cloud >>>> <cloud@lists.wikimedia.org <mailto:cloud@lists.wikimedia.org>> wrote: >>>> >>>> I switched a few "big ones" successfully, but ran into one that doesn't >>>> work: >>>> >>>> glamtools https://tools.wmflabs.org/glamtools/ >>>> <https://tools.wmflabs.org/glamtools/> is 503 but `webservice status` says >>>> "Your webservice of type php7.3 is running". >>>> >>>> On Thu, Jan 9, 2020 at 9:58 PM Bryan Davis <bd...@wikimedia.org >>>> <mailto:bd...@wikimedia.org>> wrote: >>>> I am happy to announce that a new and improved Kubernetes cluster is >>>> now available for use by beta testers on an opt-in basis. A page has >>>> been created on Wikitech [0] outlining the self-service migration >>>> process. >>>> >>>> Timeline: >>>> * 2020-01-09: 2020 Kubernetes cluster available for beta testers on an >>>> opt-in basis >>>> * 2020-01-23: 2020 Kubernetes cluster general availability for >>>> migration on an opt-in basis >>>> * 2020-02-10: Automatic migration of remaining workloads from 2016 >>>> cluster to 2020 cluster by Toolforge admins >>>> >>>> This new cluster has been a work in progress for more than a year >>>> within the Wikimedia Cloud Services team, and a top priority project >>>> for the past six months. About 35 tools, including >>>> https://tools.wmflabs.org/admin/ <https://tools.wmflabs.org/admin/>, are >>>> currently running on what we are >>>> calling the "2020 Kubernetes cluster". This new cluster is running >>>> Kubernetes v1.15.6 and Docker 19.03.4. It is also using a newer >>>> authentication and authorization method (RBAC), a new ingress routing >>>> service, and a different method of integrating with the Developer >>>> account LDAP service. We have built a new tool [1] which makes the >>>> state of the Kubernetes cluster more transparent and on par with the >>>> information that we already expose for the grid engine cluster [2]. >>>> >>>> With a significant number of tools managed by Toolforge administrators >>>> already migrated to the new cluster, we are fairly confident that the >>>> basic features used by most Kubernetes tools are covered. It is likely >>>> that a few outlying issues remain to be found as more tools move, but >>>> we have confidence that we can address them quickly. This has led us >>>> to propose a fairly short period of voluntary beta testing, followed >>>> by a short general availability opt-in migration period, and finally a >>>> complete migration of all remaining tools which will be done by the >>>> Toolforge administration team for anyone who has not migrated their >>>> self. >>>> >>>> Please help with beta testing if you have some time and are willing to >>>> get help on irc, Phabricator, and the cloud@lists.wikimedia.org >>>> <mailto:cloud@lists.wikimedia.org> >>>> mailing list for early adopter issues you may encounter. >>>> >>>> I want to publicly praise Brooke Storm and Arturo Borrero González for >>>> the hours that they have put into reading docs, building proof of >>>> concept clusters, and improving automation and processes to make the >>>> 2020 Kubernetes cluster possible. The Toolforge community can look >>>> forward to more frequent and less disruptive software upgrades in this >>>> cluster as a direct result of this work. We have some other feature >>>> improvements in planning now that I think you will all be excited to >>>> see and use later this year! >>>> >>>> [0]: >>>> https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration >>>> <https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration> >>>> [1]: https://tools.wmflabs.org/k8s-status/ >>>> <https://tools.wmflabs.org/k8s-status/> >>>> [2]: https://tools.wmflabs.org/sge-status/ >>>> <https://tools.wmflabs.org/sge-status/> >>>> >>>> Bryan (on behalf of the Toolforge admins and the Cloud Services team) >>>> -- >>>> Bryan Davis Technical Engagement Wikimedia Foundation >>>> Principal Software Engineer Boise, ID USA >>>> [[m:User:BDavis_(WMF)]] irc: bd808 >>>> >>>> _______________________________________________ >>>> Wikimedia Cloud Services announce mailing list >>>> cloud-annou...@lists.wikimedia.org >>>> <mailto:cloud-annou...@lists.wikimedia.org> (formerly >>>> labs-annou...@lists.wikimedia.org >>>> <mailto:labs-annou...@lists.wikimedia.org>) >>>> https://lists.wikimedia.org/mailman/listinfo/cloud-announce >>>> <https://lists.wikimedia.org/mailman/listinfo/cloud-announce> >>>> _______________________________________________ >>>> Wikimedia Cloud Services mailing list >>>> Cloud@lists.wikimedia.org <mailto:Cloud@lists.wikimedia.org> (formerly >>>> lab...@lists.wikimedia.org <mailto:lab...@lists.wikimedia.org>) >>>> https://lists.wikimedia.org/mailman/listinfo/cloud >>>> <https://lists.wikimedia.org/mailman/listinfo/cloud>_______________________________________________ >>>> Wikimedia Cloud Services mailing list >>>> Cloud@lists.wikimedia.org <mailto:Cloud@lists.wikimedia.org> (formerly >>>> lab...@lists.wikimedia.org <mailto:lab...@lists.wikimedia.org>) >>>> https://lists.wikimedia.org/mailman/listinfo/cloud >>>> <https://lists.wikimedia.org/mailman/listinfo/cloud> >> >
signature.asc
Description: Message signed with OpenPGP
_______________________________________________ Wikimedia Cloud Services mailing list Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/cloud
