Re: [Cloud] [Cloud-announce] [Toolforge] New Kubernetes cluster open for beta testers

[Magnus Manske via Cloud]

I switched a few "big ones" successfully, but ran into one that doesn't
work:

glamtools https://tools.wmflabs.org/glamtools/ is 503 but `webservice
status` says "Your webservice of type php7.3 is running".

On Thu, Jan 9, 2020 at 9:58 PM Bryan Davis  wrote:

> I am happy to announce that a new and improved Kubernetes cluster is
> now available for use by beta testers on an opt-in basis. A page has
> been created on Wikitech [0] outlining the self-service migration
> process.
>
> Timeline:
> * 2020-01-09: 2020 Kubernetes cluster available for beta testers on an
> opt-in basis
> * 2020-01-23: 2020 Kubernetes cluster general availability for
> migration on an opt-in basis
> * 2020-02-10: Automatic migration of remaining workloads from 2016
> cluster to 2020 cluster by Toolforge admins
>
> This new cluster has been a work in progress for more than a year
> within the Wikimedia Cloud Services team, and a top priority project
> for the past six months. About 35 tools, including
> https://tools.wmflabs.org/admin/, are currently running on what we are
> calling the "2020 Kubernetes cluster". This new cluster is running
> Kubernetes v1.15.6 and Docker 19.03.4. It is also using a newer
> authentication and authorization method (RBAC), a new ingress routing
> service, and a different method of integrating with the Developer
> account LDAP service. We have built a new tool [1] which makes the
> state of the Kubernetes cluster more transparent and on par with the
> information that we already expose for the grid engine cluster [2].
>
> With a significant number of tools managed by Toolforge administrators
> already migrated to the new cluster, we are fairly confident that the
> basic features used by most Kubernetes tools are covered. It is likely
> that a few outlying issues remain to be found as more tools move, but
> we have confidence that we can address them quickly. This has led us
> to propose a fairly short period of voluntary beta testing, followed
> by a short general availability opt-in migration period, and finally a
> complete migration of all remaining tools which will be done by the
> Toolforge administration team for anyone who has not migrated their
> self.
>
> Please help with beta testing if you have some time and are willing to
> get help on irc, Phabricator, and the cloud@lists.wikimedia.org
> mailing list for early adopter issues you may encounter.
>
> I want to publicly praise Brooke Storm and Arturo Borrero González for
> the hours that they have put into reading docs, building proof of
> concept clusters, and improving automation and processes to make the
> 2020 Kubernetes cluster possible. The Toolforge community can look
> forward to more frequent and less disruptive software upgrades in this
> cluster as a direct result of this work. We have some other feature
> improvements in planning now that I think you will all be excited to
> see and use later this year!
>
> [0]:
> https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration
> [1]: https://tools.wmflabs.org/k8s-status/
> [2]: https://tools.wmflabs.org/sge-status/
>
> Bryan (on behalf of the Toolforge admins and the Cloud Services team)
> --
> Bryan Davis  Technical Engagement  Wikimedia Foundation
> Principal Software Engineer   Boise, ID USA
> [[m:User:BDavis_(WMF)]]  irc: bd808
>
> ___
> Wikimedia Cloud Services announce mailing list
> cloud-annou...@lists.wikimedia.org (formerly
> labs-annou...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud-announce
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] [Cloud-announce] [Toolforge] New Kubernetes cluster open for beta testers

[Magnus Manske via Cloud]

And now I can't switch back:

...switch back commands...

tools.glamtools@tools-sgebastion-07:~$ unalias kubectl
tools.glamtools@tools-sgebastion-07:~$ webservice --backend=kubernetes
php7.3 start
Your job is already running
tools.glamtools@tools-sgebastion-07:~$ webservice stop
Your webservice is not running

tools.glamtools@tools-sgebastion-07:~$ webservice --backend=kubernetes
php7.3 start
Your job is already running

On Mon, Jan 13, 2020 at 3:47 PM Magnus Manske 
wrote:

> I switched a few "big ones" successfully, but ran into one that doesn't
> work:
>
> glamtools https://tools.wmflabs.org/glamtools/ is 503 but `webservice
> status` says "Your webservice of type php7.3 is running".
>
> On Thu, Jan 9, 2020 at 9:58 PM Bryan Davis  wrote:
>
>> I am happy to announce that a new and improved Kubernetes cluster is
>> now available for use by beta testers on an opt-in basis. A page has
>> been created on Wikitech [0] outlining the self-service migration
>> process.
>>
>> Timeline:
>> * 2020-01-09: 2020 Kubernetes cluster available for beta testers on an
>> opt-in basis
>> * 2020-01-23: 2020 Kubernetes cluster general availability for
>> migration on an opt-in basis
>> * 2020-02-10: Automatic migration of remaining workloads from 2016
>> cluster to 2020 cluster by Toolforge admins
>>
>> This new cluster has been a work in progress for more than a year
>> within the Wikimedia Cloud Services team, and a top priority project
>> for the past six months. About 35 tools, including
>> https://tools.wmflabs.org/admin/, are currently running on what we are
>> calling the "2020 Kubernetes cluster". This new cluster is running
>> Kubernetes v1.15.6 and Docker 19.03.4. It is also using a newer
>> authentication and authorization method (RBAC), a new ingress routing
>> service, and a different method of integrating with the Developer
>> account LDAP service. We have built a new tool [1] which makes the
>> state of the Kubernetes cluster more transparent and on par with the
>> information that we already expose for the grid engine cluster [2].
>>
>> With a significant number of tools managed by Toolforge administrators
>> already migrated to the new cluster, we are fairly confident that the
>> basic features used by most Kubernetes tools are covered. It is likely
>> that a few outlying issues remain to be found as more tools move, but
>> we have confidence that we can address them quickly. This has led us
>> to propose a fairly short period of voluntary beta testing, followed
>> by a short general availability opt-in migration period, and finally a
>> complete migration of all remaining tools which will be done by the
>> Toolforge administration team for anyone who has not migrated their
>> self.
>>
>> Please help with beta testing if you have some time and are willing to
>> get help on irc, Phabricator, and the cloud@lists.wikimedia.org
>> mailing list for early adopter issues you may encounter.
>>
>> I want to publicly praise Brooke Storm and Arturo Borrero González for
>> the hours that they have put into reading docs, building proof of
>> concept clusters, and improving automation and processes to make the
>> 2020 Kubernetes cluster possible. The Toolforge community can look
>> forward to more frequent and less disruptive software upgrades in this
>> cluster as a direct result of this work. We have some other feature
>> improvements in planning now that I think you will all be excited to
>> see and use later this year!
>>
>> [0]:
>> https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration
>> [1]: https://tools.wmflabs.org/k8s-status/
>> [2]: https://tools.wmflabs.org/sge-status/
>>
>> Bryan (on behalf of the Toolforge admins and the Cloud Services team)
>> --
>> Bryan Davis  Technical Engagement  Wikimedia Foundation
>> Principal Software Engineer   Boise, ID USA
>> [[m:User:BDavis_(WMF)]]  irc: bd808
>>
>> ___
>> Wikimedia Cloud Services announce mailing list
>> cloud-annou...@lists.wikimedia.org (formerly
>> labs-annou...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud-announce
>> ___
>> Wikimedia Cloud Services mailing list
>> Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
>> https://lists.wikimedia.org/mailman/listinfo/cloud
>
>
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud

Re: [Cloud] [Cloud-announce] [Toolforge] New Kubernetes cluster open for beta testers

[Brooke Storm]

I see a problem with at least one container image (which has nothing to do with 
the new cluster, I can see it on the old cluster as well).  It looks like I’m 
going to be trying to fix that now.
(Magnus, this is probably what you are seeing as well).

Brooke Storm
Senior SRE
Wikimedia Cloud Services
bst...@wikimedia.org 
IRC: bstorm_

> On Jan 13, 2020, at 8:47 AM, Magnus Manske via Cloud 
>  wrote:
> 
> I switched a few "big ones" successfully, but ran into one that doesn't work:
> 
> glamtools https://tools.wmflabs.org/glamtools/ 
>  is 503 but `webservice status` says 
> "Your webservice of type php7.3 is running".
> 
> On Thu, Jan 9, 2020 at 9:58 PM Bryan Davis  > wrote:
> I am happy to announce that a new and improved Kubernetes cluster is
> now available for use by beta testers on an opt-in basis. A page has
> been created on Wikitech [0] outlining the self-service migration
> process.
> 
> Timeline:
> * 2020-01-09: 2020 Kubernetes cluster available for beta testers on an
> opt-in basis
> * 2020-01-23: 2020 Kubernetes cluster general availability for
> migration on an opt-in basis
> * 2020-02-10: Automatic migration of remaining workloads from 2016
> cluster to 2020 cluster by Toolforge admins
> 
> This new cluster has been a work in progress for more than a year
> within the Wikimedia Cloud Services team, and a top priority project
> for the past six months. About 35 tools, including
> https://tools.wmflabs.org/admin/ , are 
> currently running on what we are
> calling the "2020 Kubernetes cluster". This new cluster is running
> Kubernetes v1.15.6 and Docker 19.03.4. It is also using a newer
> authentication and authorization method (RBAC), a new ingress routing
> service, and a different method of integrating with the Developer
> account LDAP service. We have built a new tool [1] which makes the
> state of the Kubernetes cluster more transparent and on par with the
> information that we already expose for the grid engine cluster [2].
> 
> With a significant number of tools managed by Toolforge administrators
> already migrated to the new cluster, we are fairly confident that the
> basic features used by most Kubernetes tools are covered. It is likely
> that a few outlying issues remain to be found as more tools move, but
> we have confidence that we can address them quickly. This has led us
> to propose a fairly short period of voluntary beta testing, followed
> by a short general availability opt-in migration period, and finally a
> complete migration of all remaining tools which will be done by the
> Toolforge administration team for anyone who has not migrated their
> self.
> 
> Please help with beta testing if you have some time and are willing to
> get help on irc, Phabricator, and the cloud@lists.wikimedia.org 
> 
> mailing list for early adopter issues you may encounter.
> 
> I want to publicly praise Brooke Storm and Arturo Borrero González for
> the hours that they have put into reading docs, building proof of
> concept clusters, and improving automation and processes to make the
> 2020 Kubernetes cluster possible. The Toolforge community can look
> forward to more frequent and less disruptive software upgrades in this
> cluster as a direct result of this work. We have some other feature
> improvements in planning now that I think you will all be excited to
> see and use later this year!
> 
> [0]: 
> https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration 
> 
> [1]: https://tools.wmflabs.org/k8s-status/ 
> 
> [2]: https://tools.wmflabs.org/sge-status/ 
> 
> 
> Bryan (on behalf of the Toolforge admins and the Cloud Services team)
> --
> Bryan Davis  Technical Engagement  Wikimedia Foundation
> Principal Software Engineer   Boise, ID USA
> [[m:User:BDavis_(WMF)]]  irc: bd808
> 
> ___
> Wikimedia Cloud Services announce mailing list
> cloud-annou...@lists.wikimedia.org 
>  (formerly 
> labs-annou...@lists.wikimedia.org )
> https://lists.wikimedia.org/mailman/listinfo/cloud-announce 
> 
> ___
> Wikimedia Cloud Services mailing list
> Cloud@lists.wikimedia.org  (formerly 
> lab...@lists.wikimedia.org )
> https://lists.wikimedia.org/mailman/listinfo/cloud 
> ___
> Wikimedia Cloud Services mailin

Re: [Cloud] [Cloud-announce] [Toolforge] New Kubernetes cluster open for beta testers

[Brooke Storm]

I suspect this will affect containers that run Debian Buster packages.  I see 
php7.3 and python3.7.
I’d suggest not even restarting web services on those runtimes until we have it 
fixed.  For anyone who has done so, we are working on it.  Any logs could be 
helpful.

Brooke Storm
Senior SRE
Wikimedia Cloud Services
bst...@wikimedia.org 
IRC: bstorm_

> On Jan 13, 2020, at 9:06 AM, Brooke Storm  wrote:
> 
> I see a problem with at least one container image (which has nothing to do 
> with the new cluster, I can see it on the old cluster as well).  It looks 
> like I’m going to be trying to fix that now.
> (Magnus, this is probably what you are seeing as well).
> 
> Brooke Storm
> Senior SRE
> Wikimedia Cloud Services
> bst...@wikimedia.org 
> IRC: bstorm_
> 
>> On Jan 13, 2020, at 8:47 AM, Magnus Manske via Cloud 
>> mailto:cloud@lists.wikimedia.org>> wrote:
>> 
>> I switched a few "big ones" successfully, but ran into one that doesn't work:
>> 
>> glamtools https://tools.wmflabs.org/glamtools/ 
>>  is 503 but `webservice status` says 
>> "Your webservice of type php7.3 is running".
>> 
>> On Thu, Jan 9, 2020 at 9:58 PM Bryan Davis > > wrote:
>> I am happy to announce that a new and improved Kubernetes cluster is
>> now available for use by beta testers on an opt-in basis. A page has
>> been created on Wikitech [0] outlining the self-service migration
>> process.
>> 
>> Timeline:
>> * 2020-01-09: 2020 Kubernetes cluster available for beta testers on an
>> opt-in basis
>> * 2020-01-23: 2020 Kubernetes cluster general availability for
>> migration on an opt-in basis
>> * 2020-02-10: Automatic migration of remaining workloads from 2016
>> cluster to 2020 cluster by Toolforge admins
>> 
>> This new cluster has been a work in progress for more than a year
>> within the Wikimedia Cloud Services team, and a top priority project
>> for the past six months. About 35 tools, including
>> https://tools.wmflabs.org/admin/ , are 
>> currently running on what we are
>> calling the "2020 Kubernetes cluster". This new cluster is running
>> Kubernetes v1.15.6 and Docker 19.03.4. It is also using a newer
>> authentication and authorization method (RBAC), a new ingress routing
>> service, and a different method of integrating with the Developer
>> account LDAP service. We have built a new tool [1] which makes the
>> state of the Kubernetes cluster more transparent and on par with the
>> information that we already expose for the grid engine cluster [2].
>> 
>> With a significant number of tools managed by Toolforge administrators
>> already migrated to the new cluster, we are fairly confident that the
>> basic features used by most Kubernetes tools are covered. It is likely
>> that a few outlying issues remain to be found as more tools move, but
>> we have confidence that we can address them quickly. This has led us
>> to propose a fairly short period of voluntary beta testing, followed
>> by a short general availability opt-in migration period, and finally a
>> complete migration of all remaining tools which will be done by the
>> Toolforge administration team for anyone who has not migrated their
>> self.
>> 
>> Please help with beta testing if you have some time and are willing to
>> get help on irc, Phabricator, and the cloud@lists.wikimedia.org 
>> 
>> mailing list for early adopter issues you may encounter.
>> 
>> I want to publicly praise Brooke Storm and Arturo Borrero González for
>> the hours that they have put into reading docs, building proof of
>> concept clusters, and improving automation and processes to make the
>> 2020 Kubernetes cluster possible. The Toolforge community can look
>> forward to more frequent and less disruptive software upgrades in this
>> cluster as a direct result of this work. We have some other feature
>> improvements in planning now that I think you will all be excited to
>> see and use later this year!
>> 
>> [0]: 
>> https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration 
>> 
>> [1]: https://tools.wmflabs.org/k8s-status/ 
>> 
>> [2]: https://tools.wmflabs.org/sge-status/ 
>> 
>> 
>> Bryan (on behalf of the Toolforge admins and the Cloud Services team)
>> --
>> Bryan Davis  Technical Engagement  Wikimedia Foundation
>> Principal Software Engineer   Boise, ID USA
>> [[m:User:BDavis_(WMF)]]  irc: bd808
>> 
>> ___
>> Wikimedia Cloud Services announce mailing list
>> cloud-annou...@lists.wikimedia.org 
>>  (formerly 
>> labs-annou...@lists.wikimedia.org 

Re: [Cloud] [Cloud-announce] [Toolforge] New Kubernetes cluster open for beta testers

[Brooke Storm]

I’ve created Phabricator task T242632 to track this.  Please coordinate there 
as well for anyone with information and time.

Brooke Storm
Senior SRE
Wikimedia Cloud Services
bst...@wikimedia.org 
IRC: bstorm_

> On Jan 13, 2020, at 9:10 AM, Brooke Storm  wrote:
> 
> I suspect this will affect containers that run Debian Buster packages.  I see 
> php7.3 and python3.7.
> I’d suggest not even restarting web services on those runtimes until we have 
> it fixed.  For anyone who has done so, we are working on it.  Any logs could 
> be helpful.
> 
> Brooke Storm
> Senior SRE
> Wikimedia Cloud Services
> bst...@wikimedia.org 
> IRC: bstorm_
> 
>> On Jan 13, 2020, at 9:06 AM, Brooke Storm > > wrote:
>> 
>> I see a problem with at least one container image (which has nothing to do 
>> with the new cluster, I can see it on the old cluster as well).  It looks 
>> like I’m going to be trying to fix that now.
>> (Magnus, this is probably what you are seeing as well).
>> 
>> Brooke Storm
>> Senior SRE
>> Wikimedia Cloud Services
>> bst...@wikimedia.org 
>> IRC: bstorm_
>> 
>>> On Jan 13, 2020, at 8:47 AM, Magnus Manske via Cloud 
>>> mailto:cloud@lists.wikimedia.org>> wrote:
>>> 
>>> I switched a few "big ones" successfully, but ran into one that doesn't 
>>> work:
>>> 
>>> glamtools https://tools.wmflabs.org/glamtools/ 
>>>  is 503 but `webservice status` says 
>>> "Your webservice of type php7.3 is running".
>>> 
>>> On Thu, Jan 9, 2020 at 9:58 PM Bryan Davis >> > wrote:
>>> I am happy to announce that a new and improved Kubernetes cluster is
>>> now available for use by beta testers on an opt-in basis. A page has
>>> been created on Wikitech [0] outlining the self-service migration
>>> process.
>>> 
>>> Timeline:
>>> * 2020-01-09: 2020 Kubernetes cluster available for beta testers on an
>>> opt-in basis
>>> * 2020-01-23: 2020 Kubernetes cluster general availability for
>>> migration on an opt-in basis
>>> * 2020-02-10: Automatic migration of remaining workloads from 2016
>>> cluster to 2020 cluster by Toolforge admins
>>> 
>>> This new cluster has been a work in progress for more than a year
>>> within the Wikimedia Cloud Services team, and a top priority project
>>> for the past six months. About 35 tools, including
>>> https://tools.wmflabs.org/admin/ , are 
>>> currently running on what we are
>>> calling the "2020 Kubernetes cluster". This new cluster is running
>>> Kubernetes v1.15.6 and Docker 19.03.4. It is also using a newer
>>> authentication and authorization method (RBAC), a new ingress routing
>>> service, and a different method of integrating with the Developer
>>> account LDAP service. We have built a new tool [1] which makes the
>>> state of the Kubernetes cluster more transparent and on par with the
>>> information that we already expose for the grid engine cluster [2].
>>> 
>>> With a significant number of tools managed by Toolforge administrators
>>> already migrated to the new cluster, we are fairly confident that the
>>> basic features used by most Kubernetes tools are covered. It is likely
>>> that a few outlying issues remain to be found as more tools move, but
>>> we have confidence that we can address them quickly. This has led us
>>> to propose a fairly short period of voluntary beta testing, followed
>>> by a short general availability opt-in migration period, and finally a
>>> complete migration of all remaining tools which will be done by the
>>> Toolforge administration team for anyone who has not migrated their
>>> self.
>>> 
>>> Please help with beta testing if you have some time and are willing to
>>> get help on irc, Phabricator, and the cloud@lists.wikimedia.org 
>>> 
>>> mailing list for early adopter issues you may encounter.
>>> 
>>> I want to publicly praise Brooke Storm and Arturo Borrero González for
>>> the hours that they have put into reading docs, building proof of
>>> concept clusters, and improving automation and processes to make the
>>> 2020 Kubernetes cluster possible. The Toolforge community can look
>>> forward to more frequent and less disruptive software upgrades in this
>>> cluster as a direct result of this work. We have some other feature
>>> improvements in planning now that I think you will all be excited to
>>> see and use later this year!
>>> 
>>> [0]: 
>>> https://wikitech.wikimedia.org/wiki/News/2020_Kubernetes_cluster_migration 
>>> 
>>> [1]: https://tools.wmflabs.org/k8s-status/ 
>>> 
>>> [2]: https://tools.wmflabs.org/sge-status/ 
>>> 
>>> 
>>> Bryan (on behalf of the Toolforge admins and the Cloud Services team)
>>> --
>>> Bryan Davis  Technica

Re: [Cloud] [Cloud-announce] [Toolforge] New Kubernetes cluster open for beta testers

[Brooke Storm]

Per that ticket, I no longer think there is any issue with the images for 
python at least.  There was an issue with some nodes (that is being/mostly 
fixed). I’ll take a look at glamtools

Brooke Storm
Senior SRE
Wikimedia Cloud Services
bst...@wikimedia.org 
IRC: bstorm_

> On Jan 13, 2020, at 9:14 AM, Brooke Storm  wrote:
> 
> I’ve created Phabricator task T242632 to track this.  Please coordinate there 
> as well for anyone with information and time.
> 
> Brooke Storm
> Senior SRE
> Wikimedia Cloud Services
> bst...@wikimedia.org 
> IRC: bstorm_
> 
>> On Jan 13, 2020, at 9:10 AM, Brooke Storm > > wrote:
>> 
>> I suspect this will affect containers that run Debian Buster packages.  I 
>> see php7.3 and python3.7.
>> I’d suggest not even restarting web services on those runtimes until we have 
>> it fixed.  For anyone who has done so, we are working on it.  Any logs could 
>> be helpful.
>> 
>> Brooke Storm
>> Senior SRE
>> Wikimedia Cloud Services
>> bst...@wikimedia.org 
>> IRC: bstorm_
>> 
>>> On Jan 13, 2020, at 9:06 AM, Brooke Storm >> > wrote:
>>> 
>>> I see a problem with at least one container image (which has nothing to do 
>>> with the new cluster, I can see it on the old cluster as well).  It looks 
>>> like I’m going to be trying to fix that now.
>>> (Magnus, this is probably what you are seeing as well).
>>> 
>>> Brooke Storm
>>> Senior SRE
>>> Wikimedia Cloud Services
>>> bst...@wikimedia.org 
>>> IRC: bstorm_
>>> 
 On Jan 13, 2020, at 8:47 AM, Magnus Manske via Cloud 
 mailto:cloud@lists.wikimedia.org>> wrote:
 
 I switched a few "big ones" successfully, but ran into one that doesn't 
 work:
 
 glamtools https://tools.wmflabs.org/glamtools/ 
  is 503 but `webservice status` says 
 "Your webservice of type php7.3 is running".
 
 On Thu, Jan 9, 2020 at 9:58 PM Bryan Davis >>> > wrote:
 I am happy to announce that a new and improved Kubernetes cluster is
 now available for use by beta testers on an opt-in basis. A page has
 been created on Wikitech [0] outlining the self-service migration
 process.
 
 Timeline:
 * 2020-01-09: 2020 Kubernetes cluster available for beta testers on an
 opt-in basis
 * 2020-01-23: 2020 Kubernetes cluster general availability for
 migration on an opt-in basis
 * 2020-02-10: Automatic migration of remaining workloads from 2016
 cluster to 2020 cluster by Toolforge admins
 
 This new cluster has been a work in progress for more than a year
 within the Wikimedia Cloud Services team, and a top priority project
 for the past six months. About 35 tools, including
 https://tools.wmflabs.org/admin/ , are 
 currently running on what we are
 calling the "2020 Kubernetes cluster". This new cluster is running
 Kubernetes v1.15.6 and Docker 19.03.4. It is also using a newer
 authentication and authorization method (RBAC), a new ingress routing
 service, and a different method of integrating with the Developer
 account LDAP service. We have built a new tool [1] which makes the
 state of the Kubernetes cluster more transparent and on par with the
 information that we already expose for the grid engine cluster [2].
 
 With a significant number of tools managed by Toolforge administrators
 already migrated to the new cluster, we are fairly confident that the
 basic features used by most Kubernetes tools are covered. It is likely
 that a few outlying issues remain to be found as more tools move, but
 we have confidence that we can address them quickly. This has led us
 to propose a fairly short period of voluntary beta testing, followed
 by a short general availability opt-in migration period, and finally a
 complete migration of all remaining tools which will be done by the
 Toolforge administration team for anyone who has not migrated their
 self.
 
 Please help with beta testing if you have some time and are willing to
 get help on irc, Phabricator, and the cloud@lists.wikimedia.org 
 
 mailing list for early adopter issues you may encounter.
 
 I want to publicly praise Brooke Storm and Arturo Borrero González for
 the hours that they have put into reading docs, building proof of
 concept clusters, and improving automation and processes to make the
 2020 Kubernetes cluster possible. The Toolforge community can look
 forward to more frequent and less disruptive software upgrades in this
 cluster as a direct result of this work. We have some other feature
 improvements in planning now that I think you will all be excited to
 see and use l