[Cloud] Re: Python upgrade on the Toolforge

On 9/28/21 8:42 PM, Roy Smith wrote: Also, please note, I'm just speaking as a Toolforge user. I can't promise what the folks who run Toolforge think about this.  They haven't told me I can't do it, but I don't know if they're totally on board with the idea. I certainly don't like the manual

[Cloud] Re: qsub waiting time on Toolforge

On 02/11/2021 19:30, Anis Elebiary wrote: I have a python script that creates a tsv for the redirect table. I submitted the job through qsub and it's been waiting for more than a day now. Is it normal for the waiting time to be this long ? And, if there's a better way to do the aforementioned t

[Cloud] Re: Storage options: NFS, Cinder, Ceph, Swift?

Hi! On 12/22/21 18:29, Sascha Brawer wrote: What storage options does the Wikimedia cloud have? Can external developers (i.e. people not employed by the Wikimedia foundation) write to Cinder and/or Swift? Either from Toolforge or from Cloud VPS? I've left more detailed replies inline. tl;dr:

[Cloud] Re: preferred proxy

On 1/26/22 18:26, Tim Moody wrote: Which is the preferred domain for dns proxies wmcloud.org or wmflabs.org ? Hi, for new services wmcloud.org is preferred. -- Taavi (User:Majavah) ___ Cloud mailing list -- cl

[Cloud] Re: No space left when doing tab-autocomplete

On 10/04/2022 21:29, Maciej Jaros wrote: I get this weird error when trying to autocomplete file and folder names: > tools.dna@tools-sgebastion-07:~$ cat /-bash: cannot create temp file for here-document: No space left on device > -bash: cannot create temp file for here-document: No space lef

[Cloud] [Cloud-announce] Toolforge Kubernetes upgrade on 2022-05-11

rs it takes for us to upgrade the entire cluster. Assuming your services are able to survive a restart, no action should be needed on your part. -- Taavi Väänänen (User:Majavah) ___ Cloud-announce mailing list -- cloud-annou...@lists.wikimedia.org

[Cloud] [Cloud-announce] Node.js v16 now available on Toolforge Kubernetes

netes#Container_images -- Taavi Väänänen, on behalf of the entire Toolforge admin team ___ Cloud-announce mailing list -- cloud-annou...@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedi

[Cloud] [Cloud-announce] Cloud VPS web proxy now enforces an IP-based rate limit

Hi all, We've just configured a simple rate limiter on the Cloud VPS web proxy that limits traffic from a single IP address to 100 requests/second. The limit is designed to be high enough that it shouldn't affect most normal workflows while also being low enough to properly block excess traffi

[Cloud] WMCS team work transparency (was: New servers don't show up on grafana-labs.wikimedia.org)

On 10/5/22 21:01, David Caro wrote: As Bryan said, this is a work in progress, we might be able to allocate some time for it in the next quarter to help Taavi and get the project going. This is news to me. In general, I feel that over the last few months, quite a lot of planning and progress

[Cloud] Re: WMCS team work transparency (was: New servers don't show up on grafana-labs.wikimedia.org)

On 10/6/22 23:16, Andrew Bogott wrote: I'm definitely interested in talking and thinking about this more. I think it is true that the cloud services staff have started coordinating more frequently in video calls, so your comment is a useful reminder that we need to redouble our efforts on post-

[Cloud] Re: ssh to tool account?

On 2/15/23 16:37, Roy Smith wrote: Am I just doing something wrong, or is toolforge set up to disallow that? Yes, all of Cloud VPS (including Toolforge) is configured to only load SSH keys from the developer account database. This particular feature was requested back in 2015 in

[Cloud] [Cloud-announce] Toolforge Kubernetes upgrade on 2023-04-03

Hi, We will be upgrading the Toolforge Kubernetes cluster next Monday (2023-04-03) starting at around 10:00 UTC. The expected impact is that tools running on the Kubernetes cluster will get restarted a couple of times over the course of the few hours it takes for us to upgrade the entire clu

[Cloud] Re: E-mail alerts for Cloud VPS instances

Hi, sorry for the delayed response. First of all, the Wikitech section you found is about the wikiprod Grafana instance at https://grafana.wikimedia.org. The WMCS Grafana instance (https://grafana.wmcloud.org) only queries data from various Prometheus instances, so for those we use Prometheus

[Cloud] [Cloud-announce] Cloud VPS statsd/Graphite service retirement + other monitoring changes

Hi, We are planning to do some changes to the Cloud VPS monitoring systems. Most notably: * The Graphite and statsd servers will be shut down. Please migrate your clients to use Prometheus metrics instead. * The Grafana instance at https://grafana-cloud.wikimedia.org (or grafana-labs.wikimed

[Cloud] Re: Cloud VPS DNS resolver on some unmaintained virtual machines

On 9/15/23 17:45, Sebastian Berlin wrote: I want to check if I understood this correctly. If an instance has the line `nameserver 172.20.255.1` in /etc/resolv.conf, it's all good? Yes, that's correct. OpenPGP_signature.asc Description: OpenPGP digital signature __

[Cloud] Re: Webservice logs?

> Cloud mailing list -- cloud@lists.wikimedia.org > List information: > https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/ > -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services Wikimedia Foundation ___

[Cloud] [Cloud-announce] Toolforge Kubernetes maintenance on 2023-10-18

. This includes tools that use the jobs framework and tools that run web services using the default Kubernetes backend. The ability to manage tools will remain operational. Taavi -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services Wikimedia Foundation

[Cloud] Re: [Cloud-announce] [IMPORTANT] Grid Engine Shutdown Timeline

I assume you are talking about the amorybot tool. That tool is running several scripts directly on the cron node using jlocal in a way that is against the rules for its use . That does have the unfortunate but unavoidab

[Cloud] [Cloud-announce] New dev.toolforge.org bastion

[0]: https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/dev.toolforge.org [1]: https://phabricator.wikimedia.org/tag/toolforge/ Taavi -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services Wikimedia Foundation ___ Cloud-announce ma

[Cloud] Re: Operating system upgrade policy

is is the specific version of the Debian Bookworm base image that your instance was created with, it means that on 2024-04-10 a new image was built to replace that one. That is unrelated to the process for dropping support for an entire OS release. Taavi -- Taa

[Cloud] Re: zsh is slow on login.toolforge.net

service name are using the g3.cores8.ram16.disk20 flavor. So there's at least no intentional additional slowness. -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services Wikimedia Foundation ___ Cloud mailing list -- cloud@li

[Cloud] Re: Parliament diagram needs PHP 7.3

Hi, The PHP 7.3 image has been emitting deprecation warnings for a while urging tools to upgrade, but has not been removed. Can you be more specific on how it is not working on that tool? Do you get an error message? Taavi -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services

[Cloud] [Cloud-announce] Cloud VPS network maintenance on 2024-05-21

are not resilient to unexpected network outages - in that case you may need to manually restart those tools after the maintenance is complete. [0]: https://phabricator.wikimedia.org/T364459 Taavi (+ the rest of the WMCS team) -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services

[Cloud] Re: Fwd: [Cloud VPS alert][wikispeech] Puppet failure on producer.wikispeech.eqiad1.wikimedia.cloud (172.16.0.200)

String value, got > Undef (file: > /srv/puppet_code/environments/production/modules/profile/manifests/simplelamp2.pp, > line: 32, column: 5) on node producer.wikispeech.eqiad1.wikimedia.cloud > WARNING: Not using cache on failed catalog > ERR: Could not retrieve catalog;

[Cloud] [Cloud-announce] New 4th generation flavors in Cloud VPS

s. [0]: https://wikitech.wikimedia.org/wiki/Help:Cloud_VPS_Instances#Instance_information [1]: https://phabricator.wikimedia.org/T364458 [2]: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication Taavi -- Taavi Väänänen (he/him) Site Reliability Engineer, Cloud Services Wikim

[Cloud] [Cloud-announce] Upcoming reboots of all Cloud VPS VMs

On Thu, Jun 13, 2024 at 8:00 PM Taavi Väänänen wrote: > We will be migrating existing VMs to the new flavors at some point in > the near future. This will most likely involve briefly shutting down > each VM as it's migrated from one network agent to the other. There > wi

[Cloud] Re: [Let's Connect] Outreach Message

On 12/12/24 10:52 AM, luce.hin...@gmail.com wrote: I am contacting you in your capacity at the list owner for the Cloud list. I would love to share a message with your mailing list, introducing our project. I would be happy for this message to be passed on directly, or to compose and send a ne

[Cloud] Re: Errors accessing QRank

Wikimedia Cloud Services provides the compute infrastructure that https://qrank.wmcloud.org and several other services depend on, but we're not directly involved with operating that particular service and don't know the specific status of it. You might have better luck contacting the project m

[Cloud] New Kubernetes runtime images available on Toolforge

.debian.org/releases/bullseye/ [2]: https://phabricator.wikimedia.org/T284590 -- Taavi Väänänen (User:Majavah) ___ Cloud mailing list -- cloud@lists.wikimedia.org List information: https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/