Hi,
since the beginning of this month we found the need to upgrade the
version of mono framework in Toolforge to something newer [0].
Affected tools/boots maintainers were aware of the upcoming changes, but
in case some of you are developing a new tool/boot, please just note the
change:
mono-com
We upgraded the Mono/.NET framework in Toolforge/GridEngine from the 3.x
version to 5.x [0].
We discovered that some tweaking is required due to some weird behavior
regarding memory allocation by the framework [1].
The first symptom you will see is your boot doing high CPU load (spins).
The fix i
Hi!
We deleted the prometheus user from LDAP and created it locally [0].
This may cause puppet failures, since there is a timeframe in which the
id/gid in /var/lib/prometheus is the old LDAP one.
We are running a massive, CloudVPS-wide deluser/adduser/chown operation
to fix this.
[0] https://ph
Hi!
Next monday 13th we will be doing some maintenance on the main Cloud VPS
deployment to merge the keystone service of both main and eqiad1
deployments (the new one that we will eventually put into production).
Toolforge users will not be affected by this outage.
Day: Monday 13th August
Start
On 07/08/18 18:24, Arturo Borrero Gonzalez wrote:
> Hi!
>
> Next monday 13th we will be doing some maintenance on the main Cloud VPS
> deployment to merge the keystone service of both main and eqiad1
> deployments (the new one that we will eventually put into production).
>
On 13/08/18 15:30, Arturo Borrero Gonzalez wrote:
> On 07/08/18 18:24, Arturo Borrero Gonzalez wrote:
>> Hi!
>>
>> Next monday 13th we will be doing some maintenance on the main Cloud VPS
>> deployment to merge the keystone service of both main and eqiad1
>> depl
Hi!
We would like to share some information regarding Wikimedia Cloud
Services plans for deprecating Ubuntu, specially Trusty.
Ubuntu Trusty's end-of-life is April 2019 and the WMF decided to
consolidate in a single operating system, which is Debian.
In Cloud VPS, projects containing Ubuntu virt
Next monday 2018-11-19 we will be rebooting several Cloud VPS
infrastructure servers [0] for maintenance and security updates.
This is just a simple reboot of servers and we don't expect any outage
or major interruptions, but some services may be down briefly:
* Horizon and Wikitech may misbehave
Hi,
next Tuesday 2018-11-20 at 17:30 UTC we will be rebooting the OSM
database (part of our data services) for maintenance and security updates.
In concrete the labstore1006.eqiad.wmnet (osmdb.eqiad.wmnet) server will
be rebooted. The other server in the cluster, labstore1007.eqiad.wmnet
has been
On 11/15/18 2:03 PM, Arturo Borrero Gonzalez wrote:
> Next monday 2018-11-19 we will be rebooting several Cloud VPS
> infrastructure servers [0] for maintenance and security updates.
>
> This is just a simple reboot of servers and we don't expect any outage
> or major int
On 11/15/18 5:58 PM, Arturo Borrero Gonzalez wrote:
> Hi,
>
> next Tuesday 2018-11-20 at 17:30 UTC we will be rebooting the OSM
> database (part of our data services) for maintenance and security updates.
>
> In concrete the labstore1006.eqiad.wmnet (osmdb.eqiad.wmnet) server
On 11/20/18 6:19 PM, Arturo Borrero Gonzalez wrote:
> On 11/15/18 5:58 PM, Arturo Borrero Gonzalez wrote:
>> Hi,
>>
>> next Tuesday 2018-11-20 at 17:30 UTC we will be rebooting the OSM
>> database (part of our data services) for maintenance and security updates.
>>
Hi,
next Tuesday, 2018-11-27 @ 17:30UTC we will reboot the
labnet1001.eqiad.wmnet server for maintenance and security updates.
This server provides virtual networking services for CloudVPS in the
main deployment (the old one, different from the eqiad1 deployment).
We won't be doing any failover p
On 11/21/18 10:54 AM, Arturo Borrero Gonzalez wrote:
> Hi,
>
> next Tuesday, 2018-11-27 @ 17:30UTC we will reboot the
> labnet1001.eqiad.wmnet server for maintenance and security updates.
>
> This server provides virtual networking services for CloudVPS in the
> main de
Hi!
Tomorrow 2018-12-20 @ 17:00 UTC (~24h from now) we will be conducting
some network maintenance in Cloud VPS (openstack).
We will be doing some works on the transport network that connects the
Neutron server to the rest of the internet. Running CloudVPS instances
will see a brief connection pr
On 12/19/18 6:16 PM, Arturo Borrero Gonzalez wrote:
> Hi!
>
> Tomorrow 2018-12-20 @ 17:00 UTC (~24h from now) we will be conducting
> some network maintenance in Cloud VPS (openstack).
>
> We will be doing some works on the transport network that connects the
> Neutron serv
(list cross-posting on purpose, sorry for that)
Hi!
Today is the deadline for Ubuntu Trusty instances running in CloudVPS
[0]. We will be shutting down the remaining instances next monday
(2019-01-21) to avoid having the weekend in-between.
This situation has been communicated in the correspondi
irt1009 as cloudvirt1009
https://phabricator.wikimedia.org/T216239
[2] ToolsDB overload and cleanup https://phabricator.wikimedia.org/T216208
[3] Replace labsdb100[4567] with instances on cloudvirt1019 and cloudvirt1020
https://phabricator.wikimedia.org/T193264
--
Arturo Borrero Gonzalez
Operations Engineer /
Borrero Gonzalez
Operations Engineer / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
gards
--
Arturo Borrero Gonzalez
Operations Engineer / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
short notice,
regards.
[0] Cloud Services: reallocate workload from rack B5-eqiad
https://phabricator.wikimedia.org/T223148
[1] Install new PDUs into b5-eqiad https://phabricator.wikimedia.org/T223126
--
Arturo Borrero Gonzalez
Operations Engineer / Wikimedia Cloud Services
Wikimedia Found
On 5/14/19 2:16 PM, Arturo Borrero Gonzalez wrote:
> Hi!
>
> on 2019-05-16 13:00 UTC there will be a maintenance operation in one of the
> Wikimedia Foundation datacenter racks that affects 2 of our servers running
> virtual machines [0]. There is a risk that this maintenanc
r references in some docs.
Where did you find a reference to this channel?
regards.
--
Arturo Borrero Gonzalez
Operations Engineer / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.or
ogging where log_title = "A.S._Roma" and
>> log_namespace = 0 and log_timestamp > 20160101000 and log_action =
>> "move"' > A.S._Roma.txt;
>>>
>>> Now, I just just get command not found.
>>>
You can read more about the `sql` comm
fast, and there may be a lot of them that
will fail while we stabilize the DNS service.
Please reach out to the WMCS team if you need more details or have any doubts.
regards.
--
Arturo Borrero Gonzalez
Operations Engineer / Wikimedia Cloud Services
Wikimedia Found
On 5/28/19 8:11 PM, Arturo Borrero Gonzalez wrote:
> Hi!
>
> On 2019-06-03 UTC+2 14:00 (next monday) we will be rebuilding the
> cloudservices1003 server,
> that holds the designate service which serves DNS request for CloudVPS and
> Toolforge.
>
> We have a backup server
gards.
[0] https://phabricator.wikimedia.org/T226778
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
On 7/23/19 8:31 PM, Arturo Borrero Gonzalez wrote:
> Hi there!
>
> There is an ongoing maintenance in the eqiad datacenter that involves changing
> power connectors of the servers. More info in this phabricator task: T226778
> [0].
[..]
>
> [0] https://phabricator.wikimedia
see
any problems related to this, please contact us.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https
prevents other major issues :-P
Bonus: some people say that the industry standard is 100 days as the maximum
uptime you may have in your servers. Some unix tools (like htop) will warn you
if the uptime is >100 (only an asterisk though).
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia
://phabricator.wikimedia.org/T153468
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
PROJECT: tools
VM: tools-sgewebgrid-lighttpd-0917 PROJECT: tools
VM: tools-sgewebgrid-lighttpd-0909 PROJECT: tools
VM: tools-sgeexec-0925 PROJECT: tools
VM: tools-sgeexec-0923 PROJECT: tools
VM: tools-sgeexec-0910 PROJECT: tools
VM: cyberbot-db-01 PROJECT: cyberbot
regards.
--
Arturo Borrero Gonzalez
Hi,
a remainder, this is happening now!
On 10/2/19 11:02 AM, Arturo Borrero Gonzalez wrote:
> Hi there,
>
> Next Wednesday 2019-10-09 at 09:00 UTC we will be doing a maintenance
> operation
> on some of our cloudvirt servers (the hypervisor servers) that involves
> rebooting
PROJECT: fastcci
VM: cvn-app8 PROJECT: cvn
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https
On 10/9/19 1:45 PM, Arturo Borrero Gonzalez wrote:
> Hello!
>
> Next Wednesday 2019-10-16 at 09:00 UTC we will be doing another maintenance
> operation on some of our cloudvirts servers (the hypervisor servers) that
> involves rebooting both the physical servers and the virtual m
-instance PROJECT: videowiki
VM: tools-sgeexec-0906 PROJECT: tools
VM: mwoffliner5 PROJECT: mwoffliner
regards
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud
On 10/16/19 12:42 PM, Zoran Dori wrote:
> Hi,
> you said 4 servers but also you said cloudvirt1028, cloudvirt1029 and
> cloudvirt1030. Where is fourth?
>
That's a typo. Sorry for that. We are rebooting *3* cloudvirts.
Good catch! :-P
regards.
--
Arturo Borrero Gonzalez
SRE
Follow-up:
We just discovered cloudvirt1014 doesn't require reboot, so this operation is
only for cloudvirt1025 and cloudvirt1026.
regards.
On 10/16/19 1:10 PM, Arturo Borrero Gonzalez wrote:
> Hello!
>
> Next Wednesday 2019-10-23 at 09:00 UTC we will be doing another maintenan
hings not affected by this change:
* webservices backend operations
* SSH bastions
* grid queues, grid jobs
* wiki-replicas, toolsdb
* other CloudVPS projects
regards.
[0] https://phabricator.wikimedia.org/T235627
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Found
On 10/16/19 3:34 PM, Arturo Borrero Gonzalez wrote:
> Follow-up:
>
> We just discovered cloudvirt1014 doesn't require reboot, so this operation is
> only for cloudvirt1025 and cloudvirt1026.
>
Reminder:
this is happening in a few minutes!
regards
--
Arturo Borrero Gonz
is why it is in scope.
>
> We sincerely apologize for the short notice.
>
Reminder, this is happening in a few minutes!
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
On 10/21/19 7:56 PM, Martin Urbanec wrote:
> Is there something you missed to say?
>
> "operation which is migrating data stored in Redis which can be tricky. The o"
That's a typo/leftover from me rewording that sentence.
Sorry for that :-)
--
Arturo Borrero Gonzal
On 10/21/19 12:16 PM, Arturo Borrero Gonzalez wrote:
> Hi there!
>
> Next Monday 2019-10-28 @ 14:30 UTC we will do a maintenance operation on
> Toolforge which consists in rebuilding the main front proxy [0] used to serve
> webservices. We expect this to be done within a 30
ntries corresponding to the window of this operation.
Other CloudVPS projects users of NFS (dumps shares, maps, etc) might also
require some checking. Please get in touch if you are a project admin of such
project.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Servic
l be disabled (probably for 20 to
> 30
> minutes.) There may also be brief network interruptions during the upgrade.
>
> Toolforge and existing VMs should be largely unaffected apart from possible
> network hiccups.
>
Reminder,
this will be happening in about 30 minutes!
;])/float(optionlist['wingrows']['left'])))
ZeroDivisionError: float division by zero
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
oothly.
As of this email, we don't have any particular metrics or insights on proxies
performances and this is something we could explore in the near future (create a
specific grafana dashboard or something).
regards.
--
Arturo Borrero Gonzalez
SR
rg/wiki/News/2020_Kubernetes_cluster_migration#Lower_default_resource_limits_for_webservice
hope that helps.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.w
Please reach out for any questions or comments.
regards.
[0] https://phabricator.wikimedia.org/T135046
[1] https://openstack-browser.toolforge.org/project/project-proxy
[2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/583098
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Servi
more details:
https://wikitech.wikimedia.org/wiki/News/CloudVPS_NAT_change
Please reach out if you have any doubts, questions, or any other issue.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia
On 4/6/20 8:00 PM, Arturo Borrero Gonzalez wrote:
> Hi there!
>
> In a few days from now (2020-04-13), the CloudVPS network will see a change
> happening that will likely go unnoticed, but it is important enough to share
> it
> with you beforehand.
>
> We will be changi
expect to keep serving legacy URLs forever, by means of redirections to the new
URLs. More information on the redirections can also be found in the wikitech
page.
The toolforge.org domain is finally here! <3
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wi
re
secure approach to host each tool webservice, from an all-shared domain to a
domain per tool.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (for
in CloudVPS, please contact
us.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
On 4/1/20 2:16 PM, Arturo Borrero Gonzalez wrote:
> Hi there!
>
> If you use a CloudVPS web proxy, this email is for you. Toolforge
> developers/users can ignore this email.
>
> We are introducing a change to eliminate the 'X-Forwarded-For' HTTP header
> that
>
://signatures.toolforge.org
https://templatedata-filler.toolforge.org
https://wordcount.toolforge.org
Please reach out for any comments, doubts or questions.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi there!
We just deployed tesseract-ocr v4.1.1 in the Toolforge grid.
The context of this update is the phabricator task T247422 [0].
Please report any issue you may find.
regards!
[0] https://phabricator.wikimedia.org/T247422
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
https://xtools.toolforge.org/
https://ytcleaner.toolforge.org/
https://zppixbot.toolforge.org/
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerl
gards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
to go in this case:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid
Run your script with jsub and it will be scheduled in a grid worker node to run
until it finishes.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
__
] https://en.wikipedia.org/wiki/Message_transfer_agent
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services announce mailing list
cloud-annou...@lists.wikimedia.org (formerly labs-annou
://wikitech.wikimedia.org/wiki/News/Toolforge.org
Please reach out if you need help or have doubts.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab
On 2020-07-06 17:59, Arturo Borrero Gonzalez wrote:
> Hi there!
>
> Tomorrow 2020-07-06 at about 10:00 UTC we will enable the legacy redirector
> and
> this migration will be completed.
>
> All requests to tools.wmflabs.org/ will be permanently redirected to
> .toolfor
Hi there,
we need to perform some unscheduled keystone maintenance right now.
Authentication to some cloud services, in particular Horizon, might be
interrupted during this maintenance period. We expect such maintenance to don't
last more than 1h.
regards.
--
Arturo Borrero Gonzale
edge network. You can find additional details in Phabricator [0].
regards.
[0] https://phabricator.wikimedia.org/T265288
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud
On 2020-10-22 17:41, Arturo Borrero Gonzalez wrote:
> Hi!
>
> There will be a general CloudVPS network maintenance on 20202-10-29, from
> 16:00
> UTC to 17:00 UTC.
>
> During the operation window, all cloud services might be intermittently down,
> inaccessible.
>
&
On 2020-10-29 16:59, Arturo Borrero Gonzalez wrote:
> On 2020-10-22 17:41, Arturo Borrero Gonzalez wrote:
>> Hi!
>>
>> There will be a general CloudVPS network maintenance on 20202-10-29, from
>> 16:00
>> UTC to 17:00 UTC.
>>
>> During the
On 2020-10-30 00:04, Maarten Dammers wrote:
> Hi Arturo,
>
> On 29-10-2020 18:30, Arturo Borrero Gonzalez wrote:
>> Let us know if you see anything weird matching the timing or somehow related
>> to
>> this operation window.
>
> This was announced as n
the
CloudVPS network to the internet.
Sorry for the short notice, we couldn't avoid scheduling this to today.
regards.
[0] https://phabricator.wikimedia.org/T265288
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Found
https://phabricator.wikimedia.org/T268669
The operation may take something between 30 minutes and 1 hours, and we are
starting soon after I finish sending this email.
Please, ping us if you see anything wrong.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Found
#wikimedia-cloud or in the cloud@lists.wikimedia.org [1]
mailing list.
regards.
[0] https://phabricator.wikimedia.org/T263284
[1] https://lists.wikimedia.org/mailman/listinfo/cloud
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia
On 12/10/20 1:33 PM, Arturo Borrero Gonzalez wrote:
Hi there!
Today 2020-12-10 @ 15:30 UTC we will perform an upgrade of the Toolforge
kubernetes cluster [0].
We don't expect any major disruption of the service, but we detected in past
upgrades that some components might be rest
r yet.
Thanks, best regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/lis
measures, somehow?
Hi,
do you know where this limit configuration can be found?
thanks for the heads up.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud
lease share a link to gerrit so I can have such patch in my radar?
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wi
On 1/25/21 11:55 AM, Arturo Borrero Gonzalez wrote:
Hello,
we are planning to change how Cloud VPS instances and Toolforge tools contact
WMF-hosted wikis, in particular the source IP address for the network connection.
The new IP address that wikis will see is 185.15.56.1.
The change is
/Help:Cloud_Services_Introduction#Communication_and_support
[2] https://phabricator.wikimedia.org/T272397
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab
e
future as we deprecate such domain.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
operation can be found in phabricator [0] and in
wikitech [1].
Regards.
[0]https://phabricator.wikimedia.org/T270704
[1]
https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/2020_Network_refresh
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
ef and everything worked.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Wikimedia Cloud Services mailing list
Cloud@lists.wikimedia.org (formerly lab...@lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud
On 5/3/21 11:27 AM, Arturo Borrero Gonzalez wrote:
Hello there,
We will be doing an upgrade to the CloudVPS edge network Thursday 2021-05-06 @
15:00 UTC that will likely impact user experience, including Toolforge.
We scheduled an 1h operation window. During that time, intermittent network
On 5/6/21 5:00 PM, Arturo Borrero Gonzalez wrote:
Reminder, this is happening now!
See you on the other side :-)
Hello from the other side.
This is now done. Sorry for the bumpy ride in Toolforge bastions.
regards
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia
On 5/7/21 7:40 AM, Sascha Brawer wrote:
Curious, does the Wikimedia cloud have some kind of monitoring system that could
have noticed and send an alert?
Yeah, we have monitoring. We could always do better with monitoring in general,
of course.
regards.
--
Arturo Borrero Gonzalez
SRE
including Toolforge tools, PAWS, and any other Cloud VPS
project using them.
More information can be found on phabricator:
https://phabricator.wikimedia.org/T286614
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
://phabricator.wikimedia.org/T294853
Sorry for the inconvenience.
regards.
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Cloud-announce mailing list -- cloud-annou...@lists.wikimedia.org
List information:
https
ation. It is true, wmcloud.org is the
current domain and wmflabs.org is considered 'legacy' and in the [slow]
process of being removed.
Hey @Tim, can you point to documentation or some information that needs
updates that could be source of confusion in the future?
thanks, regards.
--
A
things
would have been much slower and difficult for us.
Your concern about doing the migration dance twice is 100% valid, and
the only way to future-proof your tool is to remove dependency on
GridEngine and migrate it to the Kubernetes backend.
regards.
--
Arturo Borrero Gonzalez
Site Re
to the image?
See some documentation here:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Python#Kubernetes_python_jobs
I just created it, and may need some polishing, but it should work!
We will review pywikibot specific workflows and documents soon.
regards.
--
Arturo Borrero Gonzalez
email.
--
Arturo Borrero Gonzalez
Site Reliability Engineer
Wikimedia Cloud Services
Wikimedia Foundation
___
Cloud mailing list -- cloud@lists.wikimedia.org
List information:
https://lists.wikimedia.org/postorius/lists/cloud.lists.wikimedia.org/
exists, and you will see a warning if you use `containers`.
3) when listing images, the table header no longer mentions "Docker".
These changes should be mostly cosmetic, and no functional or behavioral
change is expected.
Please report any problems you may find.
regards.
--
Artu
email
notice), and some unexpected hiccups occurred. That's why the email today.
regards.
--
Arturo Borrero Gonzalez
Site Reliability Engineer
Wikimedia Cloud Services
Wikimedia Foundation
___
Cloud mailing list -- cloud@lists.wikimedia.org
List i
ide of the
transition.
I'll send another note when we finish this network maintenance is over.
regards.
[0] https://phabricator.wikimedia.org/T316284
[1] https://bugs.debian.org/989162
--
Arturo Borrero Gonzalez
Senior Site Reliability Engineer
Wikimedia Cloud Services
Wikim
On 10/6/22 12:04, Arturo Borrero Gonzalez wrote:
Hi there,
We are currently working on replacing older hardware servers with newer
ones, in particular those dedicated to cloud networking [0].
We have discovered a few shortcomings related mostly to network
interface naming in the newer
a843286ea8
Name: toolsbeta-docker-imagebuilder-01
- ID: 416f445a-cad4-45c2-b32e-f17100f93eac
Name: cloud-puppetmaster-05
- ID: 4e492051-25a3-4442-b8b9-1959f42917fe
Name: tools-k8s-worker-76
- ID: df18863a-2da7-4951-aa69-936b3d889592
Name: deployment-docker-cpjobqueue01
--
Arturo Borrero Gonza
;, and "No exceptions happened".
There was a problem in the way the puppet errors were calculated that
has been now fixed [0].
This does not affect Toolforge.
sorry for the noise,
regards.
[0] https://gerrit.wikimedia.org/r/c/operations/puppet/+/861805/
--
Arturo Borrero Gonzalez
Sen
less).
During that time, using the toolforge-jobs command line interface will most
likely fail.
regards.
[0] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework
--
Arturo Borrero Gonzalez
Senior SRE / Wikimedia Cloud Services
Wikimedia Foundation
s, in
particular Taavi (community member) and Raymond (WMF contractor).
Happy `toolforging`. Regards.
--
Arturo Borrero Gonzalez
Senior SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Cloud mailing list -- cloud@lists.wikimedia.org
List inform
gards.
[0] https://phabricator.wikimedia.org/T328539
--
Arturo Borrero Gonzalez
Senior SRE / Wikimedia Cloud Services
Wikimedia Foundation
___
Cloud mailing list -- cloud@lists.wikimedia.org
List information:
https://lists.wikimedia.org/postorius/
;re on the latest available versions. The vast
majority of tools that are only using the Jobs framework and/or the webservice
command are not affected by these changes.
This has been rescheduled to Monday 2023-04-10 to leave room for the other
operations we have.
regards.
--
Arturo Borrero Gonza
On 3/30/23 12:42, Arturo Borrero Gonzalez wrote:
On 3/28/23 00:13, Taavi Väänänen wrote:
Hi,
We will be upgrading the Toolforge Kubernetes cluster next Monday (2023-04-03)
starting at around 10:00 UTC.
The expected impact is that tools running on the Kubernetes cluster will get
restarted a
1 - 100 of 113 matches
Mail list logo
