[osuosl-openpower] Controller OS Upgrade Notification

2025-03-13 Thread Lance Albertson
collaborating with the AlmaLinux project to restore this support in AlmaLinux 9. Once these issues are resolved, we will proceed with upgrading the POWER9 and POWER10 nodes to AlmaLinux 9. Thank you for your understanding and patience. -- Lance Albertson Director Oregon State University | Open Source

[osuosl-openpower] Firmware upgrade on POWER10 systems

2025-01-27 Thread Lance Albertson
eryone a heads up in case something goes wrong. I'll be starting this upgrade later this morning. Thanks- -- Lance Albertson Director Oregon State University | Open Source Lab ___ openpower mailing list openpo...@osuosl.org https://lists.osuos

Re: [osuosl-openpower] NVMe failure affecting some VMs

2025-01-17 Thread Lance Albertson
Apologies again for this situation! Thanks- On Tue, Jan 14, 2025 at 11:03 AM Lance Albertson wrote: > All, > > One of our nodes which uses local storage with NVMe has had a failure and > all the VMs on that node are offline. Due to the way we had storage > configured on this node f

[osuosl-openpower] NVMe failure affecting some VMs

2025-01-14 Thread Lance Albertson
still be intact since they are on our Ceph cluster. I'm going to be working on getting the NVMe replaced, in the meantime, if you need us to rebuild any of your VMs, please let us know. Thanks and sorry again for the issue. -- Lance Albertson Director Oregon State University | Open

Re: [osuosl-openpower] OpenStack Yoga Upgrade - TODAY

2024-12-19 Thread Lance Albertson
I believe everything should be back to normal now. Please let me know if that isn't the case. Sorry again for the unexpected outage. On Thu, Dec 19, 2024 at 2:30 PM Lance Albertson wrote: > I've encountered an issue which caused network connectivity to be lost on > the VMs

Re: [osuosl-openpower] OpenStack Yoga Upgrade - TODAY

2024-12-19 Thread Lance Albertson
I've encountered an issue which caused network connectivity to be lost on the VMs. I'm working on resolve it right now. On Thu, Dec 19, 2024 at 11:32 AM Lance Albertson wrote: > Attention OpenStack Users, > > We will be performing an upgrade of our cluster from the Xena to

[osuosl-openpower] OpenStack Yoga Upgrade - TODAY

2024-12-19 Thread Lance Albertson
and cooperation. -- Lance Albertson Director Oregon State University | Open Source Lab ___ openpower mailing list openpo...@osuosl.org https://lists.osuosl.org/mailman/listinfo/openpower

[osuosl-openpower] OpenStack Xena Upgrade - TODAY

2024-12-11 Thread Lance Albertson
and cooperation. -- Lance Albertson Director Oregon State University | Open Source Lab ___ openpower mailing list openpo...@osuosl.org https://lists.osuosl.org/mailman/listinfo/openpower

[osuosl-openpower] OpenStack Wallaby Upgrade - TODAY

2024-12-09 Thread Lance Albertson
understanding and cooperation. -- Lance Albertson Director Oregon State University | Open Source Lab ___ openpower mailing list openpo...@osuosl.org https://lists.osuosl.org/mailman/listinfo/openpower

[osuosl-openpower] OpenStack Victoria Upgrade - TODAY

2024-12-04 Thread Lance Albertson
understanding and cooperation. -- Lance Albertson Director Oregon State University | Open Source Lab ___ openpower mailing list openpo...@osuosl.org https://lists.osuosl.org/mailman/listinfo/openpower

[osuosl-openpower] OpenStack Ussuri Upgrade - TODAY

2024-11-25 Thread Lance Albertson
utage for those VMs. However, the API interface will be down and partially available during this time period so new VM creation/deletion/etc will not work. If you have any questions or concerns, please let me know. -- Lance Albertson Director Oregon State University | Open Sourc

[Hosting] 2024 Annual OSUOSL Survey

2024-10-01 Thread Lance Albertson
our OpenPOWER hosting, please fill this out as well. It covers a different set of questions. Feel free to have multiple people from your project fill out this survey if they want to provide feedback. Thank you for your continued collaboration and support! Thanks! -- Lance Albertson Director Oregon

[osuosl-openpower] Now Providing Access to POWER10 for Open Source Projects

2024-08-19 Thread Lance Albertson
est to powerdev-requ...@osuosl.org. Make sure you put your project name and "POWER10" in the subject line so we know that this is specifically for this. Thanks to IBM for making this happen! -- Lance Albertson Director Oregon State University | Open Source Lab ___

Re: [Hosting] Potential planned 8hr network outage 08/13-08/14

2024-08-08 Thread Lance Albertson
gain! On Mon, Aug 5, 2024 at 10:33 AM Lance Albertson wrote: > I just got done having the call with LinkOregon and here are the updates I > know so far: > >1. They will be doing an on-site meeting on Wednesday with OSU >electricians to see if they can find some power to k

Re: [Hosting] Potential planned 8hr network outage 08/13-08/14

2024-08-05 Thread Lance Albertson
ge next Tuesday evening PDT. Apologies for not being able to send this out sooner. I brought up the fact that we were not notified until last week even though this apparently had been planned for months. Thanks again for your support and patience! On Fri, Aug 2, 2024 at 2:10 PM Lance Albertson

[Hosting] Potential planned 8hr network outage 08/13-08/14

2024-08-02 Thread Lance Albertson
any adjustments you can ahead of time. If you have any other questions or concerns, please contact me directly. -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.osuosl.org/m

[osuosl-openpower] ACTION REQUIRED: 2024 OSL OpenPOWER Cluster Survey

2024-05-13 Thread Lance Albertson
s for new projects if we don't hear back. Thank you sincerely, The OSU Open Source Lab OpenPOWER Team [1] https://forms.gle/D7PyAY4vYnTgmSLfA -- Lance Albertson Director Oregon State University | Open Source Lab ___ openpower mailing list openpo...@

[Hosting] FTP Server Rebuild - March 2024

2024-03-08 Thread Lance Albertson
indow:* Currently proposed schedule for the servers: - ftp-chi: 03/11/2024 - 03/15/2024 - ftp-nyc: 03/18/2024 - 03/22/2024 - ftp-osl: 04/01/2024 - 04/05/2024 *Projects affected:* All projects with hosted content on our FTP mirror network. -- Lance Albertson Director Oregon State University |

[Hosting] Issues with Email Relays (fixed now)

2024-02-03 Thread Lance Albertson
look and see how I can redeploy the change again but ensure it doesn't break any mail in the process. Thanks for your patience and sorry for the issues this may have caused. -- Lance Albertson Director Oregon State University | Open Sourc

Re: [osuosl-openpower] Controller and Hypervisor OS Upgrades

2024-01-04 Thread Lance Albertson
next. If there are any lingering issues, please let me know. On Tue, Jan 2, 2024 at 5:07 PM Lance Albertson wrote: > All, > > This month we will be going through all of our controller and hypervisor > nodes and upgrading them to AlmaLinux 8 from CentOS 7. Starting tomorrow,

[osuosl-openpower] Controller and Hypervisor OS Upgrades

2024-01-02 Thread Lance Albertson
between all the various nodes we need to do this on. If you have any questions, please let me know. -- Lance Albertson Director Oregon State University | Open Source Lab ___ openpower mailing list openpo...@osuosl.org https://lists.osuosl.org/mailman/listinfo/openpower

Re: [osuosl-openpower] OpenStack Train Upgrade - December 19, 2023 1-3PM PST (2100-2300 UTC)

2023-12-19 Thread Lance Albertson
This upgrade has been completed. Please let us know if you notice any issues. Thanks- On Tue, Dec 19, 2023 at 2:54 PM Lance Albertson wrote: > This upgrade is taking longer than expected. We have finished the upgrade > on the controller node which had the largest impact but I'm st

Re: [osuosl-openpower] OpenStack Train Upgrade - December 19, 2023 1-3PM PST (2100-2300 UTC)

2023-12-19 Thread Lance Albertson
This upgrade is taking longer than expected. We have finished the upgrade on the controller node which had the largest impact but I'm still working through the compute nodes. I imagine this will take another hour or so. Thanks- On Mon, Dec 18, 2023 at 12:52 PM Lance Albertson wrote: >

[osuosl-openpower] OpenStack Train Upgrade - December 19, 2023 1-3PM PST (2100-2300 UTC)

2023-12-18 Thread Lance Albertson
uster backend nodes to AlmaLinux 8 which will result in a larger outage. We will send out an announcement for that in a few weeks with more details. If you have any questions or concerns, please let me know. -- Lance Albertson Director Oregon State University | Open Sourc

[osuosl-openpower] Endpoint DNS changes 12/18/2023 10am-12pm PST

2023-12-13 Thread Lance Albertson
npower-openstack.osuosl.org <http://openpower-openstack.osuosl.org>*. We will be also making a redirect to make this easier for you. If you do not use the API or the web portal, you do not need to make any changes. -- Lance Albertson Director Oregon State University | Ope

Re: [Hosting] Planned Outage: Network Uplink Redundancy Setup - 1300-1400 PDT Oct 31, 2023

2023-10-31 Thread Lance Albertson
s and thanks for your patience. > > On Mon, Oct 30, 2023 at 10:27 AM Lance Albertson wrote: > >> All, >> >> Tomorrow we're planning to _finally_ change our uplink to LinkOregon to a >> redundant link (at least within our switching infrastructure). There will

Re: [Hosting] Planned Outage: Network Uplink Redundancy Setup - 1300-1400 PDT Oct 31, 2023

2023-10-31 Thread Lance Albertson
30, 2023 at 10:27 AM Lance Albertson wrote: > All, > > Tomorrow we're planning to _finally_ change our uplink to LinkOregon to a > redundant link (at least within our switching infrastructure). There will > likely be a few minutes of downtime while we change the configuration ov

[Hosting] Planned Outage: Network Uplink Redundancy Setup - 0900-1000 PDT Oct 31, 2023

2023-10-30 Thread Lance Albertson
#x27;re in the clear. Thanks for your support and patience! -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.osuosl.org/mailman/listinfo/hosting

[Hosting] Server reboots for updates

2023-09-29 Thread Lance Albertson
nly last a few minutes per reboot. Thanks for your patience and support! -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.osuosl.org/mailman/listinfo/hosting

[Hosting] Brief Internet Outage (5:55 - 6:10AM PDT)

2023-08-29 Thread Lance Albertson
s a separate datacenter from where our equipment is hosted, however it is in the same building. As far as I can tell, our systems were not directly impacted by this power event. [1] https://sn-tools.grnoc.iu.edu/public-task-viewer/?method=show_task&task_id=INC0150376 -- Lance Albertson Direc

[Hosting] oslsw3 core switch migration

2023-08-25 Thread Lance Albertson
s. If you have any questions or concerns feel free to reach out via email or IRC in #osuosl on Librachat. Thanks for your support! -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@o

[Hosting] oslsw1 host migrationg (final one!)

2023-08-10 Thread Lance Albertson
acted: almalinux-p9 ci-ppc64le-runner1 debian-riscv64-porterbox-01 gcc113 gcc114 gcc115 gcc116 gcc117 gcc118 openpower11 openpower12 openpower-gpu2 p8-java1 rockylinux-p8 rv-osuosl-01 rv-osuosl-02 rv-osuosl-03 rv-osuosl-04 rv-osuosl-05 Thanks- -- Lance Albertson Director Oregon State University |

Re: [Hosting] Maintenance - VPN Migration - 2023-08-07

2023-08-07 Thread Lance Albertson
7:13 PM Lance Albertson wrote: > > *Summary:* > > We are migrating services running on a legacy system which hosts OpenVPN > and routing for some of our NAT networks to a newer set of machines. This > will also upgrade our OpenVPN server to a newer version along with >

[Hosting] Maintenance - VPN Migration - 2023-08-07

2023-08-01 Thread Lance Albertson
- NAT routers for the following networks: - 10.0.0.0/23 - 10.1.0.0/23 - 10.1.50.0/24 - Hosts connected the subnets above will experience a brief outage -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list

Re: [Hosting] oslsw1 host migration - June 29-30 2023

2023-06-30 Thread Lance Albertson
mind that we still have a set of hosts on a third switch that still need to be moved, but this is waiting on us getting a replacement switch in place which won't happen until mid-July. Thanks for your patience and continued support! On Thu, Jun 29, 2023 at 5:02 PM Lance Albertson wrote: &

Re: [Hosting] oslsw1 host migration - June 29-30 2023

2023-06-29 Thread Lance Albertson
The moves today went well however we were not able to complete all the moves today. While we were originally planning on doing the next batch tomorrow, it will need to be postponed until Monday due to a scheduling conflict. Thanks! On Tue, Jun 27, 2023 at 10:45 AM Lance Albertson wrote: >

[Hosting] oslsw1 host migration - June 29-30 2023

2023-06-27 Thread Lance Albertson
lacement switch. Thank you for your patience! -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.osuosl.org/mailman/listinfo/hosting

[Hosting] [OUTAGE]: Backbone switch reboot - Jun 8, 2023 2:02-2:40PM PDT (2102-2140 UTC)

2023-06-08 Thread Lance Albertson
e can mitigate this in the future. We have resolved any remaining issues that were related to this. Thanks for your patience. -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org

Re: [Hosting] mysql1-vip (mysql1/mysql2) OS Upgrade | May 24th and 31st, 2023 20:00-24:00 UTC

2023-05-26 Thread Lance Albertson
o contact us directly. > > Thanks! > > -- > Ethan Hampton > Student Systems Engineer > Oregon State University | Open Source Lab > > ___ > Hosting mailing list > host...@osuosl.org

[Hosting] LinkOregon Planned Network Outages (03/27/2023 - 03/29/2023)

2023-03-28 Thread Lance Albertson
/sn-tools.ctc.grnoc.iu.edu/public-task-viewer/index.cgi?method=show_task&task_id=CHG0067260 Apologies for any issues this may cause and again for not sending this out sooner! Thanks- -- Lance Albertson Director Oregon State University | Open Source Lab __

[Hosting] Update on the NXDOMAIN DNS DDoS issues

2023-02-12 Thread Lance Albertson
an issue, please let us know by sending an email to supp...@osuosl.org. Or if you or noticing any issues since we've migrated to dnsdist in front of BIND. Thank you to everyone who pointed us in the right direction with using dnsdist. -- Lance Albertson Director Oregon State University |

[Hosting] [MAINTENANCE] Deployment of dnsdist on DNS servers

2023-02-09 Thread Lance Albertson
y problems, please let us know by sending an email to supp...@osuosl.org or pingging us on IRC. If all goes well, we'll deploy it on the two other servers (ns2 and ns3) tomorrow. Thanks for your patience and thanks for all of the help folks offered during all of this! -- Lance Albertson Direct

[Hosting] [UPDATE] NXDOMAIN DNS DDoS Issues

2023-01-26 Thread Lance Albertson
host, please reply off list and share your configuration and setup. Thanks! [1] https://dnsdist.org/ -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.osuosl.org/mailman

[Hosting] NXDOMAIN DNS DDoS Issues

2023-01-19 Thread Lance Albertson
If anyone has experience mitigating NXDOMAIN attacks using ISC BIND, please send me a direct email to see if there are any additional measures we can take. Thanks for your patience. -- Lance Albertson Director Oregon State University | Open Source Lab _

Re: [Hosting] MAINTENANCE: Core network migrations - Thursday, Dec 15 2022 2100 UTC

2022-12-15 Thread Lance Albertson
:30 PM Lance Albertson wrote: > All, > > Tomorrow starting at 1PM PST (2100 UTC) we will be migrating various fiber > uplinks to a "new" core router. There will be several small outages while I > move fiber to the "new" core router. I expect each outage should

[Hosting] MAINTENANCE: Core network migrations - Thursday, Dec 15 2022 2100 UTC

2022-12-14 Thread Lance Albertson
less if we don't run into any issues. I will send an email once everything has been finished. Please let me know if you have any questions or concerns! Thanks- -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailin

[Hosting] [OUTAGE]: Backbone switch reboot - Dec 10, 2022 5:01-5:08AM PST (1301-1308 UTC)

2022-12-10 Thread Lance Albertson
ut that when I'm ready to make that happen. Thanks and Happy Holidays everyone! -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.osuosl.org/mailman/listinfo/hosting

[Hosting] NOTICE: Campus-wide power outage (currently on generator power)

2022-11-26 Thread Lance Albertson
the few remaining systems we have running on the OCP hardware in another building that does not have generator power. If you have any questions, please let me know. Thanks- [1] https://oregonstate.edu/alert/campus-power-outage -- Lance Albertson Director Oregon State University | Open Source Lab

Re: [Hosting] [RESOLVED] DDoS on OSL DNS Servers

2022-10-31 Thread Lance Albertson
of traffic for IPv4 and IPv6 during the event. If anyone else has some recommended Bind configuration you use to mitigate this, please let me know off list. Thanks for your patience. [1] https://kb.isc.org/docs/cve-2020-8616 On Fri, Oct 28, 2022 at 10:39 PM Lance Albertson wrote: > All, &g

[Hosting] [RESOLVED] DDoS on OSL DNS Servers

2022-10-28 Thread Lance Albertson
et me know via an email to supp...@osuosl.org Thank you! -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.osuosl.org/mailman/listinfo/hosting

Re: [Hosting] RESOLVED: Core network switch reboot

2022-10-28 Thread Lance Albertson
e will be another brief outage when we switch over to the new "core" switches with MLAG. Thanks again for your patience. Hopefully I can get these all done before this switch decides to reboot again! On Mon, Sep 12, 2022 at 11:54 PM Lance Albertson wrote: > Sadly this just happened

Re: [Hosting] Core network switch reboot

2022-09-13 Thread Lance Albertson
This has been completed and everything seems to be working fine. Now keep in mind, the troublesome switch could reboot again until we figure out why it's happening. If it does, it's impact should be smaller than before at least. Thanks! On Tue, Sep 13, 2022 at 2:42 PM Lance Albert

Re: [Hosting] Core network switch reboot

2022-09-13 Thread Lance Albertson
witch 3. Move oslsw1 uplink to "new" switch 4. Move remaining backend 10g switches If anything goes wrong, I should be able to quickly revert the change. On Tue, Sep 13, 2022 at 11:08 AM Lance Albertson wrote: > All, > > I wanted to pass along more information on where we're a

Re: [Hosting] Core network switch reboot

2022-09-13 Thread Lance Albertson
working vendor, please let me know. That will certainly help our situation in the long term! I had already started working on a plan to replace these systems but it seems my time may have run out (at least for the Nexus 6k switch). Thanks all for your patience and support! On Mon, Sep 12, 2022

Re: [Hosting] On going IPv6 issues with some subnets

2022-09-13 Thread Lance Albertson
I have confirmed this has fixed the problem for IPv6 on the affected subnets. On Mon, Sep 12, 2022 at 3:15 PM Lance Albertson wrote: > All, > > LinkOregon will be pushing a fix for this tonight at midnight PDT. > > On Wed, Sep 7, 2022 at 2:22 PM Lance Albertson wrote: > >

Re: [Hosting] RESOLVED: Core network switch reboot

2022-09-12 Thread Lance Albertson
orrow morning, I'll send an update on what we're planning to do. Thanks again for your patience. On Mon, Sep 12, 2022 at 3:14 PM Lance Albertson wrote: > This happened again at approximately 10AM PDT. Since we moved our uplink > to this switch, everything went down while the switc

Re: [Hosting] On going IPv6 issues with some subnets

2022-09-12 Thread Lance Albertson
All, LinkOregon will be pushing a fix for this tonight at midnight PDT. On Wed, Sep 7, 2022 at 2:22 PM Lance Albertson wrote: > All, > > I wanted to send you an update on where we're at with the migration. We > appear to have hit some kind of bug on the LinkOregon side where

Re: [Hosting] RESOLVED: Core network switch reboot

2022-09-12 Thread Lance Albertson
On Wed, Aug 24, 2022 at 7:40 AM Lance Albertson wrote: > Unfortunately this just happened again overnight. We may need to schedule > another outage to perform some software upgrade on this switch so that this > stops happening. We'll send an announcement out once we have everything

[Hosting] On going IPv6 issues with some subnets

2022-09-07 Thread Lance Albertson
additional questions or issues since our change yesterday. Thanks! On Tue, Sep 6, 2022 at 6:04 PM Lance Albertson wrote: > All, > > I have an update on some issues we did discover after the fact. For > now, everything with IPv4 seems to be 100% OK. Please let me know if > that

Re: [Hosting] MAINTENANCE: Network Uplink migration to LinkOregon - Tuesday, Sep 6 2200 UTC

2022-09-06 Thread Lance Albertson
am missing any IPv6 subnet that isn't working, please let me know. Since this impacted a smaller set of users, I decided to move forward keeping us on the new connection while we work out these IPv6 issues and not roll back. Thanks again for all your support and patience! On Tue, Sep 6, 2022 a

Re: [Hosting] MAINTENANCE: Network Uplink migration to LinkOregon - Tuesday, Sep 6 2200 UTC

2022-09-06 Thread Lance Albertson
We should be back online now. Please let us know if there are any issues. On Tue, Sep 6, 2022 at 3:01 PM Lance Albertson wrote: > > We're going to be doing this outage here in the next 5-10 minutes. Wish us > luck! > > On Tue, Aug 30, 2022 at 3:12 PM Lance Albertson

Re: [Hosting] MAINTENANCE: Network Uplink migration to LinkOregon - Tuesday, Sep 6 2200 UTC

2022-09-06 Thread Lance Albertson
We're going to be doing this outage here in the next 5-10 minutes. Wish us luck! On Tue, Aug 30, 2022 at 3:12 PM Lance Albertson wrote: > > All, > > Next Tuesday, September 6th between 3-5PM PDT (2200-UTC) we will be > migrating our network uplink from NERO to LinkOrego

[Hosting] MAINTENANCE: Network Uplink migration to LinkOregon - Tuesday, Sep 6 2200 UTC

2022-08-30 Thread Lance Albertson
to the switches. If you have any questions or concerns, please let me know. [1] https://www.linkoregon.org/ -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.osuosl.org/mailman/listinfo/hosting

Re: [Hosting] RESOLVED: Core network switch reboot

2022-08-24 Thread Lance Albertson
22 PM Lance Albertson wrote: > All, > > It appears that one of our core network switches had a kernel panic and > rebooted which caused widespread outages throughout our infrastructure. As > of right now, everything appears to be back to normal but please let me > know if that isn&#

Re: [Hosting] UNPLANNED: Ganeti hypervisor node reboot

2022-08-05 Thread Lance Albertson
-radish.osuosl.org - ldap1.ntf.osuosl.org - mandrivausers2.osuosl.org - scripts.phpbb.com - snowdrift-app.osuosl.org - snowdrift-smtp.osuosl.org On Sat, Jul 30, 2022 at 6:23 PM Lance Albertson wrote: > All, > > One of our production Ganeti nodes (gprod3) decided to reboot on its own &

[Hosting] UNPLANNED: Ganeti hypervisor node reboot

2022-07-30 Thread Lance Albertson
-bugs.osuosl.org - lf-lists.osuosl.org - mageiavm.osuosl.org - ntpsec-service3.osuosl.org - osu1php.osuosl.org - web3.osuosl.org - www1.phpbb.com If you have any other issues related to this, please send an email to support. Thanks- -- Lance Albertson Director Oregon State

Re: [osuosl-openpower] Hypervisor server moves next week

2022-07-05 Thread Lance Albertson
able to migrate VMs without any downtime. Thanks- On Fri, Jul 1, 2022 at 4:32 PM Lance Albertson wrote: > All, > > We need to do some emergency server moves next week starting on Tuesday. > This is to help balance the power and cooling in the datacenter where these > systems are lo

[osuosl-openpower] Hypervisor server moves next week

2022-07-01 Thread Lance Albertson
s for your patience and support! -- Lance Albertson Director Oregon State University | Open Source Lab ___ openpower mailing list openpo...@osuosl.org https://lists.osuosl.org/mailman/listinfo/openpower

[Hosting] Fwd: [Link Oregon Outages] Brief Outage - Link Oregon Various Members | Task Number: INC0114335

2022-06-29 Thread Lance Albertson
noc.iu.edu> -- To unsubscribe from this group and stop receiving emails from it, send an email to outages+unsubscr...@linkoregon.org. ___ systems mailing list syst...@osuosl.org https://lists.osuosl.org/mailman/listinfo/systems -- Lance Albertson Director O

Re: [Hosting] MAINTENANCE: ftp-osl disk backplane replacement

2022-06-24 Thread Lance Albertson
This system is back up and running. All systems should be functioning normally now. On Fri, Jun 24, 2022 at 9:11 AM Lance Albertson wrote: > Taking the system down now. > > On Fri, Jun 24, 2022 at 8:59 AM Lance Albertson wrote: > >> All, >> >> We're g

Re: [Hosting] MAINTENANCE: ftp-osl disk backplane replacement

2022-06-24 Thread Lance Albertson
Taking the system down now. On Fri, Jun 24, 2022 at 8:59 AM Lance Albertson wrote: > All, > > We're going to be replacing a failing disk backplane in one of the mirrors > behind ftp.osuosl.org. Specifically, we'll be working on ftp-osl which is > currently the prima

[Hosting] MAINTENANCE: ftp-osl disk backplane replacement

2022-06-24 Thread Lance Albertson
tly referencing this host. Apologies for not sending a notice out sooner. I'll send a notice when we actually start and complete the maintenance. Thanks for your patience. -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting m

[Hosting] Fwd: [Link Oregon Outages] Maintenance - Router Firmware Upgrade | Task Number: CHG0060265

2022-06-21 Thread Lance Albertson
man/listinfo/systems -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.osuosl.org/mailman/listinfo/hosting

[Hosting] RESOLVED: Core network switch reboot

2022-05-25 Thread Lance Albertson
l.org. Apologies for the outage and we'll be looking into why this switch had a kernel panic in the first place. Thanks- -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.

[Hosting] Email issues sending to lists.osuosl.org since May 12, 2022

2022-05-18 Thread Lance Albertson
few other projects that use that domain. Unfortunately this resulted in bounced emails so those were lost. The change might take a few hours or up to a day to fully propagate. Apologies for any issues this might have caused. Thanks- -- Lance Albertson Director Oregon State University | Open

Re: [osuosl-openpower] Instance migration issues

2022-04-05 Thread Lance Albertson
you still have any issues with your VMs still, please let me know. Thanks! On Tue, Apr 5, 2022 at 11:30 AM Lance Albertson wrote: > All, > > I need to reboot some VM instances today due to an issue we discovered > while doing live migrations last week. Some instances ended up on a >

[osuosl-openpower] Instance migration issues

2022-04-05 Thread Lance Albertson
n the new hypervisor. Thankfully, it appears there is no storage loss as long as we get these instances back on a ceph-enabled hypervisor. I'm working through the instances right now and will send an update once I've completed the migration. -- Lance Albertson Director Oregon State Un

Re: [osuosl-openpower] Hypervisor reboots on NVMe hypervisors

2022-03-25 Thread Lance Albertson
Two VMs did not come back online that I just fixed. Please let me know if there are any other issues. Thanks- On Wed, Mar 23, 2022 at 10:41 AM Lance Albertson wrote: > I need to shift this to 2:30-3:30pm PDT today (2130-2230 UTC) as I have a > meeting conflict. Apologies for the addi

Re: [osuosl-openpower] Hypervisor reboots on NVMe hypervisors

2022-03-23 Thread Lance Albertson
I need to shift this to 2:30-3:30pm PDT today (2130-2230 UTC) as I have a meeting conflict. Apologies for the additional email. On Wed, Mar 23, 2022 at 9:53 AM Lance Albertson wrote: > All, > > For the few of you using the new hypervisors which use local NVMe storage, > we will b

[osuosl-openpower] Hypervisor reboots on NVMe hypervisors

2022-03-23 Thread Lance Albertson
more than 10 minutes each. Since the VMs on these nodes are using local storage, they will be shut down during the upgrade process as we cannot use live migration. If you have any questions or concerns please let me know. Thanks- -- Lance Albertson Director Oregon State University | Open Source

[Hosting] PSA: Please update polkit on all Linux systems ASAP

2022-01-27 Thread Lance Albertson
everyone to enable automatic upgrades on your systems if you're able to. [1] https://blog.qualys.com/vulnerabilities-threat-research/2022/01/25/pwnkit-local-privilege-escalation-vulnerability-discovered-in-polkits-pkexec-cve-2021-4034 [2] https://lwn.net/Articles/882609/ -- Lance Albertson Dir

Re: [osuosl-openpower] Firmware upgrades on NVMe hypervisors

2022-01-12 Thread Lance Albertson
This upgrade has been completed. Please let me know if there are any issues. Thanks- On Mon, Jan 10, 2022 at 4:31 PM Lance Albertson wrote: > All, > > For the few of you using the new hypervisors which use local NVMe storage, > we will be doing some firmware upgrades on bot

[osuosl-openpower] Firmware upgrades on NVMe hypervisors

2022-01-10 Thread Lance Albertson
we cannot use live migration. We will send an email once this has been completed. I expect each hypervisor will only be down for around 30 minutes but it could be longer. If you have any questions or concerns please let me know. Thanks- -- Lance Albertson Director Oregon State University | Open

Re: [osuosl-openpower] Backend storage issues

2021-12-19 Thread Lance Albertson
email powerdev-requ...@osuosl.org to open a ticket. Thanks for your patience! On Sun, Dec 19, 2021 at 10:21 AM Lance Albertson wrote: > All, > > It appears we had an issue with our backend storage (Ceph) overnight that > has caused some of the VMs to be offline. I'm currently worki

[osuosl-openpower] Backend storage issues

2021-12-19 Thread Lance Albertson
tly once I have an update as well. Thanks for your patience! -- Lance Albertson Director Oregon State University | Open Source Lab ___ openpower mailing list openpo...@osuosl.org https://lists.osuosl.org/mailman/listinfo/openpower

Re: [Hosting] Log4j2 Zero-day Vulnerability

2021-12-10 Thread Lance Albertson
Just a heads up, this library gets used in a lot of places for example Apache Solr, ElasticSearch, Logstash and Kafka to name a few. On Fri, Dec 10, 2021 at 6:38 PM Lance Albertson wrote: > All, > > There was a zero-day vulnerability announced today [1] affecting the Log4j > Jav

[Hosting] Log4j2 Zero-day Vulnerability

2021-12-10 Thread Lance Albertson
d.nist.gov/vuln/detail/CVE-2021-44228 -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.osuosl.org/mailman/listinfo/hosting

[osuosl-openpower] Possible service disruption caused backend storage

2021-11-15 Thread Lance Albertson
/O issues. Sorry for the issues that our mistake has caused! Thanks for your patience- -- Lance Albertson Director Oregon State University | Open Source Lab ___ openpower mailing list openpo...@osuosl.org https://lists.osuosl.org/mailman/listinfo/openpower

[Hosting] Fwd: [Link Oregon Outages] Emergency Maintenance - Link Oregon Core Node PORTPTCK | Task Number: CHG0055805

2021-11-09 Thread Lance Albertson
: LINKOR.CHG0055805.0.0 -- To unsubscribe from this group and stop receiving emails from it, send an email to outages+unsubscr...@linkoregon.org. ___ systems mailing list syst...@osuosl.org https://lists.osuosl.org/mailman/listinfo/systems -- Lance Albertson Director

Re: [osuosl-openpower] Power Outage

2021-11-02 Thread Lance Albertson
everything came back without much intervention needed. Thanks for your patience! On Tue, Nov 2, 2021 at 10:42 AM Lance Albertson wrote: > While working on adding new power to a rack next to our gear, the > power was accidentally turned off on our racks affecting all of our > OpenPower clu

[osuosl-openpower] Power Outage

2021-11-02 Thread Lance Albertson
While working on adding new power to a rack next to our gear, the power was accidentally turned off on our racks affecting all of our OpenPower cluster. We're working on bringing those systems back online as we speak. Thanks for your patience. -- Lance Albertson Director Oregon State Unive

[Hosting] Network interruption between 10:34-11:04pm PDT

2021-10-28 Thread Lance Albertson
t. Their work happens to have started and completed > before BGP converged. And it seems as though October continues to be BGP awareness month :-) Apologies for any issues this may have caused. -- Lance Albertson Director Oregon State University | Open

[osuosl-openpower] Hardware issues with P8 hypervisor (openpower2)

2021-10-25 Thread Lance Albertson
at we can reduce any further impacts. Thanks- -- Lance Albertson Director Oregon State University | Open Source Lab ___ openpower mailing list openpo...@osuosl.org https://lists.osuosl.org/mailman/listinfo/openpower

[osuosl-openpower] Hypervisor outage (openpower2)

2021-10-15 Thread Lance Albertson
blems. If you have any questions or concerns, please let me know by sending an email to powerdev-requ...@osuosl.org. Thank you! -- Lance Albertson Director Oregon State University | Open Source Lab ___ openpower mailing list openpo...@osuosl.org

[Hosting] OSL VPN Certificate expired

2021-09-24 Thread Lance Albertson
All, It appears our ca.crt and server.crt certificates have expired for our VPN. I'm working on resolving that and will be sending a new ca.crt soon that everyone will need to use. Thanks- -- Lance Albertson Director Oregon State University | Open Sourc

Re: [Hosting] mysql-vip2 (mysql3/mysql4) 5.6 -> 5.7 Upgrade, August 23, 2021 2-4PM PDT

2021-08-23 Thread Lance Albertson
The upgrade has been completed, but there is an issue with SSL connections. The new version is using self-signed certificates which is causing problems for some clients. We are preparing a fix and will have that rolled out shortly. On Thu, Aug 19, 2021 at 1:25 PM Lance Albertson wrote: >

[Hosting] mysql-vip2 (mysql3/mysql4) 5.6 -> 5.7 Upgrade, August 23, 2021 2-4PM PDT

2021-08-19 Thread Lance Albertson
equire additional steps. Thanks! -- Lance Albertson Director Oregon State University | Open Source Lab ___ Hosting mailing list host...@osuosl.org https://lists.osuosl.org/mailman/listinfo/hosting

Re: [Hosting] Unplanned Power Event

2021-08-05 Thread Lance Albertson
SL managed services. Please let me know if you do have any issues. Thanks- On Tue, Aug 3, 2021 at 3:26 PM Lance Albertson wrote: > I received an update on the issues we had in the primary data center. It > appears that there was a battery cell problem on one of the UPS's. Previou

Re: [Hosting] Unplanned Power Event

2021-08-03 Thread Lance Albertson
know. Thank you! On Sun, Aug 1, 2021 at 12:28 PM Lance Albertson wrote: > I got word that this outage was more campus wide which included impacting > the OpenCompute hosts. I went through those hosts and ensured they are back > online but let me know if I missed anything. > > OSU

Re: [Hosting] Unplanned Power Event

2021-08-01 Thread Lance Albertson
hosts to check. Thanks for your patience! On Sun, Aug 1, 2021 at 8:15 AM Lance Albertson wrote: > It seems as though we had an unplanned power event that happened in our > primary data center early this morning at 3:03AM PDT (1003 UTC) that > affected one of the two power feeds. Virtually

  1   2   3   >