Greetings everyone. I would like to provide a status update on the datacenter move.
On the plus side, we have all our data migrated to the new datacenter. All the databases and storage are available in the new location. Also, many of our services are back and functional. However, the build system pipeline is not yet ready to come back up. We are trying to debug/fix an authentication issue that is preventing ssh git access and lookaside uploads. As soon as that issue is solved, we can continue the build system bringup. That will consist of: * Testing official build for signing, bodhi handling and gating testing * opening koji and pkgs/src ssh. * Testing updates composes * Testing rawhide composes I know packagers are eager to resume their work, hopefully we will have things back up soon. Issues we know about and are working on: * The above dist git auth issues. * mailman is sometimes hitting out of memory and needs to be restarted. * Our meeting bot on matrix is unable to write log files to it's storage. * a few smaller applications haven't yet been redeployed or need some fixes. Unrelated, as we were moving, the AI scrapers started really hitting pagure.io (which was not moving or changing due to the move) which made things pretty difficult. We have blocked the 'blame' and 'history' web enpoints. If you need those, you should be able to clone the git repo and do them locally. This has moved the load back down to manageable. If you are trying to use the new src.fedoraproject.org server, you may have noticed it's ssh host keys changed. In the past we kept the old keys but this time we made a clean break. You can verify the keys by getting: https://admin.fedoraproject.org/ssh_known_hosts and adding it to your local .ssh/known_hosts. This will allow ssh to see the new host keys are signed with our CA. You can also use our sshfp records if you are using a dnssec enabled resolver. If you have noticed some other problem not mentioned above: If the issue is minor, consider just waiting and letting us know/filing a ticket on it next week if it's still happening. If it's something major, do let us know: https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/ and we will try and fix it as soon as we are able. Keep in mind we are focused on the above issues and will need to prioritze our time. We will also be having a retrospective and sharing what we learned after everything is back in a good state. Thanks again for your patience. kevin -- _______________________________________________ devel-announce mailing list -- devel-annou...@lists.fedoraproject.org To unsubscribe send an email to devel-announce-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel-annou...@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
