On April 17, 2023, we had planned maintenance that was aimed to be transparent in nature. In our current work of
standing up new infrastructure, we needed to migrate Rocky Linux 8 to 9 for our FreeIPA domain (that runs our DNS and
auth for our services internal and external).
The plan was to essentially:
* Remove one node
* Add new node, configure as needed via ansible
* Repeat the above until all nodes are migrated
As FreeIPA handles the internal DNS and responds to internal requests from our external services where necessary, this
should be fairly routine. Most internal services won’t notice IPA servers coming and going as the SRV records will have
changed and things will move on as normal.
This is mostly transparent, following the proper documentation for FreeIPA migrations between major releases of
Enterprise Linux. However, among this work, our internal firewall and unbound DNS caching were not made aware of the new
IPA systems as we added and removed nodes, and thus the problems began to start and eventually cascade externally. The
remaining 8 node that was still available was the only server responding to requests from our haproxy and unbound.
Though this was the case, this essentially caused the following issues:
* The CDN would fallback to our appropriate parameters to ensure mirror manager would still respond with at least one
mirror (dl.rockylinux.org)
* The CDN would detect our services to be back up briefly
* Some users will have success, while some would time out.
* Anyone hitting mirrors.rockylinux.org would eventually timeout
* The CDN would detect it down again and try to fallback
* The above would loop endlessly
This also unfortunately prevented us from being able to login to our VPN by normal means. The appropriate infrastructure
contacts were notified to assist in getting in and fixing the internal resolver to bring all services back online. After
all services were back online, we were able to migrate the final 8 node to 9, without further impact to our
infrastructure and our users.
Corrections have been made to the infrastructure configuration to provide better fault tolerance in the case of DNS
outages like this.
If you have any questions, please reach out.
Best,
Neil Hanlon
Infrastructure Team Lead, Rocky Enterprise Software Foundation
