Hi All,
First and foremost, we would like to thank everyone so much for your patience during our recent release.
Our new compose process exhibited a couple bugs which led to the hardlinking process not working as efficiently as it could, ultimately increasing the amount of data required to transfer for all mirrors. (Fixed)
After preemptively growing our rsync pool and the addition of a second PoP in the EU, we underestimated the load that would be placed on the tier 0 mirror services during this release and did not give enough time for the sync process to begin before announcement. This issue was further exacerbated by the above hardlinking issue and the inclusion of two more architectures worth of artifacts with ppc64le & s390x.
Additionally, it appears some file permissions may have prevented mirrors from syncing when the artifacts were moved to our production share due to an error in our script. (Fixed)
That being said, the release went really well overall!
All artifacts are served by our global CDN via Fastly, and while not ideal, these services were entirely available throughout release. This has been our stance, since the CDN is served directly from Tier 0.
In the near term, we will be looking to implement an improved tiering schema for mirrors, ensuring we're able to work with Tier 1 mirrors distributed around the globe to pre-propagate our content before the bit-flip operation which occurs on release. If you are interested in helping, please reach out!
Additionally, we will be making the following changes coinciding with a new mirroring policy (in progress):
* implement access changes on the Tier 0 rsync endpoint to allow only Tier 1 mirrors by IP address. Tier 2+ mirror operators will want to choose another mirror site to pull from. We believe this will ultimately lead to a better experience for all of our mirror operators, as they will be able to choose a mirror to pull from which is likely geographically closer to them--with less latency and higher throughput.
* launch additional rsync pops in the coming months which will coincide with other significant changes to our mirroring architecture.
We truly appreciate all the support and assistance we've received from so many people throughout this process. Your suggestions and advice on best practices have been tremendously helpful, be it privately via email or on IRC/etc. We look forward to making these infrastructure improvements in the coming months to prepare for a successful release of Rocky Linux 8.7 and 9.1 in ~Q4.
Best Regards,
Rocky Infrastructure Team
