- rocky-mirror-announce

DDoS Downloads from China Telecom/Mobile Blocks
by Louis Abel 28 Dec '23

28 Dec '23

Hello. Recently, we've received messages from several mirror operators about large amount of traffic coming in from various China IP addresses. These connections appear to be DDoS-like traffic, attempting to download the large ISO's of our releases but never actually finishing them. This obviously consumes bandwidth which can affect your mirror traffic negatively. If you find that this is happening to your mirror, we suggest putting in filters for those subnets. Even if you are not experiencing this, it may be necessary to also add them. This is, however, your call as a mirror operator to make that judgement. Below is a list of potential subnet blocks that can be used, provided by one of our operators. # IPv4 101.64.0.0/13 106.110.0.0/15 112.0.0.0/10 112.100.0.0/14 112.109.128.0/17 112.111.0.0/16 112.94.128.0/18 114.224.0.0/12 117.24.0.0/13 117.80.0.0/12 120.32.0.0/13 124.72.0.0/16 140.224.0.0/16 175.42.0.0/15 175.44.0.0/16 180.96.0.0/11 182.200.0.0/13 183.192.0.0/10 218.5.0.0/16 221.224.0.0/13 222.184.0.0/13 222.76.0.0/14 223.113.216.0/24 223.64.0.0/10 27.152.0.0/13 27.36.0.0/14 36.248.0.0/14 49.64.0.0/11 58.22.0.0/15 58.248.0.0/13 # IPv6 2408:8000::/20 2409:8000::/20 240e::/20 If you find there are other subnets to add, please do not hesitate to reply here or contact us at chat.rockylinux.org. Thank you again for supporting Rocky Linux. -- Louis Abel, RHCE RHCSIDM Infrastructure, Release Engineering, IDM

1 0

rsync node reboots
by Louis Abel 29 Nov '23

29 Nov '23

Hello mirror operators. We are performing patching and rebooting on all of our rsync endpoints. If your mirror was syncing, you may see an error of connections being closed or otherwise. If your rsync is on a regular schedule, this may not be an issue for you as it may start again at any time and resume, as you will connect to another node that is up. If you have any issues reconnecting to msync, please let us know. Thank you, -- Louis Abel, RHCE RHCSIDM Infrastructure, Release Engineering, IDM

1 0

Rocky Linux 9.3 Assets Available to Sync
by Louis Abel 15 Nov '23

15 Nov '23

Hello everyone, We are pleased to announce that Rocky Linux 9.3 assets are available to sync. We began allowing all mirrors to begin syncing within the last 48 hours from our rsync endpoints in both the US and Germany, to streamline future updates that will arrive alongside the upcoming release. US: msync.rockylinux.org EU: fra.msync.rockylinux.org For those who wish to use the FRA endpoint, the rsync modules may be different from our primary US endpoint. For those who are already using it, there is nothing you will need to change at this time. The artifacts available to all mirrors are to be kept from the public in accordance with our bitflip procedure until the release has been thoroughly tested. We have some artifacts (such as live images, containers and cloud images for ppc/s390x) that are still in the works and they will be passed out in the coming hours. If you have done the initial sync, then the follow up syncs should be very quick. If all goes well with our tests, we intend to perform the bitflip (and release Rocky Linux 9.3) on Monday during US business hours (EST, UTC-5). If anything changes, we will send a follow up email. Thank you for your continued support of Rocky Linux. If there is anything you need, please do not hesitate to reach out directly, via the mirror list, or our mattermost at https://chat.rockylinux.org/rocky-linux/channels/infrastructure. -- Louis Abel, RHCE RHCSIDM Infrastructure, Release Engineering, IDM

1 0

Rocky 8.8 Assets available to sync
by Neil Hanlon 18 May '23

18 May '23

Hello all, We are pleased to announce that Rocky Linux 8.8 is available for mirrors to sync from our mirror nodes in the US and Germany. US: msync.rockylinux.org EU: fra.msync.rockylinux.org Please note the fra nodes have a different rsync module layout which will be standardized in a future release. If you have already been using the frankfurt node, there is nothing you need to change. The artifacts available to mirrors are to be kept from the public in accordance with our bitflip procedure until the release has been thoroughly tested. If all goes well with tests, we intend to perform the bitflip (and release Rocky 8.8) tomorrow, on **Friday, May 19th, 2023**, during the afternoon of US business hours (East Coast). If anything changes to this affect, I will send a follow up email. Please see the following listing of directory sizes for this release. All artifacts should be hardlinked. ``` # du -ch --max-depth=1 8.8 102G 8.8/devel 9.3G 8.8/live 11M 8.8/plus 3.6M 8.8/ResilientStorage 4.1M 8.8/NFV 3.9M 8.8/RT 42G 8.8/images 8.8G 8.8/PowerTools 27G 8.8/isos 14M 8.8/metadata 12G 8.8/AppStream 1.9G 8.8/BaseOS 3.5M 8.8/HighAvailability 1.9M 8.8/extras 202G 8.8 202G total ``` Thank you very much for your continued support of Rocky Linux. If there is anything you need, please do not hesitate to reach out directly or via the mirrors list. Best, Neil

1 0

Rocky 9.2 Assets available to pre-sync
by Neil Hanlon 15 May '23

15 May '23

Hello all, We are pleased to announce that Rocky Linux 9.2 is available for mirrors to sync from our mirror nodes in the US and Germany. US: msync.rockylinux.org EU: fra.msync.rockylinux.org Please note the fra nodes have a different rsync module layout which will be standardized in a future release. If you have already been using the frankfurt node, there is nothing you need to change. The artifacts available to mirrors are to be kept from the public in accordance with our bitflip procedure until the release has been thoroughly tested. If all goes well with tests, we intend to perform the bitflip (and release Rocky 9.2) on Monday during US business hours (East Coast). If anything changes to this affect, I will send a follow up email. Thank you very much for your continued support of Rocky Linux. If there is anything you need, please do not hesitate to reach out directly or via the mirrors list. Best, Neil

2 2

Rocky Linux 9.1 Release Incoming
by Neil Hanlon 26 Nov '22

26 Nov '22

Hello all, We will be performing the "bitflip" on the Rocky Linux 9.1 artifacts (including replacing the symlink for `pub/rocky/9/` -> `pub/rocky/9.1`) at 04:00 UTC (or in about 3 hours). The 9.1 artifacts have been available on the tier 0 mirror at msync.rockylinux.org since late on the 24th of November and we've successfully syncronized them around several geo regions at this time. Thank you again for your continued support of Rocky Linux, and we hope this release has gone smoother than past releases. While we have not finalized the implementation of Tiering for the Rocky mirrors, we identified several bottlenecks in our network attached storage implementation which was contributing to really long delays when syncing from our mirrors. Please let us know if you encounter any issues, and remember we also have a European node in Frankfurt available at fra.msync.rockylinux.org. In the next few weeks we'll be adding a first-party node in the Asia-Pacific region as well. Best, Neil Hanlon

1 0

July Tier 0 Infrastructure Update
by Taylor Goodwill 25 Jul '22

25 Jul '22

Hi All, First and foremost, we would like to thank everyone so much for your patience during our recent release. Our new compose process exhibited a couple bugs which led to the hardlinking process not working as efficiently as it could, ultimately increasing the amount of data required to transfer for all mirrors. (Fixed) After preemptively growing our rsync pool and the addition of a second PoP in the EU, we underestimated the load that would be placed on the tier 0 mirror services during this release and did not give enough time for the sync process to begin before announcement. This issue was further exacerbated by the above hardlinking issue and the inclusion of two more architectures worth of artifacts with ppc64le & s390x. Additionally, it appears some file permissions may have prevented mirrors from syncing when the artifacts were moved to our production share due to an error in our script. (Fixed) That being said, the release went really well overall! All artifacts are served by our global CDN via Fastly, and while not ideal, these services were entirely available throughout release. This has been our stance, since the CDN is served directly from Tier 0. In the near term, we will be looking to implement an improved tiering schema for mirrors, ensuring we're able to work with Tier 1 mirrors distributed around the globe to pre-propagate our content before the bit-flip operation which occurs on release. If you are interested in helping, please reach out! Additionally, we will be making the following changes coinciding with a new mirroring policy (in progress): * implement access changes on the Tier 0 rsync endpoint to allow only Tier 1 mirrors by IP address. Tier 2+ mirror operators will want to choose another mirror site to pull from. We believe this will ultimately lead to a better experience for all of our mirror operators, as they will be able to choose a mirror to pull from which is likely geographically closer to them--with less latency and higher throughput. * launch additional rsync pops in the coming months which will coincide with other significant changes to our mirroring architecture. We truly appreciate all the support and assistance we've received from so many people throughout this process. Your suggestions and advice on best practices have been tremendously helpful, be it privately via email or on IRC/etc. We look forward to making these infrastructure improvements in the coming months to prepare for a successful release of Rocky Linux 8.7 and 9.1 in ~Q4. Best Regards, Rocky Infrastructure Team

1 0

Rocky Linux 9.0 Syncing to Tier 0 Mirrors
by Neil Hanlon 13 Jul '22

13 Jul '22

Hi everyone, We are syncing the 9.0 GA content to the production mirrors at this time and it should be available to synchronize shortly. There will be a small update for a few packages coming shortly, but we wanted to get the content available immediately. Looking at logs and metrics from the 8.6 release propagation, our tier 0 mirrors didn't seem to encounter any issues serving content as they did in prior releases, and I did not hear from anyone about synchronization troubles. If you did and/or do have any issues, please reach out. As a reminder, we have a European mirror located at fra.msync.rockylinux.org and I'll be adding a couple of nodes in other areas shortly after the release of 9.0. Like prior releases, the 9.0 GA directory will have its permissions set so that mirrors can sync the content, but not expose it to end users over HTTP or Rsync. Please ensure the permissions are set correctly on your content and services so the bitflip can occur as planned. We will release the 9.0 GA content at 14:00 UTC on Wednesday, July 14th 2022. Thank you very much for your continued support of Rocky Linux. We couldn't do this without your help! Best, Neil

1 0

[Rocky Linux] MirrorManager Maintenance over the next few days
by Neil Hanlon 12 Jul '22

12 Jul '22

Hi all, I'll be performing some maintenance on the MirrorManager servers over the next few days to prepare for the release of 9.0. Changes which are made in the UI may be rolled back due to some point restores that necessarily have to be done to integrate some of the changes for 9. I will do my best to keep them to a minimum, but please send me a message if you encounter anything extremely out of the ordinary (503 errors, etc). Best, Neil Hanlon

1 0

Rocky Linux 8.6 Available to Sync
by Neil Hanlon 16 May '22

16 May '22

Hello folks, The Rocky Linux 8.6 sources have been staged in production and are available for synchronization at this time. The final hardlink of the repositories should be finishing up here very shortly. We will perform a bitflip later on today to make everything available to the public. Thank you for your continued support of Rocky Linux! As a reminder we have a secondary synchronization point in Europe at fra.msync.rockylinux.org which can be used to pull from if it's more convenient. In combination with the release of 9 (expected next week), the infra team will also be finally getting around to deploying some much needed mirrors in different parts of the world. Lastly, I am working on some policies around mirroring that we're going to be looking to implement to ensure great user experience for people using Rocky. I am personally hoping to have these policies drafted and starting to be implemented in mid June, but we'll see what the future holds. Best, -- Neil Hanlon, Infrastructure Team Lead The Rocky Enterprise Software Foundation

1 0

2024

2023

2022