Hi folks,
With CentOS 7/Rocky 8 and "network-scripts" I was able to configure static routes using "equal cost multi-path" (ECMP) routing. /etc/sysconfig/network-scripts/route-eth0 would contain a line like
172.16.0.0/16 nexthop via 192.168.1.250 nexthop via 192.168.1.251
which would then look like:
# ip r default via 192.168.1.254 dev eth0 172.16.0.0/16 nexthop via 192.168.1.250 dev eth0 weight 1 nexthop via 192.168.1.251 dev eth0 weight 1
It seems that the NetworkManager keyfile format is not able to handle that situation.
How can I configure static ECMP routes with Rocky 8/9 and NetworkManager?
Thanks,
Uwe
Hi Uwe,
On 27.04.2023 12:34, Uwe Sauter via rocky wrote:
Hi folks,
With CentOS 7/Rocky 8 and "network-scripts" I was able to configure static routes using "equal cost multi-path" (ECMP) routing. /etc/sysconfig/network-scripts/route-eth0 would contain a line like
172.16.0.0/16 nexthop via 192.168.1.250 nexthop via 192.168.1.251
which would then look like:
# ip r default via 192.168.1.254 dev eth0 172.16.0.0/16 nexthop via 192.168.1.250 dev eth0 weight 1 nexthop via 192.168.1.251 dev eth0 weight 1
It seems that the NetworkManager keyfile format is not able to handle that situation.
How can I configure static ECMP routes with Rocky 8/9 and NetworkManager?
It's roughly the same thing in NetworkManager - the change is that ipv4 routes now support metrics, which allow you to specify the following in your [ipv4] section of the keyfile:
``` [ipv4] route1=10.0.0.250/32,10.0.0.1,50 route2=10.0.0.250/32,10.0.0.2,50 ```
Or, with nmcli:
``` nmcli con mod your-connection ipv4.routes '10.0.0.250/32 10.0.0.1 50, 10.0.0.250/32 10.0.0.2 50' ```
This will create two routes with different destinations but the same metric, making them be treated as ECMP routes.
For reference, the syntax for ipv4.routes is
``` ip[/prefix] [next-hop] [metric] [attribute=val]... [,ip[/prefix] ...] ```
Hope this helps!
--Neil
Thanks,
Uwe _______________________________________________ rocky mailing list -- rocky@lists.resf.org To unsubscribe send an email to rocky-leave@lists.resf.org
Hi Neil,
thanks for that suggestion.
How can I configure static ECMP routes with Rocky 8/9 and NetworkManager?
It's roughly the same thing in NetworkManager - the change is that ipv4 routes now support metrics, which allow you to specify the following in your [ipv4] section of the keyfile:
[ipv4] route1=10.0.0.250/32,10.0.0.1,50 route2=10.0.0.250/32,10.0.0.2,50
This will create two routes with different destinations but the same metric, making them be treated as ECMP routes.
Talking to my local network admin, he and I both agree that this is not the same behavior as before.
While adding mutiple routes to the same destination might act like ECMP it probably does not. Assuming the same metric for each entry it will prefer the first route it finds when going through the routing table.
Contratry to this ECMP using nexthop syntax will only require one entry in the routing table and the the internals of ECMP routing will then decide which gateway to select next based on weight and the previous state.
I consider this a regression of the behavior that the (upstream) distribution provides.
Using a separate systemd service that will execute a script to add/remove routes via the ip command might be a (dirty) workaround. Or putting even more distance to the distribution I might install systemd-networkd which provides true ECMP and does not require an always running service.
If there are other suggestions for how to get true ECMP routing I'd really like to hear those.
Have a nice weekend,
Uwe
On 28.04.2023 12:46, Uwe Sauter via rocky wrote:
Hi Neil,
thanks for that suggestion.
How can I configure static ECMP routes with Rocky 8/9 and NetworkManager?
It's roughly the same thing in NetworkManager - the change is that ipv4 routes now support metrics, which allow you to specify the following in your [ipv4] section of the keyfile:
[ipv4] route1=10.0.0.250/32,10.0.0.1,50 route2=10.0.0.250/32,10.0.0.2,50This will create two routes with different destinations but the same metric, making them be treated as ECMP routes.
Talking to my local network admin, he and I both agree that this is not the same behavior as before.
While adding mutiple routes to the same destination might act like ECMP it probably does not. Assuming the same metric for each entry it will prefer the first route it finds when going through the routing table.
Contratry to this ECMP using nexthop syntax will only require one entry in the routing table and the the internals of ECMP routing will then decide which gateway to select next based on weight and the previous state.
I consider this a regression of the behavior that the (upstream) distribution provides.
So, this is actually coming in NetworkManager 1.42, I got my versions mixed up. See https://networkmanager.dev/blog/networkmanager-1-42/ ) and will work as you expect. I mixed up my versions last night when writing the email.
Using a separate systemd service that will execute a script to add/remove routes via the ip command might be a (dirty) workaround. Or putting even more distance to the distribution I might install systemd-networkd which provides true ECMP and does not require an always running service.
Systemd-networkd is a good option, and one that I would recommend, generally. You may also be intrested in nmstate [1] though I'm not sure if it supports ECMP.
Personally, when I deploy ECMP, the routes are coming from an IGP, not staticly on the host. That may be an option you wish to look into, as well, but it requires changing up how you're networking your servers (routed vs switched access)
If there are other suggestions for how to get true ECMP routing I'd really like to hear those.
Have a nice weekend,
You too! --Neil
Uwe _______________________________________________ rocky mailing list -- rocky@lists.resf.org To unsubscribe send an email to rocky-leave@lists.resf.org
Hi Neil,
thanks again.
So, this is actually coming in NetworkManager 1.42, I got my versions mixed up. See https://networkmanager.dev/blog/networkmanager-1-42/ ) and will work as you expect. I mixed up my versions last night when writing the email.
Checking with Red Hat documentation it looks like this is coming with RHEL 9.2. Let's hope it won't take long to be released.
Using a separate systemd service that will execute a script to add/remove routes via the ip command might be a (dirty) workaround. Or putting even more distance to the distribution I might install systemd-networkd which provides true ECMP and does not require an always running service.
Systemd-networkd is a good option, and one that I would recommend, generally. You may also be intrested in nmstate [1] though I'm not sure if it supports ECMP.
The issue I have with using parts of systemd that are not provided by the distribution is that versions of systemd usually differ between the distribution packages and the ones coming from EPEL. Though that hopefully doesn't matter much.
The second issue I have is that RHEL and its derivatives imply that NetworkManager is installed and software is linked against. But that's a totally different rabbit hole that I don't want to descend into right now.
Best,
Uwe
Personally, when I deploy ECMP, the routes are coming from an IGP, not staticly on the host. That may be an option you wish to look into, as well, but it requires changing up how you're networking your servers (routed vs switched access)
If there are other suggestions for how to get true ECMP routing I'd really like to hear those.
Have a nice weekend,
You too! --Neil
Uwe _______________________________________________ rocky mailing list -- rocky@lists.resf.org To unsubscribe send an email to rocky-leave@lists.resf.org
-
Neil Hanlon -
Uwe Sauter