Hi Uwe,

On 27.04.2023 12:34, Uwe Sauter via rocky wrote:

Hi folks,

With CentOS 7/Rocky 8 and "network-scripts" I was able to configure static routes using "equal cost multi-path" (ECMP) routing. /etc/sysconfig/network-scripts/route-eth0 would contain a line like

172.16.0.0/16 nexthop via 192.168.1.250 nexthop via 192.168.1.251

which would then look like:

# ip r default via 192.168.1.254 dev eth0 172.16.0.0/16 nexthop via 192.168.1.250 dev eth0 weight 1 nexthop via 192.168.1.251 dev eth0 weight 1

It seems that the NetworkManager keyfile format is not able to handle that situation.

How can I configure static ECMP routes with Rocky 8/9 and NetworkManager?

It's roughly the same thing in NetworkManager - the change is that ipv4 routes now support metrics, which allow you to specify the following in your [ipv4] section of the keyfile:

``` [ipv4] route1=10.0.0.250/32,10.0.0.1,50 route2=10.0.0.250/32,10.0.0.2,50 ```

Or, with nmcli:

``` nmcli con mod your-connection ipv4.routes '10.0.0.250/32 10.0.0.1 50, 10.0.0.250/32 10.0.0.2 50' ```

This will create two routes with different destinations but the same metric, making them be treated as ECMP routes.

For reference, the syntax for ipv4.routes is

``` ip[/prefix] [next-hop] [metric] [attribute=val]... [,ip[/prefix] ...] ```

Hope this helps!

--Neil

Thanks,

Uwe _______________________________________________ rocky mailing list -- rocky@lists.resf.org To unsubscribe send an email to rocky-leave@lists.resf.org

On 28.04.2023 12:46, Uwe Sauter via rocky wrote:

Hi Neil,

thanks for that suggestion.

...

...

How can I configure static ECMP routes with Rocky 8/9 and NetworkManager?

It's roughly the same thing in NetworkManager - the change is that ipv4 routes now support metrics, which allow you to specify the following in your [ipv4] section of the keyfile:

[ipv4]
route1=10.0.0.250/32,10.0.0.1,50
route2=10.0.0.250/32,10.0.0.2,50

...

This will create two routes with different destinations but the same metric, making them be treated as ECMP routes.

Talking to my local network admin, he and I both agree that this is not the same behavior as before.

While adding mutiple routes to the same destination might act like ECMP it probably does not. Assuming the same metric for each entry it will prefer the first route it finds when going through the routing table.

Contratry to this ECMP using nexthop syntax will only require one entry in the routing table and the the internals of ECMP routing will then decide which gateway to select next based on weight and the previous state.

I consider this a regression of the behavior that the (upstream) distribution provides.

So, this is actually coming in NetworkManager 1.42, I got my versions mixed up. See https://networkmanager.dev/blog/networkmanager-1-42/ ) and will work as you expect. I mixed up my versions last night when writing the email.

Using a separate systemd service that will execute a script to add/remove routes via the ip command might be a (dirty) workaround. Or putting even more distance to the distribution I might install systemd-networkd which provides true ECMP and does not require an always running service.

Systemd-networkd is a good option, and one that I would recommend, generally. You may also be intrested in nmstate [1] though I'm not sure if it supports ECMP.

Personally, when I deploy ECMP, the routes are coming from an IGP, not staticly on the host. That may be an option you wish to look into, as well, but it requires changing up how you're networking your servers (routed vs switched access)

If there are other suggestions for how to get true ECMP routing I'd really like to hear those.

Have a nice weekend,

You too! --Neil

Uwe _______________________________________________ rocky mailing list -- rocky@lists.resf.org To unsubscribe send an email to rocky-leave@lists.resf.org

[1] https://nmstate.io/user/quick_guide.html