Dear all,
I think I've encountered a bug in lnetctl but not sure where to submit a
bug report:
Summary:
It's expected that the Lnet config on a node can be recreated on
lnet.service start up by saving the config using: lnetctl export
--backup > /etc/lnet.conf
But ordering within ymal file causes extraneous NIDs to be created when
used in combination with routing, thus breaking Lnet routing / node
communication, with server side dmesg showing "Bad dest nid n.n.n.n@o2ib
(it's my nid but on a different network)"
Environment:
Client: CentOS 7.8, Lustre 2.12.5-ib, MLNX OFED 4.9-0.1.7.1
Lnet router + server: CentOS 7.7, Lustre 2.12.4-ib, MLNX OFED 4.7-3.2.9.0
Steps to reproduce:
(Listing 1) Server side Lnet config (peer list omitted for conciseness):
https://pastebin.com/DH6HAt5a
(Listing 2) Full command listing and output on client side is reproduced
here: https://pastebin.com/h3wHyCM7
All steps below carried out on Lustre client:
1. Restart lnet service with empty /etc/lnet.conf
2. lnetctl net add: TCP network using Ethernet
3. lnetctl peer add: 2 peers with "Lnet router + server"@o2ib,tcp NIDs
4. lnetctl route add: 2 gateways to o2ib network using "Lnet router +
server"@TCP NID
5. lnetctl export: with --backup to /etc/lnet.conf; check the saved file
and confirm Lnet is configured with 2 peers and 2 gateways (Listing 2:
37-47)
6. Mount o2ib exported Lustre volume and confirm volume functioning
correctly; unmount volume
7. Restart lnet.service and check lnet configuration; finds 2 extra peer
entries that reference only TCP NID of the "Lnet router + server" along
with 2 manually configured peers that reference both o2ib and tcp NIDs
(Listing 2: 75-93)
8. Client fails to mount o2ib exported volume; server side kernel
message shows "Bad dest nid n.n.n.n@o2ib (it's my nid but on a different
network)"
9. If we reorder the peer list to go before the route list in
/etc/lnet.conf (Listing 2: 16), then lnet would be properly configured
with 2 peers on service restart and everything works as expected.
Best regards,
--
Angelos Ching
ClusterTech Limited
Tel : +852-2655-6138
Fax : +852-2994-2101
Address : Unit 211-213, Lakeside 1, 8 Science Park West Ave., Shatin, Hong Kong
Got praises or room for improvements? http://bit.ly/TellAngelos
********************************************************************************
The information contained in this e-mail and its attachments is confidential and
intended solely for the specified addressees. If you have received this email in
error, please do not read, copy, distribute, disclose or use any information of
this email in any way and please immediately notify the sender and delete this
email. Thank you for your cooperation.
********************************************************************************
_______________________________________________
lustre-discuss mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
