Hey, > I attempted to use mmcli 1.18.6 and nmcli 1.36.0 with the create and connect > bearer features of mmcli to create multiple simultaneous connections to two > different APNs with a USB cellular modem on an Ubuntu Jammy Jellyfish 22.04 > system. The ModemManager.service file had the debug and > test-multiplex-requested options activated on the ExecStart line. > > Short summary of results: > > A sequence of mmcli commands to create a bearer, connect the bearer, create a > second bearer, then connect the second bearer did result in multiple > simultaneous packet data network connections. Some subsequent configuration > with ip and route commands were necessary to make it work, but the second > connection failed after a few minutes with error > org.freedesktop.ModemManager1.Error.MobileEquipment.Unknown. >
Hum, strange about that 2nd connection dropping like that. > More details starting from the beginning: > > After booting and logging in, I plugged in the modem into the system, it > reported this configuration: > > user@ubuntu22a:~/Documents$ ifconfig > > ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > > inet6 fe80::ad94:11a5:b4a6:eada prefixlen 64 scopeid 0x20<link> > > ether 00:0c:29:df:a4:a1 txqueuelen 1000 (Ethernet) > > RX packets 0 bytes 0 (0.0 B) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 321 bytes 51137 (51.1 KB) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > device interrupt 19 base 0x2000 > > > > lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 > > inet 127.0.0.1 netmask 255.0.0.0 > > inet6 ::1 prefixlen 128 scopeid 0x10<host> > > loop txqueuelen 1000 (Local Loopback) > > RX packets 8850 bytes 637347 (637.3 KB) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 8850 bytes 637347 (637.3 KB) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > user@ubuntu22a:~/Documents$ ip link list > > 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode > DEFAULT group default qlen 1000 > > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > > 2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state > UNKNOWN mode DEFAULT group default qlen 1000 > > link/ether 00:0c:29:df:a4:a1 brd ff:ff:ff:ff:ff:ff > > altname enp2s1 > > 3: wwan0: <BROADCAST,MULTICAST,NOARP> mtu 1428 qdisc noop state DOWN mode > DEFAULT group default qlen 1000 > > link/ether 6a:d4:98:05:df:3a brd ff:ff:ff:ff:ff:ff > > > > link/ether 6a:d4:98:05:df:3a brd ff:ff:ff:ff:ff:ff > > > > user@ubuntu22a:~/Documents$ route > > Kernel IP routing table > > Destination Gateway Genmask Flags Metric Ref Use Iface > > <empty> > > > > Running the command “sudo mmcli -L” indicated that my modem was number 0. I > ran the following command: > > sudo mmcli -m 0 > –create-bearer=’apn=apnname1,ip-type=ipv4,allowed-auth=pap,user=apnuser1,password=apnpass1’ > > This was successful and created bearer number 0. > > I then ran > > sudo mmcli -m 0 -b 0 --connect > > This was also successful. > > After doing this, the system’s networking configuration looked as follows: > > user@ubuntu22a:~/Documents$ ifconfig > > <ens33 and lo omitted for brevity’s sake; they are identical to first listing> > > mbimmux0.1: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1428 > > inet6 fe80::98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid 0x20<link> > > inet6 2001:db53:18ae:0:98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid > 0x0<global> > > inet6 2001:db53:18ae:0:3911:3b6a:6a96:92a4 prefixlen 64 scopeid > 0x0<global> > > ether 9a:b0:d0:6e:dc:9f txqueuelen 1000 (Ethernet) > > RX packets 1 bytes 88 (88.0 B) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 24 bytes 4191 (4.1 KB) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > wwan0: flags=67<UP,BROADCAST,RUNNING> mtu 1428 > > inet6 fe80::98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid 0x20<link> > > ether 9a:b0:d0:6e:dc:9f txqueuelen 1000 (Ethernet) > > RX packets 1 bytes 88 (88.0 B) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 30 bytes 4271 (4.2 KB) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > user@ubuntu22a:~/Documents$ ip link list > > <1: lo and 2: ens33 omitted for brevity’s sake; they are identical to first > listing> > > 3: wwan0: <BROADCAST,UP,LOWER_UP> mtu 1428 qdisc fq_codel state UNKNOWN mode > DEFAULT group default qlen 1000 > > link/ether 9a:b0:d0:6e:dc:9f brd ff:ff:ff:ff:ff:ff > > 4: mbimmux0.1@wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1428 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > > link/ether 9a:b0:d0:6e:dc:9f brd ff:ff:ff:ff:ff:ff > > > > user@ubuntu22a:~/Documents$ route > > Kernel IP routing table > > Destination Gateway Genmask Flags Metric Ref Use Iface > > <empty> > All this seems fine to me. MM takes care of bringing up the bearer connection, and also takes care of creating the new virtual network interface that is bound to that connection. The IP configuration and routes of the new virtual network interface should be manually setup in the interface based on the details exposed in the corresponding Bearer object, MM won't do that. > > > I then ran > > sudo mmcli -m 0 > –create-bearer=’apn=apnname2,ip-type=ipv4,allowed-auth=pap,user=apnuser2,password=apnpass2’ > > This was successful and created bearer number 1. > > I then ran > > sudo mmcli -m 0 -b 1 --connect > > This was also successful. > > After doing this, the system’s networking configuration looked as follows: > > user@ubuntu22a:~/Documents$ ifconfig > > <ens33 and lo omitted for brevity’s sake; they are identical to first listing> > > mbimmux0.1: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1428 > > inet6 fe80::98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid 0x20<link> > > inet6 2001:db53:18ae:0:98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid > 0x0<global> > > inet6 2001:db53:18ae:0:3911:3b6a:6a96:92a4 prefixlen 64 scopeid > 0x0<global> > > ether 9a:b0:d0:6e:dc:9f txqueuelen 1000 (Ethernet) > > RX packets 1 bytes 88 (88.0 B) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 27 bytes 4512 (4.5 KB) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > wwan0: flags=67<UP,BROADCAST,RUNNING> mtu 1428 > > inet6 fe80::98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid 0x20<link> > > ether 9a:b0:d0:6e:dc:9f txqueuelen 1000 (Ethernet) > > RX packets 1 bytes 88 (88.0 B) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 36 bytes 4718 (4.7 KB) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > user@ubuntu22a:~/Documents$ ip link list > > <1: lo and 2: ens33 omitted for brevity’s sake; they are identical to first > listing> > > 3: wwan0: <BROADCAST,UP,LOWER_UP> mtu 1428 qdisc fq_codel state UNKNOWN mode > DEFAULT group default qlen 1000 > > link/ether 9a:b0:d0:6e:dc:9f brd ff:ff:ff:ff:ff:ff > > 4: mbimmux0.1@wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1428 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > > link/ether 9a:b0:d0:6e:dc:9f brd ff:ff:ff:ff:ff:ff > > 5: mbimmux0.2@wwan0: <BROADCAST> mtu 1428 qdisc noop state DOWN mode DEFAULT > group default qlen 1000 > > link/ether 9a:b0:d0:6e:dc:9f brd ff:ff:ff:ff:ff:ff > > > > user@ubuntu22a:~/Documents$ route > > Kernel IP routing table > > Destination Gateway Genmask Flags Metric Ref Use Iface > > <empty> > > > > I noticed that mbimmux0.2 did not appear in the “ifconfig” output and that > “ip link list” did not report it as up. So I ran “sudo ip link set mbimmux0.2 > up” and then the interface configuration looked like this: > I think that's also fine. Bringing the virtual network interface up is something the upper layers would do when setting it up. MM takes care of making sure the master interface is up, not the new virtual ones. > user@ubuntu22a:~/Documents$ ifconfig > > <ens33 and lo omitted for brevity’s sake; they are identical to first listing> > > mbimmux0.1: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1428 > > inet6 fe80::98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid 0x20<link> > > inet6 2001:db53:18ae:0:98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid > 0x0<global> > > inet6 2001:db53:18ae:0:3911:3b6a:6a96:92a4 prefixlen 64 scopeid > 0x0<global> > > ether 9a:b0:d0:6e:dc:9f txqueuelen 1000 (Ethernet) > > RX packets 1 bytes 88 (88.0 B) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 29 bytes 4726 (4.7 KB) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > mbimmux0.2: flags=67<UP,BROADCAST,RUNNING> mtu 1428 > > inet6 2001:db53:18ae:0:19fd:361e:7fdb:ea15 prefixlen 64 scopeid > 0x0<global> > > inet6 fe80::98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid 0x20<link> > > inet6 2001:db53:18ae:0:98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid > 0x0<global> > > ether 9a:b0:d0:6e:dc:9f txqueuelen 1000 (Ethernet) > > RX packets 1 bytes 88 (88.0 B) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 10 bytes 868 (868.0 B) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > wwan0: flags=67<UP,BROADCAST,RUNNING> mtu 1428 > > inet6 fe80::98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid 0x20<link> > > ether 9a:b0:d0:6e:dc:9f txqueuelen 1000 (Ethernet) > > RX packets 2 bytes 176 (176.0 B) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 49 bytes 5688 (5.6 KB) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > user@ubuntu22a:~/Documents$ ip link list > > <1: lo and 2: ens33 omitted for brevity’s sake; they are identical to first > listing> > > 3: wwan0: <BROADCAST,UP,LOWER_UP> mtu 1428 qdisc fq_codel state UNKNOWN mode > DEFAULT group default qlen 1000 > > link/ether 9a:b0:d0:6e:dc:9f brd ff:ff:ff:ff:ff:ff > > 4: mbimmux0.1@wwan0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1428 qdisc > noqueue state UP mode DEFAULT group default qlen 1000 > > link/ether 9a:b0:d0:6e:dc:9f brd ff:ff:ff:ff:ff:ff > > 5: mbimmux0.2@wwan0: <BROADCAST,UP,LOWER_UP> mtu 1428 qdisc noqueue state UP > mode DEFAULT group default qlen 1000 > > link/ether 9a:b0:d0:6e:dc:9f brd ff:ff:ff:ff:ff:ff > > > > user@ubuntu22a:~/Documents$ route > > Kernel IP routing table > > Destination Gateway Genmask Flags Metric Ref Use Iface > > <empty> > > > > The status of the modem and the two bearers was then as follows: > > user@ubuntu22a:~/Documents$ mmcli -m 0 > > -------------------------------- > > General | path: /org/freedesktop/ModemManager1/Modem/0 > > | device id: a883afd9cd8161007d0e42fe1e7a52bd9073f70d > > -------------------------------- > > Hardware | manufacturer: Undisclosed, Inc. > > | model: A101 > > | firmware revision: A101_01.2000.00 > > | h/w revision: A101_V4.5_MBIM_V4.5 > > | supported: gsm-umts, lte > > | current: gsm-umts, lte > > | equipment id: 101000000000008 > > -------------------------------- > > System | device: > /sys/devices/pci0000:00/0000:00:11.0/0000:02:03.0/usb1/1-1 > > | drivers: cdc_acm, cdc_mbim > > | plugin: generic > > | primary port: cdc-wdm0 > > | ports: cdc-wdm0 (mbim), ttyACM0 (at), ttyACM2 > (at), wwan0 (net) > > -------------------------------- > > Status | unlock retries: sim-pin2 (3) > > | state: connected > > | power state: on > > | access tech: gprs > > | signal quality: 80% (cached) > > -------------------------------- > > Modes | supported: allowed: 2g, 3g, 4g; preferred: none > > | current: allowed: 2g, 3g, 4g; preferred: none > > -------------------------------- > > IP | supported: ipv4, ipv6, ipv4v6 > > -------------------------------- > > 3GPP | imei: 101000000000008 > > | enabled locks: fixed-dialing > > | operator id: MCCMNC > > | operator name: MCCMNC > > | registration: roaming > > -------------------------------- > > 3GPP EPS | ue mode of operation: csps-2 > > -------------------------------- > > SIM | primary sim path: /org/freedesktop/ModemManager1/SIM/0 > > -------------------------------- > > Bearer | paths: /org/freedesktop/ModemManager1/Bearer/1 > > | /org/freedesktop/ModemManager1/Bearer/0 > > user@ubuntu22a:~/Documents$ mmcli -m 0 -b 0 > > ------------------------------------ > > General | path: /org/freedesktop/ModemManager1/Bearer/0 > > | type: default > > ------------------------------------ > > Status | connected: yes > > | suspended: no > > | multiplexed: yes > > | interface: mbimmux0.1 > > | ip timeout: 20 > > ------------------------------------ > > Properties | apn: apnname1 > > | roaming: allowed > > | ip type: ipv4 > > | allowed-auth: pap > > | user: apnuser1 > > | password: apnpass1 > > ------------------------------------ > > IPv4 configuration | method: static > > | address: 10.3.1.2 > > | prefix: 24 > > | gateway: 10.3.1.1 > > | dns: 208.67.222.222, 208.67.220.220 > > ------------------------------------ > > Statistics | duration: 390 > > | attempts: 1 > > | total-duration: 390 > > user@ubuntu22a:~/Documents$ mmcli -m 0 -b 1 > > ------------------------------------ > > General | path: /org/freedesktop/ModemManager1/Bearer/1 > > | type: default > > ------------------------------------ > > Status | connected: yes > > | suspended: no > > | multiplexed: yes > > | interface: mbimmux0.2 > > | ip timeout: 20 > > ------------------------------------ > > Properties | apn: apnname2 > > | roaming: allowed > > | ip type: ipv4 > > | allowed-auth: pap > > | user: apnuser2 > > | password: apnpass2 > > ------------------------------------ > > IPv4 configuration | method: static > > | address: 10.3.1.3 > > | prefix: 24 > > | gateway: 10.3.1.1 > Is this right? Are you getting in the 2nd APN connection an IP address in the same subnet as in the 1st APN connection? It wouldn't be completely wrong, but it kind of defeats the purpose of the multi APN connection really (unless the network itself has explicit routing rules for those different IPs even if in the same subnet). > | dns: 208.67.222.222, 208.67.220.220 > > ------------------------------------ > > Statistics | attempts: 2 > > | total-duration: 284 > > > > I noticed that I would need to add some route information in order to be able > to direct traffic through the mmbimux0.1 and mbimmux0.2 interfaces. This is > fine. Yes, as said, MM doesn't do any IP/route config by itself. > > However, after approximately five minutes of using and examining these > interfaces, I noticed the second bearer connection had failed with an unknown > error and mbimmux0.2 had been removed from the list of interfaces: > > user@ubuntu22a:~/Documents$ mmcli -m 0 -b 1 > > -------------------------------------- > > General | path: > /org/freedesktop/ModemManager1/Bearer/1 > > | type: default > > -------------------------------------- > > Status | connected: no > > | connection error name: > org.freedesktop.ModemManager1.Error.MobileEquipment.Unknown > > | connection error message: Unknown error > > | suspended: no > > | multiplexed: no > > | ip timeout: 20 > > -------------------------------------- > > Properties | apn: apnname2 > > | roaming: allowed > > | ip type: ipv4 > > | allowed-auth: pap > > | user: apnuser2 > > | password: apnpass2 > > -------------------------------------- > > Statistics | duration: 294 > > | attempts: 2 > > | total-duration: 578 > No idea why that error happens. That's likely an error reported by the modem itself and we're converting it to "Unknown" ourselves, but there is probably more info about the specific error happening in the MM debug log itself. You should look in the MM debug log and look for a MBIM indication reporting that disconnection at that time, and see if there's a better error description there. It could be that the modem reports Unknown itself, I wouldn't be surprised to be honest. > user@ubuntu22a:~/Documents$ ifconfig > > <ens33 and lo omitted for brevity’s sake; they are identical to first listing> > > mbimmux0.1: flags=4291<UP,BROADCAST,RUNNING,NOARP,MULTICAST> mtu 1428 > > inet6 fe80::98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid 0x20<link> > > inet6 2001:db53:18ae:0:98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid > 0x0<global> > > inet6 2001:db53:18ae:0:3911:3b6a:6a96:92a4 prefixlen 64 scopeid > 0x0<global> > > ether 9a:b0:d0:6e:dc:9f txqueuelen 1000 (Ethernet) > > RX packets 1 bytes 88 (88.0 B) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 30 bytes 4833 (4.8 KB) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > wwan0: flags=67<UP,BROADCAST,RUNNING> mtu 1428 > > inet6 fe80::98b0:d0ff:fe6e:dc9f prefixlen 64 scopeid 0x20<link> > > ether 9a:b0:d0:6e:dc:9f txqueuelen 1000 (Ethernet) > > RX packets 3 bytes 264 (264.0 B) > > RX errors 0 dropped 0 overruns 0 frame 0 > > TX packets 62 bytes 6621 (6.6 KB) > > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > > > > After it fails, running “sudo mmcli -m 0 -b 1 --connect” then “sudo ip link > set mbimmux0.2 up” puts things back in business. I did not expect to have to > keep checking and then stand it back up when it fails. > MM does not do any reconnection logic by itself, because reconnections may involve IP reconfiguration. All that should be done by upper layers, the actual network management tool should do it. > Questions: > > 1. How can I keep the second bearer connection from falling over? > If it's a network-reported disconnection, you cannot do anything. It could be the network is disconnecting you because there's no traffic ongoing or something, don't know. > 2. Is it expected that “sudo ip link set mbimmux0.2 up” would be necessary > after the second bearer connection is established? Is this the right thing to > do? > Yes, that should be fine. > 3. Is there a way to have the ifconfig and route entries managed by any of > this software to avoid the need to “manually” manage them? > You need a network management tool to do that on top of ModemManager; MM just sets everything up, and then the actual IP/route configuration should be done by upper layers. In your case it involves manually configuring the interface because "static" is the method reported, but it could also be "dhcp" or "ppp" depending on the modem type and protocol. > 4. Any other advice to help me achieve happiness? > Oh, you're very close to having everything working together :) If you script out a simple connection management logic using mmcli and iproute2 for IP/route management, you already have the basic part done. If you're up to looking at NetworkManager's wwan plugin to see how complex it would be to add the required multiplex logic support there, that would benefit everyone, not just you :) Avoiding the explicit disconnection on the device when a new connection is added would be a start. -- Aleksander https://aleksander.es
