On Mon, Jun 09, 2025 at 05:33:28PM +0100, Jonathan Cameron wrote:
> This posting is primarily about sharing the USB device emulation to get some
> early feedback.
>
> RFC reasons:
> - Known 'inaccuracies' in emulation (not obeying MTU in the to host direction
> for
> example)./
> - Not sure what to do wrt to Klaus' I2C MCTP support given that has been
> stalled
> for some time. For now only the headers are really shared between the
> two implementations.
> - This is more of an FYI / request for testing than a formal suggestion that
> this
> might be ready for upstream.
>
> Why add a CXL FM-API over MCTP over USB device?
> - Can be emulated on pretty much any host system as USB is discoverable and
> expandable. If you want a giggle, see the hacks on i386/pc and arm/virt on
> we've been using until now given only I2C controller that works is the
> aspeed
> one. e.g. https://gitlab.com/jic23/qemu/-/commit/134c2e3952b
> - Being able to talk to both the fabric management out of band interfaces
> and the in band devices on the same host makes testing much simpler.
>
> Background:
>
> Back in 2022 I posted some support for controlling the CXL fabric via the
> spec defined out of band interfaces (CXL Fabric Management API - FM-API)
> over MCTP on I2C
>
> https://lore.kernel.org/qemu-devel/20220520170128.4436-1-jonathan.came...@huawei.com/
> I reworked that on top of the NVME-MI work from Klauss.
>
> To that end I hacked the aspeed-i2c controller onto both i386/pc and arm/virt
> and posted kernel patches to enabled ACPI support for that device (more or
> less).
> It worked and has been useful in the meantime, but adding that i2c controller
> to those boards was obviously not going to be upstreamable - and to build
> reliable tests against it I don't want to carry this out of tree for ever.
> I messed around with a PCI hosted aspeed controller and might come back to
> that at some point.
>
> In the meantime, DMTF published a transport binding for MCTP over USB
> https://www.dmtf.org/sites/default/files/standards/documents/DSP0283_1.0.1.pdf
>
> Kernel support duly followed early this year: drivers/net/mctp/mctp-usb.c
> https://codeconstruct.com.au/docs/mctp-over-usb/
>
> Given the ease of adding a suitable USB controller on a PCI bus, emulating a
> suitable endpoint provides what I think is an upstreamable solution.
>
> To use this:
> -device usb-ehci,id=ehci
> -device usb-cxl-mctp,bus=ehci.0,id=fred,target=us0
>
> where target is either a CXL switch upstream port, or a type 3 device.
>
> Then install the mctp userspace tools in your guest and configure it with
>
> mctp addr add 8 dev mctpusb0
> mctp link set mctpusb0 net 11
> mctp link set mctpusb0 up
>
> systemctl start mctpd.service
> busctl call au.com.codeconstruct.MCTP1
> /au/com/codeconstruct/mctp1/interfaces/mctpusb0
> au.com.codeconstruct.MCTP.BusOwner1 SetupEndpoint ay 0
>
> I've been testing the CXL commands with
>
> https://gitlab.com/jic23/cxl-fmapi-tests
> (mostly because I still had them in my image from the i2c work)
> but libcxlmi is probably a better bet
>
I tested this patchset using libcxlmi with the following configuration:
- kernel version: https://github.com/weiny2/linux-kernel/tree/dcd-v6-2025-04-13
- QEMU: upstream ToT with MCTP USB CXL and FMAPI DCD support:
- MCTP USB CXL
(https://lore.kernel.org/linux-cxl/20250609163334.922346-1-jonathan.came...@huawei.com/T/#m21b9e0dfc689cb1890bb4d961710c23379e04902)
- note: I modified the condition on line hw/usb/dev-mctp.c:509 to allow
the FMAPI DCD Management command set (0x56) to be processed
- FMAPI DCD Management patches
(https://lore.kernel.org/linux-cxl/20250605234227.970187-1-anisa.su...@gmail.com/)
- Topology (1 DCD attached to downstream port of CXL Switch)
'-device usb-ehci,id=ehci \
-object memory-backend-file,id=cxl-mem1,mem-path=/tmp/t3_cxl1.raw,size=4G \
-object memory-backend-file,id=cxl-lsa1,mem-path=/tmp/t3_lsa1.raw,size=1M \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1,hdm_for_passthrough=true \
-device cxl-rp,port=0,bus=cxl.1,id=cxl_rp_port0,chassis=0,slot=2 \
-device
cxl-upstream,port=2,sn=1234,bus=cxl_rp_port0,id=us0,addr=0.0,multifunction=on, \
-device cxl-switch-mailbox-cci,bus=cxl_rp_port0,addr=0.1,target=us0 \
-device cxl-downstream,port=0,bus=us0,id=swport0,chassis=0,slot=4 \
-device
cxl-type3,bus=swport0,volatile-dc-memdev=cxl-mem1,id=cxl-dcd0,lsa=cxl-lsa1,num-dc-regions=2,sn=99
\
-device usb-cxl-mctp,bus=ehci.0,id=usb0,target=us0 \
-device usb-cxl-mctp,bus=ehci.0,id=usb1,target=cxl-dcd0\
-machine
cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=1k'
After creating the VM with the above topology, to configure the 2 MCTP EPs
(mctpusb0 and mctpusb1):
mctp link set mctpusb0 up
mctp addr add 50 dev mctpusb0
mctp link set mctpusb0 net 11
mctp link set mctpusb1 up
mctp addr add 51 dev mctpusb1
mctp link set mctpusb0 net 12
systemctl stop mctpd.service
systemctl start mctpd.service
busctl call au.com.codeconstruct.MCTP1
/au/com/codeconstruct/mctp1/interfaces/mctpusb0
au.com.codeconstruct.MCTP.BusOwner1 SetupEndpoint ay 0
busctl call au.com.codeconstruct.MCTP1
/au/com/codeconstruct/mctp1/interfaces/mctpusb1
au.com.codeconstruct.MCTP.BusOwner1 SetupEndpoint ay 0
To test this patchset, I used this program from libcxlmi:
https://github.com/computexpresslink/libcxlmi/blob/main/examples/cxl-mctp.c
To summarize, this program scans the D-bus for MCTP endpoints or takes the
NID:EID
of the endpoint you want to open as arguments, then sends a variety of commands
(identify, supported logs, etc.) through the endpoint(s) and prints the results,
which I verified. Both methods of opening endpoints are successful and shows
the expected output.
To ensure this works end-to-end with an FM workflow I also tested with
this program, which allows users to interactively send the FMAPI Add/Release
commands from the command line:
https://github.com/computexpresslink/libcxlmi/blob/main/examples/fmapi-mctp.c
First I created a DC Region using this ndctl command:
cxl create-region -m mem0 -d decoder0.0 -s 1G -t dynamic_ram_a
Then I ran the program to add an extent of 512MB and was able to successfully
create a DAX Device using the following commands:
daxctl create-device -r region0
daxctl reconfigure-device dax0.1 -m system-ram
The output of 'lsmem' shows 512M added:
RANGE SIZE STATE REMOVABLE BLOCK
0x0000000000000000-0x000000007fffffff 2G online yes 0-15
0x0000000100000000-0x000000027fffffff 6G online yes 32-79
0x0000001290000000-0x00000012afffffff 512M online yes 594-597
> https://github.com/computexpresslink/libcxlmi
>
> I'll post a tree with this on at gitlab.com/jic23/qemu shortly
> (cxl-<latest date>).
>
> Jonathan Cameron (3):
> hw/cxl/i2c_mctp_cxl: Initial device emulation
> docs: cxl: Add example commandline for MCTP CXL CCIs
> usb/mctp/cxl: CXL FMAPI interface via MCTP over usb.
>
> Klaus Jensen (2):
> hw/i2c: add smbus pec utility function
> hw/i2c: add mctp core
>
> MAINTAINERS | 7 +
> docs/system/devices/cxl.rst | 27 +
> include/hw/cxl/cxl_device.h | 8 +
> include/hw/i2c/mctp.h | 125 +++++
> include/hw/i2c/smbus_master.h | 2 +
> include/hw/pci-bridge/cxl_upstream_port.h | 1 +
> include/hw/usb.h | 1 +
> include/net/mctp.h | 100 ++++
> hw/cxl/cxl-mailbox-utils.c | 49 ++
> hw/cxl/i2c_mctp_cxl.c | 289 ++++++++++
> hw/i2c/mctp.c | 414 ++++++++++++++
> hw/i2c/smbus_master.c | 26 +
> hw/usb/dev-mctp.c | 639 ++++++++++++++++++++++
> hw/arm/Kconfig | 1 +
> hw/cxl/Kconfig | 4 +
> hw/cxl/meson.build | 4 +
> hw/i2c/Kconfig | 4 +
> hw/i2c/meson.build | 1 +
> hw/i2c/trace-events | 14 +
> hw/usb/Kconfig | 5 +
> hw/usb/meson.build | 1 +
> 21 files changed, 1722 insertions(+)
> create mode 100644 include/hw/i2c/mctp.h
> create mode 100644 include/net/mctp.h
> create mode 100644 hw/cxl/i2c_mctp_cxl.c
> create mode 100644 hw/i2c/mctp.c
> create mode 100644 hw/usb/dev-mctp.c
>
> --
> 2.48.1
>
