Nice article, happen to have been working on a usbip system for my Macbook M3 using similar tech.
Worth noting that, this approach is limitted on newer macOS systems: can't build libusb "userspace USB drivers" for devices that are supported by macOS. Without manually disabling some Security features, can't override drivers for system-recognized USB devices.
But this kinda expects that your USB driver is the application code too, no? This is less of a driver and more of a library + program. If I have, say, a USB to Ethernet device, how do I hook this into the ethernet adapter subsystem?
Things which are relatively standard tend to get good generic support: Ethernet devices will generally be USB/CDC/ECM or RNDIS, for example. That may Just Work (tm) if it has the right descriptors.
The userland approach is much more useful for weird or custom devices. In particular, on Windows you can do one of these user space "drivers" without having to bother with driver signing, and if you use libusb it will be portable too.
DFU - great example. If you have a USB device that has a DFU class that needs a custom driver, can dfu-util and the like hook into these userspace drivers? Or do you also need to maintain the application part?
Dfu-util is one of those "user space drivers", so if you have a nonstandard protocol you'd have to add it directly to dfu-util. There's no intermediate API.
It's not easy to set up a fake or "remapped" USB device on most OS as far as I'm aware, if you were trying to write an adapter program that modified USB packets.
dfu-util actually also just uses libusb under the hood!
Any class or device that doesn't have a driver baked into the OS can be implemented like this. And if you'd need the DFU functionality in a different application, you may be able to just simply link parts of the dfu-util tool into your application
On Linux you could create a tun/tap device from your application and translate data sent over that to requests sent to the ethernet adapter.
Of course, when you're doing these things in userspace you either need some way of communicating with the Kernel or for the other subsystems to be in userspace as well.
Not to be too facetious but a great place for communicating with the kernel where there are a ton of other driver subsystems is... the kernel.
Possibly a good addition to the article would be parallel development of an lkm. I guess it wouldn't have that windows interop but I would also be interested to see how this driver would be implemented on Windows. If it's idk 10x as many lines in the kernel vs userspace, that's a great benefit to the userspace approach.
In HFT user-space networking drivers have a long history - there is too much latency induced by switching from kernel to user space to handle networking.
> OpenOnload: A user-space network stack that intercepts socket calls to bypass the kernel network stack, accelerating standard socket operations for faster networking.
> Netmap: A framework providing a simple API for high-speed packet I/O in user space, bypassing much of the kernel overhead for efficient packet forwarding and filtering.
Perfect timing. I'm expecting to get my hands on a MOTU MIDI Express XT from my local Guitar Center within the next couple days (I paid for it when it arrived there a couple weeks ago, but they have a mandatory waiting period on used equipment to make sure it ain't stolen), which unfortunately uses some weird proprietary protocol instead of class-compliant MIDI-over-USB — so any use over USB from my PCs (nearly all of which are running Linux, OpenBSD, Haiku, or something other than Windows or macOS) is a no-go. This is okay for my immediate use cases (I just need it to route between some synth modules and controllers, without necessarily needing the PC to do any processing in-between), but it'd be cool to get the PC side of things working, too.
There's an existing out-of-tree Linux driver¹ that looks promising, but AFAICT it only does the bare minimum of exposing the MIDI ports for use with e.g. JACK, and it's also unclear how stable it is and whether it really does support the XT (the README says the kernel panic got fixed, but there are open issues about it; the README says the XT's supported, but there are open issues about that, too). I'd like to be able to create new routing presets and stuff like what the proprietary companion app can do, and I'd also like to be able to use the thing without needing to shove extra drivers into my kernel, and I'd also like to be able to use the thing on my OpenBSD and Haiku boxen, so I've been perusing LibUSB docs since a userspace USB driver that then presents the relevant MIDI ports and some tooling to reroute the MIDI ports as desired seems like something useful. This article happens to be exactly what I've been looking for w.r.t. a starting point for such a userspace driver.
I packaged that driver on the AUR, since I’ve got the same device. I can’t get the binary blob to work (admittedly, I haven’t tried very hard) and it’s not high on my list of priorities so I’m okay using it as a dumb MIDI tool.
I wish I could have read this article years ago. Reverse engineering my laptop's features would have been so much easier. My keyboard LED program is still among my favorite projects.
Really useful introduction! Working with low-level hardware APIs is challenging but rewarding. The abstraction layers in modern OS make it easier but understanding what's underneath is invaluable.
USB devices cannot directly address host memory like PCIe or FireWire, but the XHCI controller does DMA to/from host memory, and most USB device controllers have some kind of DMA between USB and the device's RAM.
Ages ago when I was trying to create a simple USB device, I found that there is very much zero information how to do it - e.g. how to correctly write descriptors and so on. The typical advice was: find similar device to what you want to make, copy its descriptors and adapt to your own device using trial and error.
Sounds like USB is a wonderful standard. Am I wrong?
Nice article, happen to have been working on a usbip system for my Macbook M3 using similar tech.
Worth noting that, this approach is limitted on newer macOS systems: can't build libusb "userspace USB drivers" for devices that are supported by macOS. Without manually disabling some Security features, can't override drivers for system-recognized USB devices.
If you are interested in doing this in golang I wrote a library to avoid needing cgo: https://github.com/kevmo314/go-usb
I use this to access UVC devices correspondingly without cgo: https://github.com/kevmo314/go-uvc
And for Rust, https://github.com/kevinmehall/nusb
But this kinda expects that your USB driver is the application code too, no? This is less of a driver and more of a library + program. If I have, say, a USB to Ethernet device, how do I hook this into the ethernet adapter subsystem?
Things which are relatively standard tend to get good generic support: Ethernet devices will generally be USB/CDC/ECM or RNDIS, for example. That may Just Work (tm) if it has the right descriptors.
The userland approach is much more useful for weird or custom devices. In particular, on Windows you can do one of these user space "drivers" without having to bother with driver signing, and if you use libusb it will be portable too.
(I maintain a small USB DFU based tool for work)
DFU - great example. If you have a USB device that has a DFU class that needs a custom driver, can dfu-util and the like hook into these userspace drivers? Or do you also need to maintain the application part?
Dfu-util is one of those "user space drivers", so if you have a nonstandard protocol you'd have to add it directly to dfu-util. There's no intermediate API.
It's not easy to set up a fake or "remapped" USB device on most OS as far as I'm aware, if you were trying to write an adapter program that modified USB packets.
dfu-util actually also just uses libusb under the hood! Any class or device that doesn't have a driver baked into the OS can be implemented like this. And if you'd need the DFU functionality in a different application, you may be able to just simply link parts of the dfu-util tool into your application
On Linux you could create a tun/tap device from your application and translate data sent over that to requests sent to the ethernet adapter.
Of course, when you're doing these things in userspace you either need some way of communicating with the Kernel or for the other subsystems to be in userspace as well.
Not to be too facetious but a great place for communicating with the kernel where there are a ton of other driver subsystems is... the kernel.
Possibly a good addition to the article would be parallel development of an lkm. I guess it wouldn't have that windows interop but I would also be interested to see how this driver would be implemented on Windows. If it's idk 10x as many lines in the kernel vs userspace, that's a great benefit to the userspace approach.
Driver signing is a killer issue on Windows; if you put your machine into dev/unsigned mode you get an ugly banner that can't be turned off.
Much easier to design the device to avoid that. E.g. by abusing USB-HID. The desktop USB missile launcher toy is USB HID, for example.
No need to pretend to be HID. Windows has WinUSB for userspace USB drivers that don't need special signing.
Arguably all these other subsystems shouldn't be in the Kernel either but that's a different topic :)
There are quite a few benefits to doing these things in userspace over the Kernel, not really necessarily just because of the code size:
- The code is much easier to write and debug, you just write code like you always would.
- Bugs don't have the possibility to taking down your entire system or introduce vulnerabilities
- Especially on Windows, everyone can do this without requiring an impossible to get driver signing certificate
In HFT user-space networking drivers have a long history - there is too much latency induced by switching from kernel to user space to handle networking.
> OpenOnload: A user-space network stack that intercepts socket calls to bypass the kernel network stack, accelerating standard socket operations for faster networking.
> Netmap: A framework providing a simple API for high-speed packet I/O in user space, bypassing much of the kernel overhead for efficient packet forwarding and filtering.
https://dysnix.com/blog/high-frequency-trading-infrastructur...
Perfect timing. I'm expecting to get my hands on a MOTU MIDI Express XT from my local Guitar Center within the next couple days (I paid for it when it arrived there a couple weeks ago, but they have a mandatory waiting period on used equipment to make sure it ain't stolen), which unfortunately uses some weird proprietary protocol instead of class-compliant MIDI-over-USB — so any use over USB from my PCs (nearly all of which are running Linux, OpenBSD, Haiku, or something other than Windows or macOS) is a no-go. This is okay for my immediate use cases (I just need it to route between some synth modules and controllers, without necessarily needing the PC to do any processing in-between), but it'd be cool to get the PC side of things working, too.
There's an existing out-of-tree Linux driver¹ that looks promising, but AFAICT it only does the bare minimum of exposing the MIDI ports for use with e.g. JACK, and it's also unclear how stable it is and whether it really does support the XT (the README says the kernel panic got fixed, but there are open issues about it; the README says the XT's supported, but there are open issues about that, too). I'd like to be able to create new routing presets and stuff like what the proprietary companion app can do, and I'd also like to be able to use the thing without needing to shove extra drivers into my kernel, and I'd also like to be able to use the thing on my OpenBSD and Haiku boxen, so I've been perusing LibUSB docs since a userspace USB driver that then presents the relevant MIDI ports and some tooling to reroute the MIDI ports as desired seems like something useful. This article happens to be exactly what I've been looking for w.r.t. a starting point for such a userspace driver.
----
¹: https://github.com/vampirefrog/motu
I packaged that driver on the AUR, since I’ve got the same device. I can’t get the binary blob to work (admittedly, I haven’t tried very hard) and it’s not high on my list of priorities so I’m okay using it as a dumb MIDI tool.
I wish I could have read this article years ago. Reverse engineering my laptop's features would have been so much easier. My keyboard LED program is still among my favorite projects.
Really useful introduction! Working with low-level hardware APIs is challenging but rewarding. The abstraction layers in modern OS make it easier but understanding what's underneath is invaluable.
The C++ looks messed up. I have yet to come across a keyboard that can type an arrow.
It's just a programming font ligature. If you copy and paste it you'll see the actual characters e.g.
(It's also modern C++ trailing return type.)Some developers like ligature based fonts. They combine 2 characters into one glyph
Thank you and the others who were kind enough to explain this. I've avoided such fonts like the plague. Didn't know it did arrows like that.
Any keyboard can type “→” if you set up a compose key :)
It's just "->" - the ligature font just renders it as a unitary arrow
Dumb question.. do USB devices support DMA? Is it done through the host? Or does the USB device always push/pull data to host memory?
USB devices cannot directly address host memory like PCIe or FireWire, but the XHCI controller does DMA to/from host memory, and most USB device controllers have some kind of DMA between USB and the device's RAM.
>>> Say you’re being handed a USB device and told to write a driver for it.
Hand it back, with a request to prove that it can't be done with one of the devices that the OS's already recognize as virtual COM ports.
Ages ago when I was trying to create a simple USB device, I found that there is very much zero information how to do it - e.g. how to correctly write descriptors and so on. The typical advice was: find similar device to what you want to make, copy its descriptors and adapt to your own device using trial and error.
Sounds like USB is a wonderful standard. Am I wrong?
Does the adb tool use libusb or a kernel driver?