Since I got my Gen 7 X1 Yoga I was following the IPU6 driver development pretty closely - how we moved from multiple patches, to DKMS, and now to "it just works". So many of us (and probably countless users in the future) owe so much to the work done by Hans de Goede. Kudos! Intel, on the other hand, could have done a much better job it seems.
Does this help with camera support on old Intel BayTrail hardware? If not, what's needed to make that work? (Asking since that HW turns out to be quite cheap and widely available on the used-items market.)
For high-level summaries, I actually appreciate such framing because the Linux ecosystem sorely lacks the coherence found in other platforms:
"USB4 V2 is supported in Windows 11 24H2"
"The Translation API is now available in macOS 15"
"To use the built-in camera in these specific laptops, you need Linux kernel 6.10+, libcamera 0.3.1+, pipewire with some downstream patches, Firefox 116..."
You get the idea. For users and application developers, "Fedora 41" represents a _coherent whole_ (roughly) and is a more productive subject to center discussions around.
USB cameras are interfaced over USB 2.0 because USB 3.0 is extremely difficult to make work from an EMI perspective running through a laptop’s hinges and lid. USB 2.0 means heavy compression to get 1080p60, and on pretty mediocre ISPs in those low cost camera controllers.
MIPI is easier from an EMI perspective and lets you use a much better ISP on the processor.
USB-2.0 is up to 480mbps which I believe is several multiples higher bit rate than even the highest quality 4k Blu-ray disks. I don't see why "heavy" compression would be required.
Edit: doing some quick math, it seems like transmitting a raw/uncompressed signal at even 720p60 far exceeds the bandwidth of USB-2.
So if the goal is to shift all transcoding to the CPU, MIPI does make sense.
Cameras use isochronous mode, which maxes out at less than half the theoretical max USB 2.0 bandwidth, and they use MJPEG compression on each frame rather than something like H.265 that can compress much more by leveraging P/B-frames.
> because USB 3.0 is extremely difficult to make work from an EMI perspective running through a laptop’s hinges and lid
Is that an issue with USB 3.0 causing interference on the 2.4GHz band and thus interfering with Wifi/BT antennas mounted in the lid? It can't just be a signal frequency issue on the USB side, since eDP signals for the display itself are much higher bandwidth.
> MIPI is easier from an EMI perspective
In the "MIPI family" (e.g. FPD-link, GMSL) there are solutions for longer lengths via coax, but MIPI CSI itself seems pretty focused on short distance links (as expected, considering its phone industry roots). It seems like the distance from processor to hinge to top-of-screen could easily get close to the nominal maximum length of MIPI CSI. Or are laptops already using MIPI-over-coax to simplify the cabling through the hinge?
At least for the Lenovo ThinkPad X1 Carbon Gen10 I know that they use a hybrid FFP/micro coax design. FFP to fit behind the display panel as the cable runs up to the sensor assembly and micro coax for fitting through the hinge.
I don't think you'll see a lot of purely coax based designs, given the trend to thin displays with small bezels.
Other laptops like the Dell XPS 13 Plus run straight FFP from the mainboard all the way to the sensor.
Making USB webcams that fit into the bezel of a laptop screen means they are basically universally terrible. In part this is because there really aren't many "native" USB imagers, almost all of the sensors are natively MIPI CSI devices, with a MIPI-USB interface that performs limited image processing and compression to present the UVC interface. Given the size/power/thermal constraints of a laptop bezel, there's very little you can do. If you compare laptop webcams to phones with similar size constraints on the sensor/optics but much more capable image processing units, phones do a much better job. Even if you have much more processing power available in a laptop CPU/GPU than a phone, once the camera feed has already been processed and heavily compressed there's nothing you can do to get that information back.
Historically, PC processors didn't have a native MIPI interface, and laptop vendors wouldn't have accepted the costs of adding an additional image processor to the motherboard or the effort to standardize a new camera interface. The first PC processors to support MIPI may have been Intel's "Silvermont" mobile/embedded processors in 2013, but didn't come to Intel laptop SoCs until some "Skylake" models with IPU3 then became fully mainstream with IPU6. Now, direct-to-SoC MIPI plus on-SoC image processing unit is widely available. Additionally, in the intervening years MS broke any all-cameras-are-UVC assumptions by using MIPI cameras on Surface devices and thus getting the software interfaces ready.
Doing the image processing in software would use too much energy. The IPU6 is a PCI device that does complete image processing, handing fully baked frames to the system. If you wanted to do what the IPU does but on the CPU, you'd need to use the vector units and they use a lot of power.
Since I got my Gen 7 X1 Yoga I was following the IPU6 driver development pretty closely - how we moved from multiple patches, to DKMS, and now to "it just works". So many of us (and probably countless users in the future) owe so much to the work done by Hans de Goede. Kudos! Intel, on the other hand, could have done a much better job it seems.
This implies Microsoft Surface pro 7+, 8 & 9 support for webcams too.
https://github.com/linux-surface/linux-surface/wiki/Camera-S...
IPU6 for image sensors feels so much like GDI print drivers from the 90s
Does this help with camera support on old Intel BayTrail hardware? If not, what's needed to make that work? (Asking since that HW turns out to be quite cheap and widely available on the used-items market.)
> Currently the IPU6 integrated in the following CPU models works if the sensor + glue hw/sw is also supported
I know I'm a little out of the loop, but why on Earth is the camera integrated to the CPU!?
My understanding is that it's not the camera that's integrated, but the ISP - image signal processor.
Isn’t driver support tied to a kernel version? Why would it be tied only to a fedora version?
Edit: the whole chain is kernel → libcamera → pipewire | pipewire-camera-consuming-app, from the article. So other distros will be getting it too
For high-level summaries, I actually appreciate such framing because the Linux ecosystem sorely lacks the coherence found in other platforms:
"USB4 V2 is supported in Windows 11 24H2"
"The Translation API is now available in macOS 15"
"To use the built-in camera in these specific laptops, you need Linux kernel 6.10+, libcamera 0.3.1+, pipewire with some downstream patches, Firefox 116..."
You get the idea. For users and application developers, "Fedora 41" represents a _coherent whole_ (roughly) and is a more productive subject to center discussions around.
Yes, and work like this really makes me appreciate the impact of work that Red Had pays for has on the greater Linux eco-system.
As someone noted, these cameras had support on Ubuntu, but those changes were not upstreamed to the different projects.
Also, if people want more info you can look at this presentation: https://archive.fosdem.org/2024/schedule/event/fosdem-2024-3...
Nobody is saying other distros won't. This is just talking about it in Fedora and what it took to make it happen.
Are there benefits to moving away from USB UVC and handling the image processing on the CPU?
USB cameras are interfaced over USB 2.0 because USB 3.0 is extremely difficult to make work from an EMI perspective running through a laptop’s hinges and lid. USB 2.0 means heavy compression to get 1080p60, and on pretty mediocre ISPs in those low cost camera controllers.
MIPI is easier from an EMI perspective and lets you use a much better ISP on the processor.
USB-2.0 is up to 480mbps which I believe is several multiples higher bit rate than even the highest quality 4k Blu-ray disks. I don't see why "heavy" compression would be required.
Edit: doing some quick math, it seems like transmitting a raw/uncompressed signal at even 720p60 far exceeds the bandwidth of USB-2.
So if the goal is to shift all transcoding to the CPU, MIPI does make sense.
Cameras use isochronous mode, which maxes out at less than half the theoretical max USB 2.0 bandwidth, and they use MJPEG compression on each frame rather than something like H.265 that can compress much more by leveraging P/B-frames.
> because USB 3.0 is extremely difficult to make work from an EMI perspective running through a laptop’s hinges and lid
Is that an issue with USB 3.0 causing interference on the 2.4GHz band and thus interfering with Wifi/BT antennas mounted in the lid? It can't just be a signal frequency issue on the USB side, since eDP signals for the display itself are much higher bandwidth.
> MIPI is easier from an EMI perspective
In the "MIPI family" (e.g. FPD-link, GMSL) there are solutions for longer lengths via coax, but MIPI CSI itself seems pretty focused on short distance links (as expected, considering its phone industry roots). It seems like the distance from processor to hinge to top-of-screen could easily get close to the nominal maximum length of MIPI CSI. Or are laptops already using MIPI-over-coax to simplify the cabling through the hinge?
At least for the Lenovo ThinkPad X1 Carbon Gen10 I know that they use a hybrid FFP/micro coax design. FFP to fit behind the display panel as the cable runs up to the sensor assembly and micro coax for fitting through the hinge. I don't think you'll see a lot of purely coax based designs, given the trend to thin displays with small bezels. Other laptops like the Dell XPS 13 Plus run straight FFP from the mainboard all the way to the sensor.
Making USB webcams that fit into the bezel of a laptop screen means they are basically universally terrible. In part this is because there really aren't many "native" USB imagers, almost all of the sensors are natively MIPI CSI devices, with a MIPI-USB interface that performs limited image processing and compression to present the UVC interface. Given the size/power/thermal constraints of a laptop bezel, there's very little you can do. If you compare laptop webcams to phones with similar size constraints on the sensor/optics but much more capable image processing units, phones do a much better job. Even if you have much more processing power available in a laptop CPU/GPU than a phone, once the camera feed has already been processed and heavily compressed there's nothing you can do to get that information back.
Historically, PC processors didn't have a native MIPI interface, and laptop vendors wouldn't have accepted the costs of adding an additional image processor to the motherboard or the effort to standardize a new camera interface. The first PC processors to support MIPI may have been Intel's "Silvermont" mobile/embedded processors in 2013, but didn't come to Intel laptop SoCs until some "Skylake" models with IPU3 then became fully mainstream with IPU6. Now, direct-to-SoC MIPI plus on-SoC image processing unit is widely available. Additionally, in the intervening years MS broke any all-cameras-are-UVC assumptions by using MIPI cameras on Surface devices and thus getting the software interfaces ready.
Cheaper camera hardware, more advanced software with computational photography?
Doing the image processing in software would use too much energy. The IPU6 is a PCI device that does complete image processing, handing fully baked frames to the system. If you wanted to do what the IPU does but on the CPU, you'd need to use the vector units and they use a lot of power.
Yeah it's only a kludge until the processing can be done on the GPU, and maybe one day, a proper driver can be developed for the ISP.
Which kernel version has the latest fixes?
Wasn't this already possible through RPMfusion?
For me it was for a bit, but it was very fragile, then eventually broke and didn't recover.
If you lack ideological purity these have been working in ChromeOS for 4 years and Ubuntu for 2 years.