A year ago, I got a Raspberry Pi Model 4B 2GB ($35 board, $1.95 heatsink, $7.95 power supply). It's fine but nothing really works and all roads lead to closed sourceville. I was astonished how bad the closed source ecosystem is on Raspberry Pi. It is too slow to play video without acceleration, but the acceleration is a mess. The supported accelerated video player (omxplayer) is buggy, awful UI, and only supports about 10% of the media files I throw at it. HDMI initialization is a mess (32bpp, anyone?). GL is a mess. Power management is awful.
The HDMI CEC story is illustrative. The normal cec-client is crappy, so I decided to make my own using their VCHIQ interface to the firmware. The open source component is purely wrappers. Understanding the wrapper is understanding nothing, you need to understand the underlying firmware driver. But *that* is not documented. And it has awful choices about how it was factored. For example, it doesn't seem to be possible to trigger a proper CEC address assignment probe. cec-client authors found the same problem and simply had it retry until it succeeds, with a comment saying they don't know why it's broken. I made it a little further at diagnosing it than they did, and I found a better (reliable) mechanism but it violates the CEC standard (even though it works). The VCHIQ CEC interface is neither high-level nor low-level.
VCHIQ itself is awful. They made a novel generalized message passing interface to interact with the firmware driver. But they made every choice wrong! To start with, it's not actually generic at all, it needs to know specially about every firmware interface. So every VCHIQ service has two open source wrappers, one within the kernel VCHIQ driver and one within the userland library. To interact with it requires a bunch of threads blocked waiting on ioctl() responses, which the Pi-provided VCHIQ libraries create under the covers without telling you. I guess threads are cheap but it's disheartening to realize how poorly all this stuff is coded, and how it forces a poor model upon your user program. I replaced tens of thousands of lines of poor code with my 1599-line CEC client.
The interface should be an interface that exposes the functions of the chip, and then the minimal hack to get it to work should be open source and use that interface! It's just that they made every design decision wrong. I think it is more than just limits of the Broadcom SoC, and more than just the Broadcom NDA. I think the engineer within Pi Foundation that did this work was a nincompoop. The sort of guy you kind of resent coming after in an open source project but you appreciate his work...but if you can't see the source then he is your enemy and you hate him.
So the good news is that the latest Pi firmware uses a more standard interface. Along with kernel mode setting they have switched to the kernel CEC layer, instead of exposing it through VCHIQ. Which is a good move (if it works), but it means all my work is already obsolete. Worst of all possible worlds from a user perspective. If I upgrade to the newest firmware, I'm basically starting over from scratch on my stupid hacks. And they never worked, so I fully expect to do as much work again with the same non-reward.
I'm not an open source purist, but it has to work. If nothing works, I need to be able to fix it! The online community can't do anything but provide magic /boot/config.txt commands to manipulate obsolete versions of these blobs...it sucks! They have the largest community but that community is on the outside of the wall and is systematically incapable of contributing.
Raspberry Pi is extremely impressive for its low price but I'd gladly pay twice as much for a board that doesn't constantly frustrate. I don't think that point on the price/performance curve is worth filling. It certainly shouldn't be the biggest noise in town.
My use case is pretty mild, maybe I didn't really need it at all. But the fact that it isn't even suitable as a media player -- and probably never will be! -- is frankly astonishing. The one really compelling thing it does properly is it plays MP3s through its line out jack. I hooked it up to a relay (controlled by its GPIO) that lets me select my speakers between the TV's line out and the Pi's line out. It works perfectly and lets me play MP3s a lot more easily than Android TV does. Other than that, it's nothing but frustration and I never use it.
So I just ordered a ROCKPro64 4GB from Pine ($79.99 board, $3.59 heatsink, $2.99 fan, $12.99 power supply). Almost exactly twice the cost. It would've been less if the 2GB $59.99 board wasn't out of stock. (FWIW, all Raspberry Pi boards have been out of stock for months, near as I can tell.) I had a good experience with a Rockchip RK3288 Asus Chromebook C201, that makes me optimistic that the open source driver experience will be more uplifting, even though the RK3399 seems to be hotter (I bought a fan, sigh).
More than to use it, I simply want to know if it's as frustrating. I am testing the hypothesis that there are better boards out there. Are we suffering under Raspberry Pi because of simple inertia? Or are the alternatives really so handicapped by their tininess that Pi deserves to dominate the cheap ARM SBC niche? Stay tuned.
It finally came, and I finally got a chance to play with it.
The heatsink was easy to install and seems secure.
I went to unplug the HDMI and USB from my pi4 to move it over and hit my first stumbling block -- the pi4 takes micro-HDMI, but the rockpro looks like standard HDMI. Had to take a timeout to go get a new cable.
I followed the directions at https://wiki.pine64.org/wiki/ROCKPro64_Software_Release#Debian. Some rockpro-specific image with the standard debian installer after. Put it on a big microsd card, and turned the thing on. It recognizes my USB keyboard, and shows the Debian installer on my HDMI TV (but does not CEC-turn-on that TV). The ethernet port works, once I discovered that I never plugged in the other end of the cable.
It thinks the TV is wider than it is, or something -- the display is missing the outer 5 characters or so but it doesn't seem to matter. It doesn't have the shell on ALT-F2 that I have learned to take for granted with the Debian installer and I don't know why not (update: I think I figured out it's because my dumb flashquark keyboard claims to be an Apple keyboard, and needs options hid_apple fnmode=0 in /etc/modprobe.d/flashquark.conf to unmangle the function keys). It is going pretty slowly and I think that is probably because the microSD is not fast. I told it to overwrite the installer on the microSD card. Hope that was a good idea.
So it acted like it installed everything successfully, and then rebooted itself, and now it is not showing anything on HDMI. I can't say I'm surprised...I really expected to need to tell Debian something about the rockpro bootloader, a subject I am ignorant of. I'm not interested in interfacing with the UART so...
Next I tried an armbian image (Armbian 22.02 Bullseye). It stalls at booting the kernel. Found this forum post, which suggests unplugging all USB for the first boot. Which worked! It is again a few characters wider than my actual screen.
Even on subsequent boots, it hangs if the USB keyboard is plugged in. Sigh.
In lieu of getting to the bottom of it, I moved the keyboard from one of the USB2 ports to the USB3 port. Now it boots but doesn't see the keyboard. Sigh.
Replaced systemd with sysv-init. Uninstalled command-not-found smartmontools vnstat haveged. Install nvi. Configure wireguard, NFS. Put logs in /run so that root can be read-only (edit /etc/init.d/mountkernfs.sh, /etc/default/tmpfs, symlinks in /var).
Subjectively, this thing is very slow in a way I've never seen before, but which I only notice when using apt. I don't think it has anything to do with I/O. For example, apt-cache search xserver on my Pi takes 1.5 seconds, but on the rockpro it takes 20 seconds! And the whole apt database is in cache. And wow it gets *much* slower if I use cpufreq-set to limit the CPU speed to 408MHz! So it seems to be the CPU that is the limit here! This is a real mystery because the rk3288 in my old Chromebook is as fast as the Pi, and the rk3399 in the rockpro is supposed to be faster by 30-70% in single-core tests.
I contrived a memory bandwidth test, and the rockpro is slightly faster at it than the Pi is. On a simple gcc task, the rockpro is 10% faster, and on a more complicated build it seems to be 30% slower than the pi. Which is well within the range, especially since they're different versions of gcc (Pi gcc 8.3.0, rockpro gcc 10.2.1).
So I have no explanation...it's 13x slower at apt but about the same at everything else? Makes me wonder if there isn't an armbian flaw in the configuration or build of apt.
It is apt! I found all sorts of armbian garbage in /etc/apt/apt.conf.d! dpkg --purge apt-file, and deleted 02-armbian-compress-indexes 02-armbian-periodic 20auto-upgrades 50unattended-upgrades. And deleted the whole directory /var/lib/apt/lists, and re-ran apt update. It turns out that they enabled some compression thing on the index, it was using lz4 which I guess is way slow. Other people complained about it too. Anyways, now apt-cache search xserver takes 0.8s!
Overall, I have to say I am not very impressed with armbian. But I don't know what struggle I really would have faced with regular Debian. Probably all I needed to do was install uboot into the SPI first, and Debian would have worked trouble-free. But maybe there's some patched driver sort of scenario that would make me long for armbian. *shrug*
Though I really do appreciate their decision to include chrony by default. I have long thought that ntpd actually isn't very good at being an ntp client, and chrony is apparently what I didn't know I wanted.
It idled overnight at 45C, which is even hotter than my pi's typical 40C. It has a decently large heatsink but no fan. It doesn't make sense to me because I thought the whole point of big.LITTLE was reduced power consumption during low load. Shoots up to 65C if I busy-loop one of the cores, compared to 55C on the pi.
Turning off extra cores with echo 0 > /sys/devices/system/cpuX/online has no effect. There is cpuidle but no driver. My chromebook idles at 28C (ambient is 21C), but its rk3288_cpuidle driver is just a stub, it is "just" using WFI. The pi kernel doesn't have cpuidle.
I found an rk3399 datasheet which says it has different power domains for the different cores. So I believe disabling the cores should reduce the temperature, but it's also possible that it is already disabling the cores as a matter of fact but something else is drawing all the power.
This is already a big difference from Pi -- I haven't hit any closed source walls. The closed bootloader isn't still running on some obscured core getting between me and anything, so I can write to the control registers directly. I've got a (leaked?) giant PDF of all of the internal control registers on the rk3399, including the PMU. OTOH, this could be an enormous amount of work and I may still not get to a satisfactory result.
If I had this as a laptop, I would probably go through the effort but I'm gonna see if I can get away with not doing it since it's just a TV box for me. It's disappointing that others have had it as a laptop (or phone!) for years and not solved it. It might mean it's a harder problem than it seems. If the Pi (with its huge user base) was this open and people hadn't solved it yet, I would think it isn't possible. For example, I have an old Intel core 2 duo laptop which is this open, and I even started writing directly to its hardware registers and eventually determined the CPU is just a hog. People really had come along before me and tried all the obvious things.
I'm not enthused about all of the black magic sort of X11 directions out there. I expect I'll go down that path eventually but I want to feel out the naive approach first. Installing xserver-xorg-video-fbdev. Added this to a file in /etc/udev/rules.d, so my non-privileged user can run X11:
SUBSYSTEM=="tty", KERNEL=="tty[0-9]*", GROUP="tty", MODE="0660"
X11 works, and mpv -vo sdl is usable, but not good -- it appears to drop frames. And --audio-device=alsa/hw:hdmisound works too but there is some sort of level adjustment causing clipping.
So effortlessly as good as my old rk3288 laptop, it is not. Maybe I'll try to get GL to work later as an output device for the software decoder.
I spent $72 for a raspi that can't really play video and which is too closed / locked down to make any progress on any issue. And I spent $111 on a rockpro that has all the same issues as the pi (and one additional issue), except that it's open enough that I have a prayer of fixing it. But here's the clincher: I simply don't want to.
And add on the fact that both of those things are hot. The raspi because it's not a mobile processor and whatever powersave features it may have are hidden behind a closed firmware and closed SoC documentation. And the rockpro...honestly, I still believe the kernel drivers just suck and don't take advantage of the available power save features but I simply don't know. It's nice to be able to see what to bang on, but discouraging to know I might bang on it a lot without any progress. My early Core 2 Duo laptop left a bad taste, I invested a good day of effort into learning that it simply unconditionally draws a ton of current even in its lowest power-save mode. Sigh.
And I was thinking, my celeron laptop is great. I'm pretty sure it has lower power draw than the raspi, and much higher performance, and it plays video without any effort, and so on. So I bought a used (2016) NUC for $97. NUC6CAYH Celeron J3455, 8GB RAM, 250GB SSD. I mean, this is barely more expensive than the raspi, less if I count the $32 I spent on SSD.
Anyways, quick review. I *hate* Intel Visual BIOS. There are features (like boot order selection) that only work with a mouse! It has laggy animations, and an awful keyboard visit order. Certain uninteresting options crash it. It is bad. It is very bad.
The NUC sends an HDMI CEC ActiveSource 1 when it turns on. That's stupid because it isn't source 1 (and it should know that). I was astonished that the nVidia Shield TV doesn't speak CEC properly (it sends a time-out power down command even if another device is the current active source, an unforced error). I was astonished that the cec-client was so just-a-useless-hack. I was astonished that the raspi CEC was so closed. And here I am astonished again. HDMI CEC is so simple, so clean, so useful, so user-facing. Why can't anyone get it right?
The NUC has a few stupid LEDs. It weighs rather a lot, I think just from all the metal "midframe" sort of stuff inside of it. I opened it up to see, and it has a huge 2.5" SSD that seems to be SATA. Wow. Was 2016 really that long ago? It has wifi on a cute little daughterboard...I suppose that wifi port is M2 and could be an SSD.
Intel Visual BIOS is an almost bottomless pit of infelicities. Through packet inspection, I learned that its tftp client tries to specify "blksize 1468", which is not in the original TFTP RFC, and is unsupported by the default tftpd. I used tftpd-hpa instead, and it was then able to download the pxelinux.0 but it still didn't work. it just showed a success message and then went into Windows. Reading about it, it seems like this is a flaw in the BIOS, supposedly it is showing a message like "Press enter to boot from downloaded image." for a fractional second, and I just can't get there in time!
PXE is so awesome, I had really come to take it for granted. But it simply didn't work. So I put the netinst iso on a USB stick, and that worked flawlessly. And Linux works great.
Put defaults.pcm.device 3 in /etc/asound.conf and it uses the HDMI audio by default. Installed xserver-xorg-video-intel, and it seems to work and be sensible from the first try. Using mpv, it effortlessly plays all my videos, synced and scaled. Got dosbox to use opengl scaling by specifying
[sdl] windowresolution=1280x720 output=openglin dosbox-0.74-3.conf. These are all things that seemed to be non-starters on the pi.
It idles at about 48C, which is slightly cooler than my pi's current 52C. It doesn't seem to be running its fan (though it does have one, I told it to be conservative about it in the BIOS settings). It does probably have a more significant heatsink than the pi, so it is fair to say the Celeron J3455 might be even worse than the pi at idling. However, it barely gets warmer when I play video, and cpufreq-info shows that it calms down pretty quickly after the video starts playing (back to its baseline 800-1000MHz).
If I run a busy loop to stress-test it, the temperature gets up to about 69C in a few minutes, but it still doesn't seem to turn on the fan (and it doesn't feel hot on the outside). Temperature drops back down to 50C almost instantly once the load is stopped. I'm not sure if there's something wrong with it that it isn't turning the fan on. That's in an ITE8987D embedded fan controller, which apparently it is an uphill battle to get Linux to talk to. Hopefully it doesn't turn out to be necessary to unravel.
The only thing that's still broken is CEC. The kernel CEC device (/dev/cec0) doesn't seem to be available. I'm pretty concerned I might not have any luck with it, tbh. Intel's documentation says:
The following Intel NUC Kits have the above external CEC header and an onboard HDMI CEC controller that the BIOS controls. The onboard HDMI CEC controller only supports bidirectional power on/off control: NUC6CAYH
That sounds awful. It sounds like the pi but worse, especially since the on/off control doesn't really work! The BIOS documentation suggests that it can't turn it on or off on request either, but only when entering/leaving sleep state S3 (suspend to RAM) or deeper.
Before I get too far trying to hack ACPI, I'm gonna try upgrading the BIOS and see if anything gets better, though.
Got AYAPLCEL.86A.0071.UEFI.zip from Intel. It's 2022, that's promising.
Ugh. It's updating but it says:
Flashing motherboard firmware: Current revision: AYAPLCEL.86A.0071.2021.1208.1638 Updating to revision: AYAPLCEL.86A.0071.2021.1208.1638
So that seems like a no-op. It's *so* buggy, I can't believe this is a maintained Intel product. The BIOS crashes so often when navigating its stupid menus that I actually feared it was unstable hardware (and Windows configurator also would crash while just sitting there), but it seems pretty solid in Linux.
It does have an option in the BIOS to tell it that it's "HDMI 2" instead of "HDMI 1". Abominable.
Anyways, Intel advertises the fact that GORITE and pulse-eight sell aftermarket CEC boards for it??? That's insane! Apparently there is a 4-pin header, which Intel documents: 5VSTBY, GND, PWR_SWITCH#, HDMI_CEC. I happen to know the HDMI_CEC standard is dead simple, I am tempted to hook up an stm32 board to the HDMI_CEC pin. Surely there is some sort of GPIO on this thing??? I mean, it is absolutely not easier to do the subset of HDMI CEC that their BIOS obviously can control than to do it right, so they must already have a GPIO somehow hooked up to it??? Would they really implement it in a closed off coprocessor??? What is this, pi??
From reading about this, apparently HDMI CEC is still a hot new thing that users are excited to have support for! This is crazy!
Anyways I guess I'll buy this stupid pulse-eight board...it is absolutely insane that Intel didn't include any way to interface with their built-in CEC adaptor, since it pretty much has to be complicated enough to allow it. The pulse-eight board lives between a CEC header and a USB header inside the NUC.
I am just absolutely flabbergasted that someone (Intel) managed to create a worse closed CEC firmware than raspi did.
Anyways it forces a decision on me, and I resent the decision probably more than the work:
There's no question, I want to build my own. I'll have GPIO leftover on the side to become the speaker input select (TV versus NUC). I also very much want to defeat the damn Intel BIOS, but there's a limitless set of hurdles there.