Asus L406 user experience

A little background. I ordered an Asus L406MA-WH02 (NB, there are older L406s with substantially different specs) for $350 on ebay to replace an Asus C201 Chromebook. The C201 is very special: $180, 1kg, 13hr+ battery (38Wh with Rockchip RK3288 ARM), 11.6" screen, but it's 5 years old and has a decently long list of minor problems.

The L406 has a 14" 1366x768 display, 1.3kg, 56Wh battery, Celeron N4000, 4GB, 64GB. So it's also light-ish and battery-ish but I've always bought 12" laptops since 2003, so it's an experiment.

It came in the mail yesterday. It's as light as expected, for its size. Not quite as thin, but I don't really care about the thinness. It is huge! I hate how huge it is. Really, it's just on the order of 2 inches wider than my old laptops, but man! I've been using it on my workbench, and the clutter kind of has these hollows for 12" laptops, but a 14" laptop...I have to move crap.

The screen is so gigantic that I feel like I have to turn my head to see the rest of it. The opposite corners live in different countries. I had the idea that people describe these 12" things as "cramped", and maybe I had always felt the same way and just never noticed without the contrast. No. I definitely prefer the smaller screen. I used to always scale up my terminal font until the terminal covered roughly the same proportion of the screen as on my last laptop, but on this one I think I might aim for the same absolute area (in inches) of the screen?

Another complaint about the screen is that it seems pretty washed out, the brights aren't very bright and the darks aren't very dark but I'm not sure that's not just an illusion. When I'm looking at any of my other laptops, their blacks aren't very dark, but they look dark in comparison to the new one?

The keyboard is fine. It has a very light feel, and I have no complaints about it. At first I thought it was bigger than my old keyboards, but I measured it and it seems to be almost identical key pitch (within 0.2mm). The keys on the edge are just pointlessly extended, but I guess that's fine. We might see how the wide keys age?

The trackpad is fine, though it's taking me a little while to get used to its approach of having two buttons delineated at the bottom. I'm thinking I'll disable that if I ever really set it up.

The battery is reported in /sys/class/power_supply/ as a 3-cell 5000mAh, apparently they really did put a 56Wh battery in the thing! Unfortunately the battery monitor on this device doesn't seem to support current_now so I will have to play some games to see how fast the various components deplete it.

Being back on x86 hardware is a neat trip. I put Linux Mint on a microSD card, but it wouldn't acknowledge it. I tried a USB stick, and it worked on the first try (just press ESC at the ASUS screen on start-up to see the list of boot options). Just dd the Mint (Xfce) ISO to the USB stick! Crazy! It has a complicated hack of different kinds of headers at the top of the stick, including a GPT table that indicates a smallish UEFI partitition (FAT32/VFAT) which contains the bootloader. It just works! Everything just works - the trackpad, the keyboard's "Fn" key, the wifi, the battery monitor, suspend-on-lid, everything.

I had no intention of using Linux Mint, I just wanted it as a test environment to gauge the size of the struggle before doing my traditional dance to get Debian installed. But even just using this "live image" Mint, I can see it is basically the same as Debian...I think I can rip out all this Xfce garbage and turn it into Devuan pretty easily. OTOH, if this is how easy it is to use UEFI with removable media, I can probably just use a stock Debian install image...

Anyways, I have a conundrum. I can't decide if I want to use this laptop. I'm not too worried about the waste of money, but I do want it to get used, so if I will be passing it on (even at a big loss), I want to be able to give it to the next guy with the stock Windows 10. So there's two things I want to do with this before I make any decisions that can't be undone.

First, I want to characterize the hardware, especially the battery consumption, to try to come up with an estimate of whether any of these 11.6" N4000 laptops might possibly satisfy, even though they all seem to have just 38Wh batteries. I can probably do that set of experiments with the Mint live image. If there's no way to get good battery life out of x86 hardware without this unusual super-sized battery in a 14" laptop, then that would really be an incentive to give the L406 a chance. OTOH, if the N4000 sips power so well that the 11.6" 38Wh laptops might satisfy then that is an indication I shouldn't even bother.

Second, I want to actually try using the thing and see how I feel about it after I've actually used it as a daily driver for a little while. That's a little complicated because I don't want to spend too much effort customizing it, and I also don't want to wipe the Windows 10. I might just install Debian on an 8GB microSD card. I think I can use USB to load the UEFI boot loader, and then instruct that to boot off of SD, then remove the USB stick after it's booted (I can't generally use the device with a USB stick stuck in it).

One concerning possibility is simply using the laptop even though I feel like it sucks. I have to weigh the three obvious kinds of suck off of eachother: flakey wifi (C201), excessive size (L406), poor battery life (12" x86 laptop). I may wind up using the C201 for many more years if that's the choice.

Installed Devuan on a microSD card. Only hassle was manually coming up with the non-free firmware-atheros package to get wifi. It automatically installed GRUB on the existing internal Windows EFI partition, which I guess is fine. Just had to copy the vmlinuz and initrd on there so it could find them (the BIOS will not talk to microSD).

It is definitely heavier, but I'm not sure it matters. After playing with it for a couple hours, the C201 feels funny, so that's a promising sign I might just get used to it. But this display is just so pointlessly large.

And it fails one actual usability test -- I can't simply carry it around by the corner of the screen the way I did the chromebooks.

I feel very good about the battery life so far. Which might actually be a good omen for finding a 12" laptop based on N4000?

I am finally fighting a long-overdue battle with fontconfig, converting some well-loved custom pcf bitmap fonts to .otb. Using fontforge. I ran into a bunch of problems and anyways eventually figured out 'xterm' will still work with the old PCF font, so converting my VGA font to OpenType is put off for another day.

And the screen doesn't seem so bad now that I've been using it for a while. I even looked at my old laptop and was able to convince myself that the black there really isn't any darker. But the L406 still seems kind of washed out, low contrast. But I also used to think the viewing angles are really bad but now that I've used it a bit, it seems more forgiving than anything I've used before. I am always adjusting the hinge on my laptop, but this one less so.

I'm still not crazy about the screen's enormous size. The web browser looks better that large, but the terminal just looks kind of crappy. Surprisingly, I'm not crazy about bigger PDF viewer either. I still can't tell if the strain of looking from one side of it to the other is a big deal.

The keyboard feels fine. It makes kind of a lot of noise. The weight required to press the keys is very low. I feel like I make more typos, and have to type more intentionally on it. It also seems to repeat keys sometimes, but only organically, never in a controlled test (unlike the Samsung Chromebook). Overall, it's very similar to the Samsung Chromebook's keyboard, which I found very decent once I got used to it. So I'm not sure if it's really a flaw in the keyboard or just my sloppy typing (which has admittedly gotten sloppier over the years).

The speakers are fine. They're clear and crisp and loud enough. I tested them once and bass was simply missing, but now it's present...?? So I guess pretty good for laptop speakers? I covered them over with tape first thing after I took it out of the box so hopefully they won't ingest metal dust. Knock on wood.

I've been using it for a couple days and I'm not revolted by it. I hate switching laptops more than I hate this laptop, I guess? I still want to do some experiments to measure power consumption, for which I will need to switch to my old laptop for a while, and I've been putting it off because it just seems like a hassle.

I mean, really, it's just a laptop. So long as it doesn't have a fan and I don't ever have to use it while it's plugged in, it's all fine.

I get caught up in these A/B preference sort of questions. Do I prefer narrow handlebars on my bike or wide ones? I don't care, but right when I switch bikes I'm always astonished at the width of the handlebars.

I've known I prefer Dvorak to QWERTY since like 1994, but how do I feel about Colemak? I switched to Colemak about 4 years ago, and I still couldn't tell if I prefered it. So I switched back to Dvorak. I switched back and forth several more times, trying to tell which I prefer. It takes about a week of agony to switch, but then it's just a keyboard, I don't like it or dislike it. I still don't know, but I got tired of trying to figure it out so I stopped. It's Dvorak today. *shrug* (If you're considering one, chose Colemak, it's much easier to learn, coming from QWERTY.)

So I'm still gonna try to estimate whether a 12" 38Wh N4000 PC laptop (like the Asus L203) would check the boxes, but I'm not planning on ditching this one for it.

At any rate, my friend told me it's easy to re-install Windows 10 if it comes to it -- supposedly the activation key is encoded in the BIOS and I don't need to worry about it? -- so I went ahead and wiped the internal SSD.

The only concrete downside of this laptop is that it's so big I am afraid to throw it around the way I'm used to. I don't particularly notice being more careful with it, but I really am. It's got a very rigid frame. We'll see how it holds up. Maybe in a year I'll be grabbing it by the screen to fling it across the room.

I never use the stock Debian kernel, even though I guess I haven't really had problems with it? I disapprove of its dependency on initrd, though I just looked in a little depth at it and it is straightforward and sensible enough (and update-initramfs is fairly trouble-free). I don't like not having a known good but not horribly bloated linux-.../.config file to modify whenever I need to add something (the bloat means slow to build). And I don't like being tied to someone else's idea of when I should update.

Anyways, I wanted WireGuard and rather than track down a module backported to 4.19, I figured I would upgrade the whole thing to 5.8. I suppose I could have just installed Debian's wireguard-dkms package and been done with it, but, you see, I didn't want to. I had to compile my own kernel back in like 1995 before I could first use Linux, because I had to modify the PCI driver not to panic on my VLB/PCI hybrid motherboard that had a bad BIOS. I'm such an old man! I just figure building your own kernel is one of the steps.

So I got the 5.8.16 kernel source and man there are a lot of config options these days! And a lot of the defaults are really painful, like -g! And then there's a ton of stuff I have just never heard of before. So just going through all the menus enabling what I thought I needed and disabling a ton of cruft took hours. After I was done, it still didn't want to boot, though it did work if I used an initrd (hah!).

After making sure all of the stuff needed to access mmc (sdhci, ahci, whatever) was built into the kernel (not dependent upon modules), it still didn't work! I could boot with init=/bin/sh and could access /dev/mmcblk0p2 and so on. From the kernel console messages, it even looked like it was booting (loading udevd etc), but it didn't show any of the messages I expected from userland.

Long story short, you need to mknod /dev/console c 5 1 in the root filesystem (before mounting devtmpfs). There's no reason sysvinit couldn't give an error message when it fails to open /dev/console (it has a valid stdout and stderr at the time), but it doesn't, so this is just something to remember. FWIW, the initrd has a default /dev/console. Sigh.

Anyways, an aside about progress. On my 486/DX4 100MHz with 4MB RAM, it took about half an hour to compile a Linux 1.2.13 kernel. Today, my quad-core AMD Ryzen 3 2200G 3.5GHz with 16GB RAM takes about 12 minutes to compile a 5.8.16 kernel. There are real reasons for that...there are more device drivers, more features, more framework. And the compiler isn't nearly as simple as gcc 2.6, either. And some of it's just my fault, not as studiously pruning the .config anymore. At some points in recent history, I've owned machines that can compile their own kernel in only 7 minutes. But still, this is pathetic.

I think it's the same dependencies-enabled problem as everything else these days. It's so easy to use a pre-existing library for something like CRC computation, so people easily throw a dependency on that in any old driver. But then that pulls in some overwrought crypto framework and it detects that it can have accelerated crypto on my processor so it wants to pull in a ton of modules, and there are so many crypto components and different users of them that it's hard to know which ones can be safely disabled. Not really that big a deal in some sense but we easily wind up with 1MB of compiled code just for what used to be a 15-line static unsigned int crc32(unsigned char *buf, int len) { ... } function that got copy-pasted haphazardly.

And make menuconfig is showing its age. There's a ton of things it could do to be more usable, especially in all the manual navigation needed to make use of the '/' search function.

On another note, since it's an x86 machine it's a no-brainer to cross-compile my kernels on my PC instead of doing that locally on the laptop.

Just once, for grins, I built the kernel natively on the L406, and it took a full hour! An hour! I'm impressed by the incredibly low performance of this Celeron N4000. I'm almost tempted to try to come up with an apples-to-apples test to benchmark it versus the RockChip.

I built a new 5.8.16 kernel for my PC as well (can you believe it worked on the first try???) so now I'm ready for wireguard. It's pretty easy to set up, a real minimum number of things you can configure. Anyways, I thought I'd take some real informal benchmarks. Time to transfer a 93MB file over my wifi using wget:

bare wlan0 21.4s 21.5s 21.0s (10% CPU)
openvpn tun0 25.5s 25.0s 25.5s (33% CPU)
wireguard wg0 22.6s 22.9s 22.9s (18% CPU)

The CPU usage estimate came from looking at the reported idle time in top. So, openvpn was already plenty good enough really but wireguard is obviously a little faster and also a little less CPU usage. I doubt I will notice the difference except maybe less state negotiation around re-connects...but really, openvpn has been doing a good job of that as well (since I got it all tuned in).

It was pretty nice setting up wireguard. I only had to go further than the quickstart guide for one tiny detail, and everything just worked on the first try, no false starts or backing up to try again.

... That was strange. I rebooted the laptop and it came up unable to mount /dev/mmcblk0p2 (root). It only saw mmcblk0boot0 mmcblk0boot1 mmcblk0rpmb (I don't know what those are all about), but neither of the GPT partitions. I simply rebooted again without changing anything and it "just worked." So there's some sort of race case or uninit memory that makes it unable to decode the GPT table??

I have been using the old C201 again to do some battery testing, and man! It is really really light, like an empty box! That's the thing I'll miss the most but I've found that in practice it doesn't seem to make a difference when I'm using it. The keyboard is actually not an improvement, but it's not bad either. It is weird to feel a keyboard with polished keys, and know that the new laptop will have them soon too. The display is so small, the text is so petite and neat and intricate. Better? Worse? It's *very* different, I can see why I gagged on the new laptop so hard.

I do feel like the white text on this old laptop is a lot brighter, it's like the same number of lumens from a big letter gets crammed into these tiny letters. Increasing the backlight brightness on the new laptop just makes the bad contrast more obvious.

In just a few minutes of poking at it while sitting in the basement (about 20ft from the wifi AP, but not remotely line-of-sight), I subjectively noticed it dropping some packets and was able to confirm it with ping. The wifi situation on the old laptop is definitely omnipresent even as I can kind of mostly ignore it.

Anyways, the battery testing. I looked in /sys/class/power_supply/[battery]/energy_now which seems to be micro-watt-hours (uWh) remaining. On the old chromebook, it also provides charge_now and current_now, which leads me to believe its "coulomb counter" is sophisticated. I'm pretty sure the one in the L406 is a dumb heuristic based on voltage or something stupid like that. After it's fully charged, for example, it takes almost half an hour of being used for it to acknowledge energy_now has decreased at all. If you ramp up power consumption by pegging the CPU then the L406 persistently under-reports current consumption for a while after that. Just a lot of signs it is a crap heuristic. Not sure about the C201.

I have had no reason to believe its overall capacity estimate is off, but I also have only gotten below about 50% once I think? Which is the same feeling I had with the C201 the first couple weeks I used it.

measure	old C201	new L406
energy_full_design	38Wh	57Wh
energy_full	32.3Wh	56.2Wh
typical consumption	~2.6W	~2.6W (2.3W-2.8W)
hypothetical current lifetime	12.4hr	21.6hr
vendor claimed lifetime	"up to 13hr"	"up to 14hr"
idle, normal backlight	2.3W	2.3W
idle, zero backlight	1.6W	1.6W
idle, 100% backlight	3.2W	3.8W
kernel build power	8.7W	6.8W
typical CPU temp	32C	32C
kernel build temp	60C	60C
kernel build time	14min	62min

They are surprisingly comparable machines. I know the C201 really does get 12 hour battery life, or better. The 21 hour figure for L406 seems reasonable. I estimated 18 hour when I first started using it, but that was before I installed the new kernel and some other changes, and back then it was typically drawing around 3.0W. I wonder if that drop in consumption was real, and if so what caused it -- a smarter kernel, smarter drivers, more specific drivers, leaving off a driver that did nothing but waste energy?

The kernel compile time is hard for me to gauge. It's not a fair comparison -- the newer kernel is probably much larger, and the newer gcc might be slower, and my L406 configuration included a bunch of useless garbage (the C201 configuration had been pruned down somewhat by Google or Asus already). But it is tempting to think that the Intel tax is being paid in performance, not in battery life or heat. That's incredible, and exactly the trade-off that would make me grudgingly accept Intel!

Anyways, the moral of the story is that the Intel N4000 seems to have "low enough" power consumption! One of these "up to 10 hour" advertised 11.6" machines with an N4000 and 38Wh battery really would have done me alright -- I could expect one of them to have similar battery life to the C201. I didn't need to switch to 14", and it is massive overkill.

It really is pretty impressive that using Linux the way I do literally gives half again more battery life than Windows? I think a lot of it is because I run my browser remotely. I don't get the impression that the "typical" Linux with systemd and gnome or whatever is particularly more efficient than Windows, or wouldn't the Pinebook Pro laptop get more than about a 6-8 hour reported battery life?

My wife's work Dell Latitude only turns the fan on high when it's working hard, and its fan is *always* on high. Windows. I snooped its network traffic and it's phoning home even when it's "sleeping." It's astounding just how much less work my computer does.

Oh, and! It makes a noise. It's from the logic board area, not the speakers. It's very quiet. A high-pitched whine or white noise with frequent clicks. More than one click a second but in no particular pattern. It's rarely absolutely quiet in here but when it is, I can hear it without even putting my head on the laptop. I wish it didn't make the noise but it's not obnoxious. Functionally, it's silent. It reminds me of the keyboard-scan whine you can hear on old HP48 calculators.

And one awesome thing about the L406 -- the trackpad's two-finger scroll is flawless. It works every time! But the trackpad is a little large so I need to set up some sort of palm rejection. And I'm still torn about having the three buttons based on where you click the hardware button.

I figured out a good apples-to-apples test that involves real life performance (make -j), with the same gcc 10.2 and the same codebase. Of course, one is building for ARM-32 and the other is building for x86-64, so it's not really the same code being generated. In my mind, that just makes the test more realistic, because I'll never build ARM code on the x86, and vice versa. For this codebase, best of 3, wallclock time, the L406 takes 85 seconds while the old C201 takes 226 seconds. So the N4000 (2017, 14nm, 4MB cache) is about 2.6x as fast as the RK3288 (2014, 28nm, 1MB cache) for roughly the same power consumption, which is what you would hope from a newer chip. Quite an accomplishment for Intel!

For grins, the same chore on my hated Lenovo U150 (core 2 duo U7300, 2009, 45nm): 147 seconds (138 seconds with gcc 4.4) -- after 7 years of progress, Intel can even outperform that without a fan, finally! On my PC (AMD Ryzen 3 2200, 2018, 14nm): 21 seconds (18 seconds with gcc 8.4). 21 seconds! That's why I do all my computing on the other end of a wifi link! The only thing heavier than make -j is a modern web browser rendering social media.

Doing these tests, I feel weird normalizing all these machines to gcc 10.2, because the changes to compiler technology happened in the same actual time-line as these old computers. But I have a suspicion gcc might have recently went downhill (seems to be confirmed). It is interesting to consider how much the actual experience improved when the slowing of software is taken into account (it's still significant).

I've come to take these speakers for granted so I wanted to record how pleased I am with them. I guess they're awful? But they're a huge improvement over my C201 speakers and I hereby praise them as tolerably usable.

The C201 speakers of course were distorted from dust ingress, and also they were distorted from the beginning by bad design -- 100% volume would cause them to clip. Bass was completely inaudible. They were generally too quiet for outdoor use. The number one thing they made was heat. I'm pretty sure that there's a certain frequency where their internal resonances turn them into a dead short, making only heat and no sound. I've certainly observed them to be making heat without sound, at any rate. Even before they were damaged.

The Samsung Chromebook speakers were just the same, as well. More heat than sound!

If this battery monitor is to be believed, the speakers are drawing 1.5W but their warmth is right at the limit of my perception. The highs are clear (as you'd expect), voice is discernable outside when cars go by (but a big truck still drowns it out). Percussion sounds complete, both the snap and the thud. Bass is, well, you can perceive the bass line if you know the song already.

In short, the "Asus SonicMaster" speakers they put in this L406 are basically the same as run of the mill smartphone speakers. Finally!

It is pretty weird to be 2020 and ranking laptop speakers by how much heat they make. I honestly did not see that one coming until I had 8 years of laptops with hot speakers. The C201 would actually cause me to pull my hand away in pain on occasion.

Wonder if they'll last since I taped over them before any ingress.

I'm still not 100% sold on this keyboard -- it's a little too easy to repeat keys. I am getting used to the lightness, though, and I can now pretty reliably type "adf" (well, "aeu" in Dvorak) without hitting the "s" ("o") in the middle. But the sound of it is really irritating. Especially the spacebar, which squeaks! In fact, I think the squeaking of the spacebar may be the only irritating feature, but man is it irritating!

I was pretty nervous to do surgery on a brand new keyboard, especially since I've never taken apart a laptop spacebar before. But I didn't feel like I had a choice. So I watched a video on youtube which suggested peeling the spacebar up from the side. That worked. The actual scissor jacks on the spacebar should be opened from the top of the key first (the top part snaps, while the bottom part slides on and can be broken by snapping).

Anyways, I got it all off in one piece, and successfully re-seated the scissor that had come undone, and found that it has two long metal stabilizer bars which connect the plastic keycap to the metal keyboard base. The hinge on the base side is metal-on-metal, which is probably the only metal-on-metal hinge in the keyboard (excepting like shift keys), so I figured that's probably the source of the unique noise.

The synthetic lube I came up with was bicycle grease, "synthetic with teflon." I put it on a fine tipped poker and got it just on the metal-on-metal hinge. Put the thing back together and it seems better. We'll see how I feel about it after it's had time to settle in, and after I've heard the keyboard in a few more sonic environments. I really hope I don't do this again.

For a work project, I have to type "VT" a bunch of times, and I discover that if you hold down RSHIFT and V then it is unable to detect a press of T. Well, the keycaps say .> and K because I use Dvorak, but you know what I mean. TV has the same problem, and holding T and V makes RSHIFT undetectable as well. And once I started questing for it, I right away found out that WN has the same flaw.

I imagine that if I sat down trying to design a keyboard scan matrix that has this flaw, it wouldn't be too hard to come up with. I should be able to forgive them. But it's lame that it's a real flaw I can't work around, and not dismissable as my fault for being a sloppy typist (it is very common to press the next key before releasing the previous key). It is surely possible to design a matrix that can detect any three simultaneous keypresses, but maybe it's harder than I want to acknowledge. But there's really no excuse when it comes to mode-shift keys like shift/ctrl/alt, they should all universally combine with any other keys without losing anything, without even losing redundancy.

Anyways this just reminds me that I want to bring my debouncing X keyboard driver onto this laptop, and I also need to figure out why sometimes palming the touchpad makes everything bad happen in the web browser while filling out a form. Do I somehow trigger a click event (it would be right mouse button)?

I think these input flaws are my only complaint about this laptop. I'd even completely forgotten that I used to hate the sound of the keyboard! Though I did just find that its wifi isn't quite wonderful in the greenhouse, but it is better than the C201 it replaced... Definitely this side of "usable."

I guess there is one thing... the wifi dies every now and then. Acts like it's associated but won't send packets. Usually assigning it to a bogus ESSID and asking it to re-associate is sufficient, but a couple times I've had to run the suspend script and re-wake it. It's roughly once a week, so it's pretty dismissable, but it surely illustrates that something isn't quite right.

Uptime is 64 days, which is nothing, but shows I haven't rebooted it since I set it up.

The keyboard continues to be the bane of this laptop. It started squeaking again, and then a couple weeks later it stopped squeaking. Pfew. And the fn button is physically sticky...not enough to matter but just enough that it feels weird if you pay attention to it for a second. And the "VT" problem continues to be a weekly frustration. And the general willingness to repeat keys is a constant but mild thorn.

But today it really took the cake. I left it off for a couple hours in the middle of the day and when I came back, there were about half a dozen keys where if you pressed them, another key would trigger at the same time. So if I pressed "yyy" (or "///"), then on the screen would appear "y/y/y/". I'm pretty sure two rows in the scan matrix got shorted together somehow. Don't know if there was dust, or moisture (it *is* humid as all get out here), or a misaligned/loose ribbon cable, or what.

I tried a few things, including a complete power cycle, and it was still failing. Got out the old C201 chromebook, and used it just long enough for the battery to start to pick up a charge, and then the new L406 magically fixed itself. I didn't percussively adjust it at all between when it was broken and when it was fixed, so that does lead one to suspect moisture? Or thermal expansion? No idea.

I am not surprised to be thinking about replacing the keyboard before anything else. And I'm perhaps even looking forward to it because I've often had the experience that the replacement keyboard is better than the one it came with originally. But it is disappointing that it's doing crap like this, when it's not even a year old yet.

On the bright side, a replacement for this keyboard seems to be readily available through multiple channels for $20. So when it does die, I think I should be able to deal with it.

The keyboard did the thing again today. This time, it didn't fix itself a couple hours later, so I'm actually back to my old chromebook. It's really incredible how light a 12" laptop is, but also it's harder to find a good spot on my lap for it. The screen is *much* higher contrast. And the wifi failure is really pronounced!

I find myself cantilevering it past the end of my knees so that it's far enough from my face. It's not so bad because it's so light but I think I might be sold on 14" laptops from here on out...

Ordered a new keyboard. $25. Knock on wood.

Ugh, the replacement keyboard is not usable because the original is plastic-riveted in. It's a dense field of rivets, approximately as many as keys. And it's layered under a couple different metal shields that are also riveted on. This is how my Chromebooks were, but people selling replacement keyboards were kind enough to send the whole plastic shell that went with it. Bummer.

To make matters somewhat stickier, all of the guts are anchored onto the bottom side of the keyboard instead of to the top of the bottom shell. So even if I do get a proper replacement keyboard, I'm gonna have to transplant the whole guts. And if I do want to try my luck at replacing the keyboard and just letting it float without rivets, it'll be that same effort.

Anyways, I cleaned the connector since I could reach it. There was some schmutz in the connector, FWIW. But it didn't fix the problem. The trouble seems to come and go but the way it is right now, if I press T and R at the same time (i.e., like typing "trash" if I don't release the T first), it generates an S event. That's in a gray area between usable and unusable. It was doing that all morning, and doing it after I took it apart, and now it just fixed itself. Intermediate problems. I hate it.

So, now that I know to look for it, people are selling the whole "palmrest top cover keyboard" assembly on ebay for $50. That's worth it if it repairs the laptop.

There's no question that the top cover part of this laptop is toast if the keyboard is toast, so I guess there's no risk to just destroying the rivets and replacing the keyboard poorly. That would at least tell me if the problem is in the keyboard (instead of the logic board, or a design kind of problem that a replacement would have as well), and if it's worthwhile to buy the whole top cover replacement.

I'm really not optimistic about getting a decent result without the rivets. The naked keyboard is extremely flexible, and already this keyboard is kind of on the edge of usability because it has such a light feel. I can only imagine what it would feel like without rivets but it doesn't inspire optimism.

I found a youtube video, someone bothered to do it. With a soldering iron, he undid each rivet one at a time. If you pry the keyboard up while you do it, the sides of the hole tend to straighten the rivet back into a little post. Then he tried putting the new keyboard in, but the ones that snagged, he had to touch up again with the iron. And then he bent each rivet back down with the iron to lock it in place. What a pain.

The thing is, I'm not convinced that cutting the rivets off is any easier than doing it "right" with the iron, so I might as well try that. Something for the weekend, I guess.

The silver lining is that the laptop is basically empty on the inside, just like a Chromebook, so it won't be too hard to do all this work, at least. It's just discouraging that I'm looking at something that might really mean the end of this laptop after less than a year, and I really have enjoyed it. It's a good step up from the failing C201. Never had a result like that before.

Oh well, I can go back to the C201 if I have to endure a long outage.

This summer, when the dewpoint exceeded 75F on June 13th, the keyboard went crazy again. It's been a pretty mild summer, in the two weeks since then I have only used the backup laptop (the Asus C201) probably 6 days. BTW, I think it has really made me appreciate the bigger laptop....mostly just because the part of the screen I'm looking at is just a little higher up? Also, I like the better wifi and usable speakers. Other than that, they're remarkably similar to this day. But the bright blacks on this L406 screen are pretty crappy.

Anyways, I ordered a used keyboard, this time with the case attached to it. And today I finally got around to installing it. Because it was humid yesterday. Anyways, I kept a diary of every screw I took out.

1. open case
   1. remove from bottom, 8x silvery ~8mm screws
   2. unscrew (but do not remove) two screws under hinge-side rubber feet (I felt out the holes and then simply pierced the rubber with the screwdriver, I did this months ago and it has been easy access since, without destroying the feet).
2. remove the battery
   1. pry straight up on battery connector
   2. remove from battery, 2x black ~5mm (apparently M2x4) screws (overlapping motherboard)
   3. unplug and unseat speakers
   4. unseat battery with speakers
   5. unseat speaker wire under battery
   6. remove battery
3. disconnect wifi modem
   1. unplug wifi antenna (slide knife under wire up to conn) (not strictly necessary)
   2. unscrew wifi modem - 1x black 2mm screw
   3. unseat wifi antenna (photo its path, maybe)
4. unplug keyboard and trackpad - lift black bar for ZIF
5. free motherboard
   1. remove screws from both hinges and MB. Note that the inside pair of screws closest to the edge of the case are bigger, like M2.5x5. The other screws are all black M2x4 (the hinge has "M2x4" stamped on it!). When repopulating these screws, make sure only to fill in screws that are marked with a triangle silkscreen. Some of the screw holes in the motherboard are filled from the top.
   2. unseat motherboard by lifting up under the battery connector
6. remove keyboard
   1. open laptop (manually unfold hinge, lift keyboard)
   2. push keyboard straight "up" (relative to its keycaps), and it should separate cleanly from MB & top
7. remove old heat sink compound from CPU and heatsink with rag, and apply new heatsink compound on the CPU
8. swap keyboards
9. reassemble
   1. hold screen upright as if in open position, with hinges open, and set the keyboard in place
   2. fold up the hinges, holding the keyboard in closed position
   3. re-seat motherboard, starting by inserting the I/O edge of it into place
10. do all the steps in reverse

The heatsink compound I bought today at Stansifers seems crappy, like it had separated. It's thin and has a clear watery element. I used it anyways, after throwing away a good amount of the watery element (the foreshots).

The new keyboard & shell looks the same as the old one. But I found a few differences. (not surprising, I was not careful to match model numbers). The new keyboard is French, which has only the effect of making the right control key send keycode 94 instead of 105. It is missing two screwholes on either side of the CPU, which seems like a normal revision. So I have two extra M2x4s in my M2 bin now. And its trackpad is different in every fine detail (but none I can tell by using it).

Anyways, crucially, it works today, it feels fine, and it doesn't have any obvious problems at the moment. It seems to have the same limited rollover as the previous one. rightshift+V+T still doesn't register the T. So I guess the only real improvement I am expecting is perhaps it will not die in humidity. But maybe it will, maybe the keyboard is just designed lame, or maybe there's a problem with the motherboard. Sigh.

Knowing that I had gotten a new keyboard and it had all the exact same frustrations (except possibly without the humiditiy failure mode, it's unclear still) encouraged me to try a different approach to improving it. I took forward my hack to xf86-input-evdev from 2013 (for my Chromebook) and I applied it to xf86-input-libinput. I had to adjust the delay up a bunch, this bounce is more like it's an actual mechanical bounce I guess, it takes up to 30ms or so to repeat.

Anyways, it was easy to dial in the level of rejection I wanted, and it is really making a night and day difference! It is catching one generally about every minute of using the laptop. Even that level of introduced errors is a big deal, and eliminating it is fantastic! I am again an almost flawless typist.

I wonder if the problem is that the keyboard is so crappy or that my tremor is causing me troubles. I guess the difference is only academic and I'm just glad to have a resolution for now.

Wow! Today, if I press t then press r (without first releasing t), then it emits an s as well: trs. This is one of the many patterns it experiences as it succumbs to humidity. It is only slightly more humid than yesterday, NWS says dewpoint 68F but really I think it's just over 70F. It is still (sorts of) usable at the moment but I know from past experience it is an illusion, it will soon be doing intolerable extrsa characters, since it's supposed to be muggy for the next 4 days.

So maybe it's a flaw with the whole class of laptops, or a whole run of keyboards. That seems kind of unlikely because they do have monsoon in Taiwan. It may, however, be a flaw with the mainboard. I guess I could determine that by checking if the keyboard connector presents a scannable matrix or if it has a controller as part of the keyboard.

Not sure I'm excited about repairing the mainboard. And it doesn't look like replacing the mainboard is an option on ebay yet, I'd have to buy a whole laptop.

It seems like, with careful detective work, I could determine which pins on the connector must be interfering, and I could resolder them on the assumption that it's just a near-short caused by faulty soldering. I guess I'm out the use of the laptop whenever it's humid anyways, so there's not a huge cost to leaving it taken-aparst on my workbench while I mess with it. But the fault may be under a BGA somewhere instead of at the connector. Sigh.

It looks a lot like I replaced an 80% functional laptop with an 80% functional laptop.

For a long time, I have noticed that this laptop has spotty wifi. It works pretty well but sometimes it isn't working: it won't associate, or it won't carry traffic, or it will carry traffic but drops 50-90% of packets. I just run my script to suspend the laptop, and wake it up and usually it works again. It's not a big deal, though it can be frustrating when it's dropping packets because maybe I'll live with bad connectivity for a while before I catch on to what is going on and apply the remedy.

I have noticed, sometimes it's more than once a day, and sometimes it's not even once a month. And today, I noticed, it acts up on the same schedule as the keyboard does! All through the winter, I don't know if I had a single incident with wifi going flakey. But this week, I've had it every day. The wifi problem is caused by the humidity too! Or, who knows, maybe it's heat.

I've noticed for a long time that there has been a bright spot towards the bottom of the LCD. It is very obvious if solid white is displayed. I didn't worry about it too much but now it has buddies. Displaying solid white across the whole screen makes it really obvious that there's about half a dozen white spots clustered around each of the hinges.

Also, the hinges themselves are getting just slightly floppy. There's a little hysteresis now before they actually change their position. It's not a big deal but it's perceptible.

I think what's happened is that my abuse of the laptop -- things like carrying it with the screen open -- are too much for a 14" screen with these hinge attachments. I think the subframe connecting the screen to the hinges is flexing like mad. I think the light spreader behind the LCD is getting either cracked or crinkled. Though it's possible it's a repairable defect like the lenses over individual LEDs migrating...but I doubt it.

So obviously, I can ignore it. Or, I can get a new screen. But if I do, I will want to invent a way to spread the load out from the hinges a little bit. Something like an aluminum plate (iow, an expired license plate) bolted to the back of the display.

But also it's just a bad omen about the 14" screen, which unfortunately I have come to like. The 12" laptop is just more amenable to being thrown around, it seems. Whodda thunk.

I wanted to replace the screen, and probably the subframe around it. So I really want to replace the whole top of the laptop. ebay right away found a $75 laptop, advertised as display works and it turns on but has other fault. Ordered it, and the resulting laptop works fine but has absolutely 0 battery life.

Opened it up. Battery is at 0.0V if it is connected to an unplugged laptop, or if it is disconnected. But if the laptop is plugged in, then it raises to 0.2V. But it never goes up from there. I think that is consistent with the laptop trying to trickle charge a deep-discharged battery, but some protection circuit or open fault within the battery has 100% isolated it.

So I bought a replacement battery for $41. And I hooked it up and it charges and discharges and works fine! For $116, I got a whole second laptop! Golly!

Note to self: when installing the new battery, the wires were too long so I had to bend them rather dramatically to get them to fit in place (which was true about the battery I removed as well). And then it was hard to get the connector to snap. But it seems secure now that it snapped. But if the connector pops off because of wire tension, I shouldn't be surprised.

... Reinstalling Linux. My USB stick still has some sort of Devuan EFI image on it, so that's what I'm using. First step is disabling secure boot in the BIOS, then disable boot to windows and boot from "UEFI Flash". The Devuan installer needs to find ath10k firmware, which it doesn't find in the most obvious way. Not sure how I solved this last time, I guess I used microSD instead of USB flash? Asked it to skip that and the wifi still enabled correctly!

Has a problem where it doesn't like deb.devuan.org because the keys are 4.5 hours in the future. Seems to be a timezone thing. Manually set the time to correct UTC using 'date' at the ALT-F2 shell. It successfully installed and did the UEFI dance and boots off its internal storage now. It is missing /sbin/iw, so I put that on the flash stick (had to reformat it, because the boot image had a weird partition table).

Upgrading devuan to 'unstable'. Installing a bunch of packages.

Oh, this is silly. Somehow, the alsamixer 'master mute' state is saved from one boot to the next, and it defaults to muted. So once per install, you have to unmute it. Far out.

vncviewer seems slightly slower. If this was happening on my old l406, I would assume the ath10k (wireless) driver is glitching and I would suspend/resume it to reset the ath10k. Incidentally fixing glitches in the ath10k driver was one of the things I hoped I would get from this process, and it looks like perhaps the opposite.

I do still need to build a custom kernel (I'm using devuan stock 6.1.0, while the old one uses my build of 5.16.13).

FWIW, the keyboard shift-V + shift-T thing where insufficient roll-over hides the second button press is present on this keyboard too. I might find out in the summer if it has the same problem with humidity as the old one.

Anyways, now it's basically identical to its partner. I guess that's kind of anti-climactic. I want to make a brace to strengthen the subframe around the display, and then I guess I can switch to this one for my daily driver.

Started investigating the subframe situation. Inside the upper half, there is a 3mm gap between the outer plastic wall of the laptop and the display. There is a cable about 1mm diameter in each of these channels (antenna on the left, webcam on the right). I am used to finding a flimsey aluminum subframe there, but on this laptop it is just the plastic clips to hold the bezel on and the cable in place. To its credit, it does have remarkably thin bezel on the sides.

At the moment, I am having trouble coming up with something that is significantly rigid that I could attach onto the backside of the display through like strings passed through holes in that channel.

The laptop has developed a series of annoying kinds of crash:

• The wifi driver freezes, requiring a reboot. Maybe just a couple times a year?
• On wake, the keyboard or mouse isn't working (I don't remember), and I have to hard-reset to get it back. More than a couple times a year?
• On wake, something isn't working and I have to ctrl-alt-backspace kill X11 to get it back (hey, that's not so bad).
• On physical motion (like picking it up), it hard freezes instantaneously. Seems to be becoming more frequent.

All said and done, I feel like I reboot it a couple times a month. It has an uptime of 5 days at the moment. The hard freeze on physical motion, I think I also observed with its twin / replacement that I got used on ebay. So I think it's an endemic problem with the model, something that bends that shouldn't, or something in that vein.

So honestly I find this pretty tolerable and I'm not inclined to do anything about it. I also haven't bothered to do anything about the bright spots in the display (from flexing). Really, the only thing that annoys me is the keyboard, which really is pretty alright and not likely to be improved upon by much.

I do use the other one whenever the humidity gets to this one's keyboard, which so far has been viable as a fall-back. That's really the only thing that makes me set it down.

The summer this year seems as humid as ever, but the old laptop has not developed the keyboard fault! An engunkification-based cure for a capacitive problem???

Anyways, my first L406 is suffering other faults. The uneven brightness of the backlight from being crinkled by the hinges sucks. And the hinges themselves are super worn too, they have a huge hysteresis range they let the screen bounce around in. And the battery is only good for about 12 hours. And the backlight maybe is fading???

But I've had it nearly 4 years, and if I get another 4 years out of its clone then that would be breaking records in affordable laptop ownership. I don't think I'll even bother to try to strengthen its hinge attachments to not crinkle this new screen as well.

Unfortunately, I don't think this new laptop will be any better about random crashes...but I have an idea about streaky wifi misbehavior. The one solid difference between the two laptops is that the new one has higher subjective wifi latency?? As if there is more buffering happening somewhere?? I have looked all through the settings for the identical ath10k_pci driver that each laptop has. I've even booted into the same kernel, but it seems the default debian kernel was doing fine.

So I have the idea I will simply get a different M.2 WiFi card. I am getting some sort of Intel AX200 or AX210 A-E keyed one, off of ebay. I'm not sure if it will work. The factory Atheros QCA9377 is not quite the full width of the M.2 slot?? There is plenty of room inside the case but I don't know if it's a restricted M.2 slot. Lottery.

But I'm still trying to think of other causes for this subjective lag, which I haven't even been able to quantify. -- Oh! Hey! I did a simultaneous tcpdump of a VNC session, and binned the number of bytes in each 10th of a second. And the old laptop is mostly 150kB..300kB in each tenth of second, but the new laptop is more like 0kB..150kB. About 10% of the samples appear to be 0 bytes, whereas less than 1% are 0 on the old laptop! So it's quantified. But straightforward file transfers, it's about the same. I'm stumped.

I've had the Intel "AX200NGW" wifi in both laptops for more than 3 weeks now. It is much more likely than the original Atheros to spontaneously disconnect -- 2-5 times a day, maybe more. Maybe a 5 second ordeal to reconnect. Maybe I should set up a script to reconnect automatically. But so far it hasn't once crashed the wifi stack. I never have to suspend or reboot the laptop just to reset it (knock on wood).

I haven't re-run the quantitative test for the VNC performance problem. Subjectively, it's about the same. I think the newer laptop just has a different buffer setting somewhere. I don't seem to care enough to try to find it.

Ambivalence is the real story here. I use one laptop until it's 20% battery, and then I use the other one. I don't need the capacity to make it through a day so having two of them is gratuitous. I don't really care which one I pick up. The new one has less hysteresis in the hinge is the only way I really can tell them apart. The bright spots from the crinkled backlight don't bother me at all. I don't seem to mind the laggy browser in VNC (browsers are always laggy).

And I remember last summer when I used the Asus C201 for a moment...I really noticed it was so much smaller, but after a moment, it's just a question of how do I set it in my lap. It's the same. I have the same software, the same user experience, either way. Really the only serious complaint I have with any of them is wifi performance.

Laptop is a commodity, I guess.

Oh! One thing I've noticed is that I can suspend the new laptop while the music is playing, and then when it resumes, the music is still playing! The old one, the sound driver would get into a bad state (painful noise) and you had to close/open the audio device (restart mpg321) to get it back. That's kind of nice I guess.

Laptop #2 -- which I got with a fully discharged (0.0V) battery that would not charge -- has eaten the new battery I had given it, I guess.

It turned itself off and would not turn on unless connected to the charger, at which point it says it is not charging. Leaving it on the charger overnight evinces no improvement. Opened it up and the battery reads 10.66V when disconnected entirely, 10.49V when connected to the laptop, and 10.52V when the laptop's AC charger is connected. So it seems it's hitting a 3.5V/cell low voltage cut off protection circuit, and refusing to charge beyond that.

I tried trickle charging it from an external supply but absolutely no current flows in that direction. Guessing there was some sort of signal on the white/yellow (presumably data or sense) cables that might enable the trickle charge recovery mode, I put the dead pack in laptop #1. No dice. I'm going to leave it charging for a while and see if it recovers but I'm not optimistic.

NB - It's trivial to insert wire wrap wire (30AWG) into the battery's connector. It catches on something pretty solidly and then you can fold it over the connector and tape it down and it's pretty durable for a moment.

I think what happens is that laptop #2 has a fault of some kind which randomly or prematurely triggers some sort of permanent charge-rejection circuit in the battery. I don't know anything about it but apparently that's a thing now, the BMS in the battery has a signal that tells it to never charge again. So basically it bricks batteries.

I haven't exactly given up on recovering the pack, resetting the BMS somehow. I am, in fact, having a lot of trouble deciding even how I would approach such a question as this. Laptop #1 is fine, laptop #2 is fine. They've each got faults. Laptop #1's battery is down to what I would estimate is less than 70% of its original capacity, which I think means it's about to take a big nose dive. That's about the most obnoxious fault out of the pile, and the fault that makes me like swapping off laptops.

Should I shop for a new laptop? Should I buy another used L406MA for $100? Should I invest a lot of diagnostic effort in the bad one? I can't even be bothered to figure out the unfavorable network setting so it seems like I just don't care.

For now, I'm spending $30 on another battery. I'm gonna put the brand new battery in laptop #1. So maybe that will put me in a situation where I don't even want laptop #2 anymore. And anyways, I'll put the half-dead battery in laptop #2, so it will be available as a spare again until it bricks that battery.

Taking apart the battery pack. Oops - I smell electrolyte. The tape is stronger than the pouch, I guess.

A bunch of chips on the board. The two chips on the left are covered in an epoxy sort of coating that makes them illegible. Rightmost: D6SC3 12A8ZA. Then 2x E1808N. On each battery, inline with its charge lead, is a 92118-VIII 34 HC85AY-1.

D6SC3 is a "lithium battery fuse", self control fuse. 3 means heater resistance code 3 - 8.3ohm heater. 12A is the current rating. 8ZA is the lot number. It connects the two side pins to eachother at 0ohm and the center pin connected through the heater at 8.8ohm. Seems good. I'm not sure if it's possible that this is a resettable fuse, but if it isn't, it hasn't been blown. And if it is, the heater would need to activate to disable it.

E1808 is an unknown.

HC85AY is a 85C thermal cutoff. I suspect it resets at room temp, and anyways it's conducting now...so it's not the problem.

The problem is probably internal to one of the two obscured chips. I scraped the epoxy off of them and the big one seems to be BQ40Z50 TI 8A1 C?97E, and the small one Q5S J2 295.

BQ40Z50 is a BMS! It talks SMBus. It has a coulomb counter!? It expects to be able to permanently disable the pack via a FUSE pin, making me thinking the self-control fuse is non-resetting and unblown. I'm getting optimistic.

Here is one of the BQ40Z50 features: "Reports the appropriate charging [current and voltage] to a smart charger." However, that doesn't explain why it wouldn't trickle charge outside of the laptop. And another feature: "Supports pre-charging/zero-volt charging", which is what you would face in a true deep discharge. It expects to work on a discharged pack.

Skimming the BQ40Z50 reference manual, a few possibilities occur to me. For example, the white or yellow wire might be !PRES, which needs to be pulled low to enable anything, to let it know that a charger is present. I haven't been able to test continuity all the way to the IC because it is under a thin coating of like epoxy. But inspection of the board found the letters C and D at the base of the white and yellow wires. So I think that's SMBUS clock (white) and data (yellow).

In principle, there are a bunch of ways to talk SMBus. Maybe I can even do it from the laptop itself.

The thing is, I just don't know that it's worthwhile. I'm really at a loss for decision-making.

But I noticed that the Asus L210 (a comparable modern 12" laptop) is only $130. So I kind of feel like maybe I'll just put the new battery in laptop #1, the old but working battery in laptop #2, and just eke out another couple years this way, without worrying about the fact that laptop #2 might eat batteries. And next time I experience dissatisfaction, I will just buy a whole new laptop.

The other thing is, since I was down to 0 portable laptops for a moment while I was playing battery switch games, I went ahead and charged the Asus C201. It still holds a charge and boots up right where I left it. But I immediately ran into out of date software (openvpn on my server changed, it's work-around-able but a pain; and I want to build something of my own which unfortunately required g++ which had gotten uninstalled somehow). And I tried to let Debian update it...it was a pain (usrmerge, natch). But then once I succeeded, nvi no longer works, because the internal-to-libc low quality random source that is used by mkstemp() now always returns 0. It's a bug in fallback code -- it tries syscall getrandom (ENOSYS), then clock_gettime64 (ENOSYS), then to clock_gettime, which succeeds (according to strace), but apparenly the idiom to use the 32-bit result has failed.

The underlying problem is that I have an old kernel on the C201 (3.14.0), and glibc maintainers aren't doing a good job at testing the fallback code that far back. The temptation, of course, is to submit a patch to glibc. Which I may yet have to do, since I have such an ancient kernel on my workbench laptop (kernel 4.1.18). Of course, upgrading the kernel on either one of these isn't out of the question.

And after that, a bug which makes Xorg server SIGSEGV. And sshd gets SIGSYS (?) immediately after spitting out a fatal error from being unable to get entropy from getrandom() (without failing over to any older interfaces).

So for the moment I've effectively bricked my C201, which makes me sad. I just can't think of any reason to fix it. It is such a cool little machine though.