Bringing SerenityOS to real hardware, one driver at a time

Bringing SerenityOS to real hardware, one driver at a time

img by Multisn8, derivative of the ladyball logo by frhun. Licensed under CC BY-SA 4.0

Many moons ago, around the time when Andreas formally resigned from being Serenity’s BDFL, I decided that I want to get involved in the project more seriously. Looking at it from a perspective of “what do I not like about this (codebase)”, the first thing that came to mind was that it runs HERE points at QEMU and not THERE points at real hardware. Obvious oversight, let’s fix it.

First thing I looked at was UEFI support, but there’s an ongoing effort by spholz which I didn’t want to hijack. For my purposes, kernel from the master branch successfully boots from GRUB running on a TianoCore UEFI runtime, so this is fine.

Debugging the OS on the same machine as your main isn’t fun, and I wanted to focus my efforts on porting it to something I’d actually want to daily drive – which limited me to recent-ish hardware. I went onto Allegro (polish online auction site) and searched for the cheapest Chromebook I could find. After discarding the literal cheapest option (due to it being a fire hazard), I ended up with a Dell 3100 (baseboard Octopus, model Fleex) for a whopping 95PLN (~25EUR). The specs are as follows:

• Intel Celeron N4020 (2 cores, no HT)
• 4GB of DDR4
• 32GB of eMMC on-board storage
• 1366×768 TN screen driven by the UHD600 IGP
• 2x USB-A, 2x USB-C, 3.5mm jack
• a keyboard that’s somehow better than Dell’s top-end business laptops

(note: starting here, i’ll refer to this laptop simply as octopus, lowercase. it’s a cute hostname)

I had a few reasons for picking a Chromebook. The most notable one is that Cr50, the security chip / embedded controller, gives literal superpowers with regards to closed-case debugging (foreshadowing…). Chromebooks are also plentiful and cheap, so not only is it easy to replace mine if I break it, it’s also an easily obtainable target for others.

cros_ec woes

Haruhi says…

“This post goes off-the-rails quite quickly. Next section isn’t about Serenity directly, rather about prep work and cursed hardware hacks that ensued.

Feel free to skip ahead if this doesn’t interest you…
But we all know this is exactly what you came here for. Strap on.”

For the uninitiated, almost all chromebooks from ~2018 onwards have one USB-C port that, with a SuzyQ cable, can be used for debugging. Cr50 exposes three ttyUSB devices:

• internal Cr50 console
• AP console (aka: chromebook’s serial port)
• cros_ec console (aka: the embedded controller)

The general idea was to use this to get easy, portable access to the serial console without needing to keep the laptop open with wires hanging off it. cros_ec can also do some cool stuff such as emulating keypresses and controlling the power state, so a “KVM” solution wasn’t out of the question.

Unfortunately, reality had different plans:

Picture 1 – my disappointment was immense and my day was ruined

I made a new SuzyQ cable to check if my old one broke. I triple-checked it with another chromebook to verify that it’s really soldered correctly. Then I went to ask others, and I learned that octopus is one of the few laptops where Cr50 doesn’t work for CCD; Dell cheaped out and didn’t include a few resistors on the board. Some have reported limited success, but only under specific constraints (“debug works only in the left port, upside down, and the right port has to have a charger connected”), but I wasn’t able to get this working at all. I shelved the project for the rest of the day out of pure disappointment.

Alternatives

Well, I already have the device, and I kinda started to like it.
Can I do anything to make it useful?

Picture 2 – lots of test pads. sadly, none of them relevant

I spent some time poking around the Nuvoton and H1B2C chips (first one implements the EC, second one the Cr50 debug bridge). As far as I understood the scraps of knowledge around the web, bridging something here would make it talk over USB, but I didn’t figure anything out. At a few points octopus stopped turning on, but thankfully there was no permanent damage – this laptop is a heavy, stable cookie. Of note: my current sources tell me that the missing resistors are only supposed to affect SPI flashing, not the USB bridge itself – therefore it is unknown why I couldn’t (and still can’t) get Cr50 debug to work at all.

Out of options and desperate for answers, I booked a flight to Japan I checked whether a generic Pi Pico board could fit inside the empty space inside the laptop. And, yes, quite easily!

Pictures 3, 4 – this would be a perfect place if not for that daughterboard… CHOMP

I found some schematics for similar laptops online, but none for exactly octopus. My board had two big debug ports – one of them turned out to be some Intel crap, containing an underdocumented JTAG and a bunch of test points irrelevant for my purposes. The other one was Google Servo, which is equally hard to search for due to name reuse – Google churned out a bunch of different debug probes under the name “Servo”, gave them really confusing names and published little to no docs about them, beyond one docs page.

Picture 5 – presumed pinout, found in some leaked schematics

In the end, I took my Glasgow, launched the UART applet with frequency detection, and started probing. The procedure was literally to slowly tap a male end of a dupont-style cable to the suspected UART TX, while Linux was sending a bunch of meows in a loop to /dev/ttyS1.

Picture 6 – the 4W TDP means that not only is this Celeron passively cooled – it barely gets warm without a heatsink. In this picture, octopus is turned on, and it’s still just 37°C CPU die

Finding the proper TX pad took me maybe a few minutes of probing. Doing the same for RX was a bit tougher: you need to be actively transmitting, and if you hit the wrong line, you’re at risk of resetting the board. Exactly that has happened two times before I found the pin – luckily, no permanent harm was done.

Picture 7 – Top terminal is on a Pi Pico, bottom terminal is octopus’ serial port over SSH

Hooray, we have both pins! I looked for two more RX/TX pins for the EC; Surprisingly, through cross-referencing pinouts I found with what I already soldered onto, it took me maybe ten minutes to find those, too. Now, onto soldering:

Picture 8 – after seeing this, WeirdTreeThing was flabbergasted that I was able to solder to a pitch this small

Thankfully, I didn’t lift any pads! I used some epoxy to secure all the wires so they wouldn’t lift; deciding against kapton tape or hot glue was instrumental for longevity of those connections, and I’m happy to report that in the past 6 months I haven’t had a single problem with those hacky connections.

Picture 9 – UV-cure epoxy changed my life. Also visible: Bringus-inspired
method of asserting dominance in the Write-Protect realm: Just Bridge It(tm)

RP2040 can also use SPI peripherals, and comparing to the Servo connector, soldering 6 cables to the flash chip was a walk in the park. I did a small hack with cutting the trace going to the write-protect pin and bridging it to ground. This gave me full write access without needing to ask the Cr50 security chip for permission.

Picture 10 – it’s not the prettiest, but who’s checking

Picture 11 – end result; I cut an additional hole in the side of the case

On the software side of things, I chose CircuitPython (because it exposes USB mass storage for script/data upload). Bridging an UART to the cdc_acm usb device was trivial, but since I hooked up the SPI flash too, it would be neat to be able to flash it, too – and that’s more complicated.

In the OSS realm, EEPROM flashing (especially SPI) is usually done through flashrom. It supports quite a bunch of different flashing methods; The one most relevant to me was serprog, which is a light protocol to proxy SPI over UART. There already was an impl for the Pico written by stacksmashing in C, but using that one would require me to reflash the pico every time I wanted to flash the BIOS…

So instead I wrote an implementation in CircuitPython. I based it heavily on the serprog Glasgow applet, because even though it’s Amaranth, it was the clearest, easiest to understand implementation I found – and parts of it I could copy 1:1.

Pictures 12, 13 – first signs of life, and first successful readSometime after the 1st flash, I randomly noticed how the datasheet said (in very big letters) that my chip was 1.8V. I… somehow never noticed that, and used 3.3V for it. Regardless, it was already soldered, glued, closed in the case… The mistake lives to this day (and hasn’t caused any harm… so far).

Picture 14 – Oops?

All of my code is consolidated into PicoCCD, a quick-and-dirty closed-case-debug solution. The repo is on my Forgejo.

Picture 15 – “i don’t need anything else here, it’s already perfect”

Finally, time for SerenityOS

For debugging purposes, I set up Alpine Linux with a bunch of basic utilities for fetching an externally-built SerenityOS kernel. Over time, this grew into a bigger contraption, which notably includes a GRUB entry that automatically downloads artifacts from my build machine, decompresses and overwrites the Kernel, and unpacks a .tar with the userspace. At the time of writing, iteration time (between making a change and testing a change) is around 20 seconds, which is quite respectable for hacking on baremetal.

Anyhoo, first test run: I prepared a GRUB boot entry (that essentially boiled down to multiboot /Kernel serial_debug), eagerly waited for it to show something, and… nothing. Neither on the screen, nor on the serial port. What gives?

Lack of any output on the screen lead me to some StackExchange post which suggested adding insmod all_video to the boot entry. It didn’t fix anything by itself (some kernel fixes were still necessary), but it was the right call anyways. Moving onto serial, the lack of output was more worrying – especially since I was getting logs from coreboot just a few seconds prior.

A primer on 16550 UART

Picture 16 – “sometimes they don’t come out and it makes me sad” ~ irth
(screencap from a CRD video on dialup VoIP, licensed under CC-BY. greetz to Gravis)

What we call “serial”, in reality can be anything from classic RS-232, through TTL UART (RS-232, but lower voltages), or in rare cases, some wholly unrelated low pin-count bus (USB is serial, technically). It has to transmit data in series (one bit at a time), as opposed to in parallel (at least 2 bits at a time, in most cases 8 or more) – but even that requirement is blurry. Originally, RS-232 also transmitted some metadata (hardware flow control, etc), but today it’s really rare to see UART with more than two wires and ground.

From the perspective of a user, the lowest common denominator is that UART transmits data, and is usually quite slow (although even this isn’t always true, see HSUART). On all but the lowest layer, the bit-by-bit nature is abstracted away, and one just sees bytes passing through – not bits. Thus, even the “serial” part becomes irrelevant.

Similarly, from the programmer’s perspective, serial is just a place to read and write bytes from. Even on the lowest abstraction layer, in virtually all cases you will send bytes. The only defining quirk of the standard is stripped away as soon as possible.

But there’s a missing link: we need something to convert those bytes into a series of bits, and then into actual electrical signals. Ever since the first IBM PC, this was the job of a serial interface chip – historically, either an 8250 or the newer drop-in replacement, 16550. Nowadays, the serial controller is tightly integrated into the chipset, but virtually all of them still use the same register map as a 40+ year old chip would. Thanks to this, as vague as it may be, serial is one of the very few standard interfaces which are present on almost all PCs in some shape or form.

16550: The interface

Historically, all external devices on the IBM PC were port-mapped to the x86 CPU. Thus, to interface with them, one would use a special instruction like outb for writes or inb for reads. This Was Widely Regarded As A Bad Move, because it made things slow as molasses. Then, MMIO (Memory-Mapped IO) came, and with it, hacks like DMA became possible – and really fast in comparison. A lot of devices moved onto MMIO for the faster speeds and lower CPU overhead.

But serial generally didn’t. By the time DMA became viable on consumer-grade machines, serial didn’t need to compete to be high-speed; It got superseded by faster parallel ports, USB, and briefly – SCSI. This put it in a great place for being a generic debug port, because as long as one can physically access the RX/TX pins, executing outb 0x3f8, 0x41 will get yield an A on the other side. The initialization sequence is around 5 instructions, and it’s trivial to implement even in small projects.

… except, as I wrote: “generally”. A lot of sources online insist that 16550-compatibles were always port-mapped, but my laptop here begs to differ:

domi@octopus:~$ doas dmesg | grep 16550
[ 0.501954] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[ 4.278345] dw-apb-uart.5: ttyS0 at MMIO 0x9112f000 (irq = 4, base_baud = 115200) is a 16550A
[ 4.299158] dw-apb-uart.6: ttyS1 at MMIO 0x180000000 (irq = 6, base_baud = 115200) is a 16550A

Figure 1 – Linux kernel logs, showing MMIO and some addresses

Digging further, I found out that the whole thing is provided by a PCI device:

domi@octopus:~$ lspci | grep UART
00:18.0 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor Serial IO UART Host Controller (rev 06)
00:18.2 Signal processing controller: Intel Corporation Celeron/Pentium Silver Processor Serial IO UART Host Controller (rev 06)

Figure 2 – lspci output

This is bad, because PCI is infamously hard to setup. Serenity has a PCI bus implementation, but we’re so early in the boot process that it’s a no-go. Even if it wasn’t, our generic PCISerialDevice so far hasn’t been used in a MMIO context; Hacking up a driver for this without any debug output would be less than ideal.

Horrors beyond our comprehension feat. port80

One of the more peculiar features of the embedded controller on chromeos devices is that it logs all writes to IO port 0x80. Traditionally, this port is used for POST status reporting. If your motherboard has a 7-segment boot code display, it’s decoding port 80! And well, nothing stops you from writing to port 80 after your BIOS finishes the POST.

#!/bin/bash
# silly hack: write to a port from userspace on Linux
if [[ ! $2 ]]; then
echo "usage: $0 "
exit 1
fi
doas dd seek=$((0x$1)) bs=1 count=1 of=/dev/port < <(xxd -p -r <<< "$2")


Figure 3 – save this as outb, chmod +x and run ./outb 80 ff, then look at your POST code display

I tested my hypothesis with the above script under linux, and sure enough, I was able to read the bytes back on the cros_ec console!

Armed with.. all of this, I got to work. Serenity starts execution at Kernel/Arch/init.cpp, so I spread a bunch of IO::out8(0x80, 1); around to track how far it got. I narrowed it down to crashing on Memory::MemoryManager::initialize(0);, and then decided that my manual debugging approach isn’t gonna cut it long-term.

What is port 0x80, if not a “serial” port?

So, one could wonder, what would happen if in the serial write routine the address grew legs and walked back from 0x3f8 to 0x80?

Picture 17 – view from the cros_ec console after enabling port80 intscroll

Answer is – briefly, a lot of bytes. Then something overflows, and cros_ec stops being able to reliably relay data. Even later, something else breaks even further and corrupts the whole cros_ec log output (not visible above). Oops?

We can work around that by inserting a bunch of dummy waitstates between writes – it sucks, it’s slow, but it will have to do. A real serial IO chip would have a big buffer to make sure this doesn’t happen, but we don’t get that luxury here.

void debug_output_cool(char ch)
{
	static bool was_cr = false;
	if (ch == 'n' && !was_cr) {
		IO::out8(serial_com1_io_port, 'r');
		for (int i=0; i<27768000; i++)
			asm volatile("nop");
	}

	IO::out8(serial_com1_io_port, ch);

	for (int i=0; i<27768000; i++)
		asm volatile("nop");

	was_cr = ch == 'r';
}

Figure 4 – artisanally crafted for loops, for a balance of “perf” and “reliability”

We also need some abstraction to automatically parse lines from cros_ec and decode the bytes as ASCII. I’m doing this with a horrifying bash oneliner:

# first, launch picocom
picocom -b 115200 --logfile /tmp/log /dev/ttyACM1

# then, in a second shell..
watch -n 1 "cat /tmp/log | tr '\r' '\n' | grep -a -i port | sed 's/.*: //;s/]//;s/0x07//;s/0x//' | cut -c 1-2 | xxd -p -r | tail -n 50"

Figure 5 – absolutely incomprehensible bash oneliner

Printing all boot messages through this makes the system go from “boots in a few seconds” to “boots in a few minutes”, but this is a price I was willing to pay.

serenity, again

Armed with boot logs, I asked for support from the community.. or that’s what I should have done. Instead, I spent a few days reading through the codebase, frantically trying to make sense of everything at once, only to finally give up and ask for help anyways. If you’re in this situation, just ask. Open-source folks usually don’t bite (as long as you search beforehand, and formulate your questions correctly), and that very much applies to all frequent Serenity developers; they’re a bunch of really nice folks :3

After some back-and-forth, spholz linked me this MR, which at the time was still open. Building from this branch made the generic framebuffer work!

Picture 19 – task failed successfully

The eMMC saga

Previously mentioned StorageManagement crash looked something like this:

136.086 [init_stage2(1:1)]: PCI: Failed to initialize SD Host Controller (PCI [0000:00:1c:00] - PCI::HardwareID [8086:31cc]): Error(errno=74)
[init_stage2(1:1)]: ASSERTION FAILED: !m_controllers.is_empty()
[init_stage2(1:1)]: ./Kernel/Devices/Storage/StorageManagement.cpp:168 in void Kernel::StorageManagement::enumerate_storage_devices()
[init_stage2(1:1)]: KERNEL PANIC! :^(

Figure 6 – SerenityOS crash debug output

As I wrote before, octopus has a 32GB eMMC chip. Serenity already had some SD drivers, so one could assume that making MMC work with those won’t be a lot of work. Unfortunately: reality struck.

What is MMC, anyways?

If you got this far, I suspect you know that a long time ago MMC was a thing. Then SD became a thing too, and MMC died within just a few years. The details on how and why all of this happened are a story in itself, a part of which I researched and wrote down in a previous blogpost.

For the purposes of this post, all one needs to know is that MMC is similar to SD (but with a different init sequence), and that it never truly died: it haunts eMMC chips.

Write the damn driver already!!

In somewhat oversimplified terms, to get an SD / MMC card to work, we need:

1. a Host Controller; they used to come in a large variety of shapes and sizes, but nowadays it’s usually an SDHCI (SD Host Controller), as specified by the SD Association
2. a bus to connect that host controller to; in our case, PCI
3. the proper protocol implemented to talk between the host and the card

As showcased by the crash log, we already had the first two. The protocol is publicly specified, but only for SD cards, not MMC – ever since that became a JEDEC standard in 2007, you need to pay up to get it in a legit way. Having to marry two diverging standards lead me into a rabbithole of comparing several different versions of both to figure out what’s what, how it evolved, and what exactly I need to implement.

As an aside: JEDEC’s standard is considered “open”, but SD Association’s is “proprietary”, just because of the SD security extensions, even though the base version is available for free. I love vendor associations.

SD vs MMC vs eMMC: the differences

Depending on the source, you’ll get vastly different stories on what differs between what. At the time, I cared most about the card initialization sequence; In serenity, we start with sending CMD0 and waiting for a response. This should pass both on SD as well as MMC. Then, we send CMD8 to begin the voltage setup – MMC doesn’t support this, so we should get an error. At that point, some sources suggest trying to reset the card (or the whole controller) and assuming that the card is MMC from here on out. Other sources, such as this wonderful forum thread suggest a much more… comprehensive approach:

According to the SD spec (SD Specifications, Part 1, Physical Layer, Simplified Specification, Version 2.00, July 27, 2006) figure 7-2: SPI Mode Initialization Flow, there are 5 possible outcomes:

1) CMD8 fails and CMD58 fails: must be MMC, thus initialize using CMD1
2) CMD8 fails and CMD58 passes: must be Ver1.x Standard Capacity SD Memory Card
3) CMD8 passes and CMD58 passes (CCS = 0): must be Ver2.00 or later Standard Capacity SD Memory Card
4) CMD8 passes and CMD58 passes (CCS = 1): must be Ver2.00 or later High Capacity SD Memory Card
5) CMD8 passes and CMD58 passes but indicates non compatible voltage range: unusable card

I’m not sure whether knowing the Version of the SD Card makes any difference to the reading/writing/erasing logic.

~ Spruce, 28th September 2006

Anyhoo, I chose to only implement basic checks – and unless we run into some weird compat issues, we probably won’t bother with the more advanced ones. I don’t exactly yearn to code up this vendor-flavored spaghetti.

The rest of the initialization flow also differs from SD, but it boils down to:

1. After the reset, set the clock to 400KHz
2. “wait for 1ms, then wait for 74 more clock cycles” (sic!)
3. Send CMD0, wait for response
4. Send CMD1 in a loop (!) until 31st bit from the response is 1 (JEDEC standard is slightly misleading on this point)
5. Once the loop finishes, save the value. That’s our Operating Conditions register
6. Continue with the SD initialization algo, sans for querying SD-specific registers
7. Optionally, probe for High-Speed compatibility and turn on one of the various HS modes (there are more than 4, along with two different ways of compatibility probing!)

My code was getting to around point 4, but no matter what I did further down the line, my card wouldn’t respond to any requests. I spent a days trying to debug what was going wrong, and I couldn’t find anything. One evening during a routine session of pulling my hair out, I randomly removed all the parts responsible for the controller reset – and while that didn’t fix it entirely, the eMMC started responding! This suggested that the reset function was to blame:

ErrorOr SDHostController::reset_host_controller()
{
m_registers->host_configuration_0 = 0;
m_registers->host_configuration_1 = m_registers->host_configuration_1 | software_reset_for_all;
if (!retry_with_timeout(
[&] {
return (m_registers->host_configuration_1 & software_reset_for_all) == 0;
})) {
return EIO;
}

return {};
}

Figure 7 – fragment of SDHostController.cpp

First weird thing about this function: no standard I found defined a host_configuration register. There was a Host Control register which was roughly in the same spot as our weird 32bit blob, but nothing seemed to fully match. AS IT TURNS OUT: the previous entity touching this code grouped a bunch of registers into two arbitrary host_configuration groups, then never fully finished initializing them – see how we just set the 1st register to 0.

Why is this important? The first group contains a smaller Power Control register, controlling the power regulator for the card. On some hardware implementations (including all that use eMMC) this register is required to power on the card itself, while other designs (that connect the power rail directly to the slot) may ignore it entirely – hence the misconfiguration.

As a temporary hack, I just stole the value initially contained in host_configuration_0 and rolled with that.

tl;dr: talking to the card is significantly easier if we power it on.

After figuring this tiny detail out, I needed a few more hours to debug and disable specific commands in the sequence that were only valid for SD cards. The rest went quite unremarkably – finally getting sensible debug output from the controller helped my nerves a lot, too.

Soon, we (really slowly) spawned a (partially corrupted) graphical session, which (unfortunately) quickly locked up. Debugging all of those issues will be described in a future part of this series (if attention span allows).

I don’t have a good picture of that semi-broken state, so please enjoy one from when I managed to unbreak the framebuffer – something I will (hopefully) describe in detail in the next post.

Picture 20 – (out of the lack of a better picture) finally! it’s alive!

This post itself took me on a learning journey that spanned around 6 months, many of them busy with other things that got in the way (wink wink, Project SERVFAIL). I plan to finish up cleaning up and upstreaming my patches sometime this year.

Huge thanks to mei, Linus, kleines Filmröllchen, Multi and WeirdTreeThing for proofreading this post!!

Equally huge thanks to everyone who survived through me hyperfocusing on this and supported me on the way ^^ you’re awesome

Comments: