From CATS to evbarm-earmv7hf: How we got our fisrt arm port supported on Anita
When I started the GSoC project, we had almost nothing to begin with. Most of the ports either required manual installation (without sysinst) or didn’t even boot. So, we decided to test as many ports as possible and get at least one port per architecture supported in anita. After about a couple of weeks of building releases and testing them, we finally decided on getting CATS as our first supported port in anita. But, there were a few major issues that needed to be fixed before I could start working on the anita library (and the controlling script).
The Problem
sysinst not working: sysinst, the menu-driven NetBSD installation tool, was not working for CATS. Everything went fine until it started decompressing sets (compressed files that contain parts of the NetBSD operating system). It used to crash randomly while doing that and we were clueless as to what was happening. So, we decided to try out manual installation.
Manual installation not working: For manual installation, what we basically did was, made (or downloaded/built) an ISO out of the sets and created a raw disk image for installing the system on. Then, we attached it to the Virtual Machine. Then we used issue a ‘newfs’ command to create partitions on the raw disk image. Sometimes we also needed ‘disklabel’ for labeling the disks. Then we used to mount the ISO and migrate to the folder containing the sets and issue a command to decompress all the sets onto the required disk image followed by making device nodes and editing a few mandatory configuration files. Now, this went as expected until we again hit the point where we decompress the sets. It got stuck again, and it used to get stuck randomly at anywhere while decompressing. We were quite puzzled as to what was actually happening.
The ‘not so jaw-dropping’ Solution
While my mentor, Christos Zoulas, was experimenting with the port, he found out that we didn’t have enough RAM attached to the virtual machine. After adding enough RAM to the Virtual Machine, we were finally able to install the system. Sounds cool? Not yet! We failed to boot it using the GENERIC kernel on subsequent reboots! Turns out the kernel couldn’t find the path to init. init, as you might be knowing is the first process to be started. From here on, the normal boot process into the user-space starts. We were startled by the fact that although, we had init present on the disk, the kernel was still unable to find it. Later on, we found out that many of the files on the disk were filled with trash and no useful information was contained in them. This was a huge disappointment.
Our final attempt at ‘Trying to bell the CATS’
Now that we were unable to figure out what was going wrong, or where the problem was, the Virtual Machine System or NetBSD port, we finally decided to conduct a Git bisect session. We were quite optimistic as our last bisect session for NetBSD/pmax was successful and we were able to fix it. After hours of building and testing, we failed to spot the error, and nothing seemed to be working.
Jared, the saviour
While we were busy trying to get CATS working, another member of The NetBSD Foundation, Jared McNeill, who happens to be an expert of IPTV platforms, was working on NetBSD/evbarm-earmv7hf port, trying to get the VEXPRESS_A15 kernel working. Since I wasn’t involved in this process, I decided to contact Jared himself to enlighten us on this issue. Quoting his own words,
“Ok, so in the start, there was a VEXPRESS_A15 kernel that didn't boot. There were two problems to start: 1. Hang at boot related to IRQ configuration in ethernet driver. The incorrect values were used for IRQ polarity and buffer type. 2. The MMC driver didn't work. The issue was that the kernel will send up to 65536-byte transfer requests, but the controller can only do up to 65535-byte transfers! This meant that small transfers worked (read disklabel etc) but once you tried to boot later it would fail.”
As you can see, although, the problem was not an easy one to solve, nonetheless, he decided to give it a shot.
“So I fixed these along with porting the board's kernel from a static configuration to use a flattened device tree (FDT) based configuration. This makes it easier to add new devices in the future. This was a good test for the FDT code as the vexpress-a15 device tree is quite complex. Lots of fixes as a result.”
Earlier, the attached device information was embedded into the kernel, he decided to have a separate Device Tree Blob passed to the kernel at boot providing the required device information. This helps in adding other devices that may come with future revisions of the hardware.
“Added support for resetting the kernel -- only a few register writes but it took a bit of time to find the right way to do it. It looks like on a real board there are a few options (different watchdog timers etc) but QEMU doesn't implement them. Next up, added SMP support! The firmware starts the CPUs and has them wait for an interrupt. When they are woken by the interrupt, they read a start vector from a register named SYS_FLAGS and jump to it. We write the address of the "cortex_mpstart" function to this register, setup the GIC distributor, issue an SGI (software generated interrupt) to all application processors, then wait for them to start. Finally we disable the GIC distributor, as it will be enabled again later on. This is also part of the platform code.”
For an ARM machine SMP boot works the way Jared has explained. While one core (you may call it primary/boot CPU) starts a reset vector which points to the memory location where the boot monitor program is stored, others (application processors) are in a Wait For Interrupt (WFI) state. When an SGI is passed through the GIC distributor registor, GICD_SGIR, by writing to it, all the other cores wake up.
“Next up, we want to support poweroff to shutdown QEMU properly from the guest. This uses the same register set as "reboot", so pretty easy to add a driver for this. Now on to device drivers. Added framebuffer (sys/arch/arm/fdt/plfb_fdt.c), keyboard/mouse, audio codec, and real-time clock drivers. Finally, we imported Linux .dts (device tree source) files into the tree and use them to generate .dtb (device tree blob) files. The dtb is passed to the kernel to describe the hardware.”
As you can see, finally he filled all the gaps and got it working like magic. I’m thankful to Jared for getting us out of trouble and clearing the path ahead, for us.