Monolinux Jiffy
A Monolinux distro for the Jiffy board!
Monolinux: https://github.com/eerimoq/monolinux
Bootloader: https://github.com/jonasblixt/punchboot
Hardware: https://github.com/jonasblixt/jiffy
Features
- Fast development cycle (a matter of seconds from source code change to entering user space).
- Everything the Linux kernel provides (networking, filesystems, drivers, etc).
- Various libraries provided by Monolinux.
- Efficient and fun unit testing with Nala.
Boot time
The embedded Linux based system enters user space 0.33 seconds after power on. The EXT4 filesystem is ready in 0.37 seconds and networking in 2.2 seconds. Very impressive for an i.MX6UL SoC with a 528 MHz ARMv7-A CPU, 1 GB DDR3 RAM and 4 GB eMMC! A reboot is even faster, only 0.26 seconds from issuing the reboot to entering user space.
Note that all software except the EXT4 filesystem are part of a secure boot chain. The system will boot even faster without secure boot, but unfortunately I've not had the oppertunity to try it.
| Measurement point | Elapsed time | Delta |
|---|---|---|
| Hardware | 1 ms | 1 ms |
| ROM code | 185 ms | 184 ms |
| Bootloader | 271 ms | 86 ms |
| Linux user space | 333 ms | 62 ms |
| EXT4 filesystem | 373 ms | 40 ms |
| TCP/IP networking | 2.2 s | 1.8 s |
More information about the system and its boot sequence:
First of all, the power is turned on. The hardware releases the reset to the i.MX6UL about 1 millisecond later, and soon the ROM code reads the bootloader from eMMC, verifies its integrity and jumps to it. This takes 185 milliseconds, which is far more than expected. It's hard to do anything about it as this is properitary NXP software.
The Punchboot bootloader is very fast out of the box, and to make it even faster the device tree patching was removed (now done compile time) and the eMMC type was changed from DDR52 to HS200. The bootloader reads the Linux kernel (3.5 MB), ramdisk (1.4 MB) and device tree (8 kB) from eMMC and verifies them in only 86 milliseconds. It then start the Linux kernel. No other bootloader was tested, not even U-Boot.
The system's tiny Linux kernel (version 4.14.78) boots in about 62 milliseconds, which is very fast. This is achieved with a minimal kernel configuration, a few patches, a minimal device tree, and uncompressed kernel and ramdisk images.
Here is a brief description of the Linux kernel patches.
- Unpack the ramdisk after drivers are probed.
- Removal of unnecessary delays in the MMC driver (I hope). This is possible since we know excatly which MMC is mounted on the board, and that it is always powered on.
- Start with MMC clock frequency at 52 MHz instaed of 400 kHz. Same reasoning as in the previous bullet, but not according to spec.
- Async MMC and FEC (Ethernet) driver probes to do other initialization in parallel.
- 10 Hz Ethernet PHY polling instead of 1 Hz. Would not be needed if the PHY sends an interrupt when its link is up.
The statically linked init process, part of the ramdisk, contains the entire application. It's implemented in C for low overhead, both in RAM and CPU time. No forks. No scripts. No shared libraries. It does however contain cURL and other libraries, which makes it about 800 kB.
The EXT4 filesystem (which is not integrity checked with dmverity) is mounted within 40 ms after entering user space. The enabler is to start the customized MMC driver early.
Networking takes by far the longest time to get ready. The main reason is that Ethernet auto-negotiation takes a significant amount of time, about 1 to 3 seconds. Users that do not need Ethernet, or can use a static link configuration, can save a bunch of time.
Below is selected messages from the Linux kernel log to better understand what is going on. There are a few user space messages in the log as well.
[ 0.000000] Booting Linux on physical CPU 0x0
[ 0.000000] Linux version 4.14.78 (erik@erik-GR8) (gcc version 9.1.0 (GCC)) #1 Tue Jun 16 19:18:46 UTC 2020
[ 0.022802] Unpacking initramfs...
[ 0.025535] mmc0: SDHCI controller on 2190000.usdhc [2190000.usdhc] using ADMA
[ 0.034262] Freeing initrd memory: 1416K
[ 0.035840] Freeing unused kernel memory: 172K
[ 0.041533] 1970-01-01 00:00:00 INFO default Successfully inserted '/root/fec.ko'.
[ 0.053621] 1970-01-01 00:00:00 INFO default Successfully inserted '/root/ext4.ko'.
[ 0.057696] fec 2188000.ethernet eth0: registered PHC device 0
[ 0.058009] mmcblk0: p1 p2 p3 p4 p5 p6
[ 0.076813] EXT4-fs (mmcblk0p3): mounted filesystem with ordered data mode. Opts: (null)
[ 0.082744] 1970-01-01 00:00:00 INFO default /ext4fs/README: +-----------------+
[ 0.082812] 1970-01-01 00:00:00 INFO default /ext4fs/README: | Monolinux Jiffy |
[ 0.082869] 1970-01-01 00:00:00 INFO default /ext4fs/README: +-----------------+
[ 0.114466] SMSC LAN8710/LAN8720 2188000.ethernet-1:01: attached PHY driver [SMSC LAN8710/LAN8720] (mii_bus:phy_addr=2188000.ethernet-1:01, irq=POLL)
[ 0.114607] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[ 1.892754] fec 2188000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
[ 1.892791] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1.893268] 1970-01-01 00:00:01 INFO dhcp_client Starting on interface 'eth0'.
[ 1.900520] 1970-01-01 00:00:01 INFO dhcp_client Received OFFER packet.
More ideas:
Compile for the Thumb instruction set for significantly smaller binaries. Has been tested, but the init process crashes.
[ 0.560005] Freeing unused kernel memory: 1024K [ 0.565155] Internal error: Oops - BUG: 0 [#1] THUMB2 [ 0.570222] Modules linked in: [ 0.573294] CPU: 0 PID: 1 Comm: init Not tainted 4.14.78 #19 [ 0.578959] Hardware name: Freescale i.MX6 Ultralite (Device Tree) [ 0.585148] task: c7009800 task.stack: c702e000 [ 0.589685] pc : [<c0205ba2>] lr : [<c0205ba2>] psr: 60000013 [ 0.595957] sp : c702ffa8 ip : 50c53c7d fp : 00000000 [ 0.601187] r10: 00000000 r9 : c702e000 r8 : c0205dc4 [ 0.606420] r7 : 000f0005 r6 : 00000000 r5 : 00000000 r4 : 000224d4 [ 0.612956] r3 : c702e000 r2 : 0002254c r1 : c702ffb0 r0 : 00000000 [ 0.619492] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment none [ 0.626633] Control: 50c53c7d Table: 871b0059 DAC: 00000051 [ 0.632387] Process init (pid: 1, stack limit = 0xc702e208) [ 0.637967] Stack: (0xc702ffa8 to 0xc7030000) [ 0.642340] ffa0: 000224d4 00000000 0002254c 00022098 beca7fda 00022094 [ 0.650531] ffc0: 000224d4 00000000 00000000 000f0005 00000000 6474e551 00000000 00000000 [ 0.658722] ffe0: 00000037 beca7df0 00010688 000111bc 20000010 0002254c 00000000 00000000 [ 0.666920] Code: 00004770 00000000 00000000 b6720000 (2008f8d9) [ 0.673020] ---[ end trace e0ea9fb97d0d2058 ]--- [ 0.677644] Kernel panic - not syncing: Fatal exception
The MMC is setup by the bootlaoder. Update the Linux MMC driver to take advantage of this. No additional setup should be needed.
Measurements
Test sequence
Unmount and power off the board.
$ umount ext4fs $ poweroff
Unplug the USB cable.
Start a logic analyzer capture.
Plug in the USB cable.
Wait for the system to start.
Stop the logic analyzer capture.
Measurement points
Hardware: RST_N high
ROM code: GPIO low
Bootloader: GPIO high
Linux: "main" printed
Filesystem: dmesg "mounted..." - dmesg "Freeing..." + "main" printed
Network: dmesg "Received OFFER..." - dmesg "Freeing..." + "main" printed
Build and run
This requires that punchboot is running and ready to execute commands.
$ ./rundocker.sh
$ make -s -j8 upload