For running ArangoDB in clusters doing performance tests we wanted to have a non virtualized set of descent hardware with fast ethernet connection, enough RAM (since thats what Arango needs) and multicore CPU. Since you need a bunch of them, cheap ARM devel boards come to mind. The original Raspberry PI (we have those) is out of the game due to V8 is not supporting it anymore. The now available PI 2 doesn’t cut it, since its ethernet NIC is connected via USB (as on the original PI). The Odroid series only have one of both: Fast ethernet or enough RAM. The Cubieboard 4 wasn’t available yet, but its Allwinner A80 SOC seemed a good choice. Then we met the Merii Optimus board, which seems to be almost the same as the PCDuino (now renamed to Arches) with the A80. While we got a bunch of them for a descent price over at Pollin, the upstream support wasn’t that good.
However, with some help of the SunXi-Linux Project we started flashing OS images to replace the preloaded Android image with the Merii Linux image. Since the userland of the Merii image is pretty sparse, we wanted something more useable. There is already a how-to on running Ubuntu which requires running a Windows host. We prefer a Linux host and want to run a Debian. Since the new Pi2 is also able to run regular Debian with ArmV7, we pick the root fs
from sjoerd.
Building the Merii SDK
The works of Merii have collected a little dust on their way from china, so it doesn’t work out of the box on a recent Debian. Our buildhost is an Amd64 Jessie using multiarch to also be able to run i386 binaries from the SDK. Verify that multiarch is working properly:
1 2 3 4 5 6 7 8 9 |
a80_MerriiLinux_optimus_v1$ ldd ./tools/pack/pctools/linux/eDragonEx/dragon linux-gate.so.1 (0xf77d8000) libdl.so.2 => /lib/i386-linux-gnu/i686/cmov/libdl.so.2 (0xf77a5000) libstdc++.so.6 => /usr/lib/i386-linux-gnu/libstdc++.so.6 (0xf76b3000) libm.so.6 => /lib/i386-linux-gnu/i686/cmov/libm.so.6 (0xf766c000) libgcc_s.so.1 => /lib/i386-linux-gnu/libgcc_s.so.1 (0xf764f000) libc.so.6 => /lib/i386-linux-gnu/i686/cmov/libc.so.6 (0xf74a5000) /lib/ld-linux.so.2 (0xf77d9000) |
Since we want to run a more recent userland with systemd
, we need to enable CONFIG_FANOTIFY – else /etc/fstab
entries for the SD-Card will fail to mount during boot. We edit linux-3.4/.config
and change the line:
1 2 |
# CONFIG_FANOTIFY is not set |
To enabled:
1 2 |
CONFIG_FANOTIFY=y |
After some fiddling, bending download URLs to places where the software still can be found, picking up dangling symlinks, and waiting for the Compiler to do its job, we get:
1 2 3 4 5 |
./build.sh -p sun9iw1p1 ./build.sh pack ls -l tools/pack/sun9iw1p1_dragonboard_optimus.img -rwxr-xr-x 1 willi willi 61M May 27 11:47 tools/pack/sun9iw1p1_dragonboard_optimus.img* |
This is an image intended to be flashed to the board using the sunxi Livesuite. As mentioned above, it only brings some busybox userland, which we want to pimp now. Use the output of dmesg to find the SD-Card device. Then flash the Raspberry pi disk image to an SD-Card:
1 2 3 4 5 6 |
bmaptool copy --nobmap jessie-rpi2-20150202.img /dev/sdd bmaptool: info: no bmap given, copy entire image to '/dev/sdd' bmaptool: info: 100% copied bmaptool: info: synchronizing '/dev/sdd' bmaptool: info: copying time: 6m 11.0s, copying speed 8.1 MiB/sec |
and mount its root filesystem:
1 2 3 |
mkdir /pi2root mount /dev/sdd2 /pi2root |
Since we need to copy some things from Merii image, and we can’t mount it directly, we locate the raw root filesystem in the buildroot and mount it using the loopback device:
1 2 3 4 |
modprobe loop # load the loopback module, just in case... mkdir orig_loopimage mount out/sun9iw1p1/linux/common/rootfs.ext4 ./orig_loopimage -t ext4 -o loop=/dev/loop1 |
Now we can access the files from the Merii root, and copy it (i.e. the kernel modules) so we have it available later on:
1 2 3 4 5 6 7 8 9 10 11 |
mkdir /pi2root/hummingroot rm -rf /pi2root/lib/modules/3.18.0-trunk-rpi2 # remove the original kernel modules, space is sparse... cd orig_loopimage tar -cO bin sbin lib opt root test usr var |tar -xC /pi2root/hummingroot mv /pi2root/hummingroot/lib/modules/* /pi2root/lib/modules cd .. umount orig_loopimage df -h Filesystem Size Used Avail Use% Mounted on /dev/sdd2 2.8G 445M 2.2G 18% /pi2root |
Sadly the generated filesystem doesn’t allow to enter the UBOOT prompt. So we need to modify kernel boot parameters in tools/pack/chips/sun9iw1p1/configs/default/env.cfg
before packing the image. I.e. we may want to make systemd
offer a rescue console (which helped me a lot, since fiddling serial console to work wouldn’t have been possible otherwise):
1 2 |
setargs_nand=setenv bootargs console=${console} root=${nand_root} init=${init} vmalloc=384M loglevel=${loglevel} partitions=${partitions} systemd.unit=rescue.target |
Now we see that the disk is 32G, but we only have 512MB on the board for the root image. So now we need to create a smaller image so it fits:
1 2 3 4 5 6 7 8 9 |
mv out/sun9iw1p1/linux/common/rootfs.ext4 out/sun9iw1p1/linux/common/rootfs.ext4.org dd if=/dev/zero of=out/sun9iw1p1/linux/common/rootfs.ext4 bs=1024 count=$((1024*512)) mkfs.ext4 out/sun9iw1p1/linux/common/rootfs.ext4 mkdir new_loopimage mount out/sun9iw1p1/linux/common/rootfs.ext4 ./new_loopimage -t ext4 -o loop=/dev/loop2 WHERE="`pwd`/new_loopimage" cd /pi2root ln -s lib/systemd/systemd init |
As Lars Vogdt describes in his excellent blog post, we fix the serial console:
1 2 |
cp lib/systemd/system/getty\@.service etc/systemd/system/getty.target.wants/getty\@ttyS0.service |
Then edit etc/systemd/system/getty.target.wants/getty@ttyS0.service
and set the correct baud rate:
1 2 3 4 5 6 7 |
... [Service] ExecStart=-/sbin/agetty %I 115200 ... [Install] Alias=getty.target.wants/getty@ttyS0.service |
We configure the ethernet adding / editing etc/network/interfaces.d/eth0
:
1 2 3 4 |
iface eth0 inet static address 192.168.1.100 gateway 192.168.1.1 |
Edit the etc/fstab
and replace the pi specific stuff:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
# <file system> <mount pt> <type> <options> <dump> <pass> /dev/mmcblk0p7 / ext4 relatime,errors=remount-ro,discard 0 1 # we want to chroot to the SD-Card later: /dev/mmcblk1p1 /mnt ext4 relatime,errors=remount-ro 0 1 proc /proc proc defaults 0 0 proc /mnt/proc proc defaults 0 1 devpts /dev/pts devpts defaults,gid=5,mode=620 0 0 devpts /mnt/dev/pts devpts defaults,gid=5,mode=620 0 1 tmpfs /tmp tmpfs defaults 0 0 tmpfs /mnt/tmp tmpfs defaults 0 1 sysfs /sys sysfs defaults 0 0 sysfs /mnt/sys sysfs defaults 0 1 |
Finally, we have to add some fix up to etc/rc.local
:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
cd /lib/modules/3.4.39 /sbin/insmod nand.ko # power up the PHY for the NIC: echo 3000000 > /sys/devices/platform/reg-15-cs-cldo1/max_microvolts echo 3000000 > /sys/devices/platform/reg-15-cs-cldo1/min_microvolts # some display driver stuff (if you need it ;-) /sbin/insmod ./disp.ko /sbin/insmod lcd.ko /sbin/insmod hdcp.ko /sbin/insmod ./bcmdhd.ko /sbin/insmod gt9xx_ts.ko /sbin/insmod sunxi-ir-rx.ko # now load the ethernet module: /sbin/insmod sunxi_gmac.ko /sbin/ifup eth0 |
Now we can copy everything into the root filesystem image:
1 2 3 4 5 6 7 |
tar -cO * |tar -xC ${WHERE} cd ${WHERE}/.. umount new_loopimage build.sh pack ls tools/pack/sun9iw1p1_dragonboard_optimus.img -lh -rwxr-xr-x 1 root root 532M May 28 14:29 tools/pack/sun9iw1p1_dragonboard_optimus.img* |
We have our beautiful Image which we can flash to the board. We first start the livesuite mentioned above, and select the image. Then we need to put the board into FEL-mode using minicom and the serial to USB cable that came with the board. If you’re able to access the UBOOT prompt, the efex command can do this. Otherwise holding down the 2 key during power up will do the job.
Some more links that helped me a lot to figure everything out:
- The very active linux sunxi community (who also helped me on IRC)
- How to Upgrade Allwinner A80 OptimusBoard Firmware (cnx-software)
Conclusion
It will take a while for the verry powerfull A80 SOC to gain ground. Part of this slow uptake is the very sparse communication policy of Allwinner and Merii. However, that seems to change a bit. With the Cubieboard 4 and the LinkSprite Arches (PCDuino) nice boards with a very active community will become available, I guess Olimex will follow soon. Expect them to arrive at exp-tech, Pollin or Watterott. Also devices with better WAF to look nice in your livingroom are about to arive.
In the next blog post we will install ArangoDB on this cluster and show some performance results. Therefore we will use fast micro-SD cards to provide additional disk-space.