TechFindings

I felt a strong desire to compile something, anything, for my WRT54GL running OpenWrt. As is often the case, in the end it is very simple, but finding the simple solution is not very easy. Ironically, the best instructions were not on the OpenWRT site (but I downloaded the Toolchain from openwrt.org).

The program I wanted to compile was a pure C program that I have written myself. Almost clean C89/ANSI code, with a few C99/Posix dependencies. No autoconfigure, no makefile.

Solution/Conclusion
I first downloaded the Toolchain for OpenWRT 12.09, brcm47xx from OpenWRT (despite I run OpenWRT 10.03.1 brcm-2.4). I unpacked it to ~/openwrt/.

Second, two environment variables:

$ PATH=$PWD/OpenWrt-Toolchain-brcm47xx-for-mipsel-gcc-4.6-linaro_uClibc-0.9.33.2/toolchain-mipsel_gcc-4.6-linaro_uClibc-0.9.33.2/bin:$PATH

$ STAGING_DIR=$PWD/OpenWrt-Toolchain-brcm47xx-for-mipsel-gcc-4.6-linaro_uClibc-0.9.33.2/toolchain-mipsel_gcc-4.6-linaro_uClibc-0.9.33.2
$ export STAGING_DIR

Third, compile

$ mipsel-openwrt-linux-uclibc-gcc program.c

Finally, send it to openwrt and test

$ scp a.out root@192.168.0.1:.
$ ssh -l root 192.168.0.1

# ./a.out

That was all I had to do, really. Now some comments on this.

Choosing a compiler/toolchain
OpenWRT gives you three download options, that all seems to be relavant if you want to compile stuff:

• OpenWRT-ImageBuilder…(how I used it for something else)
• OpenWRT-SDK…
• OpenWRT-Toolchain…

The Toolchain is a lot smaller than the others. At least for the brcm platform the SDK is named “for-linux-i486” while the Toolchain is named “for-mipsel”. That confused me because I was not sure the Toolchain was actually a cross compiler.

To confuse things more, as soon as you start reading about how to compile stuff for OpenWRT, everyone talks about the “Buildroot”. No cheating with downloading SDK or Toolchain and get an already compiled compiler! Real men compile their compilers themselves? No disrespect here… the buildroot is fantastic technology, and perhaps I will have my own one day, but right now a C compiler is all I want.

Also, it seems the Toolchain that comes with 10.03.1/brcm-2.4 (my platform) is broken (ok, I am not smart enought to make it work, cc1 complains about unknown parameter). However, in my case the toolchain for 12.09/brcm47xx also worked. I wasted much time with the 10.03.1/brcm-2.4 Toolchain. If my simple steps above dont work quite quickly for you, and you get weird errors from the compiler, download another Toolchain (perhaps even for another platform, and perhaps 12.09/brcm47xx that works for me) just to see if you can get that to work (you may not be able to use the compiled binary, but you can at least confirm that you can generate one).

I suppose it is preferred to use the Toolchain for the OpenWRT platform/version you are actually targeting, but newer toolchains for compatible platforms can also work. Perhaps the newest toolchain is always preferable.

Linking and optimizing
Compiler flags affect the size of your binary, and for OpenWRT you typically want a small binary. I guess the “-Os -s” options to the compiler is the best you can do. The binary itself is static. No dynamic linking. I think that means it only communicates with the rest of the world via Linux system calls, and as long as those have not changed in a non-compatible-way, you can compile your program with a different toolchain than was used to build OpenWRT image and packages (of course, the binary format must be good too).

The C library
What about standard library compliance? My Toolchain came with “uClibc” (although others should be possible). My program uses two things that are not C89/ANSI-C compliant (I know because Visual Studio complained):

1) snprintf(): Worked fine, I believe this is C99 standard.

Update 20140814: The CLOCK_MONOTONIC problem is fixed with BB 14.07
2) clock_gettime(): Compiled without errors or warnings, but did absolutely nothing. The input timespec struct was not modified at all when the function was called. This should be a POSIX function (not Linux specific). I guess it is either not implemented in uClibc, or I should use another clockid_t (I used CLOCK_MONOTONIC), or there is a system call behind it that does not work properly when toolchain is different from the one that build the kernel.

So, generally the compiler and uclibc worked very nicely, but some testing is required.

Build machine
I run the toolchain/cross compiler on a x64 machine running Ubuntu. The toolchain itself seems to be statically linked (ldd tells me), and built for x86 (readelf tells me). So most x86/x64 Linux machines should work just fine, and if you are on BSD you probably know how to run Linux binaries.

Toolchain limitations
For my purposes, the Toolchain was just what I needed. I do not know how to build ipk-package files, and I do not know how to build a complete OpenWRT image. Perhaps the Toolchain is not the right tool for those purposes.

QEMU
If you install QEMU you can test your OpenWRT binary on your x86/x64 Linux machine:

qemu-mipsel -L OpenWrt-Toolchain-brcm47xx-for-mipsel-gcc-4.6-linaro_uClibc-0.9.33.2/toolchain-mipsel_gcc-4.6-linaro_uClibc-0.9.33.2/ a.out

The same way, I guess it should be quite possible to run the Toolchain on a non x86-machine as well. I will write a few lines when I have compiled my OpenWRT/MIPS binaries on my QNAP/ARM running the x86 compiler/toolchain with the QEMU.