Category Archives: Raspberry Pi

Stress testing with Raspberry Pi

Posted by on 2020/02/29 No comments

I have a system – a micro service architecture platform – built on Node.js. It can run on a single computer or distributed. It is a quite small system but quite critical that it works correctly.

Under what circumstances would the system fail to work correctly? How much load can it handle? How does it behave under too heavy load?

Stress testing is difficult, and expensive. Ideally you have plenty of test clients simulating realistic usage. It can be done, but often not easily. A simple and cheap option is to run the system on less resources.

My system used to run perfectly on a Raspberry Pi. The tests work fine. I have also kept the integrationtests working (although there have been timing issues). However, the other day I tried to restore production data to the Raspberry Pi, and it failed to run properly. Problems were

  • High latency and timeouts
  • Heavy swapping
  • Escallating retries, making the situation worse

The last point is particularly interesting. Error handling is designed for stability and recovery, but it risks increasing the total load, making the system even more unstable.

I did make the system work on a RPi again, and in doing so I leant about real problems, and fixed them. It is an interesting excersise in finding problems in systems that don’t work properly, and it is a practical way to “measure first, optimize second”.

Does your system work, with a reasonable amount of production data, on a Raspberry Pi?

How to (not) set up a RPi V3 server

Posted by on 2019/07/02 No comments

A few months ago I set up a server running Archlinux on a RPi V3 using a 2.5′ USB drive for root. It is now dead.

One day I innocently did “pacman -Suy” as usual, and it didn’t restart. After that it was very unstable for days until it – the RPi V3 itself – appears broken. That is, I get very random errors, like different kernel panics, trying to boot it. I let a friend try to boot LibreELEC on it (his power, SD-card, TV) and it displayed the 4-color-splash-screen and the little lightning indicating power problem.

There are different ways to connect a USB HDD to a RPi.

  1. Let the RPi power the USB drive.
    (it works sometimes, you can try max_usb_current=1 in config.txt)
  2. Connect the USB drive to a “usb extra power cable” (google it) to take the load off the RPi
  3. Connect the USB drive to a USB hub that has external power
    (optionally, also power the RPi from the same USB hub)

With my unfortunate broken RPi V3, I used method (1) for a hard drive rated at 1.0A (the USB<->SATA-chip probably consumes some power as well). I did use a proper original RPi power supply though, but I believe I somehow stressed some component of the RPi V3 continously taking more than 1.0A from a single USB port.

Findings using USB hub to power RPi + HDD

I have an old USB Hub with a 2.0A rated power supply. I now use it as the only power source for a RPi + USB HDD (The USB Hub and RPi are connected both ways in a loop).

  • RPi V2 + 1.0A 1TB HDD: frequent under-voltage warnings
  • RPi V1 + 0.6A 320GB HDD: works perfectly

So, that particular USB Hub will drive my RPi V1, and I will find another solution for the RPi V2.

Conclusion

I have written several articles about using RPi as a server.

My sober and responsible conclusion must be: don’t (use Raspberry Pi for production Linux servers). It is simply not worth it. First it is not so cheap as you think when you have bought all cables, adapters, cases and chargers. Second, your time my not be for free. Third, performance is bad. Forth, stability is limited and don’t expect very long service time.

The cheapest NUC setup is a more rational choice.

I also believe a QNAP or Synology with virtualisation technology could be a better choice than running (multiple) RPi.

Nevertheless, I never learn, and I am now replacing my broken RPi V3 with two old Raspberry Pi (V1 + V2). I mostly use them for Syncthing and backup, I guess two is better than one, and I have unused USB Harddrives.

Archlinux vs Raspbian

I have come to like Archlinux for RPi. However, the frequent and relatively large upgrades that come with a rolling distribution feels somewhat unoptimal for a low-performance system living on an SD-card.

This time, I am back with Raspbian, because

  1. Raspbian is now based on Debian 10 (buster) right from start (I believe the Debian 9 release of Raspbian was kind of late)
  2. There is simple minimal Raspbian Buster Lite image suitable for servers or headless systems
  3. Creating an empty file named “ssh” in /boot before starting the first time lets you ssh into the brand new raspbian system, so you can easily install with neither keyboard/display or serial

I simply have nothing to complain about with Raspbian anymore.

ArchLinux on RPi with USD Harddrive

Posted by on 2019/02/03 No comments

Update 20190624: Today I ran “pacman -Suy” and ended up with a system that does not start. Two blinking green leds indicate failure to read the SD card. That is less than 5 months after I used SD-card for only /boot. I really consider not running RPi as headless servers anymore.

I have found that one of the weakest parts of a Raspberry Pi server or workstation is the SD card: it is slow and it will break sooner rather than later. There may be industrial SD cards or better SD cards, but a good old USB hard drive is just better.

With RPi v3 it shall be possible to boot straight off a USB drive! That sounded great so I got a brand new RPi v3 B+, a USB hard drive, and I installed ArchLinux on the hard drive, just as if it was a memory card. Fail. That did not work (with ArchLinux, Raspbian may be another story).

But there are levels of pain:

  1. All SD-card
  2. SD-card, but /home on USB harddrive
  3. USB harddrive, but /boot on SD-card
  4. All USB harddrive

I decided to try #3.

It turns out that when the RPi boots it runs u-boot (its like the BIOS of RPi, and many other embedded devices). At one point u-boot reads boot.scr (from the first VFAT partition of the SD card). It had the lines:

part uuid ${devtype} ${devnum}:2 uuid

setenv bootargs console=ttyS1,115200 console=tty0 root=PARTUUID=${uuid} rw rootwait smsc95xx.macaddr="${usbethaddr}"

I figured that I could do this instead:

# part uuid ${devtype} ${devnum}:2 uuid

setenv bootargs console=ttyS1,115200 console=tty0 root=/dev/sda2 rw rootwait smsc95xx.macaddr="${usbethaddr}"

However, boot.scr has a checksum so you cant just edit it. But it tells you what to do: run ./mkscr. But it is dependent on mkimage, so the procedure is:

  1. Install uboot tools
    1. ARCH: pacman -S uboot-tools
    2. Ubuntu/Debian: apt-get install u-boot-tools
  2. Edit boot.txt (not boot.scr) to your liking
  3. Run: ./mkscr

Now only /boot is on SD-card. That is quite ok with me. There is very little I/O to boot so the SD-card should survive. If I want to I can make a regular simple backup by simple file copy of /boot to a zip-file or something, and just restore that zip-file to any SD-card.

There seems to be no need to edit anything else (like fstab).

Well, the bad thing is it did not work out 100% as I hoped. The good thing is that this should work with any RPi, not just the RPi v3 that supports USB boot.

Syncthing crashes on RPi and Arch Linux

Posted by on 2018/06/28 2 comments

One of my Syncthing servers started crashing (again). It is Rapsberry Pi v2 running Arch Linux. Syncthing was 0.14.44.? I upgraded and got 0.14.48.1. Still not stable.

So I downloaded the Syncthing binary from Syncthing instead of using the one that comes with Arch Linux. That seems to work better.

During trying different things I did a database reset:

$ syncthing -reset-database     (does not start syncthing)
$ syncthing

This is not the first time Syncthing misbehaves on Raspberry Pi and I am beginning to question if it is so smart to store my files on a Raspberry Pi with a USB drive.

Raspbian – kerberos not found

Posted by on 2018/02/28 No comments

I have this very strange error on my RPi V2 with Raspbian (8.0). I suspect I will throw away the memory card and never fix it, but I will document the error for future reference.

My problem was that curl, ssh, sshd suddenly did not work. When I start the web browser I get “I/O error”. This screenshot shows (at least a symptom of) the problem.

I tried to reinstall ssh and curl:

$ apt-get install --reinstall curl

and that did not help.

Apart from this the system works ok. It shutdowns and starts properly. No I/O errors from dmesg. I doubt I will ever figure this one out. It seems to me the system is corrupt at disk level, probably an SD-card problem, and that a new install on a new SD-card is the only way forward.

Syncthing v0.14.40, Raspberry Pi, 100% CPU

Posted by on 2017/11/25 No comments

I think Syncthing is an amazing piece of software, but I ran into problem last week.

I have a library of 10 different folders, 120000 files, 42000 directories and 428GB of data.

I thought that was a little bit too much for my RPi V1 (Syncthing 0.14.40, Arch Linux), because it constantly ran at 100%. I raised Rescan Interval to several hours (so it would finish before staring over).

After startup it took about 10-15 min to get the web GUI up, and about an hour to scan all folders for the first time. Well, that is ok, but after that it still constantly used 100% CPU despite all folders were “up to date”.

It turned out it crashed and started over. I found panic logs in .config/syncthing and error messages in ./config/syncthing/index-v0.14.0.db/LOG.

Some errors indicated Bad Magic Number and Checksum Corruption. The usual reason for this seems to be hardware problem (!?!).

I upgraded my RPi V1 to an RPi V2, with little success. Then I found that I had similar problems on another RPi V2. So after shutting down Syncthing I tried the quite scary:

  $ syncthing -reset-database      ( does not start syncthing )      
  $ syncthing                      ( start syncthing )

After several hours of scanning everything seems to work perfectly!
Let us see how long that lasts.

Review: NUC vs Raspberry Pi

Posted by on 2017/09/07 2 comments

I like small, cheap, quiet computers… perhaps a little too much. For a long time I have used a Raspberry Pi V2 (QuadCore@900MHz and 1GB RAM) as a workstation. To be honest, I have not used it for web browsing, that is just too painful. But I have used it for programming and running multiple Node.js services, and a few other things.

Despite there are so many single board computers it is hard to find really good alternatives to the Raspberry Pi. And when I look into it, I find that Intel NUCs are very good options. So, I just decided to replace my RPi2 workstation with the cheapest NUC that money can currently buy: the NUC6CAY with a Celeron J3455 CPU. It sounds cheap, particularly for something server like. The interesting thing with the J3455 CPU is that it is actually Quad Core, with no hyper threading. To me it sounds amazing!

I also have an older NUC, a 54250WYKH with an i5 CPU.

Raspberry Pi V2:   ARMv7    4 Cores      900MHz                  1GB RAM
NUC                Celeron  4 Cores      1500MHz (2300 burst)    8GB RAM
NUC                i5       2 Cores (HT) 1300MHz (2600 burst)   16GB RAM

I/O is obviously superior for the NUCs (both using SSD) versus the RPI v2 having a rotating disk connected to USB. But for my purposes I think I/O and (amount of) RAM makes little difference. I think it is more about raw CPU power.

Node.js / JavaScript
When it comes to different Node.js applications, it seems the older i5 is about twice as fast as the newer Celeron (for one Core and one thread). I would say this is slightly disappointing (for the Celeron). On the other hand the Celeron is about 10x faster than the RPi V2 when it comes to Node.js code, and that is a very good reason to use a NUC rather than a Raspberry PI.

Update 2018-02-11: after a few months
I came back to my RPi2 from my cheap NUC. The difference is… everything. I really like Raspberry PIs. I have built cases for them, bought cases for them, worked on them, made servers of them. But I really must say that a NUC makes more sense: it contains everything nicely and it is so much more powerful.

You can get a Celeron NUC with 2GB RAM and a 2.5′ disk for quite little money. And from there you can go to Core i7, 32GB RAM and two hard drives: M.2 + 2.5′. And check out the Hades Canyon NUC.

I feel sorry there is basically nothing in the market like a NUC with ARM, AMD, PowerPC or Mips. The only competition is the 4 year old MacMini, which is completely an Intel machine. If you find something cool, NUC-like, not Intel, feel free to post below.

Update 2018-02-28
I ran into a new problem on my RPi. It could be anything. My guess, that I will never be able to prove, is that it is a glitch made possible by using an SD-card as root device (and possibly questionable drivers/hardware for SD on the RPi).

Update 2018-04-09
Premier Farnell has introduced a Desktop Pi. Especially promising is that together with a recent RPi you can get rid of the SD-card entirely, and only use SSD/HDD or even mSATA (over USB i presume).

Update 2020-01-05
I decided to get a QNAP instead for my small server needs.

Update 2020-03-12
Another RPi (V2), this time with a 1TB WD Pi Drive, failed to run syncthing properly. I installed syncthing on a new machine but could not sync everything from this RPi because of repeated crashes (I tried a few fixes that did not work). So my only RPi that remains in regular use is a RPI v1 used with syncthing.

Raspberry PI performance and freezes

Posted by on 2016/12/15 1 comment

On a daily basis I use a Raspberry Pi v2 (4x900MHz) with Raspian as a work station and web server. It is connected to a big display, I edit multiple files and it runs multiple Node.js instances. These Node.js processes serve HTTP and access (both read and write) local files.

I experienced regular freezes. Things that could take 2-3 seconds were listing files in a directory, opening a file, saving a file and so on.

I moved my working directory from my (high performance) SD-card to a regular spinning USB hard drive. That completely solved the problem. I experience zero freezes now, compared to plenty before.

My usual experience with Linux is that the block caching layer is highly effective: things get synced to disk when there is time to do so. I dont know if Linux handles SD-cards fundamentally different from other hard drives (syncing more often) or if the SD card (or the Raspberry Pi SD card hardware) is just slower.

So, for making real use of a Raspberry Pi I would clearly recommend a harddrive.

Raspberry Pi Server

Posted by on 2016/03/20 No comments

The Raspberry Pi has been around for some years now and it has been used in unbelievable projects. As a budget desktop computer it has not quite had the required performance (although v2 and v3 are much improving the situation over v1). However, for simple hobby server tasks the RPi can work very well.

A simple RPi (any version) setup typically requires:

  • RPi
  • SD Card
  • USB PSU + USB cable
  • Network Cable
  • External USB Drive + USB Cable (+power adapter)
  • A case

That is without display, mouse and keyboard, and you dont have a power button. It gets a bit messy.

The market is full of RPi cases that all do the same thing: nothing. They just contain the board. The market is full of mini/micro-towers for MiniITX. There are rather expensive NAS devices that come without hard drives. Why are there no small tower cases that comes with:

  • PSU
  • Slots for 1-2 hard drives (+USB to SATA converters)
  • Cabling that makes everything tidy and neat

Powering the RPi using an external hard drive
I happened to have an external USB drive with an integrated USB hub (an Iomega Minimax that was left alone when its Mac Mini died). With some wood and glue I built a simple stand for the hard drive and the RPi:

DSCN5193

DSCN5194

DSCN5196

As you can see:

  • the hard drive powers the RPi, and I can even use the hard drive power switch
  • the Ethernet and USB ports are conveniently available on the back side
  • the footprint is just slightly larger (just taller) than the hard drive itself
  • the two USB cables between RPi and harddrive are nicely contained
  • heat/ventilation should be pretty good

I have experienced no problems powering the RPi from a USB drive that it itself is connected to. It may not be a supported or recommended configuration, but for practical purposes it works for me.

Performance
I mostly run Syncthing on this RPi. The bottleneck is very much the 700MHz ARMv6 CPU, not the USB2-to-SATA-overhead.

hdparm gives me:

$ sudo /sbin/hdparm -t /dev/sda
/dev/sda:
 Timing buffered disk reads:  82 MB in  3.03 seconds =  27.09 MB/sec

$ sudo /sbin/hdparm -T /dev/sda
/dev/sda:
 Timing cached reads:   496 MB in  2.01 seconds = 247.36 MB/sec

Of course it sucks compared to what you can get in 2016, but it is not remarkably bad in anyway. And it is not so fun to live on an SD card.

The Western Digital Kit
The other day Western Digital announced both a special 314GB hard drive and accessories to make it all nice.

Plusberry Pi
There is also the interesting Plusberry Pi project.

Picocluster
Picocluster is clearly bringing new options! Not so much focus on support for a USB Hard Drive though. Some models have an HDMI port. I am not sure, but I think the idea is that you connect one RPi to the external USB/HDMI ports, and then that RPi can control the other one, if you run Picoclusters custom distribution. Not so bad, but not a KVM either. Perhaps a little modification to make a serial switch so RPi #2-5 can be controlled from #1 over serial?

Best Raspberry Pi Server Linux Distribution

Posted by on 2015/09/20 No comments

Since I got my first Raspberry Pi have have wondered: how to turn it into a proper server. Options that I have not been entirely satisfied with:

  • Arch Linux: probably a great option if you know Arch… I have been too lazy to learn.
  • Gentoo Linux: is Gentoo still relevant? Building everything on the RPi sounds very painful (slow)
  • OpenWrt: nice, but slightly too minimal for a server
  • Raspbian: nice, but a little bit too big standard installation (perhaps it does not really matter, but every apt-get upgrade takes longer time, and so on)
  • NetBSD: such a disappointment 🙁

I now found, and tested, Raspbian Unattended Netinstaller. For me, this is the shit.

If is really this simple:

  1. Format your SD-card with FAT32 (just as usual)
  2. Unpack (unzip) the raspbian-ua-netinst on your SD-card
  3. Connect the SD-card, ethernet and power to your Raspberry Pi
  4. Wait (about 25 minutes, they say, that was ok with me)
  5. SSH into your new lean Raspbian system (root/raspbian).
  6. Read under “first boot” what to do next

Clearly, you need a properly configured network (DHCP, allow fetching of packages, and you need to know what IP address it got).

The entire experience is much enhanced if you connect to your Raspberry Pi with a serial cable during the entire procedure. Jokes aside, I used a serial with my first installation. Second time when I felt confident with the process I did not bother with the serial cable.

First boot quick guide

#dpkg-reconfigure locales
#dpkg-reconfigure tzdata

/boot/config.txt: add the line
gpu_mem=16

Upgrade to jessie
For some reason, Raspbian installation is still based on wheezy, not jessie (you don’t get the latest version of Debian). I suggest, upgrade immediately:

/etc/apt/sources.list (replace wheezy with jessie, two places)

# apt-get update
# apt-get dist-upgrade

It is almost as fast as the installation itself 😉

Conclusion
I think, for the Raspberry Pi V1, Raspbian installed this way is the best server system you can have (perhaps Arch is better if you know it). For a Raspberry Pi V2, perhaps standard Debian is better (I have never used an RPi v2). Everthing I have written applies perfectly to the RPi v2 as well.