Qnap, SonarQube and Elastic Search

Update 2021-10-20: Solution in the end of the post

I use SonarQube (and SonarScanner) to analyze my JavaScript source code in a project that has almost 200k lines of code.

SonarQube is a non-trivial install and it should be an a server so different developers can access it. Thus, SonarQube is an application that it would make sense to put in a packaged container, easy to set up.

I have a QNAP, with Container Station installed. That allows me to run Docker images and Linux (LXC/LXD) image. To me, this sounds like a perfect match: Just install a SonarQube Docker container on the QNAP and be happy!

Well, that was not how they thought it.

Last time I checked the SonarQube docker image did not come with a database. That would have been the entire point! Most work related to setting up SonarQube is related to the database. Docker support data folders, so it would be easy to configure the docker container with a single datafolder for the database and everything. No. You need to docker images.

The next problem is that SonarQube comes bundled with ElasticSearch which has some remarkable system requirements. Your operating system needs to be configured to support

  • 65535 open file descriptors (link)
  • 262144 vm.max_map_count (link)

Now the first problem is that Docker in QNAP does not support this. However it works with LXC.
The second problem is that QNAP is getting rid of LXC in favour of LXD, but you cant have 65535 open file descriptors with LXD (on QNAP – hopefully they fix it). So I am stuck with unsupported LXC.

But the real problem is – who the f**k at Elastic Search thought these were reasonable requirements?

I understand that if you have hundreds of programmers working on tens of millions of code you need a powerful server. And perhaps at some point the values above make sense. But that these are minimum requirements to just START ElasticSearch? How f***ing arrogant can you be to expect people to modify /etc/security and kernel control parameters to run an in-memory database as an unpriviliged user?

The above numbers seem absolutely arbitrary (I see that it is 2^16-1 of course). How can 65535 file descriptors be kind of fine, if 32000 are not? Or 1000? I understand if you need to scale enormously. But before you need to scale to enormous amounts of data, it would be absolutely wasteful, stupid and complicated to open 50k+ files at the same time. And if 32000 file descriptors are not enough for your clients big data, how long are you going to be fine with 65535? For a few more weeks?

This is arrogant, rotten, low-quality engineering (and I will change my mind and apologize if anyone can provide a reasonable answer).

All the data SonarQube of goes to a regular database. ElasticSearch is just some kind of report-processing thing serving the frontend. I did a backup of mine today, a simple pg_dump that produces an INSERT line in a text file for every database entry. Not very optimal. My database was 36Mb. So if Elastic Search would use just 36000 file descriptors, each file correspond to 1k of actual data.

I don’t know if I am the most disappointed with the idiots at ElasticSearch, or the idiots of SonarQube who made their quite ordinary looking GUI dependent on this tyrannosaurus of a dependence.

Hopefully the QNAP people can raise the limits to ridiculous values, so nobody at ElasticSearch needs to write sensible code.

And if anyone knows a hack so you can make ElasticSearch start with lower values (at my own risk), please let me know!

Solution!

QNAP support helped me with the perhaps obvious solution. Log in as admin with ssh to the QNAP and run:

[~] # sysctl -w vm.max_map_count=262144
[~] # lx config set deb11-sonarqube limits.kernel.nofile 65535

The first command I already knew. You have to run it whenever the QNAP has restarted.

The second command is for setting the file limit in the particular container (deb11-sonarqube is the name of my container). I guess you just need to do it once (and then restart the container), and that the setting remains.

Leave a Comment


NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

Time limit is exhausted. Please reload CAPTCHA.

This site uses Akismet to reduce spam. Learn how your comment data is processed.