Author Archives: zo0ok

How fast is an Efficiency Core?

Last years more and more computer CPUs have two different types of Cores: Performance Cores and Efficiency Cores. As a consumer and an IT professional I am curious about the performance of such Efficiency Cores.

Summary

I will give the answer first. Under good conditions, an Efficiency Core can deliver about 20-25% of the performance of a Performance Core. This is comparable (slightly less) to what can be expected of getting an extra core with Hyper threading.

Good conditions are: 1) that the load/application is multi threaded, and 2) that there is sufficient memory cache to make use of another thread.

The difference with Hyper threading is of course that the Hyper thread is supposed to produce 30% extra performance, on top, when the CPU is already running at maximum (speed and power consumption) while an Efficiency Core is supposed to be able to run the system under low load, so no Performance Core needs to be active at all. However, efficiency Cores can under the right circumstances boost maximum performance.

Test CPUs

I have compared two mobile processors that are quite different. Both are rated at about 35W and both were released end 2021 to beginning 2022.

  • Apple M1 Pro (10 Cores – 8P + 2E)
  • Intel i5-1235U (10 Cores – 2P + 8E, and the 2P cores are HT so the OS sees 12 cores)

There are differences too. The Apple CPU is a premium chip in a computer that cost over EUR 2000 two years ago. The Intel CPU was found in a Chromebook currently costs less than EUR 700. I think it is safe to say that the Apple chip is much more expensive, and it also have much more cache. This is not an article about Apple vs Intel, I just want to explore Efficiency Cores, and for that purpose it was quite useful to have two processors with opposite (2+8 vs 8+2) configuration.

Test Algorithm

To test the CPUs I decided to use the Argon2 password hashing algorithm. The algorithm takes a number of parameters. For my purposes, two parameters were of particular interest:

  1. iteration count – double the count and the execution time doubles
  2. memory usage (kb) – double the memory usage and the execution time doubles (unless you run out of cache, then it gets much slower)

I am using “the reference implementation” that I downloaded a few years ago. It is possible that it is optimized for one architecture (x64) but not the other (ARM64), I do not know. I assume no floating point operations are involved so it is possible that workloads using the FPU would behave differently.

This algorithm is designed to require a significant amount of RAM, to prevent efficient parallelization on GPUs.

Single Core Calibration Benchmarks

The following table shows the iteration count that allows 100 hashing operations, in sequence, in 10s. I added a 4-core i7 with Hyper threading for comparison. I also added a column with Cache size but there may be more details involved not represented in the table. I also added two other benchmark scores, public Geekbench (typical scores, big variety there) and my own real day-to-day task – precommit – that I often wait for.

Cache MB
L2 / L3
128k1024k4096k16384kGeekbench
Single/Multi
precommit
(seconds)
M1 Pro (8P+2E)24+4 / 24210026766152300 / 1200018
i7-8809G (4C / 8HT)1 / 8290037590181500 / 4400 23
i5-1235U (2P+8E)4.5 / 124350560102222100 / 710017

I can draw two conclusions here:

  • M1 Pro seems to underperform at Argon2 compared to what would be expected from Geekbench single core performance, or my own precommit job
  • M1 Pro, with its big cache, is slowing down a bit less with bigger problem size, than the Intel chips.

For each CPU and each memory usage, I have now calibrated the iteration count so the Argon2 program always takes 0.1 seconds to run.

Adding more Cores to Benchmark

Adding more cores is now easy, as I have a command that takes 0.1 seconds to complete. If I want to activate 7 cores, I start 7 processes at the same time. As soon as any one of them dies, I start another one. Thus I always have 7 processes running (except in the end, when they die out one by one).

In an ideal CPU every extra core/process would add the same amount of work done. Lets see what happens.

On the X-axis is number of cores.
On the Y-axis is “throughput” or “performance” (measured in thousands-of-MB-iterations/second).

128k Memory

128k Memory per process is little enough to fit in the cache of all processors (not the L1 cache of course).

As we already knew, i5-1235 was ahead at 1 core and unsurprisingly the second core also performs good. After that the Efficiency cores (together with HT of the two first cores) keeps adding performance at a slower rate. The i7 displays a very linear growth over its first 4 cores, less growth when HT 5-8 kicks in, and nothing after that. M1 Pro is behind from the start and it is not until all its 8 performance cores are engaged that it has caught up with the two Intel CPUs. After 8 cores, the 2 efficiency cores of the M1 Pro give a little extra boost while i5-1235 keeps the going.

M1 Pro: 2 E-cores add about 500 each, after 8 P-cores delivered about 2500 each, so about 20% performance.
i7-8809G: 4HT add about 1500 each, after the 4 primary cores delivered about 3750 each, so about 40% performance.
i5-1235U: 8 E-cores add about as much work as the first 2 P-cores had done. It is hard to know what is HT and what is E-cores here, but about 25% performance.

1024k Memory

Now each Argon2-process uses 1MB of RAM. This should keep most of the work inside the L3 cache of all CPUs.

Again, and with much higher RAM load, the M1 Pro displays impressive linear growth over 8 cores, and very similar characteristics as for 128k. Same goes for the i7-8809G, however HT is less helpful now when the RAM pressure is higher. The i5-1235U again gets a ahead with two blazingly fast cores, after that HT (or E-cores) help a bit for a few more processes, but it is very obvious that all the 8 E-cores do not contribute to maximum performance. Up to 3 processes i5-1235U wins and M1 Pro is last. From 8 processes and up, it is the other way around.

4096k Memory

4MB per process is going to challenge the caches, especially on the Intel CPUS.

Now this looks very different. For 1 process, i5-1235U is still fastest, although the gap has tightened to both the competitors. Apple M1 Pro again manages to sustain linear growth through 8 P-cores, with some extra performance from the 2 E-cores on top. The Intel CPUs, because of smaller cache sizes, peak at 4 processes, and there is no reason to think that the E-cores contribute at all for the i5-1235U.

16384kb Memory

16Mb per process should destroy the cache of all the processors.

Again i5-1235U wins at single core performance, and Apples curve starts looking a bit more shaky. Interestingly for the i5-1235U, now when all cores suffers from too little cache, you can actually see that the E-cores contribute a bit from 4 to 10 processes.

Conclusions

First I think it is important to say that the low cost Chromebook with i5-1235U performs exceptionally well, both in absolute terms, for my specific daily (precommit) task, and for its low price.

The M1 Pro delivers impressively solid scalability up to 8 cores. If you can parallelize your problem in 8 threads/processes, you can count on the M1 Pro to deliver close to 8x performance. I suppose for heavy applications (like video processing) this is a very valuable property both to the developer and the end user. With 2 E-cores and 8 P-cores, the E-cores have little impact on maximum performance, but rather run the computer (laptop) using little power under little load. An E-core seems to have about 20-25% performance compared to a P-core.

The i7-8809G also scales very nicely over its 4 cores. In many cases HT delivers a significant extra performance (up to 40%). However, as an application developer, you may not want to create more than 4 threads if each thread is using much memory. I now understand that an HT-core typically delivers more performance than an E-core, but they serve different purposes.

The i5-1235U displays more… unpredictable… multi-process performance. The 2 main cores are awesome! But as an application developer I would hesitate how much more to parallelize. As a user of a modern operating system, having two super fast cores that get rid of heavy tasks in short time, while having a bunch of E-cores that can handle lesser tasks in the background, is appealing. I can imagine having multiple applications, tabs and services running (all on E-cores) and still getting full performance from my demanding application in focus. That is, however, quite much up to the operating system to enable.

What puzzles me with this i5-1235U CPU is – why SO MANY E-cores? The standard M1 chip has 4P+4E cores. We have noticed above that a E-core delivers about 25% of a P-core. So the M1 chip can run 1-2-3-4 E-cores, and then activate a P-core instead. Why have 8 E-cores? Perhaps there are good reasons, I would probably have made it 2P+4E and a some more cache instead.

Enjoyable Whisky (and a few other spirits)

I have been ranking whisky by testing head-to-head for a while and it is going just fine.

I sometimes have the feeling that powerful and rich whisky benefits in my list, over whisky that is just simple very enjoyable. The reason for this feeling is that there are whiskies with quite good ranking that I have a negative feeling about, and there are whiskies with quite bad ranking that I have a quite good feeling about.

So this is a new ranking list. I try just one whisky at a time while reading, watching TV or something like that, in the evening. Not right after food, and not the same day as other whisky.

The grades for each whisky are:

  1. Want to discard the dram
  2. Not wanting more
  3. Enjoyable and slightly flawed
  4. Flawless but boring
  5. Flawless and a memorable experience

I need to say that in this kind of situation a heavily peated whisky can be a bit too much. I do not think this list will make them justice. But I also do not prefer a heavily peated whisky on any occation.

List

Rated 5

Brora 38 – I cheated and had it after a Gin Tonic, rich balanced aroma, smells old. Soft rich mellow flavour. Nothing unpleasant, complex, some peat to not make it too easy, but very enjoyable.
Glen Ord 18 – fresh, fruity, soft and much to discover.

Rated 4

Bowmore 15 Golden & Elegant – I agree with the official description, peat and citrus (orange to me), not bad at all.
Dalwhinnie 15 – more white wine than malt on the nose, powerful quite complex flavour, classic fresh malty with a bit of burnt finish, not so soft but quite easy to drink.
Deanston 12 – malty, soft, surprisingly salty.
Glenburgie 15 – very easy to drink, a bit on the sweet side with hints of bourbon, not very complex and a weak 4.
Glenfarclas 17 – fresh and salty.
Glenlivet 21 Archive – flawless, but to plain for a 5.
Glenmorangie 18 – balanced, tasty, flawless, very easy to enjoy.
Glentauchers 15 – nothing extra that could justify 5.
Hazelburn 10 – dry, malty, salty, very easy to enjoy and want more of, flawless, the only thing making me not giving a 5 is that it lacks something extra.
Imperial 21 Auld Rare – Very rich, oily, soft, some orange, a very good whisky but I don’t just find the flavour to be a 5.
Johnny Walker Blue Label – it is hard to judge with completely open mind, but this is a rather complex whisky, easy to drink with much flavour. I think all the flavours that come together makes this whisky lack freshness, it becomes a bit overloaded yet underwhelming.
Laphroaig 10 – fresh, salty, peated, and if you tolerate the peat quite easy to sit and enjoy, rather flawless.
Longmorn 16 – More bourbon than sherry, very easy and nice to drink yet rich in flavour, surprisingly dry, perfect but lacking something extra required for a 5.
Miyagikyo Single Malt – balanced fruit, caramel, malt, some peat. Flawless, rather complex, but not easy. It is almost a strong 4 for balance and complexity, but a more weak 4 for enjoyability.
Talisker 10 – peated whisky is not that easy to just sit and enjoy, but this one is flawless, easy to drink, salty, peated, I give it 4.
Yamazaki Distillers Reserve – a good balance between sweet, fruity and malt, very enjoyable and very close to 5 but I lack something extra.

Rated 3

Agitator Argument Kastanj – Quite dark red color, a bit fruity nose and a sherry-like aroma. I think this much more appeals to a sherry lover. It is a bit young and odd, not quite to my tasting, but it is still quite easy to drink and nothing bad about it.
Akashi Blended Whisky – Light classic malty aroma, easy to drink soft flavour, not quite enough flavour to hide the taste of alcohol.
Andalusia Triple Distilled – More bourbon than malt, but a very nice not real bourbon, surprisingly close to a 4, an easy to drink (with some water) unusual whisky with a bit too much sweet fruity and perfume flavour to completely convince me.
Ben Bracken Speyside 5 – Quite soft, malty, a bit fruity, easy to drink. A very well crafted young budget whisky, but not really a 4.
Blantons Gold – a bit try and raw for a 4.
Bowmore 15 – complexity and nice fruits and peat, I would not mind more, but some unpleasant flavour prevent a 4.
Bowmore 18 – quite soft for a peated whisky, yet quite rich. Nothing wrong with it, but I dont feel the enthusiasm drinking it that would give it a 4.
Bushmills Black Bush – sweet caramel, vanilla and a bit glue, really nothing remarkable but it is easy to drink.
Bushmills 21 – quite much bourbon character, remembered it as softer and more fruity. Somewhat bitter finish makes it not a 4.
Chivas Regal 12 – soft, easy to drink, a bit malty and a bit chemical.
Deanston Kentucky Cask Matured – a bit too thin and alcohol smelling/tasting for a 4.
Dufftown 12 Luscious Nectar – soft, a bit fruity, a bit sweet, with some maltiness and slight bitterness in background. Very stable, but not a 4.
Dufftown 18 – balanced, quite rich, nice easy to drink, close to 4 but lacking something extra.
Glenfarclas 12 – dry, nice, easy to drink but with some odd flavours. Very close to 4.
Glenmorangie 10 – at first very nice but after a while quite thin alcohol tasting.
Glen Scotia Campbeltown Harbour – quite light, with a smell and taste of alcohol, but it is also quite soft, hint of peat and easy to enjoy despite being too charming.
Hibiki Harmony – mixed feelings about it, rather perfect but not so enjoyable.
Highland Park 18 Viking Pride – Impressive mix of peat, malt, dark fruits and citrus. It is rich and with a balance between soft and a kick. Somewhat bitter, and hints of sulphur.
J&B – Light and pure, a bit caramel and peat, works (very) surprisingly good for casual drinking.
Johnny Walker Black Label – very smooth, but a bit peated and not without character. Easy to drink but a bit boring.
Johnny Walker Gold Label – a bit oily and dirty, typical scotch, better than I remember it (and better than I have rated it), not bad enough for a 2.
Johnny Walker Red Label – a cheap overachiever. Surprisingly soft, nice hints of peat, easy to drink a bit thin and chemical but casually and standalone it is good.
Johnny Walker 18 – not bad, but not very good taste either, and something metallic about it.
Jura Superstition – not a strong 3 but I can enjoy it, this one performs much better when not compared to other whisky head to head. A bit unpolished, not smooth and uncharming, but it works.
Lagavulin 16 – fantastic in its genre, more ashy than I expected, I might prefer another whisky for another glas.
Ledaig 10 – suprisingly dry and salty at first, which is good, even a bit ashy. I enjoy drinking it, but I dont think it is a 4.
Loch Lomond Heavily Peated – sweet peated aroma, quite soft and rich, not comparing to anything else and considering it is heavily peated, I think it was nice and easy to drink.
Longrow – nice peated whisky, not quite perfected.
Macallan Sherry Oak Cask – the good thing is that the sherry is quite balanced, but the bad thing is that some flavours feel a bit off, but I could definitely have more of this despite I usually don’t like it that much.
Mackmyra Vit Ek – a bit peated on the nose but not really in the mouth, a bit sour, a hint of raw wood, overall balanced and quite nice although not very complex.
Mortlach 12 – Fruity and malty, a bit dry and spicy, but something slightly bitter metallic about it and I can not justify a 4 here.
Nikka Coffey Malt – mild aroma that reminds me of synthetic glue. Flavour is somewhat malty and salty, but that synthetic thing is still there. A decent balance between being soft and having a kick.
Old Pulteney 12 – not bad but nothing extra.
Pampero Anniversario (Rum) – Soft, sweet, easy to enjoy, a good experience but not enough for a 4.
Redbreast 15 – Both bourbon and some sherry/fruitiness, a bit heavy and raw which makes it not so easy to just casually enjoy, but it is good to drink.
Sprinkbank 12 Cask Strength – Mixed feelings here, the nose is complex with peat, a bit sour and a bit of sulphur. Also the flavour is complex and distinct, I prefer it with not so little water. It is not so easy to drink – although good – but a genuine and original whisky.
Springbank 15 Rum – It is an odd sweet-salt-sour combination, hint of peat and very little maltiness. There is nothing wrong with it, but not so charming or tasty either.
Svensk Whisky för Ukraina – Slightly peated, salty, unexpectedly tasty and I pour up some more thinking perhaps it is a 4, but a bit too odd with harsh components.
Taketsuru Pure Malt – really good, dry, salty with a hint of peat, I tasted this in a restaurant so the overall experience is not… compliant… but I am leaning towards a 4.
Tamnavulin Double Oak – a bit malty with a nice fruityness and sherry notes, although a bit thin.
Writers Tears Copper Pot – quite easy to drink but with some roughness, enjoyable, not remarkable.
Writers Tears Double Oak – to my surprise I find this one too unbalanced and not tasty enough for a 4.
Writers Tears Japanese Cask Finish – a good whisky, a bit harsh and odd to me, not a 4.

Rated 2

anCnoc Sherry Cask Finish Peated Edition – I dont much like the combination of peat and sherry, this was not so bad, but I really do not think the flavour is good and I do not want more.
Bergslagen Two Hearts – powerful and complex, almost overloaded, it has a somewhat odd sweetness, and more is not always better, and after half the glass I find it bitter.
Bushmills Original – sweet, a bit chemical, some decent complexity for a quite cheap blend, reasonably easy to drink.
Bushmills 12 – surprised, I find this to be quite bourbon-raw, and although it has complexity I rather drink something else, a strong 2.
Bushmills 16 – even more surprised, I find also this to be quite bourbon-raw, a bit bitter and sour, a strong 2.
Chivas Regal 18 – somewhat raw and not very pleasant, do not want another glas.
Crown Royal Rye – the more I drink the more chemical and unpleasant it is, weak 2.
Glen Garioch Founders Reserve – an unpleasant smell giving a somewhat sharp, raw chemical flavour makes me not want another glas, so it can not be a 3.
High Coast Berg – bitter, sweet, dry, not bad but I don’t find it very tasty, enjoyable or easy to drink. Strong two, but I rather want something else.
Hudson Manhattan Rye – finished it, did not enjoy it, bitter, sharp and nothing of the soft fresh sweetness of a bourbon.
Jim Beam Black – nice bourbon aroma, not a bad bourbon flavour, but raw and rough and not like I want more.
Johnny Walker White Walker – Sweet glue aroma, flavour is complex in a blend way, a bit malt and even hints of peat, but rather chemical alcohol flavour. Easy to drink but not particularly nice.
Lindores – at first I was very impressed with the rich, malty flavour (reminding me of Deanston 12). But there is something bitter and metallic about it, and less of a problem it is a bit perfumed.
Teeling Rum Casks – smells a bit like super glue, a bit caramel but not very sweet, and not much more. Tastes like a blend without character. Easy to drink, especially with a splash of water, but not interesting or tasty.
The Irishman – artificial and sweet, it is easy to drink, but that is it.
Tullamore Dew – some maltiness, but mostly perfume, chemicals and alcohol, and the more I drink the less I like it.

Rated 1

Floki Young Malt – Finally time for Floki. Is it as bad at some people think? The smell is not nice, it tastes better. But I poured up a big glas, added som water hoping it gets better, which it didn’t, and I will throw most of it away. So it is a 1.
Glenmorangie Sauternes – First a margarine or sulphur flavour which kind of disappears with water, but then it gets bitter-sour something. I don’t like anything about it.
Jura Seven Wood – I tasted this head to head, 3 cl for 4 tastings, and I ended up throwing away the last cl.

Code it yourself!

I have worked for 20 years creating business value and utility using software. A common discussion is: should we build this piece of software ourselves, or should we search for something that already solves our problem and download or buy that?

I will share my experiences. It is just anecdotes of course. But it is also about learning from mistakes I have seen and experienced.

BizTalk 2002

My first job was about implementing BizTalk 2002 as a message broker between several ERP systems. In hindsight, the actual requirements were:

  • Moving XML files from one network drive to another
  • Simple XML file mapping / transformation (in the actual case, this could probably have been avoided altogether with a proper architecture in the first place)
  • Error handling bad messages – what was wrong with a particular message
  • Different queues, so that many messages that are not time critical do not delay an urgent message
  • Being able to flush queues in case large amounts of messages are received in error (this happened several times)

I think a competent programmer could have built this (in 2002) using Python, IIS (for GUI) and the filesystem on Windows in a few weeks.

We were using BizTalk 2002 and MSMQ (Microsoft Message Queueing) instead of file shares and we ran into problems like:

  • GUI based BizTalk configuration stored in SQL database made automated configuration and deployment a nightmare.
  • MSMQ was limited to (somewhat less than) 2GB of total message storage, viewing and deleting messages in MSMQ was quite combersome when there were thousands or tens of thousands of messages
  • BizTalk throughput was quite horrible, since messages passing through the platform was read and written to multiple SQL tables along the way
  • Errors were logged in Windows Event Log (all in the same log), with a GUID identifying to the actual message. All failed messages (regardless of type or integration) were in a single view, identified by its guid, and only cross searching the event log would give details about a particular message.
  • All messages were being processed in a single queue, with no way to give anything priority.
  • Deleting messages that had been received in errors took literally several hours of manual work in the GUI

At the time, the license cost alone for a cluster with two BizTalk 2002-servers and two SQL Servers amounted to approximately 12-24 months of salary for a programmer.

The SQL Server Cluster is its own story. We had dozens or hundreds of SQL Servers in our organisation, but this was the only application considered critical enough for a cluster. So nobody knew how the cluster really worked and we had so many problems with it. After a few years we just turned off one of the nodes and the other node was doing fine until the plug was eventually plugged for the entire system.

BizTalk 2002 probably lived in production for 10-12 years, half of that time neither BizTalk, Windows 2000 or Windows 2000 SQL Server were supported by Microsoft.

Upgrading to BizTalk 2004 was impossible, as it was a completely different product, and it was in many ways even worse then BizTalk 2002. We did evaluate it thoroughly.

webMethods

After the disaster with BizTalk 2002 we decided to get a new message broker. By now I had learnt the real requirements and I could have built what we needed in Python on IIS (or any reasonable language and any reasonable web server). However relying on own code was to scary for management so after a long evaluation we purchased webMethods (for a license cost of more than a yearly salary for a senior programmer). webMethods was not particularly suitable out of the box, but it was a decent Java-based development and server environment, with good support for working with XML. So I built the integration platform that I knew we needed using webMethods instead of Python. This was a great success for many years. webMethods was eventually purchased by Software AG, the direction of the product changed but the core features that we had limited ourselves to using were still in place (had we many of the advanced features of webMethods our architecture and investment would have broken down much quicker). Finally, a conflict over licensing, not technical issues, made my employer invest many man hours in a 1-1-migration from webMethods to another (proprietary) platform, but by that time I was busy doing other things. If we on the other had had built the message broker we needed in Python, it could have still been running 20 years later with little need for maintenance.

SQL Server Integration Services

Data was to be exported from an Oracle database to text files, and then loaded into a MS SQL Server database. We talk about 1 GB of data every day, completely replacing the old data in place. However, the two databases had different purposes and different structures, so it was not entirely trivial.

Of course, some genious project manager decided that some genious SQL Server consultants were going to load data into SQL Server using SSIS (a new version of it, that none of the consultants had any real proven experience with). After spending about a coffee break on modelling the SQL Database and months of work with SSIS, the SSIS package was a catastrophy. No data validation, no error handling, unacceptable performance. The data exported from Oracle was not 100% good, that was obvious, but the SSIS package could not output any sensible errors that could be passed back to the Oracle people (remember, same story as with BizTalk, all happy path and no reasonable error handling is what you get when you pay Microsoft large amounts of money for advanced software).

At this point I said: I will help by writing a Python script that will validate the Oracle data. How to validate that the data can be properly loaded? I wrote a script that loaded the data into the a SQL database. It took two weeks to program in Python, and we had fully working error handling, validation and good performance. Two consultants and many months of SSIS-work left the project.

SQL Server

The same data from above was now stored in a SQL Server that was the backend for a customer facing statistics application. The data was structured in hierarchies (locations, customers, articles) and writing SQL queries for arbitrarily deep recursive hierarchies is not that nice. It did not perform well either. In 2010 a typical report could take 10 seconds to generate on a very decent Windows server. This can be done properly with star schemas and denormalized data, but that was not how the consultants thought when they struggled to get any data imported with SSIS.

I was recovering from illness and was free to explore some concepts freely. The first Raspberry Pi was popular at the time (700MHz single core ARMv6, 512MB RAM). I took the data from the SQL Database above and tried to squeeze 2 GB of SQL exported data into a file that would fit in the RAM of the Raspberry Pi. It was easy. I basically just made an array of C-structs that contained denormalised records. All strings I put in a separate memory space, removing duplicates, and referenced them from the C-structs. It all fit in a 400 Mb file on the Raspberry Pi SD-card, which gave med 100Mb for Linux, the web server and the web application.

I wrote a cgi-binary in C, and that binary mmap-ed the 400 Mb file, did a “full table scan” and returned a few basic reports. That took less than a second, much faster than the Windows Server with SQL Server on vastly better hardware. This was just a proof-of-concept, but it was surprisingly simple and straight forward to write it in C, and traversing recursive hierarchies is a pleasure in C compared to SQL.

I am quite sure a properly designed SQL database would have performed just as well. But the thing is that SQL is used because it is supposed to help, when in fact it is not particularly suitable for full-table-scan style reports and hierarchical data. Even though competent people can make fantastic things with SQL it is not easy, and in many case not easier than just writing real code instead.

Optimization

In the company we had an old resource optimisation module (travelling salesman type problem), implemented directly in an ERP system. That system was to be replaced and it was obvious that the optimisation code could not be migrated in any way. The original developer did not want to make a new implementation in a new language. Many options were evaluated and no good solutions were found.

Finally I said: give me two weeks and I will write a proof-of-concept optimiser, and if you are happy with it we can add the extra features and make it production-worthy.

Inspired by my C-CGI project above, I wrote this thing in C and it was faster and better than the old module, which had taken several minutes, even occasionally blocking parts of that ERP system. My CGI-program typically ran in less than a second. It was so fast that most other operations, that were internal to the new ERP system, were slower than calling my external C-CGI code that was an NP-hard-problem-solver.

Obviously people were sceptic to my choice of C, even more sceptic when they learnt that I used C89 ANSI C with zero dependencies (for CGI, HTTP, SOAP, etc). But I had little use for that.

Now, 10 years later, a new project was again replacing the ERP system. This time everybody was very happy to see that it took very little effort to put the C-CGI into a docker container and running it in the cloud. It was also no problem adding a few features and improvements to the 10 year old code. Imagine if this had been written with some 2013 .NET web technology and a lot of dependencies, it would have been much harder to move on.

My role in this

I understand if the reader at this point wonders why I let the projects fail miserably all the time when there was an easy solution. The truth is that as a junior developer it is hard to see problems in the big picture, even harder to change peoples mind. Even as a senior developer I had my role and responsibilities (in another projects or teams), and often the only thing I could do was share my mind and then watch the train crash. I also suffered from burnout from trying to take responsibility for things I was not responsible for.

Management often wants big vendors that they can negotiate with and complain with, rather than putting their trust in their own developers.

I now had enough experience, and there were enough failed projects, for my words to weight a bit heavier with management.

Learning from this

I was privileged in the sense that I stayed for many years in the organisation and I saw what became of the systems and code after the projects were long closed. Many developers are sent into projects, they are given limited background information, they have limited and often exaggerated experience of the project tools, and then they leave the project about the time the system goes live.

What happens a month, a year, or five years after go live, they never know. The entire industry of software consultants is missing learning from long term feedback. Instead it is obsessed with always new knowledge of new unproven versions of relatively short lived tools.

I stayed in the organisation because I wanted to learn.

I realised that I need to work with people who share my passion for delivering long term stable solutions, not jump to new projects and technology every 6-12 months.

A new way of doing things

I formulated some principles for new projects and a new platform:

  • Everything should be made as simple as possible, but not simpler (Einstein)
  • Text files over binary files, for data and programs, whenever possible
  • Use local files over any other forms of storage, whenver possible
  • Minimize work related to dependency lifecycles (upgrades, retirement of components)
  • Often it is better to rewrite something from scratch, so it is kind of more important to build replaceable small pieces of software that does one thing and does it well (unix principle), than implementing things perfectly or even documenting the internals
  • Performance matters: slow code itself cause instability, bugs, extra work and problems
  • As a learning developer, real learning is about methods, algorithms, concepts, analysis of requirements, design, testing – ideas that are valid decade after decade. Learning tools that have short life cycle is a waste of time and effort
  • Most time should be spent learning about the (business/customer) problem (domain) and writing value-adding code.
  • Working software is often replaced not because of new requirements but because the underlying technology is obsolete or not supported. This is waste. Often, progress is not possible because we are only dealing with yesterdays problems. Thus, software should be based on standards and stable products. As a developer I am intrigued by solving new problems with new code, not by replacing old working systems creating no real business value.
  • Linux is superior to Windows. People who only work with Windows or appreciate Windows, have slowly over the years been trained to bad taste when it comes to software design choices.

Apart from this defensive programming (focus on error handling) and the Agile Manifesto are good principles.

On Complexity

Another skill that is critical, when seeing any software problem, is to being able to ask yourselves how long time it would take to implement a solution in a suitable general purpose language of choice (C, Python, JavaScript) and/or using the standard library. If you know your problem can be solved in 2 days using C, with 15 lines of JS-code and JS standard library, or in two weeks using Python you may not need a dependency (that will cost you time to learn and configure, require that you keep it updated, may be upgraded or abandoned, will add complexity and possibly bugs to your code). More often than not, the solution is already available as a standard command in your linux shell, or in the or the standard library. If you just look for it and know what you have.

I have found that many productivity tools (like XML mappers) simplify the easy cases and complicate the difficult cases. If you have a lot of simple cases and a lot of stupid programmers, perhaps that makes sense. But if your work is to solve problems, you are not helped by tools that assume your problem is easy. When you have a difficult problem you don’t want to solve the problem while also working around the tool. This is somewhat similar to the happy-path and error handling. Many tools and technologies (like Promises) seem to simplify the happy-path, but when error handling is priority (which it always should be) the tools often dont help much.

Typescript is supposed to be superior to JavaScript because it checks parameter types. I learnt programming ADA so I am not impressed. If you are serious about type checking or validation, you need to check that numbers are within valid ranges, strings represent valid things, arguments are coherent with each other, objects make sense in the real world. Typescript helps some stupid simple errors, but not to define and validate real business objects in a stable way.

How is it going?

Since almost 10 years I am running a software platform with multiple applications based on Node.js and Vue.js and the principles mentioned above. Stability and performance is good, new code is deployed to production almost daily and test coverage is decent. Even better, the business has confidence in what I do so I have more or less complete freedom as developer/architect, and I spend no time on project plans, budgets, time reporting, or bureaucratic processes.

I will describe a few relevant choices.

Angular.JS

In 2014 Angular.js (Angular v1) seemed to be a good choice and it was quite trendy. Since then the people behind Angular.js have abandoned it, Vue and React have emerged, and current versions of Angular are mostly not compatible at all with Angular.js.

I adopted Angular.js completely. The good thing is that only a small subset of all features of Angular.js were used and no Angular.js extensions/plugins (such as a router) were used, so it has been quite easy to simplify the old Angular controllers and migrate them to Vue2 and later Vue3. This migration is almost complete by now.

According to my own principles I should not have used Angular.js but relied on “Vanilla” JS and direct DOM manipulation instead. I do not know, in hindsight, if that would have been better. But I didn’t have and still do not have the knowledge about direct DOM manipulation to build big complex SPAs without using Angular or Vue.

But it is interesting to note that Angular.js was the only major dependency the platform relied on, it aged badly, and replacing it has been rather costly.

I can note that Web Components (the standard) was not a realistic option in 2015, and it was not a realistic option years later when I evaluated it either. So possibly Vue3 is a good choice today.

Argon2

Password “encryption” is done using Argon2. I wrote a separate post about it. I am obviously not implementing the encryption code myself, I would never do anything like that.

HTML Canvas / Graphs / Maps

There have been some need to generate graphs or maps. This has been done with standard HTML/JavaScript Canvas. None of the graphics generated are particularly standard-looking and a simple graph/map library would not have met the requirements.

Other dependencies

There are other dependencies for things that make no sense to implement directly in JavaScript.

  • wkhtmltopdf – used to create PDF, comes with Debian
  • libxlswriter – used to create Excel files (via a small C-program)
  • trumbowyg – used for end user editing HTML capability
  • nginx – https reverse proxy
  • NPM packages (dev only)
    • mocha – a test runner – quite unnecessary dependency, especially now when Node.js has its own test capability
    • eslint – to validate JS code
    • htmllint – to validate HTML
    • c8 – to get test coverage and other statistics

Conclusions

Working in software projects, loaded with bureaucracy, unrealistic expectations and new and unproven tools that nobody masters is like driving a convertible on the highway with the roof off. It is tiring, you can not go on for long.

Working in a maintenance situation where all you do is replacing working software because the dependencies are no longer supported is the same thing.

However, when you stop relying on tools and dependencies that run out of fashion and support, when you focus on understanding and solving the real problem – not stupid invented technical problems – you start to deliver real value. That way, you build trust and can get rid of the bureaucratic overhead, it is like getting the roof back on, just cruising comfortably, making fast progress.

Whisky tasting notes 2024

Macallan Sherry Oak Cask vs Macallan Quest: Similar color and aroma. There is something un-fresh about Sherry Oak Cask that is “missing” in Quest. Surprisingly much bourbon flavour in Sherry Oak Cast, not bad flavour. Quest has a more flat, bitter flavour. Sherry Oak Cask wins.

Bushmills Black Bush vs Macallan Quest: Bushmills is paler, with a more flowery and fruity aroma. Macallan surprisingly subtle in comparison. Bushmills tastes sweet, flowery and some caramel, rather thin but not bad. Macallan has more bourbon flavour, probably some sherry but that is not what I think of after Bushmills. Another taste of Bushmills and it still holds up, and back to Macallan a bit sharp and alcohol. Bushmills wins.

Johnny Walker White Walker vs Macallan Quest: JW is paler, smells a bit of caramel, Macallan is more subtle on the nose. JW has a sweet, somewhat chemical flavour, Macallan more bourbon and the real thing. Macallan wins.

7 dlight Ichiro Mizunara Reserve vs Macallan Quest: Macallan a bit darker. Ichiro has a sweet, somewhat raw aroma, Macallan softer and more subtle. Tasting both I find the same thing, Macallan is a bit more the safe classic choice, Ichiro is not as easy to drink but it has more to offer. Ichiron wins.

Canadian Club 100 Rye vs Hudson Four Grain: Both quite dark, Canadian Club not as dark and not as red as Hudson. Canadian Club is rather fruity on the nose, I kind of think of strawberries and cherries. Hudson smells like sawing in a piece of oak wood. Tasting Canadian club is a bit underwhelming, it is not bad but not much happens, after a while I think I am drinking perhaps grappa or something. Hudson has a bit more complexity in the mouth, but not balance whatsoever. It tastes oak, bourbon and it is both sour and bitter. Of course there is more to talk about when it comes to Hudson, but more is not always better, and I think Canadian Club must win.

Miyagikyo Single Malt vs Yamazaki Distillers Reserve: Same color. Miyagikyo has nice sweet maltiness on the nose. Yamazaki is more dry, more spicy, even more malty – Miyagikyo is more fruity. I taste Miyagikyo and find it quite sweet, rich, soft, and something that resembles peat. Over to Yamazaki it is less rich, less complex, less balanced – yet good, some hints of fruitiness in the background, but next to Miyagikyo I think it fades.

Glenfiddich 12 vs Yamazaki Distillers Reserve: Similar color. Yamazaki has a more malty and rich aroma, Glenfiddich a little bit odd making me think of fusel oil. I taste Glenfiddich, it is quite light, but it has some complexity and develops nicely in the mouth. Yamazaki has a more integrated powerful and rich flavour. Both taste quite much standard malty whisky. Quite similar in flavour and quality, especially on the nose I think Yamazaki is a little bit ahead.

Dufftown Malt Masters Selection vs Glenfarclas 12: Very similar color. Dufftown has a bit fruitier nose, Glenfarclas drier more hay or mint or something. Quite similar. Dufftown has a very typical speyside flavour, not bad but nothing extra. Glenfarclas a bit more malty, bready, salty. I prefer Glenfarclas.

Dufftown Malt Masters Selection vs Glenrothes 12: Dufftown is paler. Glenrothes has a more sweet bourbon-like aroma. Dufftown has a rather clean dry speyside flavour, Glenrothes is sweeter, more flowery, kind of odd. I prefer the more classic Dufftown.

Dufftown Malt Masters Selection vs Edradour 10: Edradour is darker, with a more powerful aroma of what makes me think of sweet red fruits. Edradour has an odd flavour that makes me think of other things than whisky, and Dufftown being more dry and malty tastes like a classic Speyside. Dufftown wins.

Bulleit Rye vs Canadian Club 100 Rye: Bulleit slightly darker, but much more powerful on the nose, bourbon with some flowery hints (from the Rye typically). Canadian Club more caramel, a bit chemical. Tasting Canadian club it is smooth, caramel, a bit sweet. Bulleit Rye is much more powerful, raw, bourbon. Those who really like rye and bourbon probably prefer Bulleit, but I have to say I would rather drink Canadian Club.

Glefarclas 12 vs Glenlivet Founders Reserve: Glenfarclas wins.

Glenlivet Founders Reserve vs Glenmorangie 12 Lasanta: Glenmorangie wins.

Highgrove Organic vs Yamazaki Distillers Reserve: Highgrove is paler in color, and has lighter more pure, caramel-malty aroma. Yamazaki is sweeter, deeper, a bit oily perhaps even with a hint of peat. Tasting Highgrove it tastes a lot of citrus and without water it is not particularly soft. Yamazaki is softer, richer, it fills the mouth more, and it has more complexity. Highgrove is a bit intense without complexity, and it can not match Yamazaki. Back to Yamazaki I would guess that its flavour is dominated by some wood not typical for scotch whisky, but in a balanced way.

Chivas Regal 18 vs Tamnavulin Double Cask: Chivas is darker, at first they are quite similar on the nose, classic, a bit sweet, with Chivas a little bit more mellow and Tamnavulin a bit more sour fruity and fresh. Tamnavulin tastes soft, a bit sweet more than malty and if it is sherry it is very well balanced, rather easy to drink. Now Chivas both tastes and smells a bit peated, a bit more oily, a bit more complex and a bit more challenging. Tamnavulin is good also after Chivas, it is a bit simple though. Chivas is both more complex and balanced, and wins.

Ballantines 17 vs Glen Ord 18: Ballantines is paler, with a fresh malty aroma. Glen Ord smells, liqourice? I am a little confused about the aroma here, Ballantines is lighter and less powerful, Glen Ord is a bit heavier. Both are rather balanced on the nose. I taste Ballantines and it is light, balanced, lingering nicely, quite excellent without being anything extra. Glen Ord is a bit extra, and not so little extra, a bit oily, some citrus. Ballantines is good, but more plain than Glen Ord. Glen Ord wins.

Famous Grouse vs Glen Garioch Founders Reserve: Famouse Grouse is slightly paler, with a fresher aroma. Glen Garioch smells a bit oily in a bad way, not fresh. Tasting Famous Grouse, a bit thin, a bit chemical and a little bitter but drinkable. Glen Garioch is a bit sour, raw wood, young and unrefined. Back to Famous Grouse it tastes like a very safe choice. And Glen Garioch again, no I do not like it. Famous Grouse wins.

Glen Ord 18 vs Springbank 18: Glen Ord a bit more red. Big difference on the nose, Glen Ord malty and fresh, Springbank heavy and peated (but not that peated). Tasting Glen Ord, it is rich, malty, complex and tasty, very nice. Springbank, a bit sour and then a bit of sulphur. I do not like it. Glen Ord wins.

Glenlivet 16 Nadurra vs Macallan Whisky Makers: Glenlivet paler even if cask strength. Both have a sweet fruity sherry arama, Glenlivet a bit more fresh and sweet, Macallan a bit more velvet mellow. Glenlivet first quite much bourbon in the mouth, then sweet-sour fruit, a bit unbalanced. Macallan more balanced, dry sherry. I dont particularly like Glenlivet, Macallan wins.

Macallan Fine Oak vs Macallan Whisky Makers: Fine oak is paler, with a slightly more sour and raw aroma than Whisky Makers. Very similar flavour, Whisky Makers being more soft and balanced, and wins.

Glenfiddich 15 Solera vs Macallan Whisky Makers: Glenfiddic is paler, with more soft sweet caramel araoma, Macallan being more sherry. Glenfiddich is soft, rich, fruity, malty, lingering nicely. Macallan is more sherry. A sherry fan will probably go with Macallan, but I enjoy Glenfiddich better.

Macallan Whisky Makers vs Yamazaki Distillers Reserve: Yamazaki paler, and a bit more subtle on the nose, and Macallan is more sherry. Yamazaki tastes good, malty, classic, a bit light. Macallan tastes sherry, and I enjoy Yamazaki more.

Macallan Whisky Makers vs Redbreast 12: Redbreast a bit darker, with more bourbon aroma. Redbreast tastes a bit dry bourbon, with some fruity finish. Macallan tastes more sherry, a bit sour. I prefer the flavour of Redbreast.

Macallan Whisky Makers vs Mackmyra Reserve Elegant Ambassadör: Macallan slightly darker, more sherry aroma, Mackmyra a bit unusual wood aroma. Mackmyra also has a bit unusual woody flavour, a bit pepper, quite sweet and soft. Macallan more a classic sherry flavour. I like Mackmyra better.

Mackmyra Vit Ek vs Svensk Whisky för Ukraina: Mackmyra ha a bit more dark brown color, also with a richer, more raw wood, aroma. Tasting Svensk Whisky för Ukraina, seems young, a bit of bourbon. Mackmyra has a more rich, balanced and elegant flavour. Mackmyra wins.

Bergslagen Gast vs Mackmyra Vit Ek: Mackmyra a bit darker, Bergslagen a bit more sour peated aroma. Mackmyra is a bit softer, a bit desert wine like. Bergslagen has a fresh flavour, both smoke and sweetness. Mackmyra a bit more balanced, softer, lingers with some fruitiness. I would say this comes down to if you prefer a more peated salty whisky, or a softer more fruity whisky. Too me, Mackmyra has more quality.

Bergslagen Two Hearts vs Mackmyra Vit Ek: Very similar quite dark color. Bergslagen has a salted caramel aroma, mackmyra more like a lightly peated desert wine, more fruit. Tasting Bergslagen, quite sweet, a bit raw, lingers but not very softly. Mackmyra is softer, it really makes me think of the color yellow (plums, oranges, I dont know). Back to Bergslagen, it is a bit harsch and bitter. Mackmyra wins, its light peat, softness, and lingering fruitiness is a winning concept.

Dufftown 18 vs Mackmyra Vit Ek: Mackmyra very slightly darker. Dufftown a bit more salty and fresh on the nose, Mackmyra a bit more yellow fruit and sweet, with some peat. Dufftown has a soft, fruity, malty flavour, both balanced and complex. Mackmyra tastes much younger, a bit raw, and sour. Dufftown is less powerful, not as strong, but I think it tastes better and has more quality.

Mackmyra Vit Ek vs Yamazaki Distillers Reserve: Mackmyra slightly darker, with a more powerful aroma, but Yamazaki has a more classic malt aroma. I taste Yamazaki and find it quite classic malt with some fruity sweetness to it, perhaps most orange. Mackmyra more powerful, more sour (hint of peat). I think Yamazaki has more quality, and tastes better.

Old Pulteney 18 vs Writers Tears Japanese Cask Finished: Writers Tears very slightly darker. Old Pulteney is more malty salty on the nose, something chemical oily that I do not like so much. Writers Tears is much more fruity, perhaps on the chemical side. Tasting both quite same impression, mixed feelings about both, but I think Old Pulteney is better.

Hibiki Harmony vs Miyagikyo Single Malt: Similar quite pale color. Hibiki mostly caramel on the nose, Miyagikyo much more fruity, almost like a wine. Hibiki, soft, nutty, caramel a bit salty easy to drink. Miyagikyo immediately and unexpectedly a bit peated after Hibiki. Back to Hibiki, a bit sweet punch like, a bit synthetic. Miyagikyo is both fruity and lightly peated in an unusual way, but it works. I prefer Miyagikyo.

Hazelburn 10 vs Yamazaki Distillers Reserve: Hazelburn is paler with a somewhat more raw, salty and dry aroma. Yamazaki is more balanced, full bodied, on the nose. Hazelburn is dry in the mouth, even a bit burnt, quite light. Yamazaki has a less malty, more alcohol-flavour. Back to Hazelburn, it has an open complexity with no flavour dominating and it lingers nicely. Yamazaki tastes more closed, less developed, the flavours hidden in each other (perhaps I should have added water), resulting in something somewhat chemical. Hazelburn wins.

Glen Scotia Campbeltown Harbour vs Mackmyra Vit Ek: Mackmyra slightly darker, with a more powerful, deep and oaky aroma. Glen Scotia very subtle in comparison, a bit fruity. Tasting Glen Scotia it is light, but it has some complexity with some peaty hints. Mackmyra is a bit stronger, a bit more spicy and in your face, and not as smooth and easy as Glen Scotia. Back to Glen Scotia it is thin, makes me think of a blend. Mackmyra wins.

Glen Scotia Campbeltown Harbour vs Jura Superstition: Jura very slightly darker, with a somewhat more oily aroma. Glen Scotia a bit more fruity and peat. Jura has a nice saltiness to it, but also a raw woodiness and somthing lightly Floki-like about it. Glen Scotia is lighter, more flawless, but a bit blend-alcohol-like. Jura tastes a bit nasty (like Floki), victory to Glen Scotia.

Glenfarclas 12 vs Glen Scotia Campbeltown Harbour: Similar color. Glenfarclas more dry, a bit herblike on the nose, Glen Scotia more fruity in a somewhat artificial way. Tasting Glenfarclas it is soft, salty, balanced, a typical speyside with some odd herbiness to it. Glen Scotia is more oily, slighly peated, a bit lighter, and it tastes alcohol like a blend. Glenfarclas wins.

Chivas Regas 12 vs Oban Distillers Edition: Oban darker in color, more powerful and sweet on the nose. Tasting Chivas I find it quite thin, with a body of nutty caramel, not bad. Oban, on the other hand is sweet, raw and not so little sulphur or whatever it is that tastes old margarine. Not nice. Chivas wins.

7 dlight Three Ships vs Glen Garioch Founders Reserve: Glen Garioch a bit darker, with a slightly nasty fusel oil aroma. Three Ships has a light smokiness to it (resembles Mackmyra Vit Ek from the other day). Tasting Glen Garioch, a kind of spicy malty flavour, better than I expected. 7 dlight, a rather sour with some peat. Back to Glen Garioch, it is more classic that 7 dlight, but a bit unpleasant. Its odd, after tasting Mackmyra Vit Ek I now appreciate 7 dlight for the same reason, I think. 7 dlight wins.

Longrow vs Mackmyra Vit Ek: Longrow is paler, with a more dry more peated aroma. Mackmyra more fruity desert wine. Longrow has a dry, straight simple quite balanced, fresh and good flavour. Mackmyra more young wood, not very raw but still woody, a bit bitter with the sourness. Longrow wins.

Highgrove Organic vs Highland Park 10 Viking Scars: Highgrove is much paler, with a lighter more fresh aroma. HP has a somewhat sour aroma that I find a bit challenging at first. Highgrove light in flavour too, very fresh, clean maltiness. HP a bit more rough, dirty and oily, some of that fusel-oil smell also in the flavour, but it is kind of part of the character in a nice way. Highgrove, somewhat bitter and uncharming. HP is richer and softer, I think that makes it an HP victory.

Dufftown 18 vs Yamazaki Distillers Reserve: Dufftown a bit darker, with a more dry, fresh clean aroma. Yamazaki is a bit more powerful, spicy, nutty. Dufftown has a rather clean neutral Speyside flavour, not bad but not much write about. Yamazaki a bit more oily and more flavourful, also very balanced. Yamazaki wins.

Glen Ord 18 vs Yamazaki Distillers Reserve: Glen Ord a bit darker, and a bit more powerful and sweet arama. Both have a classic balanced aroma though, flavours quite similar, Yamazaki somewhat more accurately on spot winning narrowly.

Glen Ord 18 (2019 special release) vs Yamazaki Distillers Reserve: Very similar color and very similar aroma. Also, flavour not so different, but Glen Ord is more complex, lingers longer, and gives a richer experience. Glen Ord wins.

Arran Heavily Peated Sherry Cask vs Bowmore 18: Arran is more dark and red. Arran a bit raw wood, quite peated and sweet. Bowmore more classic malted, more dry, a bit peated but not as Arran, and more sherry rather than just sweet. Tasting Bowmore, surprisingly peated flavour and salty/dry in an elegant way. Surprisingly little peat flavour, a bit raw wood, artificially sweet, subtle balanced sherry. Bowmore wins, but two good peated sherry whiskies.

Bushmills 16 vs Writers Tears Japanese Cask Finished: Bushmills is darker with quite a bourbon aroma. Writers Tears a bit dryer (yet a bit fruity), less powerful. Back to Bushmills, yes bourbon. Writers Tears, harder to describe the aroma. Tasting Writers Tears, after adding a bit of water, a bit bitter-sweet, balanced, not overwhelming. Bushmills is more powerful, like a soft sweet balanced bourbon. Now back to Writers Tears it smells more flowery fruity. I think this thing with Japanese finish on Mizunara wood is a quite delicate things for enthusiasts and something that does not necessarily translate straight to a superior casual experience. I think Bushmills wins.

Agitator Select Cask Ex Islay vs Bunnahabhain Peat & Fruit Coopers Choice: Agitator is paler, with a kind of light, winelike, peated aroma. Bunnahabhain is more sweet with the peat in the background. Both are cask strength so I add water to both. Now Bunnahabhain is more peated, a sweet oily peat rather than smoke. Agiator changed less with water. I taste Agitator and find a dry, almost ashy peat, and a pure clean flavour. Bunnahabhain struggles more with what it wants to be, both sweet and smoky, with a bit of sulphur. Agitator wins.

Agitator Select Cask Ex Islay vs Bowmore 12: Agitator much paler, Bowmore amber. I am beginning to find orange in Bowmore and that, together with some peat, is what hits my nose first. Agitator is a bit lighter, both are peated, but when compared to each they seem to be peated to a similar degree and neither is very peated. Bowmore is more oily, chemical, heavy, and Agiator is more wine and smoke. I taste Bowmore and find it surprisingly sweet, soft, I think I can say orange with a bit of smoke. Agitator is also soft, but more salt and burnt. I find Bowmore a bit odd, perhaps they want it to taste like this, but too me it is a bit chemical. Agitator is a more refreshing peated experience. Agitator wins.

Agitator Select Cask Ex Islay vs Bowmore 15: Bowmore is a dark whisky, Agitator nearly colorless. On the nose Bowmore is more sweet, orange, oily. Agiator is more light white wine and smoke. Tasting Bowmore it is surprisingly dry, a bit peated, and with a fruity finish. Agitator is a more simple experience, it is dry and a bit of smoke. Trying Bowmore again, there is much to discover in Bowmore, it lingers nicely. I don’t completely love Bowmore, but Agitator is to plain and simple to match the more complex and interesting Bowmore.

Glen Scotia Campbeltown Harbour vs Johnny Walker White Walker: Glen Scotia somewhat more pink and clear in color, Johnny Walker more yellow. Glen Scotia a bit sour (hint of peat), almost flowery on the nose. Johnny Walker is more sweet, spicy, creamy, coffee? I taste Johnny Walker and it has a thick feeling in the mouth, like it is actually sweetened, and it has a soft, somewhat chemical easy to drink flavour. Glen Scotia is also soft, balanced, clean but with some complexity. Back to Johnny Walker it is mostly sweet, probably vanilla. Campbeltown Harbour is nice in a classic way, victory to Glen Scotia.

Nikka Coffey Malt vs Tamnavulin Double Cask: Tamnavulin slightly darker, or at least more red. Surprisingly similar aroma, both have a quite classic malt whisky aroma on the sweet side. Nikka is more vanilla, cream and spice. Tamnavulin is more fruit. I taste Nikka and it has something spiced about it, like punch. I taste Tamnavulin, and it actually tastes quite similar, yet a bit fruity. Back to Nikka it makes me think of honey rather than fruit. I find the Nikka experience more convincing somehow, Nikka wins.

Bushmills Original vs Tullamore Dew: Bushmills slightly paler, with a slightly more sweet caramel and soft aroma. Tullarmore Dew smells more pure alcohol. Bushmills also tastes a bit softer and sweeter, Tullamore is more burnt, more bitter and tastes more alcohol. Bushmills wins.

Edradour 10 vs Glen Scotia Campbeltown Harbour: Edradour very much darker, sweet fruity aroma. Glen Scotia has a light peated, salty oily aroma, and a fresh light balanced flavour. Edradour is more powerful, with a bourbon sweet fruit flavour that tastes a bit like extra-flavour rather than being well integrated. Back to Glen Scotia, a bit sweet as wall, an elegant simple easy to drink whisky with interesting flavours and complexity. Edradour, to me, is more odd. I prefer Glen Scotia.

Agitator Select Cask Ex Islay vs Deanston Kentucky Cask Matured: At least color is quite similar, both rather pale. On the nose, Deanston is sweet cream caramel, hint of fruitiness. Agitator is peated, more powerful. I taste Deanston, and even though it is light, almost diluted, it has some complexity and it is very easy to enjoy. Agitator is of course more of an aquired taste with an ashy smoke flavour. Back to Deanston, it still has flavour and quality after the more powerful peated whisky. If you want a peated whisky, go for Agitator, but in every other way I think Deanston is a better choice.

Ballantines 17 vs Glenburgie 15: Ballantines is more dry, more malty, lighter and more complex. Glenburgie is more sweet and bourbon, which makes it slightly bitter. I prefer Ballantines.

Deanston Kentucky Cask Matured vs Glenburgie 15: Deanston paler, with a soft salt caramel aroma. Glenburgie has a deeper sweeter aroma. Deanston has a soft light flavour, not so sweet. Glenburgie a bit more powerful, hints of fruit or even sherry. It is not so easy to pick a winner here, Deanston is a bit thin and kind of cheap tasting, but in that lightness there is a simple appealing complexity. Glenburgie has more of a body, but I find there is less to discover and enjoy and I prefer Deanston.

Glenburgie 15 vs Glenmorangie 10: Glenburgie is darker, with a more powerful bourbon aroma. Glenmorangie has a light aroma, a bit hay or herbs. I taste Glenmorangie and it is surprisingly salty and a bit herby, with a nutty finish. Glenburgie is more powerful, sweeter, and I feel quite sure now that there is a hint of sherry in it. Glenburgie tastes like the more complete whisky and I enjoy it more.

Glenburgie 15 vs Mortlach 12: Similar color, Mortlach has a slightly more oily rough aroma, and that difference get more clear when tasting. Glenburgie is rather soft and sweet, Mortlach is heavier, spicier and more raw. I prefer Glenburgie.

Bladnoch Adela 15 Oloroso vs Glenburgie 15: Bladnoch darker with an obvious sherry aroma, and not so little sherry flavour. Glenburgie is a more classic choice with a soft sweet malty flavour. I add little water to Bladnoch and it is a quite good sherry whisky, but I like Glenburgie better.

Glenburgie 15 vs Highgrove Organic: Glenburgie much darker, with a more full and sweet aroma. Highgrove has a light aroma, very fresh like lemons, if anything. Highgrove tastes quite much alcohol first, then comes an unusual burnt, sweet flavour of fire. Glenburgie more classic sweet soft, but also a bit bitter. Highgrove tastes quite raw and young, although it has quality too. Glenburgie is a much safer choice, I go with it.

Mackmyra Reserve Elegant Ambassadör vs Mackmyra Vit Ek: Vit Ek slightly darker, with a clear peated nose and some sweet bourbon under the peat. Ambassadör is fruity sweet, not peat at all. Tasting Ambassadör is quite sweet and nice, with a somewhat odd young raw wood. Vit Ek is not so raw wood, quite peated, with hints of bourbon. This is close, and perhaps it just comes down to personal preference or liking or not liking peat. I find Ambassadör more odd, and I find Vit Ek more balanced and unusual in a more interesting way, so it is victory to Vit Ek.

The Irishman vs Tullamore Dew: Same color, like the Irish has decided this is the right color. The nose is different though, The Irishman beinq quite fruity rich and sweet, and Tullamore Dew more smelling pure alcohol. Tullamore Dew has a quite dry wine flavour, The Irishman being more soft and easy to drink. Tullamore Dew has some bitterness and overall more flavour and complexity. I am really leaning towards The Irishman, but I could make an argument for Tullamore Dew having a more dry, manly, acquired taste for grown ups rather than the somewhat cheap sweet easy to drink Irishman. But I will not make that argument, The Irishman wins.

Glen Garioch Founders Reserve vs Tullamore Dew: Same color, Glen Garioch has a somewhat nasty fusel oil aroma while Tullamore Dew has a more alcohol aroma. Glen Garioch does not taste as bad as it smells though, and Tullamore Dew comes out as quite chemical, a bit bitter, and quite uncharming. In the end I have to pick Tullamore Dew as the winner, for an overall less negative experience.

Miyagikyo Single Malt vs Old Pulteney 18: OP very slightly darker in color. On the nose I find Miyagikyo a bit more powerful, slightly more sweet, and Old Pulteney is a bit more fresh and bourbon. Tasting is not so different, I find Miyagikyo very intergrated (rather than complex), it is like it has one very flavour and it is hard to pick out parts (like sherry, that is supposed to be in there), and it is a nice whisky flavor but it kind of tastes manufactured to perfection beyond real charm and personality. Old Pulteney is more malty, salty, fresh and more bourbon. It is easier to pick out different flavours in OP, makes a more open impression, and a lighter impression, to me. In conclusion, I really like the OP18-style of whisky, but despite that I think OP has more quality and complexity as well.

Longmorn 16 vs Writers Tears Japanese Cask Finish: Similar color. Longmorn has a bourbon aroma with some fruitiness. Writers Tears has a more dry and spicy (rather than fruity) aroma. Longmorn has a rich flavour, lingering, fruity, easy to drink, with a bit of bourbon finish, very nice. Now, when about to drink Writers Tears, I find a kind of artificial fruitiness on the nose and in the mouth I find the whisky more raw and less mature than Longmorn. Back to Longmorn, it is not overly complex but very nice. Writers Tears a bit more dry, exotic, unusual, it stands up better the second round, but with some bitterness. Longmorn wins.

Kilkerran 8 vs Tullamore Dew: Kilkerran a bit darker, with a sour too-much-sherry-aroma, and a hint of peat. Tullamore Dew has a light alcohol kind of anonymous blend aroma. I taste Tullarmore and it has a sweet body and I find it quite easy to drink. Kilkerran, this is the peat-and-sherry mix that seems to be quite popular, but I find it almost repulsive. I add more water (that usually eliminates the taste of sulphur or bad margarine) and I can recognise the young sherry-peat-qualities. I taste Tullamore Dew, and I can really taste that this is a product that is manufactured in 10000x or more quantities than Kilkerran. Kilkerran is just raw, sherry, peat, experimental, small batch. Tullamore Dew is just manufactured to specification, uncharming. I rather drink Tullamore Dew, but if you like young sherry-peat whisky you will disagree a lot.

Tobermory 12 vs Writers Tears Double Oak: Very similar color. Tobermory has a nice rich somewhat fruity wine-like aroma with a solid maltiness underneath. Writers Tears is less fruity, more caramel, fudge, and vanilla but not really bourboun. I taste Writers Tears and it has a solid, balanced flavour, quite light and fruity. Back to Tobermory it smells a bit oily now, and it tastes more burnt, so not as soft as Writers Tears. Back to Writers Tears, it has a very clear, pure and crips flavour, with something elegant close to bitterness. Tobermory, it is not peated but after Writers Tears it is a bit in that direction. Toberymory is very good, but next to Writers Tears I pick that latter one as winner.

Famous Grouse vs Tullamore Dew: Famous Grouse a bit a bit paler, with a light chemical alcohol aroma. On the nose, I have a hard time find much difference. Tullamore Dew is a bit sweeter and heavier. I taste Tullamore Dew and at first it kind of tastes alcohol like I expect non-whisky-drinkers to think of whisky, but as that first impression fades away comes some subtle notes of malt and caramel that are nice, but subtle. Famous Grouse, similar initial impression, but it mostly stays chemical. Back to Tullamore Dew, this whisky has a more rich and soft body with actual whisky flavours in it, and I do not enjoy Famous Grouse when going back to it. Tullamore Dew wins.

Old Pulteney 12 vs Tamnavulin Double Cask: Old Pulteney much paler, with a salty aroma of sea and oil. Tamnavulin is more like a sweet desert on the nose. Tasting Old Pulteney I find it fresh and kind of complex for being relatively young and light. Tamnavulin is more sweet, nice but somewhat chemical and unnatural. Old Pulteney is not perfect, but I enjoy it more, it has a more authentic, interesting and unique character, so it wins.

Finlaggan (Batch Strength) vs Loch Lomond Heavily Peated: Finlaggan is darker, with a peated oily aroma. Loch Lomond has more of a burnt wood arama. Just using my nose I prefer Finlaggan. Tasting Finlaggan I find first chocolate-caramel, and then smoke-alcohol. Not much more. Loch Lomond is more complex, more soft, more chemical but in the end the slightly better whisky.

Glen Ord 18 vs Tobermory 12: Very similar color. There is much more bourbon and oak aroma in Tobermory, of course not like a real bourbon, but head to head Glen Ord is like a Speyside malt and Tobermory like a soft bourbon, both very nice thought. I taste Tobermory and it offers a bit over everything, fruitiness, bourbon and hints of peat (I think). After that Glen Ord tasted odd, very fruity, almost perfume. Back to Tobermory it now even has a bit of the raw character of a Bourbon. Glen Ord is fine but a bit shadowed by the more powerful and distinctive Tobermory.

Glenlivet 18 vs Miyagikyo Single Malt: Glenlivet much darker with a somewhat sweet and fruity nose. Surprised, I find Miyagikyo also sweet and quite similar, in fact very similar. There is more bourbon aroma in Glenlivet and Miyagikyo is slightly more fruity-sweet. Glenlivet tastes rich, complex, soft, it lingers nicely. Miyagikyo is a bit stronger, a bit less soft, more crisp rather than mellow, and more fruity. Back to Glenlivet, a significant bourbon flavour as well now, with some bitterness. Miyagikyo, somewhat bittersweet, I find Glenlivet to be a bit better.

Glen Garioch Founders Reserve vs J&B: Glen Garioch is much darker, with a malty oily aroma. J&B is lighter on the nose, more like vodka, and it has a quite mild somewhat peated flavour. Glen Garioch is not surprisingly much more powerful in flavour, more malty. Back to J&B it is still the same thing, still enjoyable. It is no secret that there is something (fusel oil) about Glen Garoich that I do not like, but today compared to J&B I don’t find that thing so dominant, and after all Glen Garioch is a more rich experience. Glen Garioch wins.

Agitator Argument Kastanj vs Svensk Whisky för Ukraina: Agitator is darker, with a sweeter more heavy aroma. Ukraina is more bready malty. Both smell young. Tasting Ukraina, a bit raw and salty first but that is followed by nice notes of nuts, caramel and honey. Agitator is more single-flavoured, the chestnut gives a particular flavour, it is a bit sweet with some bitterness. Ukraina has a bit bourbon on the nose as well. I prefer to drink Ukraina.

Agitator Argument Kastanj vs Hazelburn 13 Oloroso: Hazelburn a bit darker, with a bit of sulphur aroma, compared to Agitator that is more young wood. Tasting Hazelburn, the sulphur dominates, but it is surprisingly dry for a sherry. Agitator is sweeter, easier to drink and has a softer finish. There is quality in Hazelburn but it cannot make up for the odd, rust-like flavour. Agitator wins.

Deanston 15 Organic vs Lindores: Both are very pale, Deanston is the most pale. Lindores has a more sweet, fruity, oaky bourbon caramel aroma. Deanston is very dry, with something somewhat unpleasant. I taste Deanston and find it quite light, somewhat bitter, not unpleasant in itself but no exactly smooth, so to say. Lindores is more sweet and fruity but it is almost immediately dominated by an odd bitter-sweet flavour, almost like tonic. Back to Deanston, given its sharp thin character, there are nice notes of bourbon and fruit. Lindores is worse, probably it is rather young and has picked up too much raw character from a cask. Deanston wins.

Akashi Blended Whisky vs Bushmills Original: Bushmills is paler, with a light fruit-caramel aroma. Akashi is even more light or subtle on the nose, more vodka like. I taste Akashi and perhaps the flavour of alcohol and a somewhat unpure distillation dominates real whisky flavours, it is at least rather soft. Bushmills has a more soft, rich, almost flowery flavour. There really is not anything good about Akashi, the best about it is that it is quite soft and subtle. Bushmills wins for at least offering the idea of experiencing a whisky.

Akashi Blended Whisky vs The Irishman: If any difference, Akashi a bit darker. Both have a quite alcohol-dominated nose, but The Irishman is more fruity and flowery, while Akashi is more dry and malty. Tasting Akashi, the dominant flavour is actually pure alcohol. The Irishman – I would say the difference in taste is less than I expected after using my nose – a bit more sweetness and less alcohol flavour. Both just seems to belong to the same blend standard – not the worst but not premium. The Irishman wins.

Lindores vs Mackmyra Brukswhisky: Two pale whiskies, Mackmyra being the most pale. Both are harsch, dry on the nose, with some bitter herb rather. Mackmyra more so, Lindores a bit sweeter. Mackmyra, this is a strange flavour, I would probably not have identified it as whisky, it is more glue and grapefruit. Lindores is more gin. This bitter-sour-dry grapefruit of Mackmyra is not that nice. Lindores has a hint of maltiness or caramel, which is nicer. Lindores wins.

Agitator Argument Kastanj vs Makers Mark: Same quite dark color. Makers Mark has a rich soft bourbon aroma, really thinking vanilla and caramel here. Agitator is a bit sour on the nose, on the brink of peated, with some fruitiness. Tasting Agitator, rather sharp and short in flavour, a bit grappa-like. Makers Mark, really big flavour that fills the whole mouth, a bit perfume and a bit bitter. Two very different whiskies, i prefer Agitator.

Glenmorangie 12 Lasanta vs Glen Scotia Campbeltown Harbour: Glenmorangie slightly darker in color, with a somewhat sweet and fruity aroma. Glen Scotia is more dry, hint of sourness and peat. I taste Lasanta and find it quite ordinary, a bit sweet, quite balanced (not as bad as I remember it). Glen Scotia is more dry, a bit peated, it opens up more in the mouth and lingers longer, probably because o lower ABV. I add some water to Lasanta and taste again, now there are hints of bourbon but a rather dull whisky. I think Glen Scotia is a flavourful, easy to drink whisky, that makes me more curious about its origin. Nothing of that can be said about Glenmorangie. Campbeltown harbour wins.

Dalmore 11 Rare Find Oloroso vs Glen Garioch Founders Reserve: Dalmore is darker, more dark brown than red or amber. Glen Garioch has a malty spicy aroma making me think of fusel oil. Dalmore has a more neutral classic aroma, absolutely with some Oloroso. Back to Glen Garioch it smells a bit like someone puked in the cask. I taste Glen Garioch and it has a malty saltiness at first, which is ok, but it becomes a bit bitter and sharp, which is not that good. Dalmore is kind of sweet, with a quite dominant flavour or bad margarine. Back to Glen Garioch, it tastes like a young bitter blend, and Dalmore first 2 seconds would beat it, if the Dalmore finish was not outright disgusting. I discard the last Dalmore, finish the Glen Garioch, barely enjoying it at all. Glen Garioch wins.

Ben Bracken Speyside vs Glen Moray Classic: Ben Bracken is darker. Both aroma and flavour, Ben Bracken is more dirty and malty, Glen Moray is more winey and a bit bourbon. Both have a bit more bitterness than desired. Ben Bracken has richer less refined flavour, while Glen Moray has a quite simple but rather elegant wine-like flavour. I can argue for either of them, but I prefer Glen Moray.

Agitator Argument Kastanj vs Glenmorangie Lasanta: Agitator very slightly darker, with a malty caramel aroma, a bit fruity as well. Glenmorangie smells more vanilla and bourbon. Agitator, at first, is quite sharp and a bit odd young tasting, some bitterness too. Glenmorangie is more sweet and fruity, more easy to drink. I prefer Glenmorangie.

Mackmyra Reserve Extra Rök Svensk Ek vs Mackmyra Vit Ek: Both quite dark, Vit Ek is paler. On the nose, more peat (or at least smoke) and wood with Reserve. Vit Ek has a nice boubon, caramel and vanilla aroma. Tasting Vit Ek is nice, it is balanced, a bit salty, a bit peated, and not bitter. Mackmyra Reserve has a more powerful aroma, a bit odd/sweet at first, but really nice middle and finish. Back to Vit Ek, after Reserve, the Vit Ek is more chemical and not so impressive. This is one of those cases where Reserve wins head to head, but perhaps Vit Ek would be a better choice for just one whisky. Maybe. Victory to Mackmyra Reserve.

Canadian Club 1858 vs Teeling Rum Cask: Canadian club darker, with a sweet chemical nose dominated by alcohol. The very pale Teeling has a more dry neutral alcohol aroma. I taste Teeling, it is easy and thin, a bit sweet. Canadian Club is more sweet, more rich, more making me think of a mixed drink, a bit chemical. Canadian Club is more powerful, but in this case I think that is a bad thing. Teeling wins.

Canadian Club 1858 vs J&B: J&B is paler, with a somewhat sour smell of alcohol. CC is sweeter, smelling syrup. I taste J&B, and it tastes a bit better than it smells, balanced, light and soft whisky flavour. Canadian Club is however more sweet, soft, and balanced, with less chemical alcohol flavour. CC wins.

Bowmore 1999 Bourbon Hand Filled vs Bowmore 15 Golden & Elegant: Hand filled is a bit paler (G&E contains artifical coloring), and has a more dry bourbon aroma. G&E is a bit more peated, more sweet and more dirty on the nose. If there are any sulphur hints in any of them, it is hidden in the quite dominant orange flavour of G&E. I have noticed significant orange flavour inte the Hand filled sample before, but not head to head with G&E. Orange or not, G&E has more flavour, is more peated, is softer, and I have to say it is a better whisky overall.

Excessive SYS CPU with NodeJS 20 on Linux

I am running a system, a collection of about 20 Node.js-processes on a single machine. Those processes do some I/O to disk and they communicate with each other using HTTP. Much of the code is almost 10 years old and this system first ran on Node 0.12. I can run the system on many different machines and I have automated tests as well.

The problem demonstrated for idle system using top

I will now illustrate the problem of excessive SYS CPU load under Node 20.10.0 compared to Node 18 on an idle system, using top.

TEST (production identical cloud VPS, Debian 11.8)

Here the system running on Node 18 has been idling for a little while.

top - 12:44:46 up 3 days, 23:21,  4 users,  load average: 0.02, 0.44, 0.35
Tasks: 109 total,   1 running, 108 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.5 us,  0.7 sy,  0.0 ni, 96.4 id,  0.1 wa,  0.0 hi,  0.3 si,  0.1 st
MiB Mem :   3910.9 total,    948.2 free,   1484.8 used,   1478.0 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   2166.2 avail Mem

Upgrading to Node.js 20.10.0 and letting the system idle a while gives:

top - 12:54:20 up 3 days, 23:30,  2 users,  load average: 0.79, 1.74, 1.16
Tasks: 108 total,   3 running, 105 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.3 us, 20.4 sy,  0.0 ni, 76.0 id,  0.0 wa,  0.0 hi,  1.3 si,  0.0 st
MiB Mem :   3910.9 total,    809.8 free,   1316.8 used,   1784.3 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   2347.7 avail Mem

As you can see, the SYS CPU load is massive under Node 20.

RPI v2, Raspbian 12.1

Here the system running on Node 18 has been idling on a RPi2 for more than 15 minutes.

top - 12:38:36 up 42 min,  2 users,  load average: 0.13, 0.11, 0.63
Tasks: 133 total,   2 running, 131 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.2 us,  1.2 sy,  0.0 ni, 95.6 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :    971.9 total,    436.0 free,    324.3 used,    263.3 buff/cache    
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.    647.6 avail Mem

This is a very under powered machine, but it is ok.

Upgrading to Node.js 20.10.0 and letting the machine idle gives:

top - 12:55:09 up 59 min,  2 users,  load average: 0.56, 1.38, 1.32
Tasks: 139 total,   1 running, 138 sleeping,   0 stopped,   0 zombie
%Cpu(s):  4.3 us, 12.6 sy,  0.0 ni, 82.7 id,  0.3 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :    971.9 total,    429.5 free,    327.9 used,    266.5 buff/cache    
MiB Swap:   8192.0 total,   8192.0 free,      0.0 used.    644.0 avail Mem

Again, a quite massive increase in SYS CPU load.

The problem demonstrated using integration tests and “time”

On the same TEST system as above, I run my integration tests on Node 18:

$ node --version
v18.13.0
$ time ./tools/local.sh integrationtest ALL -v | tail -n 1
Bad:0 Void:0 Skipped:8 Good:1543 (1551)

real 0m27.277s
user 0m17.751s
sys 0m4.251s

Changing to Node 20.10.0 instead gives:

$ node --version
v20.10.0
$ time ./tools/local.sh integrationtest ALL -v | tail -n 1
Bad:0 Void:0 Skipped:8 Good:1542 (1551)

real	0m56.958s
user	0m12.875s
sys	0m36.931s

As you can see, SYS CPU load increased dramatically.

Affected Node versions

There is never a problem with Node.js 18 or lower.

Current Node.js 20.10.0 shows the problem (on some hosts).

My tests (on one particular host) indicate that the excessive SYS CPU load was introduced with Node.js 20.3.1. The problem is still there with Node 21.

There is an interesting open Github issue.

Affected hosts

I can reproduce the problem on some computers with some configurations. Successful reproduction means that Node 18 runs fine and Node 20.10.0 runs with excessive SYS CPU load.

Hosts where problem is reproduced (Node 20 runs with excessive SYS CPU load)

  1. Raspberry Pi 2, Raspbian 12.1
  2. Intel NUC i5 4250U, Debian 12.1
  3. Cloud VPS, Glesys.com, System container VPS, x64, Debian 11.8

Host where problem is not reproduced (Node 20 runs just fine)

  1. Apple M1 Pro, macOS
  2. Dell XPS, 8th gen i7, Windows 11
  3. Raspberry Pi 2, Raspbian 11.8
  4. QNAP Container Station LXD, Celeron J1900, Debian 11.8
  5. QNAP Container Station LXD, Celeron J1900, Debian 12.4

Comments to this

On the RPi, upgrading from 11.8 to 12.1 activated the problem.
On QNAP LXD, both 11.8 and 12.4 do not show the problem.

Thus we have Debian 11.8 hosts that exhibit both behaviours, and we have Debian 12 hosts that exhibit both behaviours.

Conclusion

This problem seems quite serious.

It affects recent versions of Debian in combination with Node 20+.

I have seen no problems on macOS or Windows.

I have tested no other Linux distributions than Debian (Raspbian).

Solution

It it seems this is a kernel bug with io_uring, at least according to Node.js/Libuv people. That is consistent with my findings above about affected machines.

There is a workaround for Node.js:

UV_USE_IO_URING=0

It appears to be intentionally undocumented, which I interpret as it will be removed from Node.js when no common kernels are affected.

I will stay away from Node.js 20, at least in prodution, for a year and see how this develops.

Elemental Dice – Cerium Problem

After a few months in a box in a close, it is obvious that Cerium has a problem.

I have received a replacement Cerium die, with cerium in resin.

Debian 12 on a 2-drive NUC

After my relative success with Debian 12 on my Hades Canyon I decided to install Debian 12 on an older NUC as well, the NUC D54250WYKH with an i5-4250U. The nice thing with this NUC is that it both has an mSATA slot and room for a good old 2.5-inch drive. So I have:

  • 1 TB 5400 rpm HDD
  • 240 GB SSD

The annoying thing is that the BIOS/UEFI only wants to boot from the SATA drive, and the SATA drive shows up first in Linux. The easy way for me to install this computer would be

  • 240 GB SSD: /, /boot, /swap, /home
  • 1000 GB HDD: /home/sync (for syncthing data)

I could do a simple guided-encrypted-lvm-all-drive on the 240 GB, and a single encrypted partition on 1TB. But Debian 12 installation fails when it comes to installing GRUB, and the installed system does not boot.

Using LVM to make a logical volume spanning a small fast SSD and a large slow HDD makes no sense.

Partitioning in Debian

There is a guided option and a manual option to do Partitioning in Debian. I feel neither is good for me.

  • Guided: fails to lay out things easily on the two drives in a way that works
  • Manual: honestly, too complicated, particularly:
    • LVM and encryption hide few details, requires many steps, and hard to undo half-way
    • I understood that LILO needed to go the beginning of the drive BIOS was set to boot, and that LILO needed to see /boot (whether its own partition or root). However, with GRUB and UEFI, there are two separate extra partitions (/boot and some FAT-partition I think) and I am not allowed to control where the GRUB code goes (if anywhere). So I do not dare to set up this manually.

To make things worse (admittedly, I used the minimal.iso Debian installer which pulls things over the network to make things slower), when restarting the computer/installer there are quite many steps until my drives are detected and I can even erase partition tables and start over.

What I did

After two failed installation attempts, and several more restarts of the installer, I found a working solution.

I first erased all traces of partitions and boot code on both drives to be on the safe side. /dev/sda is the installation media.

  1. # dd if=/dev/zero of=/dev/sdb bs=1024 count=10240
  2. # dd if=/dev/zero of=/dev/sdc bs=1024 count=10240
  3. Guided non encrypted setup of 1000 TB drive, with separate /home
  4. I didn’t even install X/Gnome this time to save time

This gave me working computer that makes no use of my SSD. As root on the console I did:

  1. Backup the home directory of my non-root-user (just in case) to /root
  2. Remove /home from fstab
  3. Restart
  4. install cryptsetup and cryptsetup-run
  5. encrypt /dev/sda4 using cryptsetup (900GB+ HDD partition)
  6. encrypt /dev/sdb1 using cryptsetup (240GB SSD only partition)
  7. add entries to /etc/crypttab:
    sda4enc /dev/sda4
    sdb1enc /dev/sdb1
  8. Restart
  9. Give master encryption password (just once since I used the same)
  10. mkfs.ext4 /dev/mapper/sda4enc
  11. mkfs.ext4 /dev/mapper/sdb1enc
  12. add entries to /etc/fstab
    /dev/sdb1enc /home +options
    /dev/sda4enc /home/sync +options
  13. Restart

The result is almost 100% good. A few comments:

  • swap ended up on slow 1TB HDD, which I am fine with since I have 16GB RAM
  • root filesystem (with /usr, /root, /var, /etc and more) is not encrypted now, but I can live with having only my data (/home, /home/sync) encrypted
  • using cryptsetup/luks directly on partitions, not bothering with LVM, is much more simple
  • with /etc/crypttab and cryptsetup-run, encryption is really simple and understandable

As long as I do not run into something strange with X/Wayland/Gnome and drivers for this old NUC, I think I am good now.

What I would have wanted

I hear people have been fearing the Debian installer, up to Debian 12. I have not feared it in the past, but now I kind of do (after having issues installing two different NUCs the same week).

This is the partitioning experience I would have liked. My input/selections as [ ].

You have three drives with multiple partitions. Select all you want to keep, use as is, or delete:

/dev/sda (Debian installation media)
[KEEP] /dev/sda1 ...
[KEEP] /dev/sda2 ...
[KEEP] /dev/sda3 ...

/dev/sdb (1000 GB HITACHI)
[DELETE] /dev/sdb1  200 GB NTFS
[DELETE] /dev/sdb2  750 GB ext4
[/mnt/backup] /dev/sdb3 50 GB (just an example of something to keep

/dev/sdc (240 GB SAMSUNG)
[DELETE] /dev/sdc1  500MB FAT
[DELETE] /dev/sdc2  400MB ext2
[DELETE] /dev/sdc3  239GB ext4

With that out of the way, I would like Debian to ask me:

What device should contain 2 small partitions for boot purposes?
[X] /dev/sdb  -- 950 GB free
[ ] /dev/sdc  -- 240 GB free

Where do you want swap partitions, and what size?
[      ] /dev/sdb -- 950 GB free
[ 16GB ] /dev/sdc -- 240 GB free

Where do you want /, and what size
[      ] /dev/sdb -- 950 GB free
[ 30GB ] /dev/scd -- 224 GB free

Do you want a separate /home, and what size
[       ] /dev/sdb -- 950 GB free
[ 194GB ] /dev/scd -- 194 GB free

Do you want a separate /var, and what size
[       ] /dev/sdb -- 950 GB free
[       ] /dev/scd --   0 GB free

Do you want to set up extra non-standard mounts?
[ 950GB ] [ /home/sync ] /dev/sdb -- 950 GB free

Now it is time to choose encryption and format options:

UEFI-BOOT    500MB   [ FAT ]
/boot        500MB   [ ext2 ]
/             30GB   [ ext4 + encrypt ]
/home        194GB   [ ext4 + encrypt ]
/home/sync   950GB   [ ext4 + encrypt ]
/mnt/backup   50GB   [ KEEP ]

Finally, choose encryption password (the same, or separate).

This would have been a much better experience for me. I understand there can be more cases:

  • Computers with multiple disks may want to use LVM for to make logical volumes spanning several physical volumes. That would probably be a question between (1) and (2) above.
  • Multiple filesystems could live on a common encrypted volume, with a common encryption key, making use of LVM. That could be a question in the end:
    /usr and /var are on the same disk, do you want them to share encryption key on a common volume

Summary

I would guess that the use cases are:

  • 80% Simple 1-drive computers (Guided, automatic, defaults)
  • 10% Multi-disk servers with specific requirements (Manual, expert mode)
  • 10% 2-3 drive computers (not well supported today with Debian 12)

I am just making 80/10/10 up, of course. The unsupported 10% can be made up of:

  • Laptops or desktops that come with a small SSD and a large HDD (it happens)
  • Desktop computers with extra drives installed
  • Simple servers

Perhaps in Debian 13!

Debian 12 on Hades Canyon NUC

I have a Hades Canyon NUC (NUC8i7HVK) that I have been running Ubuntu and later Fedora on. Ubuntu has been fine for years but I didn’t want Snap (especially not for Firefox) so I tried out Fedora and that was also fine.

I realize that I did not leave Ubuntu because I did not want to have Snap, I left it because I want 100% apt. So in the long run I feel a bit alienated with Fedora and with Debian 12 out and getting good reviews I thought about giving it a try.

This desktop computer is a bit like your typical laptop when it comes to Linux, not sure everything works out of the box. I used to struggle a bit with Bluetooth and Audio, but I don’t do those things on this machine anymore. Ubuntu and Fedora are kind of already configured with proprietary non-free drivers for this NUC, but Debian is not.

TLDR

I am running Debian 12 now, installed from the “minimal.iso” debian image, and with a number of extra packages installed. The InstallingDebianOn-page for this machine is ok. All I actually did was to add non-free and contrib to sources.list and install the extra packages recommended:

I have done no extra configuration or tweaking on Debian 12, but I am not using Audio-IN, Bluetooth or Wifi so I have not tested.

Broken Live Image

I didn’t throw Fedora 38 out without doing some testing first, so I downloaded the Live image for Debian 12 and successfully tried it. Then I installed Debian 12 from the Live image (choosing install immediately at the Grub menu), which was 99% successful. But it left some Raspberry-Pi packages and some stuff in /boot, resulting in that apt could not finish rebuilding the ramdisk. Computer started, but error remained. I searched on forums, it is a known problem with the Live image, there are solutions and when I tried I just got more errors. So I ended up reinstalling Debian 12 from scratch.

minimal.iso

I downloaded the minimal.iso, convenient so I did not have to use a large USB-key, and installed from it. What a nice text/curses based installation! Then I got a non booting system!

I had to disable “Intel IGD” (I think that was how it was called) in “BIOS” (it is not BIOS anymore), becuase this machine has an Intel GPU that is not connected to any output, and with this rudimentary Debian install, somehow the system would not start.

When that was done, and I started Debian and logged in, Gnome (and neofetch I presume) reported GPU=Software. I could watch Youtube with high CPU load. That was when I installed the extra packages listed above, and since then I have been happy.

Conclusion

Debian 12 is fine on Hades Canyon NUC8i7HVK. The InstallingDebianOn-page linked above tells you more than you need. It was written from Debian 10.7.

Trying tmux

It seems screen is old and tmux is what I should use. Here are some findings and notes.

Cheat Sheet

I found a decent Cheat Sheet.

macOS backspace issue

There seems to be a problem with backspace in tmux on macOS. I installed tmux via pkgin, so if you use brew or something, perhaps the situation is different.

The simple fix I found here was to create a ~/.tmux.conf and add one line:

set -g default-terminal "screen"

or

set -g default-terminal "screen-256color"

Other solutions fixing tmux-256color with infocmp and tmp failed for me. I probably just didn’t use the right versions of the commands in the right way.

macOS resizing panes

As I understand it, panes are resized with CTRL+B+ArrowKey. But CTRL+ArrowKey does something else on macOS. I have not decided if I need to solve this yet.

Scrolling

Scrolling was always a hassle in screen. In tmux, this is a nobrainer for me (again ~/.tmux.conf):

set -g mouse on

On RHEL and downstream clones

I have been using Linux, being fascinated with Linux, since 1997. It makes me sad to see the current situation with RHEL, Alma and Rocky.

I have since long been a user of Debian and different versions of Ubuntu. Recently I have switched to Fedora on my workstations because I don’t appreciate Snap in Ubuntu.

I think Linux, how it is delivered, compared to Windows, has two advantages (apart from price):

  • Everyone can use the same version of Linux (I don’t have arbitrary limitations on my Home computer compared to my Professional computer, or my Server computer)
  • Anyone can make their own flavour (with KDE, for Gaming, for sound engineers, for servers, without systemd, for network routers and firewalls)

To me, this is about economy. Not purchase price, but about not doing the same work over and over again, on different computers, in different projects, or in different organisations. This is about maximising synergy, and minimising waste.

RHEL

RHEL is, from my perspective, about

  • Not everyone can use the same version of Linux (because RHEL is dominant but not for everyone)
  • Since last weeks, nobody should make their own flavours of RHEL

I understand it makes sense from a corporate perspective, but it makes less sense from a holistic Linux perspective. But this was kind of true for RHEL even before last weeks shutting off patches downstream.

To me, RHEL is less free, in lack of a better word. I can have it for 0 USD, I can get the source under GPL, but it still comes with strings attached that I rather don’t have.

Alma and Rocky

I have occasionally logged in to a RHEL computer but I have never done anything with Alma or Rocky. I understand if you technically want RHEL but you do not want a relationship with Red Hat, Alma or Rocky solves that. And perhaps RHEL (or Alma or Rocky) is more fit-for-purpose for you than any alternative (like Debian or Ubuntu).

I always refused to use pirated Windows because I argued that even if I pay Microsoft nothing, I am still supporting their entire ecosystem, not helping things to get better. To me, Alma and Rocky are not pirated versions of RHEL (of course not). But to me, they also do not contribute to making RHEL or any other Linux system better. And they do not make the REAL alternatives to RHEL any more viable, while supporting the RHEL ecosystem. They are just community effort to duplicate work, and from my perspective that effort could have been used for something better (like Debian – if you want free Linux).

Fedora -> CentOS Stream -> RHEL -> Clones

I kind of agree with the Red Hat position, that supporting Fedora and CentOS Stream, upstream, is their best way of serving the community. And that the clones themselves add nothing.

To me Fedora and CentOS Stream makes more sense and have more appeal, than Alma and Rocky. But I don’t need to run some enterprise applications so perhaps I do not understand.

Red Hat business model

As I understand it (and I just run Debian on my servers, so I may not know) Canonical has free download available for all versions of Ubuntu (also enterprise server versions that compete with RHEL). But you can pay for support if you want.

If Red Hat did the same, Alma and Rocky would disappear. Or they would turn into niche variants/remixes of RHEL. I have seen other places in the open source world where you need to pay for extended support, which seems to be what RHEL and the cost of RHEL is much about.

I read that Red Hat realised that customers had 1 paid RHEL computer, and 999 CentOS computers, and the support was always for the RHEL computer. That was why Red Hat moved CentOS upstream. Perhaps that was the wrong move to increase customer RHEL support loyalty, and perhaps this late move of Red Hat is also the wrong move for the same old problem.

Conclusion

Alma and Rocky exist only because Red Hat and RHEL comes with strings attached that many people do not want in the Linux world. However, there were still strings, and now Red Hat pulls them.

There are only two good solutions:

  1. Red Hat understands the real need for no strings attached
  2. People understand to move away from RHEL entirely, and truly support the real alternatives

I hope for any of these. Not for a RHEL-Alma-Rocky conflict situation.