Why in-car computer systems are so bad - Part 1

Do the people making these things even know what they are doing? Maybe not.

Introduction

Back in 2019, a large car manufacturer in Germany wanted to send lots of data over MQTT to a so-called “data warehouse” to train future self-driving work and for general business intelligence stuff. We were a team responsible for the system receiving and normalising all the data coming from each car.

Senseless bickering

Every 2 weeks, around 100 people sat around a speaker barely able to hear each other over a poor phone line. It was getting into summer and buildings in the Netherlands were never designed for the summer heatwaves the country experiences nowadays.

Eventually it was time to hear updates from the so-called “admin” team. This week they had trouble updating version numbers of all the Scala libraries and microservices. One of the dev teams were not communicating the updates to the admin team; version numbers in the HTML table in the wiki were stale. Some higher manager was wondering what was going on.

Fresh from a scrappy old-school IT services company of Linux enthusiasts I could not believe what I was hearing. I was so used to working overtime just to keep customers happy. In the chaos of it all, there was never any time to complain to anybody about not following procedures. Automation was a necessity to save me time and sanity.

The opportunity

Scripting the creation of a HTML table: what a juicy opportunity to take my attention away from the senseless bickering! The admin team had technical writers whom I was desperate to recruit anyway. The release procedure of the immense distributed monolith we were taking care of was totally undocumented; maybe they could help me write it down.

All the code for the system service was written in Scala, spread out over about 100 repositories. The teams used a SBT plugin (of course they did) to manage the version number in each repo. SBT is a bit of an abomination. It’s a huge, spawling build and compilation tool. One handy feature I discovered: I could print the version of each SBT project by running something like

sbt release version

(Or something like that). To print all versions:

for d in work/*
do
	(cd $d && echo $d && sbt release version)
done

But this was far from good enough to show to others. It was dog slow: this took several minutes for 100+ repositories. And that’s only if you had prepared the repository already. If not, you would get some obscure error and need to download the gigabytes of dependencies twice (I actually fixed this for the team! Another story for another time!). SBT and Scala were such a pain to install for something as trivial as printing a number. Lastly, sbt release version output was full of colour and inconsistent line breaks, making it hard to process the output from the shell.

Like most programming projects, it was well worth it to look at the data rather than think about code. The SBT plugin managed a file whose contents looked like:

// some comment about the version
version in ThisBuild := "1.0.0"

I knew how to extract that number without burning a hole through my over-specced Macbook Pro. I wrote an awk(1) script, and called it sbtversion:

BEGIN {
	FS = ":="
}
/^( |\t)+?version/ {
	i = index($2, "\"")
	s = substr($2, i+1)
	i = index(s, "\"")
	version = substr(s, 1, i-1)
	print version
}

All this does is extract the value between the double quotes. Now, to print the version of every repository:

for d in work/*
do
	(echo $d && sbtversion $d/version.sbt)
done

This ran in a fraction of a second. A couple of lines of shell later, I had a HTML table:

<table>
	<tr>
		<td>name</td>
		<td>version</td>
	</tr>
	...
</table>

It was so fast and anyone could run it without installing anything! I was excited to help out the admin team. So I spoke to my manager and asked how I could send this file to them.

“They can’t run any software on their laptops” my manager replied. But maybe we could update that version page instead? “Our team doesn’t have permission”. Damn. “Who does?” I asked. “Let me check and get back to you”.

A few weeks later after this same manager refused to let other teams see our code repositories, so I quit.

Lessons, if any

Since then, much of the underlying software of that big wasteful project has seen major changes:

You know when you get into a new car and you try to use the entertainment system and things barely work? Bluetooth pairing with your phone doesn’t work, the interface lags so when you tap one button then another you don’t know which button it will register… It’s enough to make you wonder whether the people making these things know what they are even doing. Perhaps this little insight helps you to stop wondering.