Why Embedded Software Development is Harder

Developers sometimes distinguish "embedded" as a special kind of software development. Maybe different than categories like "web" or "apps". Is it really? In the end, writing software looks very much the same.

There are aspects which have significant effect on the development process but it makes sense to differentiate six aspects.

External Hardware

An obvious part of developing software for custom hardware is that it requires custom hardware which is outside of your computer. That means code must be transported to the device (called flashing) and information about the behavior must flow back for debugging.

How does this impact software development? At least the initial setup requires connecting cables. Usually it involves additional complications. Where an app developer can run the app or tests with the press of a button, it usually cannot be fully automated with custom hardware and sometime physical influences, like hot weather, disturb the setup.

Distributed Systems

Web developers should agree that developing a distributed system is a special challenge. For example, time is not the same for all parts of the system and relations like "before" and "after" are not always available. Losing such basic assumptions means that trivial things become hard. For example: "Can't you just check if we have the item in stock before we confirm the customers order?"

In embedded systems you often deal with multiple devices, multiple chips, and in times of Internet of Things these can be all over the globe.

How does this impact software development? For example, you cannot assume that every running software has the same version. Then code needs to be forwards and backwards compatible. Fewer assumptions to rely on means more logic is necessary to handle edge cases.

Resource Constraints

Resource constraints hit every program sooner or later. However, most programmers use gigabytes of memory and more. If you are working on a microcontroller memory is measured in Kilobytes, so you have to save a few bytes here and there. Now resource efficiency is a daily challenge, not just for certain hot spots.

How does this impact software development? Where an app developer wouldn't think twice about introducing an abstraction layer, you consider the resource impact for your tiny microcontroller. When additional memory is absolutely necessary, you have discussions which other part of the system could be optimized or stripped down. It creates additional coordination efforts.

Real-Time

While web developers also use the term "real-time" it means something different in the embedded world: The code needs to execute fast enough to hit certain deadlines. Soft real-time means that it is acceptable to miss the deadline occasionally. For example, Google tries to answer search request within a second and a game engine tries to render 60 frames per second. To achieve this, developers are limited in algorithm choice. For example, garbage collection is convenient but without deterministic runtimes the performance is too bad in many scenarios.

Yet another level is hard real-time where even a single missed deadline can result in destruction and death. Here developers use special analysis tools to ensure timings and no shortcuts are acceptable.

How does this impact software development? No garbage collector. No heap. No standard library. At least not the usual ones, maybe you have internal proprietary stuff. Developers try to use only simple algorithms because simple often means deterministic runtime. While simple is generally considered good, when you push it to an extreme the code becomes verbose and inefficient. For example, converting an integer to an enum type must not be done with a cast but with a switch statement.

Speed improvements often don't matter. In the non-real-time world, if you optimize the core algorithm be a faster everybody is happy. In real-time world, nobody cares because we hit our deadline before and nothing changed about that. Only when a deadline miss occurs, optimization heroes are suddenly requested.

HW-SW-Co-design

Embedded implies that the hardware is somewhat unique. While you can write portable software, project management still has to accommodate with hardware development steps. At the beginning of the project, the actual hardware might be unavailable. Then you have samples but not enough for every developer and a list of known bugs which they will only fix shortly before the final release is due.

How does this impact software development? You cannot host your proprietary hardware in the cloud. A continuous integration system to run tests requires you to maintain a lab with your custom devices. Since those custom devices are in short supply, you cannot scale testing easily.

Hardware and software design work on different time scales (like weeks vs years). This leads to a chicken-and-egg problem with respect to optimizations: Why optimize the software if it is fine for the current hardware generation? Meanwhile the next hardware generation will be planned with the current software needs. Optimizations only matter when you try to cram in yet another feature but at that moment you want to deliver the feature quickly and not optimize legacy code. So there is never a suitable time to optimize.

Safety-, Security-, or Mission-Critical

Some software is used in context where people's lives are at stake. Safety-critical means people could die due to software errors. Security-critical means people could go to jail due to software errors. Mission-critical means basic necessities get broken due to software errors.

When software quality is important, it gets turned into metrics, and becomes bureaucratic. In big industries where large parts of the development are outsourced to suppliers, there are standards and regulations. For example, MISRA forbids dead code. Of course, there are static analysis to check but they are never perfect.

How does this impact software development? On the positive side, you do not need to convince anybody that unit tests are good. On the negative side, 100% test coverage is a must and can be annoying. It does not take long for a software developer to get into conflict with rules. Common sense is a weak argument against regulations.

Alltogether

Each of those aspects will complicate software development and lower productivity:

When combined the effect stacks of course. Maybe it even multiplies for exponential impact. So if you like a challenge, look for work where all of this applies, like automated driving for example. However, be aware that those challenges are not technical to a large degree. You have to deal with people, processes, and organizations instead.


Thanks to Manfred Kröhnert to suggest the debugging aspect.

Discussion on lobste.rs.

External hardware, distributed systems, resource constraints, realtime, hw-sw-codesign, and safety-criticality make it hard.