One of the hidden themes in last year's Embedded Systems Conference was that experience matters. Whether it's avoiding an error you've made before or recovering quickly from a new problem, knowing your field in depth helps you get the job done. Whether that's a marketable skill depends on circumstances beyond your control, but at least you get some satisfaction from knowing what you're doing.
This month, I'll collect some data points to show how this works out in practice.
Missing One Little Point
The fundamental illusion of all desktop and server application development methods goes something like this: Resources are free, unlimited, and readily available. Should your code run out of something, don't worry, more is on the way. Over the decades, we've seen some truly bizarre hardware designs intended to maintain that illusion.
For example, within the lifetime of those yet living, a "personal computer" with 640 KB of memory should have been, as Bill Gates evidently didn't say, enough for anyone. Intel's 8086 architecture grafted 20 physical address bits on a 16-bit datapath, inspiring all manner of bizarre memory expansion techniques culminating in the Lotus-Intel-Microsoft Expanded Memory Specification for paged access to megabytes of agonizingly expensive RAM.
The jump to 32 address bits sufficed for a mere decade, whereupon Intel's Physical Address Extension (another four bits!) gave paged access to 64 GB of physical memory. The jump to 64-bit microcontrollers, with current implementations limited to addresses of a mere 40 or 48 bits, should last longer.
On the other hand, we're chewing through two address bits per year and 2020 isn't all that far away.
Embedded development differs slightly: You don't have enough of at least one resource and you won't get any more in time to finish the job. In the bad old days you ran out of memory or datapath, nowadays there's not enough connectivity or power (which is identical to CPU speed). Sometimes, you have enough of everything, but it doesn't work quite the way you expected or has limits that your tools simply don't notice.
Some years ago, a friend called for some advice on a power-line data logger he'd built to track down utility-power glitches. It wrote a running record of voltage disturbances into a nonvolatile EEPROM, but after a week or two the memory failed. In fact, several different memory chips had failed the same way.
The EEPROM datasheet specified an endurance of 100,000 write cycles, which seemed like enough for years of recording at the expected rate. Alas, it transpired his toolchain had inadvertently put a few "hot" variables, ones updated every few seconds, into the EEPROM's address space. The arithmetic is easy:
605 × 103 sec/week=
168 hour/week × 3600 sec/hour
Perhaps surprisingly, a variable written every six seconds chews through the EEPROM's entire rated lifetime in a week. The remainder of the EEPROM might still be usable, but those addresses were hammered dead, dead, dead. Once we found the problem the fix was trivial, but it still required manual intervention.
Contemporary flash memory includes wear-leveling routines in the on-chip controller that automagically shift repeated writes at the same address to different physical cells. That's particularly helpful for applications designed without regard for memory that expires after some number of uses.
The FAT filesystem used in essentially all USB flash drives comes to mind. All file additions, deletions, or size changes also update the File Allocation Tables at fixed locations near the start of the disk image. In the absence of wear leveling, those disk sectors can fail very quickly on an active drive.
Small embedded systems often include flash memory rather than a hard disk, trading reduced storage capacity for increased mechanical durability. Although the memory can be connected directly to the CPU for use with a flash-friendly filesystem, the trend seems to be toward hardwired disk emulation through standard PC-oid interfaces.
In fact, you might well be able to plug in a stock IDE hard drive during development and replace it with a flash drive sporting an IDE interface for deployment, all without changing a single line of code. What's not to like?
Several ESC presenters described a common interaction between off-the-shelf hardware, standard operating systems, and developers assigned to their first embedded project. It seems that system event logs, so familiar in desktop and server environments and so vital during development, sometimes find their way into the finished product. That might be handy for debugging, but tends to fill up small flash drives while wearing out the underlying memory cells.
Oops.
The trick, of course, involves turning off the log-to-disk daemon or service before releasing the project. Should you forget, you'll certainly get a reminder!