A Clock That Ticks… #

Imagine a wrist watch. Beneath its panel, there are tons of gears meticulously placed in a special sequence. And under those gears, there is a motor with a battery as tiny as itself. Despite its size and complexity, this little contraption that hugs your wrist may last through your entire life.

You may wonder why and how — the answer lies in the solid, reliable engineering behind it.

So How Is It Correlated to Low Level Engineering? #

Think of your smart lamp, smart button, washing machine with Wi-Fi, and other appliances. They work tirelessly on very small amounts of electricity —obviously not the smart white appliances— and leave you with the peace of mind that you can control them anytime, everywhere, without any major crashes or faults.

If we ask the same question above, the answer is still the same.

Three Words: Fault Tolerance, Constraints, and Modularity. #

Let’s assume a small prototype board lies in front of you. It’s green, with small and big chips, lights, pins, ports, etc.

Your goal is to detect what a flower needs through humidity and temperature sensors and send the report to your phone.

Despite the simplicity of the idea, it may not be as easy as you think, because you have to consider:

  • What happens when the connection becomes unreliable?
  • What happens when the sensors stop working?
  • How do you allocate space for reports, which are not fixed in size —a report can be smaller than others if the metrics are normal?
  • What happens when a sensor sends data while the Wi-Fi chip is sending packets?
  • How do you share tight resources among requests?
  • What happens if the system needs to be scaled to multiple flowers?
  • What format should be used for reports?
  • How will the phone app handle report acquisitions, store them, and send commands to the devices?
  • What about power consumption if it runs on a battery?

All of this can be overwhelming to a developer. But sit tight — this is actually where the fun begins.

Reason in Mind, Prove in Math, Apply in Code, Test in Real Life #

First of all, do not try to take the problem as a whole and solve everything simultaneously. Instead:

  1. Divide the problem into chunks/steps.
  2. Reason about the problem — try to handle it manually in your mind and find a pattern you can exploit.
  3. Do the math for the solution you’ve proposed and see if it holds up.
  4. Solve the step with dirty code — code written very fast, without a full implementation, that works under tight assumptions.
  5. Keep track of the side effects of your dirty code.
  6. Go to the next step.
  7. Repeat until all steps are done.
  8. Measure the cons/constraints and advantages of the solution. Perhaps it has the right tradeoffs to be accepted in its context — otherwise, think of another way to approach the problem.
  9. Now tighten your code by refactoring what you’ve written to mitigate the constraints and side effects it imposed.

Ready, Set, Go! #

Initial Phase (Steps 2-7) #

  • What happens when the connection becomes unreliable?
    • Solution: Create a buffer, and hold the report until the connection is reliable enough to send. If the buffer fills up, accept the loss and drop it.
    • Cons/Constraints: Report loss.
  • What happens when the sensors stop working?
    • Solution: Send a notification that the sensors have stopped working.
    • Cons/Constraints: No logging system to see details.
  • How do you allocate space for reports?
    • Solution: Use a bump allocator.
    • Cons/Constraints: Dead zones —zones that are allocated but no longer used— and fragmentation —a phenomenon that occurs when allocation of the object cannot be done due to enough contiguous space is scattered as little chunks throughout the memory—.
  • What happens when a sensor sends data while the Wi-Fi chip is sending packets?
    • Solution: Both operations are blocking. A hard time bound will prevent blocking operations from hogging time.
    • Cons/Constraints: A hard time bound can cause timeouts for operations, leading to corrupted data sends/reads.
  • How do you share tight resources among requests?
    • Solution: Use an RTOS (Real-Time Operating System).
    • Cons/Constraints: Can be heavyweight depending on the demand it faces.
  • What happens if the system needs to be scaled to multiple flowers?
    • Solution: Create a sensors-hub architecture. Add many sensors to one hub, which also becomes the interface for managing them.
    • Cons/Constraints: More resources to manage. Single point of failure, since the hub is the only interface we use.
  • What format should be used for reports?
    • Solution: Use JSON. Simple, with parsers available for many languages.
    • Cons/Constraints: Wastes a lot of space by being human-readable. Causes more allocation and longer transmit times.
  • How will the phone app handle report acquisitions, store them, and send commands to the devices?
    • Solution: A simple local TCP server can send reports to the user’s device. The app then stores them on the phone.
    • Cons/Constraints: Local storage can bloat over time, and access to the devices is lost when the client isn’t on the same network as the devices.
  • What about power consumption if it runs on a battery?
    • Solution: Use a large Li-Po battery.
    • Cons/Constraints: Sacrifices physical space.

Build the hardware, put it in its case, write the software, flash it, observe the results, grab a coffee, and congratulate yourself regardless of whether it works or not.

Final Phase (Steps 8-9) #

  • What happens when the connection becomes unreliable?
    • New Solution: Create a buffer, and hold the report until the connection is reliable enough to send. If the buffer fills up, serialize the report and write it to disk.
    • New Cons/Constraints: Disk I/O speed and capacity become the new concerns. However, connections tend to be fixed before disk capacity fills up.
  • What happens when the sensors stop working?
    • New Solution: Send a notification that the sensors have stopped working, and send the error details to the hub.
    • New Cons/Constraints: Error-logging backpressure and disk I/O concerns again. But error logs aren’t used much over the device’s lifetime anyway.
  • How do you allocate space for reports?
    • New Solution: Use a slab allocator. Solves dead zones and drastically reduces fragmentation.
    • New Cons/Constraints: More complex logic. A hard real-time slab allocator may require more intricate math/arithmetic.
  • What happens when a sensor sends data while the Wi-Fi chip is sending packets?
    • New Solution: Detect timeouts, which prevents the device from sending malformed packets to the system.
    • New Cons/Constraints: Some data loss is inescapable.
  • How do you share tight resources among requests?
    • New Solution: Use embedded application frameworks, like Embassy, which dramatically reduce the overhead and consumption caused by an RTOS.
    • New Cons/Constraints: More verbose — it’s necessary to write more code and/or change the execution paradigm of the software.
  • What happens if the system needs to be scaled to multiple flowers?
    • Nothing needs to change. The disadvantage is real but occurs rarely enough to accept.
  • What format should be used for reports?
    • New Solution: Use Protobuf. Encoding and decoding are zero-copy.
    • New Cons/Constraints: Protobuf is a verbose serialization format that needs a schema file for the data structure, and a parser implementation may not be available in the programming language being used.
  • How will the phone app handle report acquisitions, store them, and send commands to the devices?
    • New Solution: Send and receive data and commands directly to/from the cloud, so the user can monitor or command the devices from anywhere through the client.
    • New Cons/Constraints: Cloud architecture can grow into a project of its own and must handle a lot of requests. It also requires the client and hub to authenticate, encrypt, and decrypt the data being sent and received.
  • What about power consumption if it runs on a battery?
    • New Solution: Use a smaller battery if it is possible.
    • New Cons/Constraints: Battery life can be shorter, but an optimized software stack reduces power consumption. A smaller physical footprint also means sleeker designs and lower production cost.

Finish! #

Now we have built a smart flower-monitoring system that detects the state of the flowers while you lean back on your cozy couch.

Of course, there are more questions that could be added, or different answers that handle the problems better, but don’t forget that we’re working with an example scenario here to show how astonishing a working, finished embedded ecosystem looks on the inside.

You have built a system that:

  • Heals itself when errors burst,
  • Resumes right where it left off after the lights go out,
  • Relies on math, which lets it behave deterministically,
  • Is smart enough to handle a lot of tasks, while still leaving you the flexibility and analytics to improve your quality of life,
  • Manages resources fairly and automatically, and like a boss,
  • Runs on a pinch of electricity,
  • And looks nice from outside.

Finale #

Each step was a cog, placed with the precision of a hawk’s eye; the motor and battery you assembled, the lid you placed like a veil over the crowded mechanics beneath. Now you have a wrist watch you can rely on for many years without worrying it’ll break in a foreseeable future.

And this is why I am an embedded systems engineer, and you should be too!