This article was generated with the assistance of AI to help our volunteer team share valuable insights efficiently. While we've reviewed the content for accuracy, please let us know if you spot any errors or have additional insights to contribute!
Missed the talk? No stress!
Missed our recent session with Andrew on the hidden risks of software testing and automated decision-making? This is one worth catching up on. Drawing on his own experience with a closed-loop insulin pump, Andrew showed how software can appear to be working exactly as designed while still putting a user in serious danger.
Andrew, a senior software engineering trainer and university lecturer, shared what happened after moving from manually managing his diabetes to a system that used sensor data, an insulin pump and an algorithm to adjust insulin automatically.
At first, the results looked promising. The system appeared to learn his patterns and keep his blood glucose within the desired range. But things quickly changed. The app showed insulin being delivered while the pump itself reported otherwise, and Andrew's glucose levels climbed dangerously high. After eventually intervening manually, he later experienced the opposite problem: the system had seemingly learnt from those previous events and delivered too much insulin, resulting in a severe low.
The experience led to the central message of the talk:
Your tests may be passing, but that does not mean your users are safe.
Andrew argued that this problem extends far beyond medical technology. Financial platforms, social networks, e-commerce systems and AI-powered products can all behave correctly in isolated scenarios while producing dangerous outcomes when several edge cases collide.
Key Takeaways:
Test the edge cases between the edge cases.
Individual failure scenarios may be handled correctly, but combinations of them can create completely new problems. Andrew recommends using an interaction matrix to identify where unusual conditions could overlap.Test for impact, not just implementation.
Unit tests, integration tests, coverage targets and green CI pipelines tell you whether the software behaves as expected under the scenarios you have considered. They do not prove that the resulting behaviour is safe for a real user.Define the things your system must never do.
Teams should explicitly document critical invariants such as "never charge a customer twice" or, in a medical context, "never deliver insulin when blood glucose is dangerously low". These should have dedicated tests that fail loudly when violated.AI and machine learning make this more important, not less.
A system can genuinely learn and adapt while learning the wrong thing. Teams need to test decision boundaries, monitor how models behave as conditions change and avoid assuming that greater automation automatically means greater reliability.Software engineers need to understand the people affected by their decisions.
Andrew challenged teams to go beyond feature requirements and ask a harder question: would you trust the system with your own health, money or freedom?
The session also explored practical ways teams can improve, including property-based testing, chaos engineering for business rules, observability around unusual rule interactions, controlled feature-flag roll-outs and better use of threat modelling.
Perhaps the most important shift is one of mindset. As software moves from simply processing information towards making increasingly consequential decisions, the way we test it has to evolve too.
Want to hear Andrew's full story, see the examples and dive into the discussion around AI, testing, responsibility and the future of software engineering? Watch the full session here
Join our Discord for discussion and updates, and subscribe to our monthly newsletter to stay in the loop.
Keen to contribute to the FEDSA community? Sign up here.
Otherwise, join our free Discord community to chat to other members!