Designing AI When "Almost Right" Isn't Good Enough

When I first looked at the TruDi Navigation System recall, I saw it mainly as a software accuracy problem. The system supports navigation during ear, nose, throat, and skull-base surgeries by showing where surgical instruments are positioned relative to a patient's anatomy.

In 2023, the FDA recall database posted a Class 2 device recall record involving software that could display an instrument tip in a different location from its actual position. In this context, even a small error can create serious risk.

Case reference: FDA recall database, Class 2 Device Recall TruDi Navigation System, posted October 20, 2023.

The more I studied the case, the more I realized that the real issue went beyond accuracy. It was also about trust, human oversight, and responsibility.

One assumption I had was that keeping a surgeon in control would make the system safer. If the technology is wrong, the professional can recognize the problem and override it.

But this is not always realistic. A surgeon may be relying on the navigation system specifically because the information is difficult to verify directly. If the interface presents a precise, confident result with no indication of uncertainty, the user may have little reason to question it.

This is where automation bias becomes important. Human oversight is only useful when the human can detect a problem, understand its severity, and act in time.

AI interfaces often present outputs as if every result is equally reliable. But uncertainty is part of AI, and the interface has to make that uncertainty usable.

For high-risk systems, the product should help users understand when confidence is low, when context has changed, or when the result needs additional verification.

• Confidence indicators that are visible before action is taken.
• Alerts when tracking quality, data quality, or context quality drops.
• Independent verification steps for high-impact decisions.
• Safer fallback states when the system cannot support a confident answer.

Another insight from this case was accountability. At first, I assumed the manufacturer should carry most of the responsibility. But AI-enabled products operate within a larger ecosystem involving software teams, regulators, organizations, and professional users.

Everyone plays a role. However, when responsibility is spread across many groups, it can become easy for each group to assume someone else is managing the risk.

For product teams, accountability should be defined before failure happens:

• Who monitors the system after launch?
• Who responds to safety signals?
• Who communicates limitations to users?
• Who decides when the product should no longer be trusted?

This case expanded the questions I bring into AI product design. I still care about usability and accuracy, but I now also look for the conditions around the model: risk, uncertainty, escalation, monitoring, and ownership.

• What happens when the system fails?
• Can users recognize uncertainty?
• Are we depending too heavily on human oversight?
• What safeguards exist outside the AI itself?
• Who owns the problem after launch?

These questions connect design with safety, trust, and system behavior. They also make design work more cross-functional. A trustworthy AI experience depends on what users see, what teams monitor, what the product does when confidence drops, and what happens after launch.

I initially saw the TruDi incident as a single software malfunction, but came to understand it as a broader system-level problem. My original focus was accuracy, but the case pushed me to think more carefully about human reliance, interface confidence, validation, and accountability.

The biggest shift in my thinking was around human oversight. I used to assume that keeping an experienced professional in control would provide a strong safeguard. I now understand that this protection can be limited when the user depends on the system for information that is difficult to verify independently.

I chose to turn this analysis into a portfolio article because it represents an important step in my AI learning journey. It shows how I am moving beyond understanding how AI works and beginning to think about how AI behaves inside real products and real decision-making environments.