The Therac-25 disaster serves as a poignant reminder of the vital importance of design controls and verifications in medical device development. I realize that modern frameworks (e.g., IEC 62304 and ISO 13485) have greatly enhanced oversight and traceability, but I do not think they have eliminated the risk of another "Therac" incident from occurring, especially since AI-driven and adaptive algorithms are becoming more common in medical devices. 

Current design control systems have all been developed from deterministic software, where behavior is predictable given inputs. AI systems, especially those utilized for machine learning, are adaptive and can behave predictably based on new data inputs. Thus, this can render traditional verification methods (e.g., static validation/predefined test cases) ineffective, as it is not straightforward to fully verify an algorithm that continuously learns from clinical data. 

Regulatory bodies are certainly trying to keep up. The FDA's proposed framework for AI/ML medical devices suggests a "total product lifecycle" approach, where continuous monitoring, real-world performance data, and protocols for algorithmic changes are required. This essentially shifts design controls from a static to a dynamic process, while introducing complexity and uncertainty regarding accountability in the event of failure. 

Do you think existing standards (IEC 62304) can evolve to meet the needs of adaptive AI behavior, or should we design a completely new regulatory paradigm to handle the risks of AI-driven devices?

I agree that the Therac-25 incident changed how we think about design controls; however, I think that the next "Therac moment” might come from overconfidence in verification rather than poor verification. ISO 13485 and IEC 62304 have made verification extremely structured and thorough, but this structure can sometimes lead to overconfidence since people may think that the structure is enough to account for all the nooks and crannies of verification. This is a false sense of security, and when everything is documented and checked off according to the structure, teams might assume that safety is guaranteed. 

Verification ensures that a device meets defined requirements, but those requirements themselves might be too narrowly scoped. With the advent of AI, as Krish has mentioned, these requirements can be changed and shifted in order to meet the specifics of each device. This would improve verification greatly. However, with no human oversight in a process like this, disastrous outcomes can result. Thus, I think it is important to slowly incorporate AI with human oversight to ensure verification is meeting the general requirements and other niche requirements as well to protect the public. With only AI and the current guidelines, with no critical human thinking involved, verification would still be flawed, and another incident could occur. 

Dr. Simon’s lecture emphasized traceability, but traceability alone does not equal safety. With an AI-driven system, one device’s verified output might become another’s unverified input. These unpredictable interactions can still lead to catastrophic outcomes with miscommunication between systems. Human oversight can help with this, but AI should also have inter-device verification with systems working in combination and not isolation. This could be done through simulated hospital networks or “stress-testing” devices under abnormal use conditions instead of just verifying regular conditions. AI can help with this, but once again, human oversight would be needed. 

To answer Krish’s question, I think current regulatory systems should still be around, but new ones should be created as an add-on to work with AI. These new systems should have human inputs to complement the AI, and many systems should be working together. Do you think that companies should be required to have interoperability testing as part of verification since medical devices are connected to one another? Or would this make development extremely slow since many systems interact together?

I think our current design control systems are much stronger, but not entirely error-proof. The Therac-25 tragedy definitely pushed the industry toward stricter verification, documentation, and risk management standards, which have made a huge difference. However, with AI and increasingly complex algorithms being integrated into devices, we’re entering new territory where traditional verification methods might not always apply. For example, machine learning systems can change behavior over time, which makes validation a moving target. Regulations like ISO 13485 and IEC 62304 give a solid framework, but they still rely heavily on how rigorously companies apply them in practice. So while we’re far better prepared today, I wouldn’t say a modern “Therac moment” is impossible, it is just much less likely if teams stay diligent and transparent.

Similarly, as everyone has mentioned, I also believe that this incident (Therac-25) caused a fundamental rethinking and reshaping of how the industry approaches both software safety and design. With that said, it isn't only just verification that is changing, but what exactly is being verified. In the past, deterministic code was being verified, but now, engineers (among other professions) are verifying behavior that can evolve and adapt, making validation a moving target. Our current design controls systems (such as the ISO 13485 and IEC 62304) that are in place provide a solid foundation that enforce and encourage traceability, documentation and discipline that wasn't entirely emplaced during the 1980s. As devices continued to get advanced development, so too must the SOPs and protocols. AI systems and machine learning need to have design controls that are not just a one time checklist, but more so a continuous feedback loop. I also agree with a previous sentiment that structure provided alone can create a false confidence as well. I was always of the opinion that complacency kills, and that can be true in a lot of different professions, but can be especially applicable here. I think another tragedy, such as the Therac incident, will be caused not because of a lack of standards, but because we treat them as the end all be all, rather than adapting and evolving them.

While many have raised valid points about adaptive algorithms and overconfidence in structured systems, I think a modern “Therac moment” could result less from technical gaps and more from ethical and organizational blind spots. Current design controls ensure functional safety, but they don’t really address decision accountability when multiple AI systems/ teams share responsibility.

Imagine a clinical AI that flags an anomaly, but another system overrides it, or a technician trusts the algorithm despite visible inconsistencies. Even if both devices are verified, the handoff of trust between humans and AI remains a vulnerable point. Verification can’t always capture how people interpret and act on automated outputs.

To truly prevent tragedy, we may need verification frameworks that include human factors engineering, ethics review, and cross-system communication testing - not just algorithmic validation. Should verification processes include mandatory human-in-the-loop simulations before device approval to ensure responsible use in real settings?

What contributed to the Therac-25 incidents was a reliance on old code as mentioned in previous posts. Engineers possibly assumed their code from previous projects was fine as is. With AI entering medical devices now, what is stopping engineers from thinking their AI systems are fine as is? Verification processes should definitely include human review, but then an issue arises with how these reviews would be conducted. With Therac, old code was assumed to be fine, so human reviews could take shortcuts based on the performance of an AI in past projects. The review can also find an AI to be "smart enough", which will eventually lead to another Therac moment. AI systems are not so straight-forward as code, which would require having a team with knowledge on how the AI is built. Individuals tasked with performing these verification processes may fall into the same trap which existed with the Therac engineers; assuming something is fine before it becomes not fine. Existing standards do need to be updated as AI becomes more involved in medical devices. Continuous monitoring and challenging the output of AI are necessary to ensure the safety of medical devices in the future.

AI driven and adaptive algorithms in medical devices introduce a new layer of uncertainty. Unlike deterministic systems, machine learning models can behave unpredictably outside trained data sets. Verification methods for such systems are still evolving, and traditional validation approaches may not capture edge case failures. To prevent another “Therac moment,” we need model transparency, traceable data provenance, and continuous post market performance monitoring. This is emphasized in the FDA’s proposed guidance on AI/ML based SaMD (Software as a Medical Device).

While modern design control systems like ISO 13485, IEC 62304, and FDA 21 CFR 820 are far stronger than those in the Therac-25 era—mandating rigorous verification, risk management, and traceability—they are not yet fully equipped to handle the complexities of AI-driven and adaptive medical devices. Traditional controls work well for deterministic software, but artificial intelligence introduces non-deterministic behavior, data drift, and limited explainability that can undermine conventional verification methods. Emerging frameworks such as the FDA’s Good Machine Learning Practice and the AI/ML SaMD Pre-Cert program are important steps toward continuous validation and data governance, yet regulatory science is still catching up. In essence, today’s systems are robust against another “Therac-25” rooted in coding negligence, but a “Therac moment” of the AI age could still occur if evolving algorithms are not matched by equally adaptive oversight and transparent design controls.