By Robin Roberts, Director of Health IT Regulatory Affairs, PointClickCare
LinkedIn: PointClickCare
Healthcare AI is fast and powerful . . . but can it be trusted?
Like most new technologies, AI is expanding faster than the ability to regulate it. There are no governing bodies, no universal policies or guidelines that apply to all situations and applications where it might be used. It is up to each individual organization and, sometimes, to each clinician to decide how to use it.
That makes AI the exception in healthcare, where most patient-facing technology is carefully studied and tested before being put into use. Healthcare has never dealt with anything like it. And while AI’s ultimate role is yet to be determined, there is no doubt its impact will continue to grow.
It’s incumbent on healthcare organizations and those who provide AI-powered systems to make sure the technology is safe, effective, and trusted by users and patients. This foundational layer of trust needs to be reinforced through proven results and positive experiences for patients and providers.
A survey published this year found that respondents were significantly more likely to trust in and choose medical AI in scenarios with better AI performance, FDA approval, national certification, local certification, the presence of a clinician, and the use of representative data.
A 2025 AMA survey found that 68% of physicians saw value in AI tools, and 66% were using them. But nearly half (47%) cited increased oversight from medical practitioners as the most important regulatory step to build trust in AI-generated recommendations.
Defining the trust layer
What is the foundational layer of AI trust?
It’s the combination of responsible AI practices, governance structures, and transparency mechanisms that makes AI safe to deploy in high-stakes environments, like healthcare. And it’s not sufficient for AI to merely be trustworthy; it must also be viewed as such by patients and providers in order to be accepted.
The trust layer is not a function of the AI itself, whether the model is commercial or proprietary, but of the oversight architecture around it that prevents errors and misuse by people and technology. Much of what happens in AI takes place in a “black box,” the algorithms churning away, with only the results visible to the user. A well-known example is AI-powered diagnostic imaging analysis which produces a prediction without explaining why in ways a radiologist could double-check. Without insight into how the AI reached its conclusion, it’s difficult to fully trust it. The architecture should be transparent in how it prevents harm and misuse.
This is particularly critical in healthcare, where the stakes are high, and public confidence is paramount.
Cognitive offload vs. cognitive surrender
There are two fundamental approaches to using AI in healthcare: cognitive offload and cognitive surrender.
Cognitive surrender is turning tasks over entirely to agentic AI and allowing it to function unsupervised. This is common for administrative tasks such as making appointments, sending reminder texts, and ambient listening. Generally, these are not the sort of tasks that require human supervision. Because the tasks are specific and simple, and the stakes are relatively low, AI can be largely trusted in these areas.
By contrast, cognitive offload AI doesn’t replace human users but augments them by performing tasks like collecting and analyzing data, assembling it in useful forms, creating summaries and making suggestions to the humans in the loop who retain ultimate authority. Cognitive offload is more common in clinical areas where human experience and judgment are needed, and where the stakes are highest. There is room for both types in healthcare, but it is essential that organizations match them to their appropriate uses.
AI trust in skilled nursing care
For example, in skilled nursing settings, most AI use should be of the cognitive offload model.
Skilled nursing is where some of the frailest and most vulnerable patients reside, those with little margin for error in care. While skilled nursing facilities (SNFs) are often understaffed and could benefit from the efficiency AI provides, its usage can raise trust issues among patients, their families, and staff. It’s essential that any use of AI in SNFs have effective safeguards and keep humans in charge.
A lack of trust in AI isn’t only a patient safety issue; it’s a business risk. Staff adoption depends on confidence in the tools. Errors don’t just harm residents; they undermine the entire system and expose the facility to liability. As AI becomes more deeply embedded in care workflows, trustworthiness is non-negotiable.
The human element of cognitive offload AI is particularly important for such critical tasks as Minimum Data Set assessment support, fall-risk prediction and detecting signs of deterioration among patients. The trust layer requires continuous model monitoring that guards against “drift,” hallucinations, and performance degradation in these important functions. The AI tools must be designed to keep humans involved and in control throughout their use.
Also, the architecture should have checks and balances with regular validation of outputs for both internal and third-party AI models. It’s important that AI does not have sole responsibility for monitoring itself; human experts should be integrated into the validation processes.
SNFs that contract with third-party providers who use AI should ensure that their service-level agreements include quantitative and qualitative measures to maintain accuracy and reliability over time.
Investing in AI trust creates a better future
At the moment, AI governance is largely up to the individual user. Without regulations and oversight, it can be tempting to skimp on guardrails and safety precautions and not build that trust layer.
However, regulation is coming. The ONC HTI-1 rule, which took effect in 2024, established new requirements for AI with an emphasis on transparency into how the technology is trained and the data it is trained on. While the federal government isn’t approving or banning AI algorithms, it’s providing a sort of standardized transparency “nutrition label” listing things like bias information, training data description, performance metrics etc.) so buyers can evaluate the tools. In addition, CMS Medicare Advantage rules specify that AI cannot “act alone” to terminate or deny services. In the absence of overarching federal regulation, states have taken the lead, passing multiple laws in such areas as transparency, patient protection and responsible use of AI.
SNFs that work with vendors who understand the difference between cognitive offloading and cognitive surrender and who have built AI governance and guardrails into their products will be better positioned to comply with current and future regulations.
These measures will help build trust in AI tools with patients, families and staff, while protecting the facilities and patients from errors and oversights as AI assumes an even larger role in healthcare.
