The Role of Automated Privacy Risk Assessment in Trustworthy Medical AI

As SYNTHEMA approaches its conclusion, Laura Carmichael from the University of Southampton reflects on how automated privacy risk assessment can support trustworthy AI-driven medical research, while reminding us that public trust depends on much more than technology alone.

Trust must be earned, not assumed

Building and maintaining ‘public trust and acceptance’ is essential to AI-driven medical research. The people, organisations and systems involved in supporting and carrying out AI driven medical research must show that they are ‘worthy of trust’. They must demonstrate trustworthiness so members of the public can feel confident that their trust is well placed. There are many ways in which this should be done, such as through ‘meaningful transparency’, ‘public participation’ and ‘establishing clear and consistent rules’ for data use.

Organisations must also put in place suitable safeguards to protect and respect the privacy of individuals, groups of people and communities whose data may be used for AI-driven medical research, or who may otherwise be affected. This requires effective processes and practices to identify, assess, mitigate, manage and document data protection and privacy risks systematically and in a structured way. When designed and applied appropriately, automated tools may support these privacy risk assessment processes and practices. In workshops with domain experts and practitioners, we have explored how semi-automated risk assessment tools could assist with conducting and reviewing data protection impact
assessments.

Although fundamental, privacy risk assessment is only one part of the wider effort organisations must make to demonstrate trustworthiness and help foster public trust and acceptance in AI-driven medical research.

References

European Commission. (n.d.). Artificial Intelligence in healthcare. Directorate-General for Health and Food Safety: Public Health. Available at: https://health.ec.europa.eu/ehealth-digital-health-and-care/artificial-intelligence-healthcare_en (Accessed 11 September 2026).

Information Commissioner’s Office. (2022, April). Overview of Data Protection Harms and the ICO’s
Taxonomy Available at: https://ico.org.uk/media2/about-the-ico/documents/4020144/overview-of-data-protection-harms-and-the-ico-taxonomy-v1-202204.pdf (Accessed 11 September 2026).

Jones, C., Thornton, J., & Wyatt, J.C. (2023, Autumn). Artificial intelligence and clinical decision support: clinicians’ perspectives on trust, trustworthiness, and liability. Medical Law Review, 31, 4, 501–520. https://doi.org/10.1093/medlaw/fwad013.

Milne, R., Cardoz, P., Besse, J., Bhangal, M., Chis, T., Otto, L., Sombe, I., & Tomba, G. 2026. Public Trust in Science: Overview Findings on Trust and Scientific Trustworthiness [Report]. RAND Europe: Cambridge. Available at: https://cms.wellcome.org/sites/default/files/2026-06/Wellcome_Rand_Public_Trust_in_science_Report_1_Summary.pdf  (Accessed 11 September 2026).

Modhvadia, R., & Field Reid, O. (2024, December 13). Participatory and inclusive data stewardship: A landscape review [Report]. Ada Lovelace Institute. Available at: https://www.adalovelaceinstitute.org/report/participatory-inclusive-data-stewardship/ (Accessed 11 September 2026).

National Institute of Standards and Technology (NIST), US Department of Commerce. (n.d.). Glossary: Privacy risk. Available at: https://csrc.nist.gov/glossary/term/privacy_risk (Accessed 11 September 2026).

O’Neill, O. (2018). Linking Trust to Trustworthiness. International Journal of Philosophical Studies, 26(2), 293-300. https://doi.org/10.1080/09672559.2018.1454637.

Understanding Patient Data. (n.d.). What we mean by trustworthy use of patient data. NHS Confederation. Available at: https://www.understandingpatientdata.org.uk/what-we-mean-trustworthy-use-patient-data (Accessed 11 September 2026).