Evaluation of Automated Output Checking and AI Model Risk Assessment Community of Interest#

Leads: Jim Smith (University of the West of England), Jackie Caldwell (PHS)

Proposal#

Summary#

The session is intended to give an overview of the Community of Interest on automated risk assessment. This naturally includes the risk assessment of AI models, since this is not something that can be done manually.

The aim is to: (1) introduce the Community of Interest group, and its aim of reducing barriers to adoption of (semi) automated checking, (2) make some proposals about how we plan to move forward on this alongside UK-TRE, (3) to get feedback from people present about and how they would like to se the community develop and work

Preparation#

No required preparation beyond an open mind!

If people would like some perspective on where the project has evolved from, it might be useful to skim-read the first 5-6 pages of the SACRO project’s final report

But please note this Community of Interest has a broader remit than just SACRO - for example, by design we include projects such as DataShield, as well as other approaches for assessing ML privacy leakage.

Target audience#

No specific target audience in mind - anybody interested!

Session#

Summary#

THe workshop explored projects already exploring these issues, what the priorities of the community should be, and how to align everything already happening in this space.

A name was also chosen for the community! ReBOT, Reducing Barriers to Outputs from TREs.

Next steps include setting up a Jisc mailing list for the community, and a simple accessible guide, either written or video.

Raw notes#

  • Start of a community of people looking at these tools

  • Some projects have started tackling issues: DataShield, ACRO, GRAIMatter, SACRO

  • Defining best practice

    • Aligning finding of these projects with SDAP manual

  • Remove barriers to adoption by researchers

    • Weekly drop-in sessions for ACRO, AISDC, SACRO, etc

    • Email support service

  • Name of the community? ReBOT, Reducing Barriers to Outputs from TREs

  • What does everyone think?

Next steps#

  • Create new JISC mailing list

  • Create simple accessible guide, either written or video