Standard Bank Group, working with AWS Professional Services, has deployed an automated assessment that scores each generative AI use case against international AI standards and its own policies, according to an AWS Machine Learning Blog post. The assessment produces reports and a dashboard showing what is already in place, what is working well, and what still needs mitigation. Review and approval remain with the bank's risk committee, but the committee now bases decisions on evidence collected the same way across use cases.

Why it matters

The post frames the problem directly: having an AI policy is one thing, proving one use case at a time that AI systems follow it is another. Every new generative AI application adds another system to prove, and each still needs a risk committee's approval before it reaches customers or employees. When that approval depends on reviewers reading documents and testing systems by hand, governance becomes the slowest step in the pipeline.

Standard Bank Group did not need a new AI policy. It already had a model risk practice covering concerns such as explainability and fairness, plus a change risk process that assesses each new use case according to its level of risk before go-live. Generative AI increases complexity because models and tooling change every three to six months, yet each use case must still meet the same security and compliance bar. The goal was to automate evidence gathering and testing, then feed results into the approval gates the bank already runs.

Technical details

The assessment runs entirely in Standard Bank Group's own AWS account. Documents, code, and test results stay in an Amazon Simple Storage Service (Amazon S3) bucket there. The AWS Well-Architected Tool handles intake, AWS Step Functions orchestrates the pipeline, and a foundation model on Amazon Bedrock evaluates the evidence using LLM-as-a-judge, in which a large language model checks evidence against defined criteria.

According to the post, Amazon Bedrock does not share the bank's prompts or model outputs with model providers and does not use them to train models. Access is controlled through AWS Identity and Access Management (IAM). Because Bedrock offers many foundation models through one API, the team can move to a newer judge model by changing a configuration setting instead of rebuilding the pipeline.

Each assessment moves through four steps. First, a pre-assessment agent running on AWS Fargate scans the use case's code repository and documentation and pre-populates a custom Well-Architected lens of about 40 questions, each covering a risk event such as incomplete technical documentation or insufficient human oversight. Delivery teams review the answers, correct mistakes, and add evidence the agent could not find.

Second, the pipeline examines four kinds of evidence: system documentation, policy documents, source code and the system prompt, and live calls to the application. For documents and code, it checks whether each artifact contains what its mapped control requires, with each control drawn from an international AI standard or framework or from bank policy. For live testing, the pipeline calls the application through its AWS Lambda function or API. Delivery teams supply typical user questions; the pipeline adds built-in jailbreak attempts and off-topic questions the application is expected to decline. It also sends the same prompt several times in original and reworded forms and uses an embedding model on Amazon Bedrock to check that answers stay consistent in meaning. Every response is stored in Amazon S3 so reviewers can see exactly what the application generated.

Third, the pipeline evaluates the evidence against controls. Fourth, it summarizes results in a standard, predefined format and delivers them to the people who act on them.

What to watch

Jörg Fischer, Group Chief Information and AI Officer at Standard Bank Group, is quoted in the post saying the assessment demonstrates how the bank is working with AWS to embed intelligent, automated risk management into its AI journey, and that responsible AI is not about slowing innovation down but enabling it to build and scale with confidence.

The practical takeaway is that automation targets evidence collection and testing, not the approval decision itself. Teams considering a similar approach will need to map controls to existing standards and policies, maintain the question set, and keep human review in the loop for answers the agent cannot source.

About the author

AWS Machine Learning Blog

Thoai Dispatch Curated — curated source featured on Thoai Dispatch.

More by AWS Machine Learning Blog