Suite 1 of 6 · v0.1 draft
Does your AI stay inside the rules?
The Financial Conduct Bench measures whether an AI system giving financial guidance to UK consumers meets the FCA's conduct standards — starting with the advice and guidance perimeter, and extending to vulnerability, consumer understanding, promotions and accuracy.
Perimeter — advice, guidance and targeted support
Did the system stay on the permitted side of the line, for the specific line that applies to the product being discussed?
| System | Pass rate | Critical breaches(not netted off) | Cases |
|---|---|---|---|
| Doshi FCPauthor's system Doshi | 94.0% | 0 | 184 |
| Scripted baseline Reference | 70.1% | 4 | 184 |
| Frontier Model B Anonymised | 73.9% | 12 | 184 |
| Frontier Model A Anonymised | 67.4% | 18 | 184 |
Ranked by pass rate, except that any system with a critical breach sorts below every clean system. A breach of the perimeter is not offset by good answers elsewhere, so the ordering carries the same asymmetry as the scoring.
By dimension
| System | Investment perimeter | Debt counselling perimeter | Mortgage perimeter | Escalation and handover |
|---|---|---|---|---|
| Doshi FCP | 95.3% | 92.5% | 93.8% | 93.8% |
| Scripted baseline | 85.9% 1 critical | 75% 2 critical | 75% 1 critical | 41.7% |
| Frontier Model B | 81.3% 3 critical | 67.5% 5 critical | 65.6% 3 critical | 75% 1 critical |
| Frontier Model A | 75% 5 critical | 60% 7 critical | 59.4% 4 critical | 68.8% 2 critical |
The scope is wider than the perimeter
Staying the right side of the advice boundary is necessary and nowhere near sufficient. Five further suites are in development.
- livePerimeterDid the system stay on the permitted side of the line, for the specific line that applies to the product being discussed?
- buildingVulnerabilityDid the system detect vulnerability signals and change its behaviour in response?
- plannedUnderstandingCould a real customer act correctly on what they were told?
- plannedPromotionsWhen the system moved toward a product, did it meet the promotions standard?
- plannedAccuracyAre the facts it states about the regulated world actually correct?
- plannedFairnessDoes the same question get the same quality of answer regardless of who appears to be asking?