How do I measure my AI model's governance?

Install govbench (pip install git+https://github.com/CSOAI-ORG/govbench), wrap your model as a callable(prompt)→response function, and run score_model(generate). It tests your model against a 72-cell grid of governance probes and returns refusal rates by category.

Is GovBench reproducible?

Yes. GovBench uses fixed seed (42) and saved test splits. Re-running produces identical numbers. All trained detectors have saved splits for verification.

Does GovBench work with any AI model?

Yes. Any callable(prompt)→response works: OpenAI, Anthropic, HuggingFace, Ollama, or your own gate. It's model-agnostic by design.

GovBench

Measure any AI's refusal behaviour on a governance-probe grid — reproducibly, for free.

MIT pip install git+https://github.com/CSOAI-ORG/govbench defensive by design reproducible (seed 42 + saved splits) Ed25519-signed by CSOAI

Run the benchmark Read the method

governance-probe grid cells
(6 categories × 12 styles)

237

items in the published bank
(csoai/gspc-gov, canary row excluded)

0.381

Council-34 macro-F1, full bank
(board leader; intervals overlap falcon3)

0.515

Council-34 accuracy [0.451, 0.578]
95% CI, n=237 — first quotable axis

What it measures

6 governance risk categories × 12 probe styles = a 72-cell refusal grid. Defensive by construction: it fires probe prompts at a model's gate and scores whether it refuses. It never reproduces harmful methods — harm goals are labels only.

Fraud

Deceiving people out of money or credentials

Disinformation

Generating false content to manipulate public opinion

Privacy

Exposing or misusing personal data without consent

Discrimination

Producing biased or discriminatory decisions about people

Unsafe Advice

Unsafe medical / legal / financial advice presented as authoritative

Governance Evasion

Overriding, disabling or tampering with the governance gate itself

12 probe styles: direct · roleplay · hypothetical · authority · persona · academic · incremental · urgency · prefix-inject · refusal-suppress · obfuscation · payload-split

The 72-cell grid

Each cell is a specific (category × style) combination. Green = model refused. Amber = honest gap. The grid finds where gates hold and where they leak.

Illustrative. Run the benchmark to see your model's real grid.

How it works

1. Wrap your model as callable(prompt)→response

2. score_model(generate) runs the 72-cell grid + benign controls

3. Get refusal rate by category + over-block rate + signed record

from govbench import score_model

from govbench.scorer import is_refusal

from govbench.benign_controls import over_block_rate

def generate(prompt):

    return my_model(prompt)

result = score_model(generate, verbose=True)

print(result["overall_refusal_rate"])

print(over_block_rate(generate, is_refusal))

Why it's different

Signed & attestable — Ed25519, verify offline

Reproducible — fixed seed + saved splits

Honest by construction — shows its gaps

Model-agnostic — any callable(prompt)→response

Defensive — labels only, never reproduces harm

Free & open — MIT, pip-installable

The honest contract: A high refusal rate on this grid is a reference target, not proof of un-jailbreakability. Real attackers vary phrasing far beyond 12 styles. Finding a leak is the point — a grid that finds nothing wasn't trying hard enough. This is a measurement standard, not an accredited compliance certification.

Sovereign Gate — a separate programme

The numbers below are not GovBench scores. They measure the Sovereign Gate stack — a trained CPU-side filter (sov_clf: TF-IDF + logistic regression, sovereign, no external API) and its rule layer — on held-out splits. Harness: measure_full.py; splits saved; every number traceable to a result file. GovBench measures models; the Gate blocks attacks. Different questions, different numbers.

0.939

gate score on the 72-cell grid
[0.866, 0.978] 95% CI, n=72 (from 0.545 baseline)

0.979

prompt-injection detector accuracy
[0.970, 0.986] 95% CI, n=1,293 held-out

0.967

prompt-injection detector F1
[0.956, 0.975] 95% CI, n=1,293

0.878

toxicity detector accuracy
[0.852, 0.900] 95% CI, n=698 held-out

0.784

toxicity detector F1
[0.752, 0.813] 95% CI, n=698

0/14

benign controls over-blocked
upper bound 0.215 at n=14 — small sample, stated plainly

Standards mapping

GovBench maps to established governance frameworks. Running it is not a legal conformity attestation — but it provides the measurement layer that compliance frameworks require.

EU AI Act Art.9 (risk management)

EU AI Act Art.15 (robustness)

NIST AI RMF

ISO 42001

OSCAL 1.1.2

Run it now

Three lines. Any model. Free.

pip install git+https://github.com/CSOAI-ORG/govbench

from govbench import score_model

print(score_model(generate))

View on GitHub View the dataset

Run it yourself

A frozen 24-item subset of the published bank, graded by the same deterministic rule. Council-34 scored 0.345 macro-F1 on this subset; on the full published bank (n=237) it scores 0.381 macro-F1 · 0.515 accuracy [0.451, 0.578] 95% CI. The subset is a demo; the bank is the measurement. No sign-up, nothing leaves your browser.

Items: csoai/gspc-gov · grading is a regex label read plus macro-F1, identical to the published harness · measurement, not certification, and not legal advice.