UK public sector / Nobody knows how many government organisations there are

14 August 2026 · open data registers method credit

This project needs a list of UK public bodies. Everything here starts with one: discovery generates candidate GitHub handles from organisation names, the sector labels come from what kind of body something is, and any claim that starts "across UK government" is only as good as the list underneath it. So it is worth asking where that list comes from, and how far it can be trusted.

The register

gov.uk/api/organisations is the closest thing to an authoritative answer. It is a paged JSON endpoint, published under the Open Government Licence, carrying 1,263 organisations with the GB-GOR identifiers from org-id.guide, the type of body each one is, its parent and child organisations, and - crucially - whether it still exists.

595 of those 1,263 entries are bodies that have closed. That is not a flaw in the register, it is the most useful thing in it: departments merge, agencies are renamed, functions are devolved, and a register that only listed the living would make it impossible to work out where a body's work went. Each closed entry says how it ended and, where there is one, which organisation succeeded it.

The other 299 do not reconcile the way you would expect either, and this is the second trap. They are marked exempt, which reads like a disqualification and means the opposite: a body that is very much alive but does not publish through GOV.UK because it runs its own site. ACAS, the Advanced Research and Invention Agency and the AI Security Institute are all exempt. Treat that as "not live" and you delete three hundred real organisations; treat live as the only positive value and you get the same result.

The field that looks like the answer and is not

Here is the trap, and it is a good one. The obvious way to filter out dead organisations is closed_at. It is a date field, it is named after the thing you want, and it is populated. It is also set on only 170 of the 595 closed bodies.

The fact lives in details.govuk_status, which reads live or closed, with details.govuk_closed_status saying how. Filter on closed_at and you keep 425 organisations that GOV.UK itself marks as closed, while believing you have removed them. The result is not an error message. It is a slightly-too-long list that looks entirely reasonable.

Somebody else already did this properly

odileeds/uk-government-organisations, from Open Innovations in Leeds, has been combining the two published sources into usable CSVs since August 2021. It is exactly the kind of thing that ought to exist: a scheduled job on the first of every month, a 242-line Perl script anyone can read, the raw inputs committed alongside the outputs, and a README that is honest about which source it trusts. This project's own discovery step does a worse version of the same job.

Its API-derived output is in good order — 1,091 rows, one per current organisation, exactly as documented. The HTML-derived one is not.

What went wrong

The script scrapes gov.uk/government/organisations, matching each organisation's <li> block and then, inside each block, the link that carries its name and URL. GOV.UK changed the inner markup. The outer match still works; the inner ones no longer do.

The consequences arrive in stages, and every stage looks fine on its own:

• The scraper still finds 615 organisations, so the README, which is generated from that count, still says 615. Correct.
• 590 of those 615 rows come out with a blank name and a blank URL. The identifier survives, because it is parsed from the outer block. ofsted, serious-fraud-office and supreme-court-of-the-united-kingdom are all present, and all nameless.
• The rows are then keyed by URL into a hash. Every one of those blank rows has the same blank key, so they collapse into a single entry, leaving 26 distinct organisations.
• The augmented output is written from that hash: 26 rows, against a README, in the same repository, saying 615.

Nothing threw. No step was individually unreasonable. The file has a correct header, plausible contents, and Cabinet Office at the top — it looks precisely like a working list of government departments, because that is what the survivors happen to be. It has been rebuilt on the first of the month for four and a half years: 584 rows in the February 2022 run, 25 in the March 2022 run, and never above 26 across the 46 runs since.

The bug is not the scrape breaking. Scrapes break. The bug is that a partial failure produced output that passed every check a human would casually apply, and then did it another forty-five times.

Why this is in a writeup rather than an issue

It is both. This is not a security defect and there is nothing to disclose responsibly: it is an open dataset, publicly wrong in public, and the fix is a regex. What makes it worth eight hundred words is that this site runs on the same machinery and is exposed to the same failure in the same way.

The deleted-repository fetch reads another project's JSON. The domain probe reads DNS. The harvest reads GitHub's API. Every one of those could start returning a smaller, well-formed, entirely plausible answer tomorrow, and every page on this site would render it without complaint. Two of the six traps recorded in this project's own handover notes were exactly that: a build step that reported success while doing nothing, and a weekly publish that had never once run. Neither produced an error either.

So the lesson taken from this, and now applied here: a pipeline needs to know roughly how big its own output should be. The register fetch behind this page refuses to overwrite a good local copy if the API returns fewer than 500 organisations, because "the source went strange" and "the source shrank" are the same event and only one of them is visible. That check costs three lines. It is the check that would have caught this in March 2022.

Credit, and what happens next

The reason any of this could be established — the exact month, the exact row counts, the mechanism — is that Open Innovations published the script, the inputs, the outputs and every monthly run in a public git history under an open licence. A closed pipeline with the same defect would simply be a dataset that quietly disagreed with reality, and nobody outside could ever say why.

This project will use the register directly rather than through anyone else's copy, and the register's govuk_status now drives the abolished-organisation flag so that departments which closed in 2020 stop being scored on how recently they committed. The row counts quoted above are re-fetched on every build. If the scraper is fixed, this page will say so on its own, without anybody editing it.

Method: register figures from the GOV.UK organisations API, fetched 2026-08-28, Open Government Licence v3.0. odileeds row counts fetched from the project's main branch at the same time and counted with a standard CSV reader. The February-to-March 2022 transition was established by fetching all 51 revisions of that file from the repository's history and counting each; that one figure is historical and is not recomputed on every build. No part of this required anything other than public data.

← All writeups