NaturalHistoryMuseum / Method

1. Repository inventory

The authenticated GitHub REST API was paged for every public repository in the org, capturing size, language, licence, topics, archive state, and creation and push timestamps.

fetch-repos.ps1 → data/repos.csv (190 repositories)

2. Full mirror clones

Every repository was cloned with git clone --mirror, which captures every branch, every tag and the complete history with no working tree. Analysis then runs against local history rather than API summaries.

clone-alphagov.ps1 → D:\gds\mirrors\*.git

Two failure modes shaped this script and are worth recording. Running twelve concurrent clones largest-first saturated the link and every large transfer died with curl 56 ... Connection was reset; the fix was to clone small repositories first in a wide pool and large ones last in a narrow one, with retries and backoff. Separately, genuinely empty repositories clone successfully but never gain a ref, so a refs-based completeness test rejected them forever - they now get an explicit marker file.

3. Language byte counts

GitHub's per-repository language endpoint was queried for all repositories, giving real byte counts per language instead of a single headline label.

fetch-languages.py → data/languages.json

4. Git history mining

Each mirror was mined with two git commands - one full git log across all refs, and one ls-tree at HEAD - and aggregated in process. Spawning git is the expensive part at this scale, so everything is derived from those two passes.

analyse-repos.py → data/repo-facts.json, committers.csv, commit-months.csv (189 repositories with history, 1 empty)

5. Developer documentation

The sitemap of docs.publishing.service.gov.uk was walked and every page fetched, giving the official repository taxonomy, team ownership, and the full text of the developer manual.

scrape-docs.py → data/docs-pages.json, docs-taxonomy.csv

Identity is keyed on email

Git offers no stable person identifier. Committers are collapsed by email address, so one individual using both a work and a personal address counts twice, and a shared or misconfigured address merges several people into one. Headcount figures are therefore an upper bound on distinct emails, not a census of staff.

Email domain is a proxy, not a fact

A gov.uk address strongly implies civil-service or contracted staff, but GitHub's private-email setting hides the domain for a large share of committers, and many people commit from personal accounts. Domain splits indicate scale, not precise employment.

Forked repositories are excluded from the people figures

A fork carries the upstream project's entire history. Counting those commits credits GDS with work by people who never went near it - before this exclusion, the author of reveal.js appeared in the top ten committers, and the apparent headcount was 7,507 rather than the 3,196 reported here. Forks are still counted in repository, language and size totals, where they are genuinely part of the estate.

Bot detection is pattern-based

Automated identities are matched on name and email patterns. That means the obvious [bot] suffixes, but also role accounts that look human at a glance: Package Updater and GOV.UK Continuous Integration User are between them responsible for over 41,000 commits and are unambiguously machines. The matching will still miss a bot committing under a convincingly human identity.

One repository distorts every size statistic

govuk-archive is an 8.3 GB static snapshot of www.gov.uk from 2013, not source code. It is the single largest contributor of both bytes and HTML in the whole org. Size and language-by-volume figures name it explicitly wherever it matters.

Archived repositories are excluded from practice scoring

Convention scorecards run against live repositories only. Measuring a 2013 prototype against a 2026 standard would produce a worse-looking and far less useful number.

Public repositories only

This covers what alphagov publishes. GDS also operates private repositories, and infrastructure and deployment work that never appears on GitHub. Nothing here should be read as the whole of GDS engineering.

The scripts are ordinary PowerShell and Python with no dependencies beyond git, the GitHub CLI and a Python 3 interpreter. Run in order:

powershell -File fetch-repos.ps1
powershell -File clone-alphagov.ps1     # ~19 GB, resumable
python fetch-languages.py
python scrape-docs.py
python analyse-repos.py
python build-site.py

Every step is re-runnable. The clone is resumable and skips repositories that already completed.