UK Public Sector GitHub Observatory
← all organisations

ukwa / Code

37 languages, 59.0 MB of source, and what the live estate is written in today.

Language figures come from GitHub's own per-repository byte counts, which measure source rather than a single headline label. File counts and repository sizes come from the mirrors themselves. The two tell different stories, and the difference is the point.

The codebase

37
languages in use
59.0 MB of source
19.7k
files at HEAD
across every analysed repository
0 B
on disk as mirrors
full history, every branch and tag
172.9 MB
largest repository
rclone

What stands out

One archived repository holds 19% of all the bytes.

rclone is 172.9 MB of git - a 2013 static snapshot of www.gov.uk, not source code. It single-handedly makes HTML the largest language by volume (4% of all source bytes). Any size or language statistic that does not exclude it is measuring an archive, not a codebase.

Sizes here are GitHub's reported figures.

This organisation was harvested with blobless partial clones, which fetch every commit and file path but not file contents, so there is no local size to measure against. Checked elsewhere in this dataset against 1,526 fully mirrored repositories, GitHub's reported size is accurate to within about six percent.

Ruby is the house language, but its share is falling.

Ruby is the primary language of 4% of archived repositories but only 0% of live ones. Python, Go, TypeScript and HCL have taken the difference - the signature of a shift from monolithic Rails publishing apps toward data pipelines, infrastructure-as-code and platform tooling.

Most repositories are small; a handful are enormous.

47% of repositories are under a megabyte of git data. The estate is not a monolith - it is a very long tail of small services, gems and prototypes around a few large publishing applications.

Languages, two ways

By volume of source

Bytes of code. Dominated by archived static HTML.

JavaJava: 19.9 MB19.9 MBJupyter NotebookJupyter Notebook: 10.3 MB10.3 MBJavaScriptJavaScript: 8.2 MB8.2 MBPythonPython: 6.7 MB6.7 MBGoGo: 6.2 MB6.2 MBHTMLHTML: 2.4 MB2.4 MBPHPPHP: 1.5 MB1.5 MBTypeScriptTypeScript: 1.5 MB1.5 MBCSSCSS: 906.6 kB906.6 kBASP.NETASP.NET: 316.3 kB316.3 kBVueVue: 195.7 kB195.7 kBScalaScala: 134.3 kB134.3 kBRubyRuby: 129.8 kB129.8 kBShellShell: 106.3 kB106.3 kB

By number of repositories

How many repositories name each language as their primary one. A better guide to what engineers actually work in.

JavaScriptJavaScript: 2525ShellShell: 2424HTMLHTML: 2323DockerfileDockerfile: 2323CSSCSS: 2121PythonPython: 2020JavaJava: 1919XSLTXSLT: 55RubyRuby: 55MakefileMakefile: 55ScalaScala: 44BatchfileBatchfile: 44GoGo: 33TypeScriptTypeScript: 22

Live estate versus archive

The same language ranking, split by whether the repository is still active. The shift is the clearest signal of technical direction in the whole dataset.

Active repositories

JavaJava: 22PythonPython: 00JavaScriptJavaScript: 11ShellShell: 00GoGo: 00CSSCSS: 00TypeScriptTypeScript: 00DockerfileDockerfile: 00RubyRuby: 00Jupyter NotebookJupyter Notebook: 00

Archived repositories

JavaJava: 1515PythonPython: 1111JavaScriptJavaScript: 55ShellShell: 44GoGo: 33CSSCSS: 22TypeScriptTypeScript: 22DockerfileDockerfile: 22RubyRuby: 22Jupyter NotebookJupyter Notebook: 22

Scale

Repository size distribution

Git data per repository, log-ish buckets.

1712.88.54.20< 10 kB: 2< 10 kB< 100 kB: 7< 100 kB< 1 MB: 17< 1 MB< 10 MB: 11< 10 MB< 100 MB: 16< 100 MB100 MB +: 2100 MB +

Commits against files

Each circle is a repository, sized by number of distinct authors. Both axes are logarithmic. The diagonal band is the normal relationship between codebase size and churn; outliers above it are long-lived apps with heavy iteration, below it are dumps and imports.

8.6k1.5k246.141.57.018.165.3527.64.3kSentimentalJ: 28 commits, 11 files, 6 authorsarchive-commons: 3,305 commits, 340 files, 23 authorsawesome-web-archiving: 79 commits, 1 files, 27 authorsbanana: 1,735 commits, 445 files, 59 authorsbootstrap: 3,324 commits, 211 files, 236 authorsbrowsertrix-cloud: 200 commits, 121 files, 4 authorsbrowsertrix-crawler: 92 commits, 46 files, 8 authorscrawl-test-site: 26 commits, 44 files, 2 authorsdocker-hbase-standalone: 10 commits, 5 files, 4 authorsdocker-trifecta: 22 commits, 6 files, 6 authorsdocker-zeppelin: 16 commits, 6 files, 4 authorshapy: 58 commits, 35 files, 5 authorsheritrix3: 2,635 commits, 1,014 files, 74 authorsiipc_workshop_warclight: 15 commits, 160 files, 3 authorsjruby-whois: 27 commits, 21 files, 3 authorsjupyterhub: 37 commits, 5 files, 5 authorsjupyterhub-deploy-docker: 89 commits, 22 files, 13 authorslanguage-detector: 152 commits, 172 files, 14 authorsluigi: 3,715 commits, 367 files, 516 authorsmementoweb-client-java: 78 commits, 23 files, 2 authorsmementoweb-webclient: 74 commits, 78 files, 1 authorsmrjob: 8,649 commits, 323 files, 153 authorsomeka-s-docker: 8 commits, 6 files, 2 authorsopendata: 236 commits, 108 files, 5 authorsopenwayback: 3,401 commits, 852 files, 66 authorsopenwayback-access-control: 45 commits, 149 files, 6 authorsoutbackcdx: 450 commits, 90 files, 17 authorspuppeteer: 2,387 commits, 2,573 files, 416 authorspy-wacz: 87 commits, 38 files, 11 authorspy-wasapi-client: 81 commits, 12 files, 5 authorspython-heritrix: 20 commits, 7 files, 4 authorspywb: 2,334 commits, 424 files, 67 authorsrclone: 6,356 commits, 1,687 files, 581 authorsshine: 813 commits, 1,072 files, 20 authorssiegfried: 1,129 commits, 4,263 files, 9 authorssolrwayback: 2,023 commits, 463 files, 23 authorstika: 6,070 commits, 1,424 files, 89 authorsukwa-heritrix: 577 commits, 153 files, 11 authorsukwa-pywb: 316 commits, 246 files, 8 authorsuptimerobot-prometheus-exporter: 72 commits, 21 files, 5 authorsw3act: 2,574 commits, 801 files, 30 authorswarc: 97 commits, 24 files, 5 authorswarcbase: 721 commits, 188 files, 31 authorswarclight: 87 commits, 98 files, 4 authorswarcprox: 815 commits, 35 files, 17 authorswarctools: 194 commits, 33 files, 8 authorsweb-archives: 60 commits, 49 files, 3 authorswebarchive-commons: 3,631 commits, 436 files, 51 authorswebarchive-discovery: 1,716 commits, 576 files, 40 authorswebarchive-hadoop-tools: 3,367 commits, 106 files, 22 authorswebarchive-test-suite: 9 commits, 22 files, 1 authorswebarchive-wat-mining: 7 commits, 8 files, 3 authorswebarchiving-notebooks: 14 commits, 13 files, 3 authorswren: 180 commits, 218 files, 4 authorsfiles at HEAD (log)commits (log)

The shape of a repository

Directory layout is the cheapest reliable signal of what a repository actually is. app/ with config/ and spec/ is a Rails application; src/ with test/ is Java or Node. Counted by how many repositories contain each top-level directory at HEAD.

Most common top-level directories

srcsrc: 1313docsdocs: 1010teststests: 77testtest: 66binbin: 55distdist: 44dockerdocker: 44appapp: 44resourcesresources: 33confconf: 33publicpublic: 33liblib: 33vendorvendor: 33examplesexamples: 33docdoc: 33scriptsscripts: 33cmdcmd: 33integration-testintegration-test: 33

Most common file extensions

By total file count across the estate.

.java.java: 3.9k3.9k.md.md: 2.4k2.4k.xml.xml: 2.3k2.3k.go.go: 954954.py.py: 779779.html.html: 764764.txt.txt: 742742.js.js: 713713.png.png: 471471.ts.ts: 167167.css.css: 158158.log.log: 153153.less.less: 139139.json.json: 124124.rst.rst: 118118.graph_data.graph_data: 115115

The twenty largest repositories

RepositoryLanguageGit size CommitsFilesStatus
rcloneGo172.9 MB6,3561,687archived
siegfriedGo119.7 MB1,1294,263archived
ukwa-heritrixJava99.7 MB577153archived
tikaJava97.7 MB6,0701,424archived
w3actJava71.8 MB2,574801active
ukwa-pywbJavaScript52.7 MB316246active
web-archivesJupyter Notebook33.6 MB6049archived
puppeteerTypeScript31.7 MB2,3872,573archived
solrwaybackJava26.4 MB2,023463archived
bootstrapCSS23.6 MB3,324211archived
pywbJavaScript23.1 MB2,334424archived
bananaJavaScript20.5 MB1,735445archived
mrjobPython17.4 MB8,649323archived
opendataHTML14.0 MB236108active
warclightRuby14.0 MB8798archived
openwaybackJava13.8 MB3,401852archived
webarchive-discoveryJava13.6 MB1,716576active
heritrix3Java11.4 MB2,6351,014archived
luigiPython10.0 MB3,715367archived
shineJavaScript8.5 MB8131,072archived