UK Public Sector GitHub Observatory
All 55 public repositories. Commits, authors and file counts come from the mirrored git history; type and team come from the GOV.UK Developer Documentation where it covers the repository. Type to filter; click a column heading to sort.
| Repository | Language | Git size | Commits | Authors | Files | Type | Team | Status |
|---|---|---|---|---|---|---|---|---|
| rclone "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wa | Go | 172.9 MB | 6,356 | 581 | 1,687 | archived CI docker | ||
| siegfried signature-based file format identification | Go | 119.7 MB | 1,129 | 9 | 4,263 | archived CI | ||
| ukwa-heritrix The UKWA Heritrix3 custom modules and Docker builder. | Java | 99.7 MB | 577 | 11 | 153 | archived CI docker | ||
| tika Mirror of Apache Tika | Java | 97.7 MB | 6,070 | 89 | 1,424 | archived docker | ||
| w3act w3act is an annotation and curation tool for building web archive collections | Java | 71.8 MB | 2,574 | 30 | 801 | active CI docker | ||
| ukwa-pywb | JavaScript | 52.7 MB | 316 | 8 | 246 | active CI docker | ||
| web-archives | Jupyter Notebook | 33.6 MB | 60 | 3 | 49 | archived | ||
| puppeteer Headless Chrome Node.js API | TypeScript | 31.7 MB | 2,387 | 416 | 2,573 | archived CI bot | ||
| solrwayback A search interface and wayback machine for the UKWA Solr based warc-indexer framework. | Java | 26.4 MB | 2,023 | 23 | 463 | archived CI docker | ||
| bootstrap HTML, CSS, and JS toolkit from Twitter | CSS | 23.6 MB | 3,324 | 236 | 211 | archived | ||
| pywb Core Python Web Archiving Toolkit for replay and recording of web archives | JavaScript | 23.1 MB | 2,334 | 67 | 424 | archived CI docker | ||
| banana Banana for Solr - A Port of Kibana | JavaScript | 20.5 MB | 1,735 | 59 | 445 | archived CI | ||
| mrjob Run MapReduce jobs on Hadoop or Amazon Web Services | Python | 17.4 MB | 8,649 | 153 | 323 | archived | ||
| opendata Repository of documentation about the open datasets published by the UK Web Archive. | HTML | 14.0 MB | 236 | 5 | 108 | active | ||
| warclight A Rails engine supporting the discovery of web archives. | Ruby | 14.0 MB | 87 | 4 | 98 | archived | ||
| openwayback The OpenWayback Development Project | Java | 13.8 MB | 3,401 | 66 | 852 | archived docker | ||
| webarchive-discovery Please note that the warc-indexer tool & code is now supported by NetArchiveSuite. The 'warc-ind | Java | 13.6 MB | 1,716 | 40 | 576 | active docker | ||
| heritrix3 Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web craw | Java | 11.4 MB | 2,635 | 74 | 1,014 | archived CI docker | ||
| luigi Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles depend | Python | 10.0 MB | 3,715 | 516 | 367 | archived | ||
| shine Prototype SOLR-powered web archive exploration UI. | JavaScript | 8.5 MB | 813 | 20 | 1,072 | archived docker | ||
| webarchive-commons | Java | 8.4 MB | 3,631 | 51 | 436 | archived | ||
| webarchive-hadoop-tools | Java | 7.8 MB | 3,367 | 22 | 106 | archived | ||
| archive-commons | Java | 7.7 MB | 3,305 | 23 | 340 | archived | ||
| warcbase A web archive browser built on HBase | Java | 5.7 MB | 721 | 31 | 188 | archived | ||
| webarchive-test-suite A set of test files for web archiving. | Arc | 2.5 MB | 9 | 1 | 22 | archived | ||
| language-detector Language Detection Library for Java | Java | 2.2 MB | 152 | 14 | 172 | archived | ||
| py-wacz | Python | 1.7 MB | 87 | 11 | 38 | archived CI | ||
| warcprox WARC writing MITM HTTP/S proxy | Python | 1.1 MB | 815 | 17 | 35 | archived docker | ||
| webarchiving-notebooks A collection of Jupyter notebooks for working with web archive data, tools and APIs | Jupyter Notebook | 1.1 MB | 14 | 3 | 13 | archived | ||
| browsertrix-crawler Run a high-fidelity browser-based crawler in a single Docker container | JavaScript | 970.8 kB | 92 | 8 | 46 | archived CI docker | ||
| mementoweb-webclient A simple web-based interface to Memento holdings. | Java | 778.2 kB | 74 | 1 | 78 | archived docker | ||
| crawl-test-site A simple site that uses GitHub pages to host resources for testing crawlers. | CSS | 726.0 kB | 26 | 2 | 44 | archived docker | ||
| hapy A Python wrapper around the Heritrix API. | Python | 673.8 kB | 58 | 5 | 35 | archived docker | ||
| browsertrix-cloud | TypeScript | 646.1 kB | 200 | 4 | 121 | archived docker | ||
| openwayback-access-control web access control (exclusion oracle) tools for optional use with wayback machine | JavaScript | 646.1 kB | 45 | 6 | 149 | archived | ||
| outbackcdx A Wayback RemoteResourceIndex server using RocksDB | Java | 629.8 kB | 450 | 17 | 90 | archived docker | ||
| mementoweb-client-java Core Java libraries for Memento clients. | Java | 565.2 kB | 78 | 2 | 23 | archived CI | ||
| wren Experiments in testable, scaleable crawler architectures | PHP | 548.9 kB | 180 | 4 | 218 | archived docker | ||
| jupyterhub-deploy-docker Reference deployment of JupyterHub with docker | Python | 382.0 kB | 89 | 13 | 22 | archived docker | ||
| warctools warctools | Python | 308.2 kB | 194 | 8 | 33 | archived | ||
| jruby-whois Ruby's Whois gem wrapped up as a Maven dependency for Java code, via JRuby. | Java | 208.9 kB | 27 | 3 | 21 | archived | ||
| warc Python library for reading and writing warc files | Python | 183.3 kB | 97 | 5 | 24 | archived | ||
| SentimentalJ A sentiment analysis module for node.js | Java | 181.2 kB | 28 | 6 | 11 | archived | ||
| iipc_workshop_warclight A 'vanilla' Warclight instance for workshops and demonstration. | Ruby | 142.3 kB | 15 | 3 | 160 | archived docker | ||
| python-heritrix simple python wrapper around heritrix v3.x api | Python | 126.0 kB | 20 | 4 | 7 | archived | ||
| webarchive-wat-mining WAT (web archive transform) metadata mining | Shell | 118.8 kB | 7 | 3 | 8 | archived | ||
| awesome-web-archiving An Awesome List for getting started with web archiving | - | 91.1 kB | 79 | 27 | 1 | archived | ||
| uptimerobot-prometheus-exporter A Golang Prometheus exporter for UptimeRobot monitors. | Go | 55.3 kB | 72 | 5 | 21 | archived docker | ||
| py-wasapi-client A client for the [Archive-It] WASAPI Data Transfer API | Python | 53.2 kB | 81 | 5 | 12 | archived docker | ||
| jupyterhub JupyterHub docker image for the compmodels class | Python | 43.0 kB | 37 | 5 | 5 | archived docker | ||
| docker-cantaloupe Runs 🍈 with dependancies in docker container. | Shell | 20.5 kB | 3 | 1 | 6 | archived docker | ||
| docker-trifecta Trifecta docker image | Dockerfile | 16.4 kB | 22 | 6 | 6 | archived docker | ||
| docker-zeppelin Yet Another Docker Container for Apache Zeppelin | Shell | 12.3 kB | 16 | 4 | 6 | archived docker | ||
| docker-hbase-standalone Docker automated-build providing standalone HBase 1.1.x | Shell | 9.2 kB | 10 | 4 | 5 | archived docker | ||
| omeka-s-docker | Dockerfile | 5.1 kB | 8 | 2 | 6 | archived docker |