Scrubnet

Scrubnet is an independent toolkit and knowledge hub for technical SEO specialists and web developers who want better ways to inspect websites, organise complex work and understand the changing web.

It brings together a growing collection of practical apps, technical articles and an open crawler observatory. SEO Scrubbox helps practitioners investigate technical SEO signals, rendering, indexability, site architecture and crawler access directly in the browser. ScrubGraph provides a private, local-first way to organise knowledge visually. These are the first Scrubnet apps, with more tools planned as useful ideas and workflows emerge.

Scrubnet also publishes practical guidance covering technical SEO, web development, AI search, structured publishing and crawler behaviour. The aim is to turn hands-on experience, experiments and observable evidence into information practitioners can apply to their own work.

The Scrubnet Crawler Observatory is the research layer behind part of this work. It publishes machine-readable feeds compiled from authorised public content and records verified crawler requests through live access logs. This makes it possible to examine discovery, recrawling, content freshness, request formats and the technical conditions surrounding crawler access.

Authorised websites can join the observatory for free, helping broaden the sectors, site structures and publishing patterns represented in the dataset. A crawler request demonstrates access, not indexing, ranking, preference, AI citation or subsequent use. Findings are presented as descriptive observations rather than evidence of causation.

Scrubnet is practitioner-built, independent and evidence-led: a place for better tools, clearer technical knowledge and a more observable web.