Technical SEO essentials: crawl hygiene for UK SME sites

Large sites talk about crawl budget because they have millions of URLs. A UK small business site has a smaller version of the same problem: Google spends time on junk — tag pages, internal search, soft 404s — and the service pages get less attention than they should. A quarterly pass in Search Console is enough. Our website audit will flag a slice of this; the coverage report shows the rest.

Coverage tab triage

• Excluded by noindex: intentional or accidental?
• Duplicate without user-selected canonical: fix canonical tags or parameters.
• Crawled — currently not indexed: usually quality or duplication — merge or improve.

Sitemaps, canonicals and robots.txt

The XML sitemap should list the canonical URLs you want in the index and nothing else. A canonical tag should point at the URL you would send a customer, including the right host and trailing-slash style. robots.txt can block paths that must never be crawled, such as admin or internal search. Do not use it to hide a page you still hope will rank — a blocked URL can still be indexed if something links to it, and you will not see the content quality problems. Redirect chains of three or more hops waste the crawl and dilute the signal. Point old URLs straight at the final page.

A quarterly half-hour

• Pages report: which important URLs dropped impressions.
• Coverage or page indexing: new “not indexed” reasons since last time.
• A crawl of the live site, looking for 404s in the main navigation and soft 404s that return a 200.
• One fix shipped before you add another blog post.