Home › Website SEO › Technical SEO
WEBSITE SEO / TECHNICAL SEO
Technical SEO: Complete Guide to Crawling, Indexing, Performance & Website Health
Learn how Technical SEO helps search engines crawl, render, understand and index websites through better architecture, URLs, redirects, sitemaps, robots.txt, structured data, performance, mobile usability and technical site health.
Practical explanations with copyable examples, honest limits and no guarantees about rankings.
- Crawling, indexing, rendering and ranking kept clearly separate
- robots.txt, canonicals, redirects and sitemaps explained accurately
- Audit checklists, migration checklists and prioritisation
THE BASICS
What Is Technical SEO?
Technical SEO is the work of improving a website’s technical foundations so search engines can efficiently discover, crawl, render, understand and index its pages — and so people can use the site without friction.
It’s worth being clear about what technical SEO can and can’t do. Fixing technical problems removes obstacles. It does not guarantee rankings, and no technical change on its own puts a site on page one. Search visibility depends on content quality, relevance, competition, external signals and a great deal else. What technical SEO does is make sure your pages are actually eligible — that nothing in the plumbing is preventing them from being found, processed and served.
Technical SEO is also frequently reduced to page speed. Speed matters, but it’s one area among many. The scope is considerably wider:
Four stages, not one
A lot of confused SEO advice comes from treating these as the same thing. They aren’t, and a page can succeed at one and fail at the next.
- Crawling — a crawler requests the URL
- Rendering — the page is processed, including JavaScript
- Indexing — the page may be stored for retrieval
- Ranking — systems decide what to show for a query
Crawled does not mean indexed. Indexed does not mean ranked. Keeping these separate makes diagnosing problems far easier.
DISTINCTIONS
Technical SEO vs On-Page and Off-Page SEO
The three areas are usually described as separate disciplines, but they overlap considerably and none works without the others. Excellent content on a page blocked from crawling achieves nothing; a technically flawless site with thin content achieves nothing either.
Off-page SEO covers signals from outside the site — links from other sites, brand mentions, and how a site is referenced elsewhere. It’s a different kind of work, largely outside your direct control. Claiming any one of the three matters most isn’t useful: they address different problems, and which one deserves attention depends entirely on what’s currently holding a particular site back. Content work is covered in our on-page SEO guide.
HOW SEARCH ENGINES WORK
Crawling, Rendering and Indexing
Crawling is a search engine requesting your URLs and their resources. Crawlers find URLs through links from pages they already know, XML sitemaps, redirects and previously crawled URLs. How often and how much they crawl depends on the site, its server responses and how much appears to be worth revisiting.
Crawlability is whether that’s possible at all. Server errors, slow responses, broken links, incorrect robots directives, excessive redirects, blocked CSS or JavaScript files and infinite URL spaces created by filters can all interfere.
Rendering processes the page the way a browser would, including running JavaScript. This matters because content that only exists after scripts run has to survive that step to be understood.
Indexing is the processing and potential storage of a page for retrieval. It’s not automatic — a crawled page may not be indexed, for reasons including duplication, quality assessments, canonical signals pointing elsewhere or explicit directives.
Indexability is whether a page is eligible: no noindex, an appropriate status code, no canonical pointing to a different URL, and content substantial enough to be worth storing.
JAVASCRIPT
JavaScript, Rendering and Technical SEO
JavaScript is not inherently bad for SEO — that claim is outdated and unhelpful. What’s true is that JavaScript adds a dependency: content that only appears after scripts run requires those scripts to load and execute successfully before anything can understand it.
Client-side rendering sends minimal HTML and builds the page in the browser. Server-side rendering produces HTML on the server so content is present in the initial response, with JavaScript then hydrating it for interactivity. Static rendering does that work ahead of time at build. Frameworks often let you mix strategies per route, which is usually the practical answer — render content pages on the server, leave the dashboard client-rendered.
Common JavaScript problems worth checking: content that only appears after a click or scroll interaction, navigation built from click handlers rather than real anchor elements, metadata set incorrectly or late, structured data injected inconsistently, failed API requests leaving empty pages, and client-side errors that stop rendering entirely.
The practical test is simple: view the rendered page and ask whether the important content and links are there without interaction. See our JavaScript guide and React guide for the underlying mechanics.
├── Category
│ ├── Subcategory
│ │ ├── Page
│ │ └── Page
│ └── Article
├── Category
│ ├── Subcategory
│ └── Article
└── Resources
├── Guide
└── Tools
# Readable, meaningful, consistent /website-seo/technical-seo/ /products/running-shoes/ /blog/how-search-engines-crawl/ # Harder to read and less useful as a signal /page?id=84937&cat=4&ref=nav /p/12/34/56/ /Category/Running-Shoes/Index.php # Parameters are not inherently bad — # they are appropriate for filtering, sorting and tracking /products/running-shoes?sort=price&page=2 # Pick one convention and stay with it # https vs http, www vs non-www, trailing slash or not
ARCHITECTURE
Website Architecture, Internal Links and URLs
Site architecture is how pages relate to each other. A logical hierarchy where related content sits together helps people navigate and helps crawlers understand what a section is about.
Crawl depth is how many links it takes to reach a page from an accessible starting point. There’s no magic maximum number of clicks, despite how often one gets quoted. The practical goal is that important pages are reachable through a sensible linking structure rather than buried behind pagination and filters.
Orphan pages have few or no internal links pointing to them. They’re hard to discover, invisible to users navigating the site, and often signal that something was published and forgotten. Find them by comparing sitemap URLs, crawl data, internal link data and analytics — a page in the sitemap with no internal links is a candidate.
Internal links do a lot of work: they create crawl paths, establish relationships between pages, help distribute importance and help people find related content. Use descriptive anchor text that says what the destination is about, and link because it genuinely helps the reader. See our website navigation guide.
URLs should be readable, consistent and meaningful. Query parameters aren’t inherently harmful — they’re the right tool for filtering and sorting — but unnecessary complexity makes URLs harder to work with and easier to duplicate accidentally.
CANONICALISATION
Duplicate URLs and Canonical Tags
The same content is often reachable through several URLs without anyone intending it: HTTP and HTTPS versions, www and non-www, with and without a trailing slash, with tracking parameters appended, through different category paths, or as sorted and filtered variants of a listing.
rel=”canonical” tells search engines which URL you consider the preferred version. An important nuance: it is a signal, not a command. Search engines consider it alongside other evidence — internal links, redirects, sitemap contents, the actual content — and may select a different canonical if your signals conflict. That’s usually what’s happening when a page reports a canonical different from the one you declared.
Because of that, consistency across signals matters more than the tag alone. Link internally to the canonical version, list only canonical URLs in the sitemap, and redirect the variants you don’t want reachable at all.
Where a duplicate should not exist, a redirect is stronger than a canonical. Where both versions legitimately need to be reachable — a printable page, a filtered listing — a canonical is the right tool.
Use a redirect when
- The old URL should not be reachable
- URLs changed in a migration
- Consolidating HTTP to HTTPS or www variants
Use a canonical when
- Both URLs need to stay accessible
- Filtered or sorted listing variants
- Syndicated or near-duplicate content
<!-- The preferred version of this page --> <link rel="canonical" href="https://example.com/website-seo/technical-seo/" /> <!-- Self-referencing canonicals are normal and useful --> <link rel="canonical" href="https://example.com/page/" /> <!-- Robots meta tag: an indexing directive --> <meta name="robots" content="noindex,follow"> <meta name="robots" content="index,follow"> <meta name="robots" content="noindex,nofollow"> <!-- Useful values --> <!-- noindex — do not index this page --> <!-- nofollow — do not follow links on this page --> <!-- noarchive — do not show a cached copy --> <!-- Requires the page to be crawlable to be seen at all -->
# robots.txt — crawler instructions at the site root # https://example.com/robots.txt User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /search? # Do not block CSS or JavaScript needed for rendering Allow: /wp-content/uploads/ Sitemap: https://example.com/sitemap_index.xml # robots.txt is NOT access control. The file is public, # and a disallowed URL can still appear in results if # it is linked from elsewhere. # X-Robots-Tag — for non-HTML resources such as PDFs X-Robots-Tag: noindex
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://example.com/website-seo/technical-seo/</loc>
<lastmod>2026-03-08</lastmod>
</url>
<url>
<loc>https://example.com/website-seo/core-web-vitals/</loc>
<lastmod>2026-02-21</lastmod>
</url>
</urlset>
<!-- A sitemap index for larger sites -->
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<sitemap>
<loc>https://example.com/post-sitemap.xml</loc>
</sitemap>
<sitemap>
<loc>https://example.com/page-sitemap.xml</loc>
</sitemap>
</sitemapindex>DIRECTIVES
robots.txt, Robots Meta Tags and Sitemaps
robots.txt sits at the site root and tells crawlers which paths they should not request. Two things about it are widely misunderstood. It is not access control — the file is publicly readable and well-behaved crawlers choose to honour it. And Disallow does not reliably prevent indexing: a blocked URL linked from elsewhere can still appear in results, typically without a description, because the crawler was never allowed to fetch it and see your directives.
That leads to the most common mistake in this area. To keep a page out of the index, use noindex and leave it crawlable — blocking it in robots.txt means the noindex is never seen. The two tools do different jobs and are not interchangeable.
X-Robots-Tag provides the same directives through an HTTP header, which is how you apply them to PDFs, images and other non-HTML resources.
XML sitemaps help search engines discover URLs. They should contain canonical, indexable URLs that return 200, with accurate lastmod values where you maintain them. Listing redirected, blocked or non-canonical URLs sends mixed signals. A sitemap supports discovery; it guarantees nothing about indexing. For large sites, a sitemap index splits things into manageable files. An HTML sitemap is a different thing — a page for people to navigate — and not every site needs one.
STATUS & REDIRECTS
HTTP Status Codes and Redirects
Status codes are how your server tells crawlers and browsers what happened. Using the right one is a genuine technical SEO task, not a formality.
200
OK
The page was found and returned normally.
301
Moved Permanently
The URL has permanently moved. The standard choice for migrations and consolidation.
302
Found
A temporary redirect. Not automatically harmful — just use it when the move genuinely is temporary.
304
Not Modified
Nothing changed since the client’s cached copy.
404
Not Found
No resource at this URL. Perfectly normal for genuinely removed content.
410
Gone
Deliberately removed, with no replacement. A stronger signal than 404.
500
Server Error
Something failed on the server. Repeated 500s can reduce crawling.
503
Service Unavailable
Temporarily unavailable — the right response during planned maintenance.
404 errors are not automatically a problem. Content gets removed; returning 404 for it is correct and honest. What matters is whether anything still links to it. A 404 reached from a broken internal link is a problem worth fixing; a 404 from an old external link to content that no longer exists is simply accurate.
Redirect a removed URL only when there’s a genuinely equivalent replacement. Redirecting every 404 to the homepage is a bad habit — it’s confusing for users, hides the fact that content is missing, and often produces soft 404s.
A soft 404 is a page that returns 200 while effectively saying nothing is there. It wastes crawling and hides genuine problems. If content is gone, return 404 or 410.
Redirect chains — A to B to C to the destination — add latency for users and extra requests for crawlers. Point redirects at the final destination where you can. Redirect loops, where A goes to B and B goes back to A, prevent the page loading at all and need fixing immediately.
# Apache .htaccess — single URL redirect
Redirect 301 /old-page/ /new-page/
# Force HTTPS
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}/$1 [R=301,L]
# Nginx
return 301 https://$host$request_uri;
<?php
// PHP — a permanent redirect
header("Location: https://example.com/new-page/", true, 301);
exit;
// Returning a proper 404 rather than a soft 404
http_response_code(404);
// Planned maintenance
header("Retry-After: 3600");
http_response_code(503);SECURITY
HTTPS, Mixed Content and Site Security
HTTPS encrypts the connection between client and server. It protects users, browsers flag its absence, and it’s expected as a baseline — but it does not by itself improve rankings, and any claim that adding a certificate will lift traffic should be treated sceptically.
An HTTP to HTTPS migration is a URL change and needs to be treated as one: redirect every HTTP URL to its HTTPS equivalent, update internal links and canonicals rather than relying on redirects, regenerate the sitemap, and monitor indexation afterwards.
Mixed content happens when an HTTPS page loads resources over HTTP — images, scripts, stylesheets, fonts or embeds. Browsers block or warn about it, and it’s the usual reason a migrated site shows security warnings. Hard-coded HTTP URLs in database content are the common culprit.
Broader security work — keeping software updated, controlling access, monitoring for malware — belongs alongside technical SEO because a compromised site can be flagged, deindexed or filled with injected content. It doesn’t produce rankings; it prevents losing them. Hosting configuration plays into this too; see our hosting and domains guides.
EXPERIENCE
Mobile-First Indexing, Performance and Core Web Vitals
Mobile-first indexing means the mobile version of a page is what’s primarily used for indexing. The practical implication is content parity: if your mobile version hides content, drops structured data, removes internal links or serves smaller images, that reduced version is what gets processed. Responsive design usually handles this naturally, which is one reason it’s the common approach. See our responsive design guide.
Performance affects both users and crawling efficiency. The main contributors are server response time, resource sizes, render-blocking CSS and JavaScript, unoptimised images, font loading, caching configuration and CDN use. A CDN can deliver static assets from closer infrastructure — useful, but not an automatic ranking improvement.
The current Core Web Vitals are LCP (Largest Contentful Paint, how quickly the main content appears), INP (Interaction to Next Paint, how responsive the page feels to input) and CLS (Cumulative Layout Shift, how much content moves unexpectedly). INP replaced the older first-input metric. They’re part of a broader page experience picture, not a ranking dial — and field data from real users tells you more than a single lab score. More detail in our Core Web Vitals guide and website speed guides.
Accessibility overlaps with technical SEO without being the same discipline. Semantic HTML, meaningful alt text, clear heading hierarchy, real link elements and logical structure serve both. But accessibility goes much further — keyboard operation, focus management, screen reader behaviour — and should be treated on its own terms rather than as an SEO tactic. See website accessibility.
LCP
Largest Contentful Paint — when the main content becomes visible.
INP
Interaction to Next Paint — how responsive the page is to input.
CLS
Cumulative Layout Shift — how much content moves unexpectedly.
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "WebPage",
"name": "Technical SEO Guide",
"description": "A guide to crawling, indexing and website health.",
"url": "https://example.com/website-seo/technical-seo/"
}
</script>
<!-- Breadcrumbs -->
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@type": "BreadcrumbList",
"itemListElement": [
{ "@type": "ListItem", "position": 1, "name": "Home",
"item": "https://example.com/" },
{ "@type": "ListItem", "position": 2, "name": "Website SEO",
"item": "https://example.com/website-seo/" }
]
}
</script>
<!-- Markup must describe content actually on the page. -->
<!-- Valid markup makes a page ELIGIBLE for rich results. -->
<!-- It does not guarantee they will be shown. --><!-- Every version lists every version, including itself --> <link rel="alternate" hreflang="en" href="https://example.com/en/page/" /> <link rel="alternate" hreflang="hi" href="https://example.com/hi/page/" /> <link rel="alternate" hreflang="de" href="https://example.com/de/page/" /> <link rel="alternate" hreflang="x-default" href="https://example.com/en/page/" /> <!-- Language, or language plus region --> <!-- hreflang="en" — English, any region --> <!-- hreflang="en-gb" — English, United Kingdom --> <!-- hreflang="en-in" — English, India --> <!-- Return references must be reciprocal, or the --> <!-- annotations are likely to be ignored. -->
MARKUP & LANGUAGES
Structured Data and International SEO
Structured data describes what a page is about in a format search engines can parse, using the Schema.org vocabulary — most commonly in JSON-LD. It helps with understanding and can make a page eligible for rich results.
Eligible is the operative word. Valid markup does not guarantee rich results; search engines decide what to display. And markup must accurately describe content that’s actually on the page. Marking up reviews that don’t exist or products that aren’t sold is a manual-action risk, not a shortcut. Validate with the appropriate testing tools rather than assuming. More in our schema markup guide.
hreflang indicates language and regional variants of a page so the right version can be served. The part people get wrong is that it must be reciprocal: each version references every version including itself. A one-way annotation is typically ignored, which is why hreflang problems are so common and so easily missed.
For international URL structures, subdirectories (/en/, /hi/), subdomains and country-code domains each involve trade-offs in setup, maintenance and geographic signalling. None is universally best; the right answer depends on resources, infrastructure and how distinct the markets really are.
LARGE SITES
Pagination, Faceted Navigation and Crawl Budget
Pagination appears in product listings, blog archives and category pages. Paginated pages should be crawlable with real links, each page self-canonical rather than all pointing at page one, and ideally reachable without relying solely on infinite scroll. The older advice about rel="next" and rel="prev" being required is outdated — those annotations are no longer used by Google as an indexing signal.
Faceted navigation is where large sites most often create problems for themselves. Filters combine, and a few dozen options can generate an effectively unlimited number of URLs — a crawl trap that consumes resources on near-identical pages.
Handling it requires judgment rather than a rule. Some filter combinations have genuine search demand and deserve to be indexable landing pages; most don’t. Options include canonicalising variants to the unfiltered page, using noindex on low-value combinations, blocking parameter patterns in robots.txt, and not linking to combinations you don’t want crawled. Blanket-blocking all parameters without understanding the architecture can remove pages you wanted indexed — it’s worth mapping which combinations matter first.
Crawl budget is how much crawling a site receives over a period. It’s a real consideration for very large sites with hundreds of thousands of URLs. For a small site it is almost never the problem, and worrying about it there is a distraction from things that actually matter. Server log analysis shows which URLs were actually requested, by which bots, with what status codes and response times — the most direct evidence available, and again mainly worth the effort at scale.
# Only after mapping which combinations matter. # Blanket blocking can remove pages you wanted indexed. User-agent: * # Internal search results rarely need indexing Disallow: /search? Disallow: /*?s= # Session and tracking parameters Disallow: /*?sessionid= Disallow: /*?ref= # Low-value filter combinations, where identified Disallow: /*?*sort= # Keep valuable filtered landing pages crawlable, # for example a category people actually search for: Allow: /products/shoes/blue/
PLATFORMS
Technical SEO Across Platforms
WordPress
Check permalink structure, which sitemap is active (plugins and core can conflict), canonical handling, and whether tag, author and date archives should be indexed — on many sites they generate thin duplicates. Watch for a leftover discourage-search-engines setting after launch, and for plugin and theme performance. No plugin is universally best. See our CMS guides.
WooCommerce & E-commerce
Product variations, category filters, sorting and pagination generate URL variants quickly. Decide which are indexable, canonicalise the rest, handle out-of-stock and discontinued products deliberately rather than deleting them silently, and keep product structured data accurate. Internal search pages rarely need indexing.
React & JavaScript Apps
Rendering strategy is the decision that matters. Content needing to be found should be server-rendered or statically rendered; navigation should use real anchor elements; metadata should be set per route. See our React guide.
API-Driven Sites
Headless architectures separate content from rendering. The API itself is invisible to search engines — what matters is what the frontend produces. A failed API request that leaves a page empty is a rendering problem with real consequences. See our website APIs guide.
Semantic HTML
The foundation everywhere: a proper title and meta description, one H1 with a logical heading hierarchy, real anchor elements with descriptive text, images with alt attributes, and correct canonical and robots markup. See our HTML guide.
Small sites
Get the fundamentals right and stop: HTTPS, crawlable and indexable pages, a clean URL structure, an accurate sitemap, a sensible robots.txt, correct canonicals, mobile usability, reasonable performance, sound internal linking and basic structured data. Enterprise processes like log analysis and crawl budget management are unnecessary overhead here.
Large sites
Scale changes the priorities: crawl budget becomes real, faceted navigation needs a deliberate indexation strategy, sitemap indexes replace single files, log analysis becomes worthwhile, internal linking needs to be systematic rather than manual, and automated technical QA catches regressions that manual checking never will.
AUDITING
Technical SEO Audit and Checklist
A technical SEO audit systematically checks the factors affecting crawling, indexing, rendering, usability and performance. It’s a diagnostic exercise: find what’s broken or suboptimal, understand why it matters here, and decide what’s worth fixing.
The list below is a working checklist rather than a formal standard. Not every item matters on every site, and a small brochure site can reasonably skip half of it.
Crawlability
- robots.txt reviewed and not blocking important paths
- XML sitemap accurate and submitted
- Internal links reaching important pages
- Crawl errors investigated
- Redirect chains shortened
- Crawl traps identified
Indexability
- Unintended noindex directives found
- Canonicals correct and consistent
- Duplicate URLs addressed
- Status codes appropriate
- Important content present in rendered HTML
Structure
- URL structure consistent and readable
- Site architecture logical
- Orphan pages identified
- Crawl depth reasonable for important pages
- Pagination crawlable
- Parameter handling deliberate
Performance
- LCP, INP and CLS reviewed with field data
- Server response time measured
- Images optimised and sized
- CSS and JavaScript reviewed
- Caching configured
- CDN and compression in place
Security
- HTTPS site-wide and enforced
- Mixed content resolved
- Software and plugins updated
- Forms submitted securely
Mobile
- Responsive layout working
- Content parity with desktop
- Navigation usable on small screens
- Viewport configured
Structured data
- Schema present where appropriate
- JSON-LD valid
- Markup matches visible content
- No fabricated entities
International
- hreflang reciprocal and complete
- Language and region codes correct
- URL structure consistent across versions
CRAWLABILITY [ ] robots.txt reviewed and not blocking important paths [ ] XML sitemap accurate and submitted [ ] Internal links reaching important pages [ ] Crawl errors investigated [ ] Redirect chains shortened [ ] Crawl traps identified INDEXABILITY [ ] Unintended noindex directives found [ ] Canonicals correct and consistent [ ] Duplicate URLs addressed [ ] Status codes appropriate [ ] Important content present in rendered HTML STRUCTURE [ ] URL structure consistent and readable [ ] Site architecture logical [ ] Orphan pages identified [ ] Crawl depth reasonable for important pages [ ] Pagination crawlable [ ] Parameter handling deliberate PERFORMANCE [ ] LCP, INP and CLS reviewed with field data [ ] Server response time measured [ ] Images optimised and sized [ ] CSS and JavaScript reviewed [ ] Caching configured [ ] CDN and compression in place SECURITY [ ] HTTPS site-wide and enforced [ ] Mixed content resolved [ ] Software and plugins updated [ ] Forms submitted securely MOBILE [ ] Responsive layout working [ ] Content parity with desktop [ ] Navigation usable on small screens [ ] Viewport configured STRUCTURED DATA [ ] Schema present where appropriate [ ] JSON-LD valid [ ] Markup matches visible content [ ] No fabricated entities INTERNATIONAL [ ] hreflang reciprocal and complete [ ] Language and region codes correct [ ] URL structure consistent across versions
PRIORITISATION
Prioritising Issues and Avoiding Common Mistakes
An audit that produces two hundred findings and no order of attack isn’t useful. Severity is a starting point, but the real prioritisation depends on business impact, how many URLs are affected, how important those pages are, how hard the fix is and how risky it is. A critical-sounding issue on a page nobody visits can wait; a medium issue across every product page usually can’t.
Critical
Anything preventing access, crawling or indexing of important pages — a site-wide noindex, a robots.txt blocking key sections, widespread server errors, redirect loops on live URLs.
High
Problems affecting large sections: broken canonicals across a template, HTTPS inconsistencies, mobile content missing, rendering failures on a whole page type.
Medium
Issues affecting usability, architecture or efficiency: redirect chains, orphan pages, inconsistent URLs, inefficient pagination, performance regressions.
Low
Minor improvements with limited immediate impact: small markup refinements, tidying edge-case URLs, cosmetic sitemap issues.
Common technical SEO mistakes
Blocking Important Pages
robots.txt disallowing sections that should be crawled.
Accidental noindex
A staging directive that survived launch.
Incorrect Canonicals
Pointing at the wrong URL, or all at the homepage.
Redirect Loops
Pages that never load.
Redirect Chains
Several hops where one would do.
Broken Internal Links
404s reached from your own navigation.
Orphan Pages
Published, then never linked to.
Duplicate URLs
The same content at several addresses.
Poor Architecture
Important pages buried behind filters.
Wrong Sitemap URLs
Redirected, blocked or non-canonical entries.
HTTP/HTTPS Inconsistency
Both versions reachable and linked.
Mixed Content
Insecure resources on secure pages.
Rendering Failures
Content that never appears when scripts fail.
Missing Mobile Content
Reduced mobile version treated as the primary one.
Broken hreflang
Non-reciprocal or invalid annotations.
Runaway Filter URLs
Faceted navigation generating unlimited URLs.
Slow Server Response
Delays affecting users and crawling alike.
Invalid Structured Data
Markup that doesn't match the page.
Ignoring Redesigns
URLs changed with no redirect mapping.
No Monitoring
Problems discovered months later.
LAUNCH & MIGRATION
New Websites, Migrations and Monitoring
Launches and migrations are where technical SEO damage most often happens, and almost always through omission rather than error.
For a new website, the single most common mistake is leaving staging directives in place — a site-wide noindex or a blocking robots.txt that nobody remembers to remove. It’s worth checking on launch day and again a week later.
For migrations — whether a domain change, HTTPS move, URL restructure, CMS change or redesign — the work is mapping. Inventory every existing URL, decide its destination, implement redirects, and test before launch rather than after. Update internal links to point at new URLs directly instead of relying on redirects to do it forever.
Redesigns deserve specific caution because they’re often treated as visual projects. URLs change, pages get dropped, templates lose canonicals or gain noindex, and performance regresses — all without anyone considering it an SEO change.
Monitoring is what makes the rest hold. Watch indexation, crawl and server errors, sitemap status, redirect problems, Core Web Vitals, structured data validity, broken links and organic landing pages. Technical SEO isn’t a project that finishes; sites change constantly and regressions arrive quietly.
New website checklist
- Domain and www/non-www decided and enforced
- HTTPS configured and forced
- Staging noindex and robots blocks removed
- Clean, consistent URL structure
- XML sitemap generated and submitted
- robots.txt reviewed
- Canonical tags correct
- Custom 404 page returning a real 404
- Redirects mapped from any previous URLs
- Responsive layout tested on real devices
- Core Web Vitals measured
- Structured data validated
- Internal linking in place
- Titles and meta descriptions set
- JavaScript-rendered content verified
- Search Console and analytics connected
- Backups and security reviewed
Migration checklist
- Complete inventory of existing URLs
- Old-to-new URL mapping documented
- Redirects implemented and tested before launch
- Canonicals updated to new URLs
- Internal links updated, not just redirected
- Sitemap regenerated and resubmitted
- robots.txt reviewed for the new structure
- Structured data carried across
- HTTPS and mixed content verified
- Mobile parity confirmed
- JavaScript rendering retested
- Performance compared against the old site
- 404 monitoring active
- Indexation monitored for several weeks
BEFORE LAUNCH [ ] Domain and www/non-www decided and enforced [ ] HTTPS configured and forced [ ] Staging noindex and robots blocks removed [ ] Clean, consistent URL structure [ ] XML sitemap generated and submitted [ ] robots.txt reviewed [ ] Canonical tags correct [ ] Custom 404 page returning a real 404 [ ] Redirects mapped from any previous URLs [ ] Responsive layout tested on real devices [ ] Core Web Vitals measured [ ] Structured data validated [ ] Internal linking in place [ ] Titles and meta descriptions set [ ] JavaScript-rendered content verified [ ] Search Console and analytics connected [ ] Backups and security reviewed MIGRATION [ ] Complete inventory of existing URLs [ ] Old-to-new URL mapping documented [ ] Redirects implemented and tested before launch [ ] Canonicals updated to new URLs [ ] Internal links updated, not just redirected [ ] Sitemap regenerated and resubmitted [ ] robots.txt reviewed for the new structure [ ] Structured data carried across [ ] HTTPS and mixed content verified [ ] Mobile parity confirmed [ ] JavaScript rendering retested [ ] Performance compared against the old site [ ] 404 monitoring active [ ] Indexation monitored for several weeks
TOOLS
Technical SEO Tools
Categories rather than rankings. Most technical SEO work uses several of these together, and no single tool covers everything. More in our SEO tools guide.
Search Engine Tools
Google Search Console and Bing Webmaster Tools show how a search engine actually sees your site: indexing status, sitemap processing, crawl issues, Core Web Vitals field data and search performance. URL inspection answers whether a specific page is indexed and why.
Crawlers
Desktop and cloud crawlers simulate a crawl of your site, surfacing status codes, redirect chains, duplicate titles, canonical conflicts and orphan candidates. The closest thing to seeing the site as a crawler does.
Performance Tools
PageSpeed Insights combines lab and field data; Lighthouse runs audits locally. Lighthouse scores are diagnostic signals, not ranking scores — treat them as a list of things to investigate rather than a grade.
Structured Data Validators
The Rich Results Test and Schema validators check whether markup is valid and which rich result types a page is eligible for. Eligibility is not a guarantee of display.
Browser DevTools
The Network tab for status codes and resource sizes, Console for JavaScript errors, Elements for the rendered DOM, Performance for profiling, and device emulation for mobile checks.
Log Analysis
Server logs show what bots actually requested, with status codes and timings. The most direct evidence of crawl behaviour available, and mainly worth the effort on large sites.
REFERENCE
Technical SEO Cheat Sheet and Learning Path
A quick reference for the directives, codes and metrics covered above, followed by a sensible order to learn them in.
robots.txtCrawler access instructions at the site rootnoindexDirective keeping a crawlable page out of the indexX-Robots-TagRobots directives via HTTP header, for non-HTML filesXML sitemapSupports URL discovery; no indexing guaranteerel=canonicalPreferred URL signal, not a command301Permanent redirect302Temporary redirect404Resource not found410Deliberately removed503Temporarily unavailableHTTPSEncrypted connection; expected baselinehreflangLanguage and region targeting; must be reciprocalJSON-LDStructured data formatLCPLargest Contentful PaintINPInteraction to Next PaintCLSCumulative Layout ShiftBEGINNER
- SEO fundamentals
- Crawling
- Indexing
- URL structure
- robots.txt
- XML sitemaps
INTERMEDIATE
- Canonicalisation
- Redirects
- Internal linking
- JavaScript SEO
- Structured data
- Core Web Vitals
- Mobile-first indexing
ADVANCED
- Faceted navigation
- Crawl budget
- Log analysis
- International SEO
- Large site architecture
- Automated monitoring
A diagnostic sequence
When a page isn’t appearing in search, work through these in order. The first “no” is usually your answer.
FAQ
Technical SEO: Frequently Asked Questions
Short answers to the questions people ask most about crawling, indexing, directives and technical site health.
NEXT TOPICS
Continue Learning Website SEO
On-Page SEO
Learn how page content, headings, keywords and search intent work together.
Core Web Vitals
Understand LCP, INP and CLS and how to measure them.
Schema Markup
Learn how structured data communicates information to search engines.
Image SEO
Learn how images affect discoverability and performance.
Local SEO
Learn technical and local search considerations.
SEO Tools
Explore tools for technical SEO analysis and auditing.
Website Speed
Understand performance, caching and delivery.
Website Accessibility
Learn where accessibility and technical SEO overlap.
Build a Stronger Technical SEO Foundation
Make your website easier to crawl, understand and maintain with a structured approach to technical SEO, indexing, performance, architecture and website health.