The Developer's Technical SEO Audit Checklist: 12 Checks Before Launch

Search engines first fetch /robots.txt before discovering pages. A common production misconfiguration is carrying over staging directives (such as Disallow: /) into production, which completely blocks search engines from crawling the site. Conversely, failing to disallow internal API endpoints, rate-limited webhook handlers, and private dashboard pages wastes crawler budget and risks indexing sensitive URLs.

import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://example.com';
  return {
    rules: [
      {
        userAgent: '*',
        allow: ['/', '/blog', '/blog/*'],
        disallow: ['/dashboard/', '/api/', '/auth/'],
      },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
  };
}

Your sitemap.xml is the authoritative index provided to search engines. Every URL listed in sitemap.xml must be absolute (including the https protocol), canonical, and return an HTTP 200 OK status. Never include redirected URLs, duplicate aliases, authenticated dashboard URLs, or staging domains.

import type { MetadataRoute } from 'next';

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const origin = process.env.NEXT_PUBLIC_SITE_URL || 'https://example.com';
  const articles = await getPublishedArticles();

  const blogEntries = articles.map((post) => ({
    url: `${origin}/blog/${post.slug}`,
    lastModified: new Date(post.updatedAt),
    changeFrequency: 'weekly' as const,
    priority: 0.7,
  }));

  return [
    { url: origin, lastModified: new Date(), changeFrequency: 'daily', priority: 1.0 },
    { url: `${origin}/blog`, lastModified: new Date(), changeFrequency: 'daily', priority: 0.8 },
    ...blogEntries,
  ];
}

Canonical tags tell search engines which URL represents the single authoritative source of truth. Without canonical tags, search engines may treat tracking parameters (e.g. ?utm_source=twitter or ?ref=producthunt), trailing slash variations (/blog vs /blog/), and protocol variants (http vs https) as separate duplicate pages, diluting page authority.

export async function generateMetadata({ params }: Props): Promise<Metadata> {
  const { slug } = await params;
  const post = getArticleBySlug(slug);
  if (!post) return {};

  return {
    title: post.title,
    description: post.description,
    alternates: {
      canonical: `/blog/${post.slug}`,
    },
  };
}

In Next.js App Router, omitting `metadataBase` in root layout causes Next.js to log build warnings and prevents relative metadata URLs (like Open Graph images or canonical paths) from resolving to valid absolute URLs. Setting `metadataBase: new URL(origin)` guarantees that social cards and search engines receive fully qualified URLs.

export const metadata: Metadata = {
  metadataBase: new URL(
    process.env.NEXT_PUBLIC_SITE_URL || 'https://site-sensei-virid.vercel.app'
  ),
  title: {
    default: 'SiteSensei — Free AI-Powered Website Audit Tool',
    template: '%s | SiteSensei',
  },
};

Search engines should only index public marketing pages and published editorial content. User workspace routes (/dashboard), guest audit executions, and authentication forms must be protected with `robots: { index: false, follow: false }` to avoid indexing temporary user audits or leaking customer URLs into search results.

export const metadata: Metadata = {
  robots: {
    index: false,
    follow: false,
    googleBot: {
      index: false,
      follow: false,
    },
  },
};

While Googlebot executes JavaScript, it renders pages in a two-stage process. If primary headings, article bodies, or navigational links are rendered only after client-side state loads (`useEffect` or `fetch` in client components), initial indexing can be delayed or incomplete. Render core content in React Server Components by default.

# Verify raw server HTML contains your H1 and core text without running JavaScript
curl -sL https://yoursite.com/blog/my-post | grep -E "(<h1|<article|rel=\"canonical\")"

Having both `http://yoursite.com`, `https://yoursite.com`, and `https://www.yoursite.com` return 200 OK causes search engines to split page ranking signals across three domains. Enforce a single primary origin and configure your hosting provider or edge middleware to redirect secondary variants with a single 308/301 hop.

# Check redirect behavior with curl:
curl -I http://yoursite.com
# Expected: HTTP/1.1 301 Moved Permanently -> https://yoursite.com

Search engines discover your site structure through internal links. Links pointing to nonexistent pages (like placeholder company pages or unreleased tools) trigger 404 errors during crawling, leak link equity, and create poor user experiences. Use semantic `<Link href="...">` tags with descriptive anchor text rather than generic 'click here' labels.

// Avoid: <a href="/tools/seo-checker"> (returns 404 if tool page doesn't exist)
// Correct: Deep-link directly to working tool anchor on the page
<Link
  href="/#audit-input"
  className="text-sm text-slate-400 hover:text-white"
>
  Run Instant SEO Audit
</Link>

Google's Core Web Vitals directly evaluate user experience signals. Ensure your hero banner uses `next/image` with `priority` and `fetchpriority="high"` to optimize Largest Contentful Paint (LCP). Reserve explicit image dimensions or CSS aspect ratios to eliminate Cumulative Layout Shift (CLS). Keep event callbacks lightweight to ensure responsive Interaction to Next Paint (INP).

<Image
  src="/images/hero-bg.png"
  alt=""
  fill
  priority
  quality={85}
  className="object-cover object-top"
/>

Structured data helps search engines interpret the entities, authors, and purpose of your application. Validate that all schema URLs match your active canonical domain, ensure logo assets resolve to real image files, and include BreadcrumbList and Article schemas on editorial content.

const articleSchema = {
  '@context': 'https://schema.org',
  '@type': 'TechArticle',
  headline: post.title,
  description: post.description,
  datePublished: post.publishedAt,
  dateModified: post.updatedAt,
  author: {
    '@type': 'Organization',
    name: post.author.name,
  },
};

<script
  type="application/ld+json"
  dangerouslySetInnerHTML={{ __html: JSON.stringify(articleSchema) }}
/>

While security headers are not direct ranking algorithms, website security and user safety are critical quality signals. Configuring Strict-Transport-Security (HSTS), X-Content-Type-Options (nosniff), and X-Frame-Options prevents clickjacking, MIME-type sniffing, and protocol downgrade attacks.

const securityHeaders = [
  { key: 'Strict-Transport-Security', value: 'max-age=63072000; includeSubDomains; preload' },
  { key: 'X-Content-Type-Options', value: 'nosniff' },
  { key: 'X-Frame-Options', value: 'SAMEORIGIN' },
  { key: 'Referrer-Policy', value: 'origin-when-cross-origin' },
];

Manual spot-checks frequently overlook subtle regressions like broken viewport meta tags, missing ARIA form labels, unminified assets, or missing image alt text. Running an automated audit engine catches cross-disciplinary defects in seconds.