robots.txt: Syntax, Common Mistakes, and Next.js Implementation

For over two decades, the Robots Exclusion Protocol was an informal consensus standard drafted in 1994. In September 2022, the Internet Engineering Task Force (IETF) codified the protocol as RFC 9309: Robots Exclusion Protocol. This standard defines exact syntactic rules, URI parsing behaviors, character encodings (UTF-8), and precedence algorithms that all modern search crawlers—including Googlebot, Bingbot, and Applebot—must adhere to.

As outlined in our technical SEO pre-launch checklist, any syntax ambiguity or unescaped character in /robots.txt can either shut down indexing across your entire digital estate or expose sensitive internal administration panels to public search results.

The single most frequent mistake in search architecture is confusing Disallow in robots.txt with the noindex robots meta directive. They solve two completely different engineering problems, as summarized below:

Under RFC 9309, path matching follows deterministic criteria where specificity beats file order. Here are the core syntax constructs every engineer should know:

# Standard robots.txt syntax example
User-agent: *
Allow: /
Allow: /blog/
Disallow: /api/
Disallow: /dashboard/
Disallow: /search?*
Disallow: /*.json$

# Target specific bot rules
User-agent: Googlebot-Image
Disallow: /private-assets/

# Absolute sitemap reference
Sitemap: https://example.com/sitemap.xml

Rather than maintaining a static public/robots.txt file that is prone to hardcoded errors and environment leaks, Next.js App Router provides native dynamic generation via src/app/robots.ts. This allows developers to use TypeScript, inject canonical site URLs from centralized configuration, and conditionally disallow staging branches.

import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  const isProduction = process.env.NEXT_PUBLIC_SITE_ENV === "production";
  const baseUrl = process.env.NEXT_PUBLIC_APP_URL || "https://example.com";

  // Prevent search engines from crawling staging, preview, or development deployments
  if (!isProduction) {
    return {
      rules: {
        userAgent: "*",
        disallow: "/",
      },
    };
  }

  return {
    rules: [
      {
        userAgent: "*",
        allow: ["/", "/blog", "/blog/", "/glossary", "/glossary/"],
        disallow: [
          "/dashboard/",
          "/api/",
          "/auth/",
          "/login",
          "/signup",
          "/blog?*", // Prevent crawl budget waste on client search facets
        ],
      },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
    host: baseUrl,
  };
}

Auditing crawl health across thousands of production sites reveals that simple syntax mistakes often produce outsized indexing regressions:

Before releasing major updates or framework migrations, review this verification checklist to ensure search crawler accessibility: