Robots Exclusion Protocol

Why It Matters

robots.txt prevents web scrapers and crawlers from overwhelming internal server resources with unnecessary requests to administrative endpoints, internal search queries, or staging environments. However, blocking a page in robots.txt does NOT prevent it from being indexed if external backlinks point to it; only noindex meta tags prevent indexing.

Robots.txt Standard Syntax Directives (RFC 9309)

Practical Implementation Example

Next.js MetadataRoute generates a valid, dynamic robots.txt file blocking private application routes while advertising the sitemap location to all search engines.

import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: ['/api/', '/dashboard/', '/admin/'],
      },
    ],
    sitemap: 'https://example.com/sitemap.xml',
  };
}