{"id":1338,"date":"2026-09-21T08:00:00","date_gmt":"2026-09-21T13:00:00","guid":{"rendered":"https:\/\/www.seomos.com\/?p=1338"},"modified":"2026-09-10T02:42:26","modified_gmt":"2026-09-10T07:42:26","slug":"robots-txt","status":"publish","type":"post","link":"https:\/\/www.seomos.com\/en\/blog\/robots-txt\/","title":{"rendered":"Robots.txt: How to configure it without blocking important pages"},"content":{"rendered":"
\n

Robots.txt is a text file located in the root of a host that tells compatible crawlers which routes they can request.<\/strong> Its primary function is to manage crawling, not to remove pages from the index or protect information. A blocked URL may appear in results if other sites link to it; for private content, it corresponds to authentication, and to exclude a crawlable page, it may correspond to... noindex<\/code>.<\/p>\n<\/div>\n

A single incorrectly implemented line can block a section or the entire site from search engines. That's why robots.txt should be treated like a production configuration: with a target, owner, tests, history, and recovery plan.<\/p>\n

What is the robots.txt file?<\/h2>\n

It implements the Robots Exclusion Protocol. It groups rules for one or more agents and path patterns. Recognized search engines usually respect them, but a malicious actor can ignore them. The file is public, and anyone can read the specified paths.<\/p>\n

If there is no rule blocking a URL, access is considered allowed. Having an empty file or a rule Allow: \/<\/code> It does not improve positioning; it only makes access explicit.<\/p>\n

Where it should be and what applies<\/h2>\n

It is served as https:\/\/ejemplo.com\/robots.txt<\/code>. It doesn't work in \/folder\/robots.txt<\/code>. Its scope is limited to protocol, host, and port: the file of www<\/code> It does not control a subdomain, and HTTPS does not inherit rules from HTTP.<\/p>\n

Each relevant host needs its own decision. Check the canonical domain, subdomains, staging, and CDN. The file must respond publicly and in UTF-8 text; HTTP redirects and errors can alter how it's interpreted.<\/p>\n

Basic syntax<\/h2>\n

User-agent<\/code> Start a group. Disallow<\/code> restricts a route; Allow<\/code> can open an exception within a locked pattern. Sitemap<\/code> declares an absolute URL for the sitemap. The character #<\/code> Start a comment.<\/p>\n

Routes are case-sensitive. Google supports wildcards as defined in its documentation, but other crawlers may interpret them differently. Design for the standard first and validate with the preferred crawlers.<\/p>\n

\"Rutas
The rule should be evaluated against the agent and the complete route that is actually being requested.<\/figcaption><\/figure>\n

Minimal example with commentary<\/h2>\n
User-agent: * Disallow: \/internal-search\/ Allow: \/ Sitemap: https:\/\/www.example.com\/sitemap.xml<\/code><\/pre>\n
The example allows the site but blocks a specific path. It should not be copied without reviewing the site architecture. If the internal search engine returns useful pages or the path changes, the rule would have a different effect.<\/p>\n
Allow: \/<\/code> It is redundant when there is no broader lock-up. Clarity matters: a few justified rules are safer than an ownerless historical collection.<\/p>\n
Crawling is not indexing<\/h2>\n
Crawling means requesting a resource. Indexing means processing it so it can appear in search results. Robots.txt controls the first step. Google warns that a blocked URL can still be indexed without a description if discovered through links.<\/p>\n
If you need to remove a public results page, allow tracking and delivery. noindex<\/code> until it is processed, or remove it with an appropriate status. If it contains private information, require authentication; do not rely on voluntary instruction.<\/p>\n
Robots.txt does not canonicalize<\/h2>\n
A canonical tag requires the search engine to be able to read the HTML or header. Blocking the variant can prevent that signal from being processed. For duplicates, use canonical tag<\/a>, redirects, consistent links and sitemap as appropriate.<\/p>\n
Robots can reduce access to low-value spaces, but they don't replace architecture. First, they prevent the creation of infinite combinations, and then they control the inevitable.<\/p>\n
Robots.txt is not security<\/h2>\n
The file publicly advertises paths and does not enforce them on all clients. Panels, documents, and private environments require authentication, authorization, and server configuration. Removing a path from the file does not make it secret.<\/p>\n
Also check backups, storage, and shared URLs. If there was exposure, correct access and assess the incident; editing robots.txt does not revoke existing copies or links.<\/p>\n
CSS, JavaScript, and images<\/h2>\n
Blocking necessary resources can prevent a search engine from rendering and understanding your page. Don't block entire folders of themes or scripts without checking what the templates consume. Google recommends allowing resources relevant to rendering.<\/p>\n
For images or video, the rules can influence file crawling and resource visibility, but the containing page is a separate object. Separate the objectives and test both.<\/p>\n
Parameters, filters and ecommerce<\/h2>\n
Ecommerce platforms generate order, filters, search capabilities, and tracking. Each parameter is categorized by content changes, demand, and links. Robots can restrict access to vast areas, but if they are already indexed, blocking them can freeze outdated signals.<\/p>\n
Reduce URL generation, normalize URLs, and link selected facets. Align canonical tags, sitemap, and navigation. Don't use a global rule for all URLs. ?<\/code> without checking pagination and legitimate functions.<\/p>\n
Robots.txt in WordPress<\/h2>\n
WordPress can generate a virtual file, and SEO plugins can modify it. Check the public search results, not just the editor. The option to discourage search engines may result in broad blocking or meta robots blocking, depending on the version and configuration.<\/p>\n
In production, verify after migrating, cloning, or changing the domain. Staging environments must be protected with access controls, not just robots.txt controls. Never replace server files without knowing if a virtualized layer or CDN is in place.<\/p>\n
When can a lock be useful?<\/h2>\n