When your internal search results are indexed by Google, it can create index bloat that wastes your crawl budget. To make the most of Googlebot’s time on your website and increase the chances of your high-value pages ranking for your search terms, we recommend setting up your robots.txt file to block search crawlers from accessing your internal search results.
What Are Internal Search Results?
The search function on your website creates an internal search results page each time it’s used. These search results generally have a URL featuring a question mark:
https://victorious.com/blog/?s=robotsSearch crawlers may index and surface these pages in Google search results if you don’t block them via a robots.txt directive.
Why Are My Internal Search Results Showing Up in SERPs?
Google doesn’t differentiate between your internal search result pages and any other page on your website. If they aren’t blocked from crawlers, Google will index and serve them in search results.
Isn’t Having More of My Pages Show Up in Search Results Better?
Allowing indexing of every page on your site can lead to low-value pages appearing in search results. These pages may compete against more important pages for your target keywords and negatively impact your SEO progress.
For some industries, having your internal search results indexed may be beneficial. For example, if you have a job posting website, your SEO strategy may include having your internal search results indexed to drive more traffic.
Otherwise, we recommend blocking search crawlers from indexing your internal search results in your robots.txt file.
Why Should I Use a Robots.txt Directive Instead of a Robots Meta Tag To Block Internal Search Results?
A robots.txt file tells search bots they shouldn’t access or index a particular directory. It generally lives at:
yourURL.com/robots.txt
A robots meta tag, also known as a noindex tag, tells search bots that they can crawl a page but that they shouldn’t index it. It lives in the header of the specific page it applies to.
Since internal search results will have different URLs — and the URLs are auto-generated depending on the search — you need to use a robots.txt to effectively block access to all possible URLs your search function may create.
How Do I Edit My Robots.txt To Block Internal Search Results?
Your web developer should edit your robots.txt file to add a line of code blocking access to your internal search results.
The line of code should read:
Disallow: /*?*
This code will tell search crawlers they shouldn’t access URLs that include a question mark.
Can Victorious Edit My Robots.txt File for Me?
Depending on your CMS, we may be able to edit your robots.txt file to block the indexation of internal search results. Currently, we can implement this recommendation on HubSpot, Magento, Shopify, Umbraco, and WordPress. We cannot implement this recommendation on SquareSpace.Â
If you use a different CMS, speak to your Account Manager to determine whether we can implement this audit recommendation for you.
