Editing robots.txt to determine what ends up in search engines

 Shopify hired many top-notch SEO people to help expand and improve the platform’s capabilities. One of the things that came out of that team pretty quickly was the new ability to edit the robots.txt file. Having complete control over the robots.txt gives you more ways to control what Google can and can’t do on your store. This takes away one of the most significant issues that SEOs have with Shopify.

The robot.txt file is one of the crucial tools that you can use to optimize your online store or your website. It gives a way to tell Google how you want them to crawl the site. Ideally, you use this to prevent search engines from crawling less critical pages or sections of your site. For huge ecommerce sites, this is very important.

You can find your robots.txt file on https://example.com/robots.txt. Here’s what a standard robots.txt of Shopify looks like:

User-agent: *
Disallow: /admin
Disallow: /cart
Disallow: /orders
Disallow: /checkout
Disallow: /54914154724/checkouts
Disallow: /54914154724/orders
Disallow: /carts
Disallow: /account
Disallow: /collections/*sort_by*
Disallow: /*/collections/*sort_by*
Disallow: /collections/*+*
Disallow: /collections/*%2B*
Disallow: /collections/*%2b*
Disallow: /*/collections/*+*
Disallow: /*/collections/*%2B*
Disallow: /*/collections/*%2b*
Disallow: /blogs/*+*
Disallow: /blogs/*%2B*
Disallow: /blogs/*%2b*
Disallow: /*/blogs/*+*
Disallow: /*/blogs/*%2B*
Disallow: /*/blogs/*%2b*
Disallow: /*?*oseid=*
Disallow: /*preview_theme_id*
Disallow: /*preview_script_id*
Disallow: /policies/
Disallow: /*/*?*ls=*&ls=*
Disallow: /*/*?*ls%3D*%3Fls%3D*
Disallow: /*/*?*ls%3d*%3fls%3d*
Disallow: /search
Disallow: /apple-app-site-association

Shopify automatically disallows crawling for several parts of the store. It does this pretty well, and most users probably don’t have to touch this file. But for more complex or expansive sites, it might make sense to add some rules to ensure that some more advanced features don’t generate duplicate content in the search results.

You can now edit the robots.txt liquid file by going to your Online Store admin page. From there, go to the theme section and select Actions > Edit code. Find the template section and click Add new template. Click the dropdown and select robots.txt from at the bottom. Click create template, and you can start editing.