Robots.txt is used by all Search engines (Google, Bing and other big search engines) to identify what to crawl on your site. A large number of sites have a bad Robots.txt file. Most of the time site owners have no idea that they have a bad Robots.txt file and that they need to edit it.
This blog will tell you why you need a clean Robots.txt file, what are the best practices to generate a Robots.txt file, and how to edit the Robots.txt file in Shopify.
But first let’s understand what Robots.txt file actually is and its significance in SEO.
What is Robots.txt?
You’re probably familiar with that little bit of code at the bottom of your website, the one that says things like “no-index” and “no-follow”. This code is known as “Robots.txt” and it is something that every business should have. This mini-coding language prevents search engine crawlers, like Google, from finding certain parts of your website. This prevents bots from indexing content that you don’t want search engines to find. If you have a few things you don’t want search engines to see (like your contact information or an about page) you can use Robots.txt to keep crawlers out of those areas.
A Robots.txt is a file that allows you to control how search engines and web crawlers interact with your website. A good Robots.txt file is one that is easy to read, simple to follow and is quick to load. A bad Robots.txt file makes it difficult for search engines and web crawlers to crawl your website properly and as a result, can get your site removed from search results.
A robots.txt file is a file placed on your server to tell search engine crawlers what to do. The file is an easy way to control a crawler’s access to your website. Originally designed to prevent crawlers from accessing private parts of a website (such as employees only resources), robots.txt files are now more commonly used to create a barrier between search engines and some parts of a website.
Why Should You Use Robots.txt?
There are a number of reasons why you should use robots.txt. The first is to block bots from indexing your site, which will cut down on the bandwidth load on your site and the strain on your server. This is especially important if you have a low-traffic or high-volume site. The second reason to use robots.txt is to block spiders from downloading content from your site. While Google is not a major bandwidth guzzler, the more spiders that hit your server all at once, the more stress you can put on it. It’s important to remember that Googlebot is not the culprit behind all of this stress. Rather, it’s the other, less popular spiders that are running rampant and getting a hold of your site.
A good example of why you would want to use the Robots.txt file is if you have a password-protected page on your website that includes sensitive information, like contact information or customer data. If you want search engines like Google to crawl your entire website but not find this sensitive information, you can use a Robots.txt file to prevent them from crawling the page.
How to Create a Robots.txt file ?
It is super easy to create a Robots.txt file. All you need is to create a text file in notepad , wordpad or any other text editor. Follow the below steps :-
- Open any text editor and save the file as ‘robots,’ all lowercase, being sure to choose .txt as the file type extension (in Word, choose ‘Plain Text’).
- Then, in your document, add the following two lines of text:
Robots or search engine spiders are referred to as ‘user-agents.’ The above line includes all spiders, as indicated by the asterisk (*). The Disallow line in this case has no files or folders mentioned, meaning that any directory on your site can be accessed.
- One of the robots.txt choices is to block search engine crawlers from your entire site. To accomplish this, add the following two lines to the file:
- Your robots.txt file will look like this if you want to keep spiders out of particular portions of your site:
The preceding three lines inform all robots that they are not permitted to access the about and database folders, or subdirectories. Keep in mind that each Disallow line can only have one file or folder. You are free to use as many Disallow lines as you want.
Once created, you can upload the file in the root directory in case of custom-coded websites or wordpress websites. In case of Shopify, follow the steps in the next section.
How to Edit Robots.Txt File in Shopify
In June 2021, Shopify rolled out a new feature making it possible for everyone to edit the robots.txt file from now on.
The following changes can be made to the robots.txt file by site owners:
- Allow or prevent crawling of specific URLs.
- Crawl-delay rules can be introduced for specific crawlers.
- Certain crawlers can be blocked.
So let’s see how to edit the robots.txt file in Shopify. Follow the below steps:-
- Go to Online Store > Themes in your Shopify admin panel
- Then, under Actions, select Edit Code
- Select Add a new template, and then choose robots.txt
- Click Create template
- Make the adjustments to the default template that you want.
- Save the adjustments to your published theme’s robots.txt.liquid file
Site owners can also erase the template’s contents and replace them with plain text rules, though Shopify does not encourage this because it will not be automatically updated.
You can also watch our video guide to edit robots.txt file in Shopify.
In this article, we discussed the steps that you need to keep in mind about how to edit the Robots.txt file for Shopify. The thing to remember is not to overdo it, as it may affect the website negatively. Also, Editing the robots.txt.liquid file is an unsupported adjustment, according to Shopify, thus they can’t help you with it.
About Parkhya IT Solutions
Parkhya Solutions Pvt. Ltd. is a team of more than 90+ young and active management, marketing, and information technology professionals with extensive experience in their respective industries. Our staff adheres to project management disciplines while implementing all aspects of project development.
If you want to set up your own shopify store and be your own boss selling products online, we can help. Our team of Shopify experts will guide you through the process of setting up your online store with ease.
Sign up and start your 14 day free trial today !!