Before starting this tutorial, we will define what is a robot.txt file. This file contains some lines of code which will help the crawlers to index your website on the search result. Along with it, you can easily block or restrict the page from web crawlers so that it will not be seen in the search result. Search engines first check your robot.txt file before web pages present in your website.
Format of Robot.txt file
Here I will tell you the meaning of each term.
This code is just for Google Adsense helpful for serving the related ads on your page. If you are an Adsense user then its useful for you. Keep it as it is.
In the default setting our blogs links are restricted to indexed by search crawlers that mean the web crawlers will not index our label pages link because of this code
that means the links having the keyword search just after the domain name will be ignored
This above is an example of the link of label affiliate, if we remove Disallow:/seach from the code then crawlers will access your entire blog pages. You can disallow the particular page to be indexed by the crawlers by simply putting the URL of that page like this
In this case, the search engine will ignore the about me page of my website.
How to add custom Robot.txt to blogger?
- Go to blogger and log in
- Go to setting>search preferences>crawlers & indexing
- Click the edit tab in front of robot.txt
- Paste your robot.txt code here and click on the save.
You can get your robot.txt file in Google webmaster(Google search console)
Login to Google webmaster>Crawl>robot.txt tester>Copy the code