clock  Mon - Sun 8.00 AM - 8.00 PM
play store

What is Robots.txt file

writter  Hariom Prajapati
Date  29 Jun 2022
Language  Blog
What is Robots.txt file

What is Robots.txt file

robots.txt file is a very small text file that we have to keep in the root folder of our site. This tells the search engine bots that, which part of the site to be crawl or indexed and which part not need to crawl.

If you make a slight mistake while editing or customizing it, then the search engine bots will stop crawling or indexing your site and your site will not be visible in the search results.

In this tutorial, we will see what is robots.txt file and how to create perfect robots.txt file for SEO.

Why robots.txt file is necessary for website

When the search engine bots come to the website or blog, they follow the robots.txt file and then crawl the content. But if there is no file of robots.txt in your site, then the search engine bots will start indexing and crawling all the contents of the website that you do not want to index.

The search engine bots search the robots file before indexing any website. When they do not find any instruction by the robots.txt file, then they start indexing all the contents of the website. But if any instruction is found then the search engine bots follow them then Index website or webpage accordingly.

For these reasons robots.txt file is required. If we do not give instructions to the search engine bots through this file, then they index our entire website and also index some data that you did not want to index.


Benefits of robots.txt file

  • This tell search engine bots that which part of the site to crawl and index and which part to not. 
  • A particular file, folder, image, PDF, etc. can be prevented from being indexed in the search engine.
  • Sometimes it happens that all the search engine crawlers on our website come together that affect the performance of our website, then you can get rid of this problem by adding crawl-delay in robots.txt file. 
  • You can privateize the entire section of any website.
  • Can prevent the internal search result page from appearing in the SERPs.
  • You can improve your website SEO by blocking low quality pages.


Where does robots.txt File reside in the website

If you are a wordpress user,then robots.txt file reside in the root folder of your site. If this file is not found in this location, then search engine bots start indexing your entire website.

If you want to know whether there is robots.txt file add on your website or not, then you just have to type "" in your search engine address bar.

After this a text page will open in front of you as you seeing in the screenshot given below.

This is robots.txt file of teknowize. If you do not see any such text page, then you will have to create a robots.txt file for your site.


Basic format of robots.txt file

The basic format of robots dot pst file is very simple and looks like this :-

  • User-agent: user – agent name 
  • Disallow: URL of the page you don't want to crawl ]

These two commands are considered to be a complete robots.txt file, although a robots file can contain many command of user agents and directives ( disallow, allow, crawl-delays etc.)

  • User-agent: If you want to give same instruction to all search engine bots, then use " * "  after user agent like - User-agent: *  .
  • Disallow: This prevent file and directories from being indexed. 
  • allow: This allows the search engine bots to have your content crawled and indexed.
  • Crawl-delay: How many seconds the bots has to wait before loading and crawling the page content.


Preventing all web crawlers from indexing websites


Disallow: /

Using this command in robots.txt file, you can prevent all web crawlers or bots from crawling the website.


Allow all web crawlers to index all contents

User-agent: * 


Using this command in robots.txt file, you can allow all web crawlers or bots to crawling the website.


Blocking a specific folder for a specific web crawler

User-agent: Googlebot

Disallow: /example-subfolder/

This command only prevents the Google crawler from crawling the 'example-subfolder', but if you want to block all the crawler, then your robots file like -

User-agent: *

Disallow: /example-subfolder/


Preventing any one specific page from being indexed

User-agent: *

Disallow: /page URL

This will prevent all crawlers from crawling your page URL, but if you want to block a specific crawler, then you write robots file like this -

User-agent: Googlebot

Disallow: /page URL 

This command will only prevent Google from crawling your page URL.


Add a sitemap to the robots.txt file


You can add your sitemap to robot.txt file anywhere.


Comments 0

Leave a comment

Coursera, Codeacademy, Udacity, W3Schools, Udemy, Alison, TheNewBoston, edX, P.S.Codewars,Freecodecamp, Managing technical debt blog, Scrimba, Codepen, Codepen/challenges, The Odin Project, htmlreference.​io, cssreference.​io, Frontend Mentor, Dev Challenges, MDN, Code Mentor, Coding Dojo, CSS Battle, Codier, Ace Frontend, Can I Use, CSS Tricks, 30 Seconds of Code,tutorialspoint, Neumorphism, Shaddows Brumm, Fancy Border Radius, Glow Generator, Clothoid Corners, Glassmorphism, Clipy, CSS Filters, Base64 Image, Quantity Queries, Animations, Cubic-Bezier, Keyframes, Wait Animate, Transition.Style, graphic design, web design, website design, website builder, web developer, web designer, webdesign, ecommerce website, web design company, website creator, website designer, responsive web design, web development company, best website design, web design software, web page design, build a website, web developer salary, design website, web design courses, how to design a website, web design inspiration, website layout, web designer salary, web application development, ecommerce website design, web agency, software development company, web design tutorial, web programming, design company, website design templates, what is web designing, web developer jobs, website developer, web design agency, freelance web developer, web design services, freelance web designer, graphic design websites, web solutions, ecommerce website development, free website design, web development courses, webdev, web developers, web development tools, website design services, developpeur web, web design london, website design ideas, web designing and programming, design a website, web design and development, web dev, web development services, homepage design, best designed websites, cheap website design, learn web design, web design templates, web design tools, web design jobs, website design inspiration, web design india, flash website, website developers, designer websites, website services, website design cost, good website design, site design, simple website design, cool website designs, modern website design, graphic designer websites, webcode, best web design software, website making, free web design software, mobile website design, learn web development, front end web developer, how to become a web developer, web developer portfolio, web development company in india, python web development, web development tutorial, website company, website design and development, web company, webdesigning, professional website design, affordable web design, best web design company, creative web design, top website designs, website design pricing, web developer tools, how to develop a website