In its guidelines for webmasters, Google advises webmasters not to use a robot.txt file to hide low-quality content. Remember, the goal of robots.txt is to instruct bots what to do with the content they crawl on your site. It doesn’t stop bots from crawling your website. We recommend that you don’t disallow the readme.html file in your WordPress robots.txt file. This readme file can be used by someone trying to figure out which version of WordPress you are using. If this is an individual, then they can easily access the file just by browsing it. On the other hand if someone is running a malicious query to search for a WordPress site using a specific version, then this disallow tag can protect you from such mass attacks.
How to Create And Edit Robots.txt File With Yoast SEO
Adding Your XML Sitemap to the Robots.txt File
If you are using the Yoast SEO WordPress plugin or some other plugin to generate your XML sitemap, then your plugin will try to add your sitemap automatically into the robots.txt file. However if that fails, then your plugin will show you a link to the XML Sitemaps which you can add to your EA Leads robots.txt file manually like this:If you use the popular Yoast SEO plugin, you can create (and then edit) your robots.txt file directly from the Yoast interface. However, before you can access it, you will need to enable the advanced features of Yoast SEO by going to SE
