To enable Submission to Robots.txt in Magento 2, you need to follow these steps:
- Log in to your Admin Panel.
- Go to Stores > Settings > Configuration and choose XML Sitemap in the Catalog tab.
- Open the Search Engine Submission Settings tab and set Yes in the Enable Submission to Robots.txt dropdown.
- Click on the Save Conlig button.
Robots.txt configuration in Magento 2
Go to Content > Design> Configuration.
In the opened grid, find the line Global, click the Edit link, and open the Search Engine Robots tab.
Now choose what Default Robots you need from the following:
INDEX, FOLLOW: Search engines will regularly index your pages and check the changes.
NOINDEX, FOLLOW: Search engine bot won’t index your website but it’ll still monitor changes.
INDEX, NOFOLLOW: Search bot will index your store once but never came back to track the changes.
NOINDEX, NOFOLLOW: This setting will hide your store from search engine bots.
In the Edit custom instruction of robots.txt File line, you can write the needed custom instructions. Read on to find out more about the option.
If you want to reset the default settings, you need to click the Reset To Default button, which will remove all your custom instructions.
Don’t forget to Save Configuration to apply the changes.