To enable Submission to Robots.txt in Magento 2, you need to follow these steps:
- Log in to your Admin Panel.
- Go to Stores > Settings > Configuration and choose XML Sitemap in the Catalog tab.
- Open the Search Engine Submission Settings tab and set Yes in the Enable Submission to Robots.txt dropdown.
- Click on the Save Conlig button.
Robots.txt configuration in Magento 2
Step 1
Go to Content > Design> Configuration.
Step 2
In the opened grid, find the line Global, click the Edit link, and open the Search Engine Robots tab.
Step 3
Now choose what Default Robots you need from the following:
INDEX, FOLLOW: Search engines will regularly index your pages and check the changes.
NOINDEX, FOLLOW: Search engine bot won’t index your website but it’ll still monitor changes.
INDEX, NOFOLLOW: Search bot will index your store once but never came back to track the changes.
NOINDEX, NOFOLLOW: This setting will hide your store from search engine bots.
Step 4
In the Edit custom instruction of robots.txt File line, you can write the needed custom instructions. Read on to find out more about the option.
Step 5
If you want to reset the default settings, you need to click the Reset To Default button, which will remove all your custom instructions.
Step 6
Don’t forget to Save Configuration to apply the changes.
