[FIXED] How to get one robots.txt for each store

Issue

I have a Magento 2 website with two stores. At the moment, I can edit the global website and his content is applied to both stores.

What I want to de is replace that behaviour in order to get one robot.txt file by store.

But I really have no idea how I should do that.

Currently, if I go to the back office Content > design > Configuration > (Store Edit) > Search Engine Robots

All the fields are disabled in the stores and can't be modified

But If I go on the global Content > design > Configuration > (Global Edit) > Search Engine Robots, of course, I can modify.

I also have 3 robots.txt files on my storage, but none of them seems to be matching the information saved in the global search engine robots configuration

• src/robots.txt
• src/app/design/frontend/theme1/jeunesse/robots.txt
• src/app/design/frontend/theme2/jeunesse/robots.txt

I found these two links...but none of them helped me : https://inchoo.net/online-marketing/editing-robots-txt-in-magento-2-admin/ and https://support.hypernode.com/knowledgebase/create-robots-txt-magento-2/

The first one tells me that If I have a robots.txt on my storage it should override the configurations...but looks like no considering I have robots file and they aren't showing when I go to website/robots.txt. I only find again the one in the global configuration.

The second one tells that saving the configuration should save the robots.txt file on the storage...but once again...that's not what is happening.

Thanks for your help, let me know if there is pieces of code I can show ? I really don't know which one at this point.

Solution

I'm the author of the first link. It's a 2 years old article, Magento 2 has since then introduced a few improvements to the built in robots.txt functionality.

The robots.txt content you save under Content > Design > Configuration has a "website" scope. Meaning you can edit it on website level and if you need it to vary through this config you can do it if you have multiple websites.

It is unclear from the question itself if you have multiple websites or if you have set-up multiple stores and/or storeviews under the same website.

Answered By - Toni Aničić

Answer Checked By - Robin (PHPFixing Admin)