Description
What you'd like to propose?
Making a config option to hide /, /tags, ++ from robot indexing by changing the HTML robots header.
Context:
On each page you can set robotsNoIndex: true, then it will set a meta field in the HTML header that looks like this:
<meta name="robots" content="noindex, nofollow">This is great for hinding pages from being indexed by Google and others.
But there is no similar option to hide /, /tags, and similar sites. Meaning they will naturally index the pages since they have this in the HTML header:
<meta name="robots" content="index, follow">Its possible to override the robots.txt file. But there is no guarantee that they will honor it. Quote from Google:
Warning: Don't use a robots.txt file as a means to hide your web pages (including PDFs and other text-based formats supported by Google) from Google search results.
If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex.
It would be great if there was a global option for allowing the same thing you can do on a page level.
Code of Conduct
- I agree to follow this project's Code of Conduct