Answered: How to write robots.txt file for No-indexing categories pages

Post date: 2021-06-15 02:42:31
Views: 92

Simply put a list of Disallow entries with your category names in the robots file (beware that categories are also accessible under the path "/questions"):

User-agent: *
Disallow: /category-A
Disallow: /questions/category-A
Disallow: /category-B
Disallow: /questions/category-B
...
Sitemap: https://example.org/sitemap.xml

Note that if your robots file contains a sitemap URL (which it should) you need to remove the category entries from the sitemap as well. However, the Q2A default sitemap plugin only allows for omitting the category browser, not the individual category entries (https://example.org/questions/category-A, https://example.org/questions/category-B, ...). Hence you need to either remove the sitemap (not recommended), or use a different plugin for generating it without category entries.

There is at least one more advanced sitemap plugin available, but I don't know if it supports omitting categories, since I haven't used it myself. You may need to patch the code either way.

Please click Here to read the full story.
 
Other Top and Latest Questions:
Trump trial: Ex-National Enquirer publisher David Pecker is first prosecution witness to testify
AIG Travel Guard insurance review: What you need to know
EU threatens to suspend TikTok Lite’s money-for-views program over addiction fears
Jim Cramer shares his thoughts on Tesla, Cisco Systems, Verizon, Prologis and Papa John's
Certified Addiction Counselor work experience conundrum
Hi Grandma, Relax, We have a Newborn
Here are Wednesday's biggest analyst calls: Tesla, Amazon, Airbnb, Domino's, Walmart, Disney & more
Book: Red Side Story by Jasper Fforde
Frieren: Beyond Journey's End: Conquering The Labyrinth
Answered: Nested comments on replies