Preventing WFM from showing up in search engine results
To prevent web crawlers from crawling and indexing WFM pages, these changes were made:
- A robots.txt file was added to the root path of WFM
- The robots
<META>
tag was added to all HTML pages - The
X-Robots-Tag
HTTP header was injected into the WFM header
These changes prevent WFM applications from showing up in most search engine result pages. As the changes depend on the search engine to visit the site, it can take some time for the WFM application to disappear from search results.
User-agent: Googlebot
Disallow:
User-agent: Baiduspider
Disallow: /{wfm_subroot}
User-agent: Bingbot
Disallow:
User-agent: Slurp
Disallow:
User-agent: YandexBot
Disallow:
User-agent: *
Disallow: /{wfm_subroot}
In the robots.txt file, replace {wfm_subroot} with the context root that WFM is installed on.
If you already have a generic filter for Googlebot, Baiduspider, Bingbot, Slurp, or YandexBot in your robots.txt file, ensure that {wfm_subroot} is not blocked for Googlebot, Bingbot, Slurp, and YandexBot and that it is blocked for Baiduspider and any other generic bots (User-agent: *).