Preventing WFM from showing up in search engine results

To prevent web crawlers from crawling and indexing WFM pages, these changes were made:

  • A robots.txt file was added to the root path of WFM
  • The robots <META> tag was added to all HTML pages
  • The X-Robots-Tag HTTP header was injected into the WFM header

These changes prevent WFM applications from showing up in most search engine result pages. As the changes depend on the search engine to visit the site, it can take some time for the WFM application to disappear from search results.

Note: The robots.txt file is only detected if it is at the top directory of the website. In a non-root deployment of WFM, you must create a robots.txt file at the root level with the settings specified below. If you already have a robots.txt file at the root level, then these settings would need to be added.
User-agent: Googlebot 
Disallow: 

User-agent: Baiduspider 
Disallow: /{wfm_subroot} 

User-agent: Bingbot 
Disallow: 

User-agent: Slurp 
Disallow: 

User-agent: YandexBot 
Disallow: 

User-agent: * 
Disallow: /{wfm_subroot}

In the robots.txt file, replace {wfm_subroot} with the context root that WFM is installed on.

If you already have a generic filter for Googlebot, Baiduspider, Bingbot, Slurp, or YandexBot in your robots.txt file, ensure that {wfm_subroot} is not blocked for Googlebot, Bingbot, Slurp, and YandexBot and that it is blocked for Baiduspider and any other generic bots (User-agent: *).