Hide a site entirely from search engines (Google, Bing etc.)

My company is running a few internal websites that we do not want indexed by search engines such as Google, Bing etc.
However, the websites still need to be accessible for our customers, and therefore, I do not wish to useHTTP password protection.
Obviously, I already have a robots.txt containing:
User-agent: *
Disallow: /
When I search for the domain name, it still shows up, and Google says: "A description for this result is not available because of this site's robots.txt", while Bing says "We would like to show you a description here but the site won’t allow us.".
How can I ensure that the websites are totally hidden in the search results?


This happens when Google or Bing discovers your site and has not been told not to index the site. This happens when there is a link or redirect to the site and the robots.txt restricts the search engine from the site. However, this is not the same as telling a search engine not to index the site.
Put <meta name="robots" content="noindex"> in the header of your HTML of all pages (preferable) or at least the home page and search engines should remove your site from the index in time. It can take 30-60 normally (for Google) but may take longer. It all depends upon how fast the search engine revisits your site and the processing within the search engine. It can take less than 30 days too. I just wanted to warn you that it may take some time.
For now, there is no harm except that others may discover your site. If you want to limit visitation, then perhaps another mechanism is needed. I understand wanting to keep it open and not require an account. As of right now, I am not sure I have advice on limiting visitation. But also understand that rogue spiders will also discover your site and may create links regardless of your wishes. Think about how you may control access if and when this happens - and if control is important to you.


Post a Comment

Don't Forget to comment

About Me

My photo

HI i am mohammed  a Embedded Software Developer ,SEO specialist,Electronic engineer   Read more