How to hide your website from search engine result
To prevent your web page from appearing on search engine results (google, bing, yahoo, etc.), use noindex meta tag, and DON'T use robots.txt for this purpose.
BIG NOTE here, you must remove the trailing slash, it must be <meta ...>
, NOT <meta .../>
. Google website is showing the wrong snippet, do not follow them.
To prevent search engines from crawling the page (but doesn't mean prevent them from archive and showing pages in search results)
From Google
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex
or password-protect the page.
Warning: Don't use a robots.txt file as a means to hide your web pages (including PDFs and other text-based formats supported by Google) from Google search results.
If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex
.
From Bing
The best way to block URLs (published on your site) from showing up in the Bing search results is to add a NOINDEX meta tag to the HTML header of the page.