How to hide your website from search engine result

To prevent your web page from appearing on search engine results (google, bing, yahoo, etc.), use noindex meta tag, and DON'T use robots.txt for this purpose.

Sang

May 7, 2024 • 2 min read

To prevent your web page from appearing on search engine results (google, bing, yahoo, etc.), use noindex meta tag, and DON'T use robots.txt for this purpose.

I have to make this snippet image because some systems fail to treat this as content. Instead, it parses this code as a meta header, hence, blocking this post from being shown in search engine results

BIG NOTE here, you must remove the trailing slash, it must be <meta ...>, NOT <meta .../>. Google website is showing the wrong snippet, do not follow them.

To prevent search engines from crawling the page (but doesn't mean prevent them from archive and showing pages in search results)

User-agent: *
Disallow: /

robots.txt to manage crawler traffic to your website. It does not prevent your website from being indexed

From Google

A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google. To keep a web page out of Google, block indexing with noindex or password-protect the page.

Warning: Don't use a robots.txt file as a means to hide your web pages (including PDFs and other text-based formats supported by Google) from Google search results.

If other pages point to your page with descriptive text, Google could still index the URL without visiting the page. If you want to block your page from search results, use another method such as password protection or noindex.

From Bing

The best way to block URLs (published on your site) from showing up in the Bing search results is to add a NOINDEX meta tag to the HTML header of the page.

Sign up for more like this.