Why and Who Would Want to Stop Search Engines

Why and Who Would Want to Stop Search Engines

For most websites, search engines are the major source of traffic. You may ask, why would anyone want to block search engines?

Believe it or not, there are many users who work on their sites by putting them on a live publicly accessible domain instead of creating a local development environment or creating a development site.

Some people create project management sites using WordPress. There are many people who use WordPress to create private blogs. In all these situations, you probably don’t want to be indexed by search engines and be found when you are not ready for that.

A common misconception is that if I do not have links pointing to my domain, then search engines will probably never find my website. This is not completely true.

There are many ways search engines can find a website linked elsewhere. For example:

1. Your domain name could have been previously owned by someone else and they still have some links pointing to your website now.
2. Some domain search site’s results could get indexed with your link on them.
3. There are literally thousands of pages with just list of domain names, your site can appear on one of those.

There are many things happening on the web and none of them are under your control. However, your website is still under your control, and you can instruct search engines to not follow or index your website.

Blocking Search Engines from Crawling and Indexing Your WordPress Site

WordPress comes with a built-in feature that allows you to instruct search engines not to index your site. All you need to do is visit Settings » Reading and check the box next to Search Engine Visibility option.

When this box is checked, WordPress adds this line to your website’s header:

1	<meta name='robots' content='noindex,follow' />

WordPress also modifies your site’s robots.txt file and add these lines to it:

1 2	User-agent: * Disallow: /

These lines ask robots (web crawlers) not to index your pages. However, it is totally up to search engines to accept this request or ignore it. Even though most search engines respect this, some page or random image from your site may get indexed.

How to Make Sure Your Site Doesn’t Appear in Search Results?

The most effective way to block search engines from a live website is by password protecting your entire WordPress site on the server level. This means when anyone accesses your website they are asked to provide a username and password even before they reach WordPress. This includes search engines as well. Upon login failure, they are shown 401 error and the bots turn away. Here is how to password protect an entire WordPress site.

Password Protecting an Entire Site using cPanel

If your WordPress hosting provider offers cPanel access to manage your hosting account, then you can protect your entire site using cPanel. Simply login to cPanel dashboard and then click on password protect directories.

This will bring up a popup where you need to choose the document root.

On the next screen, select the folder where your WordPress site is installed. It is usually public_html or www directory. After that, check the box next to ‘Password protect this directory’ option. Next, provide a name to the protected directory and hit the save button.

Later, you will see a success message and link to go back. Clicking on the go back link will bring you to the password protection screen. Now, you need to add a username and password which will be required to view your website.

That’s all your website is now password protected, and no one including the search engines can access your website.

Here is another tutorial on how to password protect your WordPress site. However, in this tutorial, you will be using a plugin. If for some reason you deactivate the plugin, then your site will be accessible to search engines again.

We hope this article helped you stop search engines from crawling or indexing your WordPress site.

https://www.elegantthemes.com/blog/tips-tricks/how-to-create-and-configure-your-robots-txt-file

https://www.contentkingapp.com/academy/robotstxt/

What is 301 redirect

When you need to redirect a website or individual webpage to a different location, there are a number of different tactics you can employ.

What is a redirect?

A redirect will simply send a user from the URL they originally requested to a different webpage. This works for both visitors and search engines.

Why would you use a redirect?

According to Hubspot, there are three main reasons why you would use a redirect.

1) To associate common web conventions with one URL to maximize domain authority

So if there are multiple versions of your domain name (for instance http://www.example.com and plain old example.com) there’s a chance you’ll encounter duplicate content issues. By setting up a permanent redirect (a 301 redirect) from one version of your site to your preferred version, search engines will know the correct domain to index.

2) To rebrand or rename a website with a different URL

You’ll want to set up a permanent redirect if your website name changes.

3) To direct traffic to a website from other URLs owned by the same organization

Occasionally brands may purchase website URLs that are variations on their existing brands in order to secure them. These URLs can be redirected to the proper brand domain.

There are countless other reasons why you’d set up a redirect, and it can just be for an individual webpage rather than a whole domain.

What is a 301 redirect?

As mentioned earlier, a 301 redirect is a permanent redirect. It basically tells users and search engines, “That information now resides somewhere else.”

Benefits of 301 redirect

A 301 response is ideal for correcting visitors who come to the wrong version of your domain, like non-www redirecting to www. This code should also be used when you update and move content within your site.

In terms of SEO, a link from a 301 redirected page shouldn’t lose any ranking power compared to a link from a non-301 redirected page.

There was a previously held assumption that there would be a loss of PageRank (around 15%) but this is no longer the case. As reported by Moz in August 2016:

§  In February, Google’s John Mueller announced that no PageRank is lost for 301 or 302 redirects from HTTP to HTTPS.

§  Google’s Gary Illyes stated that Google doesn’t care which redirection method you use, be it 301, 302, or 307. Google will figure it out and they all pass PageRank.

§  Gary Illyes also announced on that any 300 redirects no longer lose PageRank at all.

Problems with 301 redirect

Redirects can cause problems when the new location isn’t relevant to the old location This can be seen as a misuse of the function and is therefore a common spam indicator, and may result in a manual penalty.

Especially if you buy multiple irrelevant websites and point them all at your site.

Moz also states the following to keep in mind:

§  PageRank only forms a small part of the ranking factors used by Google

§  Try and keep every element on the page the same, except of course for the URL

§  Migrating a site to HTTPS is less likely to lose PageRank then before, but there are many others issues that can affect crawling and indexing, so do be careful.

Redirect a single page 

On page SEO

On-page and off-page SEO

SEO