How to Prevent Search Engines from Indexing WordPress Sites?

Site owners will do anything to get their websites indexed. However, you might not want search engines to crawl through your website if it’s still in development. In a case like this, it’s recommended to discourage search engines from indexing your site. Stick with us if you want to learn more about this topic!

1. Discouraging Search Engines From Indexing WordPress SitesUsing the WordPress Built-In FeatureEditing robots.txt File Manually
2. Password Protecting Your WordPress WebsiteUsing Hosting Control PanelUsing WordPress Plugins
3. Removing Indexed Page From Google

Why Would You Want To Stop Search Engines From Indexing Your Site?

There are some cases where people want to discourage search engines from indexing their sites:

• Unfinished websites — at this stage of errors and trials, it’s best not to have your website available to the public eyes.
• Restricted websites — if you plan to have an invite-only website, you do not want it to get listed on SERPs.
• Test accounts — web owners create a site duplicate for testing and trial purposes. Since these sites are not designed for the public, don’t let it get indexed by search engines.

So how do you block search engines from indexing your site? Well, take a look at several options below and try it yourself.

1. Discouraging Search Engines From Indexing WordPress Sites

The simplest way to stop search engines from indexing your website is by preventing them from crawling it. To do it, you need to edit your website directory’s robots.txt file. Here are a few ways to achieve that:

Using the WordPress Built-In Feature

Editing WordPress robots.txt is quite easy as you only need to use a WordPress built-in feature. Here’s how:

1. Login to WordPress admin area and go to Settings -> Reading.
2. Scroll down and locate the Search Engine Visibility option.
3. Check the option that says Discourage search engines from indexing this site.
4. Save Changes, and that’s it! WordPress will automatically edit its robots.txt file for you.

Editing robots.txt File Manually

If you prefer the manual option, you can use File Manager or an FTP client to edit the robots.txt file.

In this article, we’ll show you how to do it through the hPanel’s File Manager:

1. Login to hPanel and locate File Manager under the Files area.
   
   
   
2. Go to your WordPress root directory folder (in most cases, it’s public_html) and find the robots.txt file. If you can’t find it, create a new blank file.
3. Right-click on the file and select Edit.
   
   
   
4. Enter the following syntax:
   
   User-agent:
   * Disallow: /

The code above will prevent search engines from indexing your whole site. If you want to apply the disallow rule to a specific page, write the page’s subdirectory and slug. For example: Disallow /blog/food-review-2019.

The syntaxes in robots.txt files are case sensitive, so be careful when editing.

2. Password Protecting Your WordPress Website

Search engines and web crawlers don’t have access to password-protected files. Here are a few methods to password protect your WordPress site:

Using Hosting Control Panel

If you are a Hostinger client, you can password protect your website using hPanel’s Password Protect Directories tool:

1. Access hPanel and navigate to Password Protect Directories.
2. Enter your root directory into the first field.
3. Once the directory is selected, enter your username and password and click Protect.

If your root directory is public_html, leave the directory column blank

The process in cPanel is also quite similar:

1. Log in to your cPanel account and head to Directory Privacy.
   
   
   
2. Select your root directory. In our case, it’s public_html.
3. Check the Password protect this directory option, and name the protected directory. Press Save.
4. Create a new user to login to the protected website, and that’s it!
   

Using WordPress Plugins

There are tons of plugins that can help to password protect your site. However, the Password Protected plugin might just be the best one out there. It’s been tested with the new WordPress update, and it’s pretty straightforward to use.

After installing and activating the plugin, head to Settings -> Password Protected and configure the settings to match your needs.

3. Removing Indexed Page From Google

Don’t worry if Google has indexed your site. You can remove it from SERPs by following these steps:

1. Set up Google Search Console for your website.
2. Access Google Search Console of your newly added website and scroll down to Legacy tools and reports -> Removals.
3. Click the Temporarily hide button and enter the URL you want to remove from Google.
4. On a new window, choose Clear URL from cache and temporarily remove from search, then Submit Request.
   
   
   

And that’s it! Google will temporarily remove your site from search results. Make sure to apply the previous methods to prevent Google from indexing your site again.

Conclusion

There you have it! Quick and easy ways to discourage search engines from indexing your sites. Here’s a quick recap of the methods we’ve learned today:

• Edit the robots.txt file, which can be performed automatically or manually.
• Password protect your website by using a plugin or your hosting control panel.
• Remove indexed pages from Google via Google Search console.

If you have any other methods, or if you have any questions, please do let us know in the comments. Good luck!

Evolving “nofollow” – New Ways to Identify The Nature of Links

Nearly 15 years ago, the nofollow attribute was introduced as a means to help fight comment spam. It also quickly became one of Google’s recommended methods for flagging advertising-related or sponsored links. The web has evolved since nofollow was introduced in 2005 and it’s time for nofollow to evolve as well.

Today, we’re announcing two new link attributes that provide webmasters with additional ways to identify to Google Search the nature of particular links. These, along with nofollow, are summarized below:

rel="sponsored": Use the sponsored attribute to identify links on your site that were created as part of advertisements, sponsorships or other compensation agreements.

rel="ugc": UGC stands for User Generated Content, and the ugc attribute value is recommended for links within user generated content, such as comments and forum posts.

rel="nofollow": Use this attribute for cases where you want to link to a page but don’t want to imply any type of endorsement, including passing along ranking credit to another page.

When nofollow was introduced, Google would not count any link marked this way as a signal to use within our search algorithms. This has now changed. All the link attributes -- sponsored, UGC and nofollow -- are treated as hints about which links to consider or exclude within Search. We’ll use these hints -- along with other signals -- as a way to better understand how to appropriately analyze and use links within our systems.

Why not completely ignore such links, as had been the case with nofollow?

Links contain valuable information that can help us improve search, such as how the words within links describe content they point at. Looking at all the links we encounter can also help us better understand unnatural linking patterns. By shifting to a hint model, we no longer lose this important information, while still allowing site owners to indicate that some links shouldn’t be given the weight of a first-party endorsement.

We know these new attributes will generate questions, so here’s a FAQ that we hope covers most of those.

Do I need to change my existing nofollows?

No. If you use nofollow now as a way to block sponsored links, or to signify that you don’t vouch for a page you link to, that will continue to be supported. There’s absolutely no need to change any nofollow links that you already have.

Can I use more than one rel value on a link?

Yes, you can use more than one rel value on a link. For example, rel="ugc sponsored" is a perfectly valid attribute which hints that the link came from user-generated content and is sponsored. It’s also valid to use nofollow with the new attributes -- such as rel="nofollow ugc" -- if you wish to be backwards-compatible with services that don’t support the new attributes.

If I use nofollow for ads or sponsored links, do I need to change those?

No. You can keep using nofollow as a method for flagging such links to avoid possible link scheme penalties. You don't need to change any existing markup. If you have systems that append this to new links, they can continue to do so. However, we recommend switching over to rel=”sponsored” if or when it is convenient.

Do I still need to flag ad or sponsored links?

Yes. If you want to avoid a possible link scheme action, use rel=“sponsored” or rel=“nofollow” to flag these links. We prefer the use of “sponsored,” but either is fine and will be treated the same, for this purpose.

What happens if I use the wrong attribute on a link?

There’s no wrong attribute except in the case of sponsored links. If you flag a UGC link or a non-ad  link as “sponsored,” we’ll see that hint but the impact -- if any at all -- would be at most that we might not count the link as a credit for another page. In this regard, it’s no different than the status quo of many UGC and non-ad links already marked as nofollow.

It is an issue going the opposite way. Any link that is clearly an ad or sponsored should use “sponsored” or “nofollow,” as described above. Using “sponsored” is preferred, but “nofollow” is acceptable.

Why should I bother using any of these new attributes?

Using the new attributes allows us to better process links for analysis of the web. That can include your own content, if people who link to you make use of these attributes.

Won’t changing to a “hint” approach encourage link spam in comments and UGC content?

Many sites that allow third-parties to contribute to content already deter link spam in a variety of ways, including moderation tools that can be integrated into many blogging platforms and human review. The link attributes of “ugc” and “nofollow” will continue to be a further deterrent. In most cases, the move to a hint model won’t change the nature of how we treat such links.

We’ll generally treat them as we did with nofollow before and not consider them for ranking purposes. We will still continue to carefully assess how to use links within Search, just as we always have and as we’ve had to do for situations where no attributions were provided.

When do these attributes and changes go into effect?

All the link attributes, sponsored, ugc and nofollow, now work today as hints for us to incorporate for ranking purposes. For crawling and indexing purposes, nofollow will become a hint as of March 1, 2020. Those depending on nofollow solely to block a page from being indexed (which was never recommended) should use one of the much more robust mechanisms listed on our Learn how to block URLs from Google help page.

Source: Webmaster Central Blog

Google Algorithm Update 2019

A rare Google confirmation came related to a Google search algorithm update this week (12 march 2019). Google restated previous advice that there is no fix if your site was negatively impacted.

Google released a broad core search algorithm on March 12 - AKA Florida 2

Why it matters:

Google does several core ranking updates per year and confirms very few updates throughout the year. Specific to broad core updates, Google has said numerous times that you cannot do anything specific to fix your rankings. Google’s previous advice is, "there’s no ‘fix’ for pages that may perform less well other than to remain focused on building great content. Over time, it may be that your content may rise relative to other pages."
If your rankings did change recently, it may have been related to this broad core ranking update and not necessarily related to a technical change you made on your website.

What changed?

Right now it is very early and it is hard to guess what has changed. Based on the SEO chatter around this update, prior to Google confirming the update, some are saying this was again targeting the health/medical space. But, Google has said there was no specific target at medical or health sites with that August 1st update.
It is hard to know which types of sites were impacted the most right now. We will continue to monitor the situation and keep you updated on any insights we see related to this update.

Google’s previous advice.

Google has previously shared this advice around broad core algorithm updates:
"Each day, Google usually releases one or more changes designed to improve our results. Some are focused around specific improvements. Some are broad changes. Last week, we released a broad core algorithm update. We do these routinely several times per year.

As with any update, some sites may note drops or gains. There’s nothing wrong with pages that may now perform less well. Instead, it’s that changes to our systems are benefiting pages that were previously under-rewarded.

There’s no "fix" for pages that may perform less well other than to remain focused on building great content. Over time, it may be that your content may rise relative to other pages."
Source: To see more advice from Google around Google updates, see this Twitter thread.

Google's Medic Update 2018 - The Core Search Update

The big Google algorithm update, nicknamed the Medic Update, here is everything we know about it, including official information from Google and non-official insights from across the industry.

Google's Medic update and how to deal with it

The Google search algorithm update from August 1 is now fully rolled out, and here is what we know about the update, who we think was impacted and some of the analysis of what, if any, actions you may want to consider taking if you were negatively impacted.

In summary, Google is calling this a broad, global, core update, but based on much of the analysis done thus far, there seems to be a focus on health and medical sites and YMYL Your Money Your Life sites. But many sites besides those were impacted by the update. Google is telling us that there is nothing you can do to fix your site, so you should just focus on making a great experience, offer better content and a more useful website. This update has taken on the name the Medic Update because of its focus on the medical and health space. This specific focus is something Google will not confirm.

Why is it called the Medic update?

It’s called the Medic update because Barry Schwartz, one of the most prolific writers in the search industry, called it that. It doesn’t mean this update only affected medical sites.

Google has said that this update was a "broad core algorithm update" and that it does these updates "several times per year."

Google references its advice from the previous core updates, saying there’s "no ‘fix’ for pages that may perform less well, other than to remain focused on building great content. Over time, it may be that your content may rise relative to other pages." Google also said, "As with any update, some sites may note drops or gains. There’s nothing wrong with pages that may now perform less well. Instead, it’s that changes to our systems are benefiting pages that were previously under-rewarded."

Who was impacted by this update

As we explained above, Google said this is a “global” update, which implies every niche and every type of site could have been impacted. But based on the data that I’ve been seeing from surveys, multiple data companies and SEO consultants, there seems to be a focus on medical and health niches, as well as “Your Money Your Life” types of sites, with creeping into the entertainment and gaming niches as well. I’ve shown Google this data and a Google spokesperson responded by referencing the statements made above.

Source: Google Blog

Goodbye, AdWords. Hello Google Ads

Google has made the executive decision to consolidate its overwhelming abundance of ad products under three brand new umbrellas: Google Ads, Google Marketing Platform, and Google Ads Manager. Behold, logos!

“AdWords” is no more.

In a surprise rebranding of its 18-year-old PPC advertising platform, Google has renamed AdWords to the broader Google Ads, simplified its advertising products and introduced several new features.

Huge news? You bet. Cause for concern? Not at all!

That’s because we’ve broken down all the changes coming to Google advertising to give you the essential information you need.

So, what is changing?

According to Google, Google Ads users will start seeing the new brand name and logo reflected on the platform, website, billing and help center.

Instead of logging into AdWords from adwords.google.com, you will log into Google Ads from ads.google.com.

Changes to the Google Ads branding will not impact your campaign performance, navigation, or reporting.

The new Google Ads experience is going to be heavily focused on the ease of multi-channel advertising, connecting everything from search to display to video for a more seamless experience.

These new brands will help advertisers and publishers of all sizes choose the right solutions for their businesses, making it even easier for them to deliver valuable, trustworthy ads and the right experiences for consumers across devices and channels.

So, basically, Google Ads is a platform where businesses will find all they loved about AdWords and even more tools to make the process of reaching customers easier.

Rolling Out Mobile-First Indexing

Mobile-First Index Roll-out — March 26, 2018

Google announced that the mobile-first index was finally "rolling out." Since the index has been in testing for many months, and Google has suggested they are migrating sites gradually, it's unclear how much impact this specific roll-out had on the overall index. Webmaster should begin to see notifications within Google Search Console.

Source: Google Webmaster Central Blog