How to Make Google Crawl Budget Work in Your Favor

As I look at many blogs on a daily basis during my blog audits and other consulting services I provide, I run into the problem of bloggers squandering their precious time with Google bots on all sorts of links that shouldn’t be on their home page to begin with.

In order for me to explain what I mean, we first need to address the way Google crawls any website and how they determine how much time they would spend there.

The way Google bots first discover and consequently index and rank your blog pages is by following the links from other sites and within your own blog.

This is how Matt Cutts describes the process in one of his interviews:

“The best way to think about it is that the number of pages that we crawl is roughly proportional to your PageRank. So if you have a lot of incoming links on your root page, we’ll definitely crawl that. Then your root page may link to other pages, and those will get PageRank and we’ll crawl those as well. As you get deeper and deeper in your site, however, PageRank tends to decline.”

I need to remind you here that there’s a difference between the PageRank displayed in your toolbar (visible PR), which is updated every once in a blue moon and is obsolete the minute it’s updated and actual PR that Google updates on an ongoing basis.

According to the green bar, Traffic Generation Cafe has no PR, yet Google crawls my blog on a daily basis - a sure sign that my PR is higher than Google would like to show, for whatever reason.

Learn more about the difference between real and visible PR in this post:

• Mistery of Google PageRank Solved?

So, Google follows links from other websites to your home page.

As they crawl your home page, they discover other links that lead deeper into your blog. But since Google bots won’t follow all those links to the infinity, wouldn’t you want to make sure they definitely crawl the most important pages first?

And that’s precisely why what you link to from your home page and other important pages is so vital for you to maximize your face time with Google bots.

Now let’s talk about some to the links that I often see on the home page that really have no business of being there.

Admin Login

Do you have that little link at the bottom of your blog that says “WordPress Admin Login“?

Sure it might be useful to you as far as logging in to your dashboard from any page on your blog, but what if I told you that this is one of the most linked-to pages on many blogs I see?

Do you really want to waste your crawl budget on an admin page? I know, rhetorical question…

Plus, now WordPress gives you an option to display an admin bar on any page of your blog without it actually being shown to your readers.

It’s an option you can choose under Users ==> Your Profile:

So how do you get rid of the link?

Your theme is what you should look into.

I use Thesis theme on my blog and with Thesis, I would go under Design Options ==> Display Options ==> Administration.

If you are using any other theme, just poke around to see if you have a similar option.

Pagination

I had no idea this was an issue until very recently when I started seeing these kinds of most linked-to pages on different blogs:

http://www.YourSite.com/page/2/

As I did a little digging into, I realized that my blog had it as well, and it all stems from using a plugin that would display page numbers on your home page for all the posts you’ve ever written as opposed to just giving you the default “Older Posts”, “Newer Posts”.

What happens when you use a plugin like that is that you are essentially adding more and more links to your home page, sending Google bots all over the blog, as opposed to strategically showing them the posts you’d like them to crawl.

So get rid of any plugins that create pagination like that and stick with the default Wordress way to separating your blog posts into pages.

You can find this option under Settings ==> Reading.

Just choose the number of posts you’d like to show on each page and you’ll be all set to go.

By the way, the same goes for comment pagination.

When you choose to break up your comments into pages, not only do you create the extra clicks your readers have to navigate if they want to see the whole discussion, but also create a bunch of useless pages with DUPLICATE content on them.

To learn more about duplicate content and its effects on your blog, read the following post:

• Duplicate Content Phantom: Don’t Be Duped, Be Informed

Getting rid of comment pagination is pretty simple: go into Settings ==> Discussion, and make sure your “Break comments into pages” box is unchecked.

Word on Duplicate Content and Google Crawl Budget

Even though duplicate content is a common problem that you can find on pretty much any site with more than a couple of pages, there’s a good reason to be aware of it and how it affects your face time with Google bots.

Here’s another excerpt from Matt’s interview:

“Imagine we crawl three pages from a site, and then we discover that the two other pages were duplicates of the third page. We’ll drop two out of the three pages and keep only one, and that’s why it looks like it has less good content. So we might tend to not crawl quite as much from that site.”

The duplicate content post above will give you all the info you need as to how to avoid the issue of duplicate content on your site as much as possible.

Categories and Tags

This is one of the most overused blog structuring techniques I see on so many blogs.

Somehow many bloggers decide that the more categories and tags they use, the better - not quite sure why.

Your categories and tags determine your basic blog structure as well as the theme of your blog.

The more categories you use, especially generic ones or the ones that are not really related to your blog theme (see the screenshot below), the more diluted your overall blog theme will appear to both your readers and particularly Google bots, which might in turn affect your search engine rankings.

What is the benefit for your readers to see a category “focusing”? Or “success”?

Look at it from the search engine perspective as well. How are they supposed to theme your blog with generic categories like these?

My suggestions on categories:

• Keep the focus as narrow as possible - I’d suggest to stay within 5-8 categories. I know it’s an arbitrary number, but it’s a good one to keep in mind, IMHO.
• Use your KEYWORDS as category names. Take a look at mine in the navigation bar below my header - these are my categories and I would love to rank for each and every one of those keywords.
• Don’t dump each post into every imaginable category; pick one or two only.
• Don’t link to categories from your home page UNLESS they are well-researched keywords and are highly related to the theme of your blog.

TAGS:

The same goes for tags.

Tags are nothing but well-researched keywords you assign to each post hoping to rank for them at some point.

To learn more about tags and how they can affect your rankings, read this post at Kikolani.com:

• What Are Post Tags & Why You Should Be Using Them

Here’s also some additional reading on how to correctly structure your blog:

• Intelligent Site Structure for better SEO

Tag clouds

Whether it’s big or small, they are still links, and your page rank will be leaking from your main page to the tag pages.

It’s smarter to have your page rank flow to your individual posts rather than tag pages.

I would definitely remove the tag cloud, if you are still using one.

More on the topic:

• Is Google Waving Goodbye to Tag Clouds?

Affiliate Links

Affiliate marketing is how many bloggers make their money online, including yours truly.

So of course I won’t tell you to get rid of your affiliate links on your home or any other page you’d like to display them on, but at the very least, NOFOLLOW those links! Plus, make sure they open in a new window to prevent your readers from leaving your blog.

Social Media Widgets

Let’s face it: we all feel pressured to display all kinds of social media icons and widgets in our sidebars.

Their positive effect on getting connected with our readers in social media realms is debatable, yet their negative effect as far as leading both our readers and Google bots away from our blog is obvious.

Once again, I am not going to say you shouldn’t have them, but use common sense as far as how many different widgets and icons you choose to display: once again, this is the case when more is not better.

ALWAYS nofollow those links though and make sure they open in a new window as with affiliate links.

Personally, I would stay away from displaying various networks that are not widely used by most of your readers, as well as widgets that display Twitter feed, etc. Not too many of your readers find any use for them, trust me, yet they add the amount of links that lead away from your blog.

Recent/Popular/Most Commented Posts

I am definitely a big fan of displaying one of these widgets in your sidebar, but only one.

To me, popular posts widget, which is determined by the amount of comments each post has, makes the most sense to display.

If you choose to show recent posts, you are just duplicating what your home page already shows, plus it’s easy to find that info by clicking over to home page from anywhere else on the blog, whereas getting to most commented/popular posts is impossible without the widget.

DO: Link to Your Best Content

This is how you can easily improve your rankings for the posts you are trying to rank for.

Create a sidebar widget where you simply link to your best cornerstone pages using keywords as anchor text and you instantly gain some great internal backlinks leading both your readers and Google bots directly to those pages.

Take a look at how I do it on my blog (you can see the widget towards the top of the sidebar):

Marketing Takeaway

Displaying hundreds of links that send your visitors as well as Google bots on a wild goose chase looking for your best content isn’t a great way to go, as you can see.

I would strongly suggest you strategically narrow down the number of links you display on your home page that will be beneficial to your overall linking strategy.

Don’t be afraid to give your visitors fewer choices. Clean, uncluttered look definitely shows to convert much better and decrease your bounce rates, providing for easy navigation that makes sense.

Can you think of other links I forgot about? Comment to show me that you’re alive!

Image credit: TheChive.com