This is a guest post by Ben Hook. Should you wish to guest post in this blog, please visit the guest post guidelines.
There’s a lot of benefits that can come from having a site that’s regularly crawled by search engines. If you make a lot of on page changes you will benefit by seeing the results of these changes a lot quicker, it also means that if any negative results are seen due to changes then these can be quickly corrected.
So how can you increase your crawl rate? The following points are all factors that affect how regularly pages are crawled and by optimising these you can make sure you are getting crawled deeper, and more regularly.
Regular content
Search engines are much more likely to frequent a page if the content on it is regularly updated. This is one of the reasons why blogs tend to be crawled more often than static sites. If you have a static site, try to consider ways that you can include regularly updated content on the site. Think of things along the lines of Twitter feeds, recent comments or posts from your blog, or other forms of regularly updated content.
Reducing the page size
Crawling websites costs money and time. Smaller pages are quicker to crawl and so take up less resource. This means they are much more likely to be crawled regularly. Reducing your page size and making it quicker to load will also have beneficial effects on your rankings as site speed has recently become a ranking factor.
Sitemaps
Including both and XML and HTML sitemap on your site is going to make it easier for the search engines to crawl your site. While this won’t have a direct impact on how regularly your site is crawled it will make the process easier which can only help.
Duplicate content
Avoid duplicate content issues at all cost! If you have lots of canonical issues on your site then the search engines are going to spend a lot of wasted time crawling unnecessary pages. This is going to decrease how regularly your site is crawled and can also prevent the search engines from indexing a lot of your site. It’s important to get any of these issues as soon as possible.
Increase your pagerank
Pages with higher pagerank are crawled more regularly. Building links from high quality sources will help to increase how often your site is crawled. If you have certain pages on your site that you require to be crawled more often this can be dealt with using pagerank sculpting.
Reliability
If the search engines come to visit your site and it’s unavailable a lot of the time you are oign to be harming not only your rankings, but also the frequency at which they will return. You need to make sure you have a quick and reliable server and that if any maintenance is taking place the correct server headers are put in place.
If you are signed up for webmaster tools you will be able to monitor the search engines behaviour and hopefully see the increase in crawl frequency that comes as a results of these changes.
Author Bio
Ben Hook is a search engine marketer and editor of Algobreaker, a website providing SEO tips and other advice about online marketing.
If you like this article, you may also like:
Hi Ben & Lye
I do most of what you share in this article. However, could you please explain by what you mean my smaller page? Do you mean shorter articles?
Sorry if this is a basic question but as I don’t know the answer, needed to ask. Thanks for sharing Ben. Much appreciated.
Patricia Perth Australia
Patricia recently posted..So Little Time- So Much To Do
[...] This post was mentioned on Twitter by Patricia Millman, Findmyblogway. Findmyblogway said: Increase How Often Search Engines Crawl Your Site http://bit.ly/gDmRSd [...]
Ben
Very good tips listed out here
good job
Praveen recently posted..12 Must Know Guest Blogging Etiquette
Hey Ben, Thanks for the info man.
As per page rank, i think it’s the most effective way after writing great content
Olusegun recently posted..Affiliate Marketing Tips 5 – Generating Traffic and Sales with Free Viral Reports
Hi Ben, these tips could definitely attract the google bots to visit you more often. However the Google traffic might not work the best for everyone, imagine you are not talking about the things that are usually searched through the google, you want to provide your readers with something different, your experiences, point of views, ideas, tips in such case networking could be more useful for your traffic and community building.
Thanks for info.
Adam recently posted..Comments – How Important They Are To Your Success
Nice tips and i guess these are very helpful and useful ones ! I do agree there are several benefits that can come from having a site that’s crawled by search engines
Twitter: AdamPaudyal
Hey Ben, some nice tips on attracting the search engine spiders.
I agree with Adam here; providing your audience with something unique and interesting to build that network is important. Ben, you did talk about publishing regular contents in the points above. But not just regular contents. Contents that attract people like bees to honey also attract the search engine spiders. This is because search engines love the hub created by the audience of the blog.
Later on…
Adam Paudyal recently posted..Show Your Love- Be A Fan Of Endeavor Online
Hi All, thanks for your comments.
Patricia, by smaller page I mean smaller in terms of KB. There are a couple of ways to do this, consider thinks like compressing any javascript and removing any unnecessary code. Also a lot of images will cause your site to load slower. Content is always good though. Why not run your site through http://www.webpagetest.org to get an idea of how your site performs in terms of speed.
Both Adams – yes you’re right, content that attracts users is great as it helps to build loyal readers and also encourages links to your site. You need to get a balance between quality content on a regular basis to encourage users and the search engines to return often.
Thanks all of you for your comments.
Good info Ben. I never thought about how page size affects my google rank. Guess I better start crunching my images down more. But I agree that Page Rank is going to do the most for your site all around. It will by far give you the most bang for less buck.
Cool link to the WebPagetest. Running mine as we speak.
Ryan recently posted..Do I Need a Bartending License to Get a Bartending Job
Hey Ben,
Thank you for the great post, there is some really great info there for any one looking to have their site crawled by the search engine bots.
As you had stated in your post, I have found that blog commenting is a very helpful and unique way to get links back to your site, while we all know that all links are not created equal – a link is a link.
Having 1000 links back to your site from many different sources helps your site, these links can be obtained by commenting on blogs as I stated, commenting on forums or even just regular websites. Writing an article and submitting it to article submission sites could add another link or two, making a video and adding it to You Tube and adding a link in the comment section can add another link, People that start off with a new site should have a goal to get anywhere from 20-40 links a day from many different sources. What you don’t want to do or probably shy away from is getting all the links from the same site, you need diversity!
Again thank you for the great post!
I will say that you can easily get the high position in pagerank if you have reliable source where you can create good contents in your blog! Updating everyday is the key of many people to be successful!
Tracyann0312 recently posted..קורס דולות