5 Ways to Get Google to Crawl and Index Your Site
By Emma Grant | September 7th 2020
By Emma Grant 
Everyone wants to get as much traffic as possible coming to their website. Any SEO agency will confirm that it’s vital for business growth, and a crucial tool when it comes to boosting sales.
Around a third of website traffic comes from organic search. So you need to be there in those search engine results pages. The question is, how to get your website indeed by Google and the other search engines? Well, you have a choice of two ways.
One is to wait and let it happen naturally. All fine if you have a few weeks or even months to kill before you want to start getting traffic, because that’s precisely how long it could take. The other option is to take action and make it happen straight away. Get your site indexed immediately, and you’ve got more time to start building your audience.
Let’s take a look then at how to speed up that whole site indexing process. But first, a few explanatories.
What does website crawling and indexing mean?
The way Google and other search engines discover newly added website pages is by ‘crawling’. This is done by sending out software known as ‘web spiders’ which follow hyperlinks on the web, discovering new content as the go. Once a new page is detected, it is ‘indexed’, in other words, stored in a database, ready to be called up when a relevant search is conducted. The process by which an indexed page is pulled from the database to match a search query is known as an ‘algorithm’.
The thing with indexing is that you don’t want it to happen just once, plus you want to make sure that ALL your pages are indexed. The ideal scenario is for the search engines to keep re-indexing your site. This will happen whenever you add new content, and it will help to refresh your position in the search results. This is why any SEO agency will tell you that content marketing is such a crucial part of search engine optimisation.
Is your website indexed by Google? How to check…
It’s easy to check whether Google has indexed your website. Simply type the following into the Google search bar:
site:yourwebsite.com
The number highlighted shows the approximate number of pages Google has indexed. If the website isn’t indexed, you won’t see any results listed. Not indexed? Here are the top ways to get that sorted. Already indexed? Great. But chances are, there is scope to get better results. Here’s how.
How to get crawled and indexed by Google
The frequency at which your site is indexed can directly impact upon your search results, which makes it crucial to ensure all your latest content is on hand for the search spiders to crawl. It’s also essential to be aware that the faster your website loads, the quicker Google will index it. As we say all the time, speed matters. Always bear this in mind as a fundamental rule. Now onto five ways to get your site crawled and indexed by Google.
1. Create a Sitemap
A sitemap is a map of your site. It’s a document that tells search engine crawlers where they can and can’t go. Crawlers check this page to learn about your site: its size, the most important pages and where the content is. The sitemap reduces the time taken to crawl a website down from around 24 hours to just a few minutes.
The sitemap keeps Google informed as to how often to look for new pages. This is really important if you’re adding new content on a regular basis, or posting news articles that are time-sensitive. You want those pages crawled and indexed as soon as possible.
There are various ways to create a sitemap. If you use the Yoast SEO plugin, then sitemaps will be created automatically and updated whenever you add, edit or delete content. So you will never have to worry about sitemaps if you use Yoast.
If you don’t, one option is Screaming Frog. For sites with fewer than 500 pages, this is free. For larger sites, a licence is required. Another alternative is Google XML Sitemaps. This is a free plugin for WordPress sites and a great way to generate a sitemap if you’re not using Yoast.
2. Submit Your Sitemap to Google
Once you have your sitemap, you’ll need to submit it to Google via the Search Console. Here’s how.
This process informs Google about all the pages you want indexed, and can help its crawlers find the most important ones. Do be aware though that this isn’t a one-off process. Every time you add content and update your website, you’ll need to keep your sitemap up to date so that Google has a current picture of how your site looks.
3. Create a robots.txt
A robots.txt is a simple file that sits on your website and instructs search engines as to what they should and shouldn’t index.
This file is the first stop-off for search engine spiders on their journey through your website. If your robots.txt file says not to index the page, then the spider will pass it by. Any SEO agency in London will know how essential this is, because this file makes it possible to request a re-crawl, or opt pages out of crawling altogether. Vitally, it also tells Google to crawl the pages you want crawled. Excluding selected pages helps crawlers focus on the content that you actually want indexed, which can enhance the frequency at which your valuable pages are crawled for fresh information.
Here’s why the robots.txt file is so important, and how to create one for SEO.
4. Create internal links
One of the best ways to encourage crawling and indexing is via internal linking.
Search engine spiders use links to form their paths through the web. When on a site, the spiders follow any linked paths from one page to another. So the more internal linking you do, the smoother the crawling path and the easier it will be for search engines to locate new content.
In addition, make sure your navigation is nice and simple. Not only will this create a positive user experience, it will also make it more straightforward for crawlers to find and index your content. A well thought out navigation will help the search engine crawlers fathom which pages hold the most importance, and see how the individual pages relate to each other.
Other strategies you can use to organise your content with a view to making it easier to get your pages indexed include:
- Content hubs – grouped sections of similar content, such as blog or product categories.
- Bookmarks – if you publish long articles or guides, linking to the various sections courtesy of a page index with bookmarks will help both users and crawlers better navigate to relevant content.
- Content tables – similar to bookmarks, tables make for an enhanced user experience. WordPress users can make light work of this by using the Easy Table of Contents plugin.
5. Acquire inbound links
Any SEO agency will champion the use of an inbound link building strategy. This is a proven way to get a website crawled and indexed faster, as well as boosting trust and authority for SEO. How it works is simple.
The more links you have coming into your site from trusted sites, the more credible Google considers you and the better chance you’ll have of achieving good search positions.
In addition, links from other sites attract crawlers to your web pages. When one of your new pages gets a link, it will be crawled and indexed faster. Whilst you will find some links happening naturally, your ideal SEO strategy will be to be proactive in achieving the links.
Here are a few link building tactics to follow:
- Content syndication – the process of having your content re-posted on other websites with a credit to your site
- Social media sharing – posting links to your content on relevant social media platforms
- Infographic directory submissions – share your infographic content to specialist infographic directories
Looking for SEO services in London? Talk to top-rated SEO agency Figment.
If you’re looking for insight and expertise in all aspects of search engine optimisation, Figment is an SEO agency in London businesses don’t hesitate to recommend. Our specialists know all the best practice methods that will get your website crawled and indexed quickly by Google and the other search engines, so you can start enjoying a return on your investment and increased sales without delay.