“Big Data” is a buzz word I’ve heard thrown around many SEO conferences and presentations over the past year. The problem? Most presenters fail to go into detail about specific tools that can be used to sort big data effectively.
Efficiency when working with large websites is essential; there is not enough time in the day to manually check everything – from backlinks to URL’s, site pages to Meta implementation – when a website has thousands of pages. Tools come in handy, gathering and organising larger sets of online data without the stress and hassle of crunching it all manually. Where to start?
Here are my favourite SEO tools for organising large websites into bite-sized chunks of data:
Screaming Frog SEO Spider (http://www.screamingfrog.co.
This software uses your computers memory during the crawl, so if you are crawling a site with over 100, 000 URIs you will receive a ‘high memory usage’ warning message meaning you will need to allocate more RAM to the software. I would recommend having >4GB in your system so you don’t run out of resources. You can find more information on setting a higher memory limit in the Screaming Frog SEO help section.
With Google handing out penalties for dodgy and spammy links, it is a good idea to periodically check the quality of your backlinks.
The following tools simulate Google’s Web Crawlers and crawl the entire web (well most of it) and gather information on all the backlinks they find. By using these tools you can get a good indication of the links Google can see, and identify any low quality links pointing to your site. Once you have a list of low-quality links you can ask the sites where bad links appear to remove them, or disavow via Google Webmaster Tools if your site has already taken a penalty.
Majestic SEO (http://www.majesticseo.com/) is a staple backlink checker for most SEOs. Majestic split their crawl data into a “Fresh Index” with backlinks crawled over the last 90 days and a “Historic Index” of older links.
Another great tool for checking backlink profiles is ahrefs (https://ahrefs.com/). This tool also generates heaps of graphs based on the backlinks and anchor texts data, so it is great visual tool to quickly check the quality of backlinks and anchor texts being used.
A great set of tools to compare your backlinks against your competitors is Link Research Tools (http://www.linkresearchtools.com). This system gathers backlink data from a number of sources and provides tools for backlink analysis and competitor analysis.
Last but not least is Microsoft Excel, whether you love it or hate it, Excel is a vital tool to process and sort data for SEO.
You can extend Excel’s functionality with add-ins such as SEO Tools for Excel (http://nielsbosma.se/projects/seotools/) to work with SEO and other web metrics directly in Excel.
So when you are working with large amounts of data don’t freak out, just focus on extracting important information in smaller chunks and level up.