Make Big Data Small Data, It’s Easier to Chew
SFteam SFteam
Search Engine Optimisation
September 10,2013

Make Big Data Small Data, It’s Easier to Chew

“Big Data” is a buzz word I’ve heard thrown around many SEO conferences and presentations over the past year. The problem? Most presenters fail to go into detail about specific tools that can be used to sort big data effectively.

Wonka Big Data

Efficiency when working with large websites is essential; there is not enough time in the day to manually check everything – from backlinks to URL’s, site pages to Meta implementation – when a website has thousands of pages. Tools come in handy, gathering and organising larger sets of online data without the stress and hassle of crunching it all manually. Where to start?

Here are my favourite SEO tools for organising large websites into bite-sized chunks of data:

Website Crawling

Screaming Frog

Screaming Frog SEO Spider ( is a great tool for crawling large websites and gathering onsite information such as links, images, CSS, scripts and apps. Once a crawl has finished, you can create filters to sort the onsite data and focus on one small section at a time.

This software uses your computers memory during the crawl, so if you are crawling a site with over 100, 000 URIs you will receive a ‘high memory usage’ warning message meaning you will need to allocate more RAM to the software. I would recommend having >4GB in your system so you don’t run out of resources. You can find more information on setting a higher memory limit in the Screaming Frog SEO help section.

Backlinks Galore

With Google handing out penalties for dodgy and spammy links, it is a good idea to periodically check the quality of your backlinks.

The following tools simulate Google’s Web Crawlers and crawl the entire web (well most of it) and gather information on all the backlinks they find. By using these tools you can get a good indication of the links Google can see, and identify any low quality links pointing to your site. Once you have a list of  low-quality links you can ask the sites where bad links appear to remove them, or disavow via Google Webmaster Tools if your site has already taken a penalty.

Majestic SEO

Majestic SEO ( is a staple backlink checker for most SEOs. Majestic split their crawl data into a “Fresh Index” with backlinks crawled over the last 90 days and a “Historic Index” of older links.



Another great tool for checking backlink profiles is ahrefs ( This tool also generates heaps of graphs based on the backlinks and anchor texts data, so it is great visual tool to quickly check the quality of backlinks and anchor texts being used.


Competitor Link Analysis

Link Research Tools

A great set of tools to compare your backlinks against your competitors is Link Research Tools ( This system gathers backlink data from a number of sources and provides tools for backlink analysis and competitor analysis.

Microsoft Excel

Last but not least is Microsoft Excel, whether you love it or hate it, Excel is a vital tool to process and sort data for SEO.

You can extend Excel’s functionality with add-ins such as SEO Tools for Excel ( to work with SEO and other web metrics directly in Excel.

So when you are working with large amounts of data don’t freak out, just focus on extracting important information in smaller chunks and level up.

Awesome Raptor

Search Engine Optimisation
September 10,2013

Author: SFteam

Get In Touch
Get In Touch