Cromwell

Our initial analysis was conducted using data from Google Search Console in conjunction with data from log file analysis. The data showed us that Cromwell’s site was not being indexed fully because of their use of the React JavaScript framework. Google could not access the content that was only visible with JavaScript enabled so, while the framework allowed our client to have the functionality they desired in their UI, it also made it nearly impossible for Google to crawl.

Using data from Screaming Frog, we were able to collate a list of all available pages on the site. These were then categorised according to the company’s aims, identifying immediately those which could be no-indexed. In this way, we improved crawl efficiency from the outset by removing pages that didn’t need to be indexed, allowing more time for Google to access important content.

We also built a script that allowed us to pull Cromwell’s current site using Google’s headless browser “Headless Chrome”/”Puppeteer”, so we could view the site as Google’s crawlers did. Data from this script-based analysis enabled us to uncover specific rendering issues within each page and area of the site and educate the client on where problems lay.

We recommended server-side rendering as a solution to facilitate effective crawls. Server-side rendering means that Google is not expected to render the content for itself, meaning we are in full control of what we present to Googlebot. This was no small feat, with the site having more than 200,000 pages. We worked closely with our client’s development team to implement this, monitoring data on Google Search Console to see how many pages were being indexed as a result.

Most recently, we’ve been creating the perfect structure across the site for internal linking and removing duplicate content issues. To do this, we’ve used data from Screaming Frog to create a granular sitemap to categorise content across the website. This has helped us to identify which areas of the site were not being crawled and where there was a lack of internal links.

Through this approach, we have been able to identify ‘hidden’ areas and bring them to the forefront through the addition of new internal links, canonicalising or redirecting content where duplication occurred. By doing this, we were able to further optimise the crawl for Googlebot whilst improving the overall user experience too.