Using data from Screaming Frog, we were able to collate a list of all available pages on the site. These were then categorised according to the company’s aims, identifying immediately those which could be no-indexed. In this way, we improved crawl efficiency from the outset by removing pages that didn’t need to be indexed, allowing more time for Google to access important content.
We also built a script that allowed us to pull Cromwell’s current site using Google’s headless browser “Headless Chrome”/”Puppeteer”, so we could view the site as Google’s crawlers did. Data from this script-based analysis enabled us to uncover specific rendering issues within each page and area of the site and educate the client on where problems lay.
Appraise various server-side rendering options, including on-server and third-party frameworks and costings
We recommended server-side rendering as a solution to facilitate effective crawls. Server-side rendering means that Google is not expected to render the content for itself, meaning we are in full control of what we present to Googlebot. This was no small feat, with the site having more than 200,000 pages. We worked closely with our client’s development team to implement this, monitoring data on Google Search Console to see how many pages were being indexed as a result.
Update the site structure to facilitate effective internal linking
Most recently, we’ve been creating the perfect structure across the site for internal linking and removing duplicate content issues. To do this, we’ve used data from Screaming Frog to create a granular sitemap to categorise content across the website. This has helped us to identify which areas of the site were not being crawled and where there was a lack of internal links.
Through this approach, we have been able to identify ‘hidden’ areas and bring them to the forefront through the addition of new internal links, canonicalising or redirecting content where duplication occurred. By doing this, we were able to further optimise the crawl for Googlebot whilst improving the overall user experience too.