What is JavaScript?

JavaScript (JS) was initially created with the goal of making HTML web pages interact with Java web applets.

JS has since mostly been used to add animations and effects to pages to make them feel more dynamic. This is inherently different from existing HTML and CSS languages where you set the basic structure and then design how the page should look.

The computer programming language JavaScript should not be confused with another language, Java, as they are very different. JavaScript has been noted as one of the fastest-growing languages in the world, and it’s unlikely that’s going to change anytime soon. 

This comprehensive guide to JavaScript SEO is intended to arm developers and SEO professionals with the knowledge to make use of JavaScript effectively. An accompanying JavaScript Glossary has also been produced to define some common terms relating to JS.

Why should SEOs care about JavaScript?

JavaScript is being used more and more across the web to provide more dynamic experiences to web users. This trend has been growing in recent years, with websites like Airbnb running solely on JavaScript (in this case, a variant of JavaScript called React JS).

While the experiences produced by JavaScript can be really good for users, the same cannot always be said for Google and other search bots that can struggle to render JS unless managed correctly – as we’ll go on to explain in this guide. The reality is that websites seeking to use JavaScript must consider Google in their implementation, ensuring their site can be crawled, rendered and indexed effectively.

The purpose of this guide is to arm SEOs with the knowledge to understand JavaScript, manage its effective implementation and to ensure your site is JavaScript SEO friendly.

JavaScript rendering

Rendering focuses on fetching the relevant data to populate a page, and the visual layout templates and components and then merging them together to produce HTML a web browser can display. It’s here where we should introduce two fundamental concepts; server-side rendering and client-side rendering. It is critical for every SEO managing JavaScript websites to recognise the difference between the two. 

The established approach, server-side rendering, involves a browser or a search engine bot (crawler) receiving HTML that describes the page exactly. So, the content is already in place, and your browser or search engine bot must download the attached assets (CSS, images, etc) to present how the page has been designed to. As this is the traditional approach, search engines generally do not have a problem with server-side rendered content. Traditionally websites that render like this would be programmed in PHP, ASP or Ruby and might have used popular content management systems like Kentico, WordPress or Magento.  

However, the more modernised client-side rendering approach is far different. Many search engine bots have been noted to struggle with this path as a result of the primary load projects a blank HTML page, with very little content, back to the bot. The included JavaScript code then sends a request to the server and uses the data it receives back to build and render the page. 

How does JavaScript rendering work with Googlebot?

Googlebot essentially processes JavaScript in three major stages, these being;

  1. Crawling
  2. Rendering 
  3. Indexing 

As shown in Google’s diagram, Googlebot places pages within a queue for their crawling and rendering processes. From here, Googlebot fetches a URL from the crawling queue and reads the robots.txt file to see whether the URL has been disallowed or not. 

From here, Googlebot then parses the HTML response to other URL and adds to the crawling queue. Once Googlebot’s resources allow, a headless Chromium renders the page and executes the JavaScript, that we will discuss later. The rendered HTML is then used to index the page. 

However, during Google’s 2018 I/O conference, they stated that JavaScript is now being processed in two separate waves. These waves can take a couple of days – a week to properly index JavaScript-heavy websites, but Google is actively working towards rendering in near-real-time. The diagram below displays the process Googlebot goes through when indexing at current.

 

As Google is running two separate waves of indexing, it is possible for some details to be missed during the indexing process. For example, if you’re not server-side rendering crucial title tags and meta descriptions, Google may miss this on the second way, having negative implications on your organic visibility on the SERPs.

Javascript rendering with the DOM

Essentially, JavaScript rendering works when the Document Object Model (DOM) of the page has sufficiently loaded. To explain further, the DOM is the foundation of the dynamically generated page. Standard HTML pages are static where they are unchanged, whereas dynamic pages are those that have the ability to change and can be generated on-the-fly.

The JavaScript, as mentioned earlier, and resources are vital to render a page, to where the JavaScript execution is then performed. The JavaScript then makes changes within the DOM and presents the HTML code of the specific web page. The selected search engine bot will usually wait for around three seconds before taking a snapshot of the HTML code that has been generated. 

What is the difference between crawling and indexing?

Crawling and indexing are two different things that can be confused within the SEO industry. Crawling is associated with a search engine bot, such as Googlebot, discovering all the content or code that is on a web page and analyses it. Indexing, on the other hand, means that the page is likely to show up in the Search Engine Results Page (SERPs).

Despite the refinement of the bots at crawling and indexing, JavaScript does make this process a lot less efficient and more expensive. JavaScript ingrained content and links create an immense effort for crawlers to render all of the web pages. These search engines will crawl and index JavaScript generated pages, but this is likely to take longer than a static page due to the going back and forth between the crawler and indexer, however with recent announcements this is improving. 

Evergreen Google Bot

It should be noted that Googlebot is now running on the latest Chromium rendering engine (74 at the point of this post’s publication) when rendering pages for search. Moving forward, Googlebot will regularly update its rendering engine to ensure support for latest web platform features.

Inefficiencies of JavaScript for crawling and indexing

JavaScript has been scrutinised by many SEO professionals as being inefficient in comparison to other programming languages. As opposed to allowing Googlebot to download and extract links from HTML and CSS files to then index the page, JavaScript creates an extra step.

The JavaScript rendering process as a whole is a lot more complex. Ultimately, it takes an extended period of time to crawl and index JavaScript websites, then HTML ones due to the currently JavaScript rendering process and the wait pages have in the rendering queue. 

[During Google’s I/O ‘18 conference, Google officially stated that “the rendering of JavaScript-powered websites in Google Search is deferred until Googlebot has resources available to process that content”. As a result, Google clearly needs additional resources to fully crawl, render and index JavaScript-powered, client-side rendering.]

Impacts of JavaScript for SEO

Unfortunately, many JavaScript frameworks deployed without SEO considerations taken into account, this can have a drastic impact on the overall performance of the website. We have outlined the key impacts that SEOs should look out for when beginning to audit a JavaScript-powered website/landing page. 

Internal linking

With internal linking being one of the webmasters top priorities, regular anchor tags should be used within the HTML or the DOM as opposed to leveraging JavaScript functions, such as JavaScript onclick events to allow the users to traverse from web page to page across the site. Internal linking is considered as such a strong signal to search engines regarding the complete website architecture and overall importance of pages. 

Content accessibility

For content to be clear for both the user and search engine bots, the content must be indexable within the load event, require an indexable URL and use the same best practices SEO for HTML applying to render JavaScript sites. Not only this but as aligned with basic SEO practices each piece of content on a website should be unique, from the website itself and across other sources on the internet shouldn’t be duplicated. 

Obtainability

Search engines have been reported to deploy headless browsing, a type of software that can access web pages, but does not show the user and transfers the content of the web page to another program that runs on the backend. A headless browser helps to render the DOM to gain a better understanding of the user’s experience and the content situations on the page. 

However, it’s important to understand how JavaScript is executed and interacted with search engines bots, with huge organic consequences if not. An example of this is the globally renowned brand Hulu, where a significant visibility drop was seen due to a coding problem and the way the site was serving JavaScript to Googlebot. For further information on this drop off in traffic, read here.  

Rendering

JavaScript can affect the overall rendering of a web page. Therefore, if something is render-blocking, this is likely to delay the page from loading as quickly as it has the potential to. As a rule of thumb, Google recommends to completely remove or at least defer any JavaScript that interferes with the loading above the fold of content on a website.

Above the fold refers to the part of the web page that is visible when the page initially loads. The subsequent portion of the page requires scrolling is called “below the fold”. This can be apparent across a range of devices, including; desktops, mobiles, iPad’s and many more. In order to remove this, please refer to the section below that outlines key tools to use. 

Single Page Application considerations

A single page application (SPA) is a web application/ website that has been primarily designed and built to operate efficiently on the web. These pages are dynamically rewritten and loaded with the pieces you require, as opposed to loading an entire page from the server. 

The SPA approach provides a fast loading time, uses less bandwidth and provides the user with a pleasant experience by making the application behave more like a desktop application. It should be noted that there are many different SPA framework options available, dependant, on the use of the application. These include; React.js, Angular.js, Backbone.js, Preact, Next.js and hundreds more. 

When rendering SPAs, John Mueller stated the following. 

There can be many hurdles when Google attempts to crawl and index the content on the SPA. Therefore, if you’re using SPAs, it’s recommended to test out multiple times, using the Fetch command, to understand what Google is able to pick up. 

How to make your JavaScript website SEO friendly

Now that we’ve discussed the differences between indexing and crawling, as well as the impact JavaScript can have on SEO, we’ll now cover the very best practices for JavaScript SEO. 

JavaScript SEO best practises

 

5-second timeout

Although Google has not been officially stated, it’s noted that Google shouldn’t wait longer than 5 seconds, although John Mueller has indeed stated this will be challenging for many sites. Therefore, any content in by the load event, around 5 seconds, will be indexable.

Indexable URLs

Pages require indexable URLs that offer server-side support for each landing page. This includes each category, subcategory and product page. 

Descriptive metadata

With each individual page on site having a specific focus and target, they should also include descriptive titles and meta descriptions to positively help search engine bots and users to precisely detect what the page is about. Not only this, but it helps users determine if this is the most suitable page for their search query. 

Using your browser’s “Inspect” feature 

Once the rendered HTML has been obtained and meets the level of a traditional landing page expected by Google, many impacting factors will solve themselves. 

To review rendered HTML and general JavaScript elements, the inspect element of Google Chrome can be used to assist users to discover further information about the webpage that is hidden from the users view. To discover hidden JavaScript files, such as user behaviour when interacting with a web page, you can attain this information from the Sources tab of the Inspect Element. 

If you cannot see your full amount of content within the inspect element, it’s likely you’re using JavaScript to render within the browser, known as client-side rendering. 

URL Inspection tool in GSC

The URL inspection tool allows you to analyse a specific URL on your website to understand the full status of how Google is viewing it. The URL inspection tool provides the valuable data around the crawling, indexing and further information from Google’s index, such as successful indexation or structured data errors causing issues.

Improve the page loading speed

Google indicated that page speed is one of the signals used by their complex algorithm to rank pages, as well as, a faster page speed allows search engine bots to increase the number of pages helping with the overall indexation of a site. From a JavaScript point of view, making the web page more interactive and dynamic for users can come with some costs with page speed. To mitigate this, lazy loading can be advised to use for certain components, usually ones that aren’t fully required above the fold.

 

Advantages and Disadvantages of JavaScript

Advantages of JavaScript:

  • Produces the same result and is supported on all modern browsers
  • Allows for a user-friendly, highly interactive build of websites
  • Relatively easy to debug and test as JavaScript code is interpreted line by line
  • Can be used by both front-end and back-end developers

Disadvantages of JavaScript:

  • Google potentially not being able to render and index landing pages created in JavaScript
  • Requiring JavaScript to be rendered on a page can negatively impact two key areas:
    • Site speed
    • Search engine crawling and indexing

However, depending on which rendering method you use, you can reduce page load speed and make sure content is accessible to search engines for crawling and indexing.

 

JavaScript SEO Tools: Check Your JavaScript

With the endless capabilities, JavaScript has, there’s an abundance of helpful tools are available to assess and fix issues with JavaScript code. Here are just a few that you will find helpful as an SEO:

  • URL Inspection Tool

Found within Google Search Console, the URL Inspection Tool shows information on whether Google was able to crawl / index a page, whether any errors are occurring and why. 

  • Mobile-Friendly Test

Google’s mobile-friendly tester provides information regarding how easy it is for a visitor to navigate your website on a mobile device. 

  • Page Speed Insights

Google’s page speed insights tool (PSI) effectively details the performance of mobile and desktop devices. In addition to this, this tool also provides recommendations on how this can be improved. 

  • Site: Command

The site: command is one of the most direct tools to help see if Google has properly indexed your content. In order to do this, you can complete the following command into Google – site: [your website URL] “text snippet or query”

  • Diffchecker

Diffchecker is a unique tool that allows you to compare two types of text files and review the differences between both. This is especially useful for performing analysis of a webpages original source code against the rendered code. This tool delivers comprehensive comparisons to how the content has changed after being rendered.  

  • Chrome DevTools

Chrome DevTools is a set of tools for experienced web developers to build directly into Google’s Chrome browser. Chrome DevTools can help you to edit and make quick styled changes without needing to use a text editor. The help to discover problems in a fast manner, which in turn, can help to build better websites in a quicker period. 

  • DeepCrawl 

DeepCrawl can be used to crawl and render JavaScript across all the different pages of a website, in the same pattern that Google would. This intelligent tool allows you to effectively test and monitor rendering on a mass scale, in turn helping to understand is the JavaScript can be crawled and indexed by search engines bots.

  • BuiltWith

BuiltWith is a free website profiler that helps to discover what framework websites are being built with. Not only this, but it’ll also tell you any integrations with a third party website that the website has.

If you’d like to find out more information regarding JavaScript SEO or you’re undergoing some crawling or indexation issues due to JavaScript code, contact our team of technical SEO specialists, today.

Georgie Kemp

SEO Executive

Georgie is an SEO Executive and works carefully to improve a range of clients online organic presence. In her spare time, you'll catch her eating sushi, dog spotting or attempting to snowboard.

Leave a Reply

Your email address will not be published. Required fields are marked *