Getting started with SEO for Dynamic sites like Angular, React, JavaScript etc.
A lot of modern sites use Single Page Applications (dynamic sites like Angular, React, JavaScript) which has performance/UX benefits. But these sites usually return empty HTML file initially which makes SEO crawl difficult. Even though Google is getting better in crawling dynamic sites it’s not great at this moment. Worry not, you have a workaround for it.
Dynamic Rendering – switching between client-side rendered and pre-rendered content for specific user agents. When you render the app on the server first (using pre-rendering/server-side rendering) the user (and bots) get a fully rendered HTML page – problem solved.
Dynamic rendering for dummier
How Dynamic Rendering works.?
Dynamic rendering requires your web server to detect crawlers (for example, by checking the user agent). Requests from crawlers are routed to a renderer, requests from users are served normally. Where needed, the dynamic renderer serves a version of the content that's suitable to the crawler, for example, it may serve a static HTML version. You can choose to enable the dynamic renderer for all pages or on a per-page basis.
how dynamic rendering works
Implementing Dynamic Rendering
Install and configure a dynamic renderer to transform your content into static HTML that's easier for crawlers to consume. Some common dynamic renderers are Puppeteer, Rendertron, and prerender.io.
Choose the user agents that you think should receive your static HTML and refer to your specific configuration details on how to update or add user agents. Here's an example of a list of common user agents in the Rendertron middleware:
export const botUserAgents = [ 'googlebot', 'google-structured-data-testing-tool', 'bingbot', 'linkedinbot', 'mediapartners-google', ];
If pre-rendering slows down your server or you see a high number of pre-rendering requests, consider implementing a cache for pre-rendered content, or verifying that requests are from legitimate crawlers.
Determine if the user agents require desktop or mobile content. Use dynamic serving to provide the appropriate desktop or mobile version. Here's an example of how a configuration could determine if a user agent requires desktop or mobile content:
isPrerenderedUA = userAgent.matches(botUserAgents) isMobileUA = userAgent.matches(['mobile', 'android']) if (!isPrerenderedUA) { } else { servePreRendered(isMobileUA) }
In this example, use if (!isPrerenderedUA) {...} to serve regular, client-side rendered content. Use else { servePreRendered(isMobileUA)} to serve the mobile version, if needed.
Configure your server to deliver the static HTML to the crawlers that you selected. There are several ways you can do this depending on your technology; here are a few examples:
Proxy requests coming from crawlers to the dynamic renderer.
Pre-render as part of your deployment process and make your server serve the static HTML to crawlers.
Build dynamic rendering into your custom server code.
Serve static content from a pre-rendering service to crawlers.
Use a middleware for your server (for example, the rendertron middleware).
Verify your configuration
After implementing dynamic rendering, verify if everything is working by following the below steps.
Test your mobile content with the Mobile-Friendly Test to make sure Google can see your content.
Test your desktop content with Fetch as Google to make sure that the desktop content is also visible on the rendered page (the rendered page is how Googlebot sees your page).
If you use structured data, test that your structured data renders properly with the Structured Data Testing Tool.
Troubleshooting
Even after following the steps if your site isn’t appearing in Google search results.? Try troubleshooting with following steps.
Content is incomplete or looks different
What caused the issue: Your renderer might be misconfigured or your web application might be incompatible with your rendering solution. Sometimes timeouts can also cause content to not be rendered correctly.
Fix the issue: Refer to the documentation for your specific rendering solution to debug your dynamic rendering setup.
High response times
What caused the issue: Using a headless browser to render pages on demand often causes high response times, which can cause crawlers to cancel the request and not index your content. High response times can also result in crawlers reducing their crawl-rate when crawling and indexing your content.
Fix the issue
Set up a cache for the pre-rendered HTML or create a static HTML version of your content as part of your build process.
Make sure to enable the cache in your configuration (for example, by pointing crawlers to your cache).
Check that crawlers get your content quickly by using testing tools such as the Mobile-Friendly Test or webpagetest (with a custom user agent string from the list of Google Crawler user agents). Your requests should not time out.
Structured data is missing
What caused the issue: Missing the structured data user agent, or not including JSON-LD script tags in the output can cause structured data errors.
Fix the issue
Use the Structured Data Testing Tool to make sure the structured data is present on the page. Then configure the user agent for the Structured Data Testing Tool to test the pre-rendered content.
Make sure JSON-LD script tags are included in the dynamically rendered HTML of your content. Consult the documentation of your rendering solution for more information.
Smoke Tests for dynamic rendering
Here’s a little more nuance to server side rendering troubleshooting based on some real-world situations we’ve encountered.
How To Test Server Side Rendering On A New Site Before It’s Launched
It often is the case that SEOs get brought into the process well after a site has been built, but only a few days before it will be launched. We will need a way to test the new site in Google without competing in Google with the old site. For a variety of reasons we don’t want the entire new site to get crawled and indexed, but we want to know that Googlebot can index the content on a URL, that it can crawl internal links and that it can rank for relevant queries. Here’s how to do this:
Create test URLs on new site for each template (or use URLs that have already been built) and make sure they are linked from the home page.
Add a robots.txt file that allows only these test URLs to be crawled.
Here’s an example: User-Agent: Googlebot Disallow: / (this means don’t crawl the entire site) Allow: /$ (allow Gbot to crawl only the home page even though the rest of the site is blocked in the line above) Allow: /test-directory/$ (allow crawling of just the /test-directory/ URL) Allow: /test-directory/test-url (allow crawling of /test-directory/test-url)(you can add as many URLs as you want to test – the more you test, the more certain you can be, but a handful is usually fine)
Once the robots.txt is set up, verify the test site in Google Search Console.
Use the Fetch as Google tool to fetch and render the home page and request crawling of all linked URLs. We will be testing here that Google can index all of the content on the home page and can crawl the links to find the test URLs. You can view how the content on the home page looks in the Fetch tool, but I wouldn’t necessarily trust it – we sometimes see this tool out of sync with what actually appears in Google.
In a few minutes, at least the test home page should be indexed. Do exact match searches for text that appears in the title tag and in the body of the home page. If the text is generic, you may have to include site:domain.com in your query to focus only on the test domain. You are looking for your test URL to show up in the results. This is a signal that at least Google can index and understand the content on your homepage. This does not mean the page will rank well, but at least it now has a shot.
If the test links are crawlable, soon you should the test URLs linked from the home page show up in Google. Do the same tests. If they don’t show up within 24 hours, while this doesn’t necessarily mean the links aren’t crawlable, it’s at least a signal in that direction. You can also look at the text-only cache of the indexed test home page. If the links are crawlable, you should see them there.
If you want to get more data, unblock more URLs in robots.txt and request more indexing.
Once you have finished the test, request removal of the test domain in GSC via the Remove URLs tool.
We often can get this process done in 24 hours, but we recommend to clients giving it a week in case we run into any issues.
Pro-tip: If you are using Chrome and looking at a test URL for the SEO content like title tag text, often SEO extensions and viewing the source will only show the “hooks” (e.g. {metaservice.metaTitle}) and not the actual text. Open Chrome Developer Tools and look in the Elements section. The SEO stuff should be there.
Do Not Block Googlebot on Your PreRender Server
Believe it or not, we had a client do this. Someone was afraid that Googlebot was going to eat up a lot of bandwidth and cost them $. I guess they were less afraid of not making money to pay for that bandwidth.
Do Not Throttle Googlebot on Your PreRender Server
We convinced the same client to unblock Googlebot, but noticed in Google Search Console’s crawl report that pages crawled per day was very low. Again someone was trying to save money in a way that guaranteed them to lose money. There may be some threshold where you may want to limit Googlebot’s crawling, but my sense is Googlebot is pretty good at figuring that out for you.