Asynchronous HST Components and Containers

Introduction

Both HST components and HST containers can be rendered asynchronously out-of-the-box. Common use cases are:

• Content that is customized for a particular visitor (e.g. a "stores near you" component)

• Content that accesses a potentially slow external service (e.g. a Twitter feed)

• Content that should not be cached or indexed by search engines

Only live requests will actually load components asynchronously. For preview requests or views of the application in the channel manager in the CMS, components that are marked to load asynchronously will be rendered synchronously.

Configure an HST Component to be Rendered Asynchronously

Add the following JCR property to an hst:component or hst:container node to make it asynchronous:

hst:async: true

Pages that contain asynchronous components and/or containers will automatically include a Javascript file in the response. The javascript file will execute separate Ajax calls to fetch the rendered output of each asynchronous component and/or container. The only thing that needs to be present in the root renderer (jsp / freemarker) template is the following:

1) If you already have in your renderer the inclusion of hst:headContributions, then make sure you add "scripts" to the categoryExcludes. If you already have categoryExcludes, it can contain multiple categories separated by a comma.

<hst:headContributions categoryExcludes="scripts" xhtml="true"/>

2) And make sure that at the end of the renderer, for example just before the closing of the html body ( </body>) you include the following:

<hst:headContributions categoryIncludes="scripts" xhtml="true"/>

The latter will make sure that the loading of asynchronous parts in the html through ajax calls is triggered

Supported Asynchronous Modes

When configuring hst:async = true on an hst component, the default asynchronous rendering mode is by a client side ajax request that fetches the content of the asynchronous components and aggregates it in the page when done. However, you might not want the aggregation of the page to be done on the client. In that case you can also opt for ESI or SSI asynchronous modes to aggregate the asynchronous components on the server, for example in some CDN, squid, varnish, httpd, etc. The supported asynchronous modes are:

1. ajax (default)
2. ssi (Server Side Includes)
3. esi (Edge Side Includes)

You can set the asynchronous mode via property  hst:asyncmode. For example

​hst:asyncmode: esi

Restrictions

Asynchronous HST components and containers have the following restrictions:

• All descendant components of an asynchronous component will be rendered in the asynchronous request for the asynchronous ancestor.

• Asynchronous components should be independent of synchronous (i.e. 'normal') HST components. For example, you cannot have a synchronous component put something in the request context that is needed by an asynchronous component. It is possible with marking the hst:component to also have hst:standalone = false but this is strongly discouraged. Also see HST component rendering.

• Asynchronous components cannot contribute hst:headContributions since they are not rendered (they are later async rendered) when the headContributions are written to the html.

• If a client does not have Javascript enabled, asynchronous components will not be rendered if the async loading mode is ajax

A special case of the last restriction is that the output of an asynchronous component will not be indexed by search engines, as search engine bots do not execute Ajax calls. Since asynchronous calls are typically used for parts that are, for example, completely personalized, or very much depend on some external service like a Twitter feed, this is in general even better.

Although mentioned above as the third restriction, please thus realize that:

Async Components Nested Below Async Components

When marking a hst:component as hst:async = true that already has an ancestor hst:component that is marked to be aynchronous, then, when the ancestor is loaded asynchronously, its descendants are directly loaded as well, even if the descendant is marked to be asynchronous. Thus, nested asynchronous HST components are allowed, but an asynchronously rendered component will render all its descendants as well regardless whether they are asynchronous or not.

How It Works

Marking an HST component with hst:async = true, triggers the HST request processing to skip the doBeforeRender and render method of the async component and all its descendant. Instead, it will inject in the response a placeholder div with an id that contains a HST component rendering URL, and a class attribute with some obfuscated value. The aggregation valve also adds a headContribution for the asynchronous Javascript loading that is added to the html through <hst:headContributions categoryIncludes="scripts" xhtml="true"/>.

Disable Asynchronous HST Component Rendering 

Global configuration

By default, asynchronous HST component rendering is enabled. If you want to disable it globally regardless of HST component configurations explained above, set the following property to false in the HST-2 Container Configuration file: 

# flag whether the asynchronous component window rendering is enabled by default.
default.asynchronous.component.window.rendering.enabled = false

Automatic Disabling on Search Engine or Bot Requests

When page requests come from web crawlers such as search engines and various web bots, it is not desirable to exclude the content of the asynchronous HST components in the page responses because it may cause a negative impact on SEO.

Since v13.4, asynchronous HST component rendering is automatically disabled when detecting requests from web crawlers, in order to give a full page response including the content of asynchronous HST components to the web crawlers.

By default, many known web crawlers are detected by checking if the User-Agent HTTP header value contains any of the following keys:

        abachobot, accoona-ai-agent, addsugarspiderbot, adsbot-google,\
        anyapexbot, aolbuild, appengine-google, arachmo, baidu, bingpreview, b-l-i-t-z-b-o-t, becomebot, beslistbot,\
        billybobbot, bimbot, bingbot, blitzbot, boitho.com-dc, boitho.com-robot, bot, btbot,\
        catchbot, cerberian drtrs, charlotte, converacrawler, cosmos, covario-ids,\
        dataparksearch, diamondbot, digg deeper, discobot, dotbot, duckduckgo, earthcom.info, emeraldshield.com webbot,\
        envolk[its]spider, esperanzabot, exabot, fast enterprise crawler,\
        fast-webcrawler, fdse robot, findlinks, furlbot, fyberspider, g2crawler, gaisbot,\
        galaxybot, geniebot, gigabot, girafabot, googlebot, googlebot-image, googlebot-mobile,\
        googlebot-news, googlebot-video, gurujibot, happyfunbot, hl_ftien_spider,\
        holmes, htdig, iaskspider, ia_archiver, iccrawler, ichiro, igdespyder, irlbot,\
        issuecrawler, jaxified bot, jyxobot, koepabot, l.webis, lapozzbot, larbin,\
        ldspider, lexxebot, linguee bot, linkwalker, lmspider, lwp-trivial, mabontland,\
        magpie-crawler, mediapartners-google, mj12bot, mlbot, mnogosearch, mogimogi,\
        mojeekbot, moreoverbot, morning paper, msnbot, msrbot, mvaclient, mxbot, netresearchserver,\
        netseer crawler, newsgator, ng-search, nicebot, noxtrumbot, nusearch spider,\
        nutchcvs, nymesis, obot, oegp, omgilibot, omniexplorer_bot, oozbot, orbiter,\
        pagebiteshyperbot, peew, polybot, pompos, postpost, psbot, pycurl, qseero,\
        radian6, rampybot, rufusbot, sandcrawler, sbider, scoutjet, scrubby, searchsight,\
        seekbot, semanticdiscovery, sensis web crawler, 'seochat::bot', seznambot,\
        shim-crawler, shopwiki, shoula robot, silk, sitebot, snappy, sogou spider,\
        sosospider, speedy spider, sqworm, stackrambler, suggybot, surveybot, synoobot,\
        teoma, terrawizbot, thesubot, thumbnail.cz robot, tineye, truwogps, turnitinbot,\
        tweetedtimes bot, twengabot, updated, urlfilebot, vagabondo, voilabot, vortex,\
        voyager, vyu2, webcollage, websquash.com, wf84, wofindeich robot, womlpefactory,\
        xaldon_webspider, yacy, yahoo! slurp, yahoo! slurp china, yahooseeker, yahooseeker-testing,\
        yandex, yasaklibot, yeti, yodaobot, yooglifetchagent,\
        youdaobot, zao, zealbot, zspider, zyborg, abilogicbot, link valet, link validity check,\
        linkexaminer, linksmanager.com_bot, mojoo robot, notifixious, online link validator,\
        ploetz + zeller, reciprocal link system pro, rel link checker lite, sitebar, vivante link checker,\
        w3c-checklink, watchmouse, copperegg, revealuptime, bot/0.1, compspybot, feedfetcher-google, rogerbot,\
        sogou web spider, twitterbot, xenu link sleuth, crawler, spider,\
        scraper, bash, java, facebook, nutch, ruby, httpclient, optimizer, go 1.1 package,\
        megaindex, brokenlinkcheck, dlvr.it, ltx71, qwantify, python-urllib,\
        python-requests, google favicon, binlar, metauri, coccoc, disqus, xurl, netlyzer,\
        gigablastopensource, panscient, hubspot marketing grader, kimengi, libwww-perl, trove, typhoeus

To check additional User-Agent HTTP header key values in your project, set search.engine.or.bot.user.agent.patterns.extra property in the HST-2 Container Configuration file:

# Extra Comma separated Search Engine or Bot User Agent regex patterns that could be added by end projects. All must be in lower cases.
search.engine.or.bot.user.agent.patterns.extra = mycompanybot, yetanotherbot

The two new User-Agent header key values, "mycompanybot" and "yetanotherbot", will be also checked in web crawler detection by this example configuration.

API Support

Since v13.4, you may use HstRequestContext#isSearchEngineOrBotRequest() API if your application needs to know whether or not the request comes from a detected web crawler.

Troubleshooting

If your asynchronous component does not get shown in the page (and you did not configure hst:asyncmode to be esi or ssi), check out with browser development tools whether an ajax network request is done. It should be a request that contains something like

?_hn:type=component-rendering&_hn:ref=r6_r1

Note : where the _hn:ref value r6_r1 is different per component. The _hn:type=component-rendering indicates that the ajax call should be done by a Component Rendering URL. 

If you do not see this request, check out the html code of the page. It should contain in the html/head a javascript element containing something like

a425658310.b425658310.AsyncPage = { .....

and at the end of the html page it should contain

<script type="text/javascript">
//<![CDATA[
a425658310.b425658310.AsyncPage.load();
//]]>
</script>

The part  a425658310.b425658310 is different per project and randomly created. If the snippet containing  a425658310.b425658310.AsyncPage.load(); is in the html/head, asynchronous ajax calls for components won't work. To fix this problem, you need to add to the hst:headContributions in the base page template in the <head> element the categoryExcludes scripts and add to the bottom  categoryIncludes scripts. The simplified archetype project base template looks as follows to make sure asynchronous components work out of the box. 

<!doctype html>
</html>
  <head>
    <snip/>
    <@hst.headContributions categoryExcludes="htmlBodyEnd, scripts" xhtml=true/>
  </head>
  <body>
    <snip/>
    <@hst.headContributions categoryIncludes="htmlBodyEnd, scripts" xhtml=true/>
  </body>
</html>

Thus, make sure to exclude  script category in the top and include it in the bottom.

Customizing Asynchronous AJAX Component Loading

When a page is loaded, HST writes a scriptlet like the following to load the asynchronous components in ajax mode:

<script type="text/javascript">
//<![CDATA[
a425658310.b425658310.AsyncPage.load();
//]]>
</script>

Sometimes, a project needs to customize the generated scriptlet. For example, it needs to delay the JavaScript call (e.g, a425658310.b425658310.AsyncPage.load()) in order to call some other AJAX calls first and establish the visitor's state available on an external analytic system.

In this case, you can customize the generated scriptlet by configuring the ajax.asynchronous.component.windows.load.js.fragment.template property in hst-config.properties like this:

# Example customization to wrap the default JavaScript fragment in
# java.text.MessageFormat format to render asynchronous components on page load.
ajax.asynchronous.component.windows.load.js.fragment.template = registerAsyncComponentsRenderingCallback(function() '{' {0} '}');

With the custom configuration, HST now generates the scriptlet like the following:

<script type="text/javascript">
//<![CDATA[
registerAsyncComponentsRenderingCallback(function() { a425658310.b425658310.AsyncPage.load(); });
//]]>
</script>

The above example assumes that you add a custom JavaScript function, registerAsyncComponentsRenderingCallback(cb), which gets the callback function to be invoked later when it's ready to render asynchronous components. For example, the custom JavaScript function could look like this:

function registerAsyncComponentsRenderingCallback(callback) {
  // do something to establish visitor's state more here. e.g, AJAX calls on external analytic system
  // ...
  callback();
}

You can wrap the scriptlet which loads all the asynchronous components on page loading by configuring the ajax.asynchronous.component.windows.load.js.fragment.template property.