Dynamic Content Delivery Network:HTMLStream API

Background information

Example

• Scenario

  If you need to modify all anchor tags <a/> in an HTML page to link them to http://www.taobao.com, use the following code in EdgeRoutine.

• Sample code

  addEventListener('fetch', (event) => {
    event.respondWith(handle(event));
  });
  
  async function handle(event) {
    // 1. In this example, the HTML page that you want to modify is returned. 
    const response = await fetch("http://www.example.com");
    // 2. Configure the stream processing-based parser to manage HTML content. The parser supports multiple CSS selectors.
    // Specify the method for capturing the syntax and register a callback function for rewriting. 
    const htmlStream = new HTMLStream(
      response.body, // Specify the HTML flow that you want to modify.
      [[
        "a",         // The element selector. This specifies that all the anchor tags are selected. 
        {  
          // Register a callback function. The element callback function can be called in the anchor tags or in the element nodes of the Document Object Model (DOM) API. 
          // In the callback function, you can change the event object (e). 
          element: function(e) {
            // Modify the href attribute.
            e.setAttribute("href", "http://www.taobao.com");
          }
        }
      ]]);
    
    // 3. Return the modified request to the browser. HTMLStream is a readable stream.
    // You can use HTMLStream in all scenarios that support ReadableStream. 
    return new Response(htmlStream);
  }
              

• Result analysis

  The preceding sample code shows how to modify an HTML flow in real time by using the HTMLStream API. The following section describes how HTMLStream works:

  • The Fetch API retrieves a flow expression for the request. However, EdgeRoutine may not have retrieved the response body from the network layer. As a result, the frequency of garbage collection caused by data buffering is reduced.

  • HTMLStream is also a flow that supports the TransformStream operation. After HTMLStream receives a flow, HTMLStream calls the rewrite callback function to modify the specified HTML page in real time. If you want to modify an HTML page, you must acquire the raw data stream of the HTML page. You can then put the raw data stream in the HTML flow that you have created, as described in Step 2 in the preceding sample code.

    • The first parameter of HTMLStream is a stream, which represents the raw data stream of the HTML page.

    • The second parameter of HTMLStream is an array, which represents a group of rewriters. A rewriter is an array that includes a selector. The selector is used to specify the HTML content that you want to modify and select an object. Some attributes of the object will be called as callback functions. ["a" , {....}] in the preceding example is used to declare a rewriter, which represents an array whose element is 2. The string "a" in the preceding example represents the element selector that is used to locate all anchor tags in the document. The object in the preceding example specifies the callback object. If you choose to use an element selector, the object can contain the following callback functions:

      • The element callback function. The signature of this callback function is function(e). This callback function is called when the elements that are selected by the element selector are parsed.

      • The comments callback function. The signature of this callback function is function(e). This callback function is called when the comments that are nested inside the elements are parsed.

      • The text callback function. The signature of this callback function is function(e). This callback function is called when the text in the elements is parsed. This callback function can be called multiple times. Due to stream processing, HTMLStream may be processing a chunk of a string.

  • You can directly respond to HTMLStream. However, HTMLStream does not buffer data. Unlike parse5 and htmlparser2, HTMLStream does not generate a DOM tree, which significantly reduces the processing time and memory consumption. This way, HTMLStream can deliver high throughput and concurrency when it parses HTML content.

Rewriter

Syntax of element selectors

Callback functions for element selectors

Document selector

A document selector is used to select a specified document. To use a document selector, set the first element in the rewriter array to null. In the HTMLStream operation, you can configure only one document selector.

Callback functions for document selectors

Error handling

Callback parameters

Each callback function receives an object that represents the selected HTML tags or other relevant information. The object is also known as a callback parameter. This topic describes callback parameters such as Element, TextChunk, and Comments.

Element

• Definition

  This object is returned when the Element callback function is called. This object represents the selected HTML tags.

• Attributes

  • tagName(string): specifies the name of the tag.

  • attributes(iterator): returns an iterator that locates all attributes that are specified in the [name, value] format.

  • removed(bool): specifies whether to delete the element. This attribute is read-only. You can call the remove() method to delete an element. Typically, you must configure this attribute to ignore elements that are already deleted.

  • namespaceURI: the namespace URI of an element, such as, the SVG or Script element. This attribute is read-only.

• Methods

  • Modify attributes

    • getAttribute(name): queries an attribute name of a specified element.

    • setAttribute(name, value): sets an attribute name for a specified element and modifies the attribute name of a specified element.

    • hasAttribute(name): queries whether an attribute name exists in a specified element.

    • removeAttribute(name): deletes an attribute name from a specified element.

    Note

    Both the attribute name and value must be of string type.

  • Modify content

    • before(data, option): inserts content before the specified element (element tag).

    • after(data, option): inserts content after the specified element (element tag).

    • prepend(data, option): inserts content before the content of the specified element (after the opening tag of the element). Example: <div>(prepend) |aaaa|(append)</div>.

    • append(data, option): inserts content after the content of the specified element (before the closing tag of the element). Example: <div>(prepend) |aaaa|(append)</div>.

    • replace(data, option): replaces the entire element, including the tags and nested tags.

    • setInnerContent(data, option): specifies the element content and retains the tags and attributes.

    • remove(): deletes the specified element. After the element is deleted, the value of the removed attribute changes to true.

    • removeAndKeepContent(): deletes tags and attributes of the specified element and retains the content.

TextChunk

• Definition

  This object is returned when the Text callback function is called. This object represents a chunk of the selected HTML text.

• Attributes

  • removed(bool): specifies whether to delete the element. This attribute is read-only. You can call the remove() method to delete an element. Typically, you must configure this attribute to ignore elements that are already deleted.

  • text(string): specifies the text content. This attribute is read-only. The text may be a chunk of text. If the string is empty, it indicates that the last chunk of text is returned. In this case, you can merge all chunks of text.

  • lastInTextNode(bool): specifies whether it is the last chunk of text. This attribute is read-only. If the value of this attribute is true, the text attribute returns an empty string.

• Methods

  Modify content

  • before(data, option): inserts content before the specified element (element tag).

  • after(data, option): inserts content after the specified element (element tag).

  • replace(data, option): replaces the entire element, including the tags and nested tags.

  • remove(): deletes the specified element. After the element is deleted, the value of the removed attribute changes to true.

Comments

• Definition

  This object is returned when the Comments callback function is called. This object represents the comments in the selected HTML content.

• Attributes

  • removed(bool): specifies whether to delete the element. This attribute is read-only. You can call the remove() method to delete an element. Typically, you must configure this attribute to ignore elements that are already deleted.

  • text(string): specifies the existing comments or the comments to overwrite the existing ones. This attribute is readable and writable.

• Methods

  Modify content

  • before(data, option): inserts content before the specified element (element tag).

  • after(data, option): inserts content after the specified element (element tag).

  • replace(data, option): replaces the entire element, including the tags and nested tags.

  • remove(): deletes the specified element. After the element is deleted, the value of the removed attribute changes to true.

Doctype

• Definition

  This object is returned when the DOCTYPE callback function is called. This object represents the DOCTYPE of the selected HTML content.

• Attributes

  • name(string): specifies the DOCTYPE name. This attribute is ready-only.

  • publicId(string): returns a public identifier. If no public identifier exists, a value of null is returned. This attribute is read-only.

  • systemId(string): returns a system identifier. If no system identifier exists, a value of null is returned. This attribute is read-only.

Docend

• Definition

  This object is returned when the Docend callback function is called. This object represents the end of an HTML document.

• Method

  append(string, option): appends content to the end of the HTML document.