This post is meant to be a guide on the Botpress architecture, covering everything from its file structure to intents, entities, content, skills and modules.

By Alex Muchiri.

In this article, we will be exploring the open-source project architecture of Botpress, including its file structure, intents, entities, skills and modules. If you haven't already heard of Botpress, it is a highly flexible and useful platform for building chatbots. Many enterprises have used Botpress to create their very own chatbot systems.

Understanding its file structures will allow you to make custom configurations to serve your unique project needs. It will also allow you to create configurations that you can integrate with major messenger platforms like WhatsApp and Viber, besides several other platforms that are not covered in Botpress's default package. However, do keep in mind that you will need to have some basic programming skills and understand how open-source projects work to be able to get anywhere with Botpress. The languages used to create Botpress include NodeJS, Typescript, JavaScript and React.

Requirements

For this tutorial, you will need the following:

A valid GitHub account.
A Botpress server running in your Alibaba Cloud instance.
Visual Studio Code (or PyCharm) to explore the source code.

Exploring the file structure and source code

You can clone the Botpress source code from the official GitHub repository from this link: https://github.com/botpress/botpress

The first thing that we will explore is some of the most important modules in the Botpress source code. In Visual Studio code editor, navigate to src/bp/. There, you should access a number of files and directories, including the index.ts file.

Here is a snippet of the main index.ts file, which serves as an entry point to the Botpress server:

The index.ts file is the first file called when running the platform from the terminal, or when accessing it using some URL. The file is responsible for a number of important functions, which include:

Setting up the configurations
Loading the environment variables
Processing commands from the terminal

Specifically, the index.ts file is what we are interested in in the section below. It handles the Botpress server and handles command line actions:

const argv = require('yargs')
    .command(
      ['serve', '$0'],
      'Start your botpress server',
      {
        production: {
          alias: 'p',
          description: 'Whether you want to run in production mode or not',
          default: false,
          type: 'boolean'
        },
        autoMigrate: {
          description:
            'When this flag is set, Botpress will automatically migrate your content and configuration files when upgrading',
          default: false,
          type: 'boolean'
        }
      },
      argv => {
        process.IS_PRODUCTION = argv.production || yn(process.env.BP_PRODUCTION) || yn(process.env.CLUSTER_ENABLED)
        process.BPFS_STORAGE = process.core_env.BPFS_STORAGE || 'disk'

        let defaultVerbosity = process.IS_PRODUCTION ? 0 : 2
        if (!isNaN(Number(process.env.VERBOSITY_LEVEL))) {
          defaultVerbosity = Number(process.env.VERBOSITY_LEVEL)
        }

        process.AUTO_MIGRATE =
          process.env.AUTO_MIGRATE === undefined ? yn(argv.autoMigrate) : yn(process.env.AUTO_MIGRATE)

        process.VERBOSITY_LEVEL = argv.verbose ? Number(argv.verbose) : defaultVerbosity
        process.IS_LICENSED = true
        process.ASSERT_LICENSED = () => {}
        process.BOTPRESS_VERSION = metadataContent.version

        process.IS_PRO_AVAILABLE = fs.existsSync(path.resolve(process.PROJECT_LOCATION, 'pro')) || !!process.pkg
        const configPath = path.join(process.PROJECT_LOCATION, '/data/global/botpress.config.json')

        if (process.IS_PRO_AVAILABLE) {
          process.CLUSTER_ENABLED = yn(process.env.CLUSTER_ENABLED)

          if (process.env.PRO_ENABLED === undefined) {
            if (fs.existsSync(configPath)) {
              const config = require(configPath)
              process.IS_PRO_ENABLED = config.pro && config.pro.enabled
            }
          } else {
            process.IS_PRO_ENABLED = yn(process.env.PRO_ENABLED)
          }
        }

        getos.default().then(distro => {
          process.distro = distro
          require('./bootstrap')
        })
      }
    )

When initiating the Botpress server, the server command sets up all additional variables to be used when running the server and then passes the execution to the bootstrap file. As with all commands, the bootstrap file is necessary.

The bootstrap file loads the Botpress object does the following:

Receives the modules entry points. Consider the code snippet below:

  const modules: sdk.ModuleEntryPoint[] = []

  const globalConfig = await Config.getBotpressConfig()
  const loadingErrors: Error[] = []
  let modulesLog = ''

  const resolver = new ModuleResolver(logger)

  for (const entry of globalConfig.modules) {
    try {
      if (!entry.enabled) {
        modulesLog += os.EOL + `${chalk.dim('⊝')} ${entry.location} ${chalk.gray('(disabled)')}`
        continue
      }

      const moduleLocation = await resolver.resolve(entry.location)
      const rawEntry = resolver.requireModule(moduleLocation)

      const entryPoint = ModuleLoader.processModuleEntryPoint(rawEntry, entry.location)
      modules.push(entryPoint)
      process.LOADED_MODULES[entryPoint.definition.name] = moduleLocation
      modulesLog += os.EOL + `${chalk.greenBright('⦿')} ${entry.location}`
    } catch (err) {
      modulesLog += os.EOL + `${chalk.redBright('⊗')} ${entry.location} ${chalk.gray('(error)')}`
      loadingErrors.push(new FatalError(err, `Fatal error loading module "${entry.location}"`))
    }
  }
#This will handle the Botpress modules

Initiates the Botpress object through the start command:

  await Botpress.start({ modules }).catch(err => {
    logger.attachError(err).error('Error starting Botpress')
    process.exit(1)
  })
#this initiates the Botpress object

Once the module entry points have been received, they are passed on to the src/core/botpress.ts file where the actual loading of modules happens.

  private async loadModules(modules: sdk.ModuleEntryPoint[]): Promise<void> {
    const loadedModules = await this.moduleLoader.loadModules(modules)
    this.logger.info(`Loaded ${loadedModules.length} ${plur('module', loadedModules.length)}`)
  }

Other noteworthy components in this file include initializing services, cleaning up disabled modules, deploying assets, and starting server and discovering bots. Consider the app lifecycle in the snippet below:

private async initialize(options: StartOptions) {
    this.config = await this.configProvider.getBotpressConfig()

    this.trackStart()
    this.trackHeartbeat()

    setDebugScopes(process.core_env.DEBUG || (process.IS_PRODUCTION ? '' : 'bp:dialog'))

    await AppLifecycle.setDone(AppLifecycleEvents.CONFIGURATION_LOADED)

    await this.restoreDebugScope()
    await this.checkJwtSecret()
    await this.loadModules(options.modules)
    await this.migrationService.initialize()
    await this.cleanDisabledModules()
    await this.initializeServices()
    await this.checkEditionRequirements()
    await this.deployAssets()
    await this.startRealtime()
    await this.startServer()
    await this.discoverBots()

    await AppLifecycle.setDone(AppLifecycleEvents.BOTPRESS_READY)

    this.api = await createForGlobalHooks()
    await this.hookService.executeHook(new Hooks.AfterServerStart(this.api))
  }

Usually, the loaded modules are passed to the start object.

Content

Usually, a content management system (CMS) acts as the central management system to manage all bot content including all bot responses. Consider the image below for my installation's CMS:

The advantage of Botpress is that it allows you to edit your content directly without having to go through the flow editors. What is of importance to us are content type and content elements.

Type

The content type is a definition of the structure of the response sent by the bot, and usually, which varies from bot to bot. There are several different types:

Action button
Card
Carousel
Image
Single choice
Text
Dropdown

Content element

The content element contains the data of the specified content type such as some text, an image, or a card. Further, this could also include variations for the included:

{
  "id": "builtin_some-id",
  "formData": {
    "text": "👋, {{state.$r}}!",
    "variations": ["Hello, {{state.$r}}!", "Thank you for reaching out!, {{state.$r}}!", "Hi there, {{state.$r}}!"],
    "typing": true
  },
  "createdBy": "agent",
  "createdOn": "2019-09-27T20:35:21.019Z"
}

The elements of a specified content type are usually stored in one .json file in the directory storing the elements of you bot.

Modules

Botpress modules allow for the extension of the core functionalities and are reusable in any flow. Some of the core modules included out-of-the-box include:

Analytics
Basic skills
Builtin
Channel-messenger
Channel-slack
Channel-teams
Channel-telegram
Channel-web
Code editor
NLU
QNA
Scheduler
Knowledge
History

However, you could also create your own modules and integrate them into the engine. For example, you can work on a module to enable WhatsApp bot creation.

Intents and entities

Entities and intents are handled by Botpress's NLU module. The concept of intent and entities allows your bot to understand the context of a statement, derive meaning from it and respond with the appropriate action. Look at the example below:

A user wants to buy a gift for their child's birthday. They are interested in buying a car for their child.

In such a context, the intent is to buy a gift while the entity is a car. In other words, the intent is what the user wants to do, while the entity is the item associated with that intent. Intents and entities are the building blocks of intelligent conversation. Even when the situation is a bit more sophisticated, such as having multiple intents, the structure remains similar. Below is a sample bot conversation:

Customer: Hi there, I would like to buy a gift for my son
Bot: Hello too, what is your name?
Customer: I am Jenifer
Bot: Thank you Jenifer for reaching out, what kind of gift would like to buy?
Customer: I would like to buy a bicycle
Bot: That's a good choice, how old is your son?
Customer: 10 years
Bot: Fantastic, we have bikes available for that age group. Would you like anything else?
Customer: Yes, include a pair of sports shoes as well
Bot: Got it, your items will be prepared for collection
Customer: Thank you
Bot: Have a nice day Jenifer

The sample conversation above includes the elements of intents and entities.

Intents

Let's analyze the conversation. From the conversation above, we can detect what the user intended through the process of intent classification. The process is detailed in the table below.

Intents can be categories into many different types. They correspond to statements like buy bicycle, go to a movie, and buy gift, so on. Botpress enables you to add new intends to your bot from the NLU module accessible from the interface using the Create new intent function. Consider the train booking example below:

- book me a train
- I want to book a train
- I want to travel to Mombasa tomorrow
- what trains are available from Nairobi to Mombasa

Usually, it is a best practice to include between four and five utterances for each intent to improve detection accuracy.

Intent processing

Intent processing entails detecting the intent name from hooks, flow transitions and actions using the event.nlu.intent.name variable. Consider the example below:

{
  "type": "text",
  "channel": "web",
  "direction": "incoming",
  "payload": {
    "type": "text",
    "text": "good morning"
  },
  "target": "welcome bot",
  "botId": "welcome-bot",
  "threadId": "1",
  "id": some id,
  "preview": "good morning",
  "flags": {},
  "nlu": { 
    "language": "en", // language identification (from supported languages)
    "intent": { // most probable intents
      "name": "hello",
      "confidence": 1
    },
    "intents": [ identified intents, sorted by confidence
      {
        "name": "hello",
        "confidence": 1,
        "provider": "native"
      },
    ],
    "entities": [], 
    "slots": {} 
  }
}

When an intent is identified, we can then transition to a new flow, call an API or respond appropriately.

Entities

Entities are of system and custom types and are associated with intents, which are usually extracted from text. Botpress extract entities using the event.nlu.entities variable from hooks, flow transitions, and actions. Consider the example below:

The user input the following text: Let's run for some three miles.

{
  /* ... */
  entities: [
    {
      type: 'distance',
      meta: {
        confidence: 1
        provider: 'native',
        source: 'three miles',
        start: 21, 
        end: 35,
      },
      data: {
        value : 5,
        unit: 'mile',
        extras: {}
      }
    },
    {
      type: 'numeral',
      meta: {
        confidence: 1
        provider: 'native',
        source: 'three', 
        start: 21, 
        end: 29, 
      },
      data: {
        value : 3,
        extras: {}
      }
    }
  ]
}

Custom entities could either be of pattern and list types and are defined using the Entity tool in the Botpress studio Understanding module. Consider the image below:

Conclusion

In this tutorial, we have explored the Botpress file structure, as well as modules, intents and even looked at some bits of the source code. Botpress is one of the leading open-source bot frameworks in the market today. It is well supported as evidenced by the numerous releases put forward to support the framework. There are two types of licences available for the users of Botpress, commercial licenses and the AGPL licence. In the next article, we take the lessons from this tutorial on modules, intents and entities to design and setup a simple chatbot that is integrated with Facebook.

Do you have an Alibaba Cloud account? Sign up for an account and try over 40 products for free. This deal is worth up to $1300. Get Started with Alibaba Cloud to learn more.

The views expressed herein are for reference only and don't necessarily represent the official views of Alibaba Cloud.

: marcmercier June 16, 2020 at 5:47 pm

Hi Alex, thanks for doing this crash course on Botpress. I am gonna share this on the Botpress Forum (forum.botpress.com) If you have any question about the platform, please reach out to me at marc.mercier@botpress.com. Cheers

0 0

: Alex July 17, 2020 at 7:05 am

Hi Marc, please feel free to share. I'll drop you an email

0 0

Community

A Crash Course on the Botpress Architecture

Requirements

Exploring the file structure and source code

Content

Type

Content element

Modules

Intents and entities

Intents

Intent processing

Entities

Conclusion

Read previous post:

Read next post:

Alibaba Clouder

You may also like

Comments

marcmercier June 16, 2020 at 5:47 pm

Alex July 17, 2020 at 7:05 am

Alibaba Clouder

Related Products

MaxCompute

Web Hosting Solution

AI Acceleration Solution

YiDA Low-code Development Platform