BotSIM: An End-to-End Bot Simulation Toolkit for Commercial Task-Oriented Dialog Systems

Table of Contents

1. Introduction
2. Installation
3. Getting Started
4. Tutorials
5. Documentation
6. System Demo Paper and Technical Report

Introduction

BotSIM is an open-source Bot SIMulation toolkit for large-scale data-efficient end-to-end evaluation, diagnosis and remediation of commercial task-oriented dialog systems (chatbots) across multiple bot platforms. This toolkit aims to provide bot developers and practitioners with a one-stop solution to accelerate bot development and evaluation, reduce cost and time-to-market. As a modular framework, BotSIM can be extended by bot developers to support new bot platforms. As a toolkit, BotSIM offers an easy-to-use App and a suite of command line tools for bot practitioners to readily apply BotSIM's "generation-simulation-remediation" pipeline.

Key features of BotSIM include:

• Multi-stage bot evaluation: BotSIM can be used for both pre-deployment testing and potentially post-deployment performance monitoring.
• Data-efficient dialogue generation: Equipped with a deep network based paraphrasing model, BotSIM can generate an extensive set of test intent queries from the limited number of input intent utterances, which can be used to evaluate the bot intent model at scale.
• End-to-end bot evaluation via dialogue simulation: Through automatic chatbot simulation, BotSIM can identify existing issues of the bot and evaluate both the natural language understanding (NLU) performance and the end-to-end dialogue performance such as goal completion rates.
• Bot health report dashboard: The bot health report dashboard presents a multi-granularity top-down view of bot performance. Together with the analytical tools, they help bot practitioners quickly diagnose and troubleshoot their bots.
• Easy extension to new bot platform: BotSIM can be easily extended to support new bot platforms. BotSIM currently supports Salesforce Einstein BotBuilder and Google DialogFlow CX

Prerequisite

To use BotSIM, users need to have API access to the following supported bot platforms:

• Salesforce EinsteinBotBuilder LiveAgent API: org ID, button ID, deployment Id, chat endpoint
• Google DialogFlow CX API: location ID, project ID, agent ID and JSON API token

Installation

1. (Optional) Creating conda environment

   conda create -n botsim python=3.9
   conda activate botsim

2. Cloning and building dependencies

   git clone https://github.com/salesforce/botsim.git
   cd botsim
   pip install -r requirements.txt

Getting Started

Streamlit Web App

The most straightforward way of getting started with BotSIM is the Streamlit Web App. The multi-page App is developed to guide users to leverage BotSIM's "generation-simulation-remediation" pipeline for evaluation, diagnosis and remediation of their bots.

The following commands can be used to start the Streamlit Web App locally:

  export PYTHONPATH=./:$PYTHONPATH
  export DATABASE_URL="db/botsim_sqlite_demo.db"
  streamlit run botsim/streamlit_app/app.py

The App can also be deployed to GCP for access to GPU resources. Detailed deployment instructions are given in the GCP Deployment section of the documentation.

Command Line Tools

Alternatively, users can also deep-dive to learn more about BotSIM's system components through the command line tools. Details are given in the tutorial section of the code documentation.

Tutorial

We provide the following tutorials in the code documentation.

• Streamlit Web App
• Command Line Tools
• Bot Health Dashboard Navigation
• Applying Remedidation Suggestions

Documentation

For more details of the system components and advanced usages, please refer to the code documentation. We welcome the contribution from the open-source community to improve BotSIM. To support new bot platforms, please also follow the guidelines detailed in the code documentation.

System Demo and Technical Report

A dashboard demo of the Streamlit App can be accessed here. You can find more details regarding BotSIM designs in our technical report:

@article{guangsen2022-botsim-tr,
  author    = {Guangsen Wang and Junnan Li and Shafiq Joty and Steven Hoi},
  title     = {BotSIM: An End-to-End Bot Simulation Toolkit for Commercial Task-Oriented Dialog Systems},
  year      = {2022},
  url       = {https://arxiv.org/abs/2211.15916},
  archivePrefix = {arXiv},
}

The detailed system descriptions are presented in the following EMNLP 2022 system demo paper:

@article{guangsen2022-botsim-demo,
  author    = {Guangsen Wang and Samson Tan and Shafqi Joty and Gang Wu and Jimmy Au and Steven Hoi},
  title     = {BotSIM: An End-to-End Bot Simulation Framework for Commercial Task-Oriented Dialog Systems},
  year      = {2022},
  url       = {https://arxiv.org/abs/2211.11982},
  archivePrefix = {arXiv},
}

If you're using BotSIM in your research or applications, please cite the papers.

Contact Us

Feel free to contact [email protected] for any comments, issues or suggestions.

License

BSD 3-Clause License