Technology & Architecture
Let's explore how we developed Berkeley DBT into a comprehensive platform for end-to-end DBT chatbot performance testing.
Conversation Generation
At its core, DBT skills training is all about the dynamic between a skills trainer, who
imparts essential coping techniques, and a skills training client, who actively engages in
learning and applying these methods to their own emotional and interpersonal challenges.
We replicate this duo with two AI constructs: a DBT Skills Trainer Bot and a Patient Bot.
Our DBT Skills Trainer Bot, DeeBoT, utilizes a custom-tailored system prompt taking GPT-4
from "general-purpose AI" to "DBT-focused guide". This customization enables DeeBoT to
initiate interactions with a friendly greeting, assess the user's emotional state, and
choose the appropriate skill to teach based on the user’s needs.
Playing the part of a DeeBoT user is our Patient Bot. Given one of 81 pre-written starting
messages capturing various challenges a skills training client may face, Patient Bot is directed via system prompt
to continue the conversation from their perspective. Together, these two bots engage in a
simulated multi-turn conversation before the resulting synthetic conversation is passed through our evaluation process.
Evaluation
To ensure that DeeBoT adheres to established quality-of-care standards, we employed the
DBT Adherence Checklist for Individual Therapy (DBT AC-I).
This tool assesses a therapy session's structure, strategy use, stylistic approach, and
skill teaching effectiveness to verify compliance with the DBT model.
Out of the 26 individual standards outlined in the DBT AC-I, we selected 23 that were
relevant in a text-only context. We then converted each of these into a system prompt
that enables GPT-4 to analyze a conversation and assign a binary rating:
"0" for non-adherence and "1" for adherence to each standard.
To assess the relevance and accuracy of the skills DeeBoT recommends during interactions,
we created a database using regular expressions to identify which of the 46 DBT skills
listed in the DBT Skills Training Manual appear in a conversation.
After identifying the skills present, we compare them to the set we aimed to target for each specific user.
Ideally, DeeBoT's recommendations should directly match the targeted skills,
but if there is alignment with a closely related skill, it's still a good sign that DeeBoT was on the right track.