How to determine if a chat bot is learning

Abstract

This is a brief write up that describes the hypothetical requirements to measure how much a chat bot has learned.

How do I tell if a chat bot is learning?

This is an interesting question that I was recently asked. In terms of ChatterBot's codebase, the simple way to tell if a bot is learning is to create a new bot, view the data stored in it's database, then have some sort of interaction with the bot, then check the database again to see if new entries were added.

While the developer who asked this question was inquiring about the literal case of checking the bot's physical knowledge, it did get me thinking. How do I tell if a chat bot is learning? In other words, is there a way to measure the knowledge that a bot has acquired?

Variables pertaining to knowledge

There are a few hypothetical variables that effect the measure of a chat bot's knowledge.

1. The quantity of knowledge: The more information a bot has, the more responses it can provide.
2. The diversity of knowledge: A greater spread of knowledge increases the ability of the chat bot to provide an accurate response.
3. The connectedness of knowledge: The knowledge that a chat bot has is worthless if there is no connections between the information that it knows. The information needs to be connected in ways that indication how one statement response to another and how those statements are grouped together.

Conclusion

In should be possible to measure how much a chat bot has learned. This measurement would be based on the aforementioned variables and it would describe the overall quality of a chat bot's knowledge. This score would be useful in comparing the performance of different learning algorithms in chat bot's trained on the same data set. It would also be interesting to use it as a metric to gauge the data that a chat bot acquires over it's "lifetime".