Definition of estimated success rate.

[yulundu]

A “successful” dialog was defined as a dialog that met two requirements:
1) the dialog had at least 3 turns; and
2) the Let’s Go system attempted to do a backend lookup for the bus schedule.
(Note that without thorough human annotations, we have no information about whether the system filled the slots with the right information for those backend lookups -- thus the name “estimated" success.)

"The estimated success rate" was used in the email daily report, calculated by "#successful dialogs/#all dialogs" for that day. The original developer did not log in estimated success in the log files. The calculation was done post hoc using the backend lookup information.

You can locate the information about backend lookup in the '*sessions.txt’ file in the directory for each day. For example, for 20060920, you need to locate the file named “LetsGoPublic-20060920-sessions.txt”. It should be in this format:
20060720 000 15 00 18 17 167998 1 NaN
20060720 001 32 00 33 53 335999 0 NaN
20060720 002 6 00 52 17 72999 0 NaN
20060720 003 9 01 34 35 73999 1 NaN
20060720 004 29 03 43 54 282997 0 NaN
20060720 005 3 05 13 54 250999 1 NaN
20060720 006 39 05 30 36 293004 0 NaN
20060720 007 0 14 44 13 4002 Inf NaN
20060720 008 2 17 50 25 20004 1 NaN
20060720 009 8 18 58 43 78001 1 NaN

Each row contains the information for each dialog(session) during the day, and the 8th column recorded the notion of backend lookup — “1” and “Inf” means there was a backend lookup and “0” means otherwise.