Skip to content

Commit 253fea1

Browse files
alyssachvastacopybara-github
authored andcommitted
add checks and CLI to README
GitOrigin-RevId: 2d4bdb1b2d48ea70f87a1be5ede27242f712401a
1 parent fc252cc commit 253fea1

File tree

1 file changed

+16
-1
lines changed

1 file changed

+16
-1
lines changed

README.md

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,7 @@ Then to log in locally run:
4848

4949
`gcloud auth application-default login`
5050

51-
## Example Usage
51+
## Example Usage - Javascript
5252

5353
Summarize Seattle’s $15 Minimum Wage Conversation.
5454

@@ -90,6 +90,21 @@ const summary = mySensemaker.summarize(
9090
);
9191
console.log(summary.getText("MARKDOWN"));
9292
```
93+
## CLI Usage
94+
95+
There is also a simple CLI set up for testing. There are two tools:
96+
- ./runner-cli/runner.ts: takes in a CSV representing a conversation and outputs an HTML file containing the summary. The summary is best viewed as an HTML file so that the included citations can be hovered over to see the original comment and votes.
97+
- ./runner-cli/rerunner.ts: takes in a CSV representing a conversation and reruns summarization a number of times and outputs each of the summaries in one CSV. This is useful for testing consistency.
98+
99+
100+
## Running the Checks
101+
In the ./evals directory there are a number of checks that can be run on an unlabeled conversation. There are three categories of checks:
102+
- Monitoring Checks: summary generation failure rate and time to run
103+
- Quick Checks: whether the summary has an intro and conclusion, and whether all the topics and subtopics from categorization are present
104+
- Qualitative Checks: measures how often each group is mentioned
105+
106+
All three checks are run using the ./evals/run_checks.ts script.
107+
93108
94109
## Making Changes to the tools - Development
95110

0 commit comments

Comments
 (0)