Extend TOFU Dataset

[philswatton]

• We would like to extend the TOFU dataset in the following directions:
  • Amount of data
  • Different levels of granularity
  • Different levels of interconnection
• Implement functions to generate additional TOFU data deterministically given a seed number (or create separate dataset?)
• As in TOFU dataset in pipeline #1, we want to be able to manage these for creating retain/forget sets later on

[jack89roberts]

Summary of data conversation with Matt:

• People, details of individuals, relationships between people, relationships between people and organisations/non-person entities for both 1) Fake authors, and 2) Real authors. To check - are synthetic and real authors similar enough?

• Questions & answers only vs. full documents? Questions & answers only probably an ok simplification to make, but if we're generating our own data we can make profiles etc., plus experiments on real authors would probe this.

[jack89roberts]

Generate a new dataset:

1. Decide what relationships we want to have (e.g. genres, publishers, co-authors, ...)
2. Create & sample relationships (e.g. via a graph/rule-based mechanism/prompt engineering)
3. Decide what data we want to have (e.g. questions/answers, author profiles, book summaries, ...)
4. Generate data (including paraphrased & perturbed)

[jack89roberts]

Defer for now: Real dataset (authors/facts etc.)