From 3664675116c395c52352f00eb52e7a1b7e452eb5 Mon Sep 17 00:00:00 2001
From: lippings <46999249+lippings@users.noreply.github.com>
Date: Fri, 24 Jan 2020 15:35:04 +0200
Subject: [PATCH] Added description of batches in readme

---
 README.md | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/README.md b/README.md
index 8a9fac3..dc49334 100644
--- a/README.md
+++ b/README.md
@@ -28,6 +28,12 @@ is scored three times.
 
 The scores can then be used to sort the captions to choose the best captions as the final annotations.
 
+In order to ensure that a worker does not end up seeing their own submissions from earlier tasks (e.g. scoring their
+own description from the audio description task), the audio files and HITs should be divided into 10 batches.
+Each worker will then be allowed to participate in one task per batch. This can be accomplished using custom AMT
+qualifications. Once a batch of a given task is finished, the bath qualification should then be granted to the workers
+who participated in the task in that batch.
+
 ----
 
 ## Requirements
@@ -110,6 +116,12 @@ AMT Sandbox is used by default.
 8) To get submission data from AMT based on the `CURRENT` field data, run `get_results.py`. 
 The data will be written in the file listed in `CURRENT.output_data_file`.
 
+Once you have received the results for a task in a given batch, the workers who participated in that task
+should be granted the appropriate qualification. E.g. once all the results have been gathered for the
+batch 5 audio description task, the workers who participated in the batch 5 audio description task should
+be granted the batch 5 qualification. The batch 5 qualification ID is the fifth ID in the list in the field
+`BATCH` in the file `custom_qualifications.yaml`.
+
 HIT data such as rewards can be changed in the config file. 
 If you want to use task-specific requirements, you can copy the `REQUIREMENTS` field into the 
 task field directly, as is done in the `CAPTION_SCORE` task.