Merge pull request #1 from VectorInstitute/develop

Implement initial evaluation structure
VectorInstitute · Apr 22, 2024 · 6573704 · 6573704
2 parents b18934d + 69c6a90
commit 6573704
Show file tree

Hide file tree

Showing 145 changed files with 2,031 additions and 4,013 deletions.
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -1,6 +1,6 @@
 # Contributing to aieng-template
 
-Thanks for your interest in contributing to the aieng-template!
+Thanks for your interest in contributing to Vector-Eval!
 
 To submit PRs, please fill out the PR template along with the PR. If the PR
 fixes an issue, don't forget to link the PR to the issue!

diff --git a/README.md b/README.md
@@ -1,4 +1,4 @@
-# AI Engineering template
+# Vector-Eval
 
 ----------------------------------------------------------------------------------------
 
@@ -8,24 +8,23 @@
 [![codecov](https://codecov.io/gh/VectorInstitute/aieng-template/branch/main/graph/badge.svg)](https://codecov.io/gh/VectorInstitute/aieng-template)
 [![license](https://img.shields.io/github/license/VectorInstitute/aieng-template.svg)](https://github.com/VectorInstitute/aieng-template/blob/main/LICENSE)
 
-A template repo for AI Engineering projects (using ``python``)
+A repository for evaluating RAG systems.
 
 ## 🧑🏿‍💻 Developing
 
 ### Installing dependencies
 
-The development environment can be set up using
-[poetry](https://python-poetry.org/docs/#installation). Hence, make sure it is
-installed and then run:
-
+Create a new env and install the required packages:
 ```bash
-python3 -m poetry install
-source $(poetry env info --path)/bin/activate
+python3 -m venv env
+source env/bin/activate
+python3 -m pip install --upgrade pip
+python3 -m pip install -r requirements.txt
 ```
 
-In order to install dependencies for testing (codestyle, unit tests, integration tests),
-run:
+### Running evaluation
 
+Run evaluation for TASK (for e.g. pubmedqa) using SYSTEM (for e.g. basic_rag):
 ```bash
-python3 -m poetry install --with test
+python3 veval/run.py --task <TASK> --sys <SYSTEM>
 ```
diff --git a/aieng_template/bar.py b/aieng_template/bar.py
diff --git a/aieng_template/foo.py b/aieng_template/foo.py
diff --git a/docs/Makefile b/docs/Makefile
diff --git a/docs/make.bat b/docs/make.bat
diff --git a/docs/source/_static/custom.js b/docs/source/_static/custom.js
diff --git a/docs/source/_static/logos/vector_logo.png b/docs/source/_static/logos/vector_logo.png
diff --git a/docs/source/_static/require.min.js b/docs/source/_static/require.min.js
diff --git a/docs/source/_templates/base.html b/docs/source/_templates/base.html
diff --git a/docs/source/_templates/custom-class-template.rst b/docs/source/_templates/custom-class-template.rst
diff --git a/docs/source/_templates/custom-module-template.rst b/docs/source/_templates/custom-module-template.rst