Dev (#299)

* Validators refactor (#214)

* Pass metadata through guard, runner, to validator

* Validator: Change interface

- EventDetail => ValidationResult
- (key, value, schema) => (value, metadata)

* validators.Pydantic: New interface

* validators.PydanticFieldValidator: New interface

* validators: New interface for simple validators

* validators: New interface for SQL-related validators

* validators.Choice: New interface (pass schema through metadata)

* validators.SimilarToDocument: New interface

* validators.IsProfanityFree: New interface

* validators.IsHighQualityTranslation: New interface

* validators.EndsWith: New interface

* validators.ExtractedSummarySentencesMatch: New interface

* validators.ReadingTime: New interface

* validators.ExtractiveSummary: New interface

* validators.RemoveRedundantSentences: New interface

* validators.SaliencyCheck: New interface

* validators.QARelevanceLLMEval: New interface

* format

* Fix tests

* validators.Extracted*: Change behavior, add test

* validation.md: Hide Choice validator from docs

* Avoid reassigning value when no validation occurs

This is a bug in the main branch, which will be dealt with in a separate PR

* ValidationResult.outcome: Fix literals

* test_validators.test_summary_validators: Mock the embedding tests

* bump version to v0.2.0-alpha.1

* Abstract validator running into ValidatorService (#228)

* Crudely move field validation from datatypes/validators to ValidatorService

* datatypes: validate => collect_validation

* Remove unused params from collect_validation method signature, adjust logs

* Remove Choice validator (handled by skeleton reask)

* Fix embedding test import

* Implement AsyncValidatorService, that runs multiprocess

when guard is invoked asynchronously

* test_async: Test multiprocessing

* validator_service: Parallelize synchronous guard

* format

* test_guard: Test synchronous invocation multiprocessing

* Run validators on the same field in parallel when possible

* fixup tests

* format

* Add test for parallel validation on same field

* Support calling chat models with chat history instead of single user message and instructions (#234)

* Pass basemodel to sync and async runner calls (#221)

* Pass basemodel to sync and async runner calls

* remove model filters from using functions

* json_utils: Handle value type error on skeleton check (#225)

* [bug] support setting reasking instructions (#199)

* add reask instructions

* basic reasking instructions

* schema: Remove templating functionality from instructions

* Fix reask prompt compilation test

* [minor] Fix rail tests

* schema.get_reask_schema_and_prompt => get_reask_setup

* Remove examples from json reask prompt (they are in instructions)

* openai_completion_create: Add note on ignored instructions kwarg

* test_python_rail: Adjust used constants to remove examples from prompt

* constants: high_level_instructions => high_level_json_instructions

* StringSchema: Set reask instructions to something basic

* format

---------

Co-authored-by: Rafael Irgolic <[email protected]>

* make prompts optional in rail spec (#110)

* make prompts and instructions optional

* lint

* enable sending chat history with guard call

* fixing tests pt 1

* fixing tests pt 2

* fix tests

* uncomment tests

* remove temp files

---------

Co-authored-by: rafael <[email protected]>

* increment version

* Add support and docs for cohere models (#237)

* Add support and docs for cohere models

* fix tests

* fix list type

* fixing list imports

* fix tests for missing dependencies

* fix tests

* fix tests

* fix bug where instructions are generated during reasking for completion models (#239)

* Bug fixes in validator service (#243)

* fix fn calling

* functional tests with fixes in validator service

* lint

* add integration tests for string and pydantic with message history

* autoformat

---------

Co-authored-by: Caleb Courier <[email protected]>

* Create schema-wise reask instructions (#245)

* changes for setting and viewing custom reask templates

* lint

* version bump (#246)

* init commit for string formatting changes

* add formatting for instructions and prompt also

* remove comments

* Shreya/metadata bug fix (#259)

* bug fix

* fix tests

* add provenance validators (#258)

* Add provenance validators

* fix prov v0 validator

* cleanup

---------

Co-authored-by: zsimjee <[email protected]>

* version bump (#262)

* linting fixes

* bug fixes and autoformat

* update import

* update examples

* allow passing api key as kwarg

* lint fix

* add api_base arg to embedding

* lint fixes

* change var name

* bump version

* version bump

* updaate tests

* lint fix

* fix lint

* lint fix

* fix merge conflicts

* updates for tests

* update for passing unit tests and updates to examples

* fix lint

* add namespace changes for constants

* lint fix

* update last test

* move out namespace template

* lint

* casting GUARDRAILS_PROCESS_COUNT to int

* Remove `register_pydantic` and `script` tag (#280)

* Remove `register_pydantic` decorator

With failing test for nested pydantic validators

* Remove script tag

* Rewrite register_pydantic test into new Guard.from_pydantic

* Remove script docs

* Remove old pydantic docs

* Remove unused imports (and format)

* Docs: Validators refactor (#269)

* start validators concept doc

* finish validators content, sample google style docstring

* mkdocs options

* finish validator doc strings for key props and args, need to double check metadata

* finish docs on metadata, separate validators and responses

* autoformat

* ignore line length on docstring

* extra spaces

* and -> an

* update string formatting

* update rail and outputs

* fix prompt variables, escape prompt for langchain

* update syntaxin docs, update constants in docstrings

* lint fix

* update syntax in notebook

* fix syntax

* String schema (#279)

* add requirements.txt for setup

* python native string schema

* autoformat

* lint fixes

* descrition -> description

* lint fix

* update Rail classmethod name

* JsonSchema.async_validate: Correctly instantiate SkeletonReask (#284)

* Test Reasks w/ Message History (#247)

* add reasks to message history tests

* autoformat

* == None -> is None

* Choice discriminator change (#241)

* Generate choice discriminator inside union instead of outside

* Remove 'when' syntax from choice

* test_schema_to_prompt: Dont test for choice transpilation

* format

* json_utils: Change verification_failed sentinel value from staticmethod to class

* select_choice_based_on_action: Update for discriminator change

* update outputs

* update test

* Full schema reask (#248)

* Generate choice discriminator inside union instead of outside

* Remove 'when' syntax from choice

* test_schema_to_prompt: Dont test for choice transpilation

* format

* json_utils: Change verification_failed sentinel value from staticmethod to class

* format

* convert_pydantic_model_to_openai_fn: Drop extra fields

* Implement full_schema_reask option in guard

If using pydantic defaults to True, otherwise False

* test_pydantic: Use full_schema_reask=False

* guard: Fill out docstrings for __call__ and parse

* run: Fail early on missing required metadata keys (#272)

* prepend instructions to prompt (#281)

* prepend instructions to prompt

* lint, ignore venv

---------

Co-authored-by: Shreya Rajpal <[email protected]>

* Validators as functions (#249)

* validators: Allow functions as validators

* test functions as validators

* test incorrect function validator

* pydantic_utils: Validators are either classes or (validator, on_fail) tuples

* pydantic_utils: Pydantic validators are reask validators

* tests...pydantic_models: Change validator declarations to tuples

* test_validators: Test tuple validator specifications

* test_pydantic_utils: Fix test to access tuple

* Fix kwargs issue while initialising Provenance validator (#288)

* Fix kwargs issue while initialising Provenance validator

* Add formatting

* Caleb/test cov (#293)

* add reasks to message history tests

* autoformat

* == None -> is None

* start validator service unit tests

* validate tests

* start async validator unit tests

* more tests

* start more tests

* more assertions

* another test

* fix tests

* lint fixes

* 0.2.0 Migration Guide (#291)

* start migration guide

* Outline

* string format

* 0-2-migration: Add pydantic instructions

* 0-2-migration: Add choice change

* langchain and string format

* add regex

* format

* tab formatting

* indentation

* remove redundant section

* validator updates

---------

Co-authored-by: Rafael Irgolic <[email protected]>

* Document field validation logs access (#286)

* docs.logs: Add pointer to how to print logs

* logs_utils: Refactor from dataclasses to pydantic

* docs.logs: Simplify field validation logs access

* Xml/Pydantic docs parity (#292)

* chat models

* still not working

* select choice

* extracting entities

* bug free python plus bug fixes

* no secrets example

* valid sql and bug fix

* vegan mac and cheese

* trailing comma

* valid chess move

* translation

* structured data

* translation quality

* remove script fix bug

* lint fix

* test coverage

* merge cleanup branch

* notebook updates

* tweak prompt, remove interest_rates property

* restructure files (#271)

* start validators concept doc

* finish validators content, sample google style docstring

* mkdocs options

* finish validator doc strings for key props and args, need to double check metadata

* restructure files

* finish docs on metadata, separate validators and responses

* autoformat

* ignore line length on docstring

* extra spaces

* and -> an

* restructure leftnav WIP

* install from gh, API ref

* Add diag to guard

* more guard skeleton

* pydantic v1

* string schema

* backoff/tenacity docs

* last few tweaks

* lint fix

* fix mkdocs.yml merge

* copy intro as index page for now

* document __call__ vs parse

* or -> at

* formatting

* format tabs

* re-add migration guide, remove pydantic

* fix string notation

* Create examples_check.yml

---------

Co-authored-by: Caleb Courier <[email protected]>

* Num reask default (#295)

* fix num_reasks defaulting

* remove TODO

* update rail string

* Pass reask instructions for parsing (#296)

* added custom reask instructions

* fix tests

* Docs restructure (#298)

* start validators concept doc

* finish validators content, sample google style docstring

* mkdocs options

* finish validator doc strings for key props and args, need to double check metadata

* restructure files

* finish docs on metadata, separate validators and responses

* autoformat

* ignore line length on docstring

* extra spaces

* and -> an

* restructure leftnav WIP

* install from gh, API ref

* Add diag to guard

* more guard skeleton

* pydantic v1

* string schema

* backoff/tenacity docs

* last few tweaks

* lint fix

* fix mkdocs.yml merge

* copy intro as index page for now

* document __call__ vs parse

* or -> at

* formatting

* format tabs

* re-add migration guide, remove pydantic

* fix string notation

* Create examples_check.yml

* tuple pydantic examples

* add coverage for openai embeddings model selector

* relocate embeddigns tests to the correct area

* lint

---------

Co-authored-by: Caleb Courier <[email protected]>

* 0.2.0 version bump

* lint version file

* coverage

* reask utils tests

* lint fixes

* json utils tests

* lint fixes

---------

Co-authored-by: rafael <[email protected]>
Co-authored-by: Shreya Rajpal <[email protected]>
Co-authored-by: Caleb Courier <[email protected]>
Co-authored-by: Nefertiti  Rogers <[email protected]>
Co-authored-by: Nefertiti Rogers <[email protected]>
Co-authored-by: Caleb Courier <[email protected]>
Co-authored-by: Dmitry Matasov <[email protected]>
Co-authored-by: Karan Acharya <[email protected]>

Author: 8 people

.github/workflows/ci.yml

@@ -19,7 +19,7 @@ jobs:
         runs-on: ubuntu-latest
         strategy:
             matrix:
-                python-version: ['3.8', '3.9']
+                python-version: ['3.8', '3.9', '3.10', '3.11']
 
         steps:
             -   uses: actions/checkout@v2
@@ -46,7 +46,8 @@ jobs:
         runs-on: ubuntu-latest
         strategy:
             matrix:
-                python-version: ['3.8', '3.9']
+                python-version: ['3.8', '3.9', '3.10', '3.11']
+                dependencies: ['dev', 'full']
 
         steps:
             -   uses: actions/checkout@v2
@@ -63,15 +64,16 @@ jobs:
             -   name: Install Dependencies
                 run: |
                     python -m pip install --upgrade pip
-                    make dev
+                    make ${{ matrix.dependencies }}
 
             -   name: Run Pytests
                 run: |
                     make test-cov
 
             -   name: Upload to codecov.io
-                uses: codecov/codecov-action@v1
+                uses: codecov/codecov-action@v3
                 with:
+                    token: ${{ secrets.CODECOV_TOKEN }}
                     file: ./coverage.xml
                     flags: unittests
                     name: codecov-umbrella

.github/workflows/examples_check.yml

@@ -0,0 +1,34 @@
+name: Notebook Execution and Error Check
+
+on:
+  schedule:
+    - cron: "0 0 * * *"
+  workflow_dispatch:  # This enables manual triggering
+
+jobs:
+  execute_notebooks:
+    runs-on: ubuntu-latest
+    
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+
+    - name: Set up Python
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.x
+
+    - name: Install dependencies
+      run: |
+        pip install jupyter nbconvert
+
+    - name: Execute notebooks and check for errors
+      run: |
+        cd docs/examples
+        for notebook in $(ls *.ipynb); do
+          jupyter nbconvert --to notebook --execute "$notebook"
+          if [ $? -ne 0 ]; then
+            echo "Error found in $notebook"
+            exit 1
+          fi
+        done

.gitignore

@@ -2,11 +2,13 @@ openai_api_key.txt
 *__pycache__*
 data/*
 .vscode/*
+.venv/*
 .DS_Store
 *.ipynb_checkpoints/*
 *.pyc
 *.env
 *.log
+*.venv
 settings.json
 site/*
 guardrails.log
@@ -16,4 +18,8 @@ dist/*
 *.rail_output*
 .idea/*
 .cache
-scratch/
+scratch/
+.coverage*
+test.db
+test.index
+htmlcov

CONTRIBUTING.md

@@ -35,3 +35,16 @@ For convenience, consider [installing the pre-commit hooks](https://pre-commit.c
 5. Celebrate when your pull request is merged! Your changes will be available in the next Guardrails release.
 
 Thank you for your contribution and happy coding!
+
+## Documentation
+
+Docs are served via mkdocs using the mkdocstring plugin. To serve docs locally, run the following
+
+```bash
+# install dependencies
+pip install -e ".[dev]";
+
+# serve docs
+mkdocs serve;
+```
+then navigate to `localhost:8000` in your browser.

Makefile

@@ -19,7 +19,10 @@ test-basic:
 	python -c "import guardrails.version as mversion"
 
 test-cov:
-	pytest tests/ --cov=./ --cov-report=xml
+	pytest tests/ --cov=./guardrails/ --cov-report=xml
+
+view-test-cov:
+	pytest tests/ --cov=./guardrails/ --cov-report html && open htmlcov/index.html
 
 docs-serve:
 	mkdocs serve -a $(MKDOCS_SERVE_ADDR)
@@ -30,4 +33,7 @@ docs-deploy:
 dev:
 	pip install -e ".[dev]"
 
+full:
+	pip install -e ".[all]"
+
 all: autoformat lint docs test

README.md

@@ -105,11 +105,11 @@ Using `RAIL`, we:
 <prompt>
 Explain what a bank run is in a tweet.
 
-@xml_prefix_prompt
+${gr.xml_prefix_prompt}
 
-{output_schema}
+${output_schema}
 
-@json_suffix_prompt_v2_wo_none
+${gr.json_suffix_prompt_v2_wo_none}
 </prompt>
 </rail>
 ```