Initial commit

Author: Chris Newell

.gitignore

@@ -127,3 +127,10 @@ dmypy.json
 
 # Pyre type checker
 .pyre/
+
+# Eclipse
+.project
+.pydevproject
+
+# Misc
+.DS_Store

CONTRIBUTING.md

@@ -0,0 +1,17 @@
+# Contributing to Citron
+
+## Questions
+
+Questions can be raised on the [discussion board](https://github.com/bbc/citron/discussions/categories/q-a).
+
+## Reporting bugs
+
+Please report bugs on the: [issue tracker](https://github.com/bbc/citron/issues) including as much detail as possible so your bug is reproducible.
+
+## Feature suggestions
+
+Please suggest new features on the [discussion board](https://github.com/bbc/citron/discussions/categories/ideas) describing the intended usage.
+
+## Contributing code
+
+Except for small changes and docs, it is best to [suggest a feature](https://github.com/bbc/citron/discussions/categories/ideas) before submitting a pull request.

README.md

@@ -1,2 +1,78 @@
-# citron
-Citron is a quote extraction and attribution system developed by BBC R&D
+# Citron #
+
+Citron is an experimental quote extraction and attribution system created by [BBC R&D](https://www.bbc.co.uk/rd), based on a [paper](https://aclanthology.org/D13-1101/) and a [dataset](https://aclanthology.org/L16-1619/) developed by the School of Informatics at the University of Edinburgh.
+
+It can be used to extract quotes from text documents, attributing them to the appropriate speaker and resolving pronouns where necessary. Note that there can be a significant number of errors and omissions. Extracted quotes should be checked against the input text.
+
+You can run Citron using the [pre-trained model](./models/en_2021-11-15) or [train your own model](./scripts/train). You can also [evaluate its performance](./scripts/evaluate).
+
+Training and evaluating models requires data using [Citron's Annotation Format](./docs/data_format.md). Citron provides [pre-processing scripts](./scripts/preprocess) to extract suitable data from the [PARC 3.0 Corpus of Attribution Relations](https://aclanthology.org/L16-1619/). Alternatively, you can create your own data using the [Citron Annotator](./scripts/annotator) app.
+
+Technical details and potential applications are discussed in: ["Quote Extraction and Analysis for News"](./docs/DSJM_2018_paper_1.pdf).
+
+## Installation ##
+Requires Python 3.7.2 or above. The package versions shown should be installed when using the [pre-trained model](./models/en_2021-11-15).
+
+- [Install scikit-learn (1.0.*)](https://scikit-learn.org/stable/install.html)
+- [Install spaCy (3.*) and download a model](https://spacy.io/usage)    (e.g. "en_core_web_sm")
+- Download the source code: ```git clone [email protected]:bbc/citron.git```
+
+Then from the citron root directory:
+
+    python3 -m pip install -r requirements.txt
+
+Then from python3:
+
+    import nltk
+    nltk.download("names")
+
+## Usage  ##
+
+Scripts to run Citron are available in the [bin/](./bin/) directory.
+
+All scripts require the citron root directory in the PYTHONPATH.
+
+    $ export PYTHONPATH=$PYTHONPATH:/path/to/citron_root_directory
+
+### Run the Citron REST API and demonstration server ###
+    
+    $ citron-server
+        --model-path   Path to Citron model directory
+        --logfile      Path to logfile                   (Optional)
+        --port         Port for the Citron API           (Optional: default is 8080)
+        -v             Verbose mode                      (Optional)
+
+### Run Citron on the command-line ###
+
+    $ citron-extract
+        --model-path     Path to Citron model directory
+        --input-file     Path to input file                (Optional: Otherwise read from stdin)
+        --output-file    Path to output file               (Optional: Otherwise write to stdout)
+        -v               Verbose mode                      (Optional)
+
+### Use Citron in Python ###
+
+    from citron.citron import Citron
+    from citron import utils
+    
+    nlp = utils.get_parser()
+    citron = Citron(model_path, nlp)
+    doc = nlp(text)
+    quotes = citron.get_quotes(doc)
+
+## Issues and Questions ##
+Issues can be reported on the [issue tracker](https://github.com/bbc/citron/issues) and questions can be raised on the [discussion board](https://github.com/bbc/citron/discussions/categories/q-a).
+
+## Contributing ##
+
+Contributions would be welcome. Please refer to the [contributing guidelines](./CONTRIBUTING.md).
+
+## License ##
+
+Licensed under the [Apache License, Version 2.0](./LICENSE). The [pre-trained model](./models/en_2021-11-15) includes data from VerbNet 3.3 which is licensed under the [VerbNet 3.0 license](./verbnet-license.3.0.txt).
+
+## Contact ##
+
+For more information please contact: [[email protected]](mailto:[email protected])
+
+Copyright 2021 British Broadcasting Corporation.

bin/citron-extract

@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+# Copyright 2021 BBC
+# Authors: Chris Newell <[email protected]>
+#
+# License: Apache-2.0
+
+"""
+This script runs Citron on the command line.
+
+"""
+
+import argparse
+import logging
+import json
+import sys
+
+from citron.citron import Citron
+from citron.logger import logger
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Extract quotes from text",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+    )
+    parser.add_argument("-v",
+        action = "store_true",
+        default = False,
+        help = "Verbose mode"
+    )
+    parser.add_argument("--model-path", 
+        metavar = "model_path",
+        type = str,
+        required=True,
+        help = "Path to Citron model directory"
+    )
+    parser.add_argument("--input-file",
+        metavar = "input_file",
+        type = str,
+        help = "Optional: Otherwise read from stdin"
+    )
+    parser.add_argument("--output-file",
+        metavar = "output_file",
+        type = str,
+        help = "Optional: Otherwise write to stdout"
+    )
+    args = parser.parse_args()
+    
+    if args.v:
+        logger.setLevel(logging.DEBUG)
+    
+    citron = Citron(args.model_path)
+    
+    if args.input_file is None:
+        text = ""
+        
+        while True:
+            line = sys.stdin.readline()
+            
+            if not line:
+                break
+            
+            text += " " + line
+    
+    else:
+        with open(args.input_file, encoding="utf-8") as infile:
+            text = infile.read()
+    
+    results = citron.extract(text)
+    output = json.dumps(results, indent=4, sort_keys=False, ensure_ascii=False)
+    
+    if args.output_file is None:
+        print(output)
+    
+    else:
+        with open(args.output_file, "w", encoding="utf-8") as outfile:
+            outfile.write(output + "\n")   
+
+
+if __name__ == "__main__":
+    main()

bin/citron-server

@@ -0,0 +1,60 @@
+#!/usr/bin/env python3
+# Copyright 2021 BBC
+# Authors: Chris Newell <[email protected]>
+#
+# License: Apache-2.0
+
+"""
+This script starts a web server which supports the Citron
+REST API and demonstration.
+
+"""
+
+import argparse
+import logging
+
+from citron.citron import Citron
+from citron.citron import CitronWeb
+from citron.logger import logger
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="Run the Citron REST API", 
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter
+    )
+    parser.add_argument("-v",
+        action = "store_true",
+        default = False,
+        help = "Verbose mode"
+    )
+    parser.add_argument("--model-path",
+        metavar = "model_path",
+        type = str,
+        required=True,
+        help = "Path to Citron model directory"
+    )
+    parser.add_argument("--logfile",
+        metavar = "logfile",
+        type = str,
+        default = None,
+        help = "Logfile for output"
+    )
+    parser.add_argument("--port",
+        metavar = "port",
+        type = int,
+        default = 8080,
+        help = "Port for the Citron API"
+    )
+    args = parser.parse_args()
+
+    if args.v:
+        logger.setLevel(logging.DEBUG)
+    
+    citron = Citron(args.model_path)
+    web = CitronWeb(citron)
+    web.start(args.port, args.logfile)
+
+
+if __name__ == "__main__":
+    main()