Releases: explosion/spaCy
v1.6.0: Improvements to tokenizer and tests
✨ Major features and improvements
- Updated token exception handling mechanism to allow the usage of arbitrary functions as token exception matchers.
- Improve how tokenizer exceptions for English contractions and punctuations are generated.
- Update language data for Hungarian and Swedish tokenization.
- Update to use Thinc v6 to prepare for spaCy v2.0.
🔴 Bug fixes
- Fix issue #326: Tokenizer is now more consistent and handles abbreviations correctly.
- Fix issue #344: Tokenizer now handles URLs correctly.
- Fix issue #483: Period after two or more uppercase letters is split off in tokenizer exceptions.
- Fix issue #631: Add
richcmp
method toToken
. - Fix issue #718: Contractions with
She
are now handled correctly. - Fix issue #736: Times are now tokenized with correct string values.
- Fix issue #743:
Token
is now hashable. - Fix issue #744:
were
andWere
are now excluded correctly from contractions.
📋 Tests
- Modernise and reorganise all tests and remove model dependencies where possible.
- Improve test speed to ~20s for basic tests (from previously >80s) and ~100s including models (from previously >200s).
- Add fixtures for spaCy components and test utilities, e.g. to create
Doc
object manually. - Add documentation for tests to explain conventions and organisation.
👥 Contributors
Thanks to @oroszgy, @magnusburton, @guyrosin and @danielhers and for the pull requests!
v1.5.0: Alpha support for Swedish and Hungarian
✨ Major features and improvements
- NEW: Alpha support for Swedish tokenization.
- NEW: Alpha support for Hungarian tokenization.
- Update language data for Spanish tokenization.
- Speed up tokenization when no data is preloaded by caching the first 10,000 vocabulary items seen.
🔴 Bug fixes
- List the
language_data
package in thesetup.py
. - Fix missing
vec_path
declaration that was failing ifadd_vectors
was set. - Allow
Vocab
to load withoutserializer_freqs
.
📖 Documentation and examples
- NEW: spaCy Jupyter notebooks repo: ongoing collection of easy-to-run spaCy examples and tutorials.
- Fix issue #657: Generalise dependency parsing annotation specs beyond English.
- Fix various typos and inconsistencies.
👥 Contributors
Thanks to @oroszgy, @magnusburton, @jmizgajski, @aikramer2, @fnorf and @bhargavvader for the pull requests!
v1.4.0: Improved language data and alpha Dutch support
✨ Major features and improvements
- NEW: Alpha support for Dutch tokenization.
- Reorganise and improve format of language data.
- Add shared tag map, entity rules, emoticons and punctuation to language data.
- Convert entity rules, morphological rules and lemmatization rules from JSON to Python.
- Update language data for English, German, Spanish, French, Italian and Portuguese.
🔴 Bug fixes
- Fix issue #649: Update and reorganise stop lists.
- Fix issue #672: Make
token.ent_iob_
return unicode. - Fix issue #674: Add missing lemmas for contracted forms of "be" to
TOKENIZER_EXCEPTIONS
. - Fix issue #683:
Morphology
class now supplies tag map value for the special space tag if it's missing. - Fix issue #684: Ensure
spacy.en.English()
loads the Glove vector data if available. Previously was inconsistent with behaviour ofspacy.load('en')
. - Fix issue #685: Expand
TOKENIZER_EXCEPTIONS
with unicode apostrophe (’
). - Fix issue #689: Correct typo in
STOP_WORDS
. - Fix issue #691: Add tokenizer exceptions for "gonna" and "Gonna".
⚠️ Backwards incompatibilities
No changes to the public, documented API, but the previously undocumented language data and model initialisation processes have been refactored and reorganised. If you were relying on the bin/init_model.py
script, see the new spaCy Developer Resources repo. Code that references internals of the spacy.en
or spacy.de
packages should also be reviewed before updating to this version.
📖 Documentation and examples
- NEW: "Adding languages" workflow.
- NEW: "Part-of-speech tagging" workflow.
- NEW: spaCy Developer Resources repo – scripts, tools and resources for developing spaCy.
- Fix various typos and inconsistencies.
👥 Contributors
Thanks to @dafnevk, @jvdzwaan, @RvanNieuwpoort, @wrvhage, @jaspb, @savvopoulos and @davedwards for the pull requests!
v1.3.0: Improve API consistency
✨ Major features and improvements
- Add
Span.sentiment
attribute. - #658: Add
Span.noun_chunks
iterator (thanks @pokey). - #642: Let
--data-path
be specified when running download.py scripts (thanks @ExplodingCabbage). - #638: Add German stopwords (thanks @souravsingh).
- #614: Fix
PhraseMatcher
to work with newMatcher
(thanks @sadovnychyi).
🔴 Bug fixes
- Fix issue #605:
accept
argument toMatcher
now rejects matches as expected. - Fix issue #617:
Vocab.load()
now works with string paths, as well asPath
objects. - Fix issue #639: Stop words in
Language
class now used as expected. - Fix issues #656, #624:
Tokenizer
special-case rules now support arbitrary token attributes.
📖 Documentation and examples
- Add "Customizing the tokenizer" workflow.
- Add "Training the tagger, parser and entity recognizer" workflow.
- Add "Entity recognition" workflow.
- Fix various typos and inconsistencies.
👥 Contributors
Thanks to @pokey, @ExplodingCabbage, @souravsingh, @sadovnychyi, @manojsakhwar, @TiagoMRodrigues, @savkov, @pspiegelhalter, @chenb67, @kylepjohnson, @YanhaoYang, @tjrileywisc, @dechov, @wjt, @jsmootiv and @blarghmatey for the pull requests!
v1.2.0: Alpha tokenizers for Chinese, French, Spanish, Italian and Portuguese
✨ Major features and improvements
- NEW: Support Chinese tokenization, via Jieba.
- NEW: Alpha support for French, Spanish, Italian and Portuguese tokenization.
🔴 Bug fixes
- Fix issue #376: POS tags for "and/or" are now correct.
- Fix issue #578:
--force
argument on download command now operates correctly. - Fix issue #595: Lemmatization corrected for some base forms.
- Fix issue #588:
Matcher
now rejects empty patterns. - Fix issue #592: Added exception rule for tokenization of "Ph.D."
- Fix issue #599: Empty documents now considered tagged and parsed.
- Fix issue #600: Add missing
token.tag
andtoken.tag_
setters. - Fix issue #596: Added missing unicode import when compiling regexes that led to incorrect tokenization.
- Fix issue #587: Resolved bug that caused
Matcher
to sometimes segfault. - Fix issue #429: Ensure missing entity types are added to the entity recognizer.
v1.1.0: Bug fixes and adjustments
✨ Major features and improvements
- Rename new
pipeline
keyword argument ofspacy.load()
tocreate_pipeline
. - Rename new
vectors
keyword argument ofspacy.load()
toadd_vectors
.
🔴 Bug fixes
- Fix issue #544: Add
vocab.resize_vectors()
method, to support changing to vectors of different dimensionality. - Fix issue #536: Default probability was incorrect for OOV words.
- Fix issue #539: Unspecified encoding when opening some JSON files.
- Fix issue #541: GloVe vectors were being loaded incorrectly.
- Fix issue #522: Similarities and vector norms were calculated incorrectly.
- Fix issue #461:
ent_iob
attribute was incorrect after setting entities viadoc.ents
- Fix issue #459: Deserialiser failed on empty doc
- Fix issue #514: Serialization failed after adding a new entity label.
v1.0.0: Support for deep learning workflows and entity-aware rule matcher
✨ Major features and improvements
- NEW: custom processing pipelines, to support deep learning workflows
- NEW: Rule matcher now supports entity IDs and attributes
- NEW: Official/documented training APIs and
GoldParse
class - Download and use GloVe vectors by default
- Make it easier to load and unload word vectors
- Improved rule matching functionality
- Move basic data into the code, rather than the json files. This makes it simpler to use the tokenizer without the models installed, and makes adding new languages much easier.
- Replace file-system strings with
Path
objects. You can now load resources over your network, or do similar trickery, by passing any object that supports thePath
protocol.
⚠️ Backwards incompatibilities
- The data_dir keyword argument of
Language.__init__
(and its subclassesEnglish.__init__
andGerman.__init__
) has been renamed topath
. - Details of how the Language base-class and its sub-classes are loaded, and how defaults are accessed, have been heavily changed. If you have your own subclasses, you should review the changes.
- The deprecated
token.repvec
name has been removed. - The
.train()
method of Tagger and Parser has been renamed to.update()
- The previously undocumented
GoldParse
class has a new__init__()
method. The old method has been preserved inGoldParse.from_annot_tuples()
. - Previously undocumented details of the
Parser
class have changed. - The previously undocumented
get_package
andget_package_by_name
helper functions have been moved into a new module,spacy.deprecated
, in case you still need them while you update.
🔴 Bug fixes
- Fix
get_lang_class
bug when GloVe vectors are used. - Fix Issue #411:
doc.sents
raised IndexError on empty string. - Fix Issue #455: Correct lemmatization logic
- Fix Issue #371: Make
Lexeme
objects hashable - Fix Issue #469: Make
noun_chunks
detect root NPs
👥 Contributors
Thanks to @daylen, @RahulKulhari, @stared, @adamhadani, @izeye and @crawfordcomeaux for the pull requests!