diff --git a/.gitignore b/.gitignore index ddab75f..bab790c 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ openai.env *.json *.csv -*/.ipynb_checkpoints/ \ No newline at end of file +*/.ipynb_checkpoints/ +env* \ No newline at end of file diff --git a/notebooks/1_Intro to Bert.ipynb b/notebooks/1_Intro to Bert.ipynb new file mode 100644 index 0000000..94042d1 --- /dev/null +++ b/notebooks/1_Intro to Bert.ipynb @@ -0,0 +1,61 @@ +{ + "cells": [ + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Transfer learning (<-Part 1):\n", + "- What is fine-tuning? \n", + "- Why fine-tune vs starting from scratch? \n", + "- How to choose a base model (high level - Why bert?)\n", + "\n", + "#### Why Bert? (High Level)\n", + "\n", + "#### DistilBert - Why are we fine-tuning this model instead?\n", + "\n", + "DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than bert-base-uncased, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark.\n", + "\n", + "It will be faster for us to train our model on DistilBERT. Best practice to create a model quickly to establish a baseline performance and iteratively add complexity if needed. \n", + "\n", + "#### Intro to Hugging Face\n", + "\n", + "#### Fine tuning steps\n", + "- create a dataset (convert from pandas to a hugging face dataset)\n", + "- tokenize your training data with the same tokenizer used by the base model you are fine-tuning\n", + "- \n", + "\n", + "#### Alternative fine-tuning methods (high level and resources for further learning)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.6" + }, + "orig_nbformat": 4 + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/2_yelp_sentiment_classifier.ipynb b/notebooks/2_yelp_sentiment_classifier.ipynb index 23eab6a..9190440 100644 --- a/notebooks/2_yelp_sentiment_classifier.ipynb +++ b/notebooks/2_yelp_sentiment_classifier.ipynb @@ -7,23 +7,7 @@ "source": [ "# Part 2: Fine Tune a Bert Based Model for sentiment classification\n", "\n", - "In this section, we will to classify the sentiment of yelp reviews by fine-tuning distilbert\n", - "\n", - "\n", - "### Transfer learning (<-Part 1):\n", - "- What is fine-tuning? \n", - "- Why fine-tune vs starting from scratch? \n", - "- How to choose a base model (high level)\n", - "\n", - "### Why Bert?\n", - "\n", - "### Why DistilBert?\n", - "\n", - "DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than bert-base-uncased, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark.\n", - "\n", - "It will be faster for us to train our model on DistilBERT. Best practice to create a model quickly to establish a baseline performance and iteratively add complexity if needed. \n", - "\n", - "### How to fine-tune? Fine-tuning methods" + "In this section, we will fine-tune distilBert to classify the sentiment of yelp reviews" ] }, { @@ -38,7 +22,7 @@ }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 9, "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -48,16 +32,16 @@ }, "outputs": [], "source": [ - "%%capture\n", - "!pip install transformers\n", - "!pip install datasets\n", - "!pip install evaluate\n", - "!pip install torch" + "# %%capture\n", + "# !pip install transformers\n", + "# !pip install datasets\n", + "# !pip install evaluate\n", + "# !pip install torch" ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/" @@ -65,34 +49,47 @@ "id": "kSy6D93bUdcX", "outputId": "47562f08-c6c0-4b68-e905-0b0a18cf1b2f" }, + "outputs": [], + "source": [ + "# from google.colab import drive\n", + "# drive.mount('/content/drive')" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "id": "XpSP_6FAhEW9" + }, "outputs": [ { - "name": "stdout", + "name": "stderr", "output_type": "stream", "text": [ - "Mounted at /content/drive\n" + "/Users/d/repos/nlp/env/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", + " from .autonotebook import tqdm as notebook_tqdm\n" ] } ], "source": [ - "from google.colab import drive\n", - "drive.mount('/content/drive')" + "import numpy as np\n", + "import evaluate\n", + "import torch\n", + "from datasets import load_dataset\n", + "from transformers import AutoTokenizer, DistilBertForSequenceClassification, Trainer, TrainingArguments" ] }, { - "cell_type": "code", - "execution_count": null, - "metadata": { - "id": "XpSP_6FAhEW9" - }, - "outputs": [], + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, "source": [ - "import torch" + "# Step 1: Load, inspect and down-sample our dataset" ] }, { "cell_type": "code", - "execution_count": 33, + "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -226,230 +223,1669 @@ }, "outputs": [ { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "a191629c94b6468cba48382e3e274539", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Downloading builder script: 0%| | 0.00/3.10k [00:00 List[int]\n", + " | Create a mask from the two sequences passed to be used in a sequence-pair classification task. A BERT sequence\n", + " | pair mask has the following format:\n", + " | \n", + " | ```\n", + " | 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1\n", + " | | first sequence | second sequence |\n", + " | ```\n", + " | \n", + " | If `token_ids_1` is `None`, this method only returns the first portion of the mask (0s).\n", + " | \n", + " | Args:\n", + " | token_ids_0 (`List[int]`):\n", + " | List of IDs.\n", + " | token_ids_1 (`List[int]`, *optional*):\n", + " | Optional second list of IDs for sequence pairs.\n", + " | \n", + " | Returns:\n", + " | `List[int]`: List of [token type IDs](../glossary#token-type-ids) according to the given sequence(s).\n", + " | \n", + " | save_vocabulary(self, save_directory: str, filename_prefix: Optional[str] = None) -> Tuple[str]\n", + " | Save only the vocabulary of the tokenizer (vocabulary + added tokens).\n", + " | \n", + " | This method won't save the configuration and special token mappings of the tokenizer. Use\n", + " | [`~PreTrainedTokenizerFast._save_pretrained`] to save the whole state of the tokenizer.\n", + " | \n", + " | Args:\n", + " | save_directory (`str`):\n", + " | The directory in which to save the vocabulary.\n", + " | filename_prefix (`str`, *optional*):\n", + " | An optional prefix to add to the named of the saved files.\n", + " | \n", + " | Returns:\n", + " | `Tuple(str)`: Paths to the files saved.\n", + " | \n", + " | ----------------------------------------------------------------------\n", + " | Data and other attributes defined here:\n", + " | \n", + " | max_model_input_sizes = {'distilbert-base-cased': 512, 'distilbert-bas...\n", + " | \n", + " | model_input_names = ['input_ids', 'attention_mask']\n", + " | \n", + " | pretrained_init_configuration = {'distilbert-base-cased': {'do_lower_c...\n", + " | \n", + " | pretrained_vocab_files_map = {'tokenizer_file': {'distilbert-base-case...\n", + " | \n", + " | slow_tokenizer_class = int\n", + " | Size of the full vocabulary with the added tokens.\n", + " | \n", + " | convert_ids_to_tokens(self, ids: Union[int, List[int]], skip_special_tokens: bool = False) -> Union[str, List[str]]\n", + " | Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and\n", + " | added tokens.\n", + " | \n", + " | Args:\n", + " | ids (`int` or `List[int]`):\n", + " | The token id (or token ids) to convert to tokens.\n", + " | skip_special_tokens (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to remove special tokens in the decoding.\n", + " | \n", + " | Returns:\n", + " | `str` or `List[str]`: The decoded token(s).\n", + " | \n", + " | convert_tokens_to_ids(self, tokens: Union[str, List[str]]) -> Union[int, List[int]]\n", + " | Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the\n", + " | vocabulary.\n", + " | \n", + " | Args:\n", + " | tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s).\n", + " | \n", + " | Returns:\n", + " | `int` or `List[int]`: The token id or list of token ids.\n", + " | \n", + " | convert_tokens_to_string(self, tokens: List[str]) -> str\n", + " | Converts a sequence of tokens in a single string. The most simple way to do it is `\" \".join(tokens)` but we\n", + " | often want to remove sub-word tokenization artifacts at the same time.\n", + " | \n", + " | Args:\n", + " | tokens (`List[str]`): The token to join in a string.\n", + " | \n", + " | Returns:\n", + " | `str`: The joined tokens.\n", + " | \n", + " | get_added_vocab(self) -> Dict[str, int]\n", + " | Returns the added tokens in the vocabulary as a dictionary of token to index.\n", + " | \n", + " | Returns:\n", + " | `Dict[str, int]`: The added tokens.\n", + " | \n", + " | get_vocab(self) -> Dict[str, int]\n", + " | Returns the vocabulary as a dictionary of token to index.\n", + " | \n", + " | `tokenizer.get_vocab()[token]` is equivalent to `tokenizer.convert_tokens_to_ids(token)` when `token` is in the\n", + " | vocab.\n", + " | \n", + " | Returns:\n", + " | `Dict[str, int]`: The vocabulary.\n", + " | \n", + " | num_special_tokens_to_add(self, pair: bool = False) -> int\n", + " | Returns the number of added tokens when encoding a sequence with special tokens.\n", + " | \n", + " | \n", + " | \n", + " | This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put\n", + " | this inside your training loop.\n", + " | \n", + " | \n", + " | \n", + " | Args:\n", + " | pair (`bool`, *optional*, defaults to `False`):\n", + " | Whether the number of added tokens should be computed in the case of a sequence pair or a single\n", + " | sequence.\n", + " | \n", + " | Returns:\n", + " | `int`: Number of special tokens added to sequences.\n", + " | \n", + " | set_truncation_and_padding(self, padding_strategy: transformers.utils.generic.PaddingStrategy, truncation_strategy: transformers.tokenization_utils_base.TruncationStrategy, max_length: int, stride: int, pad_to_multiple_of: Optional[int])\n", + " | Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers\n", + " | library) and restore the tokenizer settings afterwards.\n", + " | \n", + " | The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a\n", + " | padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed\n", + " | section.\n", + " | \n", + " | Args:\n", + " | padding_strategy ([`~utils.PaddingStrategy`]):\n", + " | The kind of padding that will be applied to the input\n", + " | truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]):\n", + " | The kind of truncation that will be applied to the input\n", + " | max_length (`int`):\n", + " | The maximum size of a sequence.\n", + " | stride (`int`):\n", + " | The stride to use when handling overflow.\n", + " | pad_to_multiple_of (`int`, *optional*):\n", + " | If set will pad the sequence to a multiple of the provided value. This is especially useful to enable\n", + " | the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta).\n", + " | \n", + " | tokenize(self, text: str, pair: Optional[str] = None, add_special_tokens: bool = False, **kwargs) -> List[str]\n", + " | Converts a string in a sequence of tokens, replacing unknown tokens with the `unk_token`.\n", + " | \n", + " | Args:\n", + " | text (`str`):\n", + " | The sequence to be encoded.\n", + " | pair (`str`, *optional*):\n", + " | A second sequence to be encoded with the first.\n", + " | add_special_tokens (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to add the special tokens associated with the corresponding model.\n", + " | kwargs (additional keyword arguments, *optional*):\n", + " | Will be passed to the underlying model specific encode method. See details in\n", + " | [`~PreTrainedTokenizerBase.__call__`]\n", + " | \n", + " | Returns:\n", + " | `List[str]`: The list of tokens.\n", + " | \n", + " | train_new_from_iterator(self, text_iterator, vocab_size, length=None, new_special_tokens=None, special_tokens_map=None, **kwargs)\n", + " | Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline)\n", + " | as the current one.\n", + " | \n", + " | Args:\n", + " | text_iterator (generator of `List[str]`):\n", + " | The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts\n", + " | if you have everything in memory.\n", + " | vocab_size (`int`):\n", + " | The size of the vocabulary you want for your tokenizer.\n", + " | length (`int`, *optional*):\n", + " | The total number of sequences in the iterator. This is used to provide meaningful progress tracking\n", + " | new_special_tokens (list of `str` or `AddedToken`, *optional*):\n", + " | A list of new special tokens to add to the tokenizer you are training.\n", + " | special_tokens_map (`Dict[str, str]`, *optional*):\n", + " | If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special\n", + " | token name to new special token name in this argument.\n", + " | kwargs:\n", + " | Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library.\n", + " | \n", + " | Returns:\n", + " | [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on\n", + " | `text_iterator`.\n", + " | \n", + " | ----------------------------------------------------------------------\n", + " | Readonly properties inherited from transformers.tokenization_utils_fast.PreTrainedTokenizerFast:\n", + " | \n", + " | backend_tokenizer\n", + " | `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend.\n", + " | \n", + " | decoder\n", + " | `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer.\n", + " | \n", + " | is_fast\n", + " | \n", + " | vocab\n", + " | \n", + " | vocab_size\n", + " | `int`: Size of the base vocabulary (without the added tokens).\n", + " | \n", + " | ----------------------------------------------------------------------\n", + " | Data and other attributes inherited from transformers.tokenization_utils_fast.PreTrainedTokenizerFast:\n", + " | \n", + " | __annotations__ = {'can_save_slow_tokenizer': , 'slow_to...\n", + " | \n", + " | can_save_slow_tokenizer = True\n", + " | \n", + " | ----------------------------------------------------------------------\n", + " | Methods inherited from transformers.tokenization_utils_base.PreTrainedTokenizerBase:\n", + " | \n", + " | __call__(self, text: Union[str, List[str], List[List[str]]] = None, text_pair: Union[str, List[str], List[List[str]], NoneType] = None, text_target: Union[str, List[str], List[List[str]]] = None, text_pair_target: Union[str, List[str], List[List[str]], NoneType] = None, add_special_tokens: bool = True, padding: Union[bool, str, transformers.utils.generic.PaddingStrategy] = False, truncation: Union[bool, str, transformers.tokenization_utils_base.TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Union[str, transformers.utils.generic.TensorType, NoneType] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs) -> transformers.tokenization_utils_base.BatchEncoding\n", + " | Main method to tokenize and prepare for the model one or several sequence(s) or one or several pair(s) of\n", + " | sequences.\n", + " | \n", + " | Args:\n", + " | text (`str`, `List[str]`, `List[List[str]]`, *optional*):\n", + " | The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings\n", + " | (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set\n", + " | `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).\n", + " | text_pair (`str`, `List[str]`, `List[List[str]]`, *optional*):\n", + " | The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings\n", + " | (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set\n", + " | `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).\n", + " | text_target (`str`, `List[str]`, `List[List[str]]`, *optional*):\n", + " | The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a\n", + " | list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized),\n", + " | you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).\n", + " | text_pair_target (`str`, `List[str]`, `List[List[str]]`, *optional*):\n", + " | The sequence or batch of sequences to be encoded as target texts. Each sequence can be a string or a\n", + " | list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized),\n", + " | you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences).\n", + " | \n", + " | add_special_tokens (`bool`, *optional*, defaults to `True`):\n", + " | Whether or not to encode the sequences with the special tokens relative to their model.\n", + " | padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`):\n", + " | Activates and controls padding. Accepts the following values:\n", + " | \n", + " | - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single\n", + " | sequence if provided).\n", + " | - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum\n", + " | acceptable input length for the model if that argument is not provided.\n", + " | - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different\n", + " | lengths).\n", + " | truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`):\n", + " | Activates and controls truncation. Accepts the following values:\n", + " | \n", + " | - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or\n", + " | to the maximum acceptable input length for the model if that argument is not provided. This will\n", + " | truncate token by token, removing a token from the longest sequence in the pair if a pair of\n", + " | sequences (or a batch of pairs) is provided.\n", + " | - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths\n", + " | greater than the model maximum admissible input size).\n", + " | max_length (`int`, *optional*):\n", + " | Controls the maximum length to use by one of the truncation/padding parameters.\n", + " | \n", + " | If left unset or set to `None`, this will use the predefined model maximum length if a maximum length\n", + " | is required by one of the truncation/padding parameters. If the model has no specific maximum input\n", + " | length (like XLNet) truncation/padding to a maximum length will be deactivated.\n", + " | stride (`int`, *optional*, defaults to 0):\n", + " | If set to a number along with `max_length`, the overflowing tokens returned when\n", + " | `return_overflowing_tokens=True` will contain some tokens from the end of the truncated sequence\n", + " | returned to provide some overlap between truncated and overflowing sequences. The value of this\n", + " | argument defines the number of overlapping tokens.\n", + " | is_split_into_words (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the\n", + " | tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)\n", + " | which it will tokenize. This is useful for NER or token classification.\n", + " | pad_to_multiple_of (`int`, *optional*):\n", + " | If set will pad the sequence to a multiple of the provided value. This is especially useful to enable\n", + " | the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta).\n", + " | return_tensors (`str` or [`~utils.TensorType`], *optional*):\n", + " | If set, will return tensors instead of list of python integers. Acceptable values are:\n", + " | \n", + " | - `'tf'`: Return TensorFlow `tf.constant` objects.\n", + " | - `'pt'`: Return PyTorch `torch.Tensor` objects.\n", + " | - `'np'`: Return Numpy `np.ndarray` objects.\n", + " | \n", + " | return_token_type_ids (`bool`, *optional*):\n", + " | Whether to return token type IDs. If left to the default, will return the token type IDs according to\n", + " | the specific tokenizer's default, defined by the `return_outputs` attribute.\n", + " | \n", + " | [What are token type IDs?](../glossary#token-type-ids)\n", + " | return_attention_mask (`bool`, *optional*):\n", + " | Whether to return the attention mask. If left to the default, will return the attention mask according\n", + " | to the specific tokenizer's default, defined by the `return_outputs` attribute.\n", + " | \n", + " | [What are attention masks?](../glossary#attention-mask)\n", + " | return_overflowing_tokens (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return overflowing token sequences. If a pair of sequences of input ids (or a batch\n", + " | of pairs) is provided with `truncation_strategy = longest_first` or `True`, an error is raised instead\n", + " | of returning overflowing tokens.\n", + " | return_special_tokens_mask (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return special tokens mask information.\n", + " | return_offsets_mapping (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return `(char_start, char_end)` for each token.\n", + " | \n", + " | This is only available on fast tokenizers inheriting from [`PreTrainedTokenizerFast`], if using\n", + " | Python's tokenizer, this method will raise `NotImplementedError`.\n", + " | return_length (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return the lengths of the encoded inputs.\n", + " | verbose (`bool`, *optional*, defaults to `True`):\n", + " | Whether or not to print more information and warnings.\n", + " | **kwargs: passed to the `self.tokenize()` method\n", + " | \n", + " | Return:\n", + " | [`BatchEncoding`]: A [`BatchEncoding`] with the following fields:\n", + " | \n", + " | - **input_ids** -- List of token ids to be fed to a model.\n", + " | \n", + " | [What are input IDs?](../glossary#input-ids)\n", + " | \n", + " | - **token_type_ids** -- List of token type ids to be fed to a model (when `return_token_type_ids=True` or\n", + " | if *\"token_type_ids\"* is in `self.model_input_names`).\n", + " | \n", + " | [What are token type IDs?](../glossary#token-type-ids)\n", + " | \n", + " | - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when\n", + " | `return_attention_mask=True` or if *\"attention_mask\"* is in `self.model_input_names`).\n", + " | \n", + " | [What are attention masks?](../glossary#attention-mask)\n", + " | \n", + " | - **overflowing_tokens** -- List of overflowing tokens sequences (when a `max_length` is specified and\n", + " | `return_overflowing_tokens=True`).\n", + " | - **num_truncated_tokens** -- Number of tokens truncated (when a `max_length` is specified and\n", + " | `return_overflowing_tokens=True`).\n", + " | - **special_tokens_mask** -- List of 0s and 1s, with 1 specifying added special tokens and 0 specifying\n", + " | regular sequence tokens (when `add_special_tokens=True` and `return_special_tokens_mask=True`).\n", + " | - **length** -- The length of the inputs (when `return_length=True`)\n", + " | \n", + " | __repr__(self) -> str\n", + " | Return repr(self).\n", + " | \n", + " | as_target_tokenizer(self)\n", + " | Temporarily sets the tokenizer for encoding the targets. Useful for tokenizer associated to\n", + " | sequence-to-sequence models that need a slightly different processing for the labels.\n", + " | \n", + " | batch_decode(self, sequences: Union[List[int], List[List[int]], ForwardRef('np.ndarray'), ForwardRef('torch.Tensor'), ForwardRef('tf.Tensor')], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = True, **kwargs) -> List[str]\n", + " | Convert a list of lists of token ids into a list of strings by calling decode.\n", + " | \n", + " | Args:\n", + " | sequences (`Union[List[int], List[List[int]], np.ndarray, torch.Tensor, tf.Tensor]`):\n", + " | List of tokenized input ids. Can be obtained using the `__call__` method.\n", + " | skip_special_tokens (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to remove special tokens in the decoding.\n", + " | clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`):\n", + " | Whether or not to clean up the tokenization spaces.\n", + " | kwargs (additional keyword arguments, *optional*):\n", + " | Will be passed to the underlying model specific decode method.\n", + " | \n", + " | Returns:\n", + " | `List[str]`: The list of decoded sentences.\n", + " | \n", + " | batch_encode_plus(self, batch_text_or_text_pairs: Union[List[str], List[Tuple[str, str]], List[List[str]], List[Tuple[List[str], List[str]]], List[List[int]], List[Tuple[List[int], List[int]]]], add_special_tokens: bool = True, padding: Union[bool, str, transformers.utils.generic.PaddingStrategy] = False, truncation: Union[bool, str, transformers.tokenization_utils_base.TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Union[str, transformers.utils.generic.TensorType, NoneType] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs) -> transformers.tokenization_utils_base.BatchEncoding\n", + " | Tokenize and prepare for the model a list of sequences or a list of pairs of sequences.\n", + " | \n", + " | \n", + " | \n", + " | This method is deprecated, `__call__` should be used instead.\n", + " | \n", + " | \n", + " | \n", + " | Args:\n", + " | batch_text_or_text_pairs (`List[str]`, `List[Tuple[str, str]]`, `List[List[str]]`, `List[Tuple[List[str], List[str]]]`, and for not-fast tokenizers, also `List[List[int]]`, `List[Tuple[List[int], List[int]]]`):\n", + " | Batch of sequences or pair of sequences to be encoded. This can be a list of\n", + " | string/string-sequences/int-sequences or a list of pair of string/string-sequences/int-sequence (see\n", + " | details in `encode_plus`).\n", + " | \n", + " | add_special_tokens (`bool`, *optional*, defaults to `True`):\n", + " | Whether or not to encode the sequences with the special tokens relative to their model.\n", + " | padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`):\n", + " | Activates and controls padding. Accepts the following values:\n", + " | \n", + " | - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single\n", + " | sequence if provided).\n", + " | - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum\n", + " | acceptable input length for the model if that argument is not provided.\n", + " | - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different\n", + " | lengths).\n", + " | truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`):\n", + " | Activates and controls truncation. Accepts the following values:\n", + " | \n", + " | - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or\n", + " | to the maximum acceptable input length for the model if that argument is not provided. This will\n", + " | truncate token by token, removing a token from the longest sequence in the pair if a pair of\n", + " | sequences (or a batch of pairs) is provided.\n", + " | - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths\n", + " | greater than the model maximum admissible input size).\n", + " | max_length (`int`, *optional*):\n", + " | Controls the maximum length to use by one of the truncation/padding parameters.\n", + " | \n", + " | If left unset or set to `None`, this will use the predefined model maximum length if a maximum length\n", + " | is required by one of the truncation/padding parameters. If the model has no specific maximum input\n", + " | length (like XLNet) truncation/padding to a maximum length will be deactivated.\n", + " | stride (`int`, *optional*, defaults to 0):\n", + " | If set to a number along with `max_length`, the overflowing tokens returned when\n", + " | `return_overflowing_tokens=True` will contain some tokens from the end of the truncated sequence\n", + " | returned to provide some overlap between truncated and overflowing sequences. The value of this\n", + " | argument defines the number of overlapping tokens.\n", + " | is_split_into_words (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the\n", + " | tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)\n", + " | which it will tokenize. This is useful for NER or token classification.\n", + " | pad_to_multiple_of (`int`, *optional*):\n", + " | If set will pad the sequence to a multiple of the provided value. This is especially useful to enable\n", + " | the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta).\n", + " | return_tensors (`str` or [`~utils.TensorType`], *optional*):\n", + " | If set, will return tensors instead of list of python integers. Acceptable values are:\n", + " | \n", + " | - `'tf'`: Return TensorFlow `tf.constant` objects.\n", + " | - `'pt'`: Return PyTorch `torch.Tensor` objects.\n", + " | - `'np'`: Return Numpy `np.ndarray` objects.\n", + " | \n", + " | return_token_type_ids (`bool`, *optional*):\n", + " | Whether to return token type IDs. If left to the default, will return the token type IDs according to\n", + " | the specific tokenizer's default, defined by the `return_outputs` attribute.\n", + " | \n", + " | [What are token type IDs?](../glossary#token-type-ids)\n", + " | return_attention_mask (`bool`, *optional*):\n", + " | Whether to return the attention mask. If left to the default, will return the attention mask according\n", + " | to the specific tokenizer's default, defined by the `return_outputs` attribute.\n", + " | \n", + " | [What are attention masks?](../glossary#attention-mask)\n", + " | return_overflowing_tokens (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return overflowing token sequences. If a pair of sequences of input ids (or a batch\n", + " | of pairs) is provided with `truncation_strategy = longest_first` or `True`, an error is raised instead\n", + " | of returning overflowing tokens.\n", + " | return_special_tokens_mask (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return special tokens mask information.\n", + " | return_offsets_mapping (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return `(char_start, char_end)` for each token.\n", + " | \n", + " | This is only available on fast tokenizers inheriting from [`PreTrainedTokenizerFast`], if using\n", + " | Python's tokenizer, this method will raise `NotImplementedError`.\n", + " | return_length (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return the lengths of the encoded inputs.\n", + " | verbose (`bool`, *optional*, defaults to `True`):\n", + " | Whether or not to print more information and warnings.\n", + " | **kwargs: passed to the `self.tokenize()` method\n", + " | \n", + " | Return:\n", + " | [`BatchEncoding`]: A [`BatchEncoding`] with the following fields:\n", + " | \n", + " | - **input_ids** -- List of token ids to be fed to a model.\n", + " | \n", + " | [What are input IDs?](../glossary#input-ids)\n", + " | \n", + " | - **token_type_ids** -- List of token type ids to be fed to a model (when `return_token_type_ids=True` or\n", + " | if *\"token_type_ids\"* is in `self.model_input_names`).\n", + " | \n", + " | [What are token type IDs?](../glossary#token-type-ids)\n", + " | \n", + " | - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when\n", + " | `return_attention_mask=True` or if *\"attention_mask\"* is in `self.model_input_names`).\n", + " | \n", + " | [What are attention masks?](../glossary#attention-mask)\n", + " | \n", + " | - **overflowing_tokens** -- List of overflowing tokens sequences (when a `max_length` is specified and\n", + " | `return_overflowing_tokens=True`).\n", + " | - **num_truncated_tokens** -- Number of tokens truncated (when a `max_length` is specified and\n", + " | `return_overflowing_tokens=True`).\n", + " | - **special_tokens_mask** -- List of 0s and 1s, with 1 specifying added special tokens and 0 specifying\n", + " | regular sequence tokens (when `add_special_tokens=True` and `return_special_tokens_mask=True`).\n", + " | - **length** -- The length of the inputs (when `return_length=True`)\n", + " | \n", + " | decode(self, token_ids: Union[int, List[int], ForwardRef('np.ndarray'), ForwardRef('torch.Tensor'), ForwardRef('tf.Tensor')], skip_special_tokens: bool = False, clean_up_tokenization_spaces: bool = True, **kwargs) -> str\n", + " | Converts a sequence of ids in a string, using the tokenizer and vocabulary with options to remove special\n", + " | tokens and clean up tokenization spaces.\n", + " | \n", + " | Similar to doing `self.convert_tokens_to_string(self.convert_ids_to_tokens(token_ids))`.\n", + " | \n", + " | Args:\n", + " | token_ids (`Union[int, List[int], np.ndarray, torch.Tensor, tf.Tensor]`):\n", + " | List of tokenized input ids. Can be obtained using the `__call__` method.\n", + " | skip_special_tokens (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to remove special tokens in the decoding.\n", + " | clean_up_tokenization_spaces (`bool`, *optional*, defaults to `True`):\n", + " | Whether or not to clean up the tokenization spaces.\n", + " | kwargs (additional keyword arguments, *optional*):\n", + " | Will be passed to the underlying model specific decode method.\n", + " | \n", + " | Returns:\n", + " | `str`: The decoded sentence.\n", + " | \n", + " | encode(self, text: Union[str, List[str], List[int]], text_pair: Union[str, List[str], List[int], NoneType] = None, add_special_tokens: bool = True, padding: Union[bool, str, transformers.utils.generic.PaddingStrategy] = False, truncation: Union[bool, str, transformers.tokenization_utils_base.TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, return_tensors: Union[str, transformers.utils.generic.TensorType, NoneType] = None, **kwargs) -> List[int]\n", + " | Converts a string to a sequence of ids (integer), using the tokenizer and vocabulary.\n", + " | \n", + " | Same as doing `self.convert_tokens_to_ids(self.tokenize(text))`.\n", + " | \n", + " | Args:\n", + " | text (`str`, `List[str]` or `List[int]`):\n", + " | The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the\n", + " | `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids`\n", + " | method).\n", + " | text_pair (`str`, `List[str]` or `List[int]`, *optional*):\n", + " | Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using\n", + " | the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids`\n", + " | method).\n", + " | \n", + " | add_special_tokens (`bool`, *optional*, defaults to `True`):\n", + " | Whether or not to encode the sequences with the special tokens relative to their model.\n", + " | padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`):\n", + " | Activates and controls padding. Accepts the following values:\n", + " | \n", + " | - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single\n", + " | sequence if provided).\n", + " | - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum\n", + " | acceptable input length for the model if that argument is not provided.\n", + " | - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different\n", + " | lengths).\n", + " | truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`):\n", + " | Activates and controls truncation. Accepts the following values:\n", + " | \n", + " | - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or\n", + " | to the maximum acceptable input length for the model if that argument is not provided. This will\n", + " | truncate token by token, removing a token from the longest sequence in the pair if a pair of\n", + " | sequences (or a batch of pairs) is provided.\n", + " | - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths\n", + " | greater than the model maximum admissible input size).\n", + " | max_length (`int`, *optional*):\n", + " | Controls the maximum length to use by one of the truncation/padding parameters.\n", + " | \n", + " | If left unset or set to `None`, this will use the predefined model maximum length if a maximum length\n", + " | is required by one of the truncation/padding parameters. If the model has no specific maximum input\n", + " | length (like XLNet) truncation/padding to a maximum length will be deactivated.\n", + " | stride (`int`, *optional*, defaults to 0):\n", + " | If set to a number along with `max_length`, the overflowing tokens returned when\n", + " | `return_overflowing_tokens=True` will contain some tokens from the end of the truncated sequence\n", + " | returned to provide some overlap between truncated and overflowing sequences. The value of this\n", + " | argument defines the number of overlapping tokens.\n", + " | is_split_into_words (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the\n", + " | tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)\n", + " | which it will tokenize. This is useful for NER or token classification.\n", + " | pad_to_multiple_of (`int`, *optional*):\n", + " | If set will pad the sequence to a multiple of the provided value. This is especially useful to enable\n", + " | the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta).\n", + " | return_tensors (`str` or [`~utils.TensorType`], *optional*):\n", + " | If set, will return tensors instead of list of python integers. Acceptable values are:\n", + " | \n", + " | - `'tf'`: Return TensorFlow `tf.constant` objects.\n", + " | - `'pt'`: Return PyTorch `torch.Tensor` objects.\n", + " | - `'np'`: Return Numpy `np.ndarray` objects.\n", + " | \n", + " | **kwargs: Passed along to the `.tokenize()` method.\n", + " | \n", + " | Returns:\n", + " | `List[int]`, `torch.Tensor`, `tf.Tensor` or `np.ndarray`: The tokenized ids of the text.\n", + " | \n", + " | encode_plus(self, text: Union[str, List[str], List[int]], text_pair: Union[str, List[str], List[int], NoneType] = None, add_special_tokens: bool = True, padding: Union[bool, str, transformers.utils.generic.PaddingStrategy] = False, truncation: Union[bool, str, transformers.tokenization_utils_base.TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, is_split_into_words: bool = False, pad_to_multiple_of: Optional[int] = None, return_tensors: Union[str, transformers.utils.generic.TensorType, NoneType] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, **kwargs) -> transformers.tokenization_utils_base.BatchEncoding\n", + " | Tokenize and prepare for the model a sequence or a pair of sequences.\n", + " | \n", + " | \n", + " | \n", + " | This method is deprecated, `__call__` should be used instead.\n", + " | \n", + " | \n", + " | \n", + " | Args:\n", + " | text (`str`, `List[str]` or `List[int]` (the latter only for not-fast tokenizers)):\n", + " | The first sequence to be encoded. This can be a string, a list of strings (tokenized string using the\n", + " | `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids`\n", + " | method).\n", + " | text_pair (`str`, `List[str]` or `List[int]`, *optional*):\n", + " | Optional second sequence to be encoded. This can be a string, a list of strings (tokenized string using\n", + " | the `tokenize` method) or a list of integers (tokenized string ids using the `convert_tokens_to_ids`\n", + " | method).\n", + " | \n", + " | add_special_tokens (`bool`, *optional*, defaults to `True`):\n", + " | Whether or not to encode the sequences with the special tokens relative to their model.\n", + " | padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`):\n", + " | Activates and controls padding. Accepts the following values:\n", + " | \n", + " | - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single\n", + " | sequence if provided).\n", + " | - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum\n", + " | acceptable input length for the model if that argument is not provided.\n", + " | - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different\n", + " | lengths).\n", + " | truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`):\n", + " | Activates and controls truncation. Accepts the following values:\n", + " | \n", + " | - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or\n", + " | to the maximum acceptable input length for the model if that argument is not provided. This will\n", + " | truncate token by token, removing a token from the longest sequence in the pair if a pair of\n", + " | sequences (or a batch of pairs) is provided.\n", + " | - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths\n", + " | greater than the model maximum admissible input size).\n", + " | max_length (`int`, *optional*):\n", + " | Controls the maximum length to use by one of the truncation/padding parameters.\n", + " | \n", + " | If left unset or set to `None`, this will use the predefined model maximum length if a maximum length\n", + " | is required by one of the truncation/padding parameters. If the model has no specific maximum input\n", + " | length (like XLNet) truncation/padding to a maximum length will be deactivated.\n", + " | stride (`int`, *optional*, defaults to 0):\n", + " | If set to a number along with `max_length`, the overflowing tokens returned when\n", + " | `return_overflowing_tokens=True` will contain some tokens from the end of the truncated sequence\n", + " | returned to provide some overlap between truncated and overflowing sequences. The value of this\n", + " | argument defines the number of overlapping tokens.\n", + " | is_split_into_words (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the\n", + " | tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)\n", + " | which it will tokenize. This is useful for NER or token classification.\n", + " | pad_to_multiple_of (`int`, *optional*):\n", + " | If set will pad the sequence to a multiple of the provided value. This is especially useful to enable\n", + " | the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta).\n", + " | return_tensors (`str` or [`~utils.TensorType`], *optional*):\n", + " | If set, will return tensors instead of list of python integers. Acceptable values are:\n", + " | \n", + " | - `'tf'`: Return TensorFlow `tf.constant` objects.\n", + " | - `'pt'`: Return PyTorch `torch.Tensor` objects.\n", + " | - `'np'`: Return Numpy `np.ndarray` objects.\n", + " | \n", + " | return_token_type_ids (`bool`, *optional*):\n", + " | Whether to return token type IDs. If left to the default, will return the token type IDs according to\n", + " | the specific tokenizer's default, defined by the `return_outputs` attribute.\n", + " | \n", + " | [What are token type IDs?](../glossary#token-type-ids)\n", + " | return_attention_mask (`bool`, *optional*):\n", + " | Whether to return the attention mask. If left to the default, will return the attention mask according\n", + " | to the specific tokenizer's default, defined by the `return_outputs` attribute.\n", + " | \n", + " | [What are attention masks?](../glossary#attention-mask)\n", + " | return_overflowing_tokens (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return overflowing token sequences. If a pair of sequences of input ids (or a batch\n", + " | of pairs) is provided with `truncation_strategy = longest_first` or `True`, an error is raised instead\n", + " | of returning overflowing tokens.\n", + " | return_special_tokens_mask (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return special tokens mask information.\n", + " | return_offsets_mapping (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return `(char_start, char_end)` for each token.\n", + " | \n", + " | This is only available on fast tokenizers inheriting from [`PreTrainedTokenizerFast`], if using\n", + " | Python's tokenizer, this method will raise `NotImplementedError`.\n", + " | return_length (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return the lengths of the encoded inputs.\n", + " | verbose (`bool`, *optional*, defaults to `True`):\n", + " | Whether or not to print more information and warnings.\n", + " | **kwargs: passed to the `self.tokenize()` method\n", + " | \n", + " | Return:\n", + " | [`BatchEncoding`]: A [`BatchEncoding`] with the following fields:\n", + " | \n", + " | - **input_ids** -- List of token ids to be fed to a model.\n", + " | \n", + " | [What are input IDs?](../glossary#input-ids)\n", + " | \n", + " | - **token_type_ids** -- List of token type ids to be fed to a model (when `return_token_type_ids=True` or\n", + " | if *\"token_type_ids\"* is in `self.model_input_names`).\n", + " | \n", + " | [What are token type IDs?](../glossary#token-type-ids)\n", + " | \n", + " | - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when\n", + " | `return_attention_mask=True` or if *\"attention_mask\"* is in `self.model_input_names`).\n", + " | \n", + " | [What are attention masks?](../glossary#attention-mask)\n", + " | \n", + " | - **overflowing_tokens** -- List of overflowing tokens sequences (when a `max_length` is specified and\n", + " | `return_overflowing_tokens=True`).\n", + " | - **num_truncated_tokens** -- Number of tokens truncated (when a `max_length` is specified and\n", + " | `return_overflowing_tokens=True`).\n", + " | - **special_tokens_mask** -- List of 0s and 1s, with 1 specifying added special tokens and 0 specifying\n", + " | regular sequence tokens (when `add_special_tokens=True` and `return_special_tokens_mask=True`).\n", + " | - **length** -- The length of the inputs (when `return_length=True`)\n", + " | \n", + " | get_special_tokens_mask(self, token_ids_0: List[int], token_ids_1: Optional[List[int]] = None, already_has_special_tokens: bool = False) -> List[int]\n", + " | Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding\n", + " | special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods.\n", + " | \n", + " | Args:\n", + " | token_ids_0 (`List[int]`):\n", + " | List of ids of the first sequence.\n", + " | token_ids_1 (`List[int]`, *optional*):\n", + " | List of ids of the second sequence.\n", + " | already_has_special_tokens (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not the token list is already formatted with special tokens for the model.\n", + " | \n", + " | Returns:\n", + " | A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.\n", + " | \n", + " | pad(self, encoded_inputs: Union[transformers.tokenization_utils_base.BatchEncoding, List[transformers.tokenization_utils_base.BatchEncoding], Dict[str, List[int]], Dict[str, List[List[int]]], List[Dict[str, List[int]]]], padding: Union[bool, str, transformers.utils.generic.PaddingStrategy] = True, max_length: Optional[int] = None, pad_to_multiple_of: Optional[int] = None, return_attention_mask: Optional[bool] = None, return_tensors: Union[str, transformers.utils.generic.TensorType, NoneType] = None, verbose: bool = True) -> transformers.tokenization_utils_base.BatchEncoding\n", + " | Pad a single encoded input or a batch of encoded inputs up to predefined length or to the max sequence length\n", + " | in the batch.\n", + " | \n", + " | Padding side (left/right) padding token ids are defined at the tokenizer level (with `self.padding_side`,\n", + " | `self.pad_token_id` and `self.pad_token_type_id`).\n", + " | \n", + " | Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the\n", + " | text followed by a call to the `pad` method to get a padded encoding.\n", + " | \n", + " | \n", + " | \n", + " | If the `encoded_inputs` passed are dictionary of numpy arrays, PyTorch tensors or TensorFlow tensors, the\n", + " | result will use the same type unless you provide a different tensor type with `return_tensors`. In the case of\n", + " | PyTorch tensors, you will lose the specific device of your tensors however.\n", + " | \n", + " | \n", + " | \n", + " | Args:\n", + " | encoded_inputs ([`BatchEncoding`], list of [`BatchEncoding`], `Dict[str, List[int]]`, `Dict[str, List[List[int]]` or `List[Dict[str, List[int]]]`):\n", + " | Tokenized inputs. Can represent one input ([`BatchEncoding`] or `Dict[str, List[int]]`) or a batch of\n", + " | tokenized inputs (list of [`BatchEncoding`], *Dict[str, List[List[int]]]* or *List[Dict[str,\n", + " | List[int]]]*) so you can use this method during preprocessing as well as in a PyTorch Dataloader\n", + " | collate function.\n", + " | \n", + " | Instead of `List[int]` you can have tensors (numpy arrays, PyTorch tensors or TensorFlow tensors), see\n", + " | the note above for the return type.\n", + " | padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`):\n", + " | Select a strategy to pad the returned sequences (according to the model's padding side and padding\n", + " | index) among:\n", + " | \n", + " | - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single\n", + " | sequence if provided).\n", + " | - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum\n", + " | acceptable input length for the model if that argument is not provided.\n", + " | - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different\n", + " | lengths).\n", + " | max_length (`int`, *optional*):\n", + " | Maximum length of the returned list and optionally padding length (see above).\n", + " | pad_to_multiple_of (`int`, *optional*):\n", + " | If set will pad the sequence to a multiple of the provided value.\n", + " | \n", + " | This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability\n", + " | `>= 7.5` (Volta).\n", + " | return_attention_mask (`bool`, *optional*):\n", + " | Whether to return the attention mask. If left to the default, will return the attention mask according\n", + " | to the specific tokenizer's default, defined by the `return_outputs` attribute.\n", + " | \n", + " | [What are attention masks?](../glossary#attention-mask)\n", + " | return_tensors (`str` or [`~utils.TensorType`], *optional*):\n", + " | If set, will return tensors instead of list of python integers. Acceptable values are:\n", + " | \n", + " | - `'tf'`: Return TensorFlow `tf.constant` objects.\n", + " | - `'pt'`: Return PyTorch `torch.Tensor` objects.\n", + " | - `'np'`: Return Numpy `np.ndarray` objects.\n", + " | verbose (`bool`, *optional*, defaults to `True`):\n", + " | Whether or not to print more information and warnings.\n", + " | \n", + " | prepare_for_model(self, ids: List[int], pair_ids: Optional[List[int]] = None, add_special_tokens: bool = True, padding: Union[bool, str, transformers.utils.generic.PaddingStrategy] = False, truncation: Union[bool, str, transformers.tokenization_utils_base.TruncationStrategy] = None, max_length: Optional[int] = None, stride: int = 0, pad_to_multiple_of: Optional[int] = None, return_tensors: Union[str, transformers.utils.generic.TensorType, NoneType] = None, return_token_type_ids: Optional[bool] = None, return_attention_mask: Optional[bool] = None, return_overflowing_tokens: bool = False, return_special_tokens_mask: bool = False, return_offsets_mapping: bool = False, return_length: bool = False, verbose: bool = True, prepend_batch_axis: bool = False, **kwargs) -> transformers.tokenization_utils_base.BatchEncoding\n", + " | Prepares a sequence of input id, or a pair of sequences of inputs ids so that it can be used by the model. It\n", + " | adds special tokens, truncates sequences if overflowing while taking into account the special tokens and\n", + " | manages a moving window (with user defined stride) for overflowing tokens. Please Note, for *pair_ids*\n", + " | different than `None` and *truncation_strategy = longest_first* or `True`, it is not possible to return\n", + " | overflowing tokens. Such a combination of arguments will raise an error.\n", + " | \n", + " | Args:\n", + " | ids (`List[int]`):\n", + " | Tokenized input ids of the first sequence. Can be obtained from a string by chaining the `tokenize` and\n", + " | `convert_tokens_to_ids` methods.\n", + " | pair_ids (`List[int]`, *optional*):\n", + " | Tokenized input ids of the second sequence. Can be obtained from a string by chaining the `tokenize`\n", + " | and `convert_tokens_to_ids` methods.\n", + " | \n", + " | add_special_tokens (`bool`, *optional*, defaults to `True`):\n", + " | Whether or not to encode the sequences with the special tokens relative to their model.\n", + " | padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`):\n", + " | Activates and controls padding. Accepts the following values:\n", + " | \n", + " | - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single\n", + " | sequence if provided).\n", + " | - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum\n", + " | acceptable input length for the model if that argument is not provided.\n", + " | - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different\n", + " | lengths).\n", + " | truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`):\n", + " | Activates and controls truncation. Accepts the following values:\n", + " | \n", + " | - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or\n", + " | to the maximum acceptable input length for the model if that argument is not provided. This will\n", + " | truncate token by token, removing a token from the longest sequence in the pair if a pair of\n", + " | sequences (or a batch of pairs) is provided.\n", + " | - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths\n", + " | greater than the model maximum admissible input size).\n", + " | max_length (`int`, *optional*):\n", + " | Controls the maximum length to use by one of the truncation/padding parameters.\n", + " | \n", + " | If left unset or set to `None`, this will use the predefined model maximum length if a maximum length\n", + " | is required by one of the truncation/padding parameters. If the model has no specific maximum input\n", + " | length (like XLNet) truncation/padding to a maximum length will be deactivated.\n", + " | stride (`int`, *optional*, defaults to 0):\n", + " | If set to a number along with `max_length`, the overflowing tokens returned when\n", + " | `return_overflowing_tokens=True` will contain some tokens from the end of the truncated sequence\n", + " | returned to provide some overlap between truncated and overflowing sequences. The value of this\n", + " | argument defines the number of overlapping tokens.\n", + " | is_split_into_words (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not the input is already pre-tokenized (e.g., split into words). If set to `True`, the\n", + " | tokenizer assumes the input is already split into words (for instance, by splitting it on whitespace)\n", + " | which it will tokenize. This is useful for NER or token classification.\n", + " | pad_to_multiple_of (`int`, *optional*):\n", + " | If set will pad the sequence to a multiple of the provided value. This is especially useful to enable\n", + " | the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta).\n", + " | return_tensors (`str` or [`~utils.TensorType`], *optional*):\n", + " | If set, will return tensors instead of list of python integers. Acceptable values are:\n", + " | \n", + " | - `'tf'`: Return TensorFlow `tf.constant` objects.\n", + " | - `'pt'`: Return PyTorch `torch.Tensor` objects.\n", + " | - `'np'`: Return Numpy `np.ndarray` objects.\n", + " | \n", + " | return_token_type_ids (`bool`, *optional*):\n", + " | Whether to return token type IDs. If left to the default, will return the token type IDs according to\n", + " | the specific tokenizer's default, defined by the `return_outputs` attribute.\n", + " | \n", + " | [What are token type IDs?](../glossary#token-type-ids)\n", + " | return_attention_mask (`bool`, *optional*):\n", + " | Whether to return the attention mask. If left to the default, will return the attention mask according\n", + " | to the specific tokenizer's default, defined by the `return_outputs` attribute.\n", + " | \n", + " | [What are attention masks?](../glossary#attention-mask)\n", + " | return_overflowing_tokens (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return overflowing token sequences. If a pair of sequences of input ids (or a batch\n", + " | of pairs) is provided with `truncation_strategy = longest_first` or `True`, an error is raised instead\n", + " | of returning overflowing tokens.\n", + " | return_special_tokens_mask (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return special tokens mask information.\n", + " | return_offsets_mapping (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return `(char_start, char_end)` for each token.\n", + " | \n", + " | This is only available on fast tokenizers inheriting from [`PreTrainedTokenizerFast`], if using\n", + " | Python's tokenizer, this method will raise `NotImplementedError`.\n", + " | return_length (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to return the lengths of the encoded inputs.\n", + " | verbose (`bool`, *optional*, defaults to `True`):\n", + " | Whether or not to print more information and warnings.\n", + " | **kwargs: passed to the `self.tokenize()` method\n", + " | \n", + " | Return:\n", + " | [`BatchEncoding`]: A [`BatchEncoding`] with the following fields:\n", + " | \n", + " | - **input_ids** -- List of token ids to be fed to a model.\n", + " | \n", + " | [What are input IDs?](../glossary#input-ids)\n", + " | \n", + " | - **token_type_ids** -- List of token type ids to be fed to a model (when `return_token_type_ids=True` or\n", + " | if *\"token_type_ids\"* is in `self.model_input_names`).\n", + " | \n", + " | [What are token type IDs?](../glossary#token-type-ids)\n", + " | \n", + " | - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when\n", + " | `return_attention_mask=True` or if *\"attention_mask\"* is in `self.model_input_names`).\n", + " | \n", + " | [What are attention masks?](../glossary#attention-mask)\n", + " | \n", + " | - **overflowing_tokens** -- List of overflowing tokens sequences (when a `max_length` is specified and\n", + " | `return_overflowing_tokens=True`).\n", + " | - **num_truncated_tokens** -- Number of tokens truncated (when a `max_length` is specified and\n", + " | `return_overflowing_tokens=True`).\n", + " | - **special_tokens_mask** -- List of 0s and 1s, with 1 specifying added special tokens and 0 specifying\n", + " | regular sequence tokens (when `add_special_tokens=True` and `return_special_tokens_mask=True`).\n", + " | - **length** -- The length of the inputs (when `return_length=True`)\n", + " | \n", + " | prepare_seq2seq_batch(self, src_texts: List[str], tgt_texts: Optional[List[str]] = None, max_length: Optional[int] = None, max_target_length: Optional[int] = None, padding: str = 'longest', return_tensors: str = None, truncation: bool = True, **kwargs) -> transformers.tokenization_utils_base.BatchEncoding\n", + " | Prepare model inputs for translation. For best performance, translate one sentence at a time.\n", + " | \n", + " | Arguments:\n", + " | src_texts (`List[str]`):\n", + " | List of documents to summarize or source language texts.\n", + " | tgt_texts (`list`, *optional*):\n", + " | List of summaries or target language texts.\n", + " | max_length (`int`, *optional*):\n", + " | Controls the maximum length for encoder inputs (documents to summarize or source language texts) If\n", + " | left unset or set to `None`, this will use the predefined model maximum length if a maximum length is\n", + " | required by one of the truncation/padding parameters. If the model has no specific maximum input length\n", + " | (like XLNet) truncation/padding to a maximum length will be deactivated.\n", + " | max_target_length (`int`, *optional*):\n", + " | Controls the maximum length of decoder inputs (target language texts or summaries) If left unset or set\n", + " | to `None`, this will use the max_length value.\n", + " | padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `False`):\n", + " | Activates and controls padding. Accepts the following values:\n", + " | \n", + " | - `True` or `'longest'`: Pad to the longest sequence in the batch (or no padding if only a single\n", + " | sequence if provided).\n", + " | - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum\n", + " | acceptable input length for the model if that argument is not provided.\n", + " | - `False` or `'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of different\n", + " | lengths).\n", + " | return_tensors (`str` or [`~utils.TensorType`], *optional*):\n", + " | If set, will return tensors instead of list of python integers. Acceptable values are:\n", + " | \n", + " | - `'tf'`: Return TensorFlow `tf.constant` objects.\n", + " | - `'pt'`: Return PyTorch `torch.Tensor` objects.\n", + " | - `'np'`: Return Numpy `np.ndarray` objects.\n", + " | truncation (`bool`, `str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `True`):\n", + " | Activates and controls truncation. Accepts the following values:\n", + " | \n", + " | - `True` or `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or\n", + " | to the maximum acceptable input length for the model if that argument is not provided. This will\n", + " | truncate token by token, removing a token from the longest sequence in the pair if a pair of\n", + " | sequences (or a batch of pairs) is provided.\n", + " | - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `False` or `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths\n", + " | greater than the model maximum admissible input size).\n", + " | **kwargs:\n", + " | Additional keyword arguments passed along to `self.__call__`.\n", + " | \n", + " | Return:\n", + " | [`BatchEncoding`]: A [`BatchEncoding`] with the following fields:\n", + " | \n", + " | - **input_ids** -- List of token ids to be fed to the encoder.\n", + " | - **attention_mask** -- List of indices specifying which tokens should be attended to by the model.\n", + " | - **labels** -- List of token ids for tgt_texts.\n", + " | \n", + " | The full set of keys `[input_ids, attention_mask, labels]`, will only be returned if tgt_texts is passed.\n", + " | Otherwise, input_ids, attention_mask will be the only keys.\n", + " | \n", + " | push_to_hub(self, repo_id: str, use_temp_dir: Optional[bool] = None, commit_message: Optional[str] = None, private: Optional[bool] = None, use_auth_token: Union[bool, str, NoneType] = None, max_shard_size: Union[str, int, NoneType] = '10GB', create_pr: bool = False, **deprecated_kwargs) -> str\n", + " | Upload the tokenizer files to the 🤗 Model Hub while synchronizing a local clone of the repo in\n", + " | `repo_path_or_name`.\n", + " | \n", + " | Parameters:\n", + " | repo_id (`str`):\n", + " | The name of the repository you want to push your tokenizer to. It should contain your organization name\n", + " | when pushing to a given organization.\n", + " | use_temp_dir (`bool`, *optional*):\n", + " | Whether or not to use a temporary directory to store the files saved before they are pushed to the Hub.\n", + " | Will default to `True` if there is no directory named like `repo_id`, `False` otherwise.\n", + " | commit_message (`str`, *optional*):\n", + " | Message to commit while pushing. Will default to `\"Upload tokenizer\"`.\n", + " | private (`bool`, *optional*):\n", + " | Whether or not the repository created should be private.\n", + " | use_auth_token (`bool` or `str`, *optional*):\n", + " | The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated\n", + " | when running `huggingface-cli login` (stored in `~/.huggingface`). Will default to `True` if `repo_url`\n", + " | is not specified.\n", + " | max_shard_size (`int` or `str`, *optional*, defaults to `\"10GB\"`):\n", + " | Only applicable for models. The maximum size for a checkpoint before being sharded. Checkpoints shard\n", + " | will then be each of size lower than this size. If expressed as a string, needs to be digits followed\n", + " | by a unit (like `\"5MB\"`).\n", + " | create_pr (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to create a PR with the uploaded files or directly commit.\n", + " | \n", + " | Examples:\n", + " | \n", + " | ```python\n", + " | from transformers import AutoTokenizer\n", + " | \n", + " | tokenizer = AutoTokenizer.from_pretrained(\"bert-base-cased\")\n", + " | \n", + " | # Push the tokenizer to your namespace with the name \"my-finetuned-bert\".\n", + " | tokenizer.push_to_hub(\"my-finetuned-bert\")\n", + " | \n", + " | # Push the tokenizer to an organization with the name \"my-finetuned-bert\".\n", + " | tokenizer.push_to_hub(\"huggingface/my-finetuned-bert\")\n", + " | ```\n", + " | \n", + " | save_pretrained(self, save_directory: Union[str, os.PathLike], legacy_format: Optional[bool] = None, filename_prefix: Optional[str] = None, push_to_hub: bool = False, **kwargs) -> Tuple[str]\n", + " | Save the full tokenizer state.\n", + " | \n", + " | \n", + " | This method make sure the full tokenizer can then be re-loaded using the\n", + " | [`~tokenization_utils_base.PreTrainedTokenizer.from_pretrained`] class method..\n", + " | \n", + " | Warning,None This won't save modifications you may have applied to the tokenizer after the instantiation (for\n", + " | instance, modifying `tokenizer.do_lower_case` after creation).\n", + " | \n", + " | Args:\n", + " | save_directory (`str` or `os.PathLike`): The path to a directory where the tokenizer will be saved.\n", + " | legacy_format (`bool`, *optional*):\n", + " | Only applicable for a fast tokenizer. If unset (default), will save the tokenizer in the unified JSON\n", + " | format as well as in legacy format if it exists, i.e. with tokenizer specific vocabulary and a separate\n", + " | added_tokens files.\n", + " | \n", + " | If `False`, will only save the tokenizer in the unified JSON format. This format is incompatible with\n", + " | \"slow\" tokenizers (not powered by the *tokenizers* library), so the tokenizer will not be able to be\n", + " | loaded in the corresponding \"slow\" tokenizer.\n", + " | \n", + " | If `True`, will save the tokenizer in legacy format. If the \"slow\" tokenizer doesn't exits, a value\n", + " | error is raised.\n", + " | filename_prefix: (`str`, *optional*):\n", + " | A prefix to add to the names of the files saved by the tokenizer.\n", + " | push_to_hub (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the\n", + " | repository you want to push to with `repo_id` (will default to the name of `save_directory` in your\n", + " | namespace).\n", + " | kwargs:\n", + " | Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method.\n", + " | \n", + " | Returns:\n", + " | A tuple of `str`: The files saved.\n", + " | \n", + " | truncate_sequences(self, ids: List[int], pair_ids: Optional[List[int]] = None, num_tokens_to_remove: int = 0, truncation_strategy: Union[str, transformers.tokenization_utils_base.TruncationStrategy] = 'longest_first', stride: int = 0) -> Tuple[List[int], List[int], List[int]]\n", + " | Truncates a sequence pair in-place following the strategy.\n", + " | \n", + " | Args:\n", + " | ids (`List[int]`):\n", + " | Tokenized input ids of the first sequence. Can be obtained from a string by chaining the `tokenize` and\n", + " | `convert_tokens_to_ids` methods.\n", + " | pair_ids (`List[int]`, *optional*):\n", + " | Tokenized input ids of the second sequence. Can be obtained from a string by chaining the `tokenize`\n", + " | and `convert_tokens_to_ids` methods.\n", + " | num_tokens_to_remove (`int`, *optional*, defaults to 0):\n", + " | Number of tokens to remove using the truncation strategy.\n", + " | truncation_strategy (`str` or [`~tokenization_utils_base.TruncationStrategy`], *optional*, defaults to `False`):\n", + " | The strategy to follow for truncation. Can be:\n", + " | \n", + " | - `'longest_first'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will truncate\n", + " | token by token, removing a token from the longest sequence in the pair if a pair of sequences (or a\n", + " | batch of pairs) is provided.\n", + " | - `'only_first'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the first sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `'only_second'`: Truncate to a maximum length specified with the argument `max_length` or to the\n", + " | maximum acceptable input length for the model if that argument is not provided. This will only\n", + " | truncate the second sequence of a pair if a pair of sequences (or a batch of pairs) is provided.\n", + " | - `'do_not_truncate'` (default): No truncation (i.e., can output batch with sequence lengths greater\n", + " | than the model maximum admissible input size).\n", + " | stride (`int`, *optional*, defaults to 0):\n", + " | If set to a positive number, the overflowing tokens returned will contain some tokens from the main\n", + " | sequence returned. The value of this argument defines the number of additional tokens.\n", + " | \n", + " | Returns:\n", + " | `Tuple[List[int], List[int], List[int]]`: The truncated `ids`, the truncated `pair_ids` and the list of\n", + " | overflowing tokens. Note: The *longest_first* strategy returns empty list of overflowing tokens if a pair\n", + " | of sequences (or a batch of pairs) is provided.\n", + " | \n", + " | ----------------------------------------------------------------------\n", + " | Class methods inherited from transformers.tokenization_utils_base.PreTrainedTokenizerBase:\n", + " | \n", + " | from_pretrained(pretrained_model_name_or_path: Union[str, os.PathLike], *init_inputs, **kwargs) from builtins.type\n", + " | Instantiate a [`~tokenization_utils_base.PreTrainedTokenizerBase`] (or a derived class) from a predefined\n", + " | tokenizer.\n", + " | \n", + " | Args:\n", + " | pretrained_model_name_or_path (`str` or `os.PathLike`):\n", + " | Can be either:\n", + " | \n", + " | - A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co.\n", + " | Valid model ids can be located at the root-level, like `bert-base-uncased`, or namespaced under a\n", + " | user or organization name, like `dbmdz/bert-base-german-cased`.\n", + " | - A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved\n", + " | using the [`~tokenization_utils_base.PreTrainedTokenizerBase.save_pretrained`] method, e.g.,\n", + " | `./my_model_directory/`.\n", + " | - (**Deprecated**, not applicable to all derived classes) A path or url to a single saved vocabulary\n", + " | file (if and only if the tokenizer only requires a single vocabulary file like Bert or XLNet), e.g.,\n", + " | `./my_model_directory/vocab.txt`.\n", + " | cache_dir (`str` or `os.PathLike`, *optional*):\n", + " | Path to a directory in which a downloaded predefined tokenizer vocabulary files should be cached if the\n", + " | standard cache should not be used.\n", + " | force_download (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to force the (re-)download the vocabulary files and override the cached versions if they\n", + " | exist.\n", + " | resume_download (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to delete incompletely received files. Attempt to resume the download if such a file\n", + " | exists.\n", + " | proxies (`Dict[str, str]`, *optional*):\n", + " | A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128',\n", + " | 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.\n", + " | use_auth_token (`str` or *bool*, *optional*):\n", + " | The token to use as HTTP bearer authorization for remote files. If `True`, will use the token generated\n", + " | when running `huggingface-cli login` (stored in `~/.huggingface`).\n", + " | local_files_only (`bool`, *optional*, defaults to `False`):\n", + " | Whether or not to only rely on local files and not to attempt to download any files.\n", + " | revision (`str`, *optional*, defaults to `\"main\"`):\n", + " | The specific model version to use. It can be a branch name, a tag name, or a commit id, since we use a\n", + " | git-based system for storing models and other artifacts on huggingface.co, so `revision` can be any\n", + " | identifier allowed by git.\n", + " | subfolder (`str`, *optional*):\n", + " | In case the relevant files are located inside a subfolder of the model repo on huggingface.co (e.g. for\n", + " | facebook/rag-token-base), specify it here.\n", + " | inputs (additional positional arguments, *optional*):\n", + " | Will be passed along to the Tokenizer `__init__` method.\n", + " | kwargs (additional keyword arguments, *optional*):\n", + " | Will be passed to the Tokenizer `__init__` method. Can be used to set special tokens like `bos_token`,\n", + " | `eos_token`, `unk_token`, `sep_token`, `pad_token`, `cls_token`, `mask_token`,\n", + " | `additional_special_tokens`. See parameters in the `__init__` for more details.\n", + " | \n", + " | \n", + " | \n", + " | Passing `use_auth_token=True` is required when you want to use a private model.\n", + " | \n", + " | \n", + " | \n", + " | Examples:\n", + " | \n", + " | ```python\n", + " | # We can't instantiate directly the base class *PreTrainedTokenizerBase* so let's show our examples on a derived class: BertTokenizer\n", + " | # Download vocabulary from huggingface.co and cache.\n", + " | tokenizer = BertTokenizer.from_pretrained(\"bert-base-uncased\")\n", + " | \n", + " | # Download vocabulary from huggingface.co (user-uploaded) and cache.\n", + " | tokenizer = BertTokenizer.from_pretrained(\"dbmdz/bert-base-german-cased\")\n", + " | \n", + " | # If vocabulary files are in a directory (e.g. tokenizer was saved using *save_pretrained('./test/saved_model/')*)\n", + " | tokenizer = BertTokenizer.from_pretrained(\"./test/saved_model/\")\n", + " | \n", + " | # If the tokenizer uses a single vocabulary file, you can point directly to this file\n", + " | tokenizer = BertTokenizer.from_pretrained(\"./test/saved_model/my_vocab.txt\")\n", + " | \n", + " | # You can link tokens to special vocabulary when instantiating\n", + " | tokenizer = BertTokenizer.from_pretrained(\"bert-base-uncased\", unk_token=\"\")\n", + " | # You should be sure '' is in the vocabulary when doing that.\n", + " | # Otherwise use tokenizer.add_special_tokens({'unk_token': ''}) instead)\n", + " | assert tokenizer.unk_token == \"\"\n", + " | ```\n", + " | \n", + " | register_for_auto_class(auto_class='AutoTokenizer') from builtins.type\n", + " | Register this class with a given auto class. This should only be used for custom tokenizers as the ones in the\n", + " | library are already mapped with `AutoTokenizer`.\n", + " | \n", + " | \n", + " | \n", + " | This API is experimental and may have some slight breaking changes in the next releases.\n", + " | \n", + " | \n", + " | \n", + " | Args:\n", + " | auto_class (`str` or `type`, *optional*, defaults to `\"AutoTokenizer\"`):\n", + " | The auto class to register this new tokenizer with.\n", + " | \n", + " | ----------------------------------------------------------------------\n", + " | Static methods inherited from transformers.tokenization_utils_base.PreTrainedTokenizerBase:\n", + " | \n", + " | clean_up_tokenization(out_string: str) -> str\n", + " | Clean up a list of simple English tokenization artifacts like spaces before punctuations and abbreviated forms.\n", + " | \n", + " | Args:\n", + " | out_string (`str`): The text to clean up.\n", + " | \n", + " | Returns:\n", + " | `str`: The cleaned-up string.\n", + " | \n", + " | ----------------------------------------------------------------------\n", + " | Data descriptors inherited from transformers.tokenization_utils_base.PreTrainedTokenizerBase:\n", + " | \n", + " | max_len_sentences_pair\n", + " | `int`: The maximum combined length of a pair of sentences that can be fed to the model.\n", + " | \n", + " | max_len_single_sentence\n", + " | `int`: The maximum length of a sentence that can be fed to the model.\n", + " | \n", + " | ----------------------------------------------------------------------\n", + " | Data and other attributes inherited from transformers.tokenization_utils_base.PreTrainedTokenizerBase:\n", + " | \n", + " | padding_side = 'right'\n", + " | \n", + " | truncation_side = 'right'\n", + " | \n", + " | ----------------------------------------------------------------------\n", + " | Methods inherited from transformers.tokenization_utils_base.SpecialTokensMixin:\n", + " | \n", + " | add_special_tokens(self, special_tokens_dict: Dict[str, Union[str, tokenizers.AddedToken]], replace_additional_special_tokens=True) -> int\n", + " | Add a dictionary of special tokens (eos, pad, cls, etc.) to the encoder and link them to class attributes. If\n", + " | special tokens are NOT in the vocabulary, they are added to it (indexed starting from the last index of the\n", + " | current vocabulary).\n", + " | \n", + " | Note,None When adding new tokens to the vocabulary, you should make sure to also resize the token embedding\n", + " | matrix of the model so that its embedding matrix matches the tokenizer.\n", + " | \n", + " | In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method.\n", + " | \n", + " | Using `add_special_tokens` will ensure your special tokens can be used in several ways:\n", + " | \n", + " | - Special tokens are carefully handled by the tokenizer (they are never split).\n", + " | - You can easily refer to special tokens using tokenizer class attributes like `tokenizer.cls_token`. This\n", + " | makes it easy to develop model-agnostic training and fine-tuning scripts.\n", + " | \n", + " | When possible, special tokens are already registered for provided pretrained models (for instance\n", + " | [`BertTokenizer`] `cls_token` is already registered to be :obj*'[CLS]'* and XLM's one is also registered to be\n", + " | `''`).\n", + " | \n", + " | Args:\n", + " | special_tokens_dict (dictionary *str* to *str* or `tokenizers.AddedToken`):\n", + " | Keys should be in the list of predefined special attributes: [`bos_token`, `eos_token`, `unk_token`,\n", + " | `sep_token`, `pad_token`, `cls_token`, `mask_token`, `additional_special_tokens`].\n", + " | \n", + " | Tokens are only added if they are not already in the vocabulary (tested by checking if the tokenizer\n", + " | assign the index of the `unk_token` to them).\n", + " | replace_additional_special_tokens (`bool`, *optional*,, defaults to `True`):\n", + " | If `True`, the existing list of additional special tokens will be replaced by the one specified in\n", + " | `special_tokens_dict`. Otherwise, `self._additional_special_tokens` is updated. In the former case, the\n", + " | tokens will NOT be removed from the tokenizer's full vocabulary - they are only being flagged as\n", + " | non-special tokens.\n", + " | \n", + " | Returns:\n", + " | `int`: Number of tokens added to the vocabulary.\n", + " | \n", + " | Examples:\n", + " | \n", + " | ```python\n", + " | # Let's see how to add a new classification token to GPT-2\n", + " | tokenizer = GPT2Tokenizer.from_pretrained(\"gpt2\")\n", + " | model = GPT2Model.from_pretrained(\"gpt2\")\n", + " | \n", + " | special_tokens_dict = {\"cls_token\": \"\"}\n", + " | \n", + " | num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)\n", + " | print(\"We have added\", num_added_toks, \"tokens\")\n", + " | # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer.\n", + " | model.resize_token_embeddings(len(tokenizer))\n", + " | \n", + " | assert tokenizer.cls_token == \"\"\n", + " | ```\n", + " | \n", + " | add_tokens(self, new_tokens: Union[str, tokenizers.AddedToken, List[Union[str, tokenizers.AddedToken]]], special_tokens: bool = False) -> int\n", + " | Add a list of new tokens to the tokenizer class. If the new tokens are not in the vocabulary, they are added to\n", + " | it with indices starting from length of the current vocabulary and and will be isolated before the tokenization\n", + " | algorithm is applied. Added tokens and tokens from the vocabulary of the tokenization algorithm are therefore\n", + " | not treated in the same way.\n", + " | \n", + " | Note, when adding new tokens to the vocabulary, you should make sure to also resize the token embedding matrix\n", + " | of the model so that its embedding matrix matches the tokenizer.\n", + " | \n", + " | In order to do that, please use the [`~PreTrainedModel.resize_token_embeddings`] method.\n", + " | \n", + " | Args:\n", + " | new_tokens (`str`, `tokenizers.AddedToken` or a list of *str* or `tokenizers.AddedToken`):\n", + " | Tokens are only added if they are not already in the vocabulary. `tokenizers.AddedToken` wraps a string\n", + " | token to let you personalize its behavior: whether this token should only match against a single word,\n", + " | whether this token should strip all potential whitespaces on the left side, whether this token should\n", + " | strip all potential whitespaces on the right side, etc.\n", + " | special_tokens (`bool`, *optional*, defaults to `False`):\n", + " | Can be used to specify if the token is a special token. This mostly change the normalization behavior\n", + " | (special tokens like CLS or [MASK] are usually not lower-cased for instance).\n", + " | \n", + " | See details for `tokenizers.AddedToken` in HuggingFace tokenizers library.\n", + " | \n", + " | Returns:\n", + " | `int`: Number of tokens added to the vocabulary.\n", + " | \n", + " | Examples:\n", + " | \n", + " | ```python\n", + " | # Let's see how to increase the vocabulary of Bert model and tokenizer\n", + " | tokenizer = BertTokenizerFast.from_pretrained(\"bert-base-uncased\")\n", + " | model = BertModel.from_pretrained(\"bert-base-uncased\")\n", + " | \n", + " | num_added_toks = tokenizer.add_tokens([\"new_tok1\", \"my_new-tok2\"])\n", + " | print(\"We have added\", num_added_toks, \"tokens\")\n", + " | # Notice: resize_token_embeddings expect to receive the full size of the new vocabulary, i.e., the length of the tokenizer.\n", + " | model.resize_token_embeddings(len(tokenizer))\n", + " | ```\n", + " | \n", + " | sanitize_special_tokens(self) -> int\n", + " | Make sure that all the special tokens attributes of the tokenizer (`tokenizer.mask_token`,\n", + " | `tokenizer.cls_token`, etc.) are in the vocabulary.\n", + " | \n", + " | Add the missing ones to the vocabulary if needed.\n", + " | \n", + " | Return:\n", + " | `int`: The number of tokens added in the vocabulary during the operation.\n", + " | \n", + " | ----------------------------------------------------------------------\n", + " | Readonly properties inherited from transformers.tokenization_utils_base.SpecialTokensMixin:\n", + " | \n", + " | all_special_ids\n", + " | `List[int]`: List the ids of the special tokens(`''`, `''`, etc.) mapped to class attributes.\n", + " | \n", + " | all_special_tokens\n", + " | `List[str]`: All the special tokens (`''`, `''`, etc.) mapped to class attributes.\n", + " | \n", + " | Convert tokens of `tokenizers.AddedToken` type to string.\n", + " | \n", + " | all_special_tokens_extended\n", + " | `List[Union[str, tokenizers.AddedToken]]`: All the special tokens (`''`, `''`, etc.) mapped to class\n", + " | attributes.\n", + " | \n", + " | Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how\n", + " | special tokens are tokenized.\n", + " | \n", + " | pad_token_type_id\n", + " | `int`: Id of the padding token type in the vocabulary.\n", + " | \n", + " | special_tokens_map\n", + " | `Dict[str, Union[str, List[str]]]`: A dictionary mapping special token class attributes (`cls_token`,\n", + " | `unk_token`, etc.) to their values (`''`, `''`, etc.).\n", + " | \n", + " | Convert potential tokens of `tokenizers.AddedToken` type to string.\n", + " | \n", + " | special_tokens_map_extended\n", + " | `Dict[str, Union[str, tokenizers.AddedToken, List[Union[str, tokenizers.AddedToken]]]]`: A dictionary mapping\n", + " | special token class attributes (`cls_token`, `unk_token`, etc.) to their values (`''`, `''`, etc.).\n", + " | \n", + " | Don't convert tokens of `tokenizers.AddedToken` type to string so they can be used to control more finely how\n", + " | special tokens are tokenized.\n", + " | \n", + " | ----------------------------------------------------------------------\n", + " | Data descriptors inherited from transformers.tokenization_utils_base.SpecialTokensMixin:\n", + " | \n", + " | __dict__\n", + " | dictionary for instance variables (if defined)\n", + " | \n", + " | __weakref__\n", + " | list of weak references to the object (if defined)\n", + " | \n", + " | additional_special_tokens\n", + " | `List[str]`: All the additional special tokens you may want to use. Log an error if used while not having been\n", + " | set.\n", + " | \n", + " | additional_special_tokens_ids\n", + " | `List[int]`: Ids of all the additional special tokens in the vocabulary. Log an error if used while not having\n", + " | been set.\n", + " | \n", + " | bos_token\n", + " | `str`: Beginning of sentence token. Log an error if used while not having been set.\n", + " | \n", + " | bos_token_id\n", + " | `Optional[int]`: Id of the beginning of sentence token in the vocabulary. Returns `None` if the token has not\n", + " | been set.\n", + " | \n", + " | cls_token\n", + " | `str`: Classification token, to extract a summary of an input sequence leveraging self-attention along the full\n", + " | depth of the model. Log an error if used while not having been set.\n", + " | \n", + " | cls_token_id\n", + " | `Optional[int]`: Id of the classification token in the vocabulary, to extract a summary of an input sequence\n", + " | leveraging self-attention along the full depth of the model.\n", + " | \n", + " | Returns `None` if the token has not been set.\n", + " | \n", + " | eos_token\n", + " | `str`: End of sentence token. Log an error if used while not having been set.\n", + " | \n", + " | eos_token_id\n", + " | `Optional[int]`: Id of the end of sentence token in the vocabulary. Returns `None` if the token has not been\n", + " | set.\n", + " | \n", + " | mask_token\n", + " | `str`: Mask token, to use when training a model with masked-language modeling. Log an error if used while not\n", + " | having been set.\n", + " | \n", + " | mask_token_id\n", + " | `Optional[int]`: Id of the mask token in the vocabulary, used when training a model with masked-language\n", + " | modeling. Returns `None` if the token has not been set.\n", + " | \n", + " | pad_token\n", + " | `str`: Padding token. Log an error if used while not having been set.\n", + " | \n", + " | pad_token_id\n", + " | `Optional[int]`: Id of the padding token in the vocabulary. Returns `None` if the token has not been set.\n", + " | \n", + " | sep_token\n", + " | `str`: Separation token, to separate context and query in an input sequence. Log an error if used while not\n", + " | having been set.\n", + " | \n", + " | sep_token_id\n", + " | `Optional[int]`: Id of the separation token in the vocabulary, to separate context and query in an input\n", + " | sequence. Returns `None` if the token has not been set.\n", + " | \n", + " | unk_token\n", + " | `str`: Unknown token. Log an error if used while not having been set.\n", + " | \n", + " | unk_token_id\n", + " | `Optional[int]`: Id of the unknown token in the vocabulary. Returns `None` if the token has not been set.\n", + " | \n", + " | ----------------------------------------------------------------------\n", + " | Data and other attributes inherited from transformers.tokenization_utils_base.SpecialTokensMixin:\n", + " | \n", + " | SPECIAL_TOKENS_ATTRIBUTES = ['bos_token', 'eos_token', 'unk_token', 's...\n", + "\n" + ] } ], "source": [ - "dataset['train']['verse_text'][:10]" + "tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-uncased\") # uncased means lowercase\n", + "# alternative tokenizers: distilbert-base-cased (cased means case sensitive), bert-base-uncased, bert-base-cased, roberta-base\n", + "\n", + "help(tokenizer)" ] }, { "cell_type": "code", - "execution_count": 41, + "execution_count": 12, "metadata": { "colab": { "base_uri": "https://localhost:8080/", @@ -493,122 +1929,65 @@ "id": "J5YkAosPTYIc", "outputId": "ef250595-fb02-40c4-8fad-d441d25bfb26" }, + "outputs": [], + "source": [ + "def tokenize_function(examples):\n", + " \"\"\"this function takes in a batch of training (or test) examples and for each, will tokenize the text and truncate or pad it to the max length of 512 tokens. \n", + "\n", + " Args:\n", + " examples (list): List of \n", + "\n", + " Returns:\n", + " List (int): List of [input IDs] with the appropriate special tokens. The input ids are often the only required parameters to be passed to the model as input. They are token indices, numerical representations of tokens building the sequences that will be used as input by the model. (https://huggingface.co/transformers/v3.2.0/glossary.html#:~:text=The%20input%20ids%20are%20often,as%20input%20by%20the%20model.&text=The%20tokenizer%20takes%20care%20of,available%20in%20the%20tokenizer%20vocabulary.)\n", + " \"\"\"\n", + " return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Discussion Question: given the description of the tokenize_function() above, before we proceed, can you think of any potential problems with this approach? How would you go about evaluating if this approach is appropriate for our use-case? (side exercise: ex2_inspect_dataset.ipynb)" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--distilbert-base-uncased/snapshots/1c4513b2eedbda136f57676a34eea67aba266e5c/config.json\n", - "Model config DistilBertConfig {\n", - " \"_name_or_path\": \"distilbert-base-uncased\",\n", - " \"activation\": \"gelu\",\n", - " \"architectures\": [\n", - " \"DistilBertForMaskedLM\"\n", - " ],\n", - " \"attention_dropout\": 0.1,\n", - " \"dim\": 768,\n", - " \"dropout\": 0.1,\n", - " \"hidden_dim\": 3072,\n", - " \"initializer_range\": 0.02,\n", - " \"max_position_embeddings\": 512,\n", - " \"model_type\": \"distilbert\",\n", - " \"n_heads\": 12,\n", - " \"n_layers\": 6,\n", - " \"pad_token_id\": 0,\n", - " \"qa_dropout\": 0.1,\n", - " \"seq_classif_dropout\": 0.2,\n", - " \"sinusoidal_pos_embds\": false,\n", - " \"tie_weights_\": true,\n", - " \"transformers_version\": \"4.26.1\",\n", - " \"vocab_size\": 30522\n", - "}\n", - "\n", - "loading file vocab.txt from cache at /root/.cache/huggingface/hub/models--distilbert-base-uncased/snapshots/1c4513b2eedbda136f57676a34eea67aba266e5c/vocab.txt\n", - "loading file tokenizer.json from cache at /root/.cache/huggingface/hub/models--distilbert-base-uncased/snapshots/1c4513b2eedbda136f57676a34eea67aba266e5c/tokenizer.json\n", - "loading file added_tokens.json from cache at None\n", - "loading file special_tokens_map.json from cache at None\n", - "loading file tokenizer_config.json from cache at /root/.cache/huggingface/hub/models--distilbert-base-uncased/snapshots/1c4513b2eedbda136f57676a34eea67aba266e5c/tokenizer_config.json\n", - "loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--distilbert-base-uncased/snapshots/1c4513b2eedbda136f57676a34eea67aba266e5c/config.json\n", - "Model config DistilBertConfig {\n", - " \"_name_or_path\": \"distilbert-base-uncased\",\n", - " \"activation\": \"gelu\",\n", - " \"architectures\": [\n", - " \"DistilBertForMaskedLM\"\n", - " ],\n", - " \"attention_dropout\": 0.1,\n", - " \"dim\": 768,\n", - " \"dropout\": 0.1,\n", - " \"hidden_dim\": 3072,\n", - " \"initializer_range\": 0.02,\n", - " \"max_position_embeddings\": 512,\n", - " \"model_type\": \"distilbert\",\n", - " \"n_heads\": 12,\n", - " \"n_layers\": 6,\n", - " \"pad_token_id\": 0,\n", - " \"qa_dropout\": 0.1,\n", - " \"seq_classif_dropout\": 0.2,\n", - " \"sinusoidal_pos_embds\": false,\n", - " \"tie_weights_\": true,\n", - " \"transformers_version\": \"4.26.1\",\n", - " \"vocab_size\": 30522\n", - "}\n", - "\n" + " \r" ] - }, - { - "data": { - "application/vnd.jupyter.widget-view+json": { - "model_id": "603fc9e2063643b2848f538efc3edde5", - "version_major": 2, - "version_minor": 0 - }, - "text/plain": [ - "Map: 0%| | 0/892 [00:00 1\u001b[0m tokenized_datasets\n", + "\u001b[0;31mNameError\u001b[0m: name 'tokenized_datasets' is not defined" + ] } ], "source": [ - "from transformers import AutoTokenizer\n", - "\n", - "tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-uncased\")\n", - "\n", - "\n", - "def tokenize_function(examples):\n", - " return tokenizer(examples[\"verse_text\"], padding=\"max_length\", truncation=True)\n", - "\n", - "\n", - "tokenized_datasets = dataset.map(tokenize_function, batched=True)" + "tokenized_datasets" ] }, { @@ -704,9 +2083,7 @@ } ], "source": [ - "from transformers import AutoModelForSequenceClassification\n", - "\n", - "model = DistilBertForSequenceClassification.from_pretrained(\"distilbert-base-uncased\", num_labels=5)\n" + "model = DistilBertForSequenceClassification.from_pretrained(\"distilbert-base-uncased\", num_labels=5)" ] }, { @@ -730,7 +2107,7 @@ } ], "source": [ - "from transformers import TrainingArguments\n", + "\n", "\n", "training_args = TrainingArguments(output_dir=\"test_trainer\")" ] @@ -743,8 +2120,7 @@ }, "outputs": [], "source": [ - "import numpy as np\n", - "import evaluate\n", + "\n", "\n", "metric = evaluate.load(\"accuracy\")" ] @@ -784,7 +2160,7 @@ } ], "source": [ - "from transformers import TrainingArguments, Trainer\n", + "\n", "\n", "training_args = TrainingArguments(output_dir=\"test_trainer\", evaluation_strategy=\"epoch\")" ] @@ -1139,7 +2515,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.8.16" + "version": "3.9.16" }, "widgets": { "application/vnd.jupyter.widget-state+json": { diff --git a/notebooks/Inspect_DistilBERT_Vocabulary.ipynb b/notebooks/Inspect_DistilBERT_Vocabulary.ipynb new file mode 100644 index 0000000..a8545a0 --- /dev/null +++ b/notebooks/Inspect_DistilBERT_Vocabulary.ipynb @@ -0,0 +1,1258 @@ +{ + "cells": [ + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This notebook is copied from Chris McCormick's blog post: https://mccormickml.com/2019/05/14/BERT-word-embeddings-tutorial/ and applied to Distilbert so we can compare the vocabulary size and the embeddings of distilbert to BERT." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "whjogPl1KL4-" + }, + "source": [ + "## Load the Model\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "O0wWu4oEKYtc" + }, + "source": [ + "Install the huggingface implementation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 423 + }, + "id": "PzCjfNB6jksJ", + "outputId": "9aa79438-898e-4d43-8487-b0684dfe2623" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting pytorch-pretrained-bert\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d7/e0/c08d5553b89973d9a240605b9c12404bcf8227590de62bae27acbcfe076b/pytorch_pretrained_bert-0.6.2-py3-none-any.whl (123kB)\n", + "\r\u001b[K |██▋ | 10kB 18.2MB/s eta 0:00:01\r\u001b[K |█████▎ | 20kB 6.7MB/s eta 0:00:01\r\u001b[K |████████ | 30kB 9.4MB/s eta 0:00:01\r\u001b[K |██████████▋ | 40kB 5.7MB/s eta 0:00:01\r\u001b[K |█████████████▎ | 51kB 6.9MB/s eta 0:00:01\r\u001b[K |███████████████▉ | 61kB 8.1MB/s eta 0:00:01\r\u001b[K |██████████████████▌ | 71kB 9.2MB/s eta 0:00:01\r\u001b[K |█████████████████████▏ | 81kB 10.3MB/s eta 0:00:01\r\u001b[K |███████████████████████▉ | 92kB 11.4MB/s eta 0:00:01\r\u001b[K |██████████████████████████▌ | 102kB 9.1MB/s eta 0:00:01\r\u001b[K |█████████████████████████████▏ | 112kB 9.1MB/s eta 0:00:01\r\u001b[K |███████████████████████████████▊| 122kB 9.1MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 133kB 9.1MB/s \n", + "\u001b[?25hRequirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (1.3.1)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (2.21.0)\n", + "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (4.28.1)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (1.17.4)\n", + "Collecting regex\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/e3/8e/cbf2295643d7265e7883326fb4654e643bfc93b3a8a8274d8010a39d8804/regex-2019.11.1-cp36-cp36m-manylinux1_x86_64.whl (643kB)\n", + "\u001b[K |████████████████████████████████| 645kB 31.3MB/s \n", + "\u001b[?25hRequirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (1.10.18)\n", + "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert) (1.24.3)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert) (2019.9.11)\n", + "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert) (2.8)\n", + "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert) (3.0.4)\n", + "Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert) (0.2.1)\n", + "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert) (0.9.4)\n", + "Requirement already satisfied: botocore<1.14.0,>=1.13.18 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert) (1.13.18)\n", + "Requirement already satisfied: python-dateutil<2.8.1,>=2.1; python_version >= \"2.7\" in /usr/local/lib/python3.6/dist-packages (from botocore<1.14.0,>=1.13.18->boto3->pytorch-pretrained-bert) (2.6.1)\n", + "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.14.0,>=1.13.18->boto3->pytorch-pretrained-bert) (0.15.2)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil<2.8.1,>=2.1; python_version >= \"2.7\"->botocore<1.14.0,>=1.13.18->boto3->pytorch-pretrained-bert) (1.12.0)\n", + "Installing collected packages: regex, pytorch-pretrained-bert\n", + "Successfully installed pytorch-pretrained-bert-0.6.2 regex-2019.11.1\n" + ] + } + ], + "source": [ + "# !pip install pytorch-pretrained-bert" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "nwpkz_WYjscB", + "outputId": "6d4b4bf5-a2db-4126-a9b7-3e873b60fe98" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 231508/231508 [00:00<00:00, 1020506.43B/s]\n" + ] + } + ], + "source": [ + "import torch\n", + "from pytorch_pretrained_bert import BertTokenizer\n", + "\n", + "# Load pre-trained model tokenizer (vocabulary)\n", + "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "B8FD0tDpKVBF" + }, + "source": [ + "## Inspect BERT Vocabulary\n", + "--------------------------" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7UB5FRPOhinf" + }, + "source": [ + "### Vocab Dump\n", + "--------------\n", + "\n", + "Retrieve the entire list of \"tokens\" and write these out to text files so we can peruse them." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "id": "NRf7NXJGgsnG" + }, + "outputs": [], + "source": [ + "with open(\"vocabulary.txt\", 'w') as f:\n", + " \n", + " # For each token...\n", + " for token in tokenizer.vocab.keys():\n", + " \n", + " # Write it out and escape any unicode characters. \n", + " f.write(token + '\\n')\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c8geTQ3Iidsv" + }, + "source": [ + "From perusing the vocab, I'm seeing that:\n", + "\n", + "* The first 999 tokens (1-indexed) appear to be reserved, and most are of the form [unused957].\n", + " * 1 - [PAD]\n", + " * 101 - [UNK]\n", + " * 102 - [CLS]\n", + " * 103 - [SEP]\n", + " * 104 - [MASK]\n", + "* Rows 1000-1996 appear to be a dump of individual characters. \n", + " * They don't appear to be sorted by frequency (e.g., the letters of the alphabet are all in sequence).\n", + "* The first word is \"the\" at position 1997.\n", + " * From there, the words appear to be sorted by frequency. \n", + " * The top ~18 words are whole words, and then number 2016 is ##s, presumably the most common subword.\n", + " * The last whole word is at 29612, \"necessitated\"\n", + "\n", + "Some funny inclusions:\n", + "* starbucks\n", + "* triassic\n", + "* abolitionist\n", + "* 1679" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "uviYdpTZYMmd" + }, + "source": [ + "### Single Characters\n", + "---------------------" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Bu-KbpbR1L4q" + }, + "source": [ + "The following code prints out all of the single character tokens in vocabulary, as well as all of the single-character tokens preceded by '##'.\n", + "\n", + "It turns out that these are matching sets--for every standalone character there is also a '##' version. There are 997 single character tokens." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Q6ysF8LD1fbl" + }, + "source": [ + "The following cell iterates over the vocabulary, pulling out all of the single character tokens." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "YHpH2NEPy3t9" + }, + "outputs": [], + "source": [ + "one_chars = []\n", + "one_chars_hashes = []\n", + "\n", + "# For each token in the vocabulary...\n", + "for token in tokenizer.vocab.keys():\n", + " \n", + " # Record any single-character tokens.\n", + " if len(token) == 1:\n", + " one_chars.append(token)\n", + " \n", + " # Record single-character tokens preceded by the two hashes. \n", + " elif len(token) == 3 and token[0:2] == '##':\n", + " one_chars_hashes.append(token)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 470 + }, + "id": "jCzfsjHrzZYn", + "outputId": "dfc169d2-d61c-4311-f89a-184b4b23e294" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of single character tokens: 997 \n", + "\n", + "! \" # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ [ \\ ] ^ _ ` a b\n", + "c d e f g h i j k l m n o p q r s t u v w x y z { | } ~ ¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬\n", + "® ° ± ² ³ ´ µ ¶ · ¹ º » ¼ ½ ¾ ¿ × ß æ ð ÷ ø þ đ ħ ı ł ŋ œ ƒ ɐ ɑ ɒ ɔ ɕ ə ɛ ɡ ɣ ɨ\n", + "ɪ ɫ ɬ ɯ ɲ ɴ ɹ ɾ ʀ ʁ ʂ ʃ ʉ ʊ ʋ ʌ ʎ ʐ ʑ ʒ ʔ ʰ ʲ ʳ ʷ ʸ ʻ ʼ ʾ ʿ ˈ ː ˡ ˢ ˣ ˤ α β γ δ\n", + "ε ζ η θ ι κ λ μ ν ξ ο π ρ ς σ τ υ φ χ ψ ω а б в г д е ж з и к л м н о п р с т у\n", + "ф х ц ч ш щ ъ ы ь э ю я ђ є і ј љ њ ћ ӏ ա բ գ դ ե թ ի լ կ հ մ յ ն ո պ ս վ տ ր ւ\n", + "ք ־ א ב ג ד ה ו ז ח ט י ך כ ל ם מ ן נ ס ע ף פ ץ צ ק ר ש ת ، ء ا ب ة ت ث ج ح خ د\n", + "ذ ر ز س ش ص ض ط ظ ع غ ـ ف ق ك ل م ن ه و ى ي ٹ پ چ ک گ ں ھ ہ ی ے अ आ उ ए क ख ग च\n", + "ज ट ड ण त थ द ध न प ब भ म य र ल व श ष स ह ा ि ी ो । ॥ ং অ আ ই উ এ ও ক খ গ চ ছ জ\n", + "ট ড ণ ত থ দ ধ ন প ব ভ ম য র ল শ ষ স হ া ি ী ে க ச ட த ந ன ப ம ய ர ல ள வ ா ி ு ே\n", + "ை ನ ರ ಾ ක ය ර ල ව ා ก ง ต ท น พ ม ย ร ล ว ส อ า เ ་ ། ག ང ད ན པ བ མ འ ར ལ ས မ ა\n", + "ბ გ დ ე ვ თ ი კ ლ მ ნ ო რ ს ტ უ ᄀ ᄂ ᄃ ᄅ ᄆ ᄇ ᄉ ᄊ ᄋ ᄌ ᄎ ᄏ ᄐ ᄑ ᄒ ᅡ ᅢ ᅥ ᅦ ᅧ ᅩ ᅪ ᅭ ᅮ\n", + "ᅯ ᅲ ᅳ ᅴ ᅵ ᆨ ᆫ ᆯ ᆷ ᆸ ᆼ ᴬ ᴮ ᴰ ᴵ ᴺ ᵀ ᵃ ᵇ ᵈ ᵉ ᵍ ᵏ ᵐ ᵒ ᵖ ᵗ ᵘ ᵢ ᵣ ᵤ ᵥ ᶜ ᶠ ‐ ‑ ‒ – — ―\n", + "‖ ‘ ’ ‚ “ ” „ † ‡ • … ‰ ′ ″ › ‿ ⁄ ⁰ ⁱ ⁴ ⁵ ⁶ ⁷ ⁸ ⁹ ⁺ ⁻ ⁿ ₀ ₁ ₂ ₃ ₄ ₅ ₆ ₇ ₈ ₉ ₊ ₍\n", + "₎ ₐ ₑ ₒ ₓ ₕ ₖ ₗ ₘ ₙ ₚ ₛ ₜ ₤ ₩ € ₱ ₹ ℓ № ℝ ™ ⅓ ⅔ ← ↑ → ↓ ↔ ↦ ⇄ ⇌ ⇒ ∂ ∅ ∆ ∇ ∈ − ∗\n", + "∘ √ ∞ ∧ ∨ ∩ ∪ ≈ ≡ ≤ ≥ ⊂ ⊆ ⊕ ⊗ ⋅ ─ │ ■ ▪ ● ★ ☆ ☉ ♠ ♣ ♥ ♦ ♭ ♯ ⟨ ⟩ ⱼ ⺩ ⺼ ⽥ 、 。 〈 〉\n", + "《 》 「 」 『 』 〜 あ い う え お か き く け こ さ し す せ そ た ち っ つ て と な に ぬ ね の は ひ ふ へ ほ ま み\n", + "む め も や ゆ よ ら り る れ ろ を ん ァ ア ィ イ ウ ェ エ オ カ キ ク ケ コ サ シ ス セ タ チ ッ ツ テ ト ナ ニ ノ ハ\n", + "ヒ フ ヘ ホ マ ミ ム メ モ ャ ュ ョ ラ リ ル レ ロ ワ ン ・ ー 一 三 上 下 不 世 中 主 久 之 也 事 二 五 井 京 人 亻 仁\n", + "介 代 仮 伊 会 佐 侍 保 信 健 元 光 八 公 内 出 分 前 劉 力 加 勝 北 区 十 千 南 博 原 口 古 史 司 合 吉 同 名 和 囗 四\n", + "国 國 土 地 坂 城 堂 場 士 夏 外 大 天 太 夫 奈 女 子 学 宀 宇 安 宗 定 宣 宮 家 宿 寺 將 小 尚 山 岡 島 崎 川 州 巿 帝\n", + "平 年 幸 广 弘 張 彳 後 御 德 心 忄 志 忠 愛 成 我 戦 戸 手 扌 政 文 新 方 日 明 星 春 昭 智 曲 書 月 有 朝 木 本 李 村\n", + "東 松 林 森 楊 樹 橋 歌 止 正 武 比 氏 民 水 氵 氷 永 江 沢 河 治 法 海 清 漢 瀬 火 版 犬 王 生 田 男 疒 発 白 的 皇 目\n", + "相 省 真 石 示 社 神 福 禾 秀 秋 空 立 章 竹 糹 美 義 耳 良 艹 花 英 華 葉 藤 行 街 西 見 訁 語 谷 貝 貴 車 軍 辶 道 郎\n", + "郡 部 都 里 野 金 鈴 镇 長 門 間 阝 阿 陳 陽 雄 青 面 風 食 香 馬 高 龍 龸 fi fl ！ （ ） ， － ． ／ ： ？ ～\n" + ] + } + ], + "source": [ + "print('Number of single character tokens:', len(one_chars), '\\n')\n", + "\n", + "# Print all of the single characters, 40 per row.\n", + "\n", + "# For every batch of 40 tokens...\n", + "for i in range(0, len(one_chars), 40):\n", + " \n", + " # Limit the end index so we don't go past the end of the list.\n", + " end = min(i + 40, len(one_chars) + 1)\n", + " \n", + " # Print out the tokens, separated by a space.\n", + " print(' '.join(one_chars[i:end]))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 470 + }, + "id": "62MD-M1N0R-W", + "outputId": "3e5a3866-b969-4eb2-8984-8580c7b51ab9" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of single character tokens with hashes: 997 \n", + "\n", + "s a e i n o d r y t l m u h k c g p 2 z 1 b 3 f 4 6 7 x v 8 5 9 0 w j q ° ₂ а и\n", + "² ₃ ı ₁ ⁺ ½ о ه ي α е د ن ν ø р ₄ ₀ ر я ³ ι ł н ᵢ ₙ ß ة ς م − т ː ل ь к ♭ η ی в\n", + "ا × ¹ ы ה ɛ л ! \" # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \\ ] ^ _ ` { | } ~ ¡\n", + "¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ® ± ´ µ ¶ · º » ¼ ¾ ¿ æ ð ÷ þ đ ħ ŋ œ ƒ ɐ ɑ ɒ ɔ ɕ ə ɡ ɣ ɨ\n", + "ɪ ɫ ɬ ɯ ɲ ɴ ɹ ɾ ʀ ʁ ʂ ʃ ʉ ʊ ʋ ʌ ʎ ʐ ʑ ʒ ʔ ʰ ʲ ʳ ʷ ʸ ʻ ʼ ʾ ʿ ˈ ˡ ˢ ˣ ˤ β γ δ ε ζ\n", + "θ κ λ μ ξ ο π ρ σ τ υ φ χ ψ ω б г д ж з м п с у ф х ц ч ш щ ъ э ю ђ є і ј љ њ ћ\n", + "ӏ ա բ գ դ ե թ ի լ կ հ մ յ ն ո պ ս վ տ ր ւ ք ־ א ב ג ד ו ז ח ט י ך כ ל ם מ ן נ ס\n", + "ע ף פ ץ צ ק ר ש ת ، ء ب ت ث ج ح خ ذ ز س ش ص ض ط ظ ع غ ـ ف ق ك و ى ٹ پ چ ک گ ں ھ\n", + "ہ ے अ आ उ ए क ख ग च ज ट ड ण त थ द ध न प ब भ म य र ल व श ष स ह ा ि ी ो । ॥ ং অ আ\n", + "ই উ এ ও ক খ গ চ ছ জ ট ড ণ ত থ দ ধ ন প ব ভ ম য র ল শ ষ স হ া ি ী ে க ச ட த ந ன ப\n", + "ம ய ர ல ள வ ா ி ு ே ை ನ ರ ಾ ක ය ර ල ව ා ก ง ต ท น พ ม ย ร ล ว ส อ า เ ་ ། ག ང ད\n", + "ན པ བ མ འ ར ལ ས မ ა ბ გ დ ე ვ თ ი კ ლ მ ნ ო რ ს ტ უ ᄀ ᄂ ᄃ ᄅ ᄆ ᄇ ᄉ ᄊ ᄋ ᄌ ᄎ ᄏ ᄐ ᄑ\n", + "ᄒ ᅡ ᅢ ᅥ ᅦ ᅧ ᅩ ᅪ ᅭ ᅮ ᅯ ᅲ ᅳ ᅴ ᅵ ᆨ ᆫ ᆯ ᆷ ᆸ ᆼ ᴬ ᴮ ᴰ ᴵ ᴺ ᵀ ᵃ ᵇ ᵈ ᵉ ᵍ ᵏ ᵐ ᵒ ᵖ ᵗ ᵘ ᵣ ᵤ\n", + "ᵥ ᶜ ᶠ ‐ ‑ ‒ – — ― ‖ ‘ ’ ‚ “ ” „ † ‡ • … ‰ ′ ″ › ‿ ⁄ ⁰ ⁱ ⁴ ⁵ ⁶ ⁷ ⁸ ⁹ ⁻ ⁿ ₅ ₆ ₇ ₈\n", + "₉ ₊ ₍ ₎ ₐ ₑ ₒ ₓ ₕ ₖ ₗ ₘ ₚ ₛ ₜ ₤ ₩ € ₱ ₹ ℓ № ℝ ™ ⅓ ⅔ ← ↑ → ↓ ↔ ↦ ⇄ ⇌ ⇒ ∂ ∅ ∆ ∇ ∈\n", + "∗ ∘ √ ∞ ∧ ∨ ∩ ∪ ≈ ≡ ≤ ≥ ⊂ ⊆ ⊕ ⊗ ⋅ ─ │ ■ ▪ ● ★ ☆ ☉ ♠ ♣ ♥ ♦ ♯ ⟨ ⟩ ⱼ ⺩ ⺼ ⽥ 、 。 〈 〉\n", + "《 》 「 」 『 』 〜 あ い う え お か き く け こ さ し す せ そ た ち っ つ て と な に ぬ ね の は ひ ふ へ ほ ま み\n", + "む め も や ゆ よ ら り る れ ろ を ん ァ ア ィ イ ウ ェ エ オ カ キ ク ケ コ サ シ ス セ タ チ ッ ツ テ ト ナ ニ ノ ハ\n", + "ヒ フ ヘ ホ マ ミ ム メ モ ャ ュ ョ ラ リ ル レ ロ ワ ン ・ ー 一 三 上 下 不 世 中 主 久 之 也 事 二 五 井 京 人 亻 仁\n", + "介 代 仮 伊 会 佐 侍 保 信 健 元 光 八 公 内 出 分 前 劉 力 加 勝 北 区 十 千 南 博 原 口 古 史 司 合 吉 同 名 和 囗 四\n", + "国 國 土 地 坂 城 堂 場 士 夏 外 大 天 太 夫 奈 女 子 学 宀 宇 安 宗 定 宣 宮 家 宿 寺 將 小 尚 山 岡 島 崎 川 州 巿 帝\n", + "平 年 幸 广 弘 張 彳 後 御 德 心 忄 志 忠 愛 成 我 戦 戸 手 扌 政 文 新 方 日 明 星 春 昭 智 曲 書 月 有 朝 木 本 李 村\n", + "東 松 林 森 楊 樹 橋 歌 止 正 武 比 氏 民 水 氵 氷 永 江 沢 河 治 法 海 清 漢 瀬 火 版 犬 王 生 田 男 疒 発 白 的 皇 目\n", + "相 省 真 石 示 社 神 福 禾 秀 秋 空 立 章 竹 糹 美 義 耳 良 艹 花 英 華 葉 藤 行 街 西 見 訁 語 谷 貝 貴 車 軍 辶 道 郎\n", + "郡 部 都 里 野 金 鈴 镇 長 門 間 阝 阿 陳 陽 雄 青 面 風 食 香 馬 高 龍 龸 fi fl ！ （ ） ， － ． ／ ： ？ ～\n" + ] + } + ], + "source": [ + "print('Number of single character tokens with hashes:', len(one_chars_hashes), '\\n')\n", + "\n", + "# Print all of the single characters, 40 per row.\n", + "\n", + "# Strip the hash marks, since they just clutter the display.\n", + "tokens = [token.replace('##', '') for token in one_chars_hashes]\n", + "\n", + "# For every batch of 40 tokens...\n", + "for i in range(0, len(tokens), 40):\n", + " \n", + " # Limit the end index so we don't go past the end of the list.\n", + " end = min(i + 40, len(tokens) + 1)\n", + " \n", + " # Print out the tokens, separated by a space.\n", + " print(' '.join(tokens[i:end]))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "u-TFrVP9005S", + "outputId": "7e234716-83f8-424a-e0a1-b23d426ef7ca" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Are the two sets identical? True\n" + ] + } + ], + "source": [ + "print('Are the two sets identical?', set(one_chars) == set(tokens))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "uPnW2RetYc97" + }, + "source": [ + "### Subwords vs. Whole-words\n", + "Let's gather some statistics on the vocabulary.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 389 + }, + "id": "LjvrWcNTdkAW", + "outputId": "111debcc-8600-415e-9a38-9e73da02db8f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Maximum token length: 18\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoUAAAFjCAYAAABL3HHWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeViUVf8G8HuAAUSQRQdcUDGLRRQQXInXcklxRwW3QFFDM3PNBV+1en9qlCKpoZiklGhqJoiGG6ZvZoG7kjmZoLmkyLgCIswAz+8PL57XEbAZ5WHUuT/X1XU155w53zMT4e1znkUmCIIAIiIiIjJqJoZeABEREREZHkMhERERETEUEhERERFDIRERERGBoZCIiIiIwFBIRERERGAoJKKXSElJCdzc3DBnzhxDL6VS7733Htq0aWPoZbzUlEol3Nzc8PXXXxt6KUQvHIZCIqrUpEmT4ObmBqVSWeUYQRDQpUsXtGnTBkVFRTW4Omls2bIFbm5uOv0THh5u6OVWqzZt2mDYsGGGXoZOVCoVvvjiC5w6dcrQSyF6qZgZegFE9HwKDg7Gnj17sHXrVsydO7fSMRkZGfj7778xZMgQWFpa1vAKq1/79u2xaNEirbaVK1fi8uXL+PTTT7XaHR0da3Jp9IibN28iNjYWNjY28PHxMfRyiF4aDIVEVKmAgAA0aNAAO3bswMyZM2Fubl5hTFJSEoCHAfJl0KRJEzRp0kSrbdOmTbhy5Qr69+9voFUREdUMbh8TUaVMTEwwYMAA3L17F/v376/QX1BQgL1798LV1RVeXl5afZs2bUJQUBC8vLzQpk0bjBkzBidOnKi0Tnp6Ot555x20b98erVq1QteuXTF37lzcvXtXHLN+/XqEh4cjICAALVu2REBAAGbOnIlr165Vuf6ff/4ZISEh8PLyQkBAAD755BM8ePDgKb+NJ8vMzMTYsWPRtm1btGrVCn379sX69euhy1NECwoKMHLkSHh7e+PHH38U20tKSpCQkIB+/frBy8sLvr6+GDNmDE6ePKn1/kfPodu5cyf69++PVq1aoVOnTli+fDnKysqq/fPu378fYWFh8PX1hZeXFwYMGIBt27ZVGNemTRu89957UCqVCA8PR+vWrdG2bVtMnz4d9+7dqzA+MzMToaGh8Pb2RocOHfDhhx/i77//hpubGxYuXAgA2LdvH4KCggAAUVFR4nZ+ZaFdl+/j999/x/jx47V+tkaPHo2MjIzq+KqIXig8UkhEVRo4cCDi4uKQlJSEwMBArb7U1FQUFRVh0KBBWu2ffvopEhIS4OPjg2nTpqGgoACbNm3CiBEjsGrVKgQEBIhjN2zYgPnz56NBgwYYOnQoGjZsiGvXrmH//v3Izc2FnZ0dAGDNmjXw9fWFv78/7OzscO7cOWzduhUZGRnYsWMHbG1ttdbw22+/YefOnRg8eDCCgoKQkZGBb775BllZWVizZg1kMlm1fUeHDx/GmDFjYG1tjdDQUNjZ2SEtLQ3z589HdnY2Pvrooyrfm5ubi4iICOTk5ODrr79G69atAQBlZWV4//33cfDgQfTu3RtDhgzBgwcPkJSUhLCwMHz11Vfo0KGD1lw7d+7E9evXMWTIENjb22P37t1YsWIF6tati7fffrvaPu9XX32FxYsXo3379pg4cSLMzc1x4MABzJo1C9evX8f48eO1xl+6dAmjRo1Cnz59EBgYiNOnTyMpKQkajQbLli0TxymVSoSFhcHCwgKjRo1C3bp1kZaWhokTJ2rN17JlS0ydOhWff/45+vXrJ/48lf+s6PN95OTkYOTIkahduzaGDRsGJycn3L59G6dPn8aZM2cqfMdELz2BiOgJRowYIXh4eAg3btzQah88eLDg6ekp3Lp1S2w7f/684OrqKrz99tuCWq0W269fvy60bt1a6Nq1q1BaWioIgiBcvXpV8PT0FPr06SPk5+dXqFs+ThAE4f79+xX6Dx48KLi6ugpr164V2zQajeDq6iq4uroK+/fv1xr/8ccfC66ursKuXbv0+vxDhw4VPDw8quzv1auX0KpVK+HSpUtiW0lJiTB69GjB1dVVyMzMFNvHjx8v+Pn5CYIgCFlZWULnzp2Fzp07C1lZWVpzfv/994Krq6uQmpqq1f7gwQOhR48eQr9+/cS2s2fPCq6urkLbtm0FlUoltms0GqFz585CYGCgTp/Tz89PGDp06BPHXLx4UfDw8BA++uijCn0zZ84UPD09tdbg5+cnuLq6Cj///LPW2GnTpglubm5aP1OjRo0S3N3dhT/++ENsKy0tFb/HBQsWVPjMCQkJFdahz/eRnJxc6fqIjBW3j4noiYKDg1FaWqq1PZidnY1Tp06hS5cucHBwENv37dsHAIiIiIBcLhfb69evjwEDBuDKlSs4d+4cAGDXrl3QaDR4//33YW1tXaGuicn/fj1ZWVkBeHgELT8/H7dv34anpyesrKxw+vTpCu997bXX0LlzZ622sWPHAgDS0tL0/g6qcvHiRWRlZaF3795a5yKamppi3LhxAP73nTzqxIkTGD58OKytrbFx40Y0b95cq3/79u1QKBTo0KEDbt++Lf5TWFiIN954A3/88Qdu3bql9Z7evXujXr164mszMzP4+fnhypUr1baFnJqaitLSUgwcOFBrXbdv30aXLl2g0Whw5MgRrfe8+uqrWkeHAaBDhw4QBAGXL18GABQWFiIjIwP+/v5wc3MTx5mYmGDUqFFPtVZdvo/yn7v//ve/KCwsfKo6RC8Tbh8T0RN1794dderUQVJSkhistm7dCgAVto6vXr0K4GEQeFx525UrV+Dh4YFLly4BAFq0aPGPa/jll18QFxeH06dPQ61Wa/Xl5eVVGP/KK69UaGvQoAGsrKzENVaH8rlee+21Cn3ln/fxeg8ePMCoUaPg5OSEb7/9ttJAnJ2dDZVKhY4dO1ZZ++bNm6hbt674unHjxhXG2NnZQaPR4P79+7CxsdHtQz3BhQsXAAAhISFVjnk8rFa2Lnt7ewAQzxu9fv06SktL0axZswpjK2vThS7fR+fOndGtWzckJiZi8+bN8PLywuuvv47evXujadOmT1WX6EXGUEhET2RhYYE+ffrg22+/xYkTJ+Dt7Y3t27ejfv36+Ne//iV5/ZMnTyIiIgIuLi6YPn06nJ2dUatWLQDA5MmTJbmQQkoWFhbo3LkzfvjhB3z33XcYPXp0peOcnZ0xf/78KudxdnbWev3okdXHCTpc8KKL8nliY2NRu3btSsc8HuKetC4p6fJ9mJqaYsWKFTh79ix+/vlnHD16FKtWrcLKlSuxYMEC8YIWImPBUEhE/yg4OBjffvstkpKScO/ePahUKrz77rsV/uAtPzqTlZWFRo0aafVlZ2drjXFxcQHw8AKDyo7qlPvhhx9QWlqKr776Cg0bNhTbCwoKkJ+fX+l7yo9oPer69esoLCysEKaeRfm6z58/X6Gv/PNWFt4WL14MMzMzfPbZZ9BoNOJWc7mmTZvi7NmzaNu2rdY2vKGVHz2rV6+eeFFMdWjQoAFMTU1x8eLFCn2VtVXnhULAw6PVLVq0wLhx43Dr1i0MHDgQS5YsYSgko8NzConoH3l6esLDwwM7d+7Ehg0bIJPJKr03YdeuXQE8vEK1pKREbL9x4waSk5PRuHFj8ZyxwMBAyOVyfPHFFygoKKgwV/nRnKqO+MTFxVV5BOz8+fM4cOCAVlt8fDwAoFu3bv/0cXXm4uKCV199FTt37sSVK1fE9rKyMqxevRoA8NZbb1V4n4mJCaKiojBo0CDExMQgNjZWqz8oKAiFhYVYsWJFpXVv3rxZbZ9BH3379oWpqSmWLl1aYRsfeLgd/Oh/d11ZWVmhffv2+PXXX8VzToGH32NCQkKl4wFUelsbfdy5c6fCz1DdunXh5OT0zHMTvYh4pJCIdBIcHIz58+fj559/Rrt27So9uvfqq69i1KhRSEhIQGhoKHr27Cnekqa4uBgfffSRGPIaNWqEWbNmYcGCBejXrx/69++Phg0bIicnB/v27cPixYvh6uqK7t27IzExEWPGjMGQIUNgamqKQ4cOITs7u8KtaMq5ublh2rRpGDJkCBo3boyMjAzs3bsXHTt2rHBrnWf14YcfimsbNmwYbG1tkZaWhiNHjmD48OFo2bJlpe8zMTHBwoULxWBcWlqKyZMnA3h4rubPP/+MuLg4nDhxAgEBAbCzs8P169dx9OhRFBQUVHpfwGeVk5ODlStXVto3bNgwvPLKK5g1axaioqLQt29f9OnTB/Xr18fNmzehVCpx4MAB/PLLL6hTp47etadPn47hw4cjLCwMb7/9NhwcHLBv3z7xLwyPHh10dnZGvXr1sHXrVtSrVw92dnaoU6eO3qczrF+/Htu3b0fXrl3RuHFjyGQy/Prrrzh9+jSGDBmi92cgetExFBKRTvr27YtFixahuLi4wgUmj4qMjISLiws2btyI6OhoyOVy+Pj4YMKECfDz89MaGxYWhqZNm2Lt2rVYt24dNBoNHB0d4e/vDycnJwBA27ZtsWzZMqxatQqff/45atWqhddffx3r16/H4MGDK11Dq1atMGPGDCxbtgzffvstbGxsEBYWhmnTplX71mP79u2xYcMGxMbG4uuvv4ZarYaLiwvmzp2L0NDQJ75XJpPhP//5D+RyOVauXAmNRoPp06fDxMQES5cuxZYtW5CUlIS4uDiUlpZCoVCgVatWT31F7j+5du2a1r0DH9WjRw/Y29tj5MiRcHV1xddff43169fj/v37cHBwQPPmzTFjxgzxKJ6+PD098fXXXyM6Ohpr1qyBlZUV3nrrLYwYMQJ9+vTReoyiiYkJYmJiEB0djc8++wzFxcVwd3fXOxT+61//wl9//YW0tDTcvHkTcrkcTZo0wbx58zB06NCn+hxELzKZUF1nIBMREVWz9PR0hIeH4+OPP8awYcMMvRyilxrPKSQioudCUVGR1uvS0lLxCTT+/v4GWhWR8eD2MRERPRfeeustdO/eHa+++iry8/Oxd+9e/PbbbxgyZAjvG0hUA7h9TEREz4WPP/4Y6enpuHHjBkpLS9G0aVMMGDAAo0aNMtj9DomMCUMhEREREfGcQiIiIiJiKCQiIiIi8EKTanPnzn2UlXEnnoiIiJ5fJiYy2NtX/uxyhsJqUlYmMBQSERHRC4vbx0RERETEUEhEREREDIVEREREBIZCIiIiIgJDIRERERGBoZCIiIiIwFBIRERERGAoJCIiIiIwFBIRERERDPhEk8OHD2PEiBGV9u3cuRPNmzcXX584cQKLFy/G2bNnYW1tjZ49e+KDDz5ArVq1tN6nVquxbNkypKSkIC8vD+7u7pg6dSo6duxYoYaucxI9r2zt5DCXW0paQ60pwr27GklrEBHR88Hgj7kbOXIkPD09tdqcnJzEf1cqlQgPD8err76KyMhI5OTkYO3atbh69SpWrVql9b7IyEjs3bsXI0aMQNOmTZGcnIyIiAgkJiaidevWTzUn0fPKXG6JJRt7SFrjg2F7ADAUEhEZA4OHwnbt2qFbt25V9sfExMDOzg6JiYmoXfvhA5ydnZ0xd+5cpKeni0cBMzMzkZqaitmzZyM8PBwAEBQUhD59+iA6OhobNmzQe04iIiIiY/FcnFNYUFCAkpKSStt//fVXBAUFieENAPr37w8rKyvs2rVLbNu9ezfkcjlCQkLENgsLCwQHB+P48ePIzc3Ve04iIiIiY2HwUDhjxgz4+fnB29sbo0ePxrlz58S+c+fOoaSkBC1bttR6j7m5OTw8PKBUKsU2pVKJZs2aaQU9APDy8oIgCOJYfeYkIiIiMhYG2z6Wy+Xo0aMHOnXqBHt7e5w7dw5r167F8OHD8f3336NZs2ZQqVQAAIVCUeH9CoUCp06dEl+rVCqtcxEfHQdAPFKoz5z6qFvX+qneR/S8UyhsDL0EIiKqAQYLhb6+vvD19RVfd+3aFV26dMGgQYMQGxuLJUuWoKioCMDDo3iPs7CwEPsBoKioCHK5vNJxAFBcXCyO03VOfdy6VYCyMuGp3kv0NGoqrKlU+TVSh4iIpGdiIqvyQJbBt48f5e7ujo4dOyIjIwMAYGn58HYbarW6wtji4mKxv3ysRlPxKsnyMFgeDvWZk4iIiMhYPFehEAAaNGiAe/fuAfjfFm/5lu+jVCoVHB0dxdcKhULcIn58HABxrD5zEhERERmL5y4UXrlyBfb29gAAV1dXmJmZ4cyZM1pj1Go1lEolPDw8xDZ3d3dcvHgR9+/f1xp7+vRpsV/fOYmIiIiMhcFC4e3btyu0HTt2DIcPH0ZAQAAAwMbGBh07dkRKSopW2EtJSUFhYSECAwPFtsDAQGg0GmzZskVsU6vVSEpKgq+vr3gRij5zEhERERkLg11oMmXKFNSqVQutW7eGvb09zp8/j82bN8Pe3h4TJ04Ux02dOhVDhw5FWFgYQkJCkJOTg4SEBHTq1An+/v7iOG9vbwQGBiI6OhoqlQpNmjRBcnIyrl27hqioKK3aus5JREREZCxkgiAY5JLZdevWYceOHbh8+TIKCgrg4OCAgIAATJw4EQ0bNtQae+zYMURHR4vPKe7VqxemTZsGKysrrXHFxcVYunQpduzYgXv37sHNzQ3Tpk2rNOjpOqeuePUx1TSFwqZGHnPHq4+JiF4eT7r62GCh8GXDUEg1jaGQiIj09cLckoaIiIiIDIOhkIiIiIgYComIiIiIoZCIiIiIwFBIRERERGAoJCIiIiIwFBIRERERGAqJiIiICAyFRERERASGQiIiIiICQyERERERgaGQiIiIiMBQSERERERgKCQiIiIiMBQSERERERgKiYiIiAgMhUREREQEhkIiIiIiAkMhEREREQEwM/QCiF5kdnZyyOWWktbQaIpw965G0hpEREQMhUTPQC63ROLXPSStERa+BwBDIRERSYvbx0RERETEUEhEREREDIVEREREBIZCIiIiIgJDIRERERGBoZCIiIiIwFBIRERERGAoJCIiIiIwFBIRERER+EQTInoKdezMYSG3kLRGsaYYeXfVktYgIqL/YSgkIr1ZyC0wKjlQ0hoJA3YDYCgkIqop3D4mIiIiIoZCIiIiImIoJCIiIiIwFBIRERERGAqJiIiICM9ZKIyPj4ebmxv69+9foe/EiRMYNmwYvL298frrr2PBggV48OBBhXFqtRqLFy9GQEAAvLy8MHjwYKSnp1daT9c5iYiIiF52z00oVKlUiIuLg5WVVYU+pVKJ8PBwFBcXIzIyEsHBwdi8eTOmTp1aYWxkZCS++eYb9OvXD3PmzIGJiQkiIiJw8uTJp56TiIiI6GX33NyncMmSJWjZsiUEQUBeXp5WX0xMDOzs7JCYmIjatWsDAJydnTF37lykp6ejY8eOAIDMzEykpqZi9uzZCA8PBwAEBQWhT58+iI6OxoYNG/Sek4iIiMgYPBdHCjMzM7F9+3bMnj27Ql9BQQF+/fVXBAUFieENAPr37w8rKyvs2rVLbNu9ezfkcjlCQkLENgsLCwQHB+P48ePIzc3Ve04iIiIiY2DwUCgIAubPn4+goCB4eHhU6D937hxKSkrQsmVLrXZzc3N4eHhAqVSKbUqlEs2aNdMKegDg5eUFQRDEsfrMSURERGQMDL59vG3bNmRlZWHFihWV9qtUKgCAQqGo0KdQKHDq1CmtsU5OTpWOAyAeKdRnTl3VrWut93uIdKVQ2LA2ERFJyqChsKCgAEuWLMHYsWPh6OhY6ZiioiIAD4/iPc7CwkLsLx8rl8srHQcAxcXFes+pq1u3ClBWJuj9Pnqx1VRoUanyWZuIiJ6ZiYmsygNZBt0+jouLg1wux6hRo6ocY2lpCeDhrWYeV1xcLPaXj9VoNJWOA/4XDvWZk4iIiMgYGOxIYW5uLr755htMnjwZN2/eFNuLi4uh0Whw9epV2NjYiFu85Vu+j1KpVFpHGBUKhbhF/Pg4AOJYfeYkIiIiMgYGO1J469YtaDQaREdHo2vXruI/p0+fRnZ2Nrp27Yr4+Hi4urrCzMwMZ86c0Xq/Wq2GUqnUujjF3d0dFy9exP3797XGnj59WuwHoNecRERERMbAYEcKnZ2dK724ZOnSpSgsLMS///1vuLi4wMbGBh07dkRKSgrGjRsnXlmckpKCwsJCBAYGiu8NDAzE2rVrsWXLFvE+hWq1GklJSfD19RUvQtFnTiIiIiJjYLBQaGNjg27dulVo/+abb2BqaqrVN3XqVAwdOhRhYWEICQlBTk4OEhIS0KlTJ/j7+4vjvL29ERgYiOjoaKhUKjRp0gTJycm4du0aoqKitOroOicRERGRMTD4fQp14enpiYSEBJibmyMqKgpbtmzB4MGDsWzZsgpjFy1ahLCwMKSkpGDBggUoKSnB6tWr4efn99RzEhEREb3sDH6fwsclJiZW2t6mTRts2rTpH99vYWGBWbNmYdasWf84Vtc5iYiIiF52L8SRQiIiIiKSFkMhERERETEUEhERERFDIRERERGBoZCIiIiIwFBIRERERKiGUHjhwgWcPHkSpaWl1bEeIiIiIjIAnUPh+vXrMXnyZK22efPmoXfv3hg+fDj69++PO3fuVPsCiYiIiEh6OofCbdu2wdbWVnx97NgxbNmyBV26dMHYsWNx+fJlrFmzRpJFEhEREZG0dH6iydWrV9GvXz/x9d69e1G3bl0sX74cpqamKCwsxL59+zB9+nRJFkpERERE0tH5SGFhYSGsrKzE1xkZGejYsSNMTU0BAG5ubrhx40b1r5CIiIiIJKdzKFQoFMjOzgYA3LhxA+fPn0fbtm3F/vz8fJiZPXePUiYiIiIiHeic4jp16oRvv/0WpqamOHHiBExNTfHmm2+K/VlZWWjQoIEUayQiIiIiiekcCidOnIgzZ87gq6++gqmpKWbMmAEnJycAgFqtRlpamtY5h0RERET04tA5FDo4OGDLli3Izc2FlZUVrK2txb6SkhLExcWhadOmkiySiIiIiKSl90mAjo6OFdqsrKzg5+dXLQsiIiIiopqndyi8ceMGLl++jLt370IQhAr93bt3r5aFEREREVHN0TkU3r17Fx9++CHS0tIAQCsQymQyCIIAmUwGpVJZ/askIiIiIknpHArnz5+PvXv3IigoCB06dICdnZ2U6yIiIiKiGqRzKDx48CCCg4OxYMECKddDRERERAag882rS0pK4OXlJeVaiIiIiMhAdA6Fvr6+OH/+vJRrISIiIiID0TkUzpw5E6mpqfj555+lXA8RERERGYDO5xR+/vnnsLe3x9ixY+Hi4oLGjRvDxEQ7U8pkMsTFxVX7IomIiIhIWjqHwpMnT0Imk8HW1hZ37tzBnTt3KoyRyWTVujgiIiIiqhk6h8LDhw9LuQ4iIiIiMiCdzykkIiIiopeX3o+5u3nzJg4fPoybN2+iR48eqF+/PkpKSlBQUAAbGxuYmppKsU4iIiIikpBeoXDFihVYtWoVNBoNZDIZXF1dUb9+fdy/fx9vvPEGZsyYgdDQUKnWSkQEGzsLWMrNJa1RpFEj/26xpDWIiJ43OofCpKQkfPHFFxg0aBA6d+6M999/X+yztbVF586dsX//foZCIpKUpdwcvbbNk7TGzqD5yAdDIREZF53PKUxMTMSbb76JhQsXws/Pr0K/h4cHsrOzq3VxRERERFQzdA6F2dnZ6NSpU5X9Dg4OuH37drUsioiIiIhqls6h0NzcHGq1usr+nJwcWFtbV8uiiIiIiKhm6RwKPT09sX///kr71Go1duzYAR8fn2pbGBERERHVHJ1D4ahRo3DkyBF89NFHuHLlCgCgoKAAx48fxzvvvIOrV69i9OjRki2UiIiIiKSj89XHb775JiIjIxEdHY3vvvsOADBp0iQAgImJCebMmYO2bdtKs0oiIiIikpRe9ykMDw9Ht27dkJqaigsXLkAQBDRt2hR9+vRB06ZN9Sr822+/YdWqVTh79ixu3boFGxsbuLu7Y8KECfD19dUae+LECSxevBhnz56FtbU1evbsiQ8++AC1atXSGqdWq7Fs2TKkpKQgLy8P7u7umDp1Kjp27Fihvq5zEhERERkDvZ9o4uzsjHHjxlXa9+DBA51D1ZUrV1BaWoqQkBAoFArk5+djx44dCA0NRXx8PF5//XUAgFKpRHh4OF599VVERkYiJycHa9euxdWrV7Fq1SqtOSMjI7F3716MGDECTZs2RXJyMiIiIpCYmIjWrVuL4/SZk4iIiMgY6BwKY2JiMG3atCr7i4qKMG7cOKxbt06n+Xr16oVevXpptQ0bNgzdunXDunXrxFAYExMDOzs7JCYmonbt2gAeBtO5c+ciPT1dPAqYmZmJ1NRUzJ49G+Hh4QCAoKAg9OnTB9HR0diwYYPWZ9FlTiIiIiJjofOFJqtXr9YKVo9Sq9UYP348Tp48+UyLqVWrFhwcHJCXlwfg4YUsv/76K4KCgsTwBgD9+/eHlZUVdu3aJbbt3r0bcrkcISEhYpuFhQWCg4Nx/Phx5Obm6j0nERERkbHQORSOHTsWn3zyCdLS0rTaywPhkSNHsGTJEr0XUFBQgNu3b+PChQuIiYnBn3/+KR6pO3fuHEpKStCyZUut95ibm8PDwwNKpVJsUyqVaNasmVbQAwAvLy8IgiCO1WdOIiIiImOh8/bxtGnTcOPGDUyfPh1r1qxBmzZtoNFoMGnSJGRkZGDx4sXo3r273gv497//jT179gAA5HI5hg4dinfffRcAoFKpAAAKhaLC+xQKBU6dOiW+VqlUcHJyqnQcAPFIoT5zEhERERkLvS40WbhwIW7evIn33nsP33zzDWJjY3Hw4EF88sknFc4P1NWECRMwZMgQ5OTkICUlBWq1GhqNBubm5igqKgLw8Cje4ywsLMR+4OE5jXK5vNJxAFBcXCyO03VOfdSty6e5GEppiRqmZhX/e75oNZ5EobBhbSOqTURkCHqFQjMzM3zxxRcIDQ1FSEgIysrKMH/+fAQFBT31Atzc3ODm5gYA6NevHwYNGoTZs2dj+fLlsLS0BIBKH69XXFws9gOApaUlNBpNpeOA/4VDfebUx61bBSgrE57qvfRsFAob7FnzdH8p0VWPMTuhUuVXWrsmsPbzUZuI6EVnYiKr8kBWlaHw999/r3LCyb08oI8AACAASURBVJMnY9q0aejduzfc3d21xnp6ej71QuVyObp27Yq4uDgUFRWJW7zlW76PUqlUcHR0FF8rFApxi/jxcQDEsfrMSURERGQsqgyFgwYNgkwmq/KNgiBgy5Yt2LJli1b7s16oUVRUBEEQcP/+fbi6usLMzAxnzpzROl9RrVZDqVSib9++Ypu7uzsSExNx//59rYtNTp8+LfYD0GtOIiIiImNRZSicO3fuE0Phs7p9+zYcHBy02goKCrBnzx40aNAAdevWBQB07NgRKSkpGDdunBj2UlJSUFhYiMDAQPG9gYGBWLt2LbZs2SLep1CtViMpKQm+vr7iRSg2NjY6z0lERERkLKoMhaGhoZIWnjJlCiwsLNC6dWsoFApcv34dSUlJyMnJQUxMjDhu6tSpGDp0KMLCwhASEoKcnBwkJCSgU6dO8Pf3F8d5e3sjMDAQ0dHRUKlUaNKkCZKTk3Ht2jVERUVp1dZ1TiIiIiJjofdj7qpLv379kJKSgsTEROTl5cHGxgY+Pj5YtGgR2rVrJ47z9PREQkICoqOjERUVBWtrawwePLjSp6ssWrQIS5cuRUpKCu7duwc3NzesXr0afn5+WuP0mZOIiIjIGOgVCtVqNdatW4d9+/bhypUrAIDGjRvjrbfeQlhYWKW3ealKcHAwgoODdRrbpk0bbNq06R/HWVhYYNasWZg1a1a1zUlERERkDHQOhQUFBQgLC4NSqYSVlRWaNGkCADh//jxOnTqF1NRUrFu3DtbWvF8fERER0YtG58fcffHFF1Aqlfjggw+QkZGBbdu2Ydu2bUhPT8f06dOhVCoRGxsr5VqJiIiISCI6h8K0tDQMGDAAERERWtvE5ubmeOeddxAUFIS9e/dKskgiIiIikpbOoTA3Nxfe3t5V9nt5eVV6Q2giIiIiev7pHAodHBzw559/Vtl//vx52NnZVcuiiIiIiKhm6RwK33jjDXz33XfYvn17hb4dO3bgu+++Q+fOnat1cURERERUM3S++njy5Mk4dOgQZs2ahWXLlqF58+YAgOzsbFy7dg0NGjTApEmTJFsoEREREUlH5yOF9erVQ1JSEt5++22UlZXh4MGDOHjwIARBQGhoKLZu3Yp69epJuVYiIiIikoheN6+2t7fH3LlzMXfuXJSWlgIATE1NJVkYEREREdWcJx4p9PDwwI4dOyrtMzU1ZSAkIiIiekk8MRQKglBT6yAiIiIiA9L5nEIiIiIienkxFBIRERHRP19ocuHCBRw9elTnCdu2bftMCyIiIiKimvePoXDVqlVYtWqVzhMqlcpnWhARERER1bx/DIXdunWDm5tbTayFiIiIiAzkH0Nh9+7d0bdv35pYCxEREREZCC80ISIiIiKGQiIiIiJiKCQiIiIi/MM5hT/++CMcHBxqai1EREREZCBPDIWNGjWqqXUQERERkQH949XHRET0kI2dJSzlcklrFGk0yL9bJGkNIqLKMBQSEenIUi5H76SVktZIHfge8sFQSEQ1j6GQqoW9rTnMzC0krVGiLsade2pJaxARERmrKkNhbGwsunfvDldXVwDAtWvX4ODgAEtLyxpbHL04zMwtkBnXT9IaXuO3A2AoJCIikkKVt6SJjY3FuXPnxNddu3ZFWlpajSyKiIiIiGpWlaGwTp06yMvLE18LglAjCyIiIiKimlfl9rGHhwfWrFmDkpIS2NraAgCOHTuG0tLSJ04YFBRUvSskIiIiIslVGQpnz56N999/H1FRUQAAmUyGzZs3Y/PmzVVOJpPJGAqJiIiIXkBVhkJ3d3fs2bMHV65cgUqlQlhYGN599134+/vX5PqIiIiIqAY88ZY0pqamcHFxgYuLC9q2bYv27dujXbt2NbU2IiIiIqohOt+nMDExUcp1EBEREZEB6XXz6rKyMiQnJyMtLQ1Xr14FADg7O6N79+4ICgqCiUmVFzMTERER0XNM51BYVFSEiIgIHDt2DDKZDAqFAgBw8OBB/PTTT9i2bRvi4+NhYSHtUy2IiIiIqPrpfGgvLi4OR48exahRo5Ceno6ffvoJP/30EzIyMjB69GgcOXIEcXFxUq6ViIiIiCSicyjcuXMnevbsiZkzZ4r3LQQe3uR6xowZ6NmzJ1JTUyVZJBERERFJS+dQmJOT88Qrj9u2bYucnBydC2dmZuI///kPevXqBR8fH7z55puYOnUqLl26VGHsiRMnMGzYMHh7e+P111/HggUL8ODBgwrj1Go1Fi9ejICAAHh5eWHw4MFIT0+vtL6ucxIREREZA51DYZ06dXD58uUq+y9fvow6deroXPirr75CWloa/P39MWfOHAwePBhHjhxBUFAQsrOzxXFKpRLh4eEoLi5GZGQkgoODsXnzZkydOrXCnJGRkfjmm2/Qr18/zJkzByYmJoiIiMDJkye1xukzJxEREZEx0PlCE39/f2zYsAH+/v7417/+pdV36NAhbNy4EYGBgToXDg8PR3R0NMzNzcW2Xr16oW/fvoiPj8enn34KAIiJiYGdnR0SExNRu3ZtAA+veJ47dy7S09PRsWNHAA+PPKampmL27NkIDw8H8PCRe3369EF0dDQ2bNgg1tF1TiIiIiJjofORwilTpqB27doYO3YsBg4ciFmzZmHWrFkYOHAgIiIiULt2bUyaNEnnwr6+vlqBEABcXFzw2muviUcKCwoK8OuvvyIoKEgMbwDQv39/WFlZYdeuXWLb7t27IZfLERISIrZZWFggODgYx48fR25urt5zEhERERkLnUNho0aNsHXrVvTq1Qt//fUXUlJSkJKSgkuXLqF37974/vvv0ahRo2dajCAIuHnzJuzt7QEA586dQ0lJCVq2bKk1ztzcHB4eHlAqlWKbUqlEs2bNtIIeAHh5eUEQBHGsPnMSERERGQu9bl7dsGFDLFmyBIIg4Pbt2wAABwcHyGSyalnM9u3bcePGDfHcPpVKBQDiPREfpVAocOrUKfG1SqWCk5NTpeMAiEcK9ZmTiIiIyFjoFQrLyWQy1K1bt1oXkp2djf/7v/+Dn58f+vfvD+DhDbMBVNhmBh5uDZf3l4+Vy+WVjgOA4uJivefUR9261k/1PtKPQmHD2qzN2kREEniqUFjdVCoVxo0bB1tbWyxbtkx8XJ6lpSWAh7eaeVxxcbHYXz5Wo9FUOg74XzjUZ0593LpVgLIy4ane+zKoqT/EVKp81mZto6xNRFQdTExkVR7IMngozM/PR0REBPLz87Fx40atbd3yfy/f8n2USqWCo6Oj1tjyLeLHxwEQx+ozJxEREZGx0PlCEykUFxfj3XffxV9//YUvv/wSr7zyila/q6srzMzMcObMGa12tVoNpVIJDw8Psc3d3R0XL17E/fv3tcaePn1a7Nd3TiIiIiJjYbBQWFpaiilTpuDUqVNYtmwZfHx8KoyxsbFBx44dkZKSohX2UlJSUFhYqHVfxMDAQGg0GmzZskVsU6vVSEpKgq+vr3gRij5zEhERERkLg20ff/rpp9i/fz86d+6Mu3fvIiUlReyrXbs2unXrBgCYOnUqhg4dirCwMISEhCAnJwcJCQno1KkT/P39xfd4e3sjMDAQ0dHRUKlUaNKkCZKTk3Ht2jVERUVp1dZ1TiIiIiJjYbBQ+McffwAADhw4gAMHDmj1NWrUSAyFnp6eSEhIQHR0NKKiomBtbY3Bgwdj2rRpFeZctGgRli5dipSUFNy7dw9ubm5YvXo1/Pz8tMbpMycRERGRMdA5FBYUFOC9995DZGQkWrRo8cyFExMTdR7bpk0bbNq06R/HWVhYiE9aqa45iYiIiIyBzucUajQaHDlyBPfu3QMAFBYWYvbs2eIj6YiIiIjoxfXEUDhp0iR8/fXXOH36dIX7+hUXF2Pbtm2V3gaGiIiIiF4sT9w+fvDgAVasWIH8/HyYmZlBJpNh165dsLKygrOzMwTBeG/WTERERPQyeWIojI+PhyAIOHfuHH755RcsXrwYO3bswHfffQcrKyvIZDL897//ha2tLTw8PKrtGchERPT8sLGrBUu5tNclFmlKkH/3gaQ1iOjJ/vH/cplMBnd3dzg5OWHx4sVYuXIlHBwcsH//fixbtgwbNmzAunXrYG1tDV9fX3z55Zc1sW4iIqohlnIz9Pv+B0lrbA/uAz7cj8iwnhgKx4wZAz8/P/j5+aFx48YAHoZENzc3KBQKLFu2DF9++SXq1KmDo0eP4tixYzWyaCIiIiKqXk8Mhebm5khMTMTy5cthamoKmUyG5ORkABAfSWdqaopWrVqhVatWGD16tPQrJiIiIqJq98RQGBcXBwD466+/8Msvv2D+/Pk4cOAAUlJSYGFhAZlMhr1798LS0hItW7aEmZnB7oVNRERERM9Ap/sUuri4oFevXgCAZcuWYdeuXZgwYQIEQUBycjKGDh2Ktm3bIjw8XMq1EhEREZFEdL559aOaNWuGkJAQAMDKlSuRmpqKGTNmwMHBoVoXR0REREQ1Q+f9XgsLCwwYMACOjo4V+po3b47mzZtj+PDh1bo4IiIiIqoZOodCKysrREVFia+fFBKJiIiI6MXy1FeGPB4SiYhIOjZ2lrCUyyWtUaTRIP9ukaQ1iOj5xcuFiYheAJZyOfp8v0HSGj8Ev418MBQSGaunutCEiIiIiF4uDIVERERExFBIRERERAyFRERERASGQiIiIiICQyERERERgaGQiIiIiMBQSERERERgKCQiIiIiMBQSERERERgKiYiIiAgMhUREREQEhkIiIiIiAkMhEREREYGhkIiIiIjAUEhEREREYCgkIiIiIjAUEhEREREYComIiIgIDIVEREREBIZCIiIiIgJDIRERERGBoZCIiIiIYOBQmJubi+joaISFhaF169Zwc3PD4cOHKx37448/YsCAAWjVqhXefPNNxMbGoqSkpMK4vLw8zJs3Dx06dICPjw9GjBgBpVL5THMSERERvewMGgovXryI+Ph43LhxA25ublWO++mnnzBhwgTY2tpi3rx56NatG1asWIGoqCitcWVlZRg7dixSU1MRGhqKGTNm4NatWwgLC8Ply5efak4iIiIiY2BmyOKenp7IyMiAvb099u3bhwkTJlQ6btGiRWjRogXWrFkDU1NTAEDt2rWxevVqhIWFwcXFBQCwe/dunDx5EitWrEC3bt0AAD179kSPHj0QGxuLRYsW6T0nERERkTEw6JFCa2tr2NvbP3FMVlYWsrKyMGTIEDG8AcDw4cNRVlaGvXv3im179uyBo6MjunbtKrY5ODigZ8+e2LdvHzQajd5zEhERERmD5/5Ck7NnzwIAWrZsqdXu5OSE+vXri/0AoFQq4enpCZlMpjW2VatWuH//vriFrM+cRERERMbAoNvHulCpVAAAhUJRoU+hUCA3N1drbIcOHSqMc3R0BPDwwpbmzZvrNaeu6ta11vs9pD+Fwoa1WZu1WZuIJPDch8KioiIAgLm5eYU+CwsLPHjwQGtsZePK28rn0mdOXd26VYCyMgEOtpYwNZfr/X59lKo1uH2vSNIa+qqpX+YqVT5rszZrG1FtIqpeJiayKg9kPfeh0NLSEgCgVqsr9BUXF4v95WMrG1feVj5Wnzn1ZWouhypu/VO/XxeK8aEAnq9QSERERC+25/6cwvIt3vIt30epVCpxa7h8bGVbv+Vt5WP1mZOIiIjIGDz3odDDwwMAcObMGa32GzduICcnR+wHAHd3d/z+++8QBEFrbGZmJqysrNCkSRO95yQiIiIyBs99KHzttdfwyiuvYPPmzSgtLRXbN27cCBMTE3Tv3l1sCwwMRG5uLn788Uex7fbt29i9eze6du0KuVyu95xERERExsDg5xSuXLkSAJCdnQ0ASElJwfHjx1GnTh2EhoYCAGbOnInx48djzJgx6NWrF/78809s2LABQ4YMQbNmzcS5evToAR8fH8ycOROjR4+Gvb09Nm7ciLKyMkycOFGrrq5zEhERERkDg4fCZcuWab3eunUrAKBRo0ZiKOzcuTNiY2MRGxuL+fPnw8HBAePHj8d7772n9V5TU1OsXr0aixYtQmJiIoqLi9GqVSt89tlnaNq0qdZYXeckIiIiMgYGD4Xnzp3TaVy3bt3ER9c9ia2tLRYuXIiFCxdW25xEREREL7vn/pxCIiIiIpIeQyERERERMRQSEREREUMhEREREYGhkIiIiIjAUEhEREREYCgkIiIiIjAUEhEREREYComIiIgIDIVEREREBIZCIiIiIgJDIRERERGBoZCIiIiIwFBIRERERGAoJCIiIiIwFBIRERERGAqJiIiICICZoRdA1cfB1gKm5uaS1ihVq3H7XrGkNYiIytnYWcFSbippjSJNKfLvFkpag+hFwFD4EjE1N0fOyo8krVH/vf8AYCgkopphKTfFwK0ZktZIGtQB+ZJWIHoxcPuYiIiIiBgKiYiIiIihkIiIiIjAUEhEREREYCgkIiIiIjAUEhEREREYComIiIgIDIVEREREBIZCIiIiIgJDIRERERGBoZCIiIiIwGcfExERVaqOnRUs5KaS1ijWlCLvbqGkNYh0xVBIRERUCQu5KSYlX5G0xvIBjSWdn0gf3D4mIiIiIoZCIiIiImIoJCIiIiIwFBIRERERGAqJiIiICEYeCtVqNRYvXoyAgAB4eXlh8ODBSE9PN/SyiIiIiGqcUd+SJjIyEnv37sWIESPQtGlTJCcnIyIiAomJiWjdurWhl0dEREbKzq425HJpj9toNGW4e/e+pDXoxWK0oTAzMxOpqamYPXs2wsPDAQBBQUHo06cPoqOjsWHDBsMukIiIjJZcboKNW1WS1hg2SCHp/PTiMdrt4927d0MulyMkJERss7CwQHBwMI4fP47c3FwDro6IiIioZhntkUKlUolmzZqhdu3aWu1eXl4QBAFKpRKOjo46z2diIvvfv9vUfsLI6vFovUeZ2tgZrLbcRvfvq7prW1obrnZtayeD1a5T23C161oZrrajleF+zh2tbAxY23C/WxytahmstsLKwmC1Haykfczdk2rXtpL+uE1ltW3rWMFM4q3rEk0Z7uXx8X6GUNXPGwDIBEEQanAtz40+ffrAyckJa9as0WrPyspC7969sWDBAq2jiEREREQvM6PdPi4qKoJcLq/QbmHx8G+kxcXFNb0kIiIiIoMx2lBoaWkJjUZTob08DJaHQyIiIiJjYLShUKFQVHoxiUr18Govfc4nJCIiInrRGW0odHd3x8WLF3H/vvY9mk6fPi32ExERERkLow2FgYGB0Gg02LJli9imVquRlJQEX19fODlJf2UlERER0fPCaG9J4+3tjcDAQERHR0OlUqFJkyZITk7GtWvXEBUVZejlEREREdUoo70lDfDwopKlS5dix44duHfvHtzc3DBt2jT4+/sbemlERERENcqoQyERERERPWS05xQSERER0f8wFBIRERGR8V5oYki5ublYt24dTp8+jTNnzqCwsBDr1q1D+/btJa2bmZmJ5ORkHD58GNeuXYOdnR1at26NKVOmoGnTppLW/u2337Bq1SqcPXsWt27dgo2NDdzd3TFhwgT4+vpKWrsy8fHxiI6Ohru7O1JSUiSrc/jwYYwYMaLSvp07d6J58+aS1S6XmZmJ2NhYnDx5EiUlJWjcuDHCw8MxcOBAyWpGRkYiOTm5yv6DBw9KeoX/X3/9haVLl+LEiRPIy8tDw4YNERQUhPDwcJibm0tWFwBOnTqFzz//HJmZmTAxMUH79u0RGRmJJk2aVGsdfX6P/Pjjj4iNjUVWVhbq1q2L4OBgvPvuuzAze7o/AnStvXHjRmRkZCAzMxPXrl3DgAED8Omnnz5VTX1q37lzB1u3bsX+/ftx4cIFlJSUoHnz5ggPD0fPnj0lrS0IAj766COcPHkS169fR2lpKRo3bozg4GAMGzas0idpVVftx/3999/o1asXioqKsG3bNnh4eEhau0uXLvj7778rvD8iIgLTp0+XtDYA5OfnY8WKFdizZw9UKhXq1q0LPz8/xMTESFb7Sb/jAWDKlCkYP368JLWBh9dGJCQkICUlRfxzvU2bNnj//ffRrFkzvesyFBrAxYsXER8fj6ZNm8LNzQ0nT56skbpfffUVTpw4gcDAQLi5uUGlUmHDhg0ICgrC999/L2lAuXLlCkpLSxESEgKFQoH8/Hzs2LEDoaGhiI+Px+uvvy5Z7cepVCrExcXBysqqxmqOHDkSnp6eWm01cdujn376CRMmTEC7du0wefJkmJmZ4a+//sL169clrTtkyBB07NhRq00QBHz88cdo1KiRpJ/9xo0bCAkJgY2NDUJDQ2Fra4tjx45hyZIlOH/+PBYvXixZ7czMTISGhqJRo0aYOHEiysrK8O2332L48OHYtm0b6tWrV221dP09Uv4z0KFDB8ybNw9//vknVqxYgTt37mDevHmS1o6Pj0dBQQFatWolPhjgWelS+9SpU1i6dCk6deqE8ePHw8zMDHv27MGUKVNw4cIFTJgwQbLaZWVl+P333xEQEABnZ2eYmpri1KlT+OSTT3DmzBksWrRIstqP++yzz2Bi8uwbgvrU9vT0xMiRI7XaXF1dJa+dl5eHt99+G3l5eQgJCUH9+vWhUqlw9OhRSWs3b9680v+m27dvx6FDh576zzZdP/eMGTPw448/YvDgwWjRogVycnKwYcMGHDp0CDt37kTdunX1KyxQjcvPzxdu374tCIIgpKWlCa6urkJGRobkdY8fPy4UFxdrtV28eFFo2bKlMGvWLMnrP66wsFDw9/cXxo4dW6N1Z82aJYSFhQmhoaFCv379JK2VkZEhuLq6CmlpaZLWqUxeXp7QsWNHYf78+TVeuzJHjx4VXF1dhbi4OEnrfPnll4Krq6vw559/arVPnDhRaNGihaBWqyWrPWbMGKFdu3bC3bt3xbYbN24IPj4+woIFC6q1lq6/R3r16iUMGDBAKCkpEdtiYmIEd3d34eLFi5LWvnr1qlBWViYIgiD4+flVy+8ZXWpfvnxZuHr1qlZbWVmZMGLECMHLy0t48OCBZLWrMn/+fMHNzU24detWjdTOyMgQPD09hZiYGMHV1VU4e/bsU9XVp3bnzp2F8ePHP3WdZ6k9b948oUuXLuLYmqxdmbfeekvo3r27pLVVKpXg6uoqfPrpp1rt+/fvF1xdXYXvv/9e77o8p9AArK2tYW9vX+N1fX19K2ydubi44LXXXkN2dnaNr6dWrVpwcHBAXl5ejdXMzMzE9u3bMXv27BqrWa6goAAlJSU1Vm/Hjh3Iy8vD5MmTxfqCAW828MMPP0Amk6FPnz6S1il/StHjf0OuV68ezMzMYGpqKlntEydOICAgALa2tmKbo6Mj2rVrh127dlVrLV1+j2RlZSErKwtDhgzR+tzDhw9HWVkZ9u7dK1ltAGjUqBFkMtlT1XiW2o0bN0ajRo202mQyGbp164aioqJKtzirq3ZVGjZsCEEQkJ+fL3nt0tJSLFy4EKGhodVyapC+n1utVuPBgwfPXFfX2nl5eUhOTsaYMWNgb2+P4uJiqNXqGqldmczMTFy6dAl9+/aVtHZBQQEAVNiBKH9taWmpd12GQiMnCAJu3rxZYyG1oKAAt2/fxoULFxATE4M///yzwjajVARBwPz58xEUFPTU59Y8rRkzZsDPzw/e3t4YPXo0zp07J3nN9PR0vPLKK/jpp5/wxhtvwM/PD+3atUN0dDRKS0slr/8ojUaDXbt2oXXr1nB2dpa0Vtu2bQEAc+bMwR9//IHr169j+/btSE5ORkRERLVsp1VFrVbDwsKiQrulpSVUKlWlz1uX0tmzZwEALVu21Gp3cnJC/fr1xX5jcfPmTQCokd93Go0Gt2/fxvXr15GWloa1a9eicePGkv/8A8CmTZtw48YNvPfee5LXetwvv/wCHx8f+Pj4oFu3bti8ebPkNY8dOwa1Wo169eohPDwc3t7e8PHxwejRo3H58mXJ6z9u+/btAPBMoVAXzs7OaNCgARISErB//37k5OTg1KlTWLhwIZo3b46uXbvqPSfPKTRy27dvx40bNzB16tQaqffvf/8be/bsAQDI5XIMHToU7777bo3U3rZtG7KysrBixYoaqQc8/Iw9evRAp06dYG9vj3PnzmHt2rUYPnw4vv/++6c6EVhXly5dQk5ODiIjI/HOO++gRYsWOHDgAOLj41FcXIw5c+ZIVvtxhw4dwt27dyX/JQkAAQEBmDx5Mr788kvs379fbJ80adJTn0umq2bNmuHUqVMoKysTw6darUZmZiaAhyePOzo6SrqGR5Wfx6dQKCr0KRSKGg+phnT37l1s2bIF7dq1g4ODg+T1Dh06pPW7rWXLloiKipL0SDXw8HMuX74cEydORJ06dSSt9ThXV1e0adMGLi4uuHPnDr777jt8+OGHuHfvHsaOHStZ3fLgN2/ePLRs2RIxMTHIzc1FbGwsRo4ciR07dsDa2lqy+o8qLS3Frl274OXlJfkFnGZmZli+fDk++OADrYtZfHx8sH79+qc6UshQaMSys7Pxf//3f/Dz80P//v1rpOaECRMwZMgQ5OTkICUlBWq1GhqNRvIrQgsKCrBkyRKMHTu2Rv9Q9vX11bq6umvXrujSpQsGDRqE2NhYLFmyRLLahYWFuHfvHj744APxF3L37t1RWFiIjRs3Yvz48TXyhyPwcOtYLpc/05Wf+nB2dka7du3w1ltvwc7ODv/973/xxRdfwMHBAcOGDZOs7vDhw/Hxxx9j7ty5GD16NMrKyhAXFyeGs6KiIslqV6a8XmX/f1lYWFTbFt/zrqysDNOnT0d+fj7mzp1bIzW9vb2RkJCA/Px8ZGRkQKlUorCwUPK6y5cvh4ODA4YOHSp5rcetWrVK6/XAgQMxfPhwrFy5EsOGDYONjY0kdctPGVEoFIiPjxf/QtasWTOMHTsWW7durXDxi1TS09Nx8+ZNjBs3rkbq1alTBx4eHujZsye8vLxw+fJlfPnll5g8eTLWrFmj95+tyuHwGAAAEI5JREFU3D42UiqVCuPGjYOtrS2W/X979x7V8/0HcPxZzY8i0ThULtGsoY2GSB2mQi7NJUaR0Cg7CzEODnM9c+uYyYqMiRPlUtE0l4ohrGEOMtfVlBSV0EVK/f7o9D2++4akz7fNXo9z+uP7ub3en0/f8/m+Pu/b57vvFG1Se56lpSV2dna4urqyefNmkpKStNK/LygoiDp16jBhwgTFY73KBx98gK2tLWfOnFE0TsVT4t/78Lm4uFBcXMylS5cUjV8hPz+fuLg47O3ttdJsd+DAARYuXMiyZcv47LPP6NevH9988w3Dhg1j1apVPHz4ULHYbm5u+Pj4sH//fgYNGoSLiwu3b9/Gy8sLgPr16ysWuzIV34HK+lcVFRVVqybh32jp0qWcPHmS5cuXY2lpqZWYxsbG9OzZk/79+7Nw4UIcHR2ZMGFCjY3Crsz169cJCwtjzpw51Z5uqCbp6enh6elJYWGhorNsVHyPnZ2d1X7LevfujZGREefPn1cs9t9FR0ejp6fHwIEDFY/1+PFjxowZQ5cuXZgxYwZOTk5MnDiRgIAAEhMTiYqKeu1jSlL4H/T48WMmTZrE48eP+eGHHyptWtKGOnXq4OjoyOHDhxWtQbl37x4hISG4u7uTlZVFWloaaWlpFBUVUVxcTFpamqKJQmVMTEwUj1nxf31RJ2RtnXNsbCyFhYVaaToG2LFjBx07dtSY9sbBwYGCggKuXr2qaHw/Pz8SEhIIDQ1l//797N27l7KyMnR0dGjZsqWisf+u4jtQWSJy//59rdaa15b169ezY8cOZs2apfggp5dxdnamoKCAuLg4xWKsWbOGDh06YGFhobrPPXjwACi/Dyo9FVVlmjdvDih7v3nRvQ7Q6mDGJ0+ecOTIEWxtbWt0+qkXOXToEFlZWTg4OKgtt7GxoUGDBtVKhmv/UUJoVVFRET4+PqSkpLB161batm1bq+V58uQJZWVl5OfnK1ZrkZ2dTXFxMf7+/vj7+2usd3R0fKPJVasjNTVV8Vqzjh07curUKTIzM9WSkYyMDACtNR1HR0djYGCgceNSSlZWVqXnVlxcDKCVQTZGRkZ07dpV9fnUqVN89NFHWuvXVKFiQNXly5fV5snMzMwkIyND6wOutC00NJSAgADGjx+vqq2tLRUPvtUdfVwVd+/e5erVq5UOMJg8eTJNmjQhISFBsfiVSU1NBZS931R8tzMzM9WWl5aWcv/+fY05YpUSHx9Pfn6+1h6As7OzgfLzfF5ZWRmlpaXVmu1CksL/kGfPnjF9+nQuXLhAYGAgnTt31lrsnJwcjZtCXl4ehw4dwsTE5PUn2HwNLVq0qHRwydq1aykoKGDevHmYm5srEruy8z579iy//vorQ4cOVSRmBWdnZzZt2sSePXtUA4nKysrYvXs3BgYGWvn/5+TkcPr0aQYNGoS+vr7i8aC8H1FCQgK3b99We4vIgQMH0NPT01rzYYWYmBguXbpU7bcqvIl27drRtm1bwsPDGTFihGqQw86dO9HV1aVfv35aL5O2xMTEsGzZMlxcXJgzZ47W4ubm5mJoaKgxoGT37t2A5kjwmjR37lzVNCUVzpw5w/bt25k7d66ilQC5ubk0bNhQrfm2qKiIzZs3U79+fUXvNxYWFrz//vtER0fj4+OjmgEgJiaGvLw8rc1wER0djb6+Pn379tVKvIrfrQMHDqiNNI+Li6OgoIAOHTq89jElKawlgYGBAKr5Afft28e5c+do2LAhY8eOVSTmihUriI+Pp0+fPuTm5qq93q1+/fo4OTkpEhfKX/VTt25drK2tadq0KXfv3iUiIoKMjAzFfywNDQ0rPbeQkBD09PQUP299fX2sra1p3LgxN27cIDw8nMaNG+Pr66tYXCj/8Rk6dCgbN24kOzubDh068Msvv3Dy5ElmzZqllVqrmJgYSkpKtPbkDODl5cXx48dxc3NjzJgxGBkZcezYMY4fP87o0aMVfQA5ffo0GzduxM7OjkaNGnHhwgUiIyNxcXFh0KBBNR6vKveR2bNnM2XKFLy8vBg4cCDXr18nNDSUUaNGvdHo96rEjo+PVzXXP336lGvXrqn2GzJkiMZcgjUV++LFi8yePZtGjRpha2urmiKkgp2dXbWb914VOz4+nqCgIPr27UurVq0oLCzk5MmTnDx5kk8++eSNEpRXxe7Ro4fGPhVNp927d3+jmuGqnPeGDRvo378/ZmZm5ObmEhkZSUpKCosWLXqj/rRV+a7NmTOHSZMm4e7uzpAhQ7h//z4hISF06NCBTz/9VNHYUJ4Unzhxgn79+tVY3+FXxe7Tpw/t2rUjICCAtLQ0OnXqREpKCqGhoTRr1qxarzLVKavN2Wz/w15UW2FmZqY2jUZN8vDwIDExUetxAfbs2cO+ffu4efMmjx49wtDQUDWPlI2NjWJxX8bDw4NHjx4p+u7jbdu2ER0dze3bt8nLy8PY2Bh7e3t8fX0xNTVVLG6Fp0+fEhgYSFRUFFlZWbRo0YLx48drbWTiqFGjSE1N5cSJE4pPxfG8ixcvEhAQwB9//EFubi5mZma4urri5eWlaDlSUlJYsmQJV65cIT8/H3Nzc0aOHMnYsWMVGcxV1ftIbGws69ev59atWxgbG+Pq6soXX3zxRoMRqhL7Ze/AfpP3vb8qdkRExEsHsCkZ+/r162zcuJHff/+drKwsdHV1adOmDS4uLnh4eFT73cdViV2ZimvxJu8+rkrsy5cvs379eq5cuUJOTg7/+9//6NixIxMnTqRPnz7VjluV2BWOHz9OQEAA165dw8DAAEdHR7766qs36qpT1dhhYWEsXLiQoKCgGusqU5XYDx8+JDAwkGPHjpGenk79+vWxs7NjxowZ1XrokqRQCCGEEELI6GMhhBBCCCFJoRBCCCGEQJJCIYQQQgiBJIVCCCGEEAJJCoUQQgghBJIUCiGEEEIIJCkUQgghhBBIUiiE+I87evQolpaWHDx4sLaL8lYbPnw4gwcPru1iCCFeQl5zJ4T4x3ud9xXHxcXRokULBUujPStXrmTLli3ExMRgYWFR28V5pfDwcEpKShgzZkxtF0UIUQ2SFAoh/vFWrVql9vncuXOEh4czatQounTporbO2NhYm0UTzwkPD+fp06eSFArxLyVJoRDiH2/IkCFqn589e0Z4eDidO3fWWCeEEKJ6pE+hEOKtlJeXx8qVK3FwcMDKygp7e3vmzZtHZmZmlfYPDQ2lffv2zJo1i+LiYtXy8+fP4+3tjY2NDVZWVgwYMIDNmzdTWlqqtn9FH7o7d+4wdepUunbtirW1Nd7e3qSmptbouQKkp6czf/58evXqhZWVFb169WLx4sXk5uaqbbdy5UosLS1JT09nxYoV2NvbY2VlxfDhwzl9+rTGcfPy8liyZAm2trZ06tQJNzc3zp49y9SpU7G2tlZt1717d5KSkrhx4waWlpaqv0uXLqkdT1vXQwjx+qSmUAjx1ikqKmLcuHEkJSUxePBgPv74Y/7880/CwsJISEggIiKCd99994X7f/vtt2zYsAEvLy9mzZqFjo4OAAcPHmTmzJm89957fP755xgaGnL27FlWr17NzZs3Wb58udpxHj16xNixY+nRowczZ84kOTmZ0NBQpk2bRkRERI2db3JyMu7u7ujq6jJixAjMzMxITk5m586d/Pbbb+zatQsDAwO1faZPn46+vj6TJ0+msLCQrVu34uPjQ2xsLE2bNgWgtLSUKVOmkJiYyIABA+jWrRspKSl4e3tjYmKidrzFixezevVqSkpKmDFjhmp5y5YttX49hBDVI0mhEOKts3PnTpKSkvD19eXLL79ULe/SpQt+fn6sW7eOxYsXa+xXUlLC/PnziYqKYt68eXh6eqrW5eXlsWDBAnr06MGmTZvQ1S1vaHFzc6Nt27asW7cOd3d3PvzwQ9U+mZmZfP3112p97Bo0aMD333/PuXPnNPpDVteiRYuoU6cOkZGRasmug4MDHh4e7Ny5Ey8vL7V9TE1NWbt2repzp06d8PT0ZM+ePUyZMgUoT4ITExPx9PRk3rx5qm2tra3x8/NTSzSdnZ0JDg7m6dOnL2zS19b1EEJUjzQfCyHeOkeOHKFu3bpMnDhRbfnAgQMxNzcnNjZWY5/CwkJ8fHw4cOAAa9asUUsIAY4dO8ajR49wdXUlNzeXnJwc1d8nn3wCQEJCgto+9erVY/To0WrLevToAcBff/31pqcJwL179zhz5gz9+vVDR0dHrVwWFhY0a9ZMo1yAxvl169aNd955R61cR48eBWDChAlq2w4YMABTU9PXLqs2rocQovqkplAI8dZJS0vDzMxMo8kUwMLCgri4OAoLC9HX11ctX7p0Kfn5+QQGBuLo6Kix361btwDw8/N7YdysrCy1z6ampujp6akta9SoEYBGX7/qqijX9u3b2b59e6XbGBkZaSx7vlkXQE9PjwYNGqiVKy0tjXr16mk0Fevo6GBubs6FCxdeq6zauB5CiOqTpFAIIYD+/fvz008/ERwcjI2NDYaGhmrry8rKAFiwYAFt27at9BjNmzdX+/z3BKiy49WUkSNHMnDgwErXVZYcVzR/K12u52nzegghXp8khUKIt07Lli25ePGiRm0glNesNWnSRGN57969cXZ2xtfXl/Hjx7Nlyxa1GjZzc3OgvA9cz549FT+HqmrdujVQPiikpstlZmbG+fPnuXv3rlptYVlZGSkpKRrbVwzIEUL8O0mfQiHEW8fJyYmioiJ+/PFHteU///wzKSkpODk5Vbpf7969CQoK4ubNm4wfP54HDx6o1jk4OGBoaEhQUBB5eXka+xYUFFBQUFCzJ1IFpqam2NjYEB0dzdWrVzXWl5aWkpOTU61jOzg4AFR6HdPT0zW2NzAwkGZgIf7FpKZQCPHWcXNzY//+/Xz33XekpKTQuXNn1ZQ0zZs3Z+rUqS/c187OjuDgYHx8fPD09GTr1q0YGxvTsGFDVqxYwfTp03F2dmbYsGG0bNmShw8fcuvWLQ4fPkxISIja6OOaEhYWRuPGjTWWW1tbY2try7JlyxgzZgwjR45k2LBhWFpaUlJSQmpqKrGxsXh4eGiMPq4KZ2dnQkNDCQkJ4f79+6opafbu3Uu7du24c+eO2vadO3cmMTGR5cuX07FjR/T09LC3t6+0T6MQ4p9HkkIhxFunbt26bNu2jfXr13Po0CFiYmIwMjLCxcWF6dOnv3SOQiifiHnTpk1MnjyZcePGsXXrVpo0aYKTkxO7du1i06ZNREZGkpubi5GREa1atcLb25s2bdoocj7btm2rdPnEiROxtbWldevWREZGEhwczLFjx4iIiEBfXx8TExOcnZ0rHThTFbq6umzcuBF/f38OHjxIfHw87du3Jzg4mMDAQLKzs9W29/b2JiMjg6ioKEJCQigrK2PPnj2KJMpCiJqnUya9e4UQQrwmR0dHjIyMZNJpId4i0qdQCCHECz158kRjWUxMDGlpadjZ2dVCiYQQSpHmYyGEEC/k7+9PWloaXbt2xcDAgMuXLxMVFUXTpk01JrUWQvy7SfOxEEKIFzpy5AibN28mOTmZvLw8GjduTM+ePZk2bRpmZma1XTwhRA2SpFAIIYQQQkifQiGEEEIIIUmhEEIIIYRAkkIhhBBCCIEkhUIIIYQQAkkKhRBCCCEEkhQKIYQQQgjg/xp52ek5hBfjAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": { + "tags": [] + }, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "import seaborn as sns\n", + "import numpy as np\n", + "\n", + "sns.set(style='darkgrid')\n", + "\n", + "# Increase the plot size and font size.\n", + "sns.set(font_scale=1.5)\n", + "plt.rcParams[\"figure.figsize\"] = (10,5)\n", + "\n", + "# Measure the length of every token in the vocab.\n", + "token_lengths = [len(token) for token in tokenizer.vocab.keys()]\n", + "\n", + "# Plot the number of tokens of each length.\n", + "sns.countplot(token_lengths)\n", + "plt.title('Vocab Token Lengths')\n", + "plt.xlabel('Token Length')\n", + "plt.ylabel('# of Tokens')\n", + "\n", + "print('Maximum token length:', max(token_lengths))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wlDH4Wir-JFJ" + }, + "source": [ + "# New Section" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QQ8YUh75Usqj" + }, + "source": [ + "Let's look at just the tokens which begin with '##'." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "69Vf46p-Ym97" + }, + "outputs": [], + "source": [ + "num_subwords = 0\n", + "\n", + "subword_lengths = []\n", + "\n", + "# For each token in the vocabulary...\n", + "for token in tokenizer.vocab.keys():\n", + " \n", + " # If it's a subword...\n", + " if len(token) >= 2 and token[0:2] == '##':\n", + " \n", + " # Tally all subwords\n", + " num_subwords += 1\n", + "\n", + " # Measure the sub word length (without the hashes)\n", + " length = len(token) - 2\n", + "\n", + " # Record the lengths. \n", + " subword_lengths.append(length)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lakK1gF-S1Ej" + }, + "source": [ + "How many '##' tokens are there vs. the full vocab?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 50 + }, + "id": "PJh6wl2bSzaC", + "outputId": "9ee06298-f3af-4010-92b3-f50601fec3bb" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of subwords: 5,828 of 30,522\n", + "19.1%\n" + ] + } + ], + "source": [ + "vocab_size = len(tokenizer.vocab.keys())\n", + "\n", + "print('Number of subwords: {:,} of {:,}'.format(num_subwords, vocab_size))\n", + "\n", + "# Calculate the percentage of words that are '##' subwords.\n", + "prcnt = float(num_subwords) / vocab_size * 100.0\n", + "\n", + "print('%.1f%%' % prcnt)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fj_AcObWS-1S" + }, + "source": [ + "Plot the subword lengths (not including the two '##' characters)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 389 + }, + "id": "fO1GeFbhSyhD", + "outputId": "4e155231-d0a6-4b1e-caa9-638e75f6324d" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "Text(0, 0.5, '# of ## Subwords')" + ] + }, + "execution_count": 11, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoUAAAFjCAYAAABL3HHWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd1QUV/8G8Gdpi1SBgEQsKLILggqoNI0VC0ZfsSsKIpbEqCkmtiTmJPq+MbbYsMcSDRororEhFoyEqgaiErsRYwFZQVDp8/vDw/xcF3DRXRB9Pud4kr1zZ/Y7K8jDvXNnJIIgCCAiIiKit5pOTRdARERERDWPoZCIiIiIGAqJiIiIiKGQiIiIiMBQSERERERgKCQiIiIiMBQS1Zjp06dDLpfXdBkasXv3bsjlciQkJNRYDR9//DHc3d1r7P3fBpcuXYJcLse6deuq7T1/+ukneHl5IS8vr9re822Tl5cHLy8vrF+/vqZLoRrGUEj0Aunp6Zg5cyZ69uyJVq1aoW3btvD398e0adMQHx9f0+XVGmXBUZ0/QUFBNV2uRnl5eWHw4ME1XYZasrKysGzZMpw5c6amS4FCocCqVaswduxYmJiYaPTYt27dglwux969ezV2TC8vL8yaNQsAMHfu3Bf+klJYWIiWLVti06ZNAICCggK4urpi586dar2fn58f5syZI77u1KkTlixZotLv+PHjkMvl+Ouvv1TqBAATExOMGjUKK1euxIMHD9R6b3oz6dV0AUSvs7/++gtBQUHQ09NDQEAAmjVrhvz8fPzzzz+IjY2FsbExvL29a7rMWqFt27aYN2+eUtuqVatw7do1lfZ33nmnOkujZ2RlZSEsLAxGRkbw8PCo0Vp+/vlnFBcXY+jQoRo/dnR0NPT19dGpUyeNH1tdFy5cQEFBAVq3bg0ASElJQVFRkVoj3pmZmUhPTxf3/ffff3Hnzp2XHi0fNmwYwsLCEB4ejokTJ77UMaj2YygkqsTy5cvx5MkTREZGwsnJSWV7ZmZmDVRVvUpKSlBYWIg6deq80nEaNmyIhg0bKrXt3LkT165dQ9++fV/p2PTmKSwsxM6dO9G1a1eNjxICwJEjR+Dp6QkzMzONH1tdZ8+ehZGRkfhvy+nTp1G3bl00bdpUrX0BiMH99OnTkEgkLx0Kzc3N0blzZ+zYsQPjx4+Hrq7uSx2HajdOHxNV4saNG6hbt265gRAArK2txf8vm45atmyZSr9ly5ZBLpfj1q1bKtsUCgWmTp0KLy8vuLm5YeTIkTh//rxSny5dumDEiBFKbatXr4ZcLsf48eOV2ufPnw+5XI779+8rvcd3332Hjh07wtXVFR07dsR3332nMlVUNsX7xx9/YPny5fDz80PLli1x8OBBsc/27dvRs2dPuLq6olu3bti4cSO0+bTMv/76Cx988AHatm2LFi1aoE+fPvj555/Ves+8vDyEhoaiZcuWOHLkiNheUlKCjRs3om/fvmjZsiU8PDwwatQonD59Wmn/Z6+hO3ToEAICAtCiRQu89957WLx4MUpLSzV+vidOnEBwcDBat26NFi1aICAgALt27VLp5+XlhQ8++AB///03QkND4e7ujjZt2uDzzz8vdwqwbNS7VatW8Pb2xtdff427d+9CLpeLU4nHjx9Hnz59AADz5s0Tp/N79+6tcjx1Po+0tDRMmDAB7du3h6urK9q3b4+QkBD88ccfL/wckpOTcf/+fXTs2FGpPTw8HHK5HKmpqWLb48eP4erqCrlcjhs3bojtWVlZcHJywty5c5WOkZWVhTNnzsDPz0+p/Y8//kBQUBA8PDzQqlUrDBgwQKPTywCQnZ0NhUIBhUKBpKQkODk5IScnR3zt7OyMBw8eQKFQID8/X2nf3Nxccd/ExETUr18fOjo6UCgUSEhIQOPGjVFUVASFQoHHjx9XubYOHTrg7t27YuCktw9HCokq0ahRI1y/fh1RUVHo3r27Vt5jzJgxMDc3x8SJE3H//n388ssvGDFiBLZt2waZTAYA8Pb2xt69e5Gfnw9DQ0MAQFxcHHR0dJCUlISSkhLxN/v4+Hg0a9ZMnILNzc3FsGHD8M8//2DAgAFo3rw50tLSsHXrVsTHx2PHjh0qIzFz585FcXExBg8eDGNjYzRp0gQAsHHjRsyZMwdOTk6YPHkynjx5gvXr18PKykorn01ycjJGjRoFIyMjDB8+HJaWljhy5Ai+//57XL16Vem6qOdlZmZi3Lhx+Pfff7F+/Xq0adMGACAIAj7++GMcP34cvXr1wuDBg5Gfn4+IiAgEBwdj7dq18PX1VTrWoUOHcOfOHQwZMgSDBg3C4cOHsXLlSlhaWiI4OFhj51v2+Xp6emLChAmQSqWIiYnBl19+iTt37qhM66WnpyMkJAS9e/dG9+7dkZqail27dqGgoABhYWFiv7///hvBwcHQ09NDSEgI3nnnHRw9ehQfffSR0vGaN2+Ozz//HAsXLkTv3r3RoUMHAFAZTVPn87h37x5GjhwJqVSKoUOHwtbWFgqFAqmpqTh37pzKZ/y8xMREAEDLli2V2ssu14iPjxe3JScno6ioCDo6OoiPj4e9vT0AICEhAYIgqFzicezYMQiCgK5du4ptBw4cwOeff453330XY8aMgaGhIfbt24cpU6bgzp07+OCDDyqtV109evRAdna2UpuPj0+5r6dOnYrRo0eL7eX9wljRvsOHD8c333xTpdrc3NwAPP3sy75f6C0jEFGFzpw5I7i4uAgymUzo3r27MH36dCE8PFy4cuWKSt/09HRBJpMJS5cuVdm2dOlSQSaTCenp6WLbtGnTBJlMJkyYMEEoLS0V2//66y9BLpcLoaGhYltkZKQgk8mEU6dOCYIgCAUFBULLli2FL774QpDJZEJKSoogCILw8OFDwdnZWZg1a5a4748//ijIZDLhl19+Uarpl19+EWQymbBo0SKxbdeuXeK5Pn78WKl/Tk6O0KpVK8Hf319p2507dwQ3NzdBJpMJ8fHxlX+gzxkxYoQgk8kq3N6nTx/B1dVVuHbtmthWUlIijBs3TpDJZMKff/4ptk+aNElwc3MTBEEQrl27JnTp0kXo2LGjcPnyZaVjRkRECDKZTNi7d69Se0FBgdCrVy/h/fffF9suXrwoyGQyoXXr1sK9e/fE9uLiYqFr165Ct27d1DpPT09PYdCgQZX2uXnzptC8eXPh66+/Vtk2Y8YMwcXFRakGT09PQSaTCSdOnFDqO3XqVEEmkwl37twR28aMGSPI5XLh/PnzYtuzn+N3332ncs4//fSTSh1V+Tz27t1bbn3qmjhxoiCXy4Xi4mKVbe3atVP6/pg7d67QqVMnITAwUPj444/F9pkzZwouLi5CXl6e0v5jx45V+vsoKCgQvL29BR8fHyErK0tsz8/PFwICAoTmzZsrfZ6vIjExUYiNjRW//3755RchNjZW2LBhgyCTyYTt27cLsbGxQmxsrPDvv/8q7ZuSkiLExsYKBw8eFL93Y2NjhX379gkymUxYuXKluO/Vq1erXFt+fr4gk8mEzz77TCPnSrUPp4+JKuHu7o5du3ahX79+yM3Nxe7du/Hdd9+hV69eGD58ONLT01/5PcaMGQOJRCK+dnV1Rbt27RAXF4dHjx4BUB4dAZ5eT5Sfn48xY8bA1NQUcXFxAJ7+hl9SUqI0MnLkyBFYWlpiyJAhSu87ZMgQWFpaIjo6WqWmYcOGqVxDeOrUKTx58gTDhw9X2mZraytOOWpSeno6Ll68CH9/f3GkEgB0dHQwbtw4AFCaEi6TkpKCYcOGwcjICNu2bUOzZs2Utu/btw9WVlZo166dOBWnUCiQl5eHjh074vLly7h3757SPr169YKNjY34WldXF23btsWtW7dQXFyskfM9cOAAiouLMWDAAKW6FAoFunTpgqKiIpVb/tjb26tMr5b93f/zzz8Anq5o/eOPP+Dl5YXmzZuL/XR0dBASEvJStarzeZiamgJ4Oh1e9nVcFQqFAiYmJuVe2+bl5YXTp0+jqKgIwNPvC29vb/j4+Iijg8DTkUJXV1cYGxuL++bl5SEuLk5p6vj06dNQKBTi90QZqVSKUaNGobi4GMePH6/yOZSnbdu28PX1RUFBAYyNjTF48GD4+voiPz8fdevWxYABA+Dr6wtfX1/Ur19fad+WLVvC19dX/PciMDAQvr6+EAQBurq6GDZsmLivOtclPk8qlaJOnTrIysrSyLlS7cPpY6IXkMvl+OGHHwA8XeGXlJSEHTt2IDk5GR999BF27doFAwODlz6+g4NDuW2nTp3C7du34ejoCBsbGzRt2lQMhfHx8bC2toZcLkfbtm0RHx+PDz74APHx8dDR0YGnp6d4rFu3bsHV1RV6esrf7np6erC3t8eFCxdU3v/ZEPbscQCU+8OmvHN4VWXv93yoAwBHR0cAUAnlBQUFGDlyJKysrBAeHl7uIoKrV68iKytLZdrtWVlZWahXr574ukGDBip96tati5KSEuTm5sLCwkK9k6rE1atXAUAlvD9f17OeX7hTVhcAcYry7t27KC4uLvfv9GWCA6De59GhQwf06NEDW7ZswY4dO9CyZUu0a9cO77//vji9WxmJRFLhdaPe3t747bffkJKSAplMhrS0NIwcORJ2dnZYtmwZLl68iLp16+LGjRvw9/dX2vfkyZMoLCxUCoVlX2tlX1fPquhr7WVkZ2eL113GxMTAzc0Nubm5AJ7+0uXu7i7+vZmYmCj9u5KbmyuG4BMnTqBp06bQ09ODQqFATEwMZDIZSkpKoFAoYGRkJF5mUlWCICj9kkpvF4ZCoiqws7ODnZ0d+vbti8DAQJw5cwapqalo06ZNpf+QamI0ydvbG9u2bUNubi7i4+Ph5eUltv/4448oLCxEfHw8nJ2dYW5u/krv9bI/UGqagYEBunfvjsjISPz666/iiOKzBEFA/fr18b///a/C4zRq1EjpdWUrMSsKLlVVdpwlS5ZUuCL2+TBVHXWVR5331dHRwdKlS5GWlobff/8dycnJWLNmDVauXIlvv/0WAwcOrPQ9LC0t8ejRI6XrZcs8O3JeFrS8vb1haWmJOnXqIC4uTgzqz19PGB0djaZNm750IH4VVbmecMmSJejZs6fYXpXrCZ+/FlFdBQUFyM/PVxotpbcLQyHRS5BIJGjVqhXOnDmDjIwMABCDWE5Ojkr/8lYdl7l69ap4gfezbbq6ukrTR97e3tiyZQtOnDiBv/76C/379wfw9AdBfn4+jh49isuXL2PUqFFKx2rYsCGuX7+O4uJipdHC4uJi3Lhxo9zRpvKUjQ5du3ZN5YdR2SiXJpW935UrV1S2lbU9X7tEIsEPP/wAXV1dLFy4EEVFRZgwYYJSn8aNGyMlJQVt2rR5pRFeTWvcuDGAp/do1ORF/ra2ttDT08P169dVtl27dk2lTdOjRM7OznB2dsa4ceOgUCgwYMAA/Pjjjy8MhY6Ojjh8+DBu3LihMhLdsGFD2NnZIS4uDg8ePEDTpk3Fkd3WrVsjLi4OVlZWkEqlSvdaLCwsRExMDIYPH65yPAC4fPmySh0Vfa29jLCwMBQVFeHixYv44Ycf8MMPP6BevXpISUnB4sWLsXDhQjGQPf+0o2+//RZ5eXm4f/8+pkyZgi+++AIuLi74999/8fXXX2PGjBniwjR1RmLLU3bJQXkjpvR24DWFRJWIjY0td5QvPz8fsbGxAP5/6tTExATW1taIj49XGqVJT08v97q9Mj/99JNS//Pnz+OPP/6Aj4+P0rVQXl5ekEgkWLlyJYqKisQREJlMBisrK4SFhZW70tLPzw8KhQI7duxQat++fTsUCoXKbTkq0q5dOxgaGiI8PBxPnjwR2+/evYt9+/apdYyqaNiwIeRyOQ4ePCj+sAKA0tJSrFmzBgDQrVs3lf10dHTw/fffY8iQIVi6dKnKEx4CAgKQn59f7q2DACjdyqc69e7dG7q6uliyZAkKCwtVtmdnZ4vTh1UhlUrFa+3S0tLE9tLSUmzcuFGlv5GREYDyf7mpigcPHqiMVlpaWsLW1hYPHz584Uhm2SUQKSkp5W739vZGSkoKTp48qfQ17+3tjeTkZMTFxcHd3V0p+MfFxSEvL0/la97DwwOWlpbYvn270u18CgsLsWHDBujp6aFz587qnXglyq4nLC4uRt26dREQEABfX18UFRXB2toavXv3Fq8JfH5Ff9n1hHp6etDR0RGvRdTR0YGenp74urxrEdVV9lm3bdv2lc+VaieOFBJVYs6cOcjOzkaXLl0gk8lgaGgohqAbN24gICBA6Tf64cOHY/HixRgzZgz8/PyQkZGBX3/9FY6OjuIjpp53+/ZtjB49Gl26dEFmZiZ++eUXGBoaYsqUKUr9yu6XmJaWBjs7O6WRCy8vLxw4cAD6+voqo0xjxozBoUOHMGvWLFy4cAHOzs5IS0vDzp070aRJE4wZM0atz8Lc3ByffPIJ5s6di6FDhyIgIABPnjzBr7/+WuG1ia/qm2++wahRozBkyBAEBgbCwsICR44cQUJCAoYMGYJWrVqVu59EIsF3330HPT09rFixAiUlJZg8eTIAoF+/fjh58iTWrFmDP//8E++99x7q1q2LO3fuIDk5GdnZ2VoJuffu3cOKFSvK3TZ06FDY29vjyy+/xH//+1+8//776NOnD959911kZWXh77//xtGjRxETE/NSU3tffPEFkpOTERwcjOHDh+Odd95BdHS0eD3bs6OD9evXh42NDSIiImBjYwMLCwuYmpqKt6dR19atW7F792507doVjRo1go6ODuLi4nDmzBkMGDDghSOSrVu3hrW1NWJiYsRR8Wd5e3tj165duHnzpkooXLBgAR49eqRyfWZ0dDRsbW3RokULpXYDAwPMnDkTkydPxsCBAzFo0CBIpVLs27cP58+fx+TJk2Fra1ul869MUlISPDw8xM8gKSlJ7dHhpKQkODo6ijMTycnJaN68uRjmX0VMTAxsbW35DPG3GEMhUSWmT5+Oo0eP4vTp0zh8+DByc3NhamoKmUyGsWPHqvywGjt2LHJzc7F3714kJiaiWbNm+N///ofz589XGAp/+uknzJkzB8uWLUN+fj5atWqFqVOnlnvDbG9vb6SlpamMBvr4+ODAgQMqKy2Bp6tAt27diqVLl+LYsWPYvXs3rKysMHToUEyaNKlKT4sIDQ2FkZERNmzYgIULF+Ldd99FaGgoTE1N8eWXX6p9HHW1adMG4eHhCAsLw6ZNm5Cfny+GpxfdH1AikeCbb76Bvr4+Vq9ejaKiIkybNg0SiQSLFi1Cu3btsGvXLqxcuRIlJSWwtraGq6srRo4cqfHzAJ6OqJb3XFrg6WiupaUlRowYAUdHR2zYsAHh4eF49OgRLCws4ODggC+++EJc0VtVTk5O2LRpE+bPn4/169ejTp068PPzQ2hoKHr16gWpVCr2Lft85s2bh/nz5yM/Px+Ojo5VDoXt27fHtWvXcPToUdy/fx+6urpo1KgRvvrqKwQGBr5wfz09PQwaNAjr1q0Tv++eVfY9IJFIxOtrAcDFxQXm5ubIyclR+j4pLS3FsWPH0KNHj3IDaa9evWBmZobVq1dj9erVKCkpgaOjI+bNm6fRJ+6UlJTg9OnT4mUNhYWFSE1NxdSpU9XaPykpSel8k5KSyh0xr6qcnBycOHECH374IZ9m8haTCNq8GpmIiF5biYmJCAoKwsyZM1WemPM6UCgU6N69O8aNG1fuoqGqOH36NAIDA7Fhw4YX3jj7bbRy5UqsX78eUVFRGllNT7UTrykkInoLPP/ItNLSUqxbtw6A6irW14WlpSXGjx+PdevWIS8v75WOJQgCJk2apHS7JnoqLy8PGzduxPjx4xkI33IcKSQiegt06tQJXbt2hYODA/Ly8hAdHY2UlBQMHDiw0tvzENHbg6GQiOgtMHv2bJw6dQr37t1DSUkJGjVqhICAAISGhvIaMiICwFBIREREROA1hUREREQEhkIiIiIiAu9TqDEPHjxCaSln4omIiOj1paMjgYWFcbnbGAo1pLRUYCgkIiKiWovTx0RERETEUEhEREREDIVEREREBIZCIiIiIgJDIRERERGBoZCIiIiIwFBIRERERGAoJCIiIiIwFBIRERER+EQTordSXVMD6BtKa7oMtRXlFyA7t7CmyyAieqMxFBK9hfQNpTgQPKqmy1Bbr00bAIZCIiKt4vQxERERETEUEhERERFDIRERERGBoZCIiIiIwFBIRERERGAoJCIiIiLUcCjMyMjAggULEBQUBHd3d8jlciQkJKj069KlC+RyucqfBQsWqPR9+PAhZs6cCW9vb7i5uSE4OBhpaWnlvv/Ro0fRr18/tGjRAp06dUJYWBiKi4s1fp5EREREr7savU/h9evXsXbtWjRu3BhyuRxnz56tsK+LiwtGjhyp1CaTyZRel5aWYty4cbh06RJCQ0NhYWGBLVu2ICgoCLt370ajRo3EvjExMZgwYQK8vb0xc+ZMXLp0CcuXL8eDBw8wc+ZMzZ4oERER0WuuRkOhi4sL4uPjYWFhgejoaEyYMKHCvra2tujbt2+lxzt06BDOnj2L5cuXw8/PDwDg7++PHj16ICwsDPPmzRP7zps3D82bN8e6deugq6sLADA2NsaaNWsQFBQEe3v7Vz9BIiIiolqiRqePTUxMYGFhoXb/wsJCPHnypMLthw8fho2NDbp27Sq2WVpawt/fH9HR0SgqKgIAXLlyBVeuXMGQIUPEQAgAgYGBKC0tRVRU1EucDREREVHtVWsWmsTGxsLNzQ1ubm7w8/PDtm3bVPqkpaXBxcUFEolEqb1FixZ49OgRbt68CQC4cOECAMDV1VWpX7169WBraytuJyIiInpb1IpnH8tkMrRp0wb29vZ48OABtm/fjm+++QY5OTkYN26c2C8zMxPe3t4q+9vY2AB4urDFwcEBmZmZAABra2uVvtbW1sjIyKhyjVZWJlXeh4jUZ21tWtMlEBG90WpFKFy1apXS6/79+yMwMBArVqzAsGHDYGr69IdFfn4+DAwMVPYva8vPz1f6b3l9pVJppVPUFcnKykNpqVDl/ej1ZGYuhbScr4/XVUFhIR7mFKjdvzYGrMzM3JougYio1tPRkVQ4kFUrQuHzdHV1MXLkSHz22Wc4e/YsOnToAAAwNDREYWGhSv+yNkNDQ6X/lte3oKBA3E5vL6mBAUI2fFLTZaht46glANQPhURERM+rNdcUPs/W1hYAkJOTI7ZVNPVb1lY2jVw2bVw2jfyszMxMsR8RERHR26LWhsL09HQAT1cXl3FycsL58+chCMrTuKmpqTAyMhLvU+js7AwAOHfunFK/e/fu4e7du+J2IiIiorfFax8Ks7OzUVpaqtRWUFCAdevWwdjYGG5ubmJ7z549kZGRgaNHj4ptCoUChw4dQteuXaGvrw8AcHR0RNOmTbFt2zaUlJSIfbdu3QodHR10795dy2dFRERE9Hqp8WsKV6xYAQC4evUqACAyMhKnT5+GmZkZRowYgWPHjmHVqlXo0aMH7OzskJ2djYiICNy4cQPffvstjI2NxWP16NEDbm5umDp1qvhEk61bt6K0tBSTJk1Set+pU6di/PjxGD16NHr16oVLly4hPDwcQ4YMQZMmTarvAyAiIiJ6DdR4KFyyZInS6127dgEA7OzsMGLECMhkMjRt2hSRkZFQKBQwMDCAi4sLpk+fjs6dOyvtq6urizVr1mDevHnYvHkzCgoK0KJFC8ydOxeNGzdW6tu5c2eEhYUhLCwMs2fPhqWlJcaPH4+PPvpIuydMRERE9BqSCM9fgEcvhbekebNYW5vWutXHVblli7W1KQ4Ej9JiRZrVa9MG3pKGiEgDKrslzWt/TSERERERaR9DIRERERExFBIRERERQyERERERgaGQiIiIiMBQSERERERgKCQiIiIiMBQSERERERgKiYiIiAgMhUREREQEhkIiIiIiAkMhEREREYGhkIiIiIjAUEhEREREYCgkIiIiIjAUEhEREREYComIiIgIDIVEREREBIZCIiIiIoIGQqFCocCNGzc0UAoRERER1RS1Q+GePXswc+ZMpbaFCxeiXbt28Pf3x9ChQ5GXl6fxAomIiIhI+9QOhb/++iuKi4vF13/99RfWrl2LNm3aYNCgQfjrr7+wceNGbdRIRERERFqmp27HmzdvomfPnuLrQ4cOwdzcHOvWrYOBgQEkEgkOHjyIiRMnaqVQIiIiItIetUcKc3NzYWpqKr6Oi4uDr68vDAwMAACurq64ffu25iskIiIiIq1TOxRaW1vjn3/+AfB0ccnff/+NNm3aiNsfP34MXV1dzVdIRERERFqn9vSxl5cXwsPDYW5ujoSEBEgkEnTs2FHcfv36ddSrV08rRRIRERGRdqkdCj/55BOcPXsW8+fPBwCMHz8eDRo0AAAUFxcjKioK3bt3106VRERERKRVaodCW1tb7N+/H1euXIGpqSnq168vbsvPz8esWbPg5ORUpTfPyMjApk2bkJKSgnPnzuHx48fYtGkTvLy8xD4PHjzArl27cOzYMVy7dg3FxcVwcHBASEgI/P39lY63e/duzJgxo9z3Sk1NhVQqVWo7evQowsLCcOXKFVhZWWHgwIH48MMPoaen9sdCRERE9EaoUvrR1dWFXC5XaTcxMYGfn1+V3/z69etYu3YtGjduDLlcjrNnz6r0+fPPP7F48WJ06NAB48ePh56eHg4fPoxPP/0U165dw4QJE1T2+eyzz/Duu+8qtenr6yu9jomJwYQJE+Dt7Y2ZM2fi0qVLWL58OR48eKByP0YiIiKiN12FofBlVxI/O4L4Ii4uLoiPj4eFhQWio6PLDXjNmjXD4cOHYWdnJ7YFBgYiJCQEa9aswejRo2FoaKi0T8eOHeHs7Fzpe8+bNw/NmzfHunXrxAUyxsbGWLNmDYKCgmBvb6/2eRARERHVdhWGwi5dukAikVT5gGlpaWr3NTExeWGfhg0bqrRJJBL4+fkhPj4e//77LxwcHFT65OXlwcjICDo6qgusr1y5gitXrmDWrFlKK6YDAwOxatUqREVFYdy4cWqfBxEREVFtV2EonDBhgkooPHbsGNLS0uDr64tmzZoBeBqw4uLi4OTkhC5dunLiBLAAACAASURBVGi32mfcv38fAGBhYaGyLTAwEI8fP4ZUKkWnTp0wffp0pRHMCxcuAHh6b8Vn1atXD7a2tuJ2IiIiordFhaFw0qRJSq/37duHW7duYffu3SpTs+fPn0dISEi1TblmZ2djx44d8PT0hKWlpdhep04d9O/fH15eXjA2NkZKSgp+/vlnpKSkICIiQuybmZkJ4Om9F59nbW2NjIyMKtdkZfXiUU8ibbK2Nn1xp1rsTT8/IqKapvZCkzVr1mD48OHlXqvn4uKCwMBArF69Gr1799Zogc8rLS3FF198gdzcXHz99ddK2/z9/ZVWJHfr1g1t27bFuHHj8PPPP+Ozzz4D8HS1NADxaSzPkkqlePLkSZXrysrKQ2mpUOX96PVUGwNIZmau2n3f9PMjIqLy6ehIKhzIUvuJJv/88w+srKwq3P7OO++ITzzRptmzZ+PUqVOYM2dOuSuhn9exY0c0bdoUcXFxYlvZwpTCwkKV/gUFBSoLV4iIiIjedFV6zF1UVBQEQXU0rLS0FFFRUXjnnXc0WtzzwsLCsGXLFkyZMqVKI5LvvvsucnJyxNdl08Zl08jPyszMhI2NzasXS0RERFSLqB0KBw8ejMTERIwePRonT55Eeno60tPTERMTg9GjRyM5ORlDhgzRWqHh4eFYtmwZQkJCMHr06Crtm56errQgpWwK/Ny5c0r97t27h7t3777wdjZEREREbxq1rykcN24c7t+/j19++UVpKrbM8OHD8cEHH2i0uDIHDhzAf//7X/Tp0wfTp0+vsJ9CoVBaeAI8XSBz8+ZN9O3bV2xzdHRE06ZNsW3bNgwcOFC8Lc3WrVuho6PDx/URERHRW0ftUCiRSPDVV18hMDAQR48eRXp6OoCn9xHs0qULmjZt+lIFrFixAgBw9epVAEBkZCROnz4NMzMzjBgxAqmpqZg6dSrq1q0LHx8f7N27V2n/du3aidPWQ4cOhYuLC5o3bw4TExOkpqZiz549sLe3x8iRI5X2mzp1KsaPH4/Ro0ejV69euHTpEsLDwzFkyBA0adLkpc6FiIiIqLZSKxQWFhYiJSUF1tbWaNKkCcaMGaOxApYsWaL0eteuXQAAOzs7jBgxAleuXEFRUREUCgW+/PJLlf03bdokhkJ/f3+cOHECv//+O/Lz82FjY4Phw4dj4sSJMDVVXm3ZuXNnhIWFISwsDLNnz4alpSXGjx+Pjz76SGPnRkRERFRbSITyVo48p7i4GK1atcK0adMQHBxcHXXVOrwlzZvF2toUIRs+qeky1LZx1JIq35LmQPAoLVakWb02beAtaYiINOCVb0mjp6eHd955p9yVx0RERERU+6m9+rhnz544ePAgSktLtVkPEREREdUAtReaDBo0CAkJCRg1ahRGjhyJxo0bo06dOir9nn3GMBERERHVDmqHwt69e0MikUAQBCQmJlbYLy0tTSOFEREREVH1UTsUTpgwARKJRJu1EBEREVENUTsUTpo0SZt1EBEREVENUnuhCRERERG9udQeKQSA0tJSRERE4MiRI7h16xYAoEGDBujevTsCAgKgo8OMSURERFQbqR0K8/PzMXbsWCQnJ0MikcDa2hoAcPLkScTExGDPnj1Yu3YtpFKp1oolIiIiIu1Qe2hv5cqVSEpKwqhRoxAXF4eYmBjExMQgPj4eoaGhSExMxMqVK7VZKxERERFpidqh8MCBA/D398fUqVNhbm4utpuZmWHKlCnw9/fH/v37tVIkEREREWmX2qHw7t278PT0rHB727ZtcffuXY0URURERETVS+1QaGZmhps3b1a4/ebNmzAzM9NIUURERERUvdQOhb6+vggPD8fvv/+usu3UqVPYunUr2rdvr9HiiIiIiKh6qL36+NNPP8WpU6cwbtw4ODs7w9HREQBw+fJlpKWlwcLCAh9//LHWCiUiIiIi7VE7FNrZ2WHXrl1YuHAhjh8/jgsXLgAAjI2N8f7772Py5MmoX7++1golIiIiIu2p0s2r69evj4ULF0IQBCgUCgCApaUln4lMREREVMupHQpLS0vFJ5ZIJBJYWVlprSgiIiIiql5qh0JPT0+0adMG3t7e8Pb2hpOTkzbrIiIiIqJqpHYobN++PZKSknDixAlIJBKYm5vD09NTDIlNmzbVZp1EREREpEVqh8LFixcDAC5duoT4+HjxT1RUlPgsZG9vb8ybN09rxRIRERGRdqh9n8IyMpkMwcHBWLFiBRISErBkyRI4ODggIyMD+/bt00aNRERERKRlVVp9DAAKhUIcJYyLi8OtW7ego6ODVq1awdvbWxs1EhEREZGWqR0Kv//+e8THx+Py5csAno4YdunSBT4+PmjTpg1MTEy0ViQRERERaZfaoXDTpk3Q1dXFf/7zH0ycOBENGzbUZl1EREREVI3UDoWDBw9GfHw8IiMjsX//fri6uoorjz08PGBgYKDNOomIiIhIi9QOhbNmzQIA3LlzB3FxcUhISEBERARWrVoFqVQKNzc3+Pj44MMPP9RasURERESkHVVeffzuu++if//+mDt3LmJiYrB48WI0aNBAXIlcFRkZGViwYAGCgoLg7u4OuVyOhISEcvsePXoU/fr1Q4sWLdCpUyeEhYWhuLhYpd/Dhw8xc+ZMeHt7w83NDcHBwUhLS3ulYxIRERG96aq8+vjmzZvi6uOEhAQoFAoIggBzc3N4eXlV6VjXr1/H2rVr0bhxY8jlcpw9e7bcfjExMZgwYQK8vb0xc+ZMXLp0CcuXL8eDBw8wc+ZMsV9paSnGjRuHS5cuITQ0FBYWFtiyZQuCgoKwe/duNGrUqMrHJCIiInobqB0KZ8yYgYSEBNy5cweCIMDY2FjpsXfOzs5VfnMXFxfEx8fDwsIC0dHRmDBhQrn95s2bh+bNm2PdunXQ1dUFABgbG2PNmjUICgqCvb09AODQoUM4e/Ysli9fDj8/PwCAv78/evTogbCwMKUba6t7TCIiIqK3gdrTxwcOHECjRo3wySef4Ndff0ViYiJWr16NUaNGvVQgBAATExNYWFhU2ufKlSu4cuUKhgwZIoY3AAgMDERpaSmioqLEtsOHD8PGxgZdu3YV2ywtLeHv74/o6GgUFRVV+ZhEREREbwO1RwqTkpJqZIXxhQsXAACurq5K7fXq1YOtra24HQDS0tLg4uICiUSi1LdFixbYtm0bbt68CQcHhyodk4iIiOhtoHYofD4Q5ufnAwAMDQ01W9FzMjMzAQDW1tYq26ytrZGRkaHUt7ynqtjY2AB4urDFwcGhSsdUl5UVb95NNcva2rSmS9CqN/38iIhqWpUWmmRlZWHZsmWIjo5GVlYWAMDKygp+fn6YOHEi3nnnHY0XWBY+yxullEqlePLkiVLf8vqVtZUdqyrHVFdWVh5KSwXxtamZIQyl+lU+Tk3KLyhC7sP8mi7jtVAbA0hmZq7afd/08yMiovLp6EgqHMhSOxSmp6cjMDAQmZmZaNKkCdzc3AAAV69exa+//oqjR49iy5YtGn/SSdlIZGFhocq2goICpZFKQ0PDcvuVtZX1rcoxX7puqT4Cp4a/8nGq05Z5w5ELhkIiIqK3kdqhcO7cucjOzkZYWJi4srfMkSNHMHnyZMydOxdhYWEaLbBsijczM1OcBi6TmZkJd3d3pb7lTf2WtZXtX5VjEhEREb0N1F59HBcXh+HDh6sEQgDo1q0bhg0bhri4OI0WB0Bc2Xzu3Dml9nv37uHu3btKK5+dnJxw/vx5CIKg1Dc1NRVGRkbifQqrckwiIiKit4HaoVAikaBx48YVbre3t1dZ9asJjo6OaNq0KbZt24aSkhKxfevWrdDR0UH37t3Ftp49eyIjIwNHjx4V2xQKBQ4dOoSuXbtCX1+/ysckIiIiehuoPX3ctm1bJCQkYNiwYeVuT0xMhKenZ5ULWLFiBYCn1yYCQGRkJE6fPg0zMzOMGDECADB16lSMHz8eo0ePRq9evXDp0iWEh4djyJAhaNKkiXisHj16wM3NDVOnThWfaLJ161aUlpZi0qRJSu+r7jGJiIiI3gYS4fm51gqkp6cjODgYPXr0wNixY2FlZQXg6YrkNWvW4MiRI9i0aRMaNGhQpQLkcnm57XZ2djh27Jj4Ojo6GmFhYbh69SosLS0xYMAAfPTRR9DTU861OTk5mDdvHqKjo1FQUIAWLVpg+vTpcHFxUXkPdY+pjudXH1tbm9bKhSZc4fmUtbUpQjZ8UtNlqG3jqCVVXn18IHiUFivSrF6bNvBrk4hIAypbfVxhKHz2qSBlHj9+jOzsbACAmZkZAODhw4cAgLp168LY2BjR0dEaKbq2YSh8szAUvl4YComINOOlbklTv359rRVERERERK+XCkPh5s2bq7MOIiIiIqpBaq8+JiIiIqI3F0MhEREREal/SxonJ6cX3odQIpHgwoULr1wUEREREVUvtUNhQECASigsLi5Geno6UlJSIJfL+SQQIiIiolpK7VD4ww8/VLjtzJkzGD9+PL799ltN1ERERERE1Uwj1xR6eHigf//+WLBggSYOR0RERETVTGMLTezt7XH+/HlNHY6IiIiIqpHGQmFiYiKkUqmmDkdERERE1Ujtawr37NlTbnt2djbi4uJw8uRJDBw4UGOFEREREVH1UTsUTp8+HRKJBOU9KllPTw8DBw7EjBkzNFocEREREVUPtUPhpk2bVNokEgnMzc3RoEEDGBkZabQwIiIiIqo+aodCT09PbdZBRERERDXolRaa3Lt3D6mpqXj48KGm6iEiIiKiGlBpKExLS8OGDRvw4MEDpXaFQoExY8agU6dOGDJkCHx9fREWFqbVQomIiIhIeyoNhVu3bsXPP/8MCwsLpfavv/4ap06dQoMGDdCtWzeYm5tj+fLliI6O1mqxRERERKQdlV5T+Oeff6JDhw5Kbf/++y+OHTsGJycnbN++HQYGBlAoFOjfvz+2b98OPz8/rRZMRERERJpX6UhhRkYG7O3tldri4+MBAIGBgTAwMAAAWFpa4j//+Q8uXLignSqJiIiISKsqDYWPHz+GqampUltqaiokEgm8vLyU2hs2bIjs7GzNV0hEREREWldpKLS1tcXNmzeV2s6ePQszMzM0btxYqb2kpATGxsaar5CIiIiItK7SUOjq6oo9e/YgIyMDwNNAeOnSJfj4+Kj0vXLlCmxsbLRTJRERERFpVaULTcaNG4fDhw/D398fTZo0wZUrV6Cjo4Pg4GCVvidOnFCZUiYiIiKi2qHSkUInJyeEhYWhfv36uHTpEho0aIBFixbBw8NDqd/vv/+OrKwslZXKRERERFQ7vPAxd507d0bnzp0r7fPee+/h7NmzGiuKiIiIiKrXKz3mjoiIiIjeDAyFRERERPTi6ePXwfTp0xEREVHh9pMnT6JevXoICgpCYmKiyvZevXph0aJFSm2FhYVYsmQJIiMj8fDhQzg5OeGzzz4rd2U1ERER0ZuuVoTCIUOGqIQ1QRDw7bffws7ODvXq1RPb69evj08//VSpr52dncoxp0+fjqioKAQHB6Nx48aIiIjA2LFjsXnzZri7u2vnRIiIiIheU7UiFLq7u6sEteTkZDx58gR9+vRRajczM0Pfvn0rPV5qair279+PGTNmICQkBAAQEBCA3r17Y8GCBQgPD9do/URERESvu1p7TeFvv/0GiUSC3r17q2wrLi7Go0ePKtz30KFD0NfXx6BBg8Q2qVSKgQMH4vTp0+LNuomIiIjeFrUyFBYVFeHgwYNwd3dHgwYNlLZdvXoVbm5u8PDwQPv27bFq1SqUlpYq9UlLS0OTJk1UHsvXsmVLCIKAtLQ0rZ8DERER0etE7enjvLw8fPTRR5g+fTqaN2+uzZpe6NSpU8jOzlaZOm7YsCG8vLwgl8uRl5eH3377DYsWLcLt27cxa9YssV9mZqbSdYhlrK2tAeClRgqtrEyqvM/ryNratKZLoJf0pv/dvennR0RU09QOhUVFRUhMTEROTg4A4PHjx5g9ezbGjBkDBwcHrRVYnt9++w36+vrw9/dXav/++++VXvfr1w+ffPIJtm/fjpCQEDRt2hQAkJ+fD319fZXjSqVSAEBBQUGVa8rKykNpqSC+rq0/wDIzc2u6hNdCbfz7q8rf3Zt+fkREVD4dHUmFA1mVTh9//PHH2LhxI1JSUlBYWKi0raCgAHv27Kn26+8ePXqEo0ePon379rCwsHhh/9DQUAiCgISEBLHN0NAQRUVFKn3LwmBZOCQiIiJ6W1Q6UvjkyRMsX74cubm50NPTg0QiwcGDB2FkZIQGDRpAEITKdteK6OjoclcdV8TW1hYAxBFO4Ok0cXlhNjMzEwBgY2OjgUqJiIiIao9KQ+HatWshCAIuXryI2NhYzJ8/H/v27cP27dthZGQEiUSCEydOwNzcHM7OzpBIJFoveN++fTAyMkKXLl3U6p+eng4AsLS0FNucnJywefNmPHr0SGmxSUpKiridiIiI6G3ywtXHEokETk5O6N+/PwBgxYoViIyMxNixYyEIAsLDwzFgwAB4enrigw8+0GqxCoUCcXFx6NatG+rUqaO0LS8vT2WKu6SkBKtXr4aOjo7Sza979uyJoqIi7NixQ2wrLCzE7t274eHhUe4iFCIiIqI3WaUjhaNHj0br1q3RunVrNGzYEMDTkCiXy2FtbY0lS5Zg9erVMDMzQ1JSEpKTk7Va7IEDB1BcXFzu1PH58+fx+eefo3fv3mjUqBEeP36MgwcP4ty5cxg7dqxYPwC0atUKPXv2xIIFC5CZmYlGjRohIiICt2/fxpw5c7R6DkRERESvo0pDoYGBATZv3oylS5dCV1cXEolEfAZx2UpeXV1dtGjRAi1atEBoaKhWi923bx+srKzg6+ursq1+/frw8PBAVFQU7t+/Dx0dHTg6OuKHH35Av379VPrPmzcPixcvRmRkJHJyciCXy7FmzRq0bt1aq+dARERE9DqqNBSuXLkSAHDjxg3ExsZi9uzZOH78OCIjIyGVSiGRSBAVFQVDQ0O4urpCT0+7T83btm1bhdsaNmyIpUuXqn0sqVSKadOmYdq0aZoojYiIiKhWUyvF2dvbw9zcHLNnz8aSJUtga2uLI0eO4Mcff0RERAS2bNmCOnXqoFWrVti4caOWSyYiqpi5WR0YSGvFY91FhQXFyHn4pKbLIKK33Ev9y9mkSRMMGjQIP/74I1asWAEbGxskJCRo/ZpCIqIXMZDq4fuvdtZ0GVXy5f8G1nQJRETqh0KpVIp+/fqVew8/BwcHODg4IDAwUKPFEREREVH1UDsUGhkZKa3MrSwkEhEREVHt8tIX3jwfEomIiIio9nrhzauJiIiI6M2ndih8/PgxZsyYgatXr2qzHiIiIiKqAWqHwoKCAuzZswcZGRkAgPz8fISFheHWrVtaK46IiIiIqkeloXDu3Lk4cuQI7t+/DwAQBEHc9uTJEyxfvhzp6enarZCIiIiItK7ShSYxMTHizajr1asHiUSC5ORkODo6QldXVykkEhEREVHtVelI4YEDB/DHH39g2bJl8PPzgyAIWL58Od577z0MGzYMEokE586dQ3Z2dnXVS0RERERa8MJrCi0sLODn54ePPvoIABAWFobly5fD09MTgiBg4cKF8PX1RZ8+fTBr1iytF0xEREREmlfp9PE333yDNm3awMPDA8bGxgAAY2Nj+Pj4wM3NDdu3b8eCBQugp6eHpKQkPuaOiIiIqJaqNBT++eef2LlzJwRBQN26dSGRSBAbG4sGDRrAxMQEAGBlZQUfHx/07NmzWgomIiIiIs2rNBTu3bsXeXl5SElJQXx8PNauXYt169Zh3bp1sLW1hUQiwdmzZ+Hg4MDH3RERERHVYi+8ptDExATt2rVDaGgoAGDlypVYu3YtOnfuDEEQsHTpUnTs2BHdu3fHV199pfWCiYiIiEjzqvzsY6lUCh8fHzRv3hxbtmxBWFgY9PX1kZiYyGsKiYiIiGoptUOhvr4+2rZtC3NzcwCARCIB8P8LTzp27KidComIiIhI69QOhSYmJti8ebP4+vmQSERERES1V5Wnj8s8HxKJiIiIqPZ64UITIiIiInrzMRQSEREREUMhERERETEUEhEREREYComIiIgIlYTCsLAwXLp0SXx9+/Zt5OfnV0tRRERERFS9Kg2FFy9eFF937doVR44cqZaiiIiIiKh6VRgKzczM8PDhQ/G1IAjVUlB5EhISIJfLy/1z9epVpb5nzpzBsGHD0KpVK7Rr1w7//e9/8eTJE5VjFhYWYv78+Wjfvj1atmyJwYMHIy4urrpOiYiIiOi1UuHNq52dnbFu3ToUFxeLTy1JTk5GSUlJpQcMCAjQbIXPGDlyJFxcXJTa6tWrJ/5/WloaQkJC0KxZM0yfPh13797F+vXrcevWLaxatUppv+nTpyMqKgrBwcFo3LgxIiIiMHbsWGzevBnu7u5aOwciIiKi11GFoXDGjBmYOHEi5syZA+Dps463bduGbdu2VXgwiUSi1VDo6ekJPz+/Crf/+OOPqFu3LjZv3gxjY2MAQIMGDfD1118jLi4OPj4+AIDU1FTs378fM2bMQEhICICnYbZ3795YsGABwsPDtXYObwoLcwPoGUhruowqKS4swIOcwpoug4iI6LVUYSh0cnLC4cOHkZ6ejszMTAQFBeHDDz+Er69vddanIi8vD4aGhtDT01Np/+OPPzB69GgxEAJA37598f333+PgwYNiKDx06BD09fUxaNAgsZ9UKsXAgQOxaNEiZGRkwMbGpnpOqJbSM5Di9LwxNV1GlbSe+hMAhkIiIqLyVPrsY11dXdjb28Pe3h5t27aFl5cXPD09q6s2FVOmTMHjx4+hp6cHLy8vTJs2DXK5HABw8eJFFBcXw9XVVWkfAwMDODs7Iy0tTWxLS0tDkyZNlMIjALRs2RKCICAtLY2hkIiIiN4qlYbCZ23evFmbdVRKX18fPXr0QIcOHWBhYYGLFy9i/fr1CAwMxM6dO9GkSRNkZmYCAKytrVX2t7a2xp9//im+zszMVLoW8dl+AJCRkVHlGq2sTKq8z+vI2tq0pkvQqjf5/N7kcwN4fkRE2qZ2KASA0tJSRERE4MiRI7h16xaAp9fsde/eHQEBAdDR0c69sD08PODh4SG+7tq1K7p06YIBAwYgLCwMCxcuFO+haGBgoLK/VCpVusdifn4+9PX1y+0HAAUFBVWuMSsrD6Wl/79Cu7b+A5+ZmatWP57f60fdcwPe7POrjecGVO3vj4joZenoSCocyFI7FObn52Ps2LFITk6GRCIRR9VOnjyJmJgY7NmzB2vXrhWDlbY5OTnBx8cH8fHxAABDQ0MAT28187yCggJxe1nfoqKicvsBqLZzICIiInpdqD20t3LlSiQlJWHUqFGIi4tDTEwMYmJiEB8fj9DQUCQmJmLlypXarFXFu+++i5ycHAD/P/VbNo38rMzMTKVrBK2trcudIi7bl9cTEhER0dtG7VB44MAB+Pv7Y+rUqeJ9C4GnN7meMmUK/P39sX//fq0UWZH09HRYWFgAAGQyGfT09HDu3DmlPoWFhUhLS4Ozs7PY5uTkhOvXr+PRo0dKfVNSUsTtRERERG8TtUPh3bt3K1153LZtW9y9e1cjRT1PoVCotCUnJyMhIQHt27cHAJiamsLHxweRkZFKYS8yMhKPHz9Gz549xbaePXuiqKgIO3bsENsKCwuxe/dueHh4lLsIhYiIiOhNpvY1hWZmZrh582aF22/evAkzMzONFPW8Tz/9FHXq1IG7uzssLCxw+fJlbNu2DRYWFpg0aZLY77PPPsPQoUMRFBSEQYMG4e7du9iwYQM6dOigdH/FVq1aoWfPnliwYAEyMzPRqFEjRERE4Pbt2+LNuomIiIjeJmqHQl9fX4SHh8PX1xfvvfee0rZTp05h69atSqNxmuTn54d9+/Zhw4YNyMvLg6WlJXr37o1Jkyahfv36Yj8XFxds2LABCxYswJw5c2BiYoLBgwdj8uTJKsecN28eFi9ejMjISOTk5EAul2PNmjVo3bq1Vs6BiIiI6HWmdij89NNPcerUKYwbNw7Ozs5wdHQEAFy+fBlpaWmwsLDAxx9/rJUig4ODERwcrFbfNm3a4Ndff31hP6lUimnTpmHatGmvWh4RERFRrad2KLSzs8OuXbuwcOFCHD9+HBcuXAAAGBsb4/3338fkyZOVRu2IiIiIqPao0s2r69evj4ULF0IQBHHxh6WlJSQSiVaKIyIiIqLqUaVQWEYikcDKykrTtRARERFRDdHOc+mIiIiIqFZhKCQiIiIihkIiIiIiYigkIiIiIjAUEhEREREYComIiIgIVQiFeXl5CA4OFm9aTURERERvDrVDYVFRERITE5GTkwMAePz4MWbMmIGrV69qrTgiIiIiqh6VhsKPP/4YGzduREpKCgoLC5W2FRQUYM+ePcjIyNBqgURERESkfZU+0eTJkydYvnw5cnNzoaenB4lEgoMHD8LIyAgNGjSAIAjVVScRERERaVGloXDt2rUQBAEXL15EbGws5s+fj3379mH79u0wMjKCRCLBiRMnYG5uDmdnZz4DmYiIiKiWeuE1hRKJBE5OTujfvz8AYMWKFYiMjMTYsWMhCALCw8MxYMAAeHp64oMPPtB6wURERESkeZWOFI4ePRqtW7dG69at0bBhQwBPQ6JcLoe1tTWWLFmC1atXw8zMDElJSUhOTq6WoomIiIhIsyoNhQYGBti8eTOWLl0KXV1dSCQSREREAACaNm0KANDV1UWLFi3QokULhIaGar9iIiIiItK4SkPhypUrAQA3btxAbGwsZs+ejePHjyMyMhJSqRQSiQRRUVEwNDSEq6sr9PQqPRwRERERvabUuk+hvb09evXqBQBYsmQJDh48iAkTJkAQBERERGDo0KFo27YtQkJCtFkrEREREWnJSz3mrkmTJhg0aBCApwtP9u/fjylTpsDS0lKjxRERERFR9VB7vlcqlaJfiUzvmAAAHopJREFUv36wsbFR2ebg4AAHBwcEBgZqtDgiIlJmbmYAA6m0psuoksKCAuQ8LHxxRyKqUWqHQiMjI8yZM0d8XVlIJCIi7TCQSvHjjNp1+6/Jc1YDYCgket299MqQ50MiEREREdVeL3VNIRERERG9WRgKiYiIiIihkIiIiIgYComIiIgIr7DQpDqlpqYiIiICCQkJuH37NurWrQt3d3d8+umnaNy4sdgvKCgIiYmJKvv36tULixYtUmorLCzEkiVLEBkZiYcPH8LJyQmfffYZfHx8tH4+RERERK+bWhEKf/rpJ5w5cwY9e/aEXC5HZmYmwsPDERAQgJ07d8LBwUHsW79+fXz66adK+9vZ2akcc/r06YiKikJwcDAaN26MiIgIjB07Fps3b4a7u7vWz4mIiIjodVIrQmFIyP+1d+9hUVV7H8C/IAMIAgKi6YiKJoNGIGBcknN8Q/SQecHQQMVLXEwzE+uomHUq6hwvoWJqSmqlpeIVAW9pqHVS0FIUL6hJYHAUGERgEIYB3O8fvszbNNSBGNgMfD/Pw/PI2pu9v2vmwfmx19prz0RsbCyMjY3VbaNHj8bYsWOxefNmLF++XN1uaWmJ8ePH/+HxMjMzcfjwYSxZskT9aL7AwECMGTMGsbGx2LFjR4v0g4iIiKit0os5he7u7hoFIfD4ecwDBw5Edna21v61tbV4+PDh7x7v2LFjkEgk6kf1AY8X4544cSIuXLiAoqIi3YUnIiIi0gN6URQ2RBAEFBcXw9raWqM9OzsbQ4YMgbu7O3x9fbFp0yY8evRIY5+srCw4ODjA3Nxco93FxQWCICArK6vF8xMRERG1JXoxfNyQ5ORkFBYWYsGCBeo2e3t7eHl5QSaToaKiAocOHcKaNWtw9+5dxMTEqPeTy+Xo0aOH1jHt7OwA4E9dKbS17fInetH22NlZiB2hRbXn/rXnvgHsn75r7/0jag/0sijMzs5GTEwMPDw8NOYP/utf/9LYb8KECZg/fz727NmDmTNnon///gAApVIJiUSidVyT/3vIfHV1dZMz3b9fgUePBPX3+vofoFyuaNR+7F/b09i+Ae27f/rYN4D9I6LWYWho8LsXsvRu+Fgul+OVV16BlZUV1q5dC0PDP+5CWFgYBEHAuXPn1G2mpqaoqanR2re+GKwvDomIiIg6Cr26UqhQKBAZGQmFQoFdu3aph3v/yBNPPAEAKCsrU7fZ2dk1OEQsl8sBAN27d9dRYiIiIiL9oDdXCqurqzF79mzk5uYiPj5ePRT83+Tl5QEAbGxs1G1OTk7IycnRukP58uXL6u1EREREHYleFIV1dXWIiorCpUuXsHbtWgwZMkRrn4qKCqhUKq2fi4+Ph6GhocaTSgICAlBTU4O9e/eq21QqFQ4cOAB3d/cGb0IhIiIias/0Yvh4+fLlOHnyJJ577jmUlpYiKSlJvc3c3Bz+/v64du0a3nzzTYwZMwZ9+vRBZWUljh49iqtXryIyMhL29vbqn3F1dUVAQABiY2Mhl8vRp08fJCYm4u7du1i2bJkYXSQiIiISlV4UhTdu3AAAnDp1CqdOndLYJpVK4e/vj169esHd3R3Hjx9HcXExDA0NMXDgQCxfvhwTJkzQOubKlSsRFxeHpKQklJWVQSaT4dNPP4WHh0er9ImIiIioLdGLovDLL7/8r/vY29vj448/bvQxTUxMsHjxYixevLg50YiIiIjaBb2YU0hERERELYtFIRERERGxKCQiIiIiFoVEREREBBaFRERERAQWhUREREQEFoVEREREBD1Zp5CIiDoGa6vOMDLWr4+mWlUtHpRViR2DqNn06zePiIjaNSNjI1z+5LTYMZrE9dX/ETsCkU5w+JiIiIiIWBQSEREREYtCIiIiIgKLQiIiIiICi0IiIiIiAotCIiIiIgKLQiIiIiICi0IiIiIiAotCIiIiIgKLQiIiIiICi0IiIiIiAotCIiIiIgKLQiIiIiICi0IiIiIiAotCIiIiIgJgJHYAIiKijsLKyhTGxhKxYzSaSlWDsjKl2DGolbAoJCIiaiXGxhKsWrVK7BiN9uabbwJgUdhRcPiYiIiIiDp2UahSqfDRRx/B19cXLi4ueOmll5CWliZ2LCIiIqJW16GLwujoaGzbtg3jxo3D0qVLYWhoiMjISGRkZIgdjYiIiKhVddiiMDMzE4cPH8bf//53LFq0CMHBwdi2bRt69uyJ2NhYseMRERERtaoOe6PJsWPHIJFIMGnSJHWbiYkJJk6ciDVr1qCoqAjdu3cXMSEREZF+se5qAiOJsdgxGq22RoUHpdVix2gzOmxRmJWVBQcHB5ibm2u0u7i4QBAEZGVlNakoNDQ00GrrZm3ewJ5tW0P9+D3GlrYtmKRlNKV/3brYtGAS3WtK3wCgczf9ev+a0j+rrmYtmKRlNKV/ll31670DmtY/iYVpCyZpGU16/ywtWzCJ7jWlb0YSY5xLjWvBNLrlNSIKhoYqsWO0qj96Pw0EQRBaMUubMWbMGPTo0QNbt27VaL99+zZeeOEFfPjhhxpXEYmIiIjasw47p1CpVEIi0V5A1MTEBABQXc3LyURERNRxdNii0NTUFDU1NVrt9cVgfXFIRERE1BF02KLQzs4ORUVFWu1yuRwAeJMJERERdSgdtih0cnJCTk4OHj58qNF++fJl9XYiIiKijqLDFoUBAQGoqanB3r171W0qlQoHDhyAu7s7evToIWI6IiIiotbVYZekcXV1RUBAAGJjYyGXy9GnTx8kJibi7t27WLZsmdjxiIiIiFpVh12SBnh8U0lcXBxSUlJQVlYGmUyGN954A88++6zY0YiIiIhaVYcuComIiIjosQ47p5CIiIiI/h+LQiIiIiLquDea6JOioiJs374dly9fxtWrV1FZWYnt27fDy8tL7GjNlpmZicTERJw7dw53795F165d4ebmhqioKPTt21fseM125coVbNq0CdevX8f9+/dhYWEBJycnzJ07F+7u7mLH07nNmzcjNjYWTk5OSEpKEjtOs507dw7Tp09vcNuRI0cwYMCAVk6ke5mZmVi/fj0yMjJQW1sLe3t7zJw5Ey+++KLY0ZolOjoaiYmJv7v9u+++0/tVJnJzcxEXF4eLFy+ivLwcvXr1QmBgIGbOnAljY2Ox4zXbpUuXsGbNGmRmZsLQ0BBeXl6Ijo5Gnz59xI7WJE35DE9NTcX69etx+/Zt2NraYuLEiZg9ezaMjFqnXGNRqAdycnKwefNm9O3bFzKZDBkZGWJH0pktW7bg4sWLCAgIgEwmg1wux44dOxAYGIh9+/bp/YduXl4e6urqMGnSJNjZ2UGhUCAlJQWhoaHYvHkzhg0bJnZEnZHL5di4cSPMzMzEjqJzM2bMwFNPPaXRpu8FBQB8++23mDt3Ljw9PTF//nwYGRkhNzcX9+7dEztaswUHB8PHx0ejTRAEvPfee5BKpXr//hUWFmLSpEmwsLBAaGgorKys8OOPP2LVqlX46aef8NFHH4kdsVkyMzMRGhoKqVSKefPm4dGjR9i5cyemTJmCgwcPolu3bmJHbLTGfobX/z56e3vjnXfewa1bt7BhwwY8ePAA77zzTuuEFajNUygUQklJiSAIgnDixAnB0dFRSE9PFzmVbly4cEGorq7WaMvJyRGcnZ2FxYsXi5SqZVVWVgrPPvusMGvWLLGj6NTixYuFadOmCaGhocK4cePEjqMT6enpgqOjo3DixAmxo+hceXm54OPjI3zwwQdiR2k1P/zwg+Do6Chs3LhR7CjNFh8fLzg6Ogq3bt3SaJ83b54wePBgQaVSiZRMN8LDwwVPT0+htLRU3VZYWCgMGTJE+PDDD0VM1nSN/QwfPXq0MGHCBKG2tlbdtnr1asHJyUnIyclplaycU6gHunTpAmtra7FjtAh3d3etYY5+/fph4MCByM7OFilVy+rcuTNsbGxQXl4udhSdyczMRHJyMpYsWSJ2lBZTUVGB2tpasWPoTEpKCsrLyzF//nwAj/sntPPFKA4dOgQDAwOMGTNG7CjNVv80LltbW432bt26wcjICJ06dRIjls5cvHgRvr6+sLKyUrd1794dnp6eOHr0qIjJmq4xn+G3b9/G7du3ERwcrPHeTZkyBY8ePcLx48dbOiYA3mhCbZAgCCguLm5XhXBFRQVKSkrw888/Y/Xq1bh165bW0Ja+EgQBH3zwAQIDAzFo0CCx47SIhQsXwsPDA66urggLC8PNmzfFjtRsaWlp6N+/P7799lsMHz4cHh4e8PT0RGxsLOrq6sSOp3M1NTU4evQo3Nzc0Lt3b7HjNNszzzwDAFi6dClu3LiBe/fuITk5GYmJiYiMjIShoX5/vKtUKpiYmGi1m5qaQi6Xo6ioSIRULef69esAAGdnZ432Hj164IknnlBvb2mcU0htTnJyMgoLC7FgwQKxo+jMW2+9ha+//hoAIJFIEBISgtmzZ4ucSjcOHjyI27dvY8OGDWJH0TmJRIK//e1v+Otf/wpra2vcvHkTn332GaZMmYJ9+/bBwcFB7Ih/2p07d1BQUIDo6GhERERg8ODBOHXqFDZv3ozq6mosXbpU7Ig69f3336O0tBRjx44VO4pO+Pr6Yv78+YiPj8fJkyfV7a+//jrmzp0rYjLdcHBwwKVLl/Do0SN1gatSqZCZmQng8c0b3bt3FzOiTsnlcgCAnZ2d1jY7O7tWK4JZFFKbkp2djZiYGHh4eGD8+PFix9GZuXPnIjg4GAUFBUhKSoJKpUJNTY3e3yFYUVGBVatWYdasWe3qP+h67u7uGneJjxgxAn5+fggKCsL69euxatUqEdM1T2VlJcrKyvDmm29i1qxZAIBRo0ahsrISu3btwpw5c2BjYyNySt05dOgQJBIJnn/+ebGj6Ezv3r3h6emJkSNHomvXrjh9+jTWrVsHGxsbTJ48Wex4zTJlyhS89957ePvttxEWFoZHjx5h48aN6uJJqVSKnFC36vvT0GeCiYkJqqqqWiUHi0JqM+RyOV555RVYWVlh7dq1ej/88WsymQwymQwAMG7cOAQFBWHJkiX4+OOPRU7WPBs3boREIsHLL78sdpRW4+TkBB8fH6Snp4sdpVlMTU0BQGt+3dixY3Hs2DFcuXIFw4cPFyOazj18+BCpqanw9fVtN9NSDh8+jHfffRfHjh1T30k9atQoCIKAlStXYvTo0Rrz8fTN5MmTUVBQgK1bt2L//v0AHg+thoeHY9OmTTA3Nxc5oW7V/z6qVCqtbdXV1ertLa39fOqSXlMoFIiMjIRCocCWLVsavITeXkgkEowYMQLHjx/X6792i4qKsG3bNkyZMgXFxcXIz89Hfn4+qqurUVNTg/z8fJSVlYkds0X07NlT7/tW/zv226U96r/X9/792jfffIOqqqp2M3QMADt37sRTTz2ltbSOn58fKisrcePGDZGS6c6CBQtw5swZ7NixA8nJydi/fz8EQYCBgQHs7e3FjqdT9b+P9VdCf00ul7faSAyLQhJddXU1Zs+ejdzcXMTHx6N///5iR2pxSqUSgiCo7yDUR/fv30dNTQ1iY2MxYsQI9dfly5eRnZ2NESNGYPPmzWLHbBF5eXl6f8Wpft3FwsJCjfaCggIAaFdDxykpKTAzM4Ofn5/YUXSmuLi4wRuCampqAKDd3CxkZWWFoUOHqkdazp49CxcXF3Tp0kXkZLpVf5Pe1atXNdoLCwtRUFDQajfxsSgkUdXV1SEqKgqXLl3C2rVrMWTIELEj6VRJSYlWW0VFBb7++mv07NlTazkJfdK7d29s2LBB62vgwIGQSqXYsGEDAgMDxY7ZLA29fz/++CPOnTsHX19fERLpTkBAAABg37596jZBELB3716YmZm1m9/FkpISpKWlYeTIkejcubPYcXTGwcEBV69exS+//KLRfvjwYXTq1EldRLUnR44cwZUrVzBjxgyxo+jcwIED0b9/f+zevVujoN+1axcMDQ0xatSoVsnBOYV64pNPPgEA9dp9SUlJuHDhAiwtLREaGipmtGZZvnw5Tp48ieeeew6lpaUaj0YzNzeHv7+/iOmaLyoqCiYmJnBzc4OdnR3u3buHAwcOoKCgAKtXrxY7XrNYWFg0+P5s27YNnTp10vv3Dnj8/nXu3Blubm6wtrbGTz/9hN27d8Pa2hrz5s0TO16zODs7IzAwEPHx8bh//z4GDx6Mb7/9Ft9//z0WLlzYbq7EHDlyBLW1te1q6BgAwsPD8d1332Hy5MmYOnUqrKyscPr0aXz33XcICQnR6z84gcdLJsXHx2PYsGHo2rUrLl26hMTERIwdOxYvvPCC2PGarDGf4YsWLcKcOXMQHh6O0aNH49atW9ixYweCg4NbbaUDA6G9r1baTvzeX31SqVRjOQJ9M23aNJw/f77BbfreN+DxVZikpCTcvn0b5eXlsLCwwJAhQxAWFgZPT0+x47WIadOmoby8vF08+3j79u1ISUnBL7/8goqKCtjY2MDX1xfz5s1Dr169xI7XbCqVCp988gkOHjyI4uJi9O7dGzNnzkRISIjY0XQmODgYeXl5+Pe//633Czr/VmZmJtatW4esrCyUlpZCKpUiKCgI4eHhet/X3NxcxMTE4Pr163j48CH69euHSZMmITQ0VC9vQmzsZ/g333yD9evXIzs7GzY2NggKCsKrr77aas8+ZlFIRERERJxTSEREREQsComIiIgILAqJiIiICCwKiYiIiAgsComIiIgILAqJiIiICCwKiYiIiAgsComoHYqOjm43j/k6cOAAZDIZzp07J3aUVuXn54dp06aJHYOoQ+Fj7ohINHl5efj000/xww8/4N69ezA2Nka3bt3g4uKCCRMmwNvbW+yIeiU6OhqJiYlIS0uDjY2N2HH+qy+++AKWlpZ48cUXxY5CRGBRSEQiuXLlCqZNmwYjIyMEBgbiySefhFKpxJ07d3DmzBmYm5uzKGzntm/fDqlUyqKQqI1gUUhEotiwYQOqqqqQlJQEJycnre1yuVyEVK2rrq4OKpUKnTt3FjsKERHnFBKROHJzc9G1a9cGC0IAsLOzU/87Pz8fMpkM69at09pv3bp1kMlkyM/P19pWUlKCRYsWwcvLC0OGDMGMGTNw7do1jX38/PwQGhqq0RYfHw+ZTIY5c+ZotH/00UeQyWQoLi7WOMf777+P4cOHw9nZGcOHD8f777+PBw8eaPxs/dzAs2fPYsOGDfD394eLiwuOHj2q3mfPnj0ICAiAs7MzRo4ciS+++AIt9Xj63NxcLFy4EL6+vnB2doafnx9WrFiByspKjf3q52cqFAq8++678PHxwdNPP42QkBBcvnxZ67gPHjzAkiVL4OXlBTc3N0yfPh3Xr1/HtGnT4Ofnp95PJpPhP//5D86fPw+ZTKb++u37mJ2djVmzZsHNzQ0eHh54/fXXO8QfDERi4JVCIhJFnz59kJOTg+PHj2PUqFEtco6IiAhYWVnhtddeQ3FxMb766iuEhoZi9+7dcHR0BAB4e3sjOTkZSqUSpqamAIC0tDQYGhrihx9+QF1dHTp16gQASE9Px5NPPolu3boBABQKBSZPnow7d+4gKCgIgwcPRlZWFnbt2oX09HTs3bsXXbp00ci0YsUK1NbW4qWXXoK5uTkcHBwAPJ5ft2zZMjg5OeGNN95AVVUVPvvsM9ja2ur8dbl69SpmzJgBS0tLBAcHo0ePHrhx4wa+/PJLZGRk4Msvv4REItH4mfDwcNjY2GDu3LkoLS3F559/jlmzZiE1NVXdR5VKhZdffhlZWVl48cUX8fTTT+PmzZt4+eWXYWVlpXG8lStXYtmyZbC2tsbs2bPV7b+eC1lYWIjp06fD398fixYtwo0bN7B7925UVFTgs88+0/nrQtTRsSgkIlHMmTMHZ8+exbx589CvXz+4u7vj6aefhpeXFwYMGKCTc/Tq1Qvr1q2DgYEBAGDkyJGYOHEiVqxYga1btwJ4XBTu378fFy5cwLBhw6BSqZCRkYExY8YgOTkZ165dg4uLCxQKBbKysjB58mT18bds2YLc3Fz84x//wNSpU9XtgwYNQkxMDLZs2YKoqCiNTEqlEgcPHtQYMi4vL0dcXBwGDBiAhIQE9bagoCA8//zzOnktfu2tt96CnZ0d9u3bp1G0+vj44LXXXkNKSorWPL/BgwfjvffeU38/YMAAREVF4dChQwgJCQEA7N27F1lZWYiKitK4yuro6IiYmBhIpVJ12/jx47F27Vp069YN48ePbzDnnTt3sGbNGowePVrdZmhoiJ07d+Lnn39G//79m/U6EJEmDh8TkSjc3Nywf/9+TJgwAQqFAgcOHMD777+P0aNHY+rUqcjLy2v2OSIiItQFIQA4Oztj2LBhSEtLw8OHDwFAfTNLeno6ACAjIwNKpRIRERGwsLBAWloaAOD8+fOoq6vTuPnlxIkTsLGxQXBwsMZ5g4ODYWNjg2+++UYr0+TJk7XmEH7//feoqqrC1KlTNbY98cQTGDt2bHNeAi03b97EzZs3MWbMGKhUKpSUlKi/PDw8YGZmhjNnzmj93MyZMzW+r38d7ty5o247deoUOnXqhOnTp2vsO2nSJFhYWDQ5a/fu3TUKwt87LxHpBotCIhKNTCbD8uXLcfbsWZw8eRIrVqzA0KFD8eOPP+LVV1+FSqVq1vEbuuI4YMAA1NXV4e7duwAeFx79+/dXF4Xp6emws7ODTCbDM888o9FuaGgIT09P9bHy8/Ph4OAAIyPNQRcjIyP069evwcK2frj41+rn0TV05UtXV03rZWdnA3g8F9PHx0frq7KyUmPOZD17e3uN762trQEApaWl6rb8/Hx0794d5ubmGvsaGxujd+/eTc7623MCQNeuXbXOS0S6weFjImoTpFIppFIpxo8fjylTpuDixYvIzMzE0KFDNa72/VZtbW2zz+3t7Y3du3dDoVAgPT0dXl5e6vbVq1dDpVIhPT0dgwYN0pob11T18xbFFhYWhr/85S8NbrO0tNRqq59X+VstdSPMH52zpc9L1FHxSiERtSkGBgZwdXUFABQVFQGAuhArKyvT2r+hu47r1V8V+21bp06d0KtXL3Wbt7c36urqcPr0aVy5ckU9ROnj4wOlUonU1FT89NNP6mKxnr29PXJycrQK09raWuTm5jZ4pash9VfRfv7550b1oTn69u0L4PHcvGeffbbBL2dn5z91bKlUiqKiIvXQfL2ampo/fJ+IqG1gUUhEojhz5kyDV/mUSqV6Tlv90GmXLl1gZ2eH9PR0jStEeXl5Dc7bq7dlyxaN/a9du4azZ8/Cx8dHY4jTy8sLBgYG2LhxI2pqatRFoaOjI2xtbbF+/XoIgqC1mLa/vz9KSkqwd+9ejfY9e/agpKQE/v7+jXothg0bBlNTU+zYsQNVVVXq9oKCAqSkpDTqGI01ePBgODo6IiEhocHh7dra2j89NOvn54e6ujps375do33Pnj1QKBRa+5ubm3MYmKgN4fAxEYli2bJlKC0thZ+fHxwdHWFqaqougnJzcxEYGKjx/OKpU6ciLi4OERER8Pf3R1FRERISEjBw4EBcuXKlwXPcvXsX4eHh8PPzg1wux1dffQVTU1MsXLhQY7/69RKzsrIglUo1rvB5eXnhyJEjkEgkGDp0qMbPRURE4NixY4iJicH169cxaNAgZGVlYd++fXBwcEBERESjXgsrKyvMnz8fK1asQEhICAIDA1FVVYWEhAT069cP169fb+zLCuDx8jYNDVN7e3vD3d0dK1euxIwZMzBu3DgEBQVpPE3mxIkTeOONN/7UU0YmTZqEhIQExMXF4ZdfflEvSXPs2DH07dtX648AV1dX7Nu3T33ntaGhIZ577jmYmZk1+dxE1HwsColIFNHR0UhNTcWFCxfw9ddfQ6FQwMLCAo6OjoiMjNQqSiIjI6FQKJCcnIzz58/jySefxD//+U9cu3btd4vCLVu2YNmyZVi3bh2USiVcXV2xaNGiBhfM9vb2RlZWltbVQB8fHxw5cgTOzs5aN1BYWFhg165d+Pjjj3Hy5EkcOHAAtra2CAkJwbx587TWKPwjYWFhMDMzw+eff45Vq1ahZ8+eCAsLg4WFBd56661GHwd4vPh2Q4yMjODu7o5BgwYhMTER8fHxOHnyJBISEmBubg6pVIoJEybAx8enSeerZ2xsjG3btmHlypVITU3F0aNH4eLigi+++AJLly6FUqnU2H/BggUoKyvDzp07UV5eDkEQkJqayqKQSCQGAmfrEhFRC6pfysfFxUW9PiQRtT2cU0hERDrz26uBAJCQkIDy8nIMGzZMhERE1FgcPiYiIp15++23oVKp4ObmBmNjY2RkZODQoUPo27cvXnrpJbHjEdEf4PAxERHpzMGDB7Fjxw7k5uaisrIStra2GD58OObPn69+ZjQRtU0sComIiIiIcwqJiIiIiEUhEREREYFFIRERERGBRSERERERgUUhEREREYFFIREREREB+F8m3qO0ImQVEwAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": { + "tags": [] + }, + "output_type": "display_data" + } + ], + "source": [ + "sns.countplot(subword_lengths)\n", + "plt.title('Subword Token Lengths (w/o \"##\")')\n", + "plt.xlabel('Subword Length')\n", + "plt.ylabel('# of ## Subwords')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gCDYsO8qyepC" + }, + "source": [ + "### Misspellings" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "p-n_ObF14sh3", + "outputId": "f83abeb9-3a71-4ff0-84fb-b90b2edaf115" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 12, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'misspelled' in tokenizer.vocab # Right" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "FnUu6C_VjTVQ", + "outputId": "b4328880-6532-46a8-d234-7b02c400312c" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 13, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'mispelled' in tokenizer.vocab # Wrong" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "GrKICg1y45Rz", + "outputId": "21fa6373-1df7-4b5d-b166-3c4e9e3040bc" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 14, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'government' in tokenizer.vocab # Right" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "fO72IlrU5I8t", + "outputId": "c8f216b2-b2a4-4cd2-827a-bb4f23703a82" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 15, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'goverment' in tokenizer.vocab # Wrong" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "FfqU2qbpjubX", + "outputId": "bd40e7bd-d43f-4e5c-b366-3d892cd852af" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 16, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'beginning' in tokenizer.vocab # Right" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "Cvzy-sMsjs5P", + "outputId": "d95c6956-4c8e-4104-9c89-caec26157470" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 17, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'begining' in tokenizer.vocab # Wrong" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "WkqxG-675A1c", + "outputId": "81d5f64c-613b-4ec3-94b9-45abb67c7d0f" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 18, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'separate' in tokenizer.vocab # Right" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "ziFBjur15GHg", + "outputId": "1fb3d72c-daa1-40a4-bfeb-c59dad2d9744" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 19, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'seperate' in tokenizer.vocab # Wrong" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-KZHPgVsSgvz" + }, + "source": [ + "What about contractions?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "VvUru5j3jxm2", + "outputId": "4ace3552-22ba-4445-c102-5f5a9642240e" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 20, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "\"can't\" in tokenizer.vocab" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "wl_l2xKwj0_l", + "outputId": "31f882e6-a524-4801-ea2e-aa9a6cc7ba0d" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 21, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "\"cant\" in tokenizer.vocab" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "RE4TqKOlykJD" + }, + "source": [ + "### Start vs. Mid Subwords\n", + "\n", + "For single characters, there are both the individual character and the '##' version for every character. Is the same true of subwords?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "_Xf9FH7HWZya", + "outputId": "6343ff38-d934-4756-9858-582b8cebadd4" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Did not find a token for ly\n" + ] + } + ], + "source": [ + "# For each token in the vocabulary...\n", + "for token in tokenizer.vocab.keys():\n", + " \n", + " # If it's a subword...\n", + " if len(token) >= 2 and token[0:2] == '##':\n", + " if not token[2:] in tokenizer.vocab:\n", + " print('Did not find a token for', token[2:])\n", + " break" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "0rPGb9z8SmYn", + "outputId": "ee8b4c03-11ae-484a-cd6d-45b518a07939" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 23, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'##ly' in tokenizer.vocab" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "wUmQwKqfWx9K", + "outputId": "83048373-aba1-4725-f924-891887483543" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 24, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'ly' in tokenizer.vocab" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PRaaspCfyqsY" + }, + "source": [ + "### Names\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 168 + }, + "id": "s7OLeBs_zBiR", + "outputId": "3bbf6ae5-0e91-4b62-b361-9832b8cec6f1" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting wget\n", + " Downloading https://files.pythonhosted.org/packages/47/6a/62e288da7bcda82b935ff0c6cfe542970f04e29c756b0e147251b2fb251f/wget-3.2.zip\n", + "Building wheels for collected packages: wget\n", + " Building wheel for wget (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for wget: filename=wget-3.2-cp36-none-any.whl size=9681 sha256=97fcba8d47d5f818c5e3dc4b5c355474c4337b758065355989ee2b40ca73ed15\n", + " Stored in directory: /root/.cache/pip/wheels/40/15/30/7d8f7cea2902b4db79e3fea550d7d7b85ecb27ef992b618f3f\n", + "Successfully built wget\n", + "Installing collected packages: wget\n", + "Successfully installed wget-3.2\n" + ] + } + ], + "source": [ + "!pip install wget" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 50 + }, + "id": "9Mac6NiYys4g", + "outputId": "941f034e-9ed9-467a-a549-9cd8b94fe01b" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Beginning file download with wget module\n" + ] + }, + { + "data": { + "text/plain": [ + "'first-names.txt'" + ] + }, + "execution_count": 26, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "import wget\n", + "import random \n", + "\n", + "print('Beginning file download with wget module')\n", + "\n", + "url = 'http://www.gutenberg.org/files/3201/files/NAMES.TXT'\n", + "wget.download(url, 'first-names.txt')\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 50 + }, + "id": "FHF1JgrqY7Zi", + "outputId": "5cea3afc-b61a-437b-dd26-dcb4d30b7e3e" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of names: 21,985\n", + "Example: umberto\n" + ] + } + ], + "source": [ + "# Read them in.\n", + "with open('first-names.txt', 'rb') as f:\n", + " names_encoded = f.readlines()\n", + "\n", + "names = []\n", + "\n", + "# Decode the names, convert to lowercase, and strip newlines.\n", + "for name in names_encoded:\n", + " try:\n", + " names.append(name.rstrip().lower().decode('utf-8'))\n", + " except:\n", + " continue\n", + "\n", + "print('Number of names: {:,}'.format(len(names)))\n", + "print('Example:', random.choice(names))\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "6heL05KQzmCa", + "outputId": "922c4cf9-bbb6-43a3-9b32-f24e4c8a998f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "3,869 names in the vocabulary\n" + ] + } + ], + "source": [ + "num_names = 0\n", + "\n", + "# For each name in our list...\n", + "for name in names:\n", + "\n", + " # If it's in the vocab...\n", + " if name in tokenizer.vocab:\n", + " # Tally it.\n", + " num_names += 1\n", + "\n", + "print('{:,} names in the vocabulary'.format(num_names))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BM99iB36RAxX" + }, + "source": [ + "**Further Research**\n", + "* Add more modern names\n", + " * This repo / file contains some more modern names. The file download isn't working, though.\n", + " * `https://raw.githubusercontent.com/arineng/arincli/master/lib/male-first-names.txt`\n", + "* Add common names from other languages.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3HGvwsy0ynkq" + }, + "source": [ + "### Numbers" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "h5T_-gfQZKm6", + "outputId": "cfa1eb34-095e-4cdf-e4ea-006e62faa29b" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Vocab includes 881 numbers.\n" + ] + } + ], + "source": [ + "# Count how many numbers are in the vocabulary.\n", + "count = 0\n", + "\n", + "# For each token in the vocabulary...\n", + "for token in tokenizer.vocab.keys():\n", + "\n", + " # Tally if it's a number.\n", + " if token.isdigit():\n", + " count += 1\n", + " \n", + " # Any numbers >= 10,000?\n", + " if len(token) > 4:\n", + " print(token)\n", + "\n", + "print('Vocab includes {:,} numbers.'.format(count))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "-M1biDEVYjaL", + "outputId": "80b38422-6d78-404c-deed-887c6efa4cee" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Vocab includes 384 of 421 dates from 1600 - 2021\n" + ] + } + ], + "source": [ + "# Count how many dates between 1600 and 2021 are included.\n", + "count = 0 \n", + "for i in range(1600, 2021):\n", + " if str(i) in tokenizer.vocab:\n", + " count += 1\n", + "\n", + "print('Vocab includes {:,} of 421 dates from 1600 - 2021'.format(count))" + ] + } + ], + "metadata": { + "colab": { + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.16" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/notebooks/ex1_tokenizers.ipynb b/notebooks/ex1_tokenizers.ipynb new file mode 100644 index 0000000..a5f9221 --- /dev/null +++ b/notebooks/ex1_tokenizers.ipynb @@ -0,0 +1,25 @@ +{ + "cells": [ + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Deep dive (optional?) exercise: Play with tokenizers - inspect and compare them" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + } + ], + "metadata": { + "language_info": { + "name": "python" + }, + "orig_nbformat": 4 + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/ex2_count_tokens_in_dataset.ipynb b/notebooks/ex2_count_tokens_in_dataset.ipynb new file mode 100644 index 0000000..9437dd7 --- /dev/null +++ b/notebooks/ex2_count_tokens_in_dataset.ipynb @@ -0,0 +1,6983 @@ +{ + "cells": [ + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's check that our dataset is appropriate for the distilbert tokenizer. Are our examples mostly less than 512 tokens?" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "colab_type": "text", + "id": "view-in-github" + }, + "source": [ + "\"Open" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "meic0kIaymXA", + "outputId": "92d58671-ed6d-46b4-a733-6e3fe05438bf" + }, + "outputs": [], + "source": [ + "# %%capture\n", + "# !pip install transformers\n", + "# !pip install datasets\n", + "# !pip install evaluate\n", + "# !pip install torch\n", + "# !pip install matplotlib" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "id": "XpSP_6FAhEW9" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/Users/d/repos/nlp/env/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", + " from .autonotebook import tqdm as notebook_tqdm\n" + ] + } + ], + "source": [ + "from datasets import load_dataset\n", + "from transformers import AutoTokenizer\n", + "import pandas as pd\n", + "# import matplotlib.pyplot as plt" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Step 1: Load, inspect and down-sample our dataset" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 495, + "referenced_widgets": [ + "a191629c94b6468cba48382e3e274539", + "276aa6f5994e45ed993ef18b4212394c", + "b4d13a5656b048a8b7fa4d0acddceeba", + "7319933cae3246008aa376d395739e6f", + "b27f582bc82345d7b8669b56cf777dd4", + "bd26a0c9556444fa8cda018cf0c88688", + "06a151da1234476cba405feae6f92d2b", + "f4c7191d4cd44935a6d9b6484430682a", + "5b2ddc4b19984e35b4ce710124c3a364", + "a2e09c1699994048b8e587401832014d", + "336c1eab88f5455f8d11ac0b15889a13", + "7501eea7d84b4d27aae128e5844a044b", + "257612fcac154f46a14295d611681280", + "8ef573f925fd4fe2890f695755caa8bd", + "9208d75ae3c4428ca60e1c14e93aae57", + "fc19f1cde36c4322ac32c711458d0271", + "0e28ffed3f724a99bed0d3a07d146a69", + "4d2fb2f73b6c4a678cd35185a0a520b3", + "6b0ba07cc6654481a603e1362be7bc80", + "05139a04863243dc83d3f9fd0d74114b", + "a1c7b3169f4e44a8900cdbac374aee29", + "37cef5abfc9e45fe80991f74014d86c3", + "d6d047266e6f4ba09733ffb6f9a44806", + "1f342452365d47dba07dc1f10167f831", + "efe4f4c187d643e9b6f49a47f8fd5787", + "ba67d14ee081429183f7cf7e8f998871", + "c0de1c77c31844b5a2fae6f4751e96d6", + "f86082c26ab1446398f3e9b609ee25ba", + "ce893b8c74724d698838f5563f9c4d64", + "1ac4807c27e146a192a7c2023d284bcc", + "b494fbde6ba1448185329de69a024677", + "7c0013b42c1647a49c1944322f80b901", + "daad627e1eb046ea8041a63e57ff2193", + "8c8d2028126f4011b160cf9141818a2b", + "64f2866f6452428b8a5cefe9f9b60106", + "9386856839f7481eb5c4f94321f65cfd", + "d903a246b4d74783816546c0a1ba7526", + "9490ca33f6ea42ef97fc481901fed340", + "3911d344c7294faab1cadb53a252e38a", + "b25d3b38f7a547f587044adf123fdec5", + "c51add9f3f674eb6a34a4d0ef43e27e7", + "1a27ecacd77c4865b5bf0941b172342a", + "60dbb47d65474c61a2db10a2ab636818", + "d068d68e344d431d9a45ed804da66b11", + "212a56d5dbd74c44b06b72341b65ee04", + "9e65b453455040e6a5cacd6324274cc7", + "2079c7c036bd45f4af8286ea9d177969", + "b05a81f1bf3f4d52b831f42dd34e79c1", + "f438cd0761c1420db5fcb3a588d305e7", + "5374071fbada4b2d8e65fab009345259", + "390b9eac79164ecc8ec3419617625a5f", + "b28801b13d184fb391676075daa846ef", + "17646882017441eeb39a78699bbe33f6", + "9fd57c931f3a48889e37494d3d072f2a", + "11cf4d8d42c94b1aab7382e1125d3357", + "e906635962a44b798d63ac43e6213aae", + "44b9af84cd8147329fbf55a92260053f", + "fb777c1e52df48eaa76a9cbfe9235c88", + "2f162cdc30c94bb9a421661ff8f40d10", + "5199dd91d2324aca9da01c51ed812107", + "76369a70c01546dc8b9e4d24d6421c5a", + "8456cf2c339e45a2948321680498c13e", + "8641e8ef36ec491a9cec481074faa3ca", + "8692a908c7d84f3ca8e45674280e875f", + "eb966af45cf340b1926c4b903a0e58ea", + "24b2ec25a5f54deaa88a65da11cd646d", + "fac949fce47f4edcb262b62c2a1c50f3", + "e14574af4f234dd39f071c47f944ad54", + "f924ac2de1bb4789831e79551e226e29", + "d61d318d4e104434b395cbb73174037b", + "4a7fab38ff804ab4ae8215f121ee5cb4", + "402c33f8d8c94a16ba341f1ca9384627", + "1a8df02eab7c46498e1f9c60f0f5b04f", + "1417b492f37d4d938616994d6b3ca965", + "c204e25372ad4b4cb7fe390edd6f99d7", + "1a107e4030124c8bbe6b32c5f640fb95", + "a80b889df313482983002f5efe4ab2c6", + "b42d06ad9b354aefa34bf569eb578919", + "ab7bccdbe46148aeb5b91de02db55c57", + "ab09b973df504a009bd93d3473a7e878", + "69c7dcbdbcfb43e085cae897d7be872d", + "43761e8ebb0d4040b65fe74e2bd12ba1", + "8069495c5d0a4a7a8adef81247e2befe", + "de798e94efee48ce8e5cfcbd69e71956", + "7a5b67db46504053b83d857b38f78787", + "fd09ba0f798c4172a94665334bc107d8", + "dfa0242aab26425996c5c2946d31e065", + "f056a241377747f6a8982a0eed36e384", + "35a230bea6a847dcb340aa126536cf7e", + "1f835219eca344df998fece53a82b815", + "3c1ad98935f74925bceecd60d3271dfd", + "88260cb9bdce41149ded15efe96908bf", + "4c58d90f75f746f8a3dd57ea372a61ed", + "a7598e7ea47a4c24adfb2776182dbe67", + "656ab77ac4584e5cb635f721ef106a2d", + "881bdf44ce9648dcbda8391ae22d9c8c", + "e0bc075493664c40a0a3e60567f9ed8e", + "82068e9ff19946289a62c3d2d05f17b6", + "88d8c1e54880470c81fbf48d2e915058", + "bcf0036e72ac469e8a39eae22d76a586", + "0255272bfb8b43fa83aa14c5b6577487", + "f6d2587af20a47f8be304ac0b5daee75", + "521c96ab56cd49c4b6350cfeba2edc7f", + "3d0586a821e942f8a33c1eb957771cef", + "b1045684e25f42a1b673e56ce4b423bc", + "a4f36826230042d88ab6eeeb02eba2b2", + "738a30e9bc964898b4c5b4b6e0d1b25c", + "4111c760131b4d82b5edcd74bebf59b1", + "e5c10bfeb0a643299426a7a8b19fbe3c", + "d4fb3448c39149c08372c0e897ffd7f6", + "08cb8909e871484b96903a77f71a4ce7", + "32e02928eafe4375af034bb65a50e291", + "c4843dcd436b4933b0c427a28b95b34a", + "c5691040b26f4966bf6259c4de77e06d", + "98ffcd00835643e4aac3cfdde3bef4a0", + "d9cea257103e430f9e54e9856ee21267", + "185dd29085e44cdb97e7afe869559f24", + "ec18adf6d12d487fba0056f451198d47", + "251b565174da44309616141bc32c6b53", + "2ffe627ee8f043d1ada61a815b23fb69", + "f9557e28d8374e43b2aa9c9d7aebb909" + ] + }, + "id": "4SzbqB2DTUwk", + "outputId": "f2a49ef0-a577-42c8-b33d-0bce7d6aafa0" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Found cached dataset yelp_review_full (/Users/d/.cache/huggingface/datasets/yelp_review_full/yelp_review_full/1.0.0/e8e18e19d7be9e75642fc66b198abadb116f73599ec89a69ba5dd8d1e57ba0bf)\n", + "100%|██████████| 2/2 [00:00<00:00, 21.84it/s]\n", + "Loading cached split indices for dataset at /Users/d/.cache/huggingface/datasets/yelp_review_full/yelp_review_full/1.0.0/e8e18e19d7be9e75642fc66b198abadb116f73599ec89a69ba5dd8d1e57ba0bf/cache-fad11cb6e689c3c5.arrow and /Users/d/.cache/huggingface/datasets/yelp_review_full/yelp_review_full/1.0.0/e8e18e19d7be9e75642fc66b198abadb116f73599ec89a69ba5dd8d1e57ba0bf/cache-8d5301c1f758ac0d.arrow\n" + ] + } + ], + "source": [ + "dataset = load_dataset(\"yelp_review_full\")\n", + "dataset_small = dataset[\"train\"].train_test_split(train_size=100, test_size=100, seed=42, stratify_by_column=\"label\")" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Step 2: Tokenize our training and test datasets" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 1000, + "referenced_widgets": [ + "603fc9e2063643b2848f538efc3edde5", + "519d214fd7c14a9d82b57bf48dfb4982", + "49f9c038e4cc40afb08d8fc27861dfd2", + "4871b34da19a4a34b33ddc85a990fb4f", + "6f937e21f25e44ff87797c2fa6eb3005", + "a8b2baeac83247dfb51795ea750cbd5d", + "8305f804414c42a19cbb7fe1699aa6f3", + "1b5357b9d8b8438992be3d048a12b7f6", + "25232ca45bdf40089e36e858ef1b78f0", + "2db78e5dcf394fc2befc3152faee0cdb", + "91a37e0a397c425a8c16456137771c57", + "317149372b974b26bad2aa1e3361e990", + "d0b0aef997ca4d79bb6088436e34fe4a", + "7ea0157acf944ed893755a45781f8106", + "fff4a1cae63947dea76e5ded7c5d8941", + "1e5e2e6f1c784953ab049228bbddff03", + "cdaaa306ec25433b92222a46f97c0a12", + "9ad729166a7f416d93f6cd6da9e26da5", + "b68d9397030f4935b993ab85931fb599", + "98db32e0dd684b298f511b5ea5fd61f2", + "5959f54cc19c418d8b710613b8ff44d6", + "8be4350405ab4ca5a9a0d61b00632df6", + "a9c48312271f4e7f8885c03afbc1c74b", + "473fb37d5e2a4a41904341bb749375e2", + "7f3650b5468f4e1f8345c5b1cad4ef0a", + "8cee559d971c44dab2c028384b48fbda", + "8e4a965c8fc04206bf43c86c68b79dd4", + "33e12d196c7a4b3292d63fdcd622d17b", + "72e27c65f029428994cd687a71874374", + "a4106a031db9488985b290bd95036774", + "4bebd86d34f94c72bff834627e71adbc", + "5018da3ba3fd4ce986a79faa58a620c9", + "f9ab23effee947d79f0c5a1051313d7c" + ] + }, + "id": "J5YkAosPTYIc", + "outputId": "ef250595-fb02-40c4-8fad-d441d25bfb26" + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Loading cached processed dataset at /Users/d/.cache/huggingface/datasets/yelp_review_full/yelp_review_full/1.0.0/e8e18e19d7be9e75642fc66b198abadb116f73599ec89a69ba5dd8d1e57ba0bf/cache-3cb31ec9adba75d6.arrow\n", + "Map: 0%| | 0/100 [00:00 512). Running this sequence through the model will result in indexing errors\n", + " \r" + ] + } + ], + "source": [ + "tokenizer = AutoTokenizer.from_pretrained(\"distilbert-base-uncased\") # uncased means lowercase\n", + "\n", + "def tokenize_function(examples):\n", + " \"\"\"this time we won't truncate the examples so that we can evaluate the distribution of the number of tokens for our reviews\n", + " \"\"\"\n", + " # prior implementation we used for training: return tokenizer(examples[\"text\"], padding=\"max_length\", truncation=True)\n", + " return tokenizer(examples[\"text\"])\n", + "\n", + "tokenized_datasets = dataset_small.map(tokenize_function, batched=True)" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Discussion Question: given the description of the tokenize_function() above, before we proceed, can you think of any potential problems with this approach? How would you go about evaluating if this approach is appropriate for our use-case? (side exercise: ex2_inspect_dataset.ipynb)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "DatasetDict({\n", + " train: Dataset({\n", + " features: ['label', 'text', 'input_ids', 'attention_mask'],\n", + " num_rows: 100\n", + " })\n", + " test: Dataset({\n", + " features: ['label', 'text', 'input_ids', 'attention_mask'],\n", + " num_rows: 100\n", + " })\n", + "})" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tokenized_datasets" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "{'label': 1,\n", + " 'text': \"On balance I liked Hula Modern Tiki and would return. That being said, my experience with Sunday brunch at Hula was rather hit or miss.\\\\n\\\\nI debated between 3 and 4 stars in large part based on the service. For a moment I considered giving them the benefit of the doubt because they were pretty busy and giving them the fourth star. But service should be just as good when the restaurant is busy right?\\\\n\\\\nThen came the goo.\\\\n\\\\nI had French Toast with thick sliced bacon and maple syrup (according to the menu). The French toast was very good. The bacon didn't arrive and I had to ask for it - when it arrived it was good, no big deal. The big deal to me was that the menu said MAPLE SYRUP. The French Toast did not come with MAPLE SYRUP. It came with some gooey, thick, out of the bottle syrup. I'm sorry but maple FLAVORED syrup is not MAPLE SYRUP or anything like it. FAIL. I would not have ordered the French Toast if I knew it was being served with corn syrup instead of MAPLE SYRUP.\\\\n\\\\nIf you are not sure of the distinction here Hula, go somewhere that has crab on the menu. Then, instead of succulent and tender crab have yourself some of that KRAB that is some unholy amalgam of whitefish, miscellaneous proteins and crab flavoring.\\\\n\\\\nMAPLE SYRUP is runny. It comes from Maple trees, not corn fields.\",\n", + " 'input_ids': [101,\n", + " 2006,\n", + " 5703,\n", + " 1045,\n", + " 4669,\n", + " 15876,\n", + " 2721,\n", + " 2715,\n", + " 14841,\n", + " 3211,\n", + " 1998,\n", + " 2052,\n", + " 2709,\n", + " 1012,\n", + " 2008,\n", + " 2108,\n", + " 2056,\n", + " 1010,\n", + " 2026,\n", + " 3325,\n", + " 2007,\n", + " 4465,\n", + " 7987,\n", + " 4609,\n", + " 2818,\n", + " 2012,\n", + " 15876,\n", + " 2721,\n", + " 2001,\n", + " 2738,\n", + " 2718,\n", + " 2030,\n", + " 3335,\n", + " 1012,\n", + " 1032,\n", + " 1050,\n", + " 1032,\n", + " 9152,\n", + " 15268,\n", + " 2090,\n", + " 1017,\n", + " 1998,\n", + " 1018,\n", + " 3340,\n", + " 1999,\n", + " 2312,\n", + " 2112,\n", + " 2241,\n", + " 2006,\n", + " 1996,\n", + " 2326,\n", + " 1012,\n", + " 2005,\n", + " 1037,\n", + " 2617,\n", + " 1045,\n", + " 2641,\n", + " 3228,\n", + " 2068,\n", + " 1996,\n", + " 5770,\n", + " 1997,\n", + " 1996,\n", + " 4797,\n", + " 2138,\n", + " 2027,\n", + " 2020,\n", + " 3492,\n", + " 5697,\n", + " 1998,\n", + " 3228,\n", + " 2068,\n", + " 1996,\n", + " 2959,\n", + " 2732,\n", + " 1012,\n", + " 2021,\n", + " 2326,\n", + " 2323,\n", + " 2022,\n", + " 2074,\n", + " 2004,\n", + " 2204,\n", + " 2043,\n", + " 1996,\n", + " 4825,\n", + " 2003,\n", + " 5697,\n", + " 2157,\n", + " 1029,\n", + " 1032,\n", + " 1050,\n", + " 1032,\n", + " 23961,\n", + " 10222,\n", + " 2234,\n", + " 1996,\n", + " 27571,\n", + " 1012,\n", + " 1032,\n", + " 1050,\n", + " 1032,\n", + " 9152,\n", + " 2018,\n", + " 2413,\n", + " 15174,\n", + " 2007,\n", + " 4317,\n", + " 15920,\n", + " 11611,\n", + " 1998,\n", + " 11035,\n", + " 23353,\n", + " 1006,\n", + " 2429,\n", + " 2000,\n", + " 1996,\n", + " 12183,\n", + " 1007,\n", + " 1012,\n", + " 1996,\n", + " 2413,\n", + " 15174,\n", + " 2001,\n", + " 2200,\n", + " 2204,\n", + " 1012,\n", + " 1996,\n", + " 11611,\n", + " 2134,\n", + " 1005,\n", + " 1056,\n", + " 7180,\n", + " 1998,\n", + " 1045,\n", + " 2018,\n", + " 2000,\n", + " 3198,\n", + " 2005,\n", + " 2009,\n", + " 1011,\n", + " 2043,\n", + " 2009,\n", + " 3369,\n", + " 2009,\n", + " 2001,\n", + " 2204,\n", + " 1010,\n", + " 2053,\n", + " 2502,\n", + " 3066,\n", + " 1012,\n", + " 1996,\n", + " 2502,\n", + " 3066,\n", + " 2000,\n", + " 2033,\n", + " 2001,\n", + " 2008,\n", + " 1996,\n", + " 12183,\n", + " 2056,\n", + " 11035,\n", + " 23353,\n", + " 1012,\n", + " 1996,\n", + " 2413,\n", + " 15174,\n", + " 2106,\n", + " 2025,\n", + " 2272,\n", + " 2007,\n", + " 11035,\n", + " 23353,\n", + " 1012,\n", + " 2009,\n", + " 2234,\n", + " 2007,\n", + " 2070,\n", + " 27571,\n", + " 3240,\n", + " 1010,\n", + " 4317,\n", + " 1010,\n", + " 2041,\n", + " 1997,\n", + " 1996,\n", + " 5835,\n", + " 23353,\n", + " 1012,\n", + " 1045,\n", + " 1005,\n", + " 1049,\n", + " 3374,\n", + " 2021,\n", + " 11035,\n", + " 14894,\n", + " 2098,\n", + " 23353,\n", + " 2003,\n", + " 2025,\n", + " 11035,\n", + " 23353,\n", + " 2030,\n", + " 2505,\n", + " 2066,\n", + " 2009,\n", + " 1012,\n", + " 8246,\n", + " 1012,\n", + " 1045,\n", + " 2052,\n", + " 2025,\n", + " 2031,\n", + " 3641,\n", + " 1996,\n", + " 2413,\n", + " 15174,\n", + " 2065,\n", + " 1045,\n", + " 2354,\n", + " 2009,\n", + " 2001,\n", + " 2108,\n", + " 2366,\n", + " 2007,\n", + " 9781,\n", + " 23353,\n", + " 2612,\n", + " 1997,\n", + " 11035,\n", + " 23353,\n", + " 1012,\n", + " 1032,\n", + " 1050,\n", + " 1032,\n", + " 9152,\n", + " 2546,\n", + " 2017,\n", + " 2024,\n", + " 2025,\n", + " 2469,\n", + " 1997,\n", + " 1996,\n", + " 7835,\n", + " 2182,\n", + " 15876,\n", + " 2721,\n", + " 1010,\n", + " 2175,\n", + " 4873,\n", + " 2008,\n", + " 2038,\n", + " 18081,\n", + " 2006,\n", + " 1996,\n", + " 12183,\n", + " 1012,\n", + " 2059,\n", + " 1010,\n", + " 2612,\n", + " 1997,\n", + " 10514,\n", + " 9468,\n", + " 27581,\n", + " 1998,\n", + " 8616,\n", + " 18081,\n", + " 2031,\n", + " 4426,\n", + " 2070,\n", + " 1997,\n", + " 2008,\n", + " 1047,\n", + " 2527,\n", + " 2497,\n", + " 2008,\n", + " 2003,\n", + " 2070,\n", + " 4895,\n", + " 14854,\n", + " 2100,\n", + " 25933,\n", + " 27887,\n", + " 2213,\n", + " 1997,\n", + " 2317,\n", + " 7529,\n", + " 1010,\n", + " 25408,\n", + " 8171,\n", + " 1998,\n", + " 18081,\n", + " 14894,\n", + " 2075,\n", + " 1012,\n", + " 1032,\n", + " 1050,\n", + " 1032,\n", + " 13221,\n", + " 9331,\n", + " 2571,\n", + " 23353,\n", + " 2003,\n", + " 2448,\n", + " 4890,\n", + " 1012,\n", + " 2009,\n", + " 3310,\n", + " 2013,\n", + " 11035,\n", + " 3628,\n", + " 1010,\n", + " 2025,\n", + " 9781,\n", + " 4249,\n", + " 1012,\n", + " 102,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0],\n", + " 'attention_mask': [1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 1,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0,\n", + " 0]}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tokenized_datasets[\"train\"][0]" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [], + "source": [ + "train_df = pd.DataFrame(tokenized_datasets[\"train\"])" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "count 100.000000\n", + "mean 165.750000\n", + "std 148.020158\n", + "min 12.000000\n", + "25% 73.250000\n", + "50% 120.500000\n", + "75% 228.750000\n", + "max 1043.000000\n", + "Name: input_ids, dtype: float64" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "train_df[\"input_ids\"].apply(len).describe()" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAfa0lEQVR4nO3dfXST9f3/8VdL05QKaaGMFqQV5jyiQ7wBC1HnFAod43gzena8YQ4ZR49bcUB3vGFOBR2junPUuRV0DvHszMrGjuBwCPYUhXEsdxUUdKt6xMERW6bYBqiE2Hx+f/glP0MLkjZ5p6nPxzk5muu6euWTN2n7PGmTpjnnnAAAAIykJ3sBAADg64X4AAAApogPAABgivgAAACmiA8AAGCK+AAAAKaIDwAAYIr4AAAApjKSvYDjhcNh7du3T3379lVaWlqylwMAAE6Bc04HDx7U4MGDlZ5+8uc2ul187Nu3T4WFhcleBgAA6IS9e/dqyJAhJz2m28VH3759JX2xeJ/P1+nzhEIhvfzyy5o4caI8Hk+8lgcx20RjvonDbBOL+SZOKsw2EAiosLAw8n38ZLpdfBz7UYvP5+tyfGRnZ8vn83Xbf6hUxWwTi/kmDrNNLOabOKk021P5lQl+4RQAAJgiPgAAgCniAwAAmCI+AACAKeIDAACYIj4AAIAp4gMAAJgiPgAAgCniAwAAmCI+AACAKeIDAACYIj4AAIAp4gMAAJgiPgAAgKmMZC/A2tC7/5nsJcTsg8rJyV4CAABxwzMfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATHUpPiorK5WWlqbZs2dHth05ckTl5eXKy8tTnz59VFZWpqampq6uEwAA9BCdjo+tW7fqySef1MiRI6O2z5kzR6tWrdLy5cu1fv167du3T1OmTOnyQgEAQM/Qqfg4dOiQpk6dqqeeekr9+vWLbG9padGSJUv0yCOPaNy4cRo1apSWLl2q1157TZs2bYrbogEAQOrK6MwHlZeXa/LkySopKdGvf/3ryPb6+nqFQiGVlJREtg0fPlxFRUWqq6vT2LFj250rGAwqGAxGrgcCAUlSKBRSKBTqzPIiH//l/x7j7eU6fc5k6cocEuFEs0V8MN/EYbaJxXwTJxVmG8vaYo6PZcuW6fXXX9fWrVvb7WtsbFRmZqZyc3Ojtufn56uxsbHD8y1cuFDz589vt/3ll19WdnZ2rMtrp6amJur6w8VdPqW51atXJ3sJHTp+togv5ps4zDaxmG/idOfZtra2nvKxMcXH3r17NWvWLNXU1CgrKyvmhXVk7ty5qqioiFwPBAIqLCzUxIkT5fP5On3eUCikmpoaTZgwQR6PJ7J9xLy1XVpvMuyaV5rsJUQ50WwRH8w3cZhtYjHfxEmF2R77ycWpiCk+6uvrtX//fl100UWRbW1tbdqwYYP+8Ic/aO3atTp69Kiam5ujnv1oampSQUFBh+f0er3yer3ttns8nrgM+PjzBNvSunxOa931gRavfyN0jPkmDrNNLOabON15trGsK6b4GD9+vHbu3Bm1bfr06Ro+fLjuuusuFRYWyuPxqLa2VmVlZZKkhoYG7dmzR36/P5abAgAAPVRM8dG3b1+NGDEiattpp52mvLy8yPYZM2aooqJC/fv3l8/n0+233y6/39/hL5sCAICvn0692uVkHn30UaWnp6usrEzBYFClpaVatGhRvG8GAACkqC7Hx6uvvhp1PSsrS1VVVaqqqurqqQEAQA/E33YBAACmiA8AAGCK+AAAAKaIDwAAYIr4AAAApogPAABgivgAAACmiA8AAGCK+AAAAKaIDwAAYIr4AAAApogPAABgivgAAACmiA8AAGCK+AAAAKaIDwAAYIr4AAAApogPAABgivgAAACmiA8AAGCK+AAAAKaIDwAAYIr4AAAApogPAABgivgAAACmiA8AAGCK+AAAAKaIDwAAYIr4AAAApogPAABgivgAAACmiA8AAGCK+AAAAKaIDwAAYIr4AAAApogPAABgivgAAACmiA8AAGCK+AAAAKaIDwAAYIr4AAAApogPAABgivgAAACmiA8AAGCK+AAAAKaIDwAAYIr4AAAApogPAABgivgAAACmiA8AAGCK+AAAAKaIDwAAYIr4AAAApogPAABgivgAAACmiA8AAGCK+AAAAKaIDwAAYIr4AAAApogPAABgivgAAACmiA8AAGCK+AAAAKZiio/Fixdr5MiR8vl88vl88vv9eumllyL7jxw5ovLycuXl5alPnz4qKytTU1NT3BcNAABSV0zxMWTIEFVWVqq+vl7btm3TuHHjdM011+itt96SJM2ZM0erVq3S8uXLtX79eu3bt09TpkxJyMIBAEBqyojl4Kuuuirq+oIFC7R48WJt2rRJQ4YM0ZIlS1RdXa1x48ZJkpYuXapzzjlHmzZt0tixY+O3agAAkLI6/TsfbW1tWrZsmQ4fPiy/36/6+nqFQiGVlJREjhk+fLiKiopUV1cXl8UCAIDUF9MzH5K0c+dO+f1+HTlyRH369NGKFSt07rnnaseOHcrMzFRubm7U8fn5+WpsbDzh+YLBoILBYOR6IBCQJIVCIYVCoViXF3HsY48/h7eX6/Q5k6Urc0iEE80W8cF8E4fZJhbzTZxUmG0sa0tzzsX03fjo0aPas2ePWlpa9Pe//11/+tOftH79eu3YsUPTp0+PCglJKi4u1pVXXqmHHnqow/PNmzdP8+fPb7e9urpa2dnZsSwNAAAkSWtrq2688Ua1tLTI5/Od9NiY4+N4JSUlOvPMM3Xddddp/Pjx+vTTT6Oe/TjjjDM0e/ZszZkzp8OP7+iZj8LCQn388cdfufiTCYVCqqmp0YQJE+TxeCLbR8xb2+lzJsuueaXJXkKUE80W8cF8E4fZJhbzTZxUmG0gENCAAQNOKT5i/rHL8cLhsILBoEaNGiWPx6Pa2lqVlZVJkhoaGrRnzx75/f4TfrzX65XX62233ePxxGXAx58n2JbW5XNa664PtHj9G6FjzDdxmG1iMd/E6c6zjWVdMcXH3LlzNWnSJBUVFengwYOqrq7Wq6++qrVr1yonJ0czZsxQRUWF+vfvL5/Pp9tvv11+v59XugAAgIiY4mP//v368Y9/rI8++kg5OTkaOXKk1q5dqwkTJkiSHn30UaWnp6usrEzBYFClpaVatGhRQhYOAABSU0zxsWTJkpPuz8rKUlVVlaqqqrq0KAAA0HPxt10AAIAp4gMAAJgiPgAAgCniAwAAmCI+AACAKeIDAACYIj4AAIAp4gMAAJgiPgAAgCniAwAAmCI+AACAKeIDAACYIj4AAIAp4gMAAJgiPgAAgCniAwAAmCI+AACAKeIDAACYIj4AAIAp4gMAAJgiPgAAgCniAwAAmCI+AACAKeIDAACYykj2AvDVht79z2QvIYq3l9PDxdKIeWsVbEvr8JgPKicbrwoAkCp45gMAAJgiPgAAgCniAwAAmCI+AACAKeIDAACYIj4AAIAp4gMAAJgiPgAAgCniAwAAmCI+AACAKeIDAACYIj4AAIAp4gMAAJgiPgAAgCniAwAAmCI+AACAKeIDAACYIj4AAIAp4gMAAJgiPgAAgCniAwAAmCI+AACAKeIDAACYIj4AAIAp4gMAAJgiPgAAgCniAwAAmCI+AACAKeIDAACYIj4AAIAp4gMAAJgiPgAAgCniAwAAmCI+AACAKeIDAACYIj4AAIAp4gMAAJgiPgAAgCniAwAAmIopPhYuXKiLL75Yffv21cCBA3XttdeqoaEh6pgjR46ovLxceXl56tOnj8rKytTU1BTXRQMAgNQVU3ysX79e5eXl2rRpk2pqahQKhTRx4kQdPnw4csycOXO0atUqLV++XOvXr9e+ffs0ZcqUuC8cAACkpoxYDl6zZk3U9WeeeUYDBw5UfX29Lr/8crW0tGjJkiWqrq7WuHHjJElLly7VOeeco02bNmns2LHxWzkAAEhJMcXH8VpaWiRJ/fv3lyTV19crFAqppKQkcszw4cNVVFSkurq6DuMjGAwqGAxGrgcCAUlSKBRSKBTq9NqOfezx5/D2cp0+J77gTXdR/+1IV/7tvu5O9NhF1zHbxGK+iZMKs41lbWnOuU59Nw6Hw7r66qvV3NysjRs3SpKqq6s1ffr0qJiQpOLiYl155ZV66KGH2p1n3rx5mj9/frvt1dXVys7O7szSAACAsdbWVt14441qaWmRz+c76bGdfuajvLxcu3btioRHZ82dO1cVFRWR64FAQIWFhZo4ceJXLv5kQqGQampqNGHCBHk8nsj2EfPWdmm9+OIZjwdHh3XvtnQFw2kdHrNrXqnxqnqOEz120XXMNrGYb+KkwmyP/eTiVHQqPmbOnKkXX3xRGzZs0JAhQyLbCwoKdPToUTU3Nys3NzeyvampSQUFBR2ey+v1yuv1ttvu8XjiMuDjzxNs6/ibJWIXDKedcJ7d9ZMjlcTrcwDtMdvEYr6J051nG8u6Ynq1i3NOM2fO1IoVK7Ru3ToNGzYsav+oUaPk8XhUW1sb2dbQ0KA9e/bI7/fHclMAAKCHiumZj/LyclVXV+uFF15Q37591djYKEnKyclR7969lZOToxkzZqiiokL9+/eXz+fT7bffLr/fzytdAACApBjjY/HixZKkK664Imr70qVLdfPNN0uSHn30UaWnp6usrEzBYFClpaVatGhRXBYLAABSX0zxcSovjMnKylJVVZWqqqo6vSgAANBz8bddAACAqS69yRhwIkPv/meylxCzDyonJ3sJAPC1wDMfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMxRwfGzZs0FVXXaXBgwcrLS1NK1eujNrvnNN9992nQYMGqXfv3iopKdG7774br/UCAIAUF3N8HD58WOeff76qqqo63P/www/r8ccf1xNPPKHNmzfrtNNOU2lpqY4cOdLlxQIAgNSXEesHTJo0SZMmTepwn3NOjz32mH71q1/pmmuukST9+c9/Vn5+vlauXKnrr7++a6sFAAApL+b4OJndu3ersbFRJSUlkW05OTkaM2aM6urqOoyPYDCoYDAYuR4IBCRJoVBIoVCo02s59rHHn8Pby3X6nPiCN91F/ben6MrjLZ5O9NhF1zHbxGK+iZMKs41lbXGNj8bGRklSfn5+1Pb8/PzIvuMtXLhQ8+fPb7f95ZdfVnZ2dpfXVFNTE3X94eIunxL/58HR4WQvIa5Wr16d7CVEOf6xi/hhtonFfBOnO8+2tbX1lI+Na3x0xty5c1VRURG5HggEVFhYqIkTJ8rn83X6vKFQSDU1NZowYYI8Hk9k+4h5a7u0XnzxjMeDo8O6d1u6guG0ZC8nbnbNK032EiSd+LGLrmO2icV8EycVZnvsJxenIq7xUVBQIElqamrSoEGDItubmpp0wQUXdPgxXq9XXq+33XaPxxOXAR9/nmBbz/lmmWzBcFqPmmd3+4SO1+cA2mO2icV8E6c7zzaWdcX1fT6GDRumgoIC1dbWRrYFAgFt3rxZfr8/njcFAABSVMzPfBw6dEjvvfde5Pru3bu1Y8cO9e/fX0VFRZo9e7Z+/etf66yzztKwYcN07733avDgwbr22mvjuW4AAJCiYo6Pbdu26corr4xcP/b7GtOmTdMzzzyjO++8U4cPH9att96q5uZmXXbZZVqzZo2ysrLit2oAAJCyYo6PK664Qs6d+CWWaWlpeuCBB/TAAw90aWEAAKBn4m+7AAAAU8QHAAAwRXwAAABTxAcAADBFfAAAAFPEBwAAMEV8AAAAU8QHAAAwRXwAAABTxAcAADBFfAAAAFPEBwAAMEV8AAAAU8QHAAAwRXwAAABTxAcAADBFfAAAAFPEBwAAMEV8AAAAU8QHAAAwRXwAAABTxAcAADBFfAAAAFPEBwAAMEV8AAAAU8QHAAAwRXwAAABTxAcAADBFfAAAAFPEBwAAMEV8AAAAU8QHAAAwRXwAAABTxAcAADBFfAAAAFPEBwAAMEV8AAAAU8QHAAAwlZHsBQDdxdC7/5nsJUiSvL2cHi6WRsxbq2Bb2kmP/aBystGqACB+eOYDAACYIj4AAIAp4gMAAJgiPgAAgCniAwAAmCI+AACAKeIDAACY4n0+AJhK5vupxPIeKl/G+6kA8cUzHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATBEfAADAFPEBAABMER8AAMAU8QEAAEwRHwAAwBTxAQAATGUkewEAAKSyoXf/M+G34e3l9HCxNGLeWgXb0rp8vg8qJ8dhVZ3HMx8AAMAU8QEAAEwlLD6qqqo0dOhQZWVlacyYMdqyZUuibgoAAKSQhMTHX//6V1VUVOj+++/X66+/rvPPP1+lpaXav39/Im4OAACkkITExyOPPKJbbrlF06dP17nnnqsnnnhC2dnZevrppxNxcwAAIIXE/dUuR48eVX19vebOnRvZlp6erpKSEtXV1bU7PhgMKhgMRq63tLRIkg4cOKBQKNTpdYRCIbW2tuqTTz6Rx+OJbM/4/HCnz4kvZISdWlvDygilqy3c9d+6RrRY5vvJJ58YrSp+kvk52NnHbirOORlO9HW3p7N4TMf7624iHtMHDx6UJDnnvvpgF2cffvihk+Ree+21qO133HGHKy4ubnf8/fff7yRx4cKFCxcuXHrAZe/evV/ZCkl/n4+5c+eqoqIicj0cDuvAgQPKy8tTWlrn6y4QCKiwsFB79+6Vz+eLx1Lxf5htYjHfxGG2icV8EycVZuuc08GDBzV48OCvPDbu8TFgwAD16tVLTU1NUdubmppUUFDQ7niv1yuv1xu1LTc3N27r8fl83fYfKtUx28RivonDbBOL+SZOd59tTk7OKR0X9184zczM1KhRo1RbWxvZFg6HVVtbK7/fH++bAwAAKSYhP3apqKjQtGnTNHr0aBUXF+uxxx7T4cOHNX369ETcHAAASCEJiY/rrrtO//vf/3TfffepsbFRF1xwgdasWaP8/PxE3FyHvF6v7r///nY/0kHXMdvEYr6Jw2wTi/kmTk+bbZpzp/KaGAAAgPjgb7sAAABTxAcAADBFfAAAAFPEBwAAMNUj46OqqkpDhw5VVlaWxowZoy1btiR7Sd3ewoULdfHFF6tv374aOHCgrr32WjU0NEQdc+TIEZWXlysvL099+vRRWVlZuzeT27NnjyZPnqzs7GwNHDhQd9xxhz7//HPLu9LtVVZWKi0tTbNnz45sY7Zd8+GHH+pHP/qR8vLy1Lt3b5133nnatm1bZL9zTvfdd58GDRqk3r17q6SkRO+++27UOQ4cOKCpU6fK5/MpNzdXM2bM0KFDh6zvSrfS1tame++9V8OGDVPv3r115pln6sEHH4z62x3M9tRt2LBBV111lQYPHqy0tDStXLkyan+8Zvnmm2/qO9/5jrKyslRYWKiHH3440Xctdl3/ay7dy7Jly1xmZqZ7+umn3VtvveVuueUWl5ub65qampK9tG6ttLTULV261O3atcvt2LHDff/733dFRUXu0KFDkWNuu+02V1hY6Gpra922bdvc2LFj3SWXXBLZ//nnn7sRI0a4kpISt337drd69Wo3YMAAN3fu3GTcpW5py5YtbujQoW7kyJFu1qxZke3MtvMOHDjgzjjjDHfzzTe7zZs3u/fff9+tXbvWvffee5FjKisrXU5Ojlu5cqV744033NVXX+2GDRvmPvvss8gx3/ve99z555/vNm3a5P71r3+5b33rW+6GG25Ixl3qNhYsWODy8vLciy++6Hbv3u2WL1/u+vTp4373u99FjmG2p2716tXunnvucc8//7yT5FasWBG1Px6zbGlpcfn5+W7q1Klu165d7rnnnnO9e/d2Tz75pNXdPCU9Lj6Ki4tdeXl55HpbW5sbPHiwW7hwYRJXlXr279/vJLn169c755xrbm52Ho/HLV++PHLMv//9byfJ1dXVOee++MRKT093jY2NkWMWL17sfD6fCwaDtnegGzp48KA766yzXE1Njfvud78biQ9m2zV33XWXu+yyy064PxwOu4KCAvfb3/42sq25udl5vV733HPPOeece/vtt50kt3Xr1sgxL730kktLS3Mffvhh4hbfzU2ePNn95Cc/ido2ZcoUN3XqVOccs+2K4+MjXrNctGiR69evX9TXhbvuusudffbZCb5HselRP3Y5evSo6uvrVVJSEtmWnp6ukpIS1dXVJXFlqaelpUWS1L9/f0lSfX29QqFQ1GyHDx+uoqKiyGzr6up03nnnRb2ZXGlpqQKBgN566y3D1XdP5eXlmjx5ctQMJWbbVf/4xz80evRo/fCHP9TAgQN14YUX6qmnnors3717txobG6Pmm5OTozFjxkTNNzc3V6NHj44cU1JSovT0dG3evNnuznQzl1xyiWpra/XOO+9Ikt544w1t3LhRkyZNksRs4yles6yrq9Pll1+uzMzMyDGlpaVqaGjQp59+anRvvlrS/6ptPH388cdqa2tr906q+fn5+s9//pOkVaWecDis2bNn69JLL9WIESMkSY2NjcrMzGz3R//y8/PV2NgYOaaj2R/b93W2bNkyvf7669q6dWu7fcy2a95//30tXrxYFRUV+uUvf6mtW7fq5z//uTIzMzVt2rTIfDqa35fnO3DgwKj9GRkZ6t+//9d6vnfffbcCgYCGDx+uXr16qa2tTQsWLNDUqVMlidnGUbxm2djYqGHDhrU7x7F9/fr1S8j6Y9Wj4gPxUV5erl27dmnjxo3JXkqPsHfvXs2aNUs1NTXKyspK9nJ6nHA4rNGjR+s3v/mNJOnCCy/Url279MQTT2jatGlJXl1q+9vf/qZnn31W1dXV+va3v60dO3Zo9uzZGjx4MLNFl/SoH7sMGDBAvXr1avcqgaamJhUUFCRpVall5syZevHFF/XKK69oyJAhke0FBQU6evSompubo47/8mwLCgo6nP2xfV9X9fX12r9/vy666CJlZGQoIyND69ev1+OPP66MjAzl5+cz2y4YNGiQzj333Kht55xzjvbs2SPp/8/nZF8XCgoKtH///qj9n3/+uQ4cOPC1nu8dd9yhu+++W9dff73OO+883XTTTZozZ44WLlwoidnGU7xmmSpfK3pUfGRmZmrUqFGqra2NbAuHw6qtrZXf70/iyro/55xmzpypFStWaN26de2eths1apQ8Hk/UbBsaGrRnz57IbP1+v3bu3Bn1yVFTUyOfz9fum8PXyfjx47Vz507t2LEjchk9erSmTp0a+X9m23mXXnppu5eFv/POOzrjjDMkScOGDVNBQUHUfAOBgDZv3hw13+bmZtXX10eOWbduncLhsMaMGWNwL7qn1tZWpadHf5vo1auXwuGwJGYbT/Gapd/v14YNGxQKhSLH1NTU6Oyzz+42P3KR1DNfauv1et0zzzzj3n77bXfrrbe63NzcqFcJoL2f/vSnLicnx7366qvuo48+ilxaW1sjx9x2222uqKjIrVu3zm3bts35/X7n9/sj+4+9HHTixIlux44dbs2aNe4b3/gGLwftwJdf7eIcs+2KLVu2uIyMDLdgwQL37rvvumeffdZlZ2e7v/zlL5FjKisrXW5urnvhhRfcm2++6a655poOX8J44YUXus2bN7uNGze6s84662v5ctAvmzZtmjv99NMjL7V9/vnn3YABA9ydd94ZOYbZnrqDBw+67du3u+3btztJ7pFHHnHbt293//3vf51z8Zllc3Ozy8/PdzfddJPbtWuXW7ZsmcvOzualthZ+//vfu6KiIpeZmemKi4vdpk2bkr2kbk9Sh5elS5dGjvnss8/cz372M9evXz+XnZ3tfvCDH7iPPvoo6jwffPCBmzRpkuvdu7cbMGCA+8UvfuFCoZDxven+jo8PZts1q1atciNGjHBer9cNHz7c/fGPf4zaHw6H3b333uvy8/Od1+t148ePdw0NDVHHfPLJJ+6GG25wffr0cT6fz02fPt0dPHjQ8m50O4FAwM2aNcsVFRW5rKws981vftPdc889US/jZLan7pVXXunw6+y0adOcc/Gb5RtvvOEuu+wy5/V63emnn+4qKyut7uIpS3PuS29VBwAAkGA96nc+AABA90d8AAAAU8QHAAAwRXwAAABTxAcAADBFfAAAAFPEBwAAMEV8AAAAU8QHAAAwRXwAAABTxAcAADBFfAAAAFP/D9v5PrryTqFPAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "train_df[\"input_ids\"].apply(len).hist()" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": {}, + "outputs": [], + "source": [ + "train_df[\"input_ids_token_length\"] = train_df[\"input_ids\"].apply(len)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
labeltextinput_idsattention_maskinput_ids_token_length
321We came on a Friday night for dinner, and book...[101, 2057, 2234, 2006, 1037, 5958, 2305, 2005...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...1043
760This was the first time my husband and I have ...[101, 2023, 2001, 1996, 2034, 2051, 2026, 3129...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...611
774Its a typical Saturday morning. \\nYou're hungo...[101, 2049, 1037, 5171, 5095, 2851, 1012, 1032...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...611
923Icehouse is a controversially good place. I've...[101, 3256, 4580, 2003, 1037, 6801, 2135, 2204...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...544
\n", + "
" + ], + "text/plain": [ + " label text \n", + "32 1 We came on a Friday night for dinner, and book... \\\n", + "76 0 This was the first time my husband and I have ... \n", + "77 4 Its a typical Saturday morning. \\nYou're hungo... \n", + "92 3 Icehouse is a controversially good place. I've... \n", + "\n", + " input_ids \n", + "32 [101, 2057, 2234, 2006, 1037, 5958, 2305, 2005... \\\n", + "76 [101, 2023, 2001, 1996, 2034, 2051, 2026, 3129... \n", + "77 [101, 2049, 1037, 5171, 5095, 2851, 1012, 1032... \n", + "92 [101, 3256, 4580, 2003, 1037, 6801, 2135, 2204... \n", + "\n", + " attention_mask input_ids_token_length \n", + "32 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... 1043 \n", + "76 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... 611 \n", + "77 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... 611 \n", + "92 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... 544 " + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "train_df[train_df[\"input_ids_token_length\"] > 512]" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": {}, + "outputs": [], + "source": [ + "longest_review = train_df[train_df[\"input_ids_token_length\"] > 512][\"text\"].values[0]" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['We came on a Friday night for dinner, and booked a table through Open Table',\n", + " 'And we are also Landry Club member',\n", + " 'So we would think that it would be a nice dinner',\n", + " 'It turned out that the dinner was an upsetting experience because of a very bad server.',\n", + " 'nWe ordered a Capricciosa Pizza and ONE caesar salad',\n", + " 'When the salad came, we were given 2 half salads',\n", + " 'I was already feeling weird about it',\n", + " 'When we ordered, we did say A caesar salad',\n", + " 'We didn\\'t say \\\\\"two half caesar salads\\\\\"',\n", + " \"And we weren't asked if we would like to split it too\",\n", + " 'But it came out split',\n", + " \"So I wasn't sure what was going on\",\n", + " \"I didn't even know that we would be charged more for comsuming the 2 half salads\",\n", + " 'We just thought that maybe they were nice enough to split it into 2 portions',\n", + " \"We weren't asked at all so and the menu didn't even state any option of half salad of splitting salad\",\n", + " \"So we didn't have any clue that we should have asked about it before putting the salad into our mouths\",\n", + " 'Besides, the server was very not attentive to us while she was attentive to every other tables she served',\n", + " 'So we were like \\\\\"what\\'s wrong with her?\\\\\"',\n", + " 'nAt the end the check came, we were charged for 2 half salads instead of 1 full portion of caesar salad',\n", + " '1 full portion of caesar salad costs $9.99 but 2 half salads cost $11.98',\n", + " 'So of course we were upset about it! The server was being unfair to us to make the call of splitting the salad without even asking us first',\n", + " 'I feel that she should be the one paying our meal because of how she treated us (see below).',\n", + " 'nWe are really upset with the server',\n", + " 'The server always attempted to take away our food while we were still working on it',\n", + " 'When we were working on the pizza, when there was still half a pizza left on the pizza plate, the server attempted to take away our pizza and said \\\\\"you are all done?\\\\\" We said \\\\\"no, we are still working on it',\n", + " 'What kind of manner was that?? She could have asked if we are still working on the pizza without any action done first',\n", + " 'She did the same when we were working on our dessert',\n", + " 'She was so off, way too off',\n", + " 'I was nice enough to write $2 as tips for her at the end',\n", + " \"She really didn't deserve any tip as that's not called service\",\n", + " \"She doesn't deserve any gratuity when she was trying to rush people out of the restaurant\",\n", + " 'But I did observe and she was nice to all other people',\n", + " 'And I really had to come down to a conclusion that she might be a racist!\\\\nWe got so upset',\n", + " \"And I was very hungry and tired that night so I didn't have the energy to think what I could have done while we were still at the restaurant\",\n", + " 'So after taking a shower that night, my mind got clear and I was able to think',\n", + " \"Then I realized how stupid I was as I should have brought it up to the manager, and I shouldn't have given her any tip (It's not about money, it's about teaching her a lesson, if she is going to have tip with such a horrible manner, she is going to continue doing that as she thinks she's entitled)\",\n", + " \"So the other morning, we went to the restaurant and spoke with the manager about how unfair it was that we were charged for 2 half salads while we didn't ask for it\",\n", + " 'And we were upset at the server rushing us',\n", + " 'The manager felt very sorry about it so she treated us lunch afterwards',\n", + " 'We got a lasagna and a caesar salad again for lunch.',\n", + " 'nThe food is worth 4 starts honestly',\n", + " 'The pizza, the lasagna and the salad were worth 4 stars',\n", + " 'The dessert Strawberry Cassatta was worth 3 stars, it was not bad tasting, just not surprising either',\n", + " 'We were just so upset with the server so she made the whole experience 2 stars',\n", + " 'The manager was worth 4 stars as she was nice, sincere, was very reasonable and understanding us',\n", + " 'I just feel bad for them to have a bad apple being a server, who has the potential to ruin the reputation of the restaurant so badly.',\n", + " 'nWe are giving 2 stars for this review because of the bad server we had for the dinner',\n", + " '',\n", + " 'nThe lunch that the manager treated us actually deserved 4 stars, we also had a different server for that lunch too',\n", + " 'We accepted the apologies and will still try Grotto again in the future when we visit vegas, especially because we really like their pizza and lasagna',\n", + " \"We will just have to make sure that we won't have that bad server ever again in the future when visiting Grotto, and probably ask for another server if we unluckily get another bad one in the future.\"]" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "longest_review.replace(\". \", \"\\\\n\\\\\").split(\"\\\\n\\\\\")" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Run analysis on the entire dataset\n", + "You might not want to execute these cells as it may take awhile" + ] + }, + { + "cell_type": "code", + "execution_count": 67, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "data": { + "text/plain": [ + "count 650000.000000\n", + "mean 180.696091\n", + "std 163.847267\n", + "min 3.000000\n", + "25% 71.000000\n", + "50% 132.000000\n", + "75% 236.000000\n", + "max 2344.000000\n", + "Name: input_ids, dtype: float64" + ] + }, + "execution_count": 67, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tokenized_datasets = dataset.map(tokenize_function, batched=True)\n", + "train_df = pd.DataFrame(tokenized_datasets[\"train\"])\n", + "train_df[\"input_ids\"].apply(len).describe()" + ] + }, + { + "cell_type": "code", + "execution_count": 68, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 68, + "metadata": {}, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkIAAAGdCAYAAAD+JxxnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAtZUlEQVR4nO3df3BU9b3/8VcSkw0BNuGHSUj5FS8KRn5JKHFv1RENWTDjaKUdtIxNEXHkJk5DWqi5Q8MPeycUr/ywRGmvP2KntQJ3RnslNLANBq5lAQ3kCiiM9uKNvbCJBZKFAJslOd8/7uR8WYKQYMIBPs/HDDPuOe/97HvfH0he7u5JoizLsgQAAGCgaKcbAAAAcApBCAAAGIsgBAAAjEUQAgAAxiIIAQAAYxGEAACAsQhCAADAWAQhAABgrJucbuBa1tbWpiNHjqhv376Kiopyuh0AANAJlmXp5MmTSktLU3T0pV/zIQhdwpEjRzRkyBCn2wAAAFfgyy+/1ODBgy9ZQxC6hL59+0r6v0G63e5uWzccDmvLli3KyclRbGxst62LzmMPnMX8ncX8ncce9KxgMKghQ4bY38cvhSB0Ce1vh7nd7m4PQgkJCXK73fwDcAh74Czm7yzm7zz24OrozMda+LA0AAAwFkEIAAAYq0tBaPHixYqKior4M2rUKPv82bNnlZ+frwEDBqhPnz6aPn266uvrI9aoq6tTbm6uEhISlJycrPnz5+vcuXMRNdXV1ZowYYJcLpdGjBih8vLyDr2UlZVp+PDhio+PV1ZWlnbv3h1xvjO9AAAAs3X5FaE77rhDR48etf988MEH9rl58+bpvffe04YNG7Rt2zYdOXJEjz76qH2+tbVVubm5amlp0Y4dO/Tmm2+qvLxcJSUlds3hw4eVm5uryZMnq7a2VoWFhXrqqae0efNmu2bdunUqKirSokWLtGfPHo0bN05er1cNDQ2d7gUAAEBWFyxatMgaN27cRc81NjZasbGx1oYNG+xjn376qSXJ8vv9lmVZ1qZNm6zo6GgrEAjYNa+88orldrutUChkWZZlLViwwLrjjjsi1p4xY4bl9Xrt25MmTbLy8/Pt262trVZaWppVWlra6V46o6mpyZJkNTU1dfo+ndHS0mK9++67VktLS7eui85jD5zF/J3F/J3HHvSsrnz/7vJVY5999pnS0tIUHx8vj8ej0tJSDR06VDU1NQqHw8rOzrZrR40apaFDh8rv9+uuu+6S3+/XmDFjlJKSYtd4vV7NnTtXBw4c0J133im/3x+xRntNYWGhJKmlpUU1NTUqLi62z0dHRys7O1t+v1+SOtXLxYRCIYVCIft2MBiU9H+f7g+Hw10d1ddqX6s710TXsAfOYv7OYv7OYw96Vlfm2qUglJWVpfLyco0cOVJHjx7VkiVLdM8992j//v0KBAKKi4tTUlJSxH1SUlIUCAQkSYFAICIEtZ9vP3epmmAwqDNnzujEiRNqbW29aM3BgwftNS7Xy8WUlpZqyZIlHY5v2bJFCQkJX3u/K+Xz+bp9TXQNe+As5u8s5u889qBnnD59utO1XQpC06ZNs/977NixysrK0rBhw7R+/Xr16tWrK0tdk4qLi1VUVGTfbv+BTDk5Od3+c4R8Pp+mTJnCz49wCHvgLObvLObvPPagZ7W/o9MZ3+gHKiYlJem2227T559/rilTpqilpUWNjY0Rr8TU19crNTVVkpSamtrh6q72K7nOr7nw6q76+nq53W716tVLMTExiomJuWjN+WtcrpeLcblccrlcHY7Hxsb2yF/UnloXncceOIv5O4v5O4896Bldmek3+jlCp06d0l//+lcNGjRImZmZio2NVVVVlX3+0KFDqqurk8fjkSR5PB7t27cv4uoun88nt9utjIwMu+b8Ndpr2teIi4tTZmZmRE1bW5uqqqrsms70AgAA0KVXhH7605/qoYce0rBhw3TkyBEtWrRIMTExevzxx5WYmKjZs2erqKhI/fv3l9vt1rPPPiuPx2N/ODknJ0cZGRl64okntHz5cgUCAS1cuFD5+fn2KzHPPPOM1qxZowULFujJJ5/U1q1btX79elVUVNh9FBUVKS8vTxMnTtSkSZO0atUqNTc3a9asWZLUqV4AAAC6FIT+9re/6fHHH9exY8d088036+6779bOnTt18803S5JWrlyp6OhoTZ8+XaFQSF6vVy+//LJ9/5iYGG3cuFFz586Vx+NR7969lZeXp6VLl9o16enpqqio0Lx587R69WoNHjxYr776qrxer10zY8YMffXVVyopKVEgEND48eNVWVkZ8QHqy/UCAAAQZVmW5XQT16pgMKjExEQ1NTV1+4elN23apAcffJD3hh3CHjiL+TuL+TuPPehZXfn+ze8aAwAAxvpGV43hmxm9eLNCrVFOt9ElXyzLdboFAAC6Da8IAQAAYxGEAACAsQhCAADAWAQhAABgLIIQAAAwFkEIAAAYiyAEAACMRRACAADGIggBAABjEYQAAICxCEIAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMBZBCAAAGIsgBAAAjEUQAgAAxiIIAQAAYxGEAACAsQhCAADAWAQhAABgLIIQAAAwFkEIAAAYiyAEAACMRRACAADGIggBAABjEYQAAICxCEIAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMBZBCAAAGIsgBAAAjEUQAgAAxiIIAQAAYxGEAACAsQhCAADAWAQhAABgLIIQAAAwFkEIAAAYiyAEAACMRRACAADGIggBAABjEYQAAICxCEIAAMBY3ygILVu2TFFRUSosLLSPnT17Vvn5+RowYID69Omj6dOnq76+PuJ+dXV1ys3NVUJCgpKTkzV//nydO3cuoqa6uloTJkyQy+XSiBEjVF5e3uHxy8rKNHz4cMXHxysrK0u7d++OON+ZXgAAgLmuOAh9+OGH+vWvf62xY8dGHJ83b57ee+89bdiwQdu2bdORI0f06KOP2udbW1uVm5urlpYW7dixQ2+++abKy8tVUlJi1xw+fFi5ubmaPHmyamtrVVhYqKeeekqbN2+2a9atW6eioiItWrRIe/bs0bhx4+T1etXQ0NDpXgAAgNmuKAidOnVKM2fO1L/927+pX79+9vGmpia99tprWrFihe6//35lZmbqjTfe0I4dO7Rz505J0pYtW/TJJ5/od7/7ncaPH69p06bp+eefV1lZmVpaWiRJa9euVXp6ul588UXdfvvtKigo0Pe+9z2tXLnSfqwVK1Zozpw5mjVrljIyMrR27VolJCTo9ddf73QvAADAbDddyZ3y8/OVm5ur7Oxs/eIXv7CP19TUKBwOKzs72z42atQoDR06VH6/X3fddZf8fr/GjBmjlJQUu8br9Wru3Lk6cOCA7rzzTvn9/og12mva34JraWlRTU2NiouL7fPR0dHKzs6W3+/vdC8XCoVCCoVC9u1gMChJCofDCofDVzKqi2pfyxVtdduaV0t3zsFJ7c/jRnk+1xvm7yzm7zz2oGd1Za5dDkJvv/229uzZow8//LDDuUAgoLi4OCUlJUUcT0lJUSAQsGvOD0Ht59vPXaomGAzqzJkzOnHihFpbWy9ac/DgwU73cqHS0lItWbKkw/EtW7YoISHhovf5Jp6f2Nbta/a0TZs2Od1Ct/L5fE63YDTm7yzm7zz2oGecPn2607VdCkJffvmlfvzjH8vn8yk+Pr7LjV3riouLVVRUZN8OBoMaMmSIcnJy5Ha7u+1xwuGwfD6ffv5RtEJtUd227tWwf7HX6Ra6RfseTJkyRbGxsU63Yxzm7yzm7zz2oGe1v6PTGV0KQjU1NWpoaNCECRPsY62trdq+fbvWrFmjzZs3q6WlRY2NjRGvxNTX1ys1NVWSlJqa2uHqrvYruc6vufDqrvr6erndbvXq1UsxMTGKiYm5aM35a1yulwu5XC65XK4Ox2NjY3vkL2qoLUqh1usrCN1o/2B7am/ROczfWczfeexBz+jKTLv0YekHHnhA+/btU21trf1n4sSJmjlzpv3fsbGxqqqqsu9z6NAh1dXVyePxSJI8Ho/27dsXcXWXz+eT2+1WRkaGXXP+Gu017WvExcUpMzMzoqatrU1VVVV2TWZm5mV7AQAAZuvSK0J9+/bV6NGjI4717t1bAwYMsI/Pnj1bRUVF6t+/v9xut5599ll5PB77w8k5OTnKyMjQE088oeXLlysQCGjhwoXKz8+3X4155plntGbNGi1YsEBPPvmktm7dqvXr16uiosJ+3KKiIuXl5WnixImaNGmSVq1apebmZs2aNUuSlJiYeNleAACA2a7oqrFLWblypaKjozV9+nSFQiF5vV69/PLL9vmYmBht3LhRc+fOlcfjUe/evZWXl6elS5faNenp6aqoqNC8efO0evVqDR48WK+++qq83v//+ZQZM2boq6++UklJiQKBgMaPH6/KysqID1BfrhcAAGC2bxyEqqurI27Hx8errKxMZWVlX3ufYcOGXfbqo/vuu0979+69ZE1BQYEKCgq+9nxnegEAAObid40BAABjEYQAAICxCEIAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMBZBCAAAGIsgBAAAjEUQAgAAxiIIAQAAYxGEAACAsQhCAADAWAQhAABgLIIQAAAwFkEIAAAYiyAEAACMRRACAADGIggBAABjEYQAAICxCEIAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMBZBCAAAGIsgBAAAjEUQAgAAxiIIAQAAYxGEAACAsQhCAADAWAQhAABgLIIQAAAwFkEIAAAYiyAEAACMRRACAADGIggBAABjEYQAAICxCEIAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMFaXgtArr7yisWPHyu12y+12y+Px6E9/+pN9/uzZs8rPz9eAAQPUp08fTZ8+XfX19RFr1NXVKTc3VwkJCUpOTtb8+fN17ty5iJrq6mpNmDBBLpdLI0aMUHl5eYdeysrKNHz4cMXHxysrK0u7d++OON+ZXgAAgNm6FIQGDx6sZcuWqaamRh999JHuv/9+Pfzwwzpw4IAkad68eXrvvfe0YcMGbdu2TUeOHNGjjz5q37+1tVW5ublqaWnRjh079Oabb6q8vFwlJSV2zeHDh5Wbm6vJkyertrZWhYWFeuqpp7R582a7Zt26dSoqKtKiRYu0Z88ejRs3Tl6vVw0NDXbN5XoBAADoUhB66KGH9OCDD+rWW2/Vbbfdpn/5l39Rnz59tHPnTjU1Nem1117TihUrdP/99yszM1NvvPGGduzYoZ07d0qStmzZok8++US/+93vNH78eE2bNk3PP/+8ysrK1NLSIklau3at0tPT9eKLL+r2229XQUGBvve972nlypV2HytWrNCcOXM0a9YsZWRkaO3atUpISNDrr78uSZ3qBQAA4KYrvWNra6s2bNig5uZmeTwe1dTUKBwOKzs7264ZNWqUhg4dKr/fr7vuukt+v19jxoxRSkqKXeP1ejV37lwdOHBAd955p/x+f8Qa7TWFhYWSpJaWFtXU1Ki4uNg+Hx0drezsbPn9fknqVC8XEwqFFAqF7NvBYFCSFA6HFQ6Hr3BSHbWv5Yq2um3Nq6U75+Ck9udxozyf6w3zdxbzdx570LO6MtcuB6F9+/bJ4/Ho7Nmz6tOnj9555x1lZGSotrZWcXFxSkpKiqhPSUlRIBCQJAUCgYgQ1H6+/dylaoLBoM6cOaMTJ06otbX1ojUHDx6017hcLxdTWlqqJUuWdDi+ZcsWJSQkfO39rtTzE9u6fc2etmnTJqdb6FY+n8/pFozG/J3F/J3HHvSM06dPd7q2y0Fo5MiRqq2tVVNTk/793/9deXl52rZtW1eXuSYVFxerqKjIvh0MBjVkyBDl5OTI7XZ32+OEw2H5fD79/KNohdqium3dq2H/Yq/TLXSL9j2YMmWKYmNjnW7HOMzfWczfeexBz2p/R6czuhyE4uLiNGLECElSZmamPvzwQ61evVozZsxQS0uLGhsbI16Jqa+vV2pqqiQpNTW1w9Vd7VdynV9z4dVd9fX1crvd6tWrl2JiYhQTE3PRmvPXuFwvF+NyueRyuTocj42N7ZG/qKG2KIVar68gdKP9g+2pvUXnMH9nMX/nsQc9oysz/cY/R6itrU2hUEiZmZmKjY1VVVWVfe7QoUOqq6uTx+ORJHk8Hu3bty/i6i6fzye3262MjAy75vw12mva14iLi1NmZmZETVtbm6qqquyazvQCAADQpVeEiouLNW3aNA0dOlQnT57UW2+9perqam3evFmJiYmaPXu2ioqK1L9/f7ndbj377LPyeDz2h5NzcnKUkZGhJ554QsuXL1cgENDChQuVn59vvxLzzDPPaM2aNVqwYIGefPJJbd26VevXr1dFRYXdR1FRkfLy8jRx4kRNmjRJq1atUnNzs2bNmiVJneoFAACgS0GooaFBP/zhD3X06FElJiZq7Nix2rx5s6ZMmSJJWrlypaKjozV9+nSFQiF5vV69/PLL9v1jYmK0ceNGzZ07Vx6PR71791ZeXp6WLl1q16Snp6uiokLz5s3T6tWrNXjwYL366qvyev//Z1NmzJihr776SiUlJQoEAho/frwqKysjPkB9uV4AAACiLMu6/q7hvkqCwaASExPV1NTU7R+W3rRpkxbsjrnuPiP0xbJcp1voFu178OCDD/L+vAOYv7OYv/PYg57Vle/f/K4xAABgLIIQAAAwFkEIAAAYiyAEAACMRRACAADGIggBAABjEYQAAICxCEIAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMBZBCAAAGIsgBAAAjEUQAgAAxiIIAQAAYxGEAACAsQhCAADAWAQhAABgLIIQAAAwFkEIAAAYiyAEAACMRRACAADGIggBAABjEYQAAICxCEIAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMBZBCAAAGIsgBAAAjEUQAgAAxiIIAQAAYxGEAACAsQhCAADAWAQhAABgLIIQAAAwFkEIAAAYiyAEAACMRRACAADGIggBAABjEYQAAICxCEIAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMbqUhAqLS3Vt7/9bfXt21fJycl65JFHdOjQoYias2fPKj8/XwMGDFCfPn00ffp01dfXR9TU1dUpNzdXCQkJSk5O1vz583Xu3LmImurqak2YMEEul0sjRoxQeXl5h37Kyso0fPhwxcfHKysrS7t37+5yLwAAwFxdCkLbtm1Tfn6+du7cKZ/Pp3A4rJycHDU3N9s18+bN03vvvacNGzZo27ZtOnLkiB599FH7fGtrq3Jzc9XS0qIdO3bozTffVHl5uUpKSuyaw4cPKzc3V5MnT1Ztba0KCwv11FNPafPmzXbNunXrVFRUpEWLFmnPnj0aN26cvF6vGhoaOt0LAAAw201dKa6srIy4XV5eruTkZNXU1Ojee+9VU1OTXnvtNb311lu6//77JUlvvPGGbr/9du3cuVN33XWXtmzZok8++UR//vOflZKSovHjx+v555/Xz372My1evFhxcXFau3at0tPT9eKLL0qSbr/9dn3wwQdauXKlvF6vJGnFihWaM2eOZs2aJUlau3atKioq9Prrr+u5557rVC8AAMBsXQpCF2pqapIk9e/fX5JUU1OjcDis7Oxsu2bUqFEaOnSo/H6/7rrrLvn9fo0ZM0YpKSl2jdfr1dy5c3XgwAHdeeed8vv9EWu01xQWFkqSWlpaVFNTo+LiYvt8dHS0srOz5ff7O93LhUKhkEKhkH07GAxKksLhsMLh8BXN6GLa13JFW9225tXSnXNwUvvzuFGez/WG+TuL+TuPPehZXZnrFQehtrY2FRYW6jvf+Y5Gjx4tSQoEAoqLi1NSUlJEbUpKigKBgF1zfghqP99+7lI1wWBQZ86c0YkTJ9Ta2nrRmoMHD3a6lwuVlpZqyZIlHY5v2bJFCQkJXzeKK/b8xLZuX7Onbdq0yekWupXP53O6BaMxf2cxf+exBz3j9OnTna694iCUn5+v/fv364MPPrjSJa45xcXFKioqsm8Hg0ENGTJEOTk5crvd3fY44XBYPp9PP/8oWqG2qG5b92rYv9jrdAvdon0PpkyZotjYWKfbMQ7zdxbzdx570LPa39HpjCsKQgUFBdq4caO2b9+uwYMH28dTU1PV0tKixsbGiFdi6uvrlZqaatdceHVX+5Vc59dceHVXfX293G63evXqpZiYGMXExFy05vw1LtfLhVwul1wuV4fjsbGxPfIXNdQWpVDr9RWEbrR/sD21t+gc5u8s5u889qBndGWmXbpqzLIsFRQU6J133tHWrVuVnp4ecT4zM1OxsbGqqqqyjx06dEh1dXXyeDySJI/Ho3379kVc3eXz+eR2u5WRkWHXnL9Ge037GnFxccrMzIyoaWtrU1VVlV3TmV4AAIDZuvSKUH5+vt566y398Y9/VN++fe3P2iQmJqpXr15KTEzU7NmzVVRUpP79+8vtduvZZ5+Vx+OxP5yck5OjjIwMPfHEE1q+fLkCgYAWLlyo/Px8+9WYZ555RmvWrNGCBQv05JNPauvWrVq/fr0qKirsXoqKipSXl6eJEydq0qRJWrVqlZqbm+2ryDrTCwAAMFuXgtArr7wiSbrvvvsijr/xxhv60Y9+JElauXKloqOjNX36dIVCIXm9Xr388st2bUxMjDZu3Ki5c+fK4/God+/eysvL09KlS+2a9PR0VVRUaN68eVq9erUGDx6sV1991b50XpJmzJihr776SiUlJQoEAho/frwqKysjPkB9uV4AAIDZuhSELOvyl3vHx8errKxMZWVlX1szbNiwy159dN9992nv3r2XrCkoKFBBQcE36gUAAJiL3zUGAACMRRACAADGIggBAABjEYQAAICxCEIAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMBZBCAAAGIsgBAAAjEUQAgAAxiIIAQAAYxGEAACAsQhCAADAWAQhAABgLIIQAAAwFkEIAAAYiyAEAACMRRACAADGIggBAABjEYQAAICxCEIAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMBZBCAAAGIsgBAAAjEUQAgAAxiIIAQAAYxGEAACAsQhCAADAWAQhAABgLIIQAAAwFkEIAAAYiyAEAACMdZPTDeD6Mvy5Cqdb6LIvluU63QIA4BrFK0IAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMFaXg9D27dv10EMPKS0tTVFRUXr33XcjzluWpZKSEg0aNEi9evVSdna2Pvvss4ia48ePa+bMmXK73UpKStLs2bN16tSpiJqPP/5Y99xzj+Lj4zVkyBAtX768Qy8bNmzQqFGjFB8frzFjxmjTpk1d7gUAAJiry0GoublZ48aNU1lZ2UXPL1++XC+99JLWrl2rXbt2qXfv3vJ6vTp79qxdM3PmTB04cEA+n08bN27U9u3b9fTTT9vng8GgcnJyNGzYMNXU1OiFF17Q4sWL9Zvf/Mau2bFjhx5//HHNnj1be/fu1SOPPKJHHnlE+/fv71IvAADAXF3+7fPTpk3TtGnTLnrOsiytWrVKCxcu1MMPPyxJ+u1vf6uUlBS9++67euyxx/Tpp5+qsrJSH374oSZOnChJ+tWvfqUHH3xQ//qv/6q0tDT9/ve/V0tLi15//XXFxcXpjjvuUG1trVasWGEHptWrV2vq1KmaP3++JOn555+Xz+fTmjVrtHbt2k71AgAAzNblIHQphw8fViAQUHZ2tn0sMTFRWVlZ8vv9euyxx+T3+5WUlGSHIEnKzs5WdHS0du3ape9+97vy+/269957FRcXZ9d4vV798pe/1IkTJ9SvXz/5/X4VFRVFPL7X67XfqutMLxcKhUIKhUL27WAwKEkKh8MKh8PfbDjnaV/LFW1125r4ehfbu/Zj3bmv6Dzm7yzm7zz2oGd1Za7dGoQCgYAkKSUlJeJ4SkqKfS4QCCg5OTmyiZtuUv/+/SNq0tPTO6zRfq5fv34KBAKXfZzL9XKh0tJSLVmypMPxLVu2KCEh4Wue9ZV7fmJbt6+Jji787Nj5fD7fVewEF2L+zmL+zmMPesbp06c7XdutQeh6V1xcHPEqUzAY1JAhQ5STkyO3291tjxMOh+Xz+fTzj6IVaovqtnVxcfsXezsca9+DKVOmKDY21oGuzMb8ncX8ncce9Kz2d3Q6o1uDUGpqqiSpvr5egwYNso/X19dr/Pjxdk1DQ0PE/c6dO6fjx4/b909NTVV9fX1ETfvty9Wcf/5yvVzI5XLJ5XJ1OB4bG9sjf1FDbVEKtRKEetql9q6n9hadw/ydxfydxx70jK7MtFt/jlB6erpSU1NVVVVlHwsGg9q1a5c8Ho8kyePxqLGxUTU1NXbN1q1b1dbWpqysLLtm+/btEe/x+Xw+jRw5Uv369bNrzn+c9pr2x+lMLwAAwGxdDkKnTp1SbW2tamtrJf3fh5Jra2tVV1enqKgoFRYW6he/+IX+4z/+Q/v27dMPf/hDpaWl6ZFHHpEk3X777Zo6darmzJmj3bt36y9/+YsKCgr02GOPKS0tTZL0gx/8QHFxcZo9e7YOHDigdevWafXq1RFvW/34xz9WZWWlXnzxRR08eFCLFy/WRx99pIKCAknqVC8AAMBsXX5r7KOPPtLkyZPt2+3hJC8vT+Xl5VqwYIGam5v19NNPq7GxUXfffbcqKysVHx9v3+f3v/+9CgoK9MADDyg6OlrTp0/XSy+9ZJ9PTEzUli1blJ+fr8zMTA0cOFAlJSURP2voH//xH/XWW29p4cKF+ud//mfdeuutevfddzV69Gi7pjO9AAAAc3U5CN13332yrK+/7DsqKkpLly7V0qVLv7amf//+euutty75OGPHjtV//ud/XrLm+9//vr7//e9/o14AAIC5+F1jAADAWAQhAABgLIIQAAAwFkEIAAAYiyAEAACMRRACAADGIggBAABjEYQAAICxCEIAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMBZBCAAAGIsgBAAAjEUQAgAAxiIIAQAAYxGEAACAsQhCAADAWAQhAABgLIIQAAAwFkEIAAAYiyAEAACMRRACAADGIggBAABjEYQAAICxCEIAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMBZBCAAAGIsgBAAAjEUQAgAAxrrJ6QaAnjb8uYoOx1wxlpZPkkYv3qxQa5QDXV3aF8tynW4BAIzAK0IAAMBYBCEAAGAsghAAADAWQQgAABiLIAQAAIxFEAIAAMYiCAEAAGMRhAAAgLEIQgAAwFgEIQAAYCyCEAAAMBZBCAAAGIsgBAAAjMVvnweuQcOfq3C6hS77Ylmu0y0AQJcZ8YpQWVmZhg8frvj4eGVlZWn37t1OtwQAAK4BN3wQWrdunYqKirRo0SLt2bNH48aNk9frVUNDg9OtAQAAh93wb42tWLFCc+bM0axZsyRJa9euVUVFhV5//XU999xzDncH3Di68naeK8bS8knS6MWbFWqN6sGuLo238wDc0EGopaVFNTU1Ki4uto9FR0crOztbfr+/Q30oFFIoFLJvNzU1SZKOHz+ucDjcbX2Fw2GdPn1aN4Wj1drm3DcBk93UZun06Tb2wCHXyvyPHTvm2GM7qf1r0LFjxxQbG+t0O0ZiD3rWyZMnJUmWZV229oYOQn//+9/V2tqqlJSUiOMpKSk6ePBgh/rS0lItWbKkw/H09PQe6xHO+YHTDRjuWpj/wBed7gBATzp58qQSExMvWXNDB6GuKi4uVlFRkX27ra1Nx48f14ABAxQV1X3/1xoMBjVkyBB9+eWXcrvd3bYuOo89cBbzdxbzdx570LMsy9LJkyeVlpZ22dobOggNHDhQMTExqq+vjzheX1+v1NTUDvUul0sulyviWFJSUo/153a7+QfgMPbAWczfWczfeexBz7ncK0HtbuirxuLi4pSZmamqqir7WFtbm6qqquTxeBzsDAAAXAtu6FeEJKmoqEh5eXmaOHGiJk2apFWrVqm5udm+igwAAJjrhg9CM2bM0FdffaWSkhIFAgGNHz9elZWVHT5AfTW5XC4tWrSow9twuHrYA2cxf2cxf+exB9eOKKsz15YBAADcgG7ozwgBAABcCkEIAAAYiyAEAACMRRACAADGIgg5oKysTMOHD1d8fLyysrK0e/dup1u6ISxevFhRUVERf0aNGmWfP3v2rPLz8zVgwAD16dNH06dP7/DDNuvq6pSbm6uEhAQlJydr/vz5Onfu3NV+KteF7du366GHHlJaWpqioqL07rvvRpy3LEslJSUaNGiQevXqpezsbH322WcRNcePH9fMmTPldruVlJSk2bNn69SpUxE1H3/8se655x7Fx8dryJAhWr58eU8/tevC5eb/ox/9qMO/h6lTp0bUMP8rV1paqm9/+9vq27evkpOT9cgjj+jQoUMRNd31Nae6uloTJkyQy+XSiBEjVF5e3tNPzygEoats3bp1Kioq0qJFi7Rnzx6NGzdOXq9XDQ0NTrd2Q7jjjjt09OhR+88HH3xgn5s3b57ee+89bdiwQdu2bdORI0f06KOP2udbW1uVm5urlpYW7dixQ2+++abKy8tVUlLixFO55jU3N2vcuHEqKyu76Pnly5frpZde0tq1a7Vr1y717t1bXq9XZ8+etWtmzpypAwcOyOfzaePGjdq+fbuefvpp+3wwGFROTo6GDRummpoavfDCC1q8eLF+85vf9Pjzu9Zdbv6SNHXq1Ih/D3/4wx8izjP/K7dt2zbl5+dr586d8vl8CofDysnJUXNzs13THV9zDh8+rNzcXE2ePFm1tbUqLCzUU089pc2bN1/V53tDs3BVTZo0ycrPz7dvt7a2WmlpaVZpaamDXd0YFi1aZI0bN+6i5xobG63Y2Fhrw4YN9rFPP/3UkmT5/X7Lsixr06ZNVnR0tBUIBOyaV155xXK73VYoFOrR3q93kqx33nnHvt3W1malpqZaL7zwgn2ssbHRcrlc1h/+8AfLsizrk08+sSRZH374oV3zpz/9yYqKirL+93//17Isy3r55Zetfv36Rcz/Zz/7mTVy5MgefkbXlwvnb1mWlZeXZz388MNfex/m370aGhosSda2bdssy+q+rzkLFiyw7rjjjojHmjFjhuX1env6KRmDV4SuopaWFtXU1Cg7O9s+Fh0drezsbPn9fgc7u3F89tlnSktL0y233KKZM2eqrq5OklRTU6NwOBwx+1GjRmno0KH27P1+v8aMGRPxwza9Xq+CwaAOHDhwdZ/Ide7w4cMKBAIR805MTFRWVlbEvJOSkjRx4kS7Jjs7W9HR0dq1a5ddc++99youLs6u8Xq9OnTokE6cOHGVns31q7q6WsnJyRo5cqTmzp2rY8eO2eeYf/dqamqSJPXv319S933N8fv9EWu01/A9o/sQhK6iv//972ptbe3wU61TUlIUCAQc6urGkZWVpfLyclVWVuqVV17R4cOHdc899+jkyZMKBAKKi4vr8Et0z599IBC46N60n0Pntc/rUn/XA4GAkpOTI87fdNNN6t+/P3vSDaZOnarf/va3qqqq0i9/+Utt27ZN06ZNU2trqyTm353a2tpUWFio73znOxo9erQkddvXnK+rCQaDOnPmTE88HePc8L9iA+aYNm2a/d9jx45VVlaWhg0bpvXr16tXr14OdgZcfY899pj932PGjNHYsWP1D//wD6qurtYDDzzgYGc3nvz8fO3fvz/iM4m4fvCK0FU0cOBAxcTEdLhqoL6+XqmpqQ51deNKSkrSbbfdps8//1ypqalqaWlRY2NjRM35s09NTb3o3rSfQ+e1z+tSf9dTU1M7XCRw7tw5HT9+nD3pAbfccosGDhyozz//XBLz7y4FBQXauHGj3n//fQ0ePNg+3l1fc76uxu128z943YQgdBXFxcUpMzNTVVVV9rG2tjZVVVXJ4/E42NmN6dSpU/rrX/+qQYMGKTMzU7GxsRGzP3TokOrq6uzZezwe7du3L+Kbg8/nk9vtVkZGxlXv/3qWnp6u1NTUiHkHg0Ht2rUrYt6NjY2qqamxa7Zu3aq2tjZlZWXZNdu3b1c4HLZrfD6fRo4cqX79+l2lZ3Nj+Nvf/qZjx45p0KBBkpj/N2VZlgoKCvTOO+9o69atSk9PjzjfXV9zPB5PxBrtNXzP6EZOf1rbNG+//bblcrms8vJy65NPPrGefvppKykpKeKqAVyZn/zkJ1Z1dbV1+PBh6y9/+YuVnZ1tDRw40GpoaLAsy7KeeeYZa+jQodbWrVutjz76yPJ4PJbH47Hvf+7cOWv06NFWTk6OVVtba1VWVlo333yzVVxc7NRTuqadPHnS2rt3r7V3715LkrVixQpr79691v/8z/9YlmVZy5Yts5KSkqw//vGP1scff2w9/PDDVnp6unXmzBl7jalTp1p33nmntWvXLuuDDz6wbr31Vuvxxx+3zzc2NlopKSnWE088Ye3fv996++23rYSEBOvXv/71VX++15pLzf/kyZPWT3/6U8vv91uHDx+2/vznP1sTJkywbr31Vuvs2bP2Gsz/ys2dO9dKTEy0qqurraNHj9p/Tp8+bdd0x9ec//7v/7YSEhKs+fPnW59++qlVVlZmxcTEWJWVlVf1+d7ICEIO+NWvfmUNHTrUiouLsyZNmmTt3LnT6ZZuCDNmzLAGDRpkxcXFWd/61resGTNmWJ9//rl9/syZM9Y//dM/Wf369bMSEhKs7373u9bRo0cj1vjiiy+sadOmWb169bIGDhxo/eQnP7HC4fDVfirXhffff9+S1OFPXl6eZVn/dwn9z3/+cyslJcVyuVzWAw88YB06dChijWPHjlmPP/641adPH8vtdluzZs2yTp48GVHzX//1X9bdd99tuVwu61vf+pa1bNmyq/UUr2mXmv/p06etnJwc6+abb7ZiY2OtYcOGWXPmzOnwP1zM/8pdbPaSrDfeeMOu6a6vOe+//741fvx4Ky4uzrrlllsiHgPfXJRlWdbVfhUKAADgWsBnhAAAgLEIQgAAwFgEIQAAYCyCEAAAMBZBCAAAGIsgBAAAjEUQAgAAxiIIAQAAYxGEAACAsQhCAADAWAQhAABgLIIQAAAw1v8D0x6iUSmXSKkAAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "train_df[\"input_ids\"].apply(len).hist()" + ] + }, + { + "cell_type": "code", + "execution_count": 69, + "metadata": {}, + "outputs": [], + "source": [ + "train_df[\"input_ids_token_length\"] = train_df[\"input_ids\"].apply(len)" + ] + }, + { + "cell_type": "code", + "execution_count": 71, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
labeltextinput_idsattention_maskinput_ids_token_length
6144193Pour le jour de l'an 2010, nous \\u00e9tions a ...[101, 10364, 3393, 8183, 3126, 2139, 1048, 100...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...2344
5463234**Summary**\\n - Best - I have to start b...[101, 1008, 1008, 12654, 1008, 1008, 1032, 105...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...2045
4669192Torii @ Lavel, Quecec, Canada, \\u30ab\\u30ca\\u3...[101, 23413, 2072, 1030, 2474, 15985, 1010, 10...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...1966
5701024WED, MARCH 27, 2013 // 12:00PM-2:00PM \\n ...[101, 21981, 1010, 2233, 2676, 1010, 2286, 101...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...1903
6036241Was hier rumsitzt kennt das Angebot, diese bun...[101, 2001, 7632, 2121, 19379, 28032, 2480, 21...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...1896
..................
6409503The restaurant color scheme of white and brown...[101, 1996, 4825, 3609, 5679, 1997, 2317, 1998...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...513
2355802Service was absolutely stellar. However, for ...[101, 2326, 2001, 7078, 17227, 1012, 2174, 101...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...513
668692All three of my stars go to the excellent, fre...[101, 2035, 2093, 1997, 2026, 3340, 2175, 2000...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...513
4278340Absolutely horrible! I did the AYCE and starte...[101, 7078, 9202, 999, 1045, 2106, 1996, 1037,...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...513
1025281DH came to take me to lunch to celebrate a new...[101, 28144, 2234, 2000, 2202, 2033, 2000, 626...[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...513
\n", + "

30263 rows × 5 columns

\n", + "
" + ], + "text/plain": [ + " label text \n", + "614419 3 Pour le jour de l'an 2010, nous \\u00e9tions a ... \\\n", + "546323 4 **Summary**\\n - Best - I have to start b... \n", + "466919 2 Torii @ Lavel, Quecec, Canada, \\u30ab\\u30ca\\u3... \n", + "570102 4 WED, MARCH 27, 2013 // 12:00PM-2:00PM \\n ... \n", + "603624 1 Was hier rumsitzt kennt das Angebot, diese bun... \n", + "... ... ... \n", + "640950 3 The restaurant color scheme of white and brown... \n", + "235580 2 Service was absolutely stellar. However, for ... \n", + "66869 2 All three of my stars go to the excellent, fre... \n", + "427834 0 Absolutely horrible! I did the AYCE and starte... \n", + "102528 1 DH came to take me to lunch to celebrate a new... \n", + "\n", + " input_ids \n", + "614419 [101, 10364, 3393, 8183, 3126, 2139, 1048, 100... \\\n", + "546323 [101, 1008, 1008, 12654, 1008, 1008, 1032, 105... \n", + "466919 [101, 23413, 2072, 1030, 2474, 15985, 1010, 10... \n", + "570102 [101, 21981, 1010, 2233, 2676, 1010, 2286, 101... \n", + "603624 [101, 2001, 7632, 2121, 19379, 28032, 2480, 21... \n", + "... ... \n", + "640950 [101, 1996, 4825, 3609, 5679, 1997, 2317, 1998... \n", + "235580 [101, 2326, 2001, 7078, 17227, 1012, 2174, 101... \n", + "66869 [101, 2035, 2093, 1997, 2026, 3340, 2175, 2000... \n", + "427834 [101, 7078, 9202, 999, 1045, 2106, 1996, 1037,... \n", + "102528 [101, 28144, 2234, 2000, 2202, 2033, 2000, 626... \n", + "\n", + " attention_mask \n", + "614419 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... \\\n", + "546323 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... \n", + "466919 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... \n", + "570102 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... \n", + "603624 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... \n", + "... ... \n", + "640950 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... \n", + "235580 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... \n", + "66869 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... \n", + "427834 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... \n", + "102528 [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... \n", + "\n", + " input_ids_token_length \n", + "614419 2344 \n", + "546323 2045 \n", + "466919 1966 \n", + "570102 1903 \n", + "603624 1896 \n", + "... ... \n", + "640950 513 \n", + "235580 513 \n", + "66869 513 \n", + "427834 513 \n", + "102528 513 \n", + "\n", + "[30263 rows x 5 columns]" + ] + }, + "execution_count": 71, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "train_df[train_df[\"input_ids_token_length\"] > 512].sort_values(by=\"input_ids_token_length\", ascending=False)" + ] + }, + { + "cell_type": "code", + "execution_count": 76, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "0.04655846153846154" + ] + }, + "execution_count": 76, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "\n", + "train_df[train_df[\"input_ids_token_length\"] > 512].shape[0] / train_df.shape[0]" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Observation: \n", + " \n", + " Less than 5% of the reviews are longer than 512 tokens (which is the maximum length of the input for the DistilBERT model) meaning they will be truncated and we will lose information. This isn't ideal because we want to be able to classify the sentiment of the entire review, not just the first 512 tokens. \n", + "\n", + "For our use case, it seems reasonable (intuitively) that our model can assess the sentiment of a review based on the first 512 tokens. This analysis is still useful as it gives us a quantitative measure of our information loss by using Distilbert.\n", + "\n", + "##### Alternative approaches for a use-case where the entire text is important:\n", + " - We could try to split the reviews into multiple segments and then use the model to classify each segment and then aggregate the results to get a final sentiment score. \n", + " - We could try to use a model that doesn't have a maximum length, such as XLNet. This would allow us to classify the sentiment of the entire review, but it would also mean that we would need to pad the input to the maximum length of the dataset, which would be wasteful for the majority of reviews that are shorter than the maximum length. That will come at the cost of increased model size and training time." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "longest_review = train_df[train_df[\"input_ids_token_length\"] > 512][\"text\"].values[0]" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Run the tokenizer on individual examples" + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "metadata": { + "id": "bNpx6BhJaBmV" + }, + "outputs": [], + "source": [ + "\n", + "tokenized_example_not_truncated = tokenizer(longest_review, return_tensors=\"pt\") # return_tensors=\"pt\" means return pytorch tensors\n" + ] + }, + { + "cell_type": "code", + "execution_count": 44, + "metadata": { + "id": "uL_ehIYhX7FE" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'input_ids': tensor([[ 101, 2057, 2234, ..., 2925, 1012, 102]]),\n", + " 'attention_mask': tensor([[1, 1, 1, ..., 1, 1, 1]])}" + ] + }, + "execution_count": 44, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tokenized_example_not_truncated.data" + ] + }, + { + "cell_type": "code", + "execution_count": 45, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "torch.Size([1043])" + ] + }, + "execution_count": 45, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tokenized_example_not_truncated[\"input_ids\"][0].shape" + ] + }, + { + "cell_type": "code", + "execution_count": 46, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "torch.Size([512])" + ] + }, + "execution_count": 46, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tokenized_example_truncated = tokenizer(\"longest_review\", padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n", + "tokenized_example_truncated[\"input_ids\"][0].shape" + ] + }, + { + "cell_type": "code", + "execution_count": 47, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "torch.Size([512])" + ] + }, + "execution_count": 47, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tokenized_example_padded = tokenizer(\"Solid haircut\", padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n", + "tokenized_example_padded[\"input_ids\"][0].shape" + ] + }, + { + "cell_type": "code", + "execution_count": 49, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "torch.Size([512])" + ] + }, + "execution_count": 49, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tokenized_example_padded = tokenizer(\"Solid haircut\", padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n", + "tokenized_example_padded[\"input_ids\"][0].shape" + ] + }, + { + "attachments": {}, + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Inspect the string representations of the tokens" + ] + }, + { + "cell_type": "code", + "execution_count": 55, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'[CLS] solid haircut [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]'" + ] + }, + "execution_count": 55, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# decode the tokenized example\n", + "tokenized_example = tokenizer(\"Solid haircut\", padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n", + "tokenizer.decode(tokenized_example[\"input_ids\"][0])" + ] + }, + { + "cell_type": "code", + "execution_count": 66, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'[CLS] dana engebretson embedding [SEP] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD] [PAD]'" + ] + }, + "execution_count": 66, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# try your name\n", + "tokenized_example = tokenizer(\"Dana Engebretson embedding\", padding=\"max_length\", truncation=True, return_tensors=\"pt\")\n", + "tokenizer.decode(tokenized_example[\"input_ids\"][0]) # why is this not breaking engebretson into separate tokens like below with tokenizer.tokenize?" + ] + }, + { + "cell_type": "code", + "execution_count": 65, + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['dana', 'eng', '##eb', '##ret', '##son']" + ] + }, + "execution_count": 65, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "tokenizer.tokenize(\"Dana Engebretson\")" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "accelerator": "GPU", + "colab": { + "authorship_tag": "ABX9TyNy747vVqwhy2qH6PYwLuCv", + "include_colab_link": true, + "provenance": [] + }, + "gpuClass": "standard", + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.9.16" + }, + "widgets": { + "application/vnd.jupyter.widget-state+json": { + "0255272bfb8b43fa83aa14c5b6577487": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_b1045684e25f42a1b673e56ce4b423bc", + "placeholder": "​", + "style": "IPY_MODEL_a4f36826230042d88ab6eeeb02eba2b2", + "value": "Generating test split: 0%" + } + }, + "05139a04863243dc83d3f9fd0d74114b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "06a151da1234476cba405feae6f92d2b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "08cb8909e871484b96903a77f71a4ce7": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_32e02928eafe4375af034bb65a50e291", + "IPY_MODEL_c4843dcd436b4933b0c427a28b95b34a", + "IPY_MODEL_c5691040b26f4966bf6259c4de77e06d" + ], + "layout": "IPY_MODEL_98ffcd00835643e4aac3cfdde3bef4a0" + } + }, + "0e28ffed3f724a99bed0d3a07d146a69": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "11cf4d8d42c94b1aab7382e1125d3357": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "1417b492f37d4d938616994d6b3ca965": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "17646882017441eeb39a78699bbe33f6": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "185dd29085e44cdb97e7afe869559f24": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "1a107e4030124c8bbe6b32c5f640fb95": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "1a27ecacd77c4865b5bf0941b172342a": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "1a8df02eab7c46498e1f9c60f0f5b04f": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "1ac4807c27e146a192a7c2023d284bcc": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "1b5357b9d8b8438992be3d048a12b7f6": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "1e5e2e6f1c784953ab049228bbddff03": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": "hidden", + "width": null + } + }, + "1f342452365d47dba07dc1f10167f831": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_f86082c26ab1446398f3e9b609ee25ba", + "placeholder": "​", + "style": "IPY_MODEL_ce893b8c74724d698838f5563f9c4d64", + "value": "Downloading readme: 100%" + } + }, + "1f835219eca344df998fece53a82b815": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_a7598e7ea47a4c24adfb2776182dbe67", + "placeholder": "​", + "style": "IPY_MODEL_656ab77ac4584e5cb635f721ef106a2d", + "value": "Generating validation split: 0%" + } + }, + "2079c7c036bd45f4af8286ea9d177969": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_b28801b13d184fb391676075daa846ef", + "max": 19312, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_17646882017441eeb39a78699bbe33f6", + "value": 19312 + } + }, + "212a56d5dbd74c44b06b72341b65ee04": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_9e65b453455040e6a5cacd6324274cc7", + "IPY_MODEL_2079c7c036bd45f4af8286ea9d177969", + "IPY_MODEL_b05a81f1bf3f4d52b831f42dd34e79c1" + ], + "layout": "IPY_MODEL_f438cd0761c1420db5fcb3a588d305e7" + } + }, + "24b2ec25a5f54deaa88a65da11cd646d": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "251b565174da44309616141bc32c6b53": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "25232ca45bdf40089e36e858ef1b78f0": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "257612fcac154f46a14295d611681280": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_0e28ffed3f724a99bed0d3a07d146a69", + "placeholder": "​", + "style": "IPY_MODEL_4d2fb2f73b6c4a678cd35185a0a520b3", + "value": "Downloading metadata: 100%" + } + }, + "276aa6f5994e45ed993ef18b4212394c": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_bd26a0c9556444fa8cda018cf0c88688", + "placeholder": "​", + "style": "IPY_MODEL_06a151da1234476cba405feae6f92d2b", + "value": "Downloading builder script: 100%" + } + }, + "2db78e5dcf394fc2befc3152faee0cdb": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "2f162cdc30c94bb9a421661ff8f40d10": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_eb966af45cf340b1926c4b903a0e58ea", + "placeholder": "​", + "style": "IPY_MODEL_24b2ec25a5f54deaa88a65da11cd646d", + "value": " 4.75k/? [00:00<00:00, 213kB/s]" + } + }, + "2ffe627ee8f043d1ada61a815b23fb69": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "317149372b974b26bad2aa1e3361e990": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_d0b0aef997ca4d79bb6088436e34fe4a", + "IPY_MODEL_7ea0157acf944ed893755a45781f8106", + "IPY_MODEL_fff4a1cae63947dea76e5ded7c5d8941" + ], + "layout": "IPY_MODEL_1e5e2e6f1c784953ab049228bbddff03" + } + }, + "32e02928eafe4375af034bb65a50e291": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_d9cea257103e430f9e54e9856ee21267", + "placeholder": "​", + "style": "IPY_MODEL_185dd29085e44cdb97e7afe869559f24", + "value": "100%" + } + }, + "336c1eab88f5455f8d11ac0b15889a13": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "33e12d196c7a4b3292d63fdcd622d17b": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "35a230bea6a847dcb340aa126536cf7e": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_1f835219eca344df998fece53a82b815", + "IPY_MODEL_3c1ad98935f74925bceecd60d3271dfd", + "IPY_MODEL_88260cb9bdce41149ded15efe96908bf" + ], + "layout": "IPY_MODEL_4c58d90f75f746f8a3dd57ea372a61ed" + } + }, + "37cef5abfc9e45fe80991f74014d86c3": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "390b9eac79164ecc8ec3419617625a5f": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "3911d344c7294faab1cadb53a252e38a": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "3c1ad98935f74925bceecd60d3271dfd": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_881bdf44ce9648dcbda8391ae22d9c8c", + "max": 105, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_e0bc075493664c40a0a3e60567f9ed8e", + "value": 105 + } + }, + "3d0586a821e942f8a33c1eb957771cef": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": "hidden", + "width": null + } + }, + "402c33f8d8c94a16ba341f1ca9384627": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "4111c760131b4d82b5edcd74bebf59b1": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "43761e8ebb0d4040b65fe74e2bd12ba1": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": "hidden", + "width": null + } + }, + "44b9af84cd8147329fbf55a92260053f": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_76369a70c01546dc8b9e4d24d6421c5a", + "placeholder": "​", + "style": "IPY_MODEL_8456cf2c339e45a2948321680498c13e", + "value": "Downloading data: " + } + }, + "473fb37d5e2a4a41904341bb749375e2": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_33e12d196c7a4b3292d63fdcd622d17b", + "placeholder": "​", + "style": "IPY_MODEL_72e27c65f029428994cd687a71874374", + "value": "Map: 0%" + } + }, + "4871b34da19a4a34b33ddc85a990fb4f": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_2db78e5dcf394fc2befc3152faee0cdb", + "placeholder": "​", + "style": "IPY_MODEL_91a37e0a397c425a8c16456137771c57", + "value": " 892/892 [00:00<00:00, 1679.26 examples/s]" + } + }, + "49f9c038e4cc40afb08d8fc27861dfd2": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_1b5357b9d8b8438992be3d048a12b7f6", + "max": 892, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_25232ca45bdf40089e36e858ef1b78f0", + "value": 892 + } + }, + "4a7fab38ff804ab4ae8215f121ee5cb4": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "4bebd86d34f94c72bff834627e71adbc": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "4c58d90f75f746f8a3dd57ea372a61ed": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": "hidden", + "width": null + } + }, + "4d2fb2f73b6c4a678cd35185a0a520b3": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "5018da3ba3fd4ce986a79faa58a620c9": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "5199dd91d2324aca9da01c51ed812107": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "519d214fd7c14a9d82b57bf48dfb4982": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_a8b2baeac83247dfb51795ea750cbd5d", + "placeholder": "​", + "style": "IPY_MODEL_8305f804414c42a19cbb7fe1699aa6f3", + "value": "Map: 100%" + } + }, + "521c96ab56cd49c4b6350cfeba2edc7f": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_e5c10bfeb0a643299426a7a8b19fbe3c", + "placeholder": "​", + "style": "IPY_MODEL_d4fb3448c39149c08372c0e897ffd7f6", + "value": " 0/104 [00:00<?, ? examples/s]" + } + }, + "5374071fbada4b2d8e65fab009345259": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "5959f54cc19c418d8b710613b8ff44d6": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "5b2ddc4b19984e35b4ce710124c3a364": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "603fc9e2063643b2848f538efc3edde5": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_519d214fd7c14a9d82b57bf48dfb4982", + "IPY_MODEL_49f9c038e4cc40afb08d8fc27861dfd2", + "IPY_MODEL_4871b34da19a4a34b33ddc85a990fb4f" + ], + "layout": "IPY_MODEL_6f937e21f25e44ff87797c2fa6eb3005" + } + }, + "60dbb47d65474c61a2db10a2ab636818": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "64f2866f6452428b8a5cefe9f9b60106": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_3911d344c7294faab1cadb53a252e38a", + "placeholder": "​", + "style": "IPY_MODEL_b25d3b38f7a547f587044adf123fdec5", + "value": "Downloading data files: 100%" + } + }, + "656ab77ac4584e5cb635f721ef106a2d": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "69c7dcbdbcfb43e085cae897d7be872d": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_dfa0242aab26425996c5c2946d31e065", + "placeholder": "​", + "style": "IPY_MODEL_f056a241377747f6a8982a0eed36e384", + "value": " 0/892 [00:00<?, ? examples/s]" + } + }, + "6b0ba07cc6654481a603e1362be7bc80": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "6f937e21f25e44ff87797c2fa6eb3005": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": "hidden", + "width": null + } + }, + "72e27c65f029428994cd687a71874374": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "7319933cae3246008aa376d395739e6f": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_a2e09c1699994048b8e587401832014d", + "placeholder": "​", + "style": "IPY_MODEL_336c1eab88f5455f8d11ac0b15889a13", + "value": " 3.10k/3.10k [00:00<00:00, 178kB/s]" + } + }, + "738a30e9bc964898b4c5b4b6e0d1b25c": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "7501eea7d84b4d27aae128e5844a044b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_257612fcac154f46a14295d611681280", + "IPY_MODEL_8ef573f925fd4fe2890f695755caa8bd", + "IPY_MODEL_9208d75ae3c4428ca60e1c14e93aae57" + ], + "layout": "IPY_MODEL_fc19f1cde36c4322ac32c711458d0271" + } + }, + "76369a70c01546dc8b9e4d24d6421c5a": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "7a5b67db46504053b83d857b38f78787": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "7c0013b42c1647a49c1944322f80b901": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "7ea0157acf944ed893755a45781f8106": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_b68d9397030f4935b993ab85931fb599", + "max": 105, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_98db32e0dd684b298f511b5ea5fd61f2", + "value": 105 + } + }, + "7f3650b5468f4e1f8345c5b1cad4ef0a": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_a4106a031db9488985b290bd95036774", + "max": 104, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_4bebd86d34f94c72bff834627e71adbc", + "value": 104 + } + }, + "8069495c5d0a4a7a8adef81247e2befe": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "82068e9ff19946289a62c3d2d05f17b6": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "8305f804414c42a19cbb7fe1699aa6f3": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "8456cf2c339e45a2948321680498c13e": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "8641e8ef36ec491a9cec481074faa3ca": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "8692a908c7d84f3ca8e45674280e875f": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "881bdf44ce9648dcbda8391ae22d9c8c": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "88260cb9bdce41149ded15efe96908bf": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_82068e9ff19946289a62c3d2d05f17b6", + "placeholder": "​", + "style": "IPY_MODEL_88d8c1e54880470c81fbf48d2e915058", + "value": " 0/105 [00:00<?, ? examples/s]" + } + }, + "88d8c1e54880470c81fbf48d2e915058": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "8be4350405ab4ca5a9a0d61b00632df6": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "8c8d2028126f4011b160cf9141818a2b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_64f2866f6452428b8a5cefe9f9b60106", + "IPY_MODEL_9386856839f7481eb5c4f94321f65cfd", + "IPY_MODEL_d903a246b4d74783816546c0a1ba7526" + ], + "layout": "IPY_MODEL_9490ca33f6ea42ef97fc481901fed340" + } + }, + "8cee559d971c44dab2c028384b48fbda": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_5018da3ba3fd4ce986a79faa58a620c9", + "placeholder": "​", + "style": "IPY_MODEL_f9ab23effee947d79f0c5a1051313d7c", + "value": " 0/104 [00:00<?, ? examples/s]" + } + }, + "8e4a965c8fc04206bf43c86c68b79dd4": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": "hidden", + "width": null + } + }, + "8ef573f925fd4fe2890f695755caa8bd": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_6b0ba07cc6654481a603e1362be7bc80", + "max": 2098, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_05139a04863243dc83d3f9fd0d74114b", + "value": 2098 + } + }, + "91a37e0a397c425a8c16456137771c57": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "9208d75ae3c4428ca60e1c14e93aae57": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_a1c7b3169f4e44a8900cdbac374aee29", + "placeholder": "​", + "style": "IPY_MODEL_37cef5abfc9e45fe80991f74014d86c3", + "value": " 2.10k/2.10k [00:00<00:00, 119kB/s]" + } + }, + "9386856839f7481eb5c4f94321f65cfd": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_c51add9f3f674eb6a34a4d0ef43e27e7", + "max": 3, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_1a27ecacd77c4865b5bf0941b172342a", + "value": 3 + } + }, + "9490ca33f6ea42ef97fc481901fed340": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "98db32e0dd684b298f511b5ea5fd61f2": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "98ffcd00835643e4aac3cfdde3bef4a0": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "9ad729166a7f416d93f6cd6da9e26da5": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "9e65b453455040e6a5cacd6324274cc7": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_5374071fbada4b2d8e65fab009345259", + "placeholder": "​", + "style": "IPY_MODEL_390b9eac79164ecc8ec3419617625a5f", + "value": "Downloading data: " + } + }, + "9fd57c931f3a48889e37494d3d072f2a": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "a191629c94b6468cba48382e3e274539": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_276aa6f5994e45ed993ef18b4212394c", + "IPY_MODEL_b4d13a5656b048a8b7fa4d0acddceeba", + "IPY_MODEL_7319933cae3246008aa376d395739e6f" + ], + "layout": "IPY_MODEL_b27f582bc82345d7b8669b56cf777dd4" + } + }, + "a1c7b3169f4e44a8900cdbac374aee29": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "a2e09c1699994048b8e587401832014d": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "a4106a031db9488985b290bd95036774": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "a4f36826230042d88ab6eeeb02eba2b2": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "a7598e7ea47a4c24adfb2776182dbe67": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "a80b889df313482983002f5efe4ab2c6": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "a8b2baeac83247dfb51795ea750cbd5d": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "a9c48312271f4e7f8885c03afbc1c74b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_473fb37d5e2a4a41904341bb749375e2", + "IPY_MODEL_7f3650b5468f4e1f8345c5b1cad4ef0a", + "IPY_MODEL_8cee559d971c44dab2c028384b48fbda" + ], + "layout": "IPY_MODEL_8e4a965c8fc04206bf43c86c68b79dd4" + } + }, + "ab09b973df504a009bd93d3473a7e878": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_7a5b67db46504053b83d857b38f78787", + "max": 892, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_fd09ba0f798c4172a94665334bc107d8", + "value": 892 + } + }, + "ab7bccdbe46148aeb5b91de02db55c57": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_8069495c5d0a4a7a8adef81247e2befe", + "placeholder": "​", + "style": "IPY_MODEL_de798e94efee48ce8e5cfcbd69e71956", + "value": "Generating train split: 0%" + } + }, + "b05a81f1bf3f4d52b831f42dd34e79c1": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_9fd57c931f3a48889e37494d3d072f2a", + "placeholder": "​", + "style": "IPY_MODEL_11cf4d8d42c94b1aab7382e1125d3357", + "value": " 40.6k/? [00:00<00:00, 1.60MB/s]" + } + }, + "b1045684e25f42a1b673e56ce4b423bc": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "b25d3b38f7a547f587044adf123fdec5": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "b27f582bc82345d7b8669b56cf777dd4": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "b28801b13d184fb391676075daa846ef": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "b42d06ad9b354aefa34bf569eb578919": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_ab7bccdbe46148aeb5b91de02db55c57", + "IPY_MODEL_ab09b973df504a009bd93d3473a7e878", + "IPY_MODEL_69c7dcbdbcfb43e085cae897d7be872d" + ], + "layout": "IPY_MODEL_43761e8ebb0d4040b65fe74e2bd12ba1" + } + }, + "b494fbde6ba1448185329de69a024677": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "b4d13a5656b048a8b7fa4d0acddceeba": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_f4c7191d4cd44935a6d9b6484430682a", + "max": 3097, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_5b2ddc4b19984e35b4ce710124c3a364", + "value": 3097 + } + }, + "b68d9397030f4935b993ab85931fb599": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "ba67d14ee081429183f7cf7e8f998871": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_7c0013b42c1647a49c1944322f80b901", + "placeholder": "​", + "style": "IPY_MODEL_daad627e1eb046ea8041a63e57ff2193", + "value": " 5.51k/5.51k [00:00<00:00, 356kB/s]" + } + }, + "bcf0036e72ac469e8a39eae22d76a586": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_0255272bfb8b43fa83aa14c5b6577487", + "IPY_MODEL_f6d2587af20a47f8be304ac0b5daee75", + "IPY_MODEL_521c96ab56cd49c4b6350cfeba2edc7f" + ], + "layout": "IPY_MODEL_3d0586a821e942f8a33c1eb957771cef" + } + }, + "bd26a0c9556444fa8cda018cf0c88688": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "c0de1c77c31844b5a2fae6f4751e96d6": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "c204e25372ad4b4cb7fe390edd6f99d7": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "c4843dcd436b4933b0c427a28b95b34a": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_ec18adf6d12d487fba0056f451198d47", + "max": 3, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_251b565174da44309616141bc32c6b53", + "value": 3 + } + }, + "c51add9f3f674eb6a34a4d0ef43e27e7": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "c5691040b26f4966bf6259c4de77e06d": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_2ffe627ee8f043d1ada61a815b23fb69", + "placeholder": "​", + "style": "IPY_MODEL_f9557e28d8374e43b2aa9c9d7aebb909", + "value": " 3/3 [00:00<00:00, 122.01it/s]" + } + }, + "cdaaa306ec25433b92222a46f97c0a12": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "ce893b8c74724d698838f5563f9c4d64": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "d068d68e344d431d9a45ed804da66b11": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "d0b0aef997ca4d79bb6088436e34fe4a": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_cdaaa306ec25433b92222a46f97c0a12", + "placeholder": "​", + "style": "IPY_MODEL_9ad729166a7f416d93f6cd6da9e26da5", + "value": "Map: 100%" + } + }, + "d4fb3448c39149c08372c0e897ffd7f6": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "d61d318d4e104434b395cbb73174037b": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_1a107e4030124c8bbe6b32c5f640fb95", + "placeholder": "​", + "style": "IPY_MODEL_a80b889df313482983002f5efe4ab2c6", + "value": " 4.56k/? [00:00<00:00, 166kB/s]" + } + }, + "d6d047266e6f4ba09733ffb6f9a44806": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_1f342452365d47dba07dc1f10167f831", + "IPY_MODEL_efe4f4c187d643e9b6f49a47f8fd5787", + "IPY_MODEL_ba67d14ee081429183f7cf7e8f998871" + ], + "layout": "IPY_MODEL_c0de1c77c31844b5a2fae6f4751e96d6" + } + }, + "d903a246b4d74783816546c0a1ba7526": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_60dbb47d65474c61a2db10a2ab636818", + "placeholder": "​", + "style": "IPY_MODEL_d068d68e344d431d9a45ed804da66b11", + "value": " 3/3 [00:01<00:00, 1.70it/s]" + } + }, + "d9cea257103e430f9e54e9856ee21267": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "daad627e1eb046ea8041a63e57ff2193": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "de798e94efee48ce8e5cfcbd69e71956": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "dfa0242aab26425996c5c2946d31e065": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "e0bc075493664c40a0a3e60567f9ed8e": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "e14574af4f234dd39f071c47f944ad54": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_402c33f8d8c94a16ba341f1ca9384627", + "placeholder": "​", + "style": "IPY_MODEL_1a8df02eab7c46498e1f9c60f0f5b04f", + "value": "Downloading data: " + } + }, + "e5c10bfeb0a643299426a7a8b19fbe3c": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "e906635962a44b798d63ac43e6213aae": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_44b9af84cd8147329fbf55a92260053f", + "IPY_MODEL_fb777c1e52df48eaa76a9cbfe9235c88", + "IPY_MODEL_2f162cdc30c94bb9a421661ff8f40d10" + ], + "layout": "IPY_MODEL_5199dd91d2324aca9da01c51ed812107" + } + }, + "eb966af45cf340b1926c4b903a0e58ea": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "ec18adf6d12d487fba0056f451198d47": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "efe4f4c187d643e9b6f49a47f8fd5787": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_1ac4807c27e146a192a7c2023d284bcc", + "max": 5508, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_b494fbde6ba1448185329de69a024677", + "value": 5508 + } + }, + "f056a241377747f6a8982a0eed36e384": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "f438cd0761c1420db5fcb3a588d305e7": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "f4c7191d4cd44935a6d9b6484430682a": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "f6d2587af20a47f8be304ac0b5daee75": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_738a30e9bc964898b4c5b4b6e0d1b25c", + "max": 104, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_4111c760131b4d82b5edcd74bebf59b1", + "value": 104 + } + }, + "f86082c26ab1446398f3e9b609ee25ba": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "f924ac2de1bb4789831e79551e226e29": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_1417b492f37d4d938616994d6b3ca965", + "max": 2440, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_c204e25372ad4b4cb7fe390edd6f99d7", + "value": 2440 + } + }, + "f9557e28d8374e43b2aa9c9d7aebb909": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "f9ab23effee947d79f0c5a1051313d7c": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "DescriptionStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "DescriptionStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "description_width": "" + } + }, + "fac949fce47f4edcb262b62c2a1c50f3": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HBoxModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HBoxModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HBoxView", + "box_style": "", + "children": [ + "IPY_MODEL_e14574af4f234dd39f071c47f944ad54", + "IPY_MODEL_f924ac2de1bb4789831e79551e226e29", + "IPY_MODEL_d61d318d4e104434b395cbb73174037b" + ], + "layout": "IPY_MODEL_4a7fab38ff804ab4ae8215f121ee5cb4" + } + }, + "fb777c1e52df48eaa76a9cbfe9235c88": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "FloatProgressModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "FloatProgressModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "ProgressView", + "bar_style": "success", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_8641e8ef36ec491a9cec481074faa3ca", + "max": 2511, + "min": 0, + "orientation": "horizontal", + "style": "IPY_MODEL_8692a908c7d84f3ca8e45674280e875f", + "value": 2511 + } + }, + "fc19f1cde36c4322ac32c711458d0271": { + "model_module": "@jupyter-widgets/base", + "model_module_version": "1.2.0", + "model_name": "LayoutModel", + "state": { + "_model_module": "@jupyter-widgets/base", + "_model_module_version": "1.2.0", + "_model_name": "LayoutModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "LayoutView", + "align_content": null, + "align_items": null, + "align_self": null, + "border": null, + "bottom": null, + "display": null, + "flex": null, + "flex_flow": null, + "grid_area": null, + "grid_auto_columns": null, + "grid_auto_flow": null, + "grid_auto_rows": null, + "grid_column": null, + "grid_gap": null, + "grid_row": null, + "grid_template_areas": null, + "grid_template_columns": null, + "grid_template_rows": null, + "height": null, + "justify_content": null, + "justify_items": null, + "left": null, + "margin": null, + "max_height": null, + "max_width": null, + "min_height": null, + "min_width": null, + "object_fit": null, + "object_position": null, + "order": null, + "overflow": null, + "overflow_x": null, + "overflow_y": null, + "padding": null, + "right": null, + "top": null, + "visibility": null, + "width": null + } + }, + "fd09ba0f798c4172a94665334bc107d8": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "ProgressStyleModel", + "state": { + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "ProgressStyleModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/base", + "_view_module_version": "1.2.0", + "_view_name": "StyleView", + "bar_color": null, + "description_width": "" + } + }, + "fff4a1cae63947dea76e5ded7c5d8941": { + "model_module": "@jupyter-widgets/controls", + "model_module_version": "1.5.0", + "model_name": "HTMLModel", + "state": { + "_dom_classes": [], + "_model_module": "@jupyter-widgets/controls", + "_model_module_version": "1.5.0", + "_model_name": "HTMLModel", + "_view_count": null, + "_view_module": "@jupyter-widgets/controls", + "_view_module_version": "1.5.0", + "_view_name": "HTMLView", + "description": "", + "description_tooltip": null, + "layout": "IPY_MODEL_5959f54cc19c418d8b710613b8ff44d6", + "placeholder": "​", + "style": "IPY_MODEL_8be4350405ab4ca5a9a0d61b00632df6", + "value": " 105/105 [00:00<00:00, 932.00 examples/s]" + } + } + } + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/notebooks/ref_only/Inspect_BERT_Vocabulary.ipynb b/notebooks/ref_only/Inspect_BERT_Vocabulary.ipynb new file mode 100644 index 0000000..2fcc479 --- /dev/null +++ b/notebooks/ref_only/Inspect_BERT_Vocabulary.ipynb @@ -0,0 +1,1328 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "whjogPl1KL4-" + }, + "source": [ + "## Load the Model\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "O0wWu4oEKYtc" + }, + "source": [ + "Install the huggingface implementation." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 423 + }, + "id": "PzCjfNB6jksJ", + "outputId": "9aa79438-898e-4d43-8487-b0684dfe2623", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting pytorch-pretrained-bert\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/d7/e0/c08d5553b89973d9a240605b9c12404bcf8227590de62bae27acbcfe076b/pytorch_pretrained_bert-0.6.2-py3-none-any.whl (123kB)\n", + "\r\u001b[K |██▋ | 10kB 18.2MB/s eta 0:00:01\r\u001b[K |█████▎ | 20kB 6.7MB/s eta 0:00:01\r\u001b[K |████████ | 30kB 9.4MB/s eta 0:00:01\r\u001b[K |██████████▋ | 40kB 5.7MB/s eta 0:00:01\r\u001b[K |█████████████▎ | 51kB 6.9MB/s eta 0:00:01\r\u001b[K |███████████████▉ | 61kB 8.1MB/s eta 0:00:01\r\u001b[K |██████████████████▌ | 71kB 9.2MB/s eta 0:00:01\r\u001b[K |█████████████████████▏ | 81kB 10.3MB/s eta 0:00:01\r\u001b[K |███████████████████████▉ | 92kB 11.4MB/s eta 0:00:01\r\u001b[K |██████████████████████████▌ | 102kB 9.1MB/s eta 0:00:01\r\u001b[K |█████████████████████████████▏ | 112kB 9.1MB/s eta 0:00:01\r\u001b[K |███████████████████████████████▊| 122kB 9.1MB/s eta 0:00:01\r\u001b[K |████████████████████████████████| 133kB 9.1MB/s \n", + "\u001b[?25hRequirement already satisfied: torch>=0.4.1 in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (1.3.1)\n", + "Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (2.21.0)\n", + "Requirement already satisfied: tqdm in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (4.28.1)\n", + "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (1.17.4)\n", + "Collecting regex\n", + "\u001b[?25l Downloading https://files.pythonhosted.org/packages/e3/8e/cbf2295643d7265e7883326fb4654e643bfc93b3a8a8274d8010a39d8804/regex-2019.11.1-cp36-cp36m-manylinux1_x86_64.whl (643kB)\n", + "\u001b[K |████████████████████████████████| 645kB 31.3MB/s \n", + "\u001b[?25hRequirement already satisfied: boto3 in /usr/local/lib/python3.6/dist-packages (from pytorch-pretrained-bert) (1.10.18)\n", + "Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert) (1.24.3)\n", + "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert) (2019.9.11)\n", + "Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert) (2.8)\n", + "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->pytorch-pretrained-bert) (3.0.4)\n", + "Requirement already satisfied: s3transfer<0.3.0,>=0.2.0 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert) (0.2.1)\n", + "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert) (0.9.4)\n", + "Requirement already satisfied: botocore<1.14.0,>=1.13.18 in /usr/local/lib/python3.6/dist-packages (from boto3->pytorch-pretrained-bert) (1.13.18)\n", + "Requirement already satisfied: python-dateutil<2.8.1,>=2.1; python_version >= \"2.7\" in /usr/local/lib/python3.6/dist-packages (from botocore<1.14.0,>=1.13.18->boto3->pytorch-pretrained-bert) (2.6.1)\n", + "Requirement already satisfied: docutils<0.16,>=0.10 in /usr/local/lib/python3.6/dist-packages (from botocore<1.14.0,>=1.13.18->boto3->pytorch-pretrained-bert) (0.15.2)\n", + "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.6/dist-packages (from python-dateutil<2.8.1,>=2.1; python_version >= \"2.7\"->botocore<1.14.0,>=1.13.18->boto3->pytorch-pretrained-bert) (1.12.0)\n", + "Installing collected packages: regex, pytorch-pretrained-bert\n", + "Successfully installed pytorch-pretrained-bert-0.6.2 regex-2019.11.1\n" + ] + } + ], + "source": [ + "!pip install pytorch-pretrained-bert" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "nwpkz_WYjscB", + "outputId": "6d4b4bf5-a2db-4126-a9b7-3e873b60fe98", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "100%|██████████| 231508/231508 [00:00<00:00, 907607.65B/s]\n" + ] + } + ], + "source": [ + "import torch\n", + "from pytorch_pretrained_bert import BertTokenizer\n", + "\n", + "# Load pre-trained model tokenizer (vocabulary)\n", + "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "B8FD0tDpKVBF" + }, + "source": [ + "## Inspect BERT Vocabulary\n", + "--------------------------" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7UB5FRPOhinf" + }, + "source": [ + "### Vocab Dump\n", + "--------------\n", + "\n", + "Retrieve the entire list of \"tokens\" and write these out to text files so we can peruse them." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "NRf7NXJGgsnG", + "vscode": { + "languageId": "python" + } + }, + "outputs": [], + "source": [ + "with open(\"vocabulary.txt\", 'w') as f:\n", + " \n", + " # For each token...\n", + " for token in tokenizer.vocab.keys():\n", + " \n", + " # Write it out and escape any unicode characters. \n", + " f.write(token + '\\n')\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c8geTQ3Iidsv" + }, + "source": [ + "From perusing the vocab, I'm seeing that:\n", + "\n", + "* The first 999 tokens (1-indexed) appear to be reserved, and most are of the form [unused957].\n", + " * 1 - [PAD]\n", + " * 101 - [UNK]\n", + " * 102 - [CLS]\n", + " * 103 - [SEP]\n", + " * 104 - [MASK]\n", + "* Rows 1000-1996 appear to be a dump of individual characters. \n", + " * They don't appear to be sorted by frequency (e.g., the letters of the alphabet are all in sequence).\n", + "* The first word is \"the\" at position 1997.\n", + " * From there, the words appear to be sorted by frequency. \n", + " * The top ~18 words are whole words, and then number 2016 is ##s, presumably the most common subword.\n", + " * The last whole word is at 29612, \"necessitated\"\n", + "\n", + "Some funny inclusions:\n", + "* starbucks\n", + "* triassic\n", + "* abolitionist\n", + "* 1679" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "uviYdpTZYMmd" + }, + "source": [ + "### Single Characters\n", + "---------------------" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Bu-KbpbR1L4q" + }, + "source": [ + "The following code prints out all of the single character tokens in vocabulary, as well as all of the single-character tokens preceded by '##'.\n", + "\n", + "It turns out that these are matching sets--for every standalone character there is also a '##' version. There are 997 single character tokens." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Q6ysF8LD1fbl" + }, + "source": [ + "The following cell iterates over the vocabulary, pulling out all of the single character tokens." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "YHpH2NEPy3t9", + "vscode": { + "languageId": "python" + } + }, + "outputs": [], + "source": [ + "one_chars = []\n", + "one_chars_hashes = []\n", + "\n", + "# For each token in the vocabulary...\n", + "for token in tokenizer.vocab.keys():\n", + " \n", + " # Record any single-character tokens.\n", + " if len(token) == 1:\n", + " one_chars.append(token)\n", + " \n", + " # Record single-character tokens preceded by the two hashes. \n", + " elif len(token) == 3 and token[0:2] == '##':\n", + " one_chars_hashes.append(token)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 470 + }, + "id": "jCzfsjHrzZYn", + "outputId": "dfc169d2-d61c-4311-f89a-184b4b23e294", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of single character tokens: 997 \n", + "\n", + "! \" # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ [ \\ ] ^ _ ` a b\n", + "c d e f g h i j k l m n o p q r s t u v w x y z { | } ~ ¡ ¢ £ ¤ ¥ ¦ § ¨ © ª « ¬\n", + "® ° ± ² ³ ´ µ ¶ · ¹ º » ¼ ½ ¾ ¿ × ß æ ð ÷ ø þ đ ħ ı ł ŋ œ ƒ ɐ ɑ ɒ ɔ ɕ ə ɛ ɡ ɣ ɨ\n", + "ɪ ɫ ɬ ɯ ɲ ɴ ɹ ɾ ʀ ʁ ʂ ʃ ʉ ʊ ʋ ʌ ʎ ʐ ʑ ʒ ʔ ʰ ʲ ʳ ʷ ʸ ʻ ʼ ʾ ʿ ˈ ː ˡ ˢ ˣ ˤ α β γ δ\n", + "ε ζ η θ ι κ λ μ ν ξ ο π ρ ς σ τ υ φ χ ψ ω а б в г д е ж з и к л м н о п р с т у\n", + "ф х ц ч ш щ ъ ы ь э ю я ђ є і ј љ њ ћ ӏ ա բ գ դ ե թ ի լ կ հ մ յ ն ո պ ս վ տ ր ւ\n", + "ք ־ א ב ג ד ה ו ז ח ט י ך כ ל ם מ ן נ ס ע ף פ ץ צ ק ר ש ת ، ء ا ب ة ت ث ج ح خ د\n", + "ذ ر ز س ش ص ض ط ظ ع غ ـ ف ق ك ل م ن ه و ى ي ٹ پ چ ک گ ں ھ ہ ی ے अ आ उ ए क ख ग च\n", + "ज ट ड ण त थ द ध न प ब भ म य र ल व श ष स ह ा ि ी ो । ॥ ং অ আ ই উ এ ও ক খ গ চ ছ জ\n", + "ট ড ণ ত থ দ ধ ন প ব ভ ম য র ল শ ষ স হ া ি ী ে க ச ட த ந ன ப ம ய ர ல ள வ ா ி ு ே\n", + "ை ನ ರ ಾ ක ය ර ල ව ා ก ง ต ท น พ ม ย ร ล ว ส อ า เ ་ ། ག ང ད ན པ བ མ འ ར ལ ས မ ა\n", + "ბ გ დ ე ვ თ ი კ ლ მ ნ ო რ ს ტ უ ᄀ ᄂ ᄃ ᄅ ᄆ ᄇ ᄉ ᄊ ᄋ ᄌ ᄎ ᄏ ᄐ ᄑ ᄒ ᅡ ᅢ ᅥ ᅦ ᅧ ᅩ ᅪ ᅭ ᅮ\n", + "ᅯ ᅲ ᅳ ᅴ ᅵ ᆨ ᆫ ᆯ ᆷ ᆸ ᆼ ᴬ ᴮ ᴰ ᴵ ᴺ ᵀ ᵃ ᵇ ᵈ ᵉ ᵍ ᵏ ᵐ ᵒ ᵖ ᵗ ᵘ ᵢ ᵣ ᵤ ᵥ ᶜ ᶠ ‐ ‑ ‒ – — ―\n", + "‖ ‘ ’ ‚ “ ” „ † ‡ • … ‰ ′ ″ › ‿ ⁄ ⁰ ⁱ ⁴ ⁵ ⁶ ⁷ ⁸ ⁹ ⁺ ⁻ ⁿ ₀ ₁ ₂ ₃ ₄ ₅ ₆ ₇ ₈ ₉ ₊ ₍\n", + "₎ ₐ ₑ ₒ ₓ ₕ ₖ ₗ ₘ ₙ ₚ ₛ ₜ ₤ ₩ € ₱ ₹ ℓ № ℝ ™ ⅓ ⅔ ← ↑ → ↓ ↔ ↦ ⇄ ⇌ ⇒ ∂ ∅ ∆ ∇ ∈ − ∗\n", + "∘ √ ∞ ∧ ∨ ∩ ∪ ≈ ≡ ≤ ≥ ⊂ ⊆ ⊕ ⊗ ⋅ ─ │ ■ ▪ ● ★ ☆ ☉ ♠ ♣ ♥ ♦ ♭ ♯ ⟨ ⟩ ⱼ ⺩ ⺼ ⽥ 、 。 〈 〉\n", + "《 》 「 」 『 』 〜 あ い う え お か き く け こ さ し す せ そ た ち っ つ て と な に ぬ ね の は ひ ふ へ ほ ま み\n", + "む め も や ゆ よ ら り る れ ろ を ん ァ ア ィ イ ウ ェ エ オ カ キ ク ケ コ サ シ ス セ タ チ ッ ツ テ ト ナ ニ ノ ハ\n", + "ヒ フ ヘ ホ マ ミ ム メ モ ャ ュ ョ ラ リ ル レ ロ ワ ン ・ ー 一 三 上 下 不 世 中 主 久 之 也 事 二 五 井 京 人 亻 仁\n", + "介 代 仮 伊 会 佐 侍 保 信 健 元 光 八 公 内 出 分 前 劉 力 加 勝 北 区 十 千 南 博 原 口 古 史 司 合 吉 同 名 和 囗 四\n", + "国 國 土 地 坂 城 堂 場 士 夏 外 大 天 太 夫 奈 女 子 学 宀 宇 安 宗 定 宣 宮 家 宿 寺 將 小 尚 山 岡 島 崎 川 州 巿 帝\n", + "平 年 幸 广 弘 張 彳 後 御 德 心 忄 志 忠 愛 成 我 戦 戸 手 扌 政 文 新 方 日 明 星 春 昭 智 曲 書 月 有 朝 木 本 李 村\n", + "東 松 林 森 楊 樹 橋 歌 止 正 武 比 氏 民 水 氵 氷 永 江 沢 河 治 法 海 清 漢 瀬 火 版 犬 王 生 田 男 疒 発 白 的 皇 目\n", + "相 省 真 石 示 社 神 福 禾 秀 秋 空 立 章 竹 糹 美 義 耳 良 艹 花 英 華 葉 藤 行 街 西 見 訁 語 谷 貝 貴 車 軍 辶 道 郎\n", + "郡 部 都 里 野 金 鈴 镇 長 門 間 阝 阿 陳 陽 雄 青 面 風 食 香 馬 高 龍 龸 fi fl ！ （ ） ， － ． ／ ： ？ ～\n" + ] + } + ], + "source": [ + "print('Number of single character tokens:', len(one_chars), '\\n')\n", + "\n", + "# Print all of the single characters, 40 per row.\n", + "\n", + "# For every batch of 40 tokens...\n", + "for i in range(0, len(one_chars), 40):\n", + " \n", + " # Limit the end index so we don't go past the end of the list.\n", + " end = min(i + 40, len(one_chars) + 1)\n", + " \n", + " # Print out the tokens, separated by a space.\n", + " print(' '.join(one_chars[i:end]))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 470 + }, + "id": "62MD-M1N0R-W", + "outputId": "3e5a3866-b969-4eb2-8984-8580c7b51ab9", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of single character tokens with hashes: 997 \n", + "\n", + "s a e i n o d r y t l m u h k c g p 2 z 1 b 3 f 4 6 7 x v 8 5 9 0 w j q ° ₂ а и\n", + "² ₃ ı ₁ ⁺ ½ о ه ي α е د ن ν ø р ₄ ₀ ر я ³ ι ł н ᵢ ₙ ß ة ς م − т ː ل ь к ♭ η ی в\n", + "ا × ¹ ы ה ɛ л ! \" # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \\ ] ^ _ ` { | } ~ ¡\n", + "¢ £ ¤ ¥ ¦ § ¨ © ª « ¬ ® ± ´ µ ¶ · º » ¼ ¾ ¿ æ ð ÷ þ đ ħ ŋ œ ƒ ɐ ɑ ɒ ɔ ɕ ə ɡ ɣ ɨ\n", + "ɪ ɫ ɬ ɯ ɲ ɴ ɹ ɾ ʀ ʁ ʂ ʃ ʉ ʊ ʋ ʌ ʎ ʐ ʑ ʒ ʔ ʰ ʲ ʳ ʷ ʸ ʻ ʼ ʾ ʿ ˈ ˡ ˢ ˣ ˤ β γ δ ε ζ\n", + "θ κ λ μ ξ ο π ρ σ τ υ φ χ ψ ω б г д ж з м п с у ф х ц ч ш щ ъ э ю ђ є і ј љ њ ћ\n", + "ӏ ա բ գ դ ե թ ի լ կ հ մ յ ն ո պ ս վ տ ր ւ ք ־ א ב ג ד ו ז ח ט י ך כ ל ם מ ן נ ס\n", + "ע ף פ ץ צ ק ר ש ת ، ء ب ت ث ج ح خ ذ ز س ش ص ض ط ظ ع غ ـ ف ق ك و ى ٹ پ چ ک گ ں ھ\n", + "ہ ے अ आ उ ए क ख ग च ज ट ड ण त थ द ध न प ब भ म य र ल व श ष स ह ा ि ी ो । ॥ ং অ আ\n", + "ই উ এ ও ক খ গ চ ছ জ ট ড ণ ত থ দ ধ ন প ব ভ ম য র ল শ ষ স হ া ি ী ে க ச ட த ந ன ப\n", + "ம ய ர ல ள வ ா ி ு ே ை ನ ರ ಾ ක ය ර ල ව ා ก ง ต ท น พ ม ย ร ล ว ส อ า เ ་ ། ག ང ད\n", + "ན པ བ མ འ ར ལ ས မ ა ბ გ დ ე ვ თ ი კ ლ მ ნ ო რ ს ტ უ ᄀ ᄂ ᄃ ᄅ ᄆ ᄇ ᄉ ᄊ ᄋ ᄌ ᄎ ᄏ ᄐ ᄑ\n", + "ᄒ ᅡ ᅢ ᅥ ᅦ ᅧ ᅩ ᅪ ᅭ ᅮ ᅯ ᅲ ᅳ ᅴ ᅵ ᆨ ᆫ ᆯ ᆷ ᆸ ᆼ ᴬ ᴮ ᴰ ᴵ ᴺ ᵀ ᵃ ᵇ ᵈ ᵉ ᵍ ᵏ ᵐ ᵒ ᵖ ᵗ ᵘ ᵣ ᵤ\n", + "ᵥ ᶜ ᶠ ‐ ‑ ‒ – — ― ‖ ‘ ’ ‚ “ ” „ † ‡ • … ‰ ′ ″ › ‿ ⁄ ⁰ ⁱ ⁴ ⁵ ⁶ ⁷ ⁸ ⁹ ⁻ ⁿ ₅ ₆ ₇ ₈\n", + "₉ ₊ ₍ ₎ ₐ ₑ ₒ ₓ ₕ ₖ ₗ ₘ ₚ ₛ ₜ ₤ ₩ € ₱ ₹ ℓ № ℝ ™ ⅓ ⅔ ← ↑ → ↓ ↔ ↦ ⇄ ⇌ ⇒ ∂ ∅ ∆ ∇ ∈\n", + "∗ ∘ √ ∞ ∧ ∨ ∩ ∪ ≈ ≡ ≤ ≥ ⊂ ⊆ ⊕ ⊗ ⋅ ─ │ ■ ▪ ● ★ ☆ ☉ ♠ ♣ ♥ ♦ ♯ ⟨ ⟩ ⱼ ⺩ ⺼ ⽥ 、 。 〈 〉\n", + "《 》 「 」 『 』 〜 あ い う え お か き く け こ さ し す せ そ た ち っ つ て と な に ぬ ね の は ひ ふ へ ほ ま み\n", + "む め も や ゆ よ ら り る れ ろ を ん ァ ア ィ イ ウ ェ エ オ カ キ ク ケ コ サ シ ス セ タ チ ッ ツ テ ト ナ ニ ノ ハ\n", + "ヒ フ ヘ ホ マ ミ ム メ モ ャ ュ ョ ラ リ ル レ ロ ワ ン ・ ー 一 三 上 下 不 世 中 主 久 之 也 事 二 五 井 京 人 亻 仁\n", + "介 代 仮 伊 会 佐 侍 保 信 健 元 光 八 公 内 出 分 前 劉 力 加 勝 北 区 十 千 南 博 原 口 古 史 司 合 吉 同 名 和 囗 四\n", + "国 國 土 地 坂 城 堂 場 士 夏 外 大 天 太 夫 奈 女 子 学 宀 宇 安 宗 定 宣 宮 家 宿 寺 將 小 尚 山 岡 島 崎 川 州 巿 帝\n", + "平 年 幸 广 弘 張 彳 後 御 德 心 忄 志 忠 愛 成 我 戦 戸 手 扌 政 文 新 方 日 明 星 春 昭 智 曲 書 月 有 朝 木 本 李 村\n", + "東 松 林 森 楊 樹 橋 歌 止 正 武 比 氏 民 水 氵 氷 永 江 沢 河 治 法 海 清 漢 瀬 火 版 犬 王 生 田 男 疒 発 白 的 皇 目\n", + "相 省 真 石 示 社 神 福 禾 秀 秋 空 立 章 竹 糹 美 義 耳 良 艹 花 英 華 葉 藤 行 街 西 見 訁 語 谷 貝 貴 車 軍 辶 道 郎\n", + "郡 部 都 里 野 金 鈴 镇 長 門 間 阝 阿 陳 陽 雄 青 面 風 食 香 馬 高 龍 龸 fi fl ！ （ ） ， － ． ／ ： ？ ～\n" + ] + } + ], + "source": [ + "print('Number of single character tokens with hashes:', len(one_chars_hashes), '\\n')\n", + "\n", + "# Print all of the single characters, 40 per row.\n", + "\n", + "# Strip the hash marks, since they just clutter the display.\n", + "tokens = [token.replace('##', '') for token in one_chars_hashes]\n", + "\n", + "# For every batch of 40 tokens...\n", + "for i in range(0, len(tokens), 40):\n", + " \n", + " # Limit the end index so we don't go past the end of the list.\n", + " end = min(i + 40, len(tokens) + 1)\n", + " \n", + " # Print out the tokens, separated by a space.\n", + " print(' '.join(tokens[i:end]))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "u-TFrVP9005S", + "outputId": "7e234716-83f8-424a-e0a1-b23d426ef7ca", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Are the two sets identical? True\n" + ] + } + ], + "source": [ + "print('Are the two sets identical?', set(one_chars) == set(tokens))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "uPnW2RetYc97" + }, + "source": [ + "### Subwords vs. Whole-words\n", + "Let's gather some statistics on the vocabulary.\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 389 + }, + "id": "LjvrWcNTdkAW", + "outputId": "111debcc-8600-415e-9a38-9e73da02db8f", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Maximum token length: 18\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoUAAAFjCAYAAABL3HHWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdeViUVf8G8HuAAUSQRQdcUDGLRRQQXInXcklxRwW3QFFDM3PNBV+1en9qlCKpoZiklGhqJoiGG6ZvZoG7kjmZoLmkyLgCIswAz+8PL57XEbAZ5WHUuT/X1XU155w53zMT4e1znkUmCIIAIiIiIjJqJoZeABEREREZHkMhERERETEUEhERERFDIRERERGBoZCIiIiIwFBIRERERGAoJKKXSElJCdzc3DBnzhxDL6VS7733Htq0aWPoZbzUlEol3Nzc8PXXXxt6KUQvHIZCIqrUpEmT4ObmBqVSWeUYQRDQpUsXtGnTBkVFRTW4Omls2bIFbm5uOv0THh5u6OVWqzZt2mDYsGGGXoZOVCoVvvjiC5w6dcrQSyF6qZgZegFE9HwKDg7Gnj17sHXrVsydO7fSMRkZGfj7778xZMgQWFpa1vAKq1/79u2xaNEirbaVK1fi8uXL+PTTT7XaHR0da3Jp9IibN28iNjYWNjY28PHxMfRyiF4aDIVEVKmAgAA0aNAAO3bswMyZM2Fubl5hTFJSEoCHAfJl0KRJEzRp0kSrbdOmTbhy5Qr69+9voFUREdUMbh8TUaVMTEwwYMAA3L17F/v376/QX1BQgL1798LV1RVeXl5afZs2bUJQUBC8vLzQpk0bjBkzBidOnKi0Tnp6Ot555x20b98erVq1QteuXTF37lzcvXtXHLN+/XqEh4cjICAALVu2REBAAGbOnIlr165Vuf6ff/4ZISEh8PLyQkBAAD755BM8ePDgKb+NJ8vMzMTYsWPRtm1btGrVCn379sX69euhy1NECwoKMHLkSHh7e+PHH38U20tKSpCQkIB+/frBy8sLvr6+GDNmDE6ePKn1/kfPodu5cyf69++PVq1aoVOnTli+fDnKysqq/fPu378fYWFh8PX1hZeXFwYMGIBt27ZVGNemTRu89957UCqVCA8PR+vWrdG2bVtMnz4d9+7dqzA+MzMToaGh8Pb2RocOHfDhhx/i77//hpubGxYuXAgA2LdvH4KCggAAUVFR4nZ+ZaFdl+/j999/x/jx47V+tkaPHo2MjIzq+KqIXig8UkhEVRo4cCDi4uKQlJSEwMBArb7U1FQUFRVh0KBBWu2ffvopEhIS4OPjg2nTpqGgoACbNm3CiBEjsGrVKgQEBIhjN2zYgPnz56NBgwYYOnQoGjZsiGvXrmH//v3Izc2FnZ0dAGDNmjXw9fWFv78/7OzscO7cOWzduhUZGRnYsWMHbG1ttdbw22+/YefOnRg8eDCCgoKQkZGBb775BllZWVizZg1kMlm1fUeHDx/GmDFjYG1tjdDQUNjZ2SEtLQ3z589HdnY2Pvrooyrfm5ubi4iICOTk5ODrr79G69atAQBlZWV4//33cfDgQfTu3RtDhgzBgwcPkJSUhLCwMHz11Vfo0KGD1lw7d+7E9evXMWTIENjb22P37t1YsWIF6tati7fffrvaPu9XX32FxYsXo3379pg4cSLMzc1x4MABzJo1C9evX8f48eO1xl+6dAmjRo1Cnz59EBgYiNOnTyMpKQkajQbLli0TxymVSoSFhcHCwgKjRo1C3bp1kZaWhokTJ2rN17JlS0ydOhWff/45+vXrJ/48lf+s6PN95OTkYOTIkahduzaGDRsGJycn3L59G6dPn8aZM2cqfMdELz2BiOgJRowYIXh4eAg3btzQah88eLDg6ekp3Lp1S2w7f/684OrqKrz99tuCWq0W269fvy60bt1a6Nq1q1BaWioIgiBcvXpV8PT0FPr06SPk5+dXqFs+ThAE4f79+xX6Dx48KLi6ugpr164V2zQajeDq6iq4uroK+/fv1xr/8ccfC66ursKuXbv0+vxDhw4VPDw8quzv1auX0KpVK+HSpUtiW0lJiTB69GjB1dVVyMzMFNvHjx8v+Pn5CYIgCFlZWULnzp2Fzp07C1lZWVpzfv/994Krq6uQmpqq1f7gwQOhR48eQr9+/cS2s2fPCq6urkLbtm0FlUoltms0GqFz585CYGCgTp/Tz89PGDp06BPHXLx4UfDw8BA++uijCn0zZ84UPD09tdbg5+cnuLq6Cj///LPW2GnTpglubm5aP1OjRo0S3N3dhT/++ENsKy0tFb/HBQsWVPjMCQkJFdahz/eRnJxc6fqIjBW3j4noiYKDg1FaWqq1PZidnY1Tp06hS5cucHBwENv37dsHAIiIiIBcLhfb69evjwEDBuDKlSs4d+4cAGDXrl3QaDR4//33YW1tXaGuicn/fj1ZWVkBeHgELT8/H7dv34anpyesrKxw+vTpCu997bXX0LlzZ622sWPHAgDS0tL0/g6qcvHiRWRlZaF3795a5yKamppi3LhxAP73nTzqxIkTGD58OKytrbFx40Y0b95cq3/79u1QKBTo0KEDbt++Lf5TWFiIN954A3/88Qdu3bql9Z7evXujXr164mszMzP4+fnhypUr1baFnJqaitLSUgwcOFBrXbdv30aXLl2g0Whw5MgRrfe8+uqrWkeHAaBDhw4QBAGXL18GABQWFiIjIwP+/v5wc3MTx5mYmGDUqFFPtVZdvo/yn7v//ve/KCwsfKo6RC8Tbh8T0RN1794dderUQVJSkhistm7dCgAVto6vXr0K4GEQeFx525UrV+Dh4YFLly4BAFq0aPGPa/jll18QFxeH06dPQ61Wa/Xl5eVVGP/KK69UaGvQoAGsrKzENVaH8rlee+21Cn3ln/fxeg8ePMCoUaPg5OSEb7/9ttJAnJ2dDZVKhY4dO1ZZ++bNm6hbt674unHjxhXG2NnZQaPR4P79+7CxsdHtQz3BhQsXAAAhISFVjnk8rFa2Lnt7ewAQzxu9fv06SktL0axZswpjK2vThS7fR+fOndGtWzckJiZi8+bN8PLywuuvv47evXujadOmT1WX6EXGUEhET2RhYYE+ffrg22+/xYkTJ+Dt7Y3t27ejfv36+Ne//iV5/ZMnTyIiIgIuLi6YPn06nJ2dUatWLQDA5MmTJbmQQkoWFhbo3LkzfvjhB3z33XcYPXp0peOcnZ0xf/78KudxdnbWev3okdXHCTpc8KKL8nliY2NRu3btSsc8HuKetC4p6fJ9mJqaYsWKFTh79ix+/vlnHD16FKtWrcLKlSuxYMEC8YIWImPBUEhE/yg4OBjffvstkpKScO/ePahUKrz77rsV/uAtPzqTlZWFRo0aafVlZ2drjXFxcQHw8AKDyo7qlPvhhx9QWlqKr776Cg0bNhTbCwoKkJ+fX+l7yo9oPer69esoLCysEKaeRfm6z58/X6Gv/PNWFt4WL14MMzMzfPbZZ9BoNOJWc7mmTZvi7NmzaNu2rdY2vKGVHz2rV6+eeFFMdWjQoAFMTU1x8eLFCn2VtVXnhULAw6PVLVq0wLhx43Dr1i0MHDgQS5YsYSgko8NzConoH3l6esLDwwM7d+7Ehg0bIJPJKr03YdeuXQE8vEK1pKREbL9x4waSk5PRuHFj8ZyxwMBAyOVyfPHFFygoKKgwV/nRnKqO+MTFxVV5BOz8+fM4cOCAVlt8fDwAoFu3bv/0cXXm4uKCV199FTt37sSVK1fE9rKyMqxevRoA8NZbb1V4n4mJCaKiojBo0CDExMQgNjZWqz8oKAiFhYVYsWJFpXVv3rxZbZ9BH3379oWpqSmWLl1aYRsfeLgd/Oh/d11ZWVmhffv2+PXXX8VzToGH32NCQkKl4wFUelsbfdy5c6fCz1DdunXh5OT0zHMTvYh4pJCIdBIcHIz58+fj559/Rrt27So9uvfqq69i1KhRSEhIQGhoKHr27Cnekqa4uBgfffSRGPIaNWqEWbNmYcGCBejXrx/69++Phg0bIicnB/v27cPixYvh6uqK7t27IzExEWPGjMGQIUNgamqKQ4cOITs7u8KtaMq5ublh2rRpGDJkCBo3boyMjAzs3bsXHTt2rHBrnWf14YcfimsbNmwYbG1tkZaWhiNHjmD48OFo2bJlpe8zMTHBwoULxWBcWlqKyZMnA3h4rubPP/+MuLg4nDhxAgEBAbCzs8P169dx9OhRFBQUVHpfwGeVk5ODlStXVto3bNgwvPLKK5g1axaioqLQt29f9OnTB/Xr18fNmzehVCpx4MAB/PLLL6hTp47etadPn47hw4cjLCwMb7/9NhwcHLBv3z7xLwyPHh10dnZGvXr1sHXrVtSrVw92dnaoU6eO3qczrF+/Htu3b0fXrl3RuHFjyGQy/Prrrzh9+jSGDBmi92cgetExFBKRTvr27YtFixahuLi4wgUmj4qMjISLiws2btyI6OhoyOVy+Pj4YMKECfDz89MaGxYWhqZNm2Lt2rVYt24dNBoNHB0d4e/vDycnJwBA27ZtsWzZMqxatQqff/45atWqhddffx3r16/H4MGDK11Dq1atMGPGDCxbtgzffvstbGxsEBYWhmnTplX71mP79u2xYcMGxMbG4uuvv4ZarYaLiwvmzp2L0NDQJ75XJpPhP//5D+RyOVauXAmNRoPp06fDxMQES5cuxZYtW5CUlIS4uDiUlpZCoVCgVatWT31F7j+5du2a1r0DH9WjRw/Y29tj5MiRcHV1xddff43169fj/v37cHBwQPPmzTFjxgzxKJ6+PD098fXXXyM6Ohpr1qyBlZUV3nrrLYwYMQJ9+vTReoyiiYkJYmJiEB0djc8++wzFxcVwd3fXOxT+61//wl9//YW0tDTcvHkTcrkcTZo0wbx58zB06NCn+hxELzKZUF1nIBMREVWz9PR0hIeH4+OPP8awYcMMvRyilxrPKSQioudCUVGR1uvS0lLxCTT+/v4GWhWR8eD2MRERPRfeeustdO/eHa+++iry8/Oxd+9e/PbbbxgyZAjvG0hUA7h9TEREz4WPP/4Y6enpuHHjBkpLS9G0aVMMGDAAo0aNMtj9DomMCUMhEREREfGcQiIiIiJiKCQiIiIi8EKTanPnzn2UlXEnnoiIiJ5fJiYy2NtX/uxyhsJqUlYmMBQSERHRC4vbx0RERETEUEhEREREDIVEREREBIZCIiIiIgJDIRERERGBoZCIiIiIwFBIRERERGAoJCIiIiIwFBIRERERDPhEk8OHD2PEiBGV9u3cuRPNmzcXX584cQKLFy/G2bNnYW1tjZ49e+KDDz5ArVq1tN6nVquxbNkypKSkIC8vD+7u7pg6dSo6duxYoYaucxI9r2zt5DCXW0paQ60pwr27GklrEBHR88Hgj7kbOXIkPD09tdqcnJzEf1cqlQgPD8err76KyMhI5OTkYO3atbh69SpWrVql9b7IyEjs3bsXI0aMQNOmTZGcnIyIiAgkJiaidevWTzUn0fPKXG6JJRt7SFrjg2F7ADAUEhEZA4OHwnbt2qFbt25V9sfExMDOzg6JiYmoXfvhA5ydnZ0xd+5cpKeni0cBMzMzkZqaitmzZyM8PBwAEBQUhD59+iA6OhobNmzQe04iIiIiY/FcnFNYUFCAkpKSStt//fVXBAUFieENAPr37w8rKyvs2rVLbNu9ezfkcjlCQkLENgsLCwQHB+P48ePIzc3Ve04iIiIiY2HwUDhjxgz4+fnB29sbo0ePxrlz58S+c+fOoaSkBC1bttR6j7m5OTw8PKBUKsU2pVKJZs2aaQU9APDy8oIgCOJYfeYkIiIiMhYG2z6Wy+Xo0aMHOnXqBHt7e5w7dw5r167F8OHD8f3336NZs2ZQqVQAAIVCUeH9CoUCp06dEl+rVCqtcxEfHQdAPFKoz5z6qFvX+qneR/S8UyhsDL0EIiKqAQYLhb6+vvD19RVfd+3aFV26dMGgQYMQGxuLJUuWoKioCMDDo3iPs7CwEPsBoKioCHK5vNJxAFBcXCyO03VOfdy6VYCyMuGp3kv0NGoqrKlU+TVSh4iIpGdiIqvyQJbBt48f5e7ujo4dOyIjIwMAYGn58HYbarW6wtji4mKxv3ysRlPxKsnyMFgeDvWZk4iIiMhYPFehEAAaNGiAe/fuAfjfFm/5lu+jVCoVHB0dxdcKhULcIn58HABxrD5zEhERERmL5y4UXrlyBfb29gAAV1dXmJmZ4cyZM1pj1Go1lEolPDw8xDZ3d3dcvHgR9+/f1xp7+vRpsV/fOYmIiIiMhcFC4e3btyu0HTt2DIcPH0ZAQAAAwMbGBh07dkRKSopW2EtJSUFhYSECAwPFtsDAQGg0GmzZskVsU6vVSEpKgq+vr3gRij5zEhERERkLg11oMmXKFNSqVQutW7eGvb09zp8/j82bN8Pe3h4TJ04Ux02dOhVDhw5FWFgYQkJCkJOTg4SEBHTq1An+/v7iOG9vbwQGBiI6OhoqlQpNmjRBcnIyrl27hqioKK3aus5JREREZCxkgiAY5JLZdevWYceOHbh8+TIKCgrg4OCAgIAATJw4EQ0bNtQae+zYMURHR4vPKe7VqxemTZsGKysrrXHFxcVYunQpduzYgXv37sHNzQ3Tpk2rNOjpOqeuePUx1TSFwqZGHnPHq4+JiF4eT7r62GCh8GXDUEg1jaGQiIj09cLckoaIiIiIDIOhkIiIiIgYComIiIiIoZCIiIiIwFBIRERERGAoJCIiIiIwFBIRERERGAqJiIiICAyFRERERASGQiIiIiICQyERERERgaGQiIiIiMBQSERERERgKCQiIiIiMBQSERERERgKiYiIiAgMhUREREQEhkIiIiIiAkMhEREREQEwM/QCiF5kdnZyyOWWktbQaIpw965G0hpEREQMhUTPQC63ROLXPSStERa+BwBDIRERSYvbx0RERETEUEhEREREDIVEREREBIZCIiIiIgJDIRERERGBoZCIiIiIwFBIRERERGAoJCIiIiIwFBIRERER+EQTInoKdezMYSG3kLRGsaYYeXfVktYgIqL/YSgkIr1ZyC0wKjlQ0hoJA3YDYCgkIqop3D4mIiIiIoZCIiIiImIoJCIiIiIwFBIRERERGAqJiIiICM9ZKIyPj4ebmxv69+9foe/EiRMYNmwYvL298frrr2PBggV48OBBhXFqtRqLFy9GQEAAvLy8MHjwYKSnp1daT9c5iYiIiF52z00oVKlUiIuLg5WVVYU+pVKJ8PBwFBcXIzIyEsHBwdi8eTOmTp1aYWxkZCS++eYb9OvXD3PmzIGJiQkiIiJw8uTJp56TiIiI6GX33NyncMmSJWjZsiUEQUBeXp5WX0xMDOzs7JCYmIjatWsDAJydnTF37lykp6ejY8eOAIDMzEykpqZi9uzZCA8PBwAEBQWhT58+iI6OxoYNG/Sek4iIiMgYPBdHCjMzM7F9+3bMnj27Ql9BQQF+/fVXBAUFieENAPr37w8rKyvs2rVLbNu9ezfkcjlCQkLENgsLCwQHB+P48ePIzc3Ve04iIiIiY2DwUCgIAubPn4+goCB4eHhU6D937hxKSkrQsmVLrXZzc3N4eHhAqVSKbUqlEs2aNdMKegDg5eUFQRDEsfrMSURERGQMDL59vG3bNmRlZWHFihWV9qtUKgCAQqGo0KdQKHDq1CmtsU5OTpWOAyAeKdRnTl3VrWut93uIdKVQ2LA2ERFJyqChsKCgAEuWLMHYsWPh6OhY6ZiioiIAD4/iPc7CwkLsLx8rl8srHQcAxcXFes+pq1u3ClBWJuj9Pnqx1VRoUanyWZuIiJ6ZiYmsygNZBt0+jouLg1wux6hRo6ocY2lpCeDhrWYeV1xcLPaXj9VoNJWOA/4XDvWZk4iIiMgYGOxIYW5uLr755htMnjwZN2/eFNuLi4uh0Whw9epV2NjYiFu85Vu+j1KpVFpHGBUKhbhF/Pg4AOJYfeYkIiIiMgYGO1J469YtaDQaREdHo2vXruI/p0+fRnZ2Nrp27Yr4+Hi4urrCzMwMZ86c0Xq/Wq2GUqnUujjF3d0dFy9exP3797XGnj59WuwHoNecRERERMbAYEcKnZ2dK724ZOnSpSgsLMS///1vuLi4wMbGBh07dkRKSgrGjRsnXlmckpKCwsJCBAYGiu8NDAzE2rVrsWXLFvE+hWq1GklJSfD19RUvQtFnTiIiIiJjYLBQaGNjg27dulVo/+abb2BqaqrVN3XqVAwdOhRhYWEICQlBTk4OEhIS0KlTJ/j7+4vjvL29ERgYiOjoaKhUKjRp0gTJycm4du0aoqKitOroOicRERGRMTD4fQp14enpiYSEBJibmyMqKgpbtmzB4MGDsWzZsgpjFy1ahLCwMKSkpGDBggUoKSnB6tWr4efn99RzEhEREb3sDH6fwsclJiZW2t6mTRts2rTpH99vYWGBWbNmYdasWf84Vtc5iYiIiF52L8SRQiIiIiKSFkMhERERETEUEhERERFDIRERERGBoZCIiIiIwFBIRERERKiGUHjhwgWcPHkSpaWl1bEeIiIiIjIAnUPh+vXrMXnyZK22efPmoXfv3hg+fDj69++PO3fuVPsCiYiIiEh6OofCbdu2wdbWVnx97NgxbNmyBV26dMHYsWNx+fJlrFmzRpJFEhEREZG0dH6iydWrV9GvXz/x9d69e1G3bl0sX74cpqamKCwsxL59+zB9+nRJFkpERERE0tH5SGFhYSGsrKzE1xkZGejYsSNMTU0BAG5ubrhx40b1r5CIiIiIJKdzKFQoFMjOzgYA3LhxA+fPn0fbtm3F/vz8fJiZPXePUiYiIiIiHeic4jp16oRvv/0WpqamOHHiBExNTfHmm2+K/VlZWWjQoIEUayQiIiIiiekcCidOnIgzZ87gq6++gqmpKWbMmAEnJycAgFqtRlpamtY5h0RERET04tA5FDo4OGDLli3Izc2FlZUVrK2txb6SkhLExcWhadOmkiySiIiIiKSl90mAjo6OFdqsrKzg5+dXLQsiIiIiopqndyi8ceMGLl++jLt370IQhAr93bt3r5aFEREREVHN0TkU3r17Fx9++CHS0tIAQCsQymQyCIIAmUwGpVJZ/askIiIiIknpHArnz5+PvXv3IigoCB06dICdnZ2U6yIiIiKiGqRzKDx48CCCg4OxYMECKddDRERERAag882rS0pK4OXlJeVaiIiIiMhAdA6Fvr6+OH/+vJRrISIiIiID0TkUzpw5E6mpqfj555+lXA8RERERGYDO5xR+/vnnsLe3x9ixY+Hi4oLGjRvDxEQ7U8pkMsTFxVX7IomIiIhIWjqHwpMnT0Imk8HW1hZ37tzBnTt3KoyRyWTVujgiIiIiqhk6h8LDhw9LuQ4iIiIiMiCdzykkIiIiopeX3o+5u3nzJg4fPoybN2+iR48eqF+/PkpKSlBQUAAbGxuYmppKsU4iIiIikpBeoXDFihVYtWoVNBoNZDIZXF1dUb9+fdy/fx9vvPEGZsyYgdDQUKnWSkQEGzsLWMrNJa1RpFEj/26xpDWIiJ43OofCpKQkfPHFFxg0aBA6d+6M999/X+yztbVF586dsX//foZCIpKUpdwcvbbNk7TGzqD5yAdDIREZF53PKUxMTMSbb76JhQsXws/Pr0K/h4cHsrOzq3VxRERERFQzdA6F2dnZ6NSpU5X9Dg4OuH37drUsioiIiIhqls6h0NzcHGq1usr+nJwcWFtbV8uiiIiIiKhm6RwKPT09sX///kr71Go1duzYAR8fn2pbGBERERHVHJ1D4ahRo3DkyBF89NFHuHLlCgCgoKAAx48fxzvvvIOrV69i9OjRki2UiIiIiKSj89XHb775JiIjIxEdHY3vvvsOADBp0iQAgImJCebMmYO2bdtKs0oiIiIikpRe9ykMDw9Ht27dkJqaigsXLkAQBDRt2hR9+vRB06ZN9Sr822+/YdWqVTh79ixu3boFGxsbuLu7Y8KECfD19dUae+LECSxevBhnz56FtbU1evbsiQ8++AC1atXSGqdWq7Fs2TKkpKQgLy8P7u7umDp1Kjp27Fihvq5zEhERERkDvZ9o4uzsjHHjxlXa9+DBA51D1ZUrV1BaWoqQkBAoFArk5+djx44dCA0NRXx8PF5//XUAgFKpRHh4OF599VVERkYiJycHa9euxdWrV7Fq1SqtOSMjI7F3716MGDECTZs2RXJyMiIiIpCYmIjWrVuL4/SZk4iIiMgY6BwKY2JiMG3atCr7i4qKMG7cOKxbt06n+Xr16oVevXpptQ0bNgzdunXDunXrxFAYExMDOzs7JCYmonbt2gAeBtO5c+ciPT1dPAqYmZmJ1NRUzJ49G+Hh4QCAoKAg9OnTB9HR0diwYYPWZ9FlTiIiIiJjofOFJqtXr9YKVo9Sq9UYP348Tp48+UyLqVWrFhwcHJCXlwfg4YUsv/76K4KCgsTwBgD9+/eHlZUVdu3aJbbt3r0bcrkcISEhYpuFhQWCg4Nx/Phx5Obm6j0nERERkbHQORSOHTsWn3zyCdLS0rTaywPhkSNHsGTJEr0XUFBQgNu3b+PChQuIiYnBn3/+KR6pO3fuHEpKStCyZUut95ibm8PDwwNKpVJsUyqVaNasmVbQAwAvLy8IgiCO1WdOIiIiImOh8/bxtGnTcOPGDUyfPh1r1qxBmzZtoNFoMGnSJGRkZGDx4sXo3r273gv497//jT179gAA5HI5hg4dinfffRcAoFKpAAAKhaLC+xQKBU6dOiW+VqlUcHJyqnQcAPFIoT5zEhERERkLvS40WbhwIW7evIn33nsP33zzDWJjY3Hw4EF88sknFc4P1NWECRMwZMgQ5OTkICUlBWq1GhqNBubm5igqKgLw8Cje4ywsLMR+4OE5jXK5vNJxAFBcXCyO03VOfdSty6e5GEppiRqmZhX/e75oNZ5EobBhbSOqTURkCHqFQjMzM3zxxRcIDQ1FSEgIysrKMH/+fAQFBT31Atzc3ODm5gYA6NevHwYNGoTZs2dj+fLlsLS0BIBKH69XXFws9gOApaUlNBpNpeOA/4VDfebUx61bBSgrE57qvfRsFAob7FnzdH8p0VWPMTuhUuVXWrsmsPbzUZuI6EVnYiKr8kBWlaHw999/r3LCyb08oI8AACAASURBVJMnY9q0aejduzfc3d21xnp6ej71QuVyObp27Yq4uDgUFRWJW7zlW76PUqlUcHR0FF8rFApxi/jxcQDEsfrMSURERGQsqgyFgwYNgkwmq/KNgiBgy5Yt2LJli1b7s16oUVRUBEEQcP/+fbi6usLMzAxnzpzROl9RrVZDqVSib9++Ypu7uzsSExNx//59rYtNTp8+LfYD0GtOIiIiImNRZSicO3fuE0Phs7p9+zYcHBy02goKCrBnzx40aNAAdevWBQB07NgRKSkpGDdunBj2UlJSUFhYiMDAQPG9gYGBWLt2LbZs2SLep1CtViMpKQm+vr7iRSg2NjY6z0lERERkLKoMhaGhoZIWnjJlCiwsLNC6dWsoFApcv34dSUlJyMnJQUxMjDhu6tSpGDp0KMLCwhASEoKcnBwkJCSgU6dO8Pf3F8d5e3sjMDAQ0dHRUKlUaNKkCZKTk3Ht2jVERUVp1dZ1TiIiIiJjofdj7qpLv379kJKSgsTEROTl5cHGxgY+Pj5YtGgR2rVrJ47z9PREQkICoqOjERUVBWtrawwePLjSp6ssWrQIS5cuRUpKCu7duwc3NzesXr0afn5+WuP0mZOIiIjIGOgVCtVqNdatW4d9+/bhypUrAIDGjRvjrbfeQlhYWKW3ealKcHAwgoODdRrbpk0bbNq06R/HWVhYYNasWZg1a1a1zUlERERkDHQOhQUFBQgLC4NSqYSVlRWaNGkCADh//jxOnTqF1NRUrFu3DtbWvF8fERER0YtG58fcffHFF1Aqlfjggw+QkZGBbdu2Ydu2bUhPT8f06dOhVCoRGxsr5VqJiIiISCI6h8K0tDQMGDAAERERWtvE5ubmeOeddxAUFIS9e/dKskgiIiIikpbOoTA3Nxfe3t5V9nt5eVV6Q2giIiIiev7pHAodHBzw559/Vtl//vx52NnZVcuiiIiIiKhm6RwK33jjDXz33XfYvn17hb4dO3bgu+++Q+fOnat1cURERERUM3S++njy5Mk4dOgQZs2ahWXLlqF58+YAgOzsbFy7dg0NGjTApEmTJFsoEREREUlH5yOF9erVQ1JSEt5++22UlZXh4MGDOHjwIARBQGhoKLZu3Yp69epJuVYiIiIikoheN6+2t7fH3LlzMXfuXJSWlgIATE1NJVkYEREREdWcJx4p9PDwwI4dOyrtMzU1ZSAkIiIiekk8MRQKglBT6yAiIiIiA9L5nEIiIiIienkxFBIRERHRP19ocuHCBRw9elTnCdu2bftMCyIiIiKimvePoXDVqlVYtWqVzhMqlcpnWhARERER1bx/DIXdunWDm5tbTayFiIiIiAzkH0Nh9+7d0bdv35pYCxEREREZCC80ISIiIiKGQiIiIiJiKCQiIiIi/MM5hT/++CMcHBxqai1EREREZCBPDIWNGjWqqXUQERERkQH949XHRET0kI2dJSzlcklrFGk0yL9bJGkNIqLKMBQSEenIUi5H76SVktZIHfge8sFQSEQ1j6GQqoW9rTnMzC0krVGiLsade2pJaxARERmrKkNhbGwsunfvDldXVwDAtWvX4ODgAEtLyxpbHL04zMwtkBnXT9IaXuO3A2AoJCIikkKVt6SJjY3FuXPnxNddu3ZFWlpajSyKiIiIiGpWlaGwTp06yMvLE18LglAjCyIiIiKimlfl9rGHhwfWrFmDkpIS2NraAgCOHTuG0tLSJ04YFBRUvSskIiIiIslVGQpnz56N999/H1FRUQAAmUyGzZs3Y/PmzVVOJpPJGAqJiIiIXkBVhkJ3d3fs2bMHV65cgUqlQlhYGN599134+/vX5PqIiIiIqAY88ZY0pqamcHFxgYuLC9q2bYv27dujXbt2NbU2IiIiIqohOt+nMDExUcp1EBEREZEB6XXz6rKyMiQnJyMtLQ1Xr14FADg7O6N79+4ICgqCiUmVFzMTERER0XNM51BYVFSEiIgIHDt2DDKZDAqFAgBw8OBB/PTTT9i2bRvi4+NhYSHtUy2IiIiIqPrpfGgvLi4OR48exahRo5Ceno6ffvoJP/30EzIyMjB69GgcOXIEcXFxUq6ViIiIiCSicyjcuXMnevbsiZkzZ4r3LQQe3uR6xowZ6NmzJ1JTUyVZJBERERFJS+dQmJOT88Qrj9u2bYucnBydC2dmZuI///kPevXqBR8fH7z55puYOnUqLl26VGHsiRMnMGzYMHh7e+P111/HggUL8ODBgwrj1Go1Fi9ejICAAHh5eWHw4MFIT0+vtL6ucxIREREZA51DYZ06dXD58uUq+y9fvow6deroXPirr75CWloa/P39MWfOHAwePBhHjhxBUFAQsrOzxXFKpRLh4eEoLi5GZGQkgoODsXnzZkydOrXCnJGRkfjmm2/Qr18/zJkzByYmJoiIiMDJkye1xukzJxEREZEx0PlCE39/f2zYsAH+/v7417/+pdV36NAhbNy4EYGBgToXDg8PR3R0NMzNzcW2Xr16oW/fvoiPj8enn34KAIiJiYGdnR0SExNRu3ZtAA+veJ47dy7S09PRsWNHAA+PPKampmL27NkIDw8H8PCRe3369EF0dDQ2bNgg1tF1TiIiIiJjofORwilTpqB27doYO3YsBg4ciFmzZmHWrFkYOHAgIiIiULt2bUyaNEnnwr6+vlqBEABcXFzw2muviUcKCwoK8OuvvyIoKEgMbwDQv39/WFlZYdeuXWLb7t27IZfLERISIrZZWFggODgYx48fR25urt5zEhERERkLnUNho0aNsHXrVvTq1Qt//fUXUlJSkJKSgkuXLqF37974/vvv0ahRo2dajCAIuHnzJuzt7QEA586dQ0lJCVq2bKk1ztzcHB4eHlAqlWKbUqlEs2bNtIIeAHh5eUEQBHGsPnMSERERGQu9bl7dsGFDLFmyBIIg4Pbt2wAABwcHyGSyalnM9u3bcePGDfHcPpVKBQDiPREfpVAocOrUKfG1SqWCk5NTpeMAiEcK9ZmTiIiIyFjoFQrLyWQy1K1bt1oXkp2djf/7v/+Dn58f+vfvD+DhDbMBVNhmBh5uDZf3l4+Vy+WVjgOA4uJivefUR9261k/1PtKPQmHD2qzN2kREEniqUFjdVCoVxo0bB1tbWyxbtkx8XJ6lpSWAh7eaeVxxcbHYXz5Wo9FUOg74XzjUZ0593LpVgLIy4ane+zKoqT/EVKp81mZto6xNRFQdTExkVR7IMngozM/PR0REBPLz87Fx40atbd3yfy/f8n2USqWCo6Oj1tjyLeLHxwEQx+ozJxEREZGx0PlCEykUFxfj3XffxV9//YUvv/wSr7zyila/q6srzMzMcObMGa12tVoNpVIJDw8Psc3d3R0XL17E/fv3tcaePn1a7Nd3TiIiIiJjYbBQWFpaiilTpuDUqVNYtmwZfHx8KoyxsbFBx44dkZKSohX2UlJSUFhYqHVfxMDAQGg0GmzZskVsU6vVSEpKgq+vr3gRij5zEhERERkLg20ff/rpp9i/fz86d+6Mu3fvIiUlReyrXbs2unXrBgCYOnUqhg4dirCwMISEhCAnJwcJCQno1KkT/P39xfd4e3sjMDAQ0dHRUKlUaNKkCZKTk3Ht2jVERUVp1dZ1TiIiIiJjYbBQ+McffwAADhw4gAMHDmj1NWrUSAyFnp6eSEhIQHR0NKKiomBtbY3Bgwdj2rRpFeZctGgRli5dipSUFNy7dw9ubm5YvXo1/Pz8tMbpMycRERGRMdA5FBYUFOC9995DZGQkWrRo8cyFExMTdR7bpk0bbNq06R/HWVhYiE9aqa45iYiIiIyBzucUajQaHDlyBPfu3QMAFBYWYvbs2eIj6YiIiIjoxfXEUDhp0iR8/fXXOH36dIX7+hUXF2Pbtm2V3gaGiIiIiF4sT9w+fvDgAVasWIH8/HyYmZlBJpNh165dsLKygrOzMwTBeG/WTERERPQyeWIojI+PhyAIOHfuHH755RcsXrwYO3bswHfffQcrKyvIZDL897//ha2tLTw8PKrtGchERPT8sLGrBUu5tNclFmlKkH/3gaQ1iOjJ/vH/cplMBnd3dzg5OWHx4sVYuXIlHBwcsH//fixbtgwbNmzAunXrYG1tDV9fX3z55Zc1sW4iIqohlnIz9Pv+B0lrbA/uAz7cj8iwnhgKx4wZAz8/P/j5+aFx48YAHoZENzc3KBQKLFu2DF9++SXq1KmDo0eP4tixYzWyaCIiIiKqXk8Mhebm5khMTMTy5cthamoKmUyG5ORkABAfSWdqaopWrVqhVatWGD16tPQrJiIiIqJq98RQGBcXBwD466+/8Msvv2D+/Pk4cOAAUlJSYGFhAZlMhr1798LS0hItW7aEmZnB7oVNRERERM9Ap/sUuri4oFevXgCAZcuWYdeuXZgwYQIEQUBycjKGDh2Ktm3bIjw8XMq1EhEREZFEdL559aOaNWuGkJAQAMDKlSuRmpqKGTNmwMHBoVoXR0REREQ1Q+f9XgsLCwwYMACOjo4V+po3b47mzZtj+PDh1bo4IiIiIqoZOodCKysrREVFia+fFBKJiIiI6MXy1FeGPB4SiYhIOjZ2lrCUyyWtUaTRIP9ukaQ1iOj5xcuFiYheAJZyOfp8v0HSGj8Ev418MBQSGaunutCEiIiIiF4uDIVERERExFBIRERERAyFRERERASGQiIiIiICQyERERERgaGQiIiIiMBQSERERERgKCQiIiIiMBQSERERERgKiYiIiAgMhUREREQEhkIiIiIiAkMhEREREYGhkIiIiIjAUEhEREREYCgkIiIiIjAUEhEREREYComIiIgIDIVEREREBIZCIiIiIgJDIRERERGBoZCIiIiIYOBQmJubi+joaISFhaF169Zwc3PD4cOHKx37448/YsCAAWjVqhXefPNNxMbGoqSkpMK4vLw8zJs3Dx06dICPjw9GjBgBpVL5THMSERERvewMGgovXryI+Ph43LhxA25ublWO++mnnzBhwgTY2tpi3rx56NatG1asWIGoqCitcWVlZRg7dixSU1MRGhqKGTNm4NatWwgLC8Ply5efak4iIiIiY2BmyOKenp7IyMiAvb099u3bhwkTJlQ6btGiRWjRogXWrFkDU1NTAEDt2rWxevVqhIWFwcXFBQCwe/dunDx5EitWrEC3bt0AAD179kSPHj0QGxuLRYsW6T0nERERkTEw6JFCa2tr2NvbP3FMVlYWsrKyMGTIEDG8AcDw4cNRVlaGvXv3im179uyBo6MjunbtKrY5ODigZ8+e2LdvHzQajd5zEhERERmD5/5Ck7NnzwIAWrZsqdXu5OSE+vXri/0AoFQq4enpCZlMpjW2VatWuH//vriFrM+cRERERMbAoNvHulCpVAAAhUJRoU+hUCA3N1drbIcOHSqMc3R0BPDwwpbmzZvrNaeu6ta11vs9pD+Fwoa1WZu1WZuIJPDch8KioiIAgLm5eYU+CwsLPHjwQGtsZePK28rn0mdOXd26VYCyMgEOtpYwNZfr/X59lKo1uH2vSNIa+qqpX+YqVT5rszZrG1FtIqpeJiayKg9kPfeh0NLSEgCgVqsr9BUXF4v95WMrG1feVj5Wnzn1ZWouhypu/VO/XxeK8aEAnq9QSERERC+25/6cwvIt3vIt30epVCpxa7h8bGVbv+Vt5WP1mZOIiIjIGDz3odDDwwMAcObMGa32GzduICcnR+wHAHd3d/z+++8QBEFrbGZmJqysrNCkSRO95yQiIiIyBs99KHzttdfwyiuvYPPmzSgtLRXbN27cCBMTE3Tv3l1sCwwMRG5uLn788Uex7fbt29i9eze6du0KuVyu95xERERExsDg5xSuXLkSAJCdnQ0ASElJwfHjx1GnTh2EhoYCAGbOnInx48djzJgx6NWrF/78809s2LABQ4YMQbNmzcS5evToAR8fH8ycOROjR4+Gvb09Nm7ciLKyMkycOFGrrq5zEhERERkDg4fCZcuWab3eunUrAKBRo0ZiKOzcuTNiY2MRGxuL+fPnw8HBAePHj8d7772n9V5TU1OsXr0aixYtQmJiIoqLi9GqVSt89tlnaNq0qdZYXeckIiIiMgYGD4Xnzp3TaVy3bt3ER9c9ia2tLRYuXIiFCxdW25xEREREL7vn/pxCIiIiIpIeQyERERERMRQSEREREUMhEREREYGhkIiIiIjAUEhEREREYCgkIiIiIjAUEhEREREYComIiIgIDIVEREREBIZCIiIiIgJDIRERERGBoZCIiIiIwFBIRERERGAoJCIiIiIwFBIRERERGAqJiIiICICZoRdA1cfB1gKm5uaS1ihVq3H7XrGkNYiIytnYWcFSbippjSJNKfLvFkpag+hFwFD4EjE1N0fOyo8krVH/vf8AYCgkopphKTfFwK0ZktZIGtQB+ZJWIHoxcPuYiIiIiBgKiYiIiIihkIiIiIjAUEhEREREYCgkIiIiIjAUEhEREREYComIiIgIDIVEREREBIZCIiIiIgJDIRERERGBoZCIiIiIwGcfExERVaqOnRUs5KaS1ijWlCLvbqGkNYh0xVBIRERUCQu5KSYlX5G0xvIBjSWdn0gf3D4mIiIiIoZCIiIiImIoJCIiIiIwFBIRERERGAqJiIiICEYeCtVqNRYvXoyAgAB4eXlh8ODBSE9PN/SyiIiIiGqcUd+SJjIyEnv37sWIESPQtGlTJCcnIyIiAomJiWjdurWhl0dEREbKzq425HJpj9toNGW4e/e+pDXoxWK0oTAzMxOpqamYPXs2wsPDAQBBQUHo06cPoqOjsWHDBsMukIiIjJZcboKNW1WS1hg2SCHp/PTiMdrt4927d0MulyMkJERss7CwQHBwMI4fP47c3FwDro6IiIioZhntkUKlUolmzZqhdu3aWu1eXl4QBAFKpRKOjo46z2diIvvfv9vUfsLI6vFovUeZ2tgZrLbcRvfvq7prW1obrnZtayeD1a5T23C161oZrrajleF+zh2tbAxY23C/WxytahmstsLKwmC1Haykfczdk2rXtpL+uE1ltW3rWMFM4q3rEk0Z7uXx8X6GUNXPGwDIBEEQanAtz40+ffrAyckJa9as0WrPyspC7969sWDBAq2jiEREREQvM6PdPi4qKoJcLq/QbmHx8G+kxcXFNb0kIiIiIoMx2lBoaWkJjUZTob08DJaHQyIiIiJjYLShUKFQVHoxiUr18Govfc4nJCIiInrRGW0odHd3x8WLF3H/vvY9mk6fPi32ExERERkLow2FgYGB0Gg02LJli9imVquRlJQEX19fODlJf2UlERER0fPCaG9J4+3tjcDAQERHR0OlUqFJkyZITk7GtWvXEBUVZejlEREREdUoo70lDfDwopKlS5dix44duHfvHtzc3DBt2jT4+/sbemlERERENcqoQyERERERPWS05xQSERER0f8wFBIRERGR8V5oYki5ublYt24dTp8+jTNnzqCwsBDr1q1D+/btJa2bmZmJ5ORkHD58GNeuXYOdnR1at26NKVOmoGnTppLW/u2337Bq1SqcPXsWt27dgo2NDdzd3TFhwgT4+vpKWrsy8fHxiI6Ohru7O1JSUiSrc/jwYYwYMaLSvp07d6J58+aS1S6XmZmJ2NhYnDx5EiUlJWjcuDHCw8MxcOBAyWpGRkYiOTm5yv6DBw9KeoX/X3/9haVLl+LEiRPIy8tDw4YNERQUhPDwcJibm0tWFwBOnTqFzz//HJmZmTAxMUH79u0RGRmJJk2aVGsdfX6P/Pjjj4iNjUVWVhbq1q2L4OBgvPvuuzAze7o/AnStvXHjRmRkZCAzMxPXrl3DgAED8Omnnz5VTX1q37lzB1u3bsX+/ftx4cIFlJSUoHnz5ggPD0fPnj0lrS0IAj766COcPHkS169fR2lpKRo3bozg4GAMGzas0idpVVftx/3999/o1asXioqKsG3bNnh4eEhau0uXLvj7778rvD8iIgLTp0+XtDYA5OfnY8WKFdizZw9UKhXq1q0LPz8/xMTESFb7Sb/jAWDKlCkYP368JLWBh9dGJCQkICUlRfxzvU2bNnj//ffRrFkzvesyFBrAxYsXER8fj6ZNm8LNzQ0nT56skbpfffUVTpw4gcDAQLi5uUGlUmHDhg0ICgrC999/L2lAuXLlCkpLSxESEgKFQoH8/Hzs2LEDoaGhiI+Px+uvvy5Z7cepVCrExcXBysqqxmqOHDkSnp6eWm01cdujn376CRMmTEC7du0wefJkmJmZ4a+//sL169clrTtkyBB07NhRq00QBHz88cdo1KiRpJ/9xo0bCAkJgY2NDUJDQ2Fra4tjx45hyZIlOH/+PBYvXixZ7czMTISGhqJRo0aYOHEiysrK8O2332L48OHYtm0b6tWrV221dP09Uv4z0KFDB8ybNw9//vknVqxYgTt37mDevHmS1o6Pj0dBQQFatWolPhjgWelS+9SpU1i6dCk6deqE8ePHw8zMDHv27MGUKVNw4cIFTJgwQbLaZWVl+P333xEQEABnZ2eYmpri1KlT+OSTT3DmzBksWrRIstqP++yzz2Bi8uwbgvrU9vT0xMiRI7XaXF1dJa+dl5eHt99+G3l5eQgJCUH9+vWhUqlw9OhRSWs3b9680v+m27dvx6FDh576zzZdP/eMGTPw448/YvDgwWjRogVycnKwYcMGHDp0CDt37kTdunX1KyxQjcvPzxdu374tCIIgpKWlCa6urkJGRobkdY8fPy4UFxdrtV28eFFo2bKlMGvWLMnrP66wsFDw9/cXxo4dW6N1Z82aJYSFhQmhoaFCv379JK2VkZEhuLq6CmlpaZLWqUxeXp7QsWNHYf78+TVeuzJHjx4VXF1dhbi4OEnrfPnll4Krq6vw559/arVPnDhRaNGihaBWqyWrPWbMGKFdu3bC3bt3xbYbN24IPj4+woIFC6q1lq6/R3r16iUMGDBAKCkpEdtiYmIEd3d34eLFi5LWvnr1qlBWViYIgiD4+flVy+8ZXWpfvnxZuHr1qlZbWVmZMGLECMHLy0t48OCBZLWrMn/+fMHNzU24detWjdTOyMgQPD09hZiYGMHV1VU4e/bsU9XVp3bnzp2F8ePHP3WdZ6k9b948oUuXLuLYmqxdmbfeekvo3r27pLVVKpXg6uoqfPrpp1rt+/fvF1xdXYXvv/9e77o8p9AArK2tYW9vX+N1fX19K2ydubi44LXXXkN2dnaNr6dWrVpwcHBAXl5ejdXMzMzE9u3bMXv27BqrWa6goAAlJSU1Vm/Hjh3Iy8vD5MmTxfqCAW828MMPP0Amk6FPnz6S1il/StHjf0OuV68ezMzMYGpqKlntEydOICAgALa2tmKbo6Mj2rVrh127dlVrLV1+j2RlZSErKwtDhgzR+tzDhw9HWVkZ9u7dK1ltAGjUqBFkMtlT1XiW2o0bN0ajRo202mQyGbp164aioqJKtzirq3ZVGjZsCEEQkJ+fL3nt0tJSLFy4EKGhodVyapC+n1utVuPBgwfPXFfX2nl5eUhOTsaYMWNgb2+P4uJiqNXqGqldmczMTFy6dAl9+/aVtHZBQQEAVNiBKH9taWmpd12GQiMnCAJu3rxZYyG1oKAAt2/fxoULFxATE4M///yzwjajVARBwPz58xEUFPTU59Y8rRkzZsDPzw/e3t4YPXo0zp07J3nN9PR0vPLKK/jpp5/wxhtvwM/PD+3atUN0dDRKS0slr/8ojUaDXbt2oXXr1nB2dpa0Vtu2bQEAc+bMwR9//IHr169j+/btSE5ORkRERLVsp1VFrVbDwsKiQrulpSVUKlWlz1uX0tmzZwEALVu21Gp3cnJC/fr1xX5jcfPmTQCokd93Go0Gt2/fxvXr15GWloa1a9eicePGkv/8A8CmTZtw48YNvPfee5LXetwvv/wCHx8f+Pj4oFu3bti8ebPkNY8dOwa1Wo169eohPDwc3t7e8PHxwejRo3H58mXJ6z9u+/btAPBMoVAXzs7OaNCgARISErB//37k5OTg1KlTWLhwIZo3b46uXbvqPSfPKTRy27dvx40bNzB16tQaqffvf/8be/bsAQDI5XIMHToU7777bo3U3rZtG7KysrBixYoaqQc8/Iw9evRAp06dYG9vj3PnzmHt2rUYPnw4vv/++6c6EVhXly5dQk5ODiIjI/HOO++gRYsWOHDgAOLj41FcXIw5c+ZIVvtxhw4dwt27dyX/JQkAAQEBmDx5Mr788kvs379fbJ80adJTn0umq2bNmuHUqVMoKysTw6darUZmZiaAhyePOzo6SrqGR5Wfx6dQKCr0KRSKGg+phnT37l1s2bIF7dq1g4ODg+T1Dh06pPW7rWXLloiKipL0SDXw8HMuX74cEydORJ06dSSt9ThXV1e0adMGLi4uuHPnDr777jt8+OGHuHfvHsaOHStZ3fLgN2/ePLRs2RIxMTHIzc1FbGwsRo4ciR07dsDa2lqy+o8qLS3Frl274OXlJfkFnGZmZli+fDk++OADrYtZfHx8sH79+qc6UshQaMSys7Pxf//3f/Dz80P//v1rpOaECRMwZMgQ5OTkICUlBWq1GhqNRvIrQgsKCrBkyRKMHTu2Rv9Q9vX11bq6umvXrujSpQsGDRqE2NhYLFmyRLLahYWFuHfvHj744APxF3L37t1RWFiIjRs3Yvz48TXyhyPwcOtYLpc/05Wf+nB2dka7du3w1ltvwc7ODv/973/xxRdfwMHBAcOGDZOs7vDhw/Hxxx9j7ty5GD16NMrKyhAXFyeGs6KiIslqV6a8XmX/f1lYWFTbFt/zrqysDNOnT0d+fj7mzp1bIzW9vb2RkJCA/Px8ZGRkQKlUorCwUPK6y5cvh4ODA4YOHSp5rcetWrVK6/XAgQMxfPhwrFy5EsOGDYONjY0kdctPGVEoFIiPjxf/QtasWTOMHTsWW7durXDxi1TS09Nx8+ZNjBs3rkbq1alTBx4eHujZsye8vLxw+fJlfPnll5g8eTLWrFmj95+tyuHwGAAAEI5JREFU3D42UiqVCuPGjYOtrS2W/X979x7V8/0HcPxZzY8i0ThULtGsoY2GSB2mQi7NJUaR0Cg7CzEODnM9c+uYyYqMiRPlUtE0l4ohrGEOMtfVlBSV0EVK/f7o9D2++4akz7fNXo9z+uP7ub3en0/f8/m+Pu/b57vvFG1Se56lpSV2dna4urqyefNmkpKStNK/LygoiDp16jBhwgTFY73KBx98gK2tLWfOnFE0TsVT4t/78Lm4uFBcXMylS5cUjV8hPz+fuLg47O3ttdJsd+DAARYuXMiyZcv47LPP6NevH9988w3Dhg1j1apVPHz4ULHYbm5u+Pj4sH//fgYNGoSLiwu3b9/Gy8sLgPr16ysWuzIV34HK+lcVFRVVqybh32jp0qWcPHmS5cuXY2lpqZWYxsbG9OzZk/79+7Nw4UIcHR2ZMGFCjY3Crsz169cJCwtjzpw51Z5uqCbp6enh6elJYWGhorNsVHyPnZ2d1X7LevfujZGREefPn1cs9t9FR0ejp6fHwIEDFY/1+PFjxowZQ5cuXZgxYwZOTk5MnDiRgIAAEhMTiYqKeu1jSlL4H/T48WMmTZrE48eP+eGHHyptWtKGOnXq4OjoyOHDhxWtQbl37x4hISG4u7uTlZVFWloaaWlpFBUVUVxcTFpamqKJQmVMTEwUj1nxf31RJ2RtnXNsbCyFhYVaaToG2LFjBx07dtSY9sbBwYGCggKuXr2qaHw/Pz8SEhIIDQ1l//797N27l7KyMnR0dGjZsqWisf+u4jtQWSJy//59rdaa15b169ezY8cOZs2apfggp5dxdnamoKCAuLg4xWKsWbOGDh06YGFhobrPPXjwACi/Dyo9FVVlmjdvDih7v3nRvQ7Q6mDGJ0+ecOTIEWxtbWt0+qkXOXToEFlZWTg4OKgtt7GxoUGDBtVKhmv/UUJoVVFRET4+PqSkpLB161batm1bq+V58uQJZWVl5OfnK1ZrkZ2dTXFxMf7+/vj7+2usd3R0fKPJVasjNTVV8Vqzjh07curUKTIzM9WSkYyMDACtNR1HR0djYGCgceNSSlZWVqXnVlxcDKCVQTZGRkZ07dpV9fnUqVN89NFHWuvXVKFiQNXly5fV5snMzMwkIyND6wOutC00NJSAgADGjx+vqq2tLRUPvtUdfVwVd+/e5erVq5UOMJg8eTJNmjQhISFBsfiVSU1NBZS931R8tzMzM9WWl5aWcv/+fY05YpUSHx9Pfn6+1h6As7OzgfLzfF5ZWRmlpaXVmu1CksL/kGfPnjF9+nQuXLhAYGAgnTt31lrsnJwcjZtCXl4ehw4dwsTE5PUn2HwNLVq0qHRwydq1aykoKGDevHmYm5srEruy8z579iy//vorQ4cOVSRmBWdnZzZt2sSePXtUA4nKysrYvXs3BgYGWvn/5+TkcPr0aQYNGoS+vr7i8aC8H1FCQgK3b99We4vIgQMH0NPT01rzYYWYmBguXbpU7bcqvIl27drRtm1bwsPDGTFihGqQw86dO9HV1aVfv35aL5O2xMTEsGzZMlxcXJgzZ47W4ubm5mJoaKgxoGT37t2A5kjwmjR37lzVNCUVzpw5w/bt25k7d66ilQC5ubk0bNhQrfm2qKiIzZs3U79+fUXvNxYWFrz//vtER0fj4+OjmgEgJiaGvLw8rc1wER0djb6+Pn379tVKvIrfrQMHDqiNNI+Li6OgoIAOHTq89jElKawlgYGBAKr5Afft28e5c+do2LAhY8eOVSTmihUriI+Pp0+fPuTm5qq93q1+/fo4OTkpEhfKX/VTt25drK2tadq0KXfv3iUiIoKMjAzFfywNDQ0rPbeQkBD09PQUP299fX2sra1p3LgxN27cIDw8nMaNG+Pr66tYXCj/8Rk6dCgbN24kOzubDh068Msvv3Dy5ElmzZqllVqrmJgYSkpKtPbkDODl5cXx48dxc3NjzJgxGBkZcezYMY4fP87o0aMVfQA5ffo0GzduxM7OjkaNGnHhwgUiIyNxcXFh0KBBNR6vKveR2bNnM2XKFLy8vBg4cCDXr18nNDSUUaNGvdHo96rEjo+PVzXXP336lGvXrqn2GzJkiMZcgjUV++LFi8yePZtGjRpha2urmiKkgp2dXbWb914VOz4+nqCgIPr27UurVq0oLCzk5MmTnDx5kk8++eSNEpRXxe7Ro4fGPhVNp927d3+jmuGqnPeGDRvo378/ZmZm5ObmEhkZSUpKCosWLXqj/rRV+a7NmTOHSZMm4e7uzpAhQ7h//z4hISF06NCBTz/9VNHYUJ4Unzhxgn79+tVY3+FXxe7Tpw/t2rUjICCAtLQ0OnXqREpKCqGhoTRr1qxarzLVKavN2Wz/w15UW2FmZqY2jUZN8vDwIDExUetxAfbs2cO+ffu4efMmjx49wtDQUDWPlI2NjWJxX8bDw4NHjx4p+u7jbdu2ER0dze3bt8nLy8PY2Bh7e3t8fX0xNTVVLG6Fp0+fEhgYSFRUFFlZWbRo0YLx48drbWTiqFGjSE1N5cSJE4pPxfG8ixcvEhAQwB9//EFubi5mZma4urri5eWlaDlSUlJYsmQJV65cIT8/H3Nzc0aOHMnYsWMVGcxV1ftIbGws69ev59atWxgbG+Pq6soXX3zxRoMRqhL7Ze/AfpP3vb8qdkRExEsHsCkZ+/r162zcuJHff/+drKwsdHV1adOmDS4uLnh4eFT73cdViV2ZimvxJu8+rkrsy5cvs379eq5cuUJOTg7/+9//6NixIxMnTqRPnz7VjluV2BWOHz9OQEAA165dw8DAAEdHR7766qs36qpT1dhhYWEsXLiQoKCgGusqU5XYDx8+JDAwkGPHjpGenk79+vWxs7NjxowZ1XrokqRQCCGEEELI6GMhhBBCCCFJoRBCCCGEQJJCIYQQQgiBJIVCCCGEEAJJCoUQQgghBJIUCiGEEEIIJCkUQgghhBBIUiiE+I87evQolpaWHDx4sLaL8lYbPnw4gwcPru1iCCFeQl5zJ4T4x3ud9xXHxcXRokULBUujPStXrmTLli3ExMRgYWFR28V5pfDwcEpKShgzZkxtF0UIUQ2SFAoh/vFWrVql9vncuXOEh4czatQounTporbO2NhYm0UTzwkPD+fp06eSFArxLyVJoRDiH2/IkCFqn589e0Z4eDidO3fWWCeEEKJ6pE+hEOKtlJeXx8qVK3FwcMDKygp7e3vmzZtHZmZmlfYPDQ2lffv2zJo1i+LiYtXy8+fP4+3tjY2NDVZWVgwYMIDNmzdTWlqqtn9FH7o7d+4wdepUunbtirW1Nd7e3qSmptbouQKkp6czf/58evXqhZWVFb169WLx4sXk5uaqbbdy5UosLS1JT09nxYoV2NvbY2VlxfDhwzl9+rTGcfPy8liyZAm2trZ06tQJNzc3zp49y9SpU7G2tlZt1717d5KSkrhx4waWlpaqv0uXLqkdT1vXQwjx+qSmUAjx1ikqKmLcuHEkJSUxePBgPv74Y/7880/CwsJISEggIiKCd99994X7f/vtt2zYsAEvLy9mzZqFjo4OAAcPHmTmzJm89957fP755xgaGnL27FlWr17NzZs3Wb58udpxHj16xNixY+nRowczZ84kOTmZ0NBQpk2bRkRERI2db3JyMu7u7ujq6jJixAjMzMxITk5m586d/Pbbb+zatQsDAwO1faZPn46+vj6TJ0+msLCQrVu34uPjQ2xsLE2bNgWgtLSUKVOmkJiYyIABA+jWrRspKSl4e3tjYmKidrzFixezevVqSkpKmDFjhmp5y5YttX49hBDVI0mhEOKts3PnTpKSkvD19eXLL79ULe/SpQt+fn6sW7eOxYsXa+xXUlLC/PnziYqKYt68eXh6eqrW5eXlsWDBAnr06MGmTZvQ1S1vaHFzc6Nt27asW7cOd3d3PvzwQ9U+mZmZfP3112p97Bo0aMD333/PuXPnNPpDVteiRYuoU6cOkZGRasmug4MDHh4e7Ny5Ey8vL7V9TE1NWbt2repzp06d8PT0ZM+ePUyZMgUoT4ITExPx9PRk3rx5qm2tra3x8/NTSzSdnZ0JDg7m6dOnL2zS19b1EEJUjzQfCyHeOkeOHKFu3bpMnDhRbfnAgQMxNzcnNjZWY5/CwkJ8fHw4cOAAa9asUUsIAY4dO8ajR49wdXUlNzeXnJwc1d8nn3wCQEJCgto+9erVY/To0WrLevToAcBff/31pqcJwL179zhz5gz9+vVDR0dHrVwWFhY0a9ZMo1yAxvl169aNd955R61cR48eBWDChAlq2w4YMABTU9PXLqs2rocQovqkplAI8dZJS0vDzMxMo8kUwMLCgri4OAoLC9HX11ctX7p0Kfn5+QQGBuLo6Kix361btwDw8/N7YdysrCy1z6ampujp6akta9SoEYBGX7/qqijX9u3b2b59e6XbGBkZaSx7vlkXQE9PjwYNGqiVKy0tjXr16mk0Fevo6GBubs6FCxdeq6zauB5CiOqTpFAIIYD+/fvz008/ERwcjI2NDYaGhmrry8rKAFiwYAFt27at9BjNmzdX+/z3BKiy49WUkSNHMnDgwErXVZYcVzR/K12u52nzegghXp8khUKIt07Lli25ePGiRm0glNesNWnSRGN57969cXZ2xtfXl/Hjx7Nlyxa1GjZzc3OgvA9cz549FT+HqmrdujVQPiikpstlZmbG+fPnuXv3rlptYVlZGSkpKRrbVwzIEUL8O0mfQiHEW8fJyYmioiJ+/PFHteU///wzKSkpODk5Vbpf7969CQoK4ubNm4wfP54HDx6o1jk4OGBoaEhQUBB5eXka+xYUFFBQUFCzJ1IFpqam2NjYEB0dzdWrVzXWl5aWkpOTU61jOzg4AFR6HdPT0zW2NzAwkGZgIf7FpKZQCPHWcXNzY//+/Xz33XekpKTQuXNn1ZQ0zZs3Z+rUqS/c187OjuDgYHx8fPD09GTr1q0YGxvTsGFDVqxYwfTp03F2dmbYsGG0bNmShw8fcuvWLQ4fPkxISIja6OOaEhYWRuPGjTWWW1tbY2try7JlyxgzZgwjR45k2LBhWFpaUlJSQmpqKrGxsXh4eGiMPq4KZ2dnQkNDCQkJ4f79+6opafbu3Uu7du24c+eO2vadO3cmMTGR5cuX07FjR/T09LC3t6+0T6MQ4p9HkkIhxFunbt26bNu2jfXr13Po0CFiYmIwMjLCxcWF6dOnv3SOQiifiHnTpk1MnjyZcePGsXXrVpo0aYKTkxO7du1i06ZNREZGkpubi5GREa1atcLb25s2bdoocj7btm2rdPnEiROxtbWldevWREZGEhwczLFjx4iIiEBfXx8TExOcnZ0rHThTFbq6umzcuBF/f38OHjxIfHw87du3Jzg4mMDAQLKzs9W29/b2JiMjg6ioKEJCQigrK2PPnj2KJMpCiJqnUya9e4UQQrwmR0dHjIyMZNJpId4i0qdQCCHECz158kRjWUxMDGlpadjZ2dVCiYQQSpHmYyGEEC/k7+9PWloaXbt2xcDAgMuXLxMVFUXTpk01JrUWQvy7SfOxEEKIFzpy5AibN28mOTmZvLw8GjduTM+ePZk2bRpmZma1XTwhRA2SpFAIIYQQQkifQiGEEEIIIUmhEEIIIYRAkkIhhBBCCIEkhUIIIYQQAkkKhRBCCCEEkhQKIYQQQgjg/xp52ek5hBfjAAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": { + "tags": [] + }, + "output_type": "display_data" + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "import seaborn as sns\n", + "import numpy as np\n", + "\n", + "sns.set(style='darkgrid')\n", + "\n", + "# Increase the plot size and font size.\n", + "sns.set(font_scale=1.5)\n", + "plt.rcParams[\"figure.figsize\"] = (10,5)\n", + "\n", + "# Measure the length of every token in the vocab.\n", + "token_lengths = [len(token) for token in tokenizer.vocab.keys()]\n", + "\n", + "# Plot the number of tokens of each length.\n", + "sns.countplot(token_lengths)\n", + "plt.title('Vocab Token Lengths')\n", + "plt.xlabel('Token Length')\n", + "plt.ylabel('# of Tokens')\n", + "\n", + "print('Maximum token length:', max(token_lengths))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wlDH4Wir-JFJ" + }, + "source": [ + "# New Section" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "QQ8YUh75Usqj" + }, + "source": [ + "Let's look at just the tokens which begin with '##'." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "69Vf46p-Ym97", + "vscode": { + "languageId": "python" + } + }, + "outputs": [], + "source": [ + "num_subwords = 0\n", + "\n", + "subword_lengths = []\n", + "\n", + "# For each token in the vocabulary...\n", + "for token in tokenizer.vocab.keys():\n", + " \n", + " # If it's a subword...\n", + " if len(token) >= 2 and token[0:2] == '##':\n", + " \n", + " # Tally all subwords\n", + " num_subwords += 1\n", + "\n", + " # Measure the sub word length (without the hashes)\n", + " length = len(token) - 2\n", + "\n", + " # Record the lengths. \n", + " subword_lengths.append(length)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lakK1gF-S1Ej" + }, + "source": [ + "How many '##' tokens are there vs. the full vocab?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 50 + }, + "id": "PJh6wl2bSzaC", + "outputId": "9ee06298-f3af-4010-92b3-f50601fec3bb", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of subwords: 5,828 of 30,522\n", + "19.1%\n" + ] + } + ], + "source": [ + "vocab_size = len(tokenizer.vocab.keys())\n", + "\n", + "print('Number of subwords: {:,} of {:,}'.format(num_subwords, vocab_size))\n", + "\n", + "# Calculate the percentage of words that are '##' subwords.\n", + "prcnt = float(num_subwords) / vocab_size * 100.0\n", + "\n", + "print('%.1f%%' % prcnt)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fj_AcObWS-1S" + }, + "source": [ + "Plot the subword lengths (not including the two '##' characters)." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 389 + }, + "id": "fO1GeFbhSyhD", + "outputId": "4e155231-d0a6-4b1e-caa9-638e75f6324d", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "Text(0, 0.5, '# of ## Subwords')" + ] + }, + "execution_count": 11, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAoUAAAFjCAYAAABL3HHWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd1QUV/8G8Gdpi1SBgEQsKLILggqoNI0VC0ZfsSsKIpbEqCkmtiTmJPq+MbbYsMcSDRororEhFoyEqgaiErsRYwFZQVDp8/vDw/xcF3DRXRB9Pud4kr1zZ/Y7K8jDvXNnJIIgCCAiIiKit5pOTRdARERERDWPoZCIiIiIGAqJiIiIiKGQiIiIiMBQSERERERgKCQiIiIiMBQS1Zjp06dDLpfXdBkasXv3bsjlciQkJNRYDR9//DHc3d1r7P3fBpcuXYJcLse6deuq7T1/+ukneHl5IS8vr9re822Tl5cHLy8vrF+/vqZLoRrGUEj0Aunp6Zg5cyZ69uyJVq1aoW3btvD398e0adMQHx9f0+XVGmXBUZ0/QUFBNV2uRnl5eWHw4ME1XYZasrKysGzZMpw5c6amS4FCocCqVaswduxYmJiYaPTYt27dglwux969ezV2TC8vL8yaNQsAMHfu3Bf+klJYWIiWLVti06ZNAICCggK4urpi586dar2fn58f5syZI77u1KkTlixZotLv+PHjkMvl+Ouvv1TqBAATExOMGjUKK1euxIMHD9R6b3oz6dV0AUSvs7/++gtBQUHQ09NDQEAAmjVrhvz8fPzzzz+IjY2FsbExvL29a7rMWqFt27aYN2+eUtuqVatw7do1lfZ33nmnOkujZ2RlZSEsLAxGRkbw8PCo0Vp+/vlnFBcXY+jQoRo/dnR0NPT19dGpUyeNH1tdFy5cQEFBAVq3bg0ASElJQVFRkVoj3pmZmUhPTxf3/ffff3Hnzp2XHi0fNmwYwsLCEB4ejokTJ77UMaj2YygkqsTy5cvx5MkTREZGwsnJSWV7ZmZmDVRVvUpKSlBYWIg6deq80nEaNmyIhg0bKrXt3LkT165dQ9++fV/p2PTmKSwsxM6dO9G1a1eNjxICwJEjR+Dp6QkzMzONH1tdZ8+ehZGRkfhvy+nTp1G3bl00bdpUrX0BiMH99OnTkEgkLx0Kzc3N0blzZ+zYsQPjx4+Hrq7uSx2HajdOHxNV4saNG6hbt265gRAArK2txf8vm45atmyZSr9ly5ZBLpfj1q1bKtsUCgWmTp0KLy8vuLm5YeTIkTh//rxSny5dumDEiBFKbatXr4ZcLsf48eOV2ufPnw+5XI779+8rvcd3332Hjh07wtXVFR07dsR3332nMlVUNsX7xx9/YPny5fDz80PLli1x8OBBsc/27dvRs2dPuLq6olu3bti4cSO0+bTMv/76Cx988AHatm2LFi1aoE+fPvj555/Ves+8vDyEhoaiZcuWOHLkiNheUlKCjRs3om/fvmjZsiU8PDwwatQonD59Wmn/Z6+hO3ToEAICAtCiRQu89957WLx4MUpLSzV+vidOnEBwcDBat26NFi1aICAgALt27VLp5+XlhQ8++AB///03QkND4e7ujjZt2uDzzz8vdwqwbNS7VatW8Pb2xtdff427d+9CLpeLU4nHjx9Hnz59AADz5s0Tp/N79+6tcjx1Po+0tDRMmDAB7du3h6urK9q3b4+QkBD88ccfL/wckpOTcf/+fXTs2FGpPTw8HHK5HKmpqWLb48eP4erqCrlcjhs3bojtWVlZcHJywty5c5WOkZWVhTNnzsDPz0+p/Y8//kBQUBA8PDzQqlUrDBgwQKPTywCQnZ0NhUIBhUKBpKQkODk5IScnR3zt7OyMBw8eQKFQID8/X2nf3Nxccd/ExETUr18fOjo6UCgUSEhIQOPGjVFUVASFQoHHjx9XubYOHTrg7t27YuCktw9HCokq0ahRI1y/fh1RUVHo3r27Vt5jzJgxMDc3x8SJE3H//n388ssvGDFiBLZt2waZTAYA8Pb2xt69e5Gfnw9DQ0MAQFxcHHR0dJCUlISSkhLxN/v4+Hg0a9ZMnILNzc3FsGHD8M8//2DAgAFo3rw50tLSsHXrVsTHx2PHjh0qIzFz585FcXExBg8eDGNjYzRp0gQAsHHjRsyZMwdOTk6YPHkynjx5gvXr18PKykorn01ycjJGjRoFIyMjDB8+HJaWljhy5Ai+//57XL16Vem6qOdlZmZi3Lhx+Pfff7F+/Xq0adMGACAIAj7++GMcP34cvXr1wuDBg5Gfn4+IiAgEBwdj7dq18PX1VTrWoUOHcOfOHQwZMgSDBg3C4cOHsXLlSlhaWiI4OFhj51v2+Xp6emLChAmQSqWIiYnBl19+iTt37qhM66WnpyMkJAS9e/dG9+7dkZqail27dqGgoABhYWFiv7///hvBwcHQ09NDSEgI3nnnHRw9ehQfffSR0vGaN2+Ozz//HAsXLkTv3r3RoUMHAFAZTVPn87h37x5GjhwJqVSKoUOHwtbWFgqFAqmpqTh37pzKZ/y8xMREAEDLli2V2ssu14iPjxe3JScno6ioCDo6OoiPj4e9vT0AICEhAYIgqFzicezYMQiCgK5du4ptBw4cwOeff453330XY8aMgaGhIfbt24cpU6bgzp07+OCDDyqtV109evRAdna2UpuPj0+5r6dOnYrRo0eL7eX9wljRvsOHD8c333xTpdrc3NwAPP3sy75f6C0jEFGFzpw5I7i4uAgymUzo3r27MH36dCE8PFy4cuWKSt/09HRBJpMJS5cuVdm2dOlSQSaTCenp6WLbtGnTBJlMJkyYMEEoLS0V2//66y9BLpcLoaGhYltkZKQgk8mEU6dOCYIgCAUFBULLli2FL774QpDJZEJKSoogCILw8OFDwdnZWZg1a5a4748//ijIZDLhl19+Uarpl19+EWQymbBo0SKxbdeuXeK5Pn78WKl/Tk6O0KpVK8Hf319p2507dwQ3NzdBJpMJ8fHxlX+gzxkxYoQgk8kq3N6nTx/B1dVVuHbtmthWUlIijBs3TpDJZMKff/4ptk+aNElwc3MTBEEQrl27JnTp0kXo2LGjcPnyZaVjRkRECDKZTNi7d69Se0FBgdCrVy/h/fffF9suXrwoyGQyoXXr1sK9e/fE9uLiYqFr165Ct27d1DpPT09PYdCgQZX2uXnzptC8eXPh66+/Vtk2Y8YMwcXFRakGT09PQSaTCSdOnFDqO3XqVEEmkwl37twR28aMGSPI5XLh/PnzYtuzn+N3332ncs4//fSTSh1V+Tz27t1bbn3qmjhxoiCXy4Xi4mKVbe3atVP6/pg7d67QqVMnITAwUPj444/F9pkzZwouLi5CXl6e0v5jx45V+vsoKCgQvL29BR8fHyErK0tsz8/PFwICAoTmzZsrfZ6vIjExUYiNjRW//3755RchNjZW2LBhgyCTyYTt27cLsbGxQmxsrPDvv/8q7ZuSkiLExsYKBw8eFL93Y2NjhX379gkymUxYuXKluO/Vq1erXFt+fr4gk8mEzz77TCPnSrUPp4+JKuHu7o5du3ahX79+yM3Nxe7du/Hdd9+hV69eGD58ONLT01/5PcaMGQOJRCK+dnV1Rbt27RAXF4dHjx4BUB4dAZ5eT5Sfn48xY8bA1NQUcXFxAJ7+hl9SUqI0MnLkyBFYWlpiyJAhSu87ZMgQWFpaIjo6WqWmYcOGqVxDeOrUKTx58gTDhw9X2mZraytOOWpSeno6Ll68CH9/f3GkEgB0dHQwbtw4AFCaEi6TkpKCYcOGwcjICNu2bUOzZs2Utu/btw9WVlZo166dOBWnUCiQl5eHjh074vLly7h3757SPr169YKNjY34WldXF23btsWtW7dQXFyskfM9cOAAiouLMWDAAKW6FAoFunTpgqKiIpVb/tjb26tMr5b93f/zzz8Anq5o/eOPP+Dl5YXmzZuL/XR0dBASEvJStarzeZiamgJ4Oh1e9nVcFQqFAiYmJuVe2+bl5YXTp0+jqKgIwNPvC29vb/j4+Iijg8DTkUJXV1cYGxuL++bl5SEuLk5p6vj06dNQKBTi90QZqVSKUaNGobi4GMePH6/yOZSnbdu28PX1RUFBAYyNjTF48GD4+voiPz8fdevWxYABA+Dr6wtfX1/Ur19fad+WLVvC19dX/PciMDAQvr6+EAQBurq6GDZsmLivOtclPk8qlaJOnTrIysrSyLlS7cPpY6IXkMvl+OGHHwA8XeGXlJSEHTt2IDk5GR999BF27doFAwODlz6+g4NDuW2nTp3C7du34ejoCBsbGzRt2lQMhfHx8bC2toZcLkfbtm0RHx+PDz74APHx8dDR0YGnp6d4rFu3bsHV1RV6esrf7np6erC3t8eFCxdU3v/ZEPbscQCU+8OmvHN4VWXv93yoAwBHR0cAUAnlBQUFGDlyJKysrBAeHl7uIoKrV68iKytLZdrtWVlZWahXr574ukGDBip96tati5KSEuTm5sLCwkK9k6rE1atXAUAlvD9f17OeX7hTVhcAcYry7t27KC4uLvfv9GWCA6De59GhQwf06NEDW7ZswY4dO9CyZUu0a9cO77//vji9WxmJRFLhdaPe3t747bffkJKSAplMhrS0NIwcORJ2dnZYtmwZLl68iLp16+LGjRvw9/dX2vfkyZMoLCxUCoVlX2tlX1fPquhr7WVkZ2eL113GxMTAzc0Nubm5AJ7+0uXu7i7+vZmYmCj9u5KbmyuG4BMnTqBp06bQ09ODQqFATEwMZDIZSkpKoFAoYGRkJF5mUlWCICj9kkpvF4ZCoiqws7ODnZ0d+vbti8DAQJw5cwapqalo06ZNpf+QamI0ydvbG9u2bUNubi7i4+Ph5eUltv/4448oLCxEfHw8nJ2dYW5u/krv9bI/UGqagYEBunfvjsjISPz666/iiOKzBEFA/fr18b///a/C4zRq1EjpdWUrMSsKLlVVdpwlS5ZUuCL2+TBVHXWVR5331dHRwdKlS5GWlobff/8dycnJWLNmDVauXIlvv/0WAwcOrPQ9LC0t8ejRI6XrZcs8O3JeFrS8vb1haWmJOnXqIC4uTgzqz19PGB0djaZNm750IH4VVbmecMmSJejZs6fYXpXrCZ+/FlFdBQUFyM/PVxotpbcLQyHRS5BIJGjVqhXOnDmDjIwMABCDWE5Ojkr/8lYdl7l69ap4gfezbbq6ukrTR97e3tiyZQtOnDiBv/76C/379wfw9AdBfn4+jh49isuXL2PUqFFKx2rYsCGuX7+O4uJipdHC4uJi3Lhxo9zRpvKUjQ5du3ZN5YdR2SiXJpW935UrV1S2lbU9X7tEIsEPP/wAXV1dLFy4EEVFRZgwYYJSn8aNGyMlJQVt2rR5pRFeTWvcuDGAp/do1ORF/ra2ttDT08P169dVtl27dk2lTdOjRM7OznB2dsa4ceOgUCgwYMAA/Pjjjy8MhY6Ojjh8+DBu3LihMhLdsGFD2NnZIS4uDg8ePEDTpk3Fkd3WrVsjLi4OVlZWkEqlSvdaLCwsRExMDIYPH65yPAC4fPmySh0Vfa29jLCwMBQVFeHixYv44Ycf8MMPP6BevXpISUnB4sWLsXDhQjGQPf+0o2+//RZ5eXm4f/8+pkyZgi+++AIuLi74999/8fXXX2PGjBniwjR1RmLLU3bJQXkjpvR24DWFRJWIjY0td5QvPz8fsbGxAP5/6tTExATW1taIj49XGqVJT08v97q9Mj/99JNS//Pnz+OPP/6Aj4+P0rVQXl5ekEgkWLlyJYqKisQREJlMBisrK4SFhZW70tLPzw8KhQI7duxQat++fTsUCoXKbTkq0q5dOxgaGiI8PBxPnjwR2+/evYt9+/apdYyqaNiwIeRyOQ4ePCj+sAKA0tJSrFmzBgDQrVs3lf10dHTw/fffY8iQIVi6dKnKEx4CAgKQn59f7q2DACjdyqc69e7dG7q6uliyZAkKCwtVtmdnZ4vTh1UhlUrFa+3S0tLE9tLSUmzcuFGlv5GREYDyf7mpigcPHqiMVlpaWsLW1hYPHz584Uhm2SUQKSkp5W739vZGSkoKTp48qfQ17+3tjeTkZMTFxcHd3V0p+MfFxSEvL0/la97DwwOWlpbYvn270u18CgsLsWHDBujp6aFz587qnXglyq4nLC4uRt26dREQEABfX18UFRXB2toavXv3Fq8JfH5Ff9n1hHp6etDR0RGvRdTR0YGenp74urxrEdVV9lm3bdv2lc+VaieOFBJVYs6cOcjOzkaXLl0gk8lgaGgohqAbN24gICBA6Tf64cOHY/HixRgzZgz8/PyQkZGBX3/9FY6OjuIjpp53+/ZtjB49Gl26dEFmZiZ++eUXGBoaYsqUKUr9yu6XmJaWBjs7O6WRCy8vLxw4cAD6+voqo0xjxozBoUOHMGvWLFy4cAHOzs5IS0vDzp070aRJE4wZM0atz8Lc3ByffPIJ5s6di6FDhyIgIABPnjzBr7/+WuG1ia/qm2++wahRozBkyBAEBgbCwsICR44cQUJCAoYMGYJWrVqVu59EIsF3330HPT09rFixAiUlJZg8eTIAoF+/fjh58iTWrFmDP//8E++99x7q1q2LO3fuIDk5GdnZ2VoJuffu3cOKFSvK3TZ06FDY29vjyy+/xH//+1+8//776NOnD959911kZWXh77//xtGjRxETE/NSU3tffPEFkpOTERwcjOHDh+Odd95BdHS0eD3bs6OD9evXh42NDSIiImBjYwMLCwuYmpqKt6dR19atW7F792507doVjRo1go6ODuLi4nDmzBkMGDDghSOSrVu3hrW1NWJiYsRR8Wd5e3tj165duHnzpkooXLBgAR49eqRyfWZ0dDRsbW3RokULpXYDAwPMnDkTkydPxsCBAzFo0CBIpVLs27cP58+fx+TJk2Fra1ul869MUlISPDw8xM8gKSlJ7dHhpKQkODo6ijMTycnJaN68uRjmX0VMTAxsbW35DPG3GEMhUSWmT5+Oo0eP4vTp0zh8+DByc3NhamoKmUyGsWPHqvywGjt2LHJzc7F3714kJiaiWbNm+N///ofz589XGAp/+uknzJkzB8uWLUN+fj5atWqFqVOnlnvDbG9vb6SlpamMBvr4+ODAgQMqKy2Bp6tAt27diqVLl+LYsWPYvXs3rKysMHToUEyaNKlKT4sIDQ2FkZERNmzYgIULF+Ldd99FaGgoTE1N8eWXX6p9HHW1adMG4eHhCAsLw6ZNm5Cfny+GpxfdH1AikeCbb76Bvr4+Vq9ejaKiIkybNg0SiQSLFi1Cu3btsGvXLqxcuRIlJSWwtraGq6srRo4cqfHzAJ6OqJb3XFrg6WiupaUlRowYAUdHR2zYsAHh4eF49OgRLCws4ODggC+++EJc0VtVTk5O2LRpE+bPn4/169ejTp068PPzQ2hoKHr16gWpVCr2Lft85s2bh/nz5yM/Px+Ojo5VDoXt27fHtWvXcPToUdy/fx+6urpo1KgRvvrqKwQGBr5wfz09PQwaNAjr1q0Tv++eVfY9IJFIxOtrAcDFxQXm5ubIyclR+j4pLS3FsWPH0KNHj3IDaa9evWBmZobVq1dj9erVKCkpgaOjI+bNm6fRJ+6UlJTg9OnT4mUNhYWFSE1NxdSpU9XaPykpSel8k5KSyh0xr6qcnBycOHECH374IZ9m8haTCNq8GpmIiF5biYmJCAoKwsyZM1WemPM6UCgU6N69O8aNG1fuoqGqOH36NAIDA7Fhw4YX3jj7bbRy5UqsX78eUVFRGllNT7UTrykkInoLPP/ItNLSUqxbtw6A6irW14WlpSXGjx+PdevWIS8v75WOJQgCJk2apHS7JnoqLy8PGzduxPjx4xkI33IcKSQiegt06tQJXbt2hYODA/Ly8hAdHY2UlBQMHDiw0tvzENHbg6GQiOgtMHv2bJw6dQr37t1DSUkJGjVqhICAAISGhvIaMiICwFBIREREROA1hUREREQEhkIiIiIiAu9TqDEPHjxCaSln4omIiOj1paMjgYWFcbnbGAo1pLRUYCgkIiKiWovTx0RERETEUEhEREREDIVEREREBIZCIiIiIgJDIRERERGBoZCIiIiIwFBIRERERGAoJCIiIiIwFBIRERER+EQTordSXVMD6BtKa7oMtRXlFyA7t7CmyyAieqMxFBK9hfQNpTgQPKqmy1Bbr00bAIZCIiKt4vQxERERETEUEhERERFDIRERERGBoZCIiIiIwFBIRERERGAoJCIiIiLUcCjMyMjAggULEBQUBHd3d8jlciQkJKj069KlC+RyucqfBQsWqPR9+PAhZs6cCW9vb7i5uSE4OBhpaWnlvv/Ro0fRr18/tGjRAp06dUJYWBiKi4s1fp5EREREr7savU/h9evXsXbtWjRu3BhyuRxnz56tsK+LiwtGjhyp1CaTyZRel5aWYty4cbh06RJCQ0NhYWGBLVu2ICgoCLt370ajRo3EvjExMZgwYQK8vb0xc+ZMXLp0CcuXL8eDBw8wc+ZMzZ4oERER0WuuRkOhi4sL4uPjYWFhgejoaEyYMKHCvra2tujbt2+lxzt06BDOnj2L5cuXw8/PDwDg7++PHj16ICwsDPPmzRP7zps3D82bN8e6deugq6sLADA2NsaaNWsQFBQEe3v7Vz9BIiIiolqiRqePTUxMYGFhoXb/wsJCPHnypMLthw8fho2NDbp27Sq2WVpawt/fH9HR0SgqKgIAXLlyBVeuXMGQIUPEQAgAgYGBKC0tRVRU1EucDREREVHtVWsWmsTGxsLNzQ1ubm7w8/PDtm3bVPqkpaXBxcUFEolEqb1FixZ49OgRbt68CQC4cOECAMDV1VWpX7169WBraytuJyIiInpb1IpnH8tkMrRp0wb29vZ48OABtm/fjm+++QY5OTkYN26c2C8zMxPe3t4q+9vY2AB4urDFwcEBmZmZAABra2uVvtbW1sjIyKhyjVZWJlXeh4jUZ21tWtMlEBG90WpFKFy1apXS6/79+yMwMBArVqzAsGHDYGr69IdFfn4+DAwMVPYva8vPz1f6b3l9pVJppVPUFcnKykNpqVDl/ej1ZGYuhbScr4/XVUFhIR7mFKjdvzYGrMzM3JougYio1tPRkVQ4kFUrQuHzdHV1MXLkSHz22Wc4e/YsOnToAAAwNDREYWGhSv+yNkNDQ6X/lte3oKBA3E5vL6mBAUI2fFLTZaht46glANQPhURERM+rNdcUPs/W1hYAkJOTI7ZVNPVb1lY2jVw2bVw2jfyszMxMsR8RERHR26LWhsL09HQAT1cXl3FycsL58+chCMrTuKmpqTAyMhLvU+js7AwAOHfunFK/e/fu4e7du+J2IiIiorfFax8Ks7OzUVpaqtRWUFCAdevWwdjYGG5ubmJ7z549kZGRgaNHj4ptCoUChw4dQteuXaGvrw8AcHR0RNOmTbFt2zaUlJSIfbdu3QodHR10795dy2dFRERE9Hqp8WsKV6xYAQC4evUqACAyMhKnT5+GmZkZRowYgWPHjmHVqlXo0aMH7OzskJ2djYiICNy4cQPffvstjI2NxWP16NEDbm5umDp1qvhEk61bt6K0tBSTJk1Set+pU6di/PjxGD16NHr16oVLly4hPDwcQ4YMQZMmTarvAyAiIiJ6DdR4KFyyZInS6127dgEA7OzsMGLECMhkMjRt2hSRkZFQKBQwMDCAi4sLpk+fjs6dOyvtq6urizVr1mDevHnYvHkzCgoK0KJFC8ydOxeNGzdW6tu5c2eEhYUhLCwMs2fPhqWlJcaPH4+PPvpIuydMRERE9BqSCM9fgEcvhbekebNYW5vWutXHVblli7W1KQ4Ej9JiRZrVa9MG3pKGiEgDKrslzWt/TSERERERaR9DIRERERExFBIRERERQyERERERgaGQiIiIiMBQSERERERgKCQiIiIiMBQSERERERgKiYiIiAgMhUREREQEhkIiIiIiAkMhEREREYGhkIiIiIjAUEhEREREYCgkIiIiIjAUEhEREREYComIiIgIDIVEREREBIZCIiIiIoIGQqFCocCNGzc0UAoRERER1RS1Q+GePXswc+ZMpbaFCxeiXbt28Pf3x9ChQ5GXl6fxAomIiIhI+9QOhb/++iuKi4vF13/99RfWrl2LNm3aYNCgQfjrr7+wceNGbdRIRERERFqmp27HmzdvomfPnuLrQ4cOwdzcHOvWrYOBgQEkEgkOHjyIiRMnaqVQIiIiItIetUcKc3NzYWpqKr6Oi4uDr68vDAwMAACurq64ffu25iskIiIiIq1TOxRaW1vjn3/+AfB0ccnff/+NNm3aiNsfP34MXV1dzVdIRERERFqn9vSxl5cXwsPDYW5ujoSEBEgkEnTs2FHcfv36ddSrV08rRRIRERGRdqkdCj/55BOcPXsW8+fPBwCMHz8eDRo0AAAUFxcjKioK3bt3106VRERERKRVaodCW1tb7N+/H1euXIGpqSnq168vbsvPz8esWbPg5ORUpTfPyMjApk2bkJKSgnPnzuHx48fYtGkTvLy8xD4PHjzArl27cOzYMVy7dg3FxcVwcHBASEgI/P39lY63e/duzJgxo9z3Sk1NhVQqVWo7evQowsLCcOXKFVhZWWHgwIH48MMPoaen9sdCRERE9EaoUvrR1dWFXC5XaTcxMYGfn1+V3/z69etYu3YtGjduDLlcjrNnz6r0+fPPP7F48WJ06NAB48ePh56eHg4fPoxPP/0U165dw4QJE1T2+eyzz/Duu+8qtenr6yu9jomJwYQJE+Dt7Y2ZM2fi0qVLWL58OR48eKByP0YiIiKiN12FofBlVxI/O4L4Ii4uLoiPj4eFhQWio6PLDXjNmjXD4cOHYWdnJ7YFBgYiJCQEa9aswejRo2FoaKi0T8eOHeHs7Fzpe8+bNw/NmzfHunXrxAUyxsbGWLNmDYKCgmBvb6/2eRARERHVdhWGwi5dukAikVT5gGlpaWr3NTExeWGfhg0bqrRJJBL4+fkhPj4e//77LxwcHFT65OXlwcjICDo6qgusr1y5gitXrmDWrFlKK6YDAwOxatUqREVFYdy4cWqfBxEREVFtV2EonDBhgkooPHbsGNLS0uDr64tmzZoBeBqw4uLi4OTkhC5dunLiBLAAACAASURBVGi32mfcv38fAGBhYaGyLTAwEI8fP4ZUKkWnTp0wffp0pRHMCxcuAHh6b8Vn1atXD7a2tuJ2IiIiordFhaFw0qRJSq/37duHW7duYffu3SpTs+fPn0dISEi1TblmZ2djx44d8PT0hKWlpdhep04d9O/fH15eXjA2NkZKSgp+/vlnpKSkICIiQuybmZkJ4Om9F59nbW2NjIyMKtdkZfXiUU8ibbK2Nn1xp1rsTT8/IqKapvZCkzVr1mD48OHlXqvn4uKCwMBArF69Gr1799Zogc8rLS3FF198gdzcXHz99ddK2/z9/ZVWJHfr1g1t27bFuHHj8PPPP+Ozzz4D8HS1NADxaSzPkkqlePLkSZXrysrKQ2mpUOX96PVUGwNIZmau2n3f9PMjIqLy6ehIKhzIUvuJJv/88w+srKwq3P7OO++ITzzRptmzZ+PUqVOYM2dOuSuhn9exY0c0bdoUcXFxYlvZwpTCwkKV/gUFBSoLV4iIiIjedFV6zF1UVBQEQXU0rLS0FFFRUXjnnXc0WtzzwsLCsGXLFkyZMqVKI5LvvvsucnJyxNdl08Zl08jPyszMhI2NzasXS0RERFSLqB0KBw8ejMTERIwePRonT55Eeno60tPTERMTg9GjRyM5ORlDhgzRWqHh4eFYtmwZQkJCMHr06Crtm56errQgpWwK/Ny5c0r97t27h7t3777wdjZEREREbxq1rykcN24c7t+/j19++UVpKrbM8OHD8cEHH2i0uDIHDhzAf//7X/Tp0wfTp0+vsJ9CoVBaeAI8XSBz8+ZN9O3bV2xzdHRE06ZNsW3bNgwcOFC8Lc3WrVuho6PDx/URERHRW0ftUCiRSPDVV18hMDAQR48eRXp6OoCn9xHs0qULmjZt+lIFrFixAgBw9epVAEBkZCROnz4NMzMzjBgxAqmpqZg6dSrq1q0LHx8f7N27V2n/du3aidPWQ4cOhYuLC5o3bw4TExOkpqZiz549sLe3x8iRI5X2mzp1KsaPH4/Ro0ejV69euHTpEsLDwzFkyBA0adLkpc6FiIiIqLZSKxQWFhYiJSUF1tbWaNKkCcaMGaOxApYsWaL0eteuXQAAOzs7jBgxAleuXEFRUREUCgW+/PJLlf03bdokhkJ/f3+cOHECv//+O/Lz82FjY4Phw4dj4sSJMDVVXm3ZuXNnhIWFISwsDLNnz4alpSXGjx+Pjz76SGPnRkRERFRbSITyVo48p7i4GK1atcK0adMQHBxcHXXVOrwlzZvF2toUIRs+qeky1LZx1JIq35LmQPAoLVakWb02beAtaYiINOCVb0mjp6eHd955p9yVx0RERERU+6m9+rhnz544ePAgSktLtVkPEREREdUAtReaDBo0CAkJCRg1ahRGjhyJxo0bo06dOir9nn3GMBERERHVDmqHwt69e0MikUAQBCQmJlbYLy0tTSOFEREREVH1UTsUTpgwARKJRJu1EBEREVENUTsUTpo0SZt1EBEREVENUnuhCRERERG9udQeKQSA0tJSRERE4MiRI7h16xYAoEGDBujevTsCAgKgo8OMSURERFQbqR0K8/PzMXbsWCQnJ0MikcDa2hoAcPLkScTExGDPnj1Yu3YtpFKp1oolIiIiIu1Qe2hv5cqVSEpKwqhRoxAXF4eYmBjExMQgPj4eoaGhSExMxMqVK7VZKxERERFpidqh8MCBA/D398fUqVNhbm4utpuZmWHKlCnw9/fH/v37tVIkEREREWmX2qHw7t278PT0rHB727ZtcffuXY0URURERETVS+1QaGZmhps3b1a4/ebNmzAzM9NIUURERERUvdQOhb6+vggPD8fvv/+usu3UqVPYunUr2rdvr9HiiIiIiKh6qL36+NNPP8WpU6cwbtw4ODs7w9HREQBw+fJlpKWlwcLCAh9//LHWCiUiIiIi7VE7FNrZ2WHXrl1YuHAhjh8/jgsXLgAAjI2N8f7772Py5MmoX7++1golIiIiIu2p0s2r69evj4ULF0IQBCgUCgCApaUln4lMREREVMupHQpLS0vFJ5ZIJBJYWVlprSgiIiIiql5qh0JPT0+0adMG3t7e8Pb2hpOTkzbrIiIiIqJqpHYobN++PZKSknDixAlIJBKYm5vD09NTDIlNmzbVZp1EREREpEVqh8LFixcDAC5duoT4+HjxT1RUlPgsZG9vb8ybN09rxRIRERGRdqh9n8IyMpkMwcHBWLFiBRISErBkyRI4ODggIyMD+/bt00aNRERERKRlVVp9DAAKhUIcJYyLi8OtW7ego6ODVq1awdvbWxs1EhEREZGWqR0Kv//+e8THx+Py5csAno4YdunSBT4+PmjTpg1MTEy0ViQRERERaZfaoXDTpk3Q1dXFf/7zH0ycOBENGzbUZl1EREREVI3UDoWDBw9GfHw8IiMjsX//fri6uoorjz08PGBgYKDNOomIiIhIi9QOhbNmzQIA3LlzB3FxcUhISEBERARWrVoFqVQKNzc3+Pj44MMPP9RasURERESkHVVeffzuu++if//+mDt3LmJiYrB48WI0aNBAXIlcFRkZGViwYAGCgoLg7u4OuVyOhISEcvsePXoU/fr1Q4sWLdCpUyeEhYWhuLhYpd/Dhw8xc+ZMeHt7w83NDcHBwUhLS3ulYxIRERG96aq8+vjmzZvi6uOEhAQoFAoIggBzc3N4eXlV6VjXr1/H2rVr0bhxY8jlcpw9e7bcfjExMZgwYQK8vb0xc+ZMXLp0CcuXL8eDBw8wc+ZMsV9paSnGjRuHS5cuITQ0FBYWFtiyZQuCgoKwe/duNGrUqMrHJCIiInobqB0KZ8yYgYSEBNy5cweCIMDY2FjpsXfOzs5VfnMXFxfEx8fDwsIC0dHRmDBhQrn95s2bh+bNm2PdunXQ1dUFABgbG2PNmjUICgqCvb09AODQoUM4e/Ysli9fDj8/PwCAv78/evTogbCwMKUba6t7TCIiIqK3gdrTxwcOHECjRo3wySef4Ndff0ViYiJWr16NUaNGvVQgBAATExNYWFhU2ufKlSu4cuUKhgwZIoY3AAgMDERpaSmioqLEtsOHD8PGxgZdu3YV2ywtLeHv74/o6GgUFRVV+ZhEREREbwO1RwqTkpJqZIXxhQsXAACurq5K7fXq1YOtra24HQDS0tLg4uICiUSi1LdFixbYtm0bbt68CQcHhyodk4iIiOhtoHYofD4Q5ufnAwAMDQ01W9FzMjMzAQDW1tYq26ytrZGRkaHUt7ynqtjY2AB4urDFwcGhSsdUl5UVb95NNcva2rSmS9CqN/38iIhqWpUWmmRlZWHZsmWIjo5GVlYWAMDKygp+fn6YOHEi3nnnHY0XWBY+yxullEqlePLkiVLf8vqVtZUdqyrHVFdWVh5KSwXxtamZIQyl+lU+Tk3KLyhC7sP8mi7jtVAbA0hmZq7afd/08yMiovLp6EgqHMhSOxSmp6cjMDAQmZmZaNKkCdzc3AAAV69exa+//oqjR49iy5YtGn/SSdlIZGFhocq2goICpZFKQ0PDcvuVtZX1rcoxX7puqT4Cp4a/8nGq05Z5w5ELhkIiIqK3kdqhcO7cucjOzkZYWJi4srfMkSNHMHnyZMydOxdhYWEaLbBsijczM1OcBi6TmZkJd3d3pb7lTf2WtZXtX5VjEhEREb0N1F59HBcXh+HDh6sEQgDo1q0bhg0bhri4OI0WB0Bc2Xzu3Dml9nv37uHu3btKK5+dnJxw/vx5CIKg1Dc1NRVGRkbifQqrckwiIiKit4HaoVAikaBx48YVbre3t1dZ9asJjo6OaNq0KbZt24aSkhKxfevWrdDR0UH37t3Ftp49eyIjIwNHjx4V2xQKBQ4dOoSuXbtCX1+/ysckIiIiehuoPX3ctm1bJCQkYNiwYeVuT0xMhKenZ5ULWLFiBYCn1yYCQGRkJE6fPg0zMzOMGDECADB16lSMHz8eo0ePRq9evXDp0iWEh4djyJAhaNKkiXisHj16wM3NDVOnThWfaLJ161aUlpZi0qRJSu+r7jGJiIiI3gYS4fm51gqkp6cjODgYPXr0wNixY2FlZQXg6YrkNWvW4MiRI9i0aRMaNGhQpQLkcnm57XZ2djh27Jj4Ojo6GmFhYbh69SosLS0xYMAAfPTRR9DTU861OTk5mDdvHqKjo1FQUIAWLVpg+vTpcHFxUXkPdY+pjudXH1tbm9bKhSZc4fmUtbUpQjZ8UtNlqG3jqCVVXn18IHiUFivSrF6bNvBrk4hIAypbfVxhKHz2qSBlHj9+jOzsbACAmZkZAODhw4cAgLp168LY2BjR0dEaKbq2YSh8szAUvl4YComINOOlbklTv359rRVERERERK+XCkPh5s2bq7MOIiIiIqpBaq8+JiIiIqI3F0MhEREREal/SxonJ6cX3odQIpHgwoULr1wUEREREVUvtUNhQECASigsLi5Geno6UlJSIJfL+SQQIiIiolpK7VD4ww8/VLjtzJkzGD9+PL799ltN1ERERERE1Uwj1xR6eHigf//+WLBggSYOR0RERETVTGMLTezt7XH+/HlNHY6IiIiIqpHGQmFiYiKkUqmmDkdERERE1Ujtawr37NlTbnt2djbi4uJw8uRJDBw4UGOFEREREVH1UTsUTp8+HRKJBOU9KllPTw8DBw7EjBkzNFocEREREVUPtUPhpk2bVNokEgnMzc3RoEEDGBkZabQwIiIiIqo+aodCT09PbdZBRERERDXolRaa3Lt3D6mpqXj48KGm6iEiIiKiGlBpKExLS8OGDRvw4MEDpXaFQoExY8agU6dOGDJkCHx9fREWFqbVQomIiIhIeyoNhVu3bsXPP/8MCwsLpfavv/4ap06dQoMGDdCtWzeYm5tj+fLliI6O1mqxRERERKQdlV5T+Oeff6JDhw5Kbf/++y+OHTsGJycnbN++HQYGBlAoFOjfvz+2b98OPz8/rRZMRERERJpX6UhhRkYG7O3tldri4+MBAIGBgTAwMAAAWFpa4j//+Q8uXLignSqJiIiISKsqDYWPHz+GqampUltqaiokEgm8vLyU2hs2bIjs7GzNV0hEREREWldpKLS1tcXNmzeV2s6ePQszMzM0btxYqb2kpATGxsaar5CIiIiItK7SUOjq6oo9e/YgIyMDwNNAeOnSJfj4+Kj0vXLlCmxsbLRTJRERERFpVaULTcaNG4fDhw/D398fTZo0wZUrV6Cjo4Pg4GCVvidOnFCZUiYiIiKi2qHSkUInJyeEhYWhfv36uHTpEho0aIBFixbBw8NDqd/vv/+OrKwslZXKRERERFQ7vPAxd507d0bnzp0r7fPee+/h7NmzGiuKiIiIiKrXKz3mjoiIiIjeDAyFRERERPTi6ePXwfTp0xEREVHh9pMnT6JevXoICgpCYmKiyvZevXph0aJFSm2FhYVYsmQJIiMj8fDhQzg5OeGzzz4rd2U1ERER0ZuuVoTCIUOGqIQ1QRDw7bffws7ODvXq1RPb69evj08//VSpr52dncoxp0+fjqioKAQHB6Nx48aIiIjA2LFjsXnzZri7u2vnRIiIiIheU7UiFLq7u6sEteTkZDx58gR9+vRRajczM0Pfvn0rPV5qair279+PGTNmICQkBAAQEBCA3r17Y8GCBQgPD9do/URERESvu1p7TeFvv/0GiUSC3r17q2wrLi7Go0ePKtz30KFD0NfXx6BBg8Q2qVSKgQMH4vTp0+LNuomIiIjeFrUyFBYVFeHgwYNwd3dHgwYNlLZdvXoVbm5u8PDwQPv27bFq1SqUlpYq9UlLS0OTJk1UHsvXsmVLCIKAtLQ0rZ8DERER0etE7enjvLw8fPTRR5g+fTqaN2+uzZpe6NSpU8jOzlaZOm7YsCG8vLwgl8uRl5eH3377DYsWLcLt27cxa9YssV9mZqbSdYhlrK2tAeClRgqtrEyqvM/ryNratKZLoJf0pv/dvennR0RU09QOhUVFRUhMTEROTg4A4PHjx5g9ezbGjBkDBwcHrRVYnt9++w36+vrw9/dXav/++++VXvfr1w+ffPIJtm/fjpCQEDRt2hQAkJ+fD319fZXjSqVSAEBBQUGVa8rKykNpqSC+rq0/wDIzc2u6hNdCbfz7q8rf3Zt+fkREVD4dHUmFA1mVTh9//PHH2LhxI1JSUlBYWKi0raCgAHv27Kn26+8ePXqEo0ePon379rCwsHhh/9DQUAiCgISEBLHN0NAQRUVFKn3LwmBZOCQiIiJ6W1Q6UvjkyRMsX74cubm50NPTg0QiwcGDB2FkZIQGDRpAEITKdteK6OjoclcdV8TW1hYAxBFO4Ok0cXlhNjMzEwBgY2OjgUqJiIiIao9KQ+HatWshCAIuXryI2NhYzJ8/H/v27cP27dthZGQEiUSCEydOwNzcHM7OzpBIJFoveN++fTAyMkKXLl3U6p+eng4AsLS0FNucnJywefNmPHr0SGmxSUpKiridiIiI6G3ywtXHEokETk5O6N+/PwBgxYoViIyMxNixYyEIAsLDwzFgwAB4enrigw8+0GqxCoUCcXFx6NatG+rUqaO0LS8vT2WKu6SkBKtXr4aOjo7Sza979uyJoqIi7NixQ2wrLCzE7t274eHhUe4iFCIiIqI3WaUjhaNHj0br1q3RunVrNGzYEMDTkCiXy2FtbY0lS5Zg9erVMDMzQ1JSEpKTk7Va7IEDB1BcXFzu1PH58+fx+eefo3fv3mjUqBEeP36MgwcP4ty5cxg7dqxYPwC0atUKPXv2xIIFC5CZmYlGjRohIiICt2/fxpw5c7R6DkRERESvo0pDoYGBATZv3oylS5dCV1cXEolEfAZx2UpeXV1dtGjRAi1atEBoaKhWi923bx+srKzg6+ursq1+/frw8PBAVFQU7t+/Dx0dHTg6OuKHH35Av379VPrPmzcPixcvRmRkJHJyciCXy7FmzRq0bt1aq+dARERE9DqqNBSuXLkSAHDjxg3ExsZi9uzZOH78OCIjIyGVSiGRSBAVFQVDQ0O4urpCT0+7T83btm1bhdsaNmyIpUuXqn0sqVSKadOmYdq0aZoojYiIiKhWUyvF2dvbw9zcHLNnz8aSJUtga2uLI0eO4Mcff0RERAS2bNmCOnXqoFWrVti4caOWSyYiqpi5WR0YSGvFY91FhQXFyHn4pKbLIKK33Ev9y9mkSRMMGjQIP/74I1asWAEbGxskJCRo/ZpCIqIXMZDq4fuvdtZ0GVXy5f8G1nQJRETqh0KpVIp+/fqVew8/BwcHODg4IDAwUKPFEREREVH1UDsUGhkZKa3MrSwkEhEREVHt8tIX3jwfEomIiIio9nrhzauJiIiI6M2ndih8/PgxZsyYgatXr2qzHiIiIiKqAWqHwoKCAuzZswcZGRkAgPz8fISFheHWrVtaK46IiIiIqkeloXDu3Lk4cuQI7t+/DwAQBEHc9uTJEyxfvhzp6enarZCIiIiItK7ShSYxMTHizajr1asHiUSC5ORkODo6QldXVykkEhEREVHtVelI4YEDB/DHH39g2bJl8PPzgyAIWL58Od577z0MGzYMEokE586dQ3Z2dnXVS0RERERa8MJrCi0sLODn54ePPvoIABAWFobly5fD09MTgiBg4cKF8PX1RZ8+fTBr1iytF0xEREREmlfp9PE333yDNm3awMPDA8bGxgAAY2Nj+Pj4wM3NDdu3b8eCBQugp6eHpKQkPuaOiIiIqJaqNBT++eef2LlzJwRBQN26dSGRSBAbG4sGDRrAxMQEAGBlZQUfHx/07NmzWgomIiIiIs2rNBTu3bsXeXl5SElJQXx8PNauXYt169Zh3bp1sLW1hUQiwdmzZ+Hg4MDH3RERERHVYi+8ptDExATt2rVDaGgoAGDlypVYu3YtOnfuDEEQsHTpUnTs2BHdu3fHV199pfWCiYiIiEjzqvzsY6lUCh8fHzRv3hxbtmxBWFgY9PX1kZiYyGsKiYiIiGoptUOhvr4+2rZtC3NzcwCARCIB8P8LTzp27KidComIiIhI69QOhSYmJti8ebP4+vmQSERERES1V5Wnj8s8HxKJiIiIqPZ64UITIiIiInrzMRQSEREREUMhERERETEUEhEREREYComIiIgIlYTCsLAwXLp0SXx9+/Zt5OfnV0tRRERERFS9Kg2FFy9eFF937doVR44cqZaiiIiIiKh6VRgKzczM8PDhQ/G1IAjVUlB5EhISIJfLy/1z9epVpb5nzpzBsGHD0KpVK7Rr1w7//e9/8eTJE5VjFhYWYv78+Wjfvj1atmyJwYMHIy4urrpOiYiIiOi1UuHNq52dnbFu3ToUFxeLTy1JTk5GSUlJpQcMCAjQbIXPGDlyJFxcXJTa6tWrJ/5/WloaQkJC0KxZM0yfPh13797F+vXrcevWLaxatUppv+nTpyMqKgrBwcFo3LgxIiIiMHbsWGzevBnu7u5aOwciIiKi11GFoXDGjBmYOHEi5syZA+Dps463bduGbdu2VXgwiUSi1VDo6ekJPz+/Crf/+OOPqFu3LjZv3gxjY2MAQIMGDfD1118jLi4OPj4+AIDU1FTs378fM2bMQEhICICnYbZ3795YsGABwsPDtXYObwoLcwPoGUhruowqKS4swIOcwpoug4iI6LVUYSh0cnLC4cOHkZ6ejszMTAQFBeHDDz+Er69vddanIi8vD4aGhtDT01Np/+OPPzB69GgxEAJA37598f333+PgwYNiKDx06BD09fUxaNAgsZ9UKsXAgQOxaNEiZGRkwMbGpnpOqJbSM5Di9LwxNV1GlbSe+hMAhkIiIqLyVPrsY11dXdjb28Pe3h5t27aFl5cXPD09q6s2FVOmTMHjx4+hp6cHLy8vTJs2DXK5HABw8eJFFBcXw9XVVWkfAwMDODs7Iy0tTWxLS0tDkyZNlMIjALRs2RKCICAtLY2hkIiIiN4qlYbCZ23evFmbdVRKX18fPXr0QIcOHWBhYYGLFy9i/fr1CAwMxM6dO9GkSRNkZmYCAKytrVX2t7a2xp9//im+zszMVLoW8dl+AJCRkVHlGq2sTKq8z+vI2tq0pkvQqjf5/N7kcwN4fkRE2qZ2KASA0tJSRERE4MiRI7h16xaAp9fsde/eHQEBAdDR0c69sD08PODh4SG+7tq1K7p06YIBAwYgLCwMCxcuFO+haGBgoLK/VCpVusdifn4+9PX1y+0HAAUFBVWuMSsrD6Wl/79Cu7b+A5+ZmatWP57f60fdcwPe7POrjecGVO3vj4joZenoSCocyFI7FObn52Ps2LFITk6GRCIRR9VOnjyJmJgY7NmzB2vXrhWDlbY5OTnBx8cH8fHxAABDQ0MAT28187yCggJxe1nfoqKicvsBqLZzICIiInpdqD20t3LlSiQlJWHUqFGIi4tDTEwMYmJiEB8fj9DQUCQmJmLlypXarFXFu+++i5ycHAD/P/VbNo38rMzMTKVrBK2trcudIi7bl9cTEhER0dtG7VB44MAB+Pv7Y+rUqeJ9C4GnN7meMmUK/P39sX//fq0UWZH09HRYWFgAAGQyGfT09HDu3DmlPoWFhUhLS4Ozs7PY5uTkhOvXr+PRo0dKfVNSUsTtRERERG8TtUPh3bt3K1153LZtW9y9e1cjRT1PoVCotCUnJyMhIQHt27cHAJiamsLHxweRkZFKYS8yMhKPHz9Gz549xbaePXuiqKgIO3bsENsKCwuxe/dueHh4lLsIhYiIiOhNpvY1hWZmZrh582aF22/evAkzMzONFPW8Tz/9FHXq1IG7uzssLCxw+fJlbNu2DRYWFpg0aZLY77PPPsPQoUMRFBSEQYMG4e7du9iwYQM6dOigdH/FVq1aoWfPnliwYAEyMzPRqFEjRERE4Pbt2+LNuomIiIjeJmqHQl9fX4SHh8PX1xfvvfee0rZTp05h69atSqNxmuTn54d9+/Zhw4YNyMvLg6WlJXr37o1Jkyahfv36Yj8XFxds2LABCxYswJw5c2BiYoLBgwdj8uTJKsecN28eFi9ejMjISOTk5EAul2PNmjVo3bq1Vs6BiIiI6HWmdij89NNPcerUKYwbNw7Ozs5wdHQEAFy+fBlpaWmwsLDAxx9/rJUig4ODERwcrFbfNm3a4Ndff31hP6lUimnTpmHatGmvWh4RERFRrad2KLSzs8OuXbuwcOFCHD9+HBcuXAAAGBsb4/3338fkyZOVRu2IiIiIqPao0s2r69evj4ULF0IQBHHxh6WlJSQSiVaKIyIiIqLqUaVQWEYikcDKykrTtRARERFRDdHOc+mIiIiIqFZhKCQiIiIihkIiIiIiYigkIiIiIjAUEhEREREYComIiIgIVQiFeXl5CA4OFm9aTURERERvDrVDYVFRERITE5GTkwMAePz4MWbMmIGrV69qrTgiIiIiqh6VhsKPP/4YGzduREpKCgoLC5W2FRQUYM+ePcjIyNBqgURERESkfZU+0eTJkydYvnw5cnNzoaenB4lEgoMHD8LIyAgNGjSAIAjVVScRERERaVGloXDt2rUQBAEXL15EbGws5s+fj3379mH79u0wMjKCRCLBiRMnYG5uDmdnZz4DmYiIiKiWeuE1hRKJBE5OTujfvz8AYMWKFYiMjMTYsWMhCALCw8MxYMAAeHp64oMPPtB6wURERESkeZWOFI4ePRqtW7dG69at0bBhQwBPQ6JcLoe1tTWWLFmC1atXw8zMDElJSUhOTq6WoomIiIhIsyoNhQYGBti8eTOWLl0KXV1dSCQSREREAACaNm0KANDV1UWLFi3QokULhIaGar9iIiIiItK4SkPhypUrAQA3btxAbGwsZs+ejePHjyMyMhJSqRQSiQRRUVEwNDSEq6sr9PQqPRwRERERvabUuk+hvb09evXqBQBYsmQJDh48iAkTJkAQBERERGDo0KFo27YtQkJCtFkrEREREWnJSz3mrkmTJhg0aBCApwtP9u/fjylTpsDS0lKjxRERERFR9VB7vlcqlaJfiUzvmAAAHopJREFUv36wsbFR2ebg4AAHBwcEBgZqtDgiIlJmbmYAA6m0psuoksKCAuQ8LHxxRyKqUWqHQiMjI8yZM0d8XVlIJCIi7TCQSvHjjNp1+6/Jc1YDYCgket299MqQ50MiEREREdVeL3VNIRERERG9WRgKiYiIiIihkIiIiIgYComIiIgIr7DQpDqlpqYiIiICCQkJuH37NurWrQt3d3d8+umnaNy4sdgvKCgIiYmJKvv36tULixYtUmorLCzEkiVLEBkZiYcPH8LJyQmfffYZfHx8tH4+RERERK+bWhEKf/rpJ5w5cwY9e/aEXC5HZmYmwsPDERAQgJ07d8LBwUHsW79+fXz66adK+9vZ2akcc/r06YiKikJwcDAaN26MiIgIjB07Fps3b4a7u7vWz4mIiIjodVIrQmFIyP+1d+9hUVV7H8C/IAMIAgKi6YiKJoNGIGBcknN8Q/SQecHQQMVLXEwzE+uomHUq6hwvoWJqSmqlpeIVAW9pqHVS0FIUL6hJYHAUGERgEIYB3O8fvszbNNSBGNgMfD/Pw/PI2pu9v2vmwfmx19prz0RsbCyMjY3VbaNHj8bYsWOxefNmLF++XN1uaWmJ8ePH/+HxMjMzcfjwYSxZskT9aL7AwECMGTMGsbGx2LFjR4v0g4iIiKit0os5he7u7hoFIfD4ecwDBw5Edna21v61tbV4+PDh7x7v2LFjkEgk6kf1AY8X4544cSIuXLiAoqIi3YUnIiIi0gN6URQ2RBAEFBcXw9raWqM9OzsbQ4YMgbu7O3x9fbFp0yY8evRIY5+srCw4ODjA3Nxco93FxQWCICArK6vF8xMRERG1JXoxfNyQ5ORkFBYWYsGCBeo2e3t7eHl5QSaToaKiAocOHcKaNWtw9+5dxMTEqPeTy+Xo0aOH1jHt7OwA4E9dKbS17fInetH22NlZiB2hRbXn/rXnvgHsn75r7/0jag/0sijMzs5GTEwMPDw8NOYP/utf/9LYb8KECZg/fz727NmDmTNnon///gAApVIJiUSidVyT/3vIfHV1dZMz3b9fgUePBPX3+vofoFyuaNR+7F/b09i+Ae27f/rYN4D9I6LWYWho8LsXsvRu+Fgul+OVV16BlZUV1q5dC0PDP+5CWFgYBEHAuXPn1G2mpqaoqanR2re+GKwvDomIiIg6Cr26UqhQKBAZGQmFQoFdu3aph3v/yBNPPAEAKCsrU7fZ2dk1OEQsl8sBAN27d9dRYiIiIiL9oDdXCqurqzF79mzk5uYiPj5ePRT83+Tl5QEAbGxs1G1OTk7IycnRukP58uXL6u1EREREHYleFIV1dXWIiorCpUuXsHbtWgwZMkRrn4qKCqhUKq2fi4+Ph6GhocaTSgICAlBTU4O9e/eq21QqFQ4cOAB3d/cGb0IhIiIias/0Yvh4+fLlOHnyJJ577jmUlpYiKSlJvc3c3Bz+/v64du0a3nzzTYwZMwZ9+vRBZWUljh49iqtXryIyMhL29vbqn3F1dUVAQABiY2Mhl8vRp08fJCYm4u7du1i2bJkYXSQiIiISlV4UhTdu3AAAnDp1CqdOndLYJpVK4e/vj169esHd3R3Hjx9HcXExDA0NMXDgQCxfvhwTJkzQOubKlSsRFxeHpKQklJWVQSaT4dNPP4WHh0er9ImIiIioLdGLovDLL7/8r/vY29vj448/bvQxTUxMsHjxYixevLg50YiIiIjaBb2YU0hERERELYtFIRERERGxKCQiIiIiFoVEREREBBaFRERERAQWhUREREQEFoVEREREBD1Zp5CIiDoGa6vOMDLWr4+mWlUtHpRViR2DqNn06zePiIjaNSNjI1z+5LTYMZrE9dX/ETsCkU5w+JiIiIiIWBQSEREREYtCIiIiIgKLQiIiIiICi0IiIiIiAotCIiIiIgKLQiIiIiICi0IiIiIiAotCIiIiIgKLQiIiIiICi0IiIiIiAotCIiIiIgKLQiIiIiICi0IiIiIiAotCIiIiIgJgJHYAIiKijsLKyhTGxhKxYzSaSlWDsjKl2DGolbAoJCIiaiXGxhKsWrVK7BiN9uabbwJgUdhRcPiYiIiIiDp2UahSqfDRRx/B19cXLi4ueOmll5CWliZ2LCIiIqJW16GLwujoaGzbtg3jxo3D0qVLYWhoiMjISGRkZIgdjYiIiKhVddiiMDMzE4cPH8bf//53LFq0CMHBwdi2bRt69uyJ2NhYseMRERERtaoOe6PJsWPHIJFIMGnSJHWbiYkJJk6ciDVr1qCoqAjdu3cXMSEREZF+se5qAiOJsdgxGq22RoUHpdVix2gzOmxRmJWVBQcHB5ibm2u0u7i4QBAEZGVlNakoNDQ00GrrZm3ewJ5tW0P9+D3GlrYtmKRlNKV/3brYtGAS3WtK3wCgczf9ev+a0j+rrmYtmKRlNKV/ll31670DmtY/iYVpCyZpGU16/ywtWzCJ7jWlb0YSY5xLjWvBNLrlNSIKhoYqsWO0qj96Pw0EQRBaMUubMWbMGPTo0QNbt27VaL99+zZeeOEFfPjhhxpXEYmIiIjasw47p1CpVEIi0V5A1MTEBABQXc3LyURERNRxdNii0NTUFDU1NVrt9cVgfXFIRERE1BF02KLQzs4ORUVFWu1yuRwAeJMJERERdSgdtih0cnJCTk4OHj58qNF++fJl9XYiIiKijqLDFoUBAQGoqanB3r171W0qlQoHDhyAu7s7evToIWI6IiIiotbVYZekcXV1RUBAAGJjYyGXy9GnTx8kJibi7t27WLZsmdjxiIiIiFpVh12SBnh8U0lcXBxSUlJQVlYGmUyGN954A88++6zY0YiIiIhaVYcuComIiIjosQ47p5CIiIiI/h+LQiIiIiLquDea6JOioiJs374dly9fxtWrV1FZWYnt27fDy8tL7GjNlpmZicTERJw7dw53795F165d4ebmhqioKPTt21fseM125coVbNq0CdevX8f9+/dhYWEBJycnzJ07F+7u7mLH07nNmzcjNjYWTk5OSEpKEjtOs507dw7Tp09vcNuRI0cwYMCAVk6ke5mZmVi/fj0yMjJQW1sLe3t7zJw5Ey+++KLY0ZolOjoaiYmJv7v9u+++0/tVJnJzcxEXF4eLFy+ivLwcvXr1QmBgIGbOnAljY2Ox4zXbpUuXsGbNGmRmZsLQ0BBeXl6Ijo5Gnz59xI7WJE35DE9NTcX69etx+/Zt2NraYuLEiZg9ezaMjFqnXGNRqAdycnKwefNm9O3bFzKZDBkZGWJH0pktW7bg4sWLCAgIgEwmg1wux44dOxAYGIh9+/bp/YduXl4e6urqMGnSJNjZ2UGhUCAlJQWhoaHYvHkzhg0bJnZEnZHL5di4cSPMzMzEjqJzM2bMwFNPPaXRpu8FBQB8++23mDt3Ljw9PTF//nwYGRkhNzcX9+7dEztaswUHB8PHx0ejTRAEvPfee5BKpXr//hUWFmLSpEmwsLBAaGgorKys8OOPP2LVqlX46aef8NFHH4kdsVkyMzMRGhoKqVSKefPm4dGjR9i5cyemTJmCgwcPolu3bmJHbLTGfobX/z56e3vjnXfewa1bt7BhwwY8ePAA77zzTuuEFajNUygUQklJiSAIgnDixAnB0dFRSE9PFzmVbly4cEGorq7WaMvJyRGcnZ2FxYsXi5SqZVVWVgrPPvusMGvWLLGj6NTixYuFadOmCaGhocK4cePEjqMT6enpgqOjo3DixAmxo+hceXm54OPjI3zwwQdiR2k1P/zwg+Do6Chs3LhR7CjNFh8fLzg6Ogq3bt3SaJ83b54wePBgQaVSiZRMN8LDwwVPT0+htLRU3VZYWCgMGTJE+PDDD0VM1nSN/QwfPXq0MGHCBKG2tlbdtnr1asHJyUnIyclplaycU6gHunTpAmtra7FjtAh3d3etYY5+/fph4MCByM7OFilVy+rcuTNsbGxQXl4udhSdyczMRHJyMpYsWSJ2lBZTUVGB2tpasWPoTEpKCsrLyzF//nwAj/sntPPFKA4dOgQDAwOMGTNG7CjNVv80LltbW432bt26wcjICJ06dRIjls5cvHgRvr6+sLKyUrd1794dnp6eOHr0qIjJmq4xn+G3b9/G7du3ERwcrPHeTZkyBY8ePcLx48dbOiYA3mhCbZAgCCguLm5XhXBFRQVKSkrw888/Y/Xq1bh165bW0Ja+EgQBH3zwAQIDAzFo0CCx47SIhQsXwsPDA66urggLC8PNmzfFjtRsaWlp6N+/P7799lsMHz4cHh4e8PT0RGxsLOrq6sSOp3M1NTU4evQo3Nzc0Lt3b7HjNNszzzwDAFi6dClu3LiBe/fuITk5GYmJiYiMjIShoX5/vKtUKpiYmGi1m5qaQi6Xo6ioSIRULef69esAAGdnZ432Hj164IknnlBvb2mcU0htTnJyMgoLC7FgwQKxo+jMW2+9ha+//hoAIJFIEBISgtmzZ4ucSjcOHjyI27dvY8OGDWJH0TmJRIK//e1v+Otf/wpra2vcvHkTn332GaZMmYJ9+/bBwcFB7Ih/2p07d1BQUIDo6GhERERg8ODBOHXqFDZv3ozq6mosXbpU7Ig69f3336O0tBRjx44VO4pO+Pr6Yv78+YiPj8fJkyfV7a+//jrmzp0rYjLdcHBwwKVLl/Do0SN1gatSqZCZmQng8c0b3bt3FzOiTsnlcgCAnZ2d1jY7O7tWK4JZFFKbkp2djZiYGHh4eGD8+PFix9GZuXPnIjg4GAUFBUhKSoJKpUJNTY3e3yFYUVGBVatWYdasWe3qP+h67u7uGneJjxgxAn5+fggKCsL69euxatUqEdM1T2VlJcrKyvDmm29i1qxZAIBRo0ahsrISu3btwpw5c2BjYyNySt05dOgQJBIJnn/+ebGj6Ezv3r3h6emJkSNHomvXrjh9+jTWrVsHGxsbTJ48Wex4zTJlyhS89957ePvttxEWFoZHjx5h48aN6uJJqVSKnFC36vvT0GeCiYkJqqqqWiUHi0JqM+RyOV555RVYWVlh7dq1ej/88WsymQwymQwAMG7cOAQFBWHJkiX4+OOPRU7WPBs3boREIsHLL78sdpRW4+TkBB8fH6Snp4sdpVlMTU0BQGt+3dixY3Hs2DFcuXIFw4cPFyOazj18+BCpqanw9fVtN9NSDh8+jHfffRfHjh1T30k9atQoCIKAlStXYvTo0Rrz8fTN5MmTUVBQgK1bt2L//v0AHg+thoeHY9OmTTA3Nxc5oW7V/z6qVCqtbdXV1ertLa39fOqSXlMoFIiMjIRCocCWLVsavITeXkgkEowYMQLHjx/X6792i4qKsG3bNkyZMgXFxcXIz89Hfn4+qqurUVNTg/z8fJSVlYkds0X07NlT7/tW/zv226U96r/X9/792jfffIOqqqp2M3QMADt37sRTTz2ltbSOn58fKisrcePGDZGS6c6CBQtw5swZ7NixA8nJydi/fz8EQYCBgQHs7e3FjqdT9b+P9VdCf00ul7faSAyLQhJddXU1Zs+ejdzcXMTHx6N///5iR2pxSqUSgiCo7yDUR/fv30dNTQ1iY2MxYsQI9dfly5eRnZ2NESNGYPPmzWLHbBF5eXl6f8Wpft3FwsJCjfaCggIAaFdDxykpKTAzM4Ofn5/YUXSmuLi4wRuCampqAKDd3CxkZWWFoUOHqkdazp49CxcXF3Tp0kXkZLpVf5Pe1atXNdoLCwtRUFDQajfxsSgkUdXV1SEqKgqXLl3C2rVrMWTIELEj6VRJSYlWW0VFBb7++mv07NlTazkJfdK7d29s2LBB62vgwIGQSqXYsGEDAgMDxY7ZLA29fz/++CPOnTsHX19fERLpTkBAAABg37596jZBELB3716YmZm1m9/FkpISpKWlYeTIkejcubPYcXTGwcEBV69exS+//KLRfvjwYXTq1EldRLUnR44cwZUrVzBjxgyxo+jcwIED0b9/f+zevVujoN+1axcMDQ0xatSoVsnBOYV64pNPPgEA9dp9SUlJuHDhAiwtLREaGipmtGZZvnw5Tp48ieeeew6lpaUaj0YzNzeHv7+/iOmaLyoqCiYmJnBzc4OdnR3u3buHAwcOoKCgAKtXrxY7XrNYWFg0+P5s27YNnTp10vv3Dnj8/nXu3Blubm6wtrbGTz/9hN27d8Pa2hrz5s0TO16zODs7IzAwEPHx8bh//z4GDx6Mb7/9Ft9//z0WLlzYbq7EHDlyBLW1te1q6BgAwsPD8d1332Hy5MmYOnUqrKyscPr0aXz33XcICQnR6z84gcdLJsXHx2PYsGHo2rUrLl26hMTERIwdOxYvvPCC2PGarDGf4YsWLcKcOXMQHh6O0aNH49atW9ixYweCg4NbbaUDA6G9r1baTvzeX31SqVRjOQJ9M23aNJw/f77BbfreN+DxVZikpCTcvn0b5eXlsLCwwJAhQxAWFgZPT0+x47WIadOmoby8vF08+3j79u1ISUnBL7/8goqKCtjY2MDX1xfz5s1Dr169xI7XbCqVCp988gkOHjyI4uJi9O7dGzNnzkRISIjY0XQmODgYeXl5+Pe//633Czr/VmZmJtatW4esrCyUlpZCKpUiKCgI4eHhet/X3NxcxMTE4Pr163j48CH69euHSZMmITQ0VC9vQmzsZ/g333yD9evXIzs7GzY2NggKCsKrr77aas8+ZlFIRERERJxTSEREREQsComIiIgILAqJiIiICCwKiYiIiAgsComIiIgILAqJiIiICCwKiYiIiAgsComoHYqOjm43j/k6cOAAZDIZzp07J3aUVuXn54dp06aJHYOoQ+Fj7ohINHl5efj000/xww8/4N69ezA2Nka3bt3g4uKCCRMmwNvbW+yIeiU6OhqJiYlIS0uDjY2N2HH+qy+++AKWlpZ48cUXxY5CRGBRSEQiuXLlCqZNmwYjIyMEBgbiySefhFKpxJ07d3DmzBmYm5uzKGzntm/fDqlUyqKQqI1gUUhEotiwYQOqqqqQlJQEJycnre1yuVyEVK2rrq4OKpUKnTt3FjsKERHnFBKROHJzc9G1a9cGC0IAsLOzU/87Pz8fMpkM69at09pv3bp1kMlkyM/P19pWUlKCRYsWwcvLC0OGDMGMGTNw7do1jX38/PwQGhqq0RYfHw+ZTIY5c+ZotH/00UeQyWQoLi7WOMf777+P4cOHw9nZGcOHD8f777+PBw8eaPxs/dzAs2fPYsOGDfD394eLiwuOHj2q3mfPnj0ICAiAs7MzRo4ciS+++AIt9Xj63NxcLFy4EL6+vnB2doafnx9WrFiByspKjf3q52cqFAq8++678PHxwdNPP42QkBBcvnxZ67gPHjzAkiVL4OXlBTc3N0yfPh3Xr1/HtGnT4Ofnp95PJpPhP//5D86fPw+ZTKb++u37mJ2djVmzZsHNzQ0eHh54/fXXO8QfDERi4JVCIhJFnz59kJOTg+PHj2PUqFEtco6IiAhYWVnhtddeQ3FxMb766iuEhoZi9+7dcHR0BAB4e3sjOTkZSqUSpqamAIC0tDQYGhrihx9+QF1dHTp16gQASE9Px5NPPolu3boBABQKBSZPnow7d+4gKCgIgwcPRlZWFnbt2oX09HTs3bsXXbp00ci0YsUK1NbW4qWXXoK5uTkcHBwAPJ5ft2zZMjg5OeGNN95AVVUVPvvsM9ja2ur8dbl69SpmzJgBS0tLBAcHo0ePHrhx4wa+/PJLZGRk4Msvv4REItH4mfDwcNjY2GDu3LkoLS3F559/jlmzZiE1NVXdR5VKhZdffhlZWVl48cUX8fTTT+PmzZt4+eWXYWVlpXG8lStXYtmyZbC2tsbs2bPV7b+eC1lYWIjp06fD398fixYtwo0bN7B7925UVFTgs88+0/nrQtTRsSgkIlHMmTMHZ8+exbx589CvXz+4u7vj6aefhpeXFwYMGKCTc/Tq1Qvr1q2DgYEBAGDkyJGYOHEiVqxYga1btwJ4XBTu378fFy5cwLBhw6BSqZCRkYExY8YgOTkZ165dg4uLCxQKBbKysjB58mT18bds2YLc3Fz84x//wNSpU9XtgwYNQkxMDLZs2YKoqCiNTEqlEgcPHtQYMi4vL0dcXBwGDBiAhIQE9bagoCA8//zzOnktfu2tt96CnZ0d9u3bp1G0+vj44LXXXkNKSorWPL/BgwfjvffeU38/YMAAREVF4dChQwgJCQEA7N27F1lZWYiKitK4yuro6IiYmBhIpVJ12/jx47F27Vp069YN48ePbzDnnTt3sGbNGowePVrdZmhoiJ07d+Lnn39G//79m/U6EJEmDh8TkSjc3Nywf/9+TJgwAQqFAgcOHMD777+P0aNHY+rUqcjLy2v2OSIiItQFIQA4Oztj2LBhSEtLw8OHDwFAfTNLeno6ACAjIwNKpRIRERGwsLBAWloaAOD8+fOoq6vTuPnlxIkTsLGxQXBwsMZ5g4ODYWNjg2+++UYr0+TJk7XmEH7//feoqqrC1KlTNbY98cQTGDt2bHNeAi03b97EzZs3MWbMGKhUKpSUlKi/PDw8YGZmhjNnzmj93MyZMzW+r38d7ty5o247deoUOnXqhOnTp2vsO2nSJFhYWDQ5a/fu3TUKwt87LxHpBotCIhKNTCbD8uXLcfbsWZw8eRIrVqzA0KFD8eOPP+LVV1+FSqVq1vEbuuI4YMAA1NXV4e7duwAeFx79+/dXF4Xp6emws7ODTCbDM888o9FuaGgIT09P9bHy8/Ph4OAAIyPNQRcjIyP069evwcK2frj41+rn0TV05UtXV03rZWdnA3g8F9PHx0frq7KyUmPOZD17e3uN762trQEApaWl6rb8/Hx0794d5ubmGvsaGxujd+/eTc7623MCQNeuXbXOS0S6weFjImoTpFIppFIpxo8fjylTpuDixYvIzMzE0KFDNa72/VZtbW2zz+3t7Y3du3dDoVAgPT0dXl5e6vbVq1dDpVIhPT0dgwYN0pob11T18xbFFhYWhr/85S8NbrO0tNRqq59X+VstdSPMH52zpc9L1FHxSiERtSkGBgZwdXUFABQVFQGAuhArKyvT2r+hu47r1V8V+21bp06d0KtXL3Wbt7c36urqcPr0aVy5ckU9ROnj4wOlUonU1FT89NNP6mKxnr29PXJycrQK09raWuTm5jZ4pash9VfRfv7550b1oTn69u0L4PHcvGeffbbBL2dn5z91bKlUiqKiIvXQfL2ampo/fJ+IqG1gUUhEojhz5kyDV/mUSqV6Tlv90GmXLl1gZ2eH9PR0jStEeXl5Dc7bq7dlyxaN/a9du4azZ8/Cx8dHY4jTy8sLBgYG2LhxI2pqatRFoaOjI2xtbbF+/XoIgqC1mLa/vz9KSkqwd+9ejfY9e/agpKQE/v7+jXothg0bBlNTU+zYsQNVVVXq9oKCAqSkpDTqGI01ePBgODo6IiEhocHh7dra2j89NOvn54e6ujps375do33Pnj1QKBRa+5ubm3MYmKgN4fAxEYli2bJlKC0thZ+fHxwdHWFqaqougnJzcxEYGKjx/OKpU6ciLi4OERER8Pf3R1FRERISEjBw4EBcuXKlwXPcvXsX4eHh8PPzg1wux1dffQVTU1MsXLhQY7/69RKzsrIglUo1rvB5eXnhyJEjkEgkGDp0qMbPRURE4NixY4iJicH169cxaNAgZGVlYd++fXBwcEBERESjXgsrKyvMnz8fK1asQEhICAIDA1FVVYWEhAT069cP169fb+zLCuDx8jYNDVN7e3vD3d0dK1euxIwZMzBu3DgEBQVpPE3mxIkTeOONN/7UU0YmTZqEhIQExMXF4ZdfflEvSXPs2DH07dtX648AV1dX7Nu3T33ntaGhIZ577jmYmZk1+dxE1HwsColIFNHR0UhNTcWFCxfw9ddfQ6FQwMLCAo6OjoiMjNQqSiIjI6FQKJCcnIzz58/jySefxD//+U9cu3btd4vCLVu2YNmyZVi3bh2USiVcXV2xaNGiBhfM9vb2RlZWltbVQB8fHxw5cgTOzs5aN1BYWFhg165d+Pjjj3Hy5EkcOHAAtra2CAkJwbx587TWKPwjYWFhMDMzw+eff45Vq1ahZ8+eCAsLg4WFBd56661GHwd4vPh2Q4yMjODu7o5BgwYhMTER8fHxOHnyJBISEmBubg6pVIoJEybAx8enSeerZ2xsjG3btmHlypVITU3F0aNH4eLigi+++AJLly6FUqnU2H/BggUoKyvDzp07UV5eDkEQkJqayqKQSCQGAmfrEhFRC6pfysfFxUW9PiQRtT2cU0hERDrz26uBAJCQkIDy8nIMGzZMhERE1FgcPiYiIp15++23oVKp4ObmBmNjY2RkZODQoUPo27cvXnrpJbHjEdEf4PAxERHpzMGDB7Fjxw7k5uaisrIStra2GD58OObPn69+ZjQRtU0sComIiIiIcwqJiIiIiEUhEREREYFFIRERERGBRSERERERgUUhEREREYFFIREREREB+F8m3qO0ImQVEwAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": { + "tags": [] + }, + "output_type": "display_data" + } + ], + "source": [ + "sns.countplot(subword_lengths)\n", + "plt.title('Subword Token Lengths (w/o \"##\")')\n", + "plt.xlabel('Subword Length')\n", + "plt.ylabel('# of ## Subwords')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "gCDYsO8qyepC" + }, + "source": [ + "### Misspellings" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "p-n_ObF14sh3", + "outputId": "f83abeb9-3a71-4ff0-84fb-b90b2edaf115", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 12, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'misspelled' in tokenizer.vocab # Right" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "FnUu6C_VjTVQ", + "outputId": "b4328880-6532-46a8-d234-7b02c400312c", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 13, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'mispelled' in tokenizer.vocab # Wrong" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "GrKICg1y45Rz", + "outputId": "21fa6373-1df7-4b5d-b166-3c4e9e3040bc", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 14, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'government' in tokenizer.vocab # Right" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "fO72IlrU5I8t", + "outputId": "c8f216b2-b2a4-4cd2-827a-bb4f23703a82", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 15, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'goverment' in tokenizer.vocab # Wrong" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "FfqU2qbpjubX", + "outputId": "bd40e7bd-d43f-4e5c-b366-3d892cd852af", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 16, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'beginning' in tokenizer.vocab # Right" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "Cvzy-sMsjs5P", + "outputId": "d95c6956-4c8e-4104-9c89-caec26157470", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 17, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'begining' in tokenizer.vocab # Wrong" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "WkqxG-675A1c", + "outputId": "81d5f64c-613b-4ec3-94b9-45abb67c7d0f", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 18, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'separate' in tokenizer.vocab # Right" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "ziFBjur15GHg", + "outputId": "1fb3d72c-daa1-40a4-bfeb-c59dad2d9744", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 19, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'seperate' in tokenizer.vocab # Wrong" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-KZHPgVsSgvz" + }, + "source": [ + "What about contractions?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "VvUru5j3jxm2", + "outputId": "4ace3552-22ba-4445-c102-5f5a9642240e", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 20, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "\"can't\" in tokenizer.vocab" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "wl_l2xKwj0_l", + "outputId": "31f882e6-a524-4801-ea2e-aa9a6cc7ba0d", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 21, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "\"cant\" in tokenizer.vocab" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "RE4TqKOlykJD" + }, + "source": [ + "### Start vs. Mid Subwords\n", + "\n", + "For single characters, there are both the individual character and the '##' version for every character. Is the same true of subwords?" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "_Xf9FH7HWZya", + "outputId": "6343ff38-d934-4756-9858-582b8cebadd4", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Did not find a token for ly\n" + ] + } + ], + "source": [ + "# For each token in the vocabulary...\n", + "for token in tokenizer.vocab.keys():\n", + " \n", + " # If it's a subword...\n", + " if len(token) >= 2 and token[0:2] == '##':\n", + " if not token[2:] in tokenizer.vocab:\n", + " print('Did not find a token for', token[2:])\n", + " break" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "0rPGb9z8SmYn", + "outputId": "ee8b4c03-11ae-484a-cd6d-45b518a07939", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "True" + ] + }, + "execution_count": 23, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'##ly' in tokenizer.vocab" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "wUmQwKqfWx9K", + "outputId": "83048373-aba1-4725-f924-891887483543", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "data": { + "text/plain": [ + "False" + ] + }, + "execution_count": 24, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "'ly' in tokenizer.vocab" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "PRaaspCfyqsY" + }, + "source": [ + "### Names\n", + "\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 168 + }, + "id": "s7OLeBs_zBiR", + "outputId": "3bbf6ae5-0e91-4b62-b361-9832b8cec6f1", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Collecting wget\n", + " Downloading https://files.pythonhosted.org/packages/47/6a/62e288da7bcda82b935ff0c6cfe542970f04e29c756b0e147251b2fb251f/wget-3.2.zip\n", + "Building wheels for collected packages: wget\n", + " Building wheel for wget (setup.py) ... \u001b[?25l\u001b[?25hdone\n", + " Created wheel for wget: filename=wget-3.2-cp36-none-any.whl size=9681 sha256=97fcba8d47d5f818c5e3dc4b5c355474c4337b758065355989ee2b40ca73ed15\n", + " Stored in directory: /root/.cache/pip/wheels/40/15/30/7d8f7cea2902b4db79e3fea550d7d7b85ecb27ef992b618f3f\n", + "Successfully built wget\n", + "Installing collected packages: wget\n", + "Successfully installed wget-3.2\n" + ] + } + ], + "source": [ + "!pip install wget" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 50 + }, + "id": "9Mac6NiYys4g", + "outputId": "941f034e-9ed9-467a-a549-9cd8b94fe01b", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Beginning file download with wget module\n" + ] + }, + { + "data": { + "text/plain": [ + "'first-names.txt'" + ] + }, + "execution_count": 26, + "metadata": { + "tags": [] + }, + "output_type": "execute_result" + } + ], + "source": [ + "import wget\n", + "import random \n", + "\n", + "print('Beginning file download with wget module')\n", + "\n", + "url = 'http://www.gutenberg.org/files/3201/files/NAMES.TXT'\n", + "wget.download(url, 'first-names.txt')\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 50 + }, + "id": "FHF1JgrqY7Zi", + "outputId": "5cea3afc-b61a-437b-dd26-dcb4d30b7e3e", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Number of names: 21,985\n", + "Example: umberto\n" + ] + } + ], + "source": [ + "# Read them in.\n", + "with open('first-names.txt', 'rb') as f:\n", + " names_encoded = f.readlines()\n", + "\n", + "names = []\n", + "\n", + "# Decode the names, convert to lowercase, and strip newlines.\n", + "for name in names_encoded:\n", + " try:\n", + " names.append(name.rstrip().lower().decode('utf-8'))\n", + " except:\n", + " continue\n", + "\n", + "print('Number of names: {:,}'.format(len(names)))\n", + "print('Example:', random.choice(names))\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "6heL05KQzmCa", + "outputId": "922c4cf9-bbb6-43a3-9b32-f24e4c8a998f", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "3,869 names in the vocabulary\n" + ] + } + ], + "source": [ + "num_names = 0\n", + "\n", + "# For each name in our list...\n", + "for name in names:\n", + "\n", + " # If it's in the vocab...\n", + " if name in tokenizer.vocab:\n", + " # Tally it.\n", + " num_names += 1\n", + "\n", + "print('{:,} names in the vocabulary'.format(num_names))" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "BM99iB36RAxX" + }, + "source": [ + "**Further Research**\n", + "* Add more modern names\n", + " * This repo / file contains some more modern names. The file download isn't working, though.\n", + " * `https://raw.githubusercontent.com/arineng/arincli/master/lib/male-first-names.txt`\n", + "* Add common names from other languages.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "3HGvwsy0ynkq" + }, + "source": [ + "### Numbers" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "h5T_-gfQZKm6", + "outputId": "cfa1eb34-095e-4cdf-e4ea-006e62faa29b", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Vocab includes 881 numbers.\n" + ] + } + ], + "source": [ + "# Count how many numbers are in the vocabulary.\n", + "count = 0\n", + "\n", + "# For each token in the vocabulary...\n", + "for token in tokenizer.vocab.keys():\n", + "\n", + " # Tally if it's a number.\n", + " if token.isdigit():\n", + " count += 1\n", + " \n", + " # Any numbers >= 10,000?\n", + " if len(token) > 4:\n", + " print(token)\n", + "\n", + "print('Vocab includes {:,} numbers.'.format(count))" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 34 + }, + "id": "-M1biDEVYjaL", + "outputId": "80b38422-6d78-404c-deed-887c6efa4cee", + "vscode": { + "languageId": "python" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Vocab includes 384 of 421 dates from 1600 - 2021\n" + ] + } + ], + "source": [ + "# Count how many dates between 1600 and 2021 are included.\n", + "count = 0 \n", + "for i in range(1600, 2021):\n", + " if str(i) in tokenizer.vocab:\n", + " count += 1\n", + "\n", + "print('Vocab includes {:,} of 421 dates from 1600 - 2021'.format(count))" + ] + } + ], + "metadata": { + "colab": { + "provenance": [], + "toc_visible": true + }, + "kernelspec": { + "display_name": "Python 3", + "name": "python3" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/notebooks/More_SOTA_Deep_Learning_Models.ipynb b/notebooks/ref_only/More_SOTA_Deep_Learning_Models.ipynb similarity index 100% rename from notebooks/More_SOTA_Deep_Learning_Models.ipynb rename to notebooks/ref_only/More_SOTA_Deep_Learning_Models.ipynb diff --git a/notebooks/NLP_intro.ipynb b/notebooks/ref_only/NLP_intro.ipynb similarity index 100% rename from notebooks/NLP_intro.ipynb rename to notebooks/ref_only/NLP_intro.ipynb diff --git a/notebooks/Poetry_w_BERT.ipynb b/notebooks/ref_only/Poetry_w_BERT.ipynb similarity index 100% rename from notebooks/Poetry_w_BERT.ipynb rename to notebooks/ref_only/Poetry_w_BERT.ipynb diff --git a/notebooks/Poetry_w_BERT_gpt_vs_human.ipynb b/notebooks/ref_only/Poetry_w_BERT_gpt_vs_human.ipynb similarity index 100% rename from notebooks/Poetry_w_BERT_gpt_vs_human.ipynb rename to notebooks/ref_only/Poetry_w_BERT_gpt_vs_human.ipynb diff --git a/notebooks/chatgpt-haikus.ipynb b/notebooks/ref_only/chatgpt-haikus.ipynb similarity index 100% rename from notebooks/chatgpt-haikus.ipynb rename to notebooks/ref_only/chatgpt-haikus.ipynb diff --git a/notebooks/clean_reddit_haiku_dataset.ipynb b/notebooks/ref_only/clean_reddit_haiku_dataset.ipynb similarity index 100% rename from notebooks/clean_reddit_haiku_dataset.ipynb rename to notebooks/ref_only/clean_reddit_haiku_dataset.ipynb diff --git a/notebooks/combine_haiku_datasets.ipynb b/notebooks/ref_only/combine_haiku_datasets.ipynb similarity index 100% rename from notebooks/combine_haiku_datasets.ipynb rename to notebooks/ref_only/combine_haiku_datasets.ipynb diff --git a/notebooks/fine_tune_distilbert_glue_text_classification.ipynb b/notebooks/ref_only/fine_tune_distilbert_glue_text_classification.ipynb similarity index 100% rename from notebooks/fine_tune_distilbert_glue_text_classification.ipynb rename to notebooks/ref_only/fine_tune_distilbert_glue_text_classification.ipynb diff --git a/notebooks/fine_tuning_bert.ipynb b/notebooks/ref_only/fine_tuning_bert.ipynb similarity index 100% rename from notebooks/fine_tuning_bert.ipynb rename to notebooks/ref_only/fine_tuning_bert.ipynb diff --git a/notebooks/hugging_face_poetry_datasets.ipynb b/notebooks/ref_only/hugging_face_poetry_datasets.ipynb similarity index 100% rename from notebooks/hugging_face_poetry_datasets.ipynb rename to notebooks/ref_only/hugging_face_poetry_datasets.ipynb diff --git a/notebooks/notes.md b/notebooks/ref_only/notes.md similarity index 100% rename from notebooks/notes.md rename to notebooks/ref_only/notes.md diff --git a/notebooks/poetry_setfit.ipynb b/notebooks/ref_only/poetry_setfit.ipynb similarity index 100% rename from notebooks/poetry_setfit.ipynb rename to notebooks/ref_only/poetry_setfit.ipynb diff --git a/notebooks/reddit_haiku_dataset.ipynb b/notebooks/ref_only/reddit_haiku_dataset.ipynb similarity index 100% rename from notebooks/reddit_haiku_dataset.ipynb rename to notebooks/ref_only/reddit_haiku_dataset.ipynb diff --git a/notebooks/vocabulary.txt b/notebooks/vocabulary.txt new file mode 100644 index 0000000..fb14027 --- /dev/null +++ b/notebooks/vocabulary.txt @@ -0,0 +1,30522 @@ +[PAD] +[unused0] +[unused1] +[unused2] +[unused3] +[unused4] +[unused5] +[unused6] +[unused7] +[unused8] +[unused9] +[unused10] +[unused11] +[unused12] +[unused13] +[unused14] +[unused15] +[unused16] +[unused17] +[unused18] +[unused19] +[unused20] +[unused21] +[unused22] +[unused23] +[unused24] +[unused25] +[unused26] +[unused27] +[unused28] +[unused29] +[unused30] +[unused31] +[unused32] +[unused33] +[unused34] +[unused35] +[unused36] +[unused37] +[unused38] +[unused39] +[unused40] +[unused41] +[unused42] +[unused43] +[unused44] +[unused45] +[unused46] +[unused47] +[unused48] +[unused49] +[unused50] +[unused51] +[unused52] +[unused53] +[unused54] +[unused55] +[unused56] +[unused57] +[unused58] +[unused59] +[unused60] +[unused61] +[unused62] +[unused63] +[unused64] +[unused65] +[unused66] +[unused67] +[unused68] +[unused69] +[unused70] +[unused71] +[unused72] +[unused73] +[unused74] +[unused75] +[unused76] +[unused77] +[unused78] +[unused79] +[unused80] +[unused81] +[unused82] +[unused83] +[unused84] +[unused85] +[unused86] +[unused87] +[unused88] +[unused89] +[unused90] +[unused91] +[unused92] +[unused93] +[unused94] +[unused95] +[unused96] +[unused97] +[unused98] +[UNK] +[CLS] +[SEP] +[MASK] +[unused99] +[unused100] +[unused101] +[unused102] +[unused103] +[unused104] +[unused105] +[unused106] +[unused107] +[unused108] +[unused109] +[unused110] +[unused111] +[unused112] +[unused113] +[unused114] +[unused115] +[unused116] +[unused117] +[unused118] +[unused119] +[unused120] +[unused121] +[unused122] +[unused123] +[unused124] +[unused125] +[unused126] +[unused127] +[unused128] +[unused129] +[unused130] +[unused131] +[unused132] +[unused133] +[unused134] +[unused135] +[unused136] +[unused137] +[unused138] +[unused139] +[unused140] +[unused141] +[unused142] +[unused143] +[unused144] +[unused145] +[unused146] +[unused147] +[unused148] +[unused149] +[unused150] +[unused151] +[unused152] +[unused153] +[unused154] +[unused155] +[unused156] +[unused157] +[unused158] +[unused159] +[unused160] +[unused161] +[unused162] +[unused163] +[unused164] +[unused165] +[unused166] +[unused167] +[unused168] +[unused169] +[unused170] +[unused171] +[unused172] +[unused173] +[unused174] +[unused175] +[unused176] +[unused177] +[unused178] +[unused179] +[unused180] +[unused181] +[unused182] +[unused183] +[unused184] +[unused185] +[unused186] +[unused187] +[unused188] +[unused189] +[unused190] +[unused191] +[unused192] +[unused193] +[unused194] +[unused195] +[unused196] +[unused197] +[unused198] +[unused199] +[unused200] +[unused201] +[unused202] +[unused203] +[unused204] +[unused205] +[unused206] +[unused207] +[unused208] +[unused209] +[unused210] +[unused211] +[unused212] +[unused213] +[unused214] +[unused215] +[unused216] +[unused217] +[unused218] +[unused219] +[unused220] +[unused221] +[unused222] +[unused223] +[unused224] +[unused225] +[unused226] +[unused227] +[unused228] +[unused229] +[unused230] +[unused231] +[unused232] +[unused233] +[unused234] +[unused235] +[unused236] +[unused237] +[unused238] +[unused239] +[unused240] +[unused241] +[unused242] +[unused243] +[unused244] +[unused245] +[unused246] +[unused247] +[unused248] +[unused249] +[unused250] +[unused251] +[unused252] +[unused253] +[unused254] +[unused255] +[unused256] +[unused257] +[unused258] +[unused259] +[unused260] +[unused261] +[unused262] +[unused263] +[unused264] +[unused265] +[unused266] +[unused267] +[unused268] +[unused269] +[unused270] +[unused271] +[unused272] +[unused273] +[unused274] +[unused275] +[unused276] +[unused277] +[unused278] +[unused279] +[unused280] +[unused281] +[unused282] +[unused283] +[unused284] +[unused285] +[unused286] +[unused287] +[unused288] +[unused289] +[unused290] +[unused291] +[unused292] +[unused293] +[unused294] +[unused295] +[unused296] +[unused297] +[unused298] +[unused299] +[unused300] +[unused301] +[unused302] +[unused303] +[unused304] +[unused305] +[unused306] +[unused307] +[unused308] +[unused309] +[unused310] +[unused311] +[unused312] +[unused313] +[unused314] +[unused315] +[unused316] +[unused317] +[unused318] +[unused319] +[unused320] +[unused321] +[unused322] +[unused323] +[unused324] +[unused325] +[unused326] +[unused327] +[unused328] +[unused329] +[unused330] +[unused331] +[unused332] +[unused333] +[unused334] +[unused335] +[unused336] +[unused337] +[unused338] +[unused339] +[unused340] +[unused341] +[unused342] +[unused343] +[unused344] +[unused345] +[unused346] +[unused347] +[unused348] +[unused349] +[unused350] +[unused351] +[unused352] +[unused353] +[unused354] +[unused355] +[unused356] +[unused357] +[unused358] +[unused359] +[unused360] +[unused361] +[unused362] +[unused363] +[unused364] +[unused365] +[unused366] +[unused367] +[unused368] +[unused369] +[unused370] +[unused371] +[unused372] +[unused373] +[unused374] +[unused375] +[unused376] +[unused377] +[unused378] +[unused379] +[unused380] +[unused381] +[unused382] +[unused383] +[unused384] +[unused385] +[unused386] +[unused387] +[unused388] +[unused389] +[unused390] +[unused391] +[unused392] +[unused393] +[unused394] +[unused395] +[unused396] +[unused397] +[unused398] +[unused399] +[unused400] +[unused401] +[unused402] +[unused403] +[unused404] +[unused405] +[unused406] +[unused407] +[unused408] +[unused409] +[unused410] +[unused411] +[unused412] +[unused413] +[unused414] +[unused415] +[unused416] +[unused417] +[unused418] +[unused419] +[unused420] +[unused421] +[unused422] +[unused423] +[unused424] +[unused425] +[unused426] +[unused427] +[unused428] +[unused429] +[unused430] +[unused431] +[unused432] +[unused433] +[unused434] +[unused435] +[unused436] +[unused437] +[unused438] +[unused439] +[unused440] +[unused441] +[unused442] +[unused443] +[unused444] +[unused445] +[unused446] +[unused447] +[unused448] +[unused449] +[unused450] +[unused451] +[unused452] +[unused453] +[unused454] +[unused455] +[unused456] +[unused457] +[unused458] +[unused459] +[unused460] +[unused461] +[unused462] +[unused463] +[unused464] +[unused465] +[unused466] +[unused467] +[unused468] +[unused469] +[unused470] +[unused471] +[unused472] +[unused473] +[unused474] +[unused475] +[unused476] +[unused477] +[unused478] +[unused479] +[unused480] +[unused481] +[unused482] +[unused483] +[unused484] +[unused485] +[unused486] +[unused487] +[unused488] +[unused489] +[unused490] +[unused491] +[unused492] +[unused493] +[unused494] +[unused495] +[unused496] +[unused497] +[unused498] +[unused499] +[unused500] +[unused501] +[unused502] +[unused503] +[unused504] +[unused505] +[unused506] +[unused507] +[unused508] +[unused509] +[unused510] +[unused511] +[unused512] +[unused513] +[unused514] +[unused515] +[unused516] +[unused517] +[unused518] +[unused519] +[unused520] +[unused521] +[unused522] +[unused523] +[unused524] +[unused525] +[unused526] +[unused527] +[unused528] +[unused529] +[unused530] +[unused531] +[unused532] +[unused533] +[unused534] +[unused535] +[unused536] +[unused537] +[unused538] +[unused539] +[unused540] +[unused541] +[unused542] +[unused543] +[unused544] +[unused545] +[unused546] +[unused547] +[unused548] +[unused549] +[unused550] +[unused551] +[unused552] +[unused553] +[unused554] +[unused555] +[unused556] +[unused557] +[unused558] +[unused559] +[unused560] +[unused561] +[unused562] +[unused563] +[unused564] +[unused565] +[unused566] +[unused567] +[unused568] +[unused569] +[unused570] +[unused571] +[unused572] +[unused573] +[unused574] +[unused575] +[unused576] +[unused577] +[unused578] +[unused579] +[unused580] +[unused581] +[unused582] +[unused583] +[unused584] +[unused585] +[unused586] +[unused587] +[unused588] +[unused589] +[unused590] +[unused591] +[unused592] +[unused593] +[unused594] +[unused595] +[unused596] +[unused597] +[unused598] +[unused599] +[unused600] +[unused601] +[unused602] +[unused603] +[unused604] +[unused605] +[unused606] +[unused607] +[unused608] +[unused609] +[unused610] +[unused611] +[unused612] +[unused613] +[unused614] +[unused615] +[unused616] +[unused617] +[unused618] +[unused619] +[unused620] +[unused621] +[unused622] +[unused623] +[unused624] +[unused625] +[unused626] +[unused627] +[unused628] +[unused629] +[unused630] +[unused631] +[unused632] +[unused633] +[unused634] +[unused635] +[unused636] +[unused637] +[unused638] +[unused639] +[unused640] +[unused641] +[unused642] +[unused643] +[unused644] +[unused645] +[unused646] +[unused647] +[unused648] +[unused649] +[unused650] +[unused651] +[unused652] +[unused653] +[unused654] +[unused655] +[unused656] +[unused657] +[unused658] +[unused659] +[unused660] +[unused661] +[unused662] +[unused663] +[unused664] +[unused665] +[unused666] +[unused667] +[unused668] +[unused669] +[unused670] +[unused671] +[unused672] +[unused673] +[unused674] +[unused675] +[unused676] +[unused677] +[unused678] +[unused679] +[unused680] +[unused681] +[unused682] +[unused683] +[unused684] +[unused685] +[unused686] +[unused687] +[unused688] +[unused689] +[unused690] +[unused691] +[unused692] +[unused693] +[unused694] +[unused695] +[unused696] +[unused697] +[unused698] +[unused699] +[unused700] +[unused701] +[unused702] +[unused703] +[unused704] +[unused705] +[unused706] +[unused707] +[unused708] +[unused709] +[unused710] +[unused711] +[unused712] +[unused713] +[unused714] +[unused715] +[unused716] +[unused717] +[unused718] +[unused719] +[unused720] +[unused721] +[unused722] +[unused723] +[unused724] +[unused725] +[unused726] +[unused727] +[unused728] +[unused729] +[unused730] +[unused731] +[unused732] +[unused733] +[unused734] +[unused735] +[unused736] +[unused737] +[unused738] +[unused739] +[unused740] +[unused741] +[unused742] +[unused743] +[unused744] +[unused745] +[unused746] +[unused747] +[unused748] +[unused749] +[unused750] +[unused751] +[unused752] +[unused753] +[unused754] +[unused755] +[unused756] +[unused757] +[unused758] +[unused759] +[unused760] +[unused761] +[unused762] +[unused763] +[unused764] +[unused765] +[unused766] +[unused767] +[unused768] +[unused769] +[unused770] +[unused771] +[unused772] +[unused773] +[unused774] +[unused775] +[unused776] +[unused777] +[unused778] +[unused779] +[unused780] +[unused781] +[unused782] +[unused783] +[unused784] +[unused785] +[unused786] +[unused787] +[unused788] +[unused789] +[unused790] +[unused791] +[unused792] +[unused793] +[unused794] +[unused795] +[unused796] +[unused797] +[unused798] +[unused799] +[unused800] +[unused801] +[unused802] +[unused803] +[unused804] +[unused805] +[unused806] +[unused807] +[unused808] +[unused809] +[unused810] +[unused811] +[unused812] +[unused813] +[unused814] +[unused815] +[unused816] +[unused817] +[unused818] +[unused819] +[unused820] +[unused821] +[unused822] +[unused823] +[unused824] +[unused825] +[unused826] +[unused827] +[unused828] +[unused829] +[unused830] +[unused831] +[unused832] +[unused833] +[unused834] +[unused835] +[unused836] +[unused837] +[unused838] +[unused839] +[unused840] +[unused841] +[unused842] +[unused843] +[unused844] +[unused845] +[unused846] +[unused847] +[unused848] +[unused849] +[unused850] +[unused851] +[unused852] +[unused853] +[unused854] +[unused855] +[unused856] +[unused857] +[unused858] +[unused859] +[unused860] +[unused861] +[unused862] +[unused863] +[unused864] +[unused865] +[unused866] +[unused867] +[unused868] +[unused869] +[unused870] +[unused871] +[unused872] +[unused873] +[unused874] +[unused875] +[unused876] +[unused877] +[unused878] +[unused879] +[unused880] +[unused881] +[unused882] +[unused883] +[unused884] +[unused885] +[unused886] +[unused887] +[unused888] +[unused889] +[unused890] +[unused891] +[unused892] +[unused893] +[unused894] +[unused895] +[unused896] +[unused897] +[unused898] +[unused899] +[unused900] +[unused901] +[unused902] +[unused903] +[unused904] +[unused905] +[unused906] +[unused907] +[unused908] +[unused909] +[unused910] +[unused911] +[unused912] +[unused913] +[unused914] +[unused915] +[unused916] +[unused917] +[unused918] +[unused919] +[unused920] +[unused921] +[unused922] +[unused923] +[unused924] +[unused925] +[unused926] +[unused927] +[unused928] +[unused929] +[unused930] +[unused931] +[unused932] +[unused933] +[unused934] +[unused935] +[unused936] +[unused937] +[unused938] +[unused939] +[unused940] +[unused941] +[unused942] +[unused943] +[unused944] +[unused945] +[unused946] +[unused947] +[unused948] +[unused949] +[unused950] +[unused951] +[unused952] +[unused953] +[unused954] +[unused955] +[unused956] +[unused957] +[unused958] +[unused959] +[unused960] +[unused961] +[unused962] +[unused963] +[unused964] +[unused965] +[unused966] +[unused967] +[unused968] +[unused969] +[unused970] +[unused971] +[unused972] +[unused973] +[unused974] +[unused975] +[unused976] +[unused977] +[unused978] +[unused979] +[unused980] +[unused981] +[unused982] +[unused983] +[unused984] +[unused985] +[unused986] +[unused987] +[unused988] +[unused989] +[unused990] +[unused991] +[unused992] +[unused993] +! +" +# +$ +% +& +' +( +) +* ++ +, +- +. +/ +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +: +; +< += +> +? +@ +[ +\ +] +^ +_ +` +a +b +c +d +e +f +g +h +i +j +k +l +m +n +o +p +q +r +s +t +u +v +w +x +y +z +{ +| +} +~ +¡ +¢ +£ +¤ +¥ +¦ +§ +¨ +© +ª +« +¬ +® +° +± +² +³ +´ +µ +¶ +· +¹ +º +» +¼ +½ +¾ +¿ +× +ß +æ +ð +÷ +ø +þ +đ +ħ +ı +ł +ŋ +œ +ƒ +ɐ +ɑ +ɒ +ɔ +ɕ +ə +ɛ +ɡ +ɣ +ɨ +ɪ +ɫ +ɬ +ɯ +ɲ +ɴ +ɹ +ɾ +ʀ +ʁ +ʂ +ʃ +ʉ +ʊ +ʋ +ʌ +ʎ +ʐ +ʑ +ʒ +ʔ +ʰ +ʲ +ʳ +ʷ +ʸ +ʻ +ʼ +ʾ +ʿ +ˈ +ː +ˡ +ˢ +ˣ +ˤ +α +β +γ +δ +ε +ζ +η +θ +ι +κ +λ +μ +ν +ξ +ο +π +ρ +ς +σ +τ +υ +φ +χ +ψ +ω +а +б +в +г +д +е +ж +з +и +к +л +м +н +о +п +р +с +т +у +ф +х +ц +ч +ш +щ +ъ +ы +ь +э +ю +я +ђ +є +і +ј +љ +њ +ћ +ӏ +ա +բ +գ +դ +ե +թ +ի +լ +կ +հ +մ +յ +ն +ո +պ +ս +վ +տ +ր +ւ +ք +־ +א +ב +ג +ד +ה +ו +ז +ח +ט +י +ך +כ +ל +ם +מ +ן +נ +ס +ע +ף +פ +ץ +צ +ק +ר +ש +ת +، +ء +ا +ب +ة +ت +ث +ج +ح +خ +د +ذ +ر +ز +س +ش +ص +ض +ط +ظ +ع +غ +ـ +ف +ق +ك +ل +م +ن +ه +و +ى +ي +ٹ +پ +چ +ک +گ +ں +ھ +ہ +ی +ے +अ +आ +उ +ए +क +ख +ग +च +ज +ट +ड +ण +त +थ +द +ध +न +प +ब +भ +म +य +र +ल +व +श +ष +स +ह +ा +ि +ी +ो +। +॥ +ং +অ +আ +ই +উ +এ +ও +ক +খ +গ +চ +ছ +জ +ট +ড +ণ +ত +থ +দ +ধ +ন +প +ব +ভ +ম +য +র +ল +শ +ষ +স +হ +া +ি +ী +ে +க +ச +ட +த +ந +ன +ப +ம +ய +ர +ல +ள +வ +ா +ி +ு +ே +ை +ನ +ರ +ಾ +ක +ය +ර +ල +ව +ා +ก +ง +ต +ท +น +พ +ม +ย +ร +ล +ว +ส +อ +า +เ +་ +། +ག +ང +ད +ན +པ +བ +མ +འ +ར +ལ +ས +မ +ა +ბ +გ +დ +ე +ვ +თ +ი +კ +ლ +მ +ნ +ო +რ +ს +ტ +უ +ᄀ +ᄂ +ᄃ +ᄅ +ᄆ +ᄇ +ᄉ +ᄊ +ᄋ +ᄌ +ᄎ +ᄏ +ᄐ +ᄑ +ᄒ +ᅡ +ᅢ +ᅥ +ᅦ +ᅧ +ᅩ +ᅪ +ᅭ +ᅮ +ᅯ +ᅲ +ᅳ +ᅴ +ᅵ +ᆨ +ᆫ +ᆯ +ᆷ +ᆸ +ᆼ +ᴬ +ᴮ +ᴰ +ᴵ +ᴺ +ᵀ +ᵃ +ᵇ +ᵈ +ᵉ +ᵍ +ᵏ +ᵐ +ᵒ +ᵖ +ᵗ +ᵘ +ᵢ +ᵣ +ᵤ +ᵥ +ᶜ +ᶠ +‐ +‑ +‒ +– +— +― +‖ +‘ +’ +‚ +“ +” +„ +† +‡ +• +… +‰ +′ +″ +› +‿ +⁄ +⁰ +ⁱ +⁴ +⁵ +⁶ +⁷ +⁸ +⁹ +⁺ +⁻ +ⁿ +₀ +₁ +₂ +₃ +₄ +₅ +₆ +₇ +₈ +₉ +₊ +₍ +₎ +ₐ +ₑ +ₒ +ₓ +ₕ +ₖ +ₗ +ₘ +ₙ +ₚ +ₛ +ₜ +₤ +₩ +€ +₱ +₹ +ℓ +№ +ℝ +™ +⅓ +⅔ +← +↑ +→ +↓ +↔ +↦ +⇄ +⇌ +⇒ +∂ +∅ +∆ +∇ +∈ +− +∗ +∘ +√ +∞ +∧ +∨ +∩ +∪ +≈ +≡ +≤ +≥ +⊂ +⊆ +⊕ +⊗ +⋅ +─ +│ +■ +▪ +● +★ +☆ +☉ +♠ +♣ +♥ +♦ +♭ +♯ +⟨ +⟩ +ⱼ +⺩ +⺼ +⽥ +、 +。 +〈 +〉 +《 +》 +「 +」 +『 +』 +〜 +あ +い +う +え +お +か +き +く +け +こ +さ +し +す +せ +そ +た +ち +っ +つ +て +と +な +に +ぬ +ね +の +は +ひ +ふ +へ +ほ +ま +み +む +め +も +や +ゆ +よ +ら +り +る +れ +ろ +を +ん +ァ +ア +ィ +イ +ウ +ェ +エ +オ +カ +キ +ク +ケ +コ +サ +シ +ス +セ +タ +チ +ッ +ツ +テ +ト +ナ +ニ +ノ +ハ +ヒ +フ +ヘ +ホ +マ +ミ +ム +メ +モ +ャ +ュ +ョ +ラ +リ +ル +レ +ロ +ワ +ン +・ +ー +一 +三 +上 +下 +不 +世 +中 +主 +久 +之 +也 +事 +二 +五 +井 +京 +人 +亻 +仁 +介 +代 +仮 +伊 +会 +佐 +侍 +保 +信 +健 +元 +光 +八 +公 +内 +出 +分 +前 +劉 +力 +加 +勝 +北 +区 +十 +千 +南 +博 +原 +口 +古 +史 +司 +合 +吉 +同 +名 +和 +囗 +四 +国 +國 +土 +地 +坂 +城 +堂 +場 +士 +夏 +外 +大 +天 +太 +夫 +奈 +女 +子 +学 +宀 +宇 +安 +宗 +定 +宣 +宮 +家 +宿 +寺 +將 +小 +尚 +山 +岡 +島 +崎 +川 +州 +巿 +帝 +平 +年 +幸 +广 +弘 +張 +彳 +後 +御 +德 +心 +忄 +志 +忠 +愛 +成 +我 +戦 +戸 +手 +扌 +政 +文 +新 +方 +日 +明 +星 +春 +昭 +智 +曲 +書 +月 +有 +朝 +木 +本 +李 +村 +東 +松 +林 +森 +楊 +樹 +橋 +歌 +止 +正 +武 +比 +氏 +民 +水 +氵 +氷 +永 +江 +沢 +河 +治 +法 +海 +清 +漢 +瀬 +火 +版 +犬 +王 +生 +田 +男 +疒 +発 +白 +的 +皇 +目 +相 +省 +真 +石 +示 +社 +神 +福 +禾 +秀 +秋 +空 +立 +章 +竹 +糹 +美 +義 +耳 +良 +艹 +花 +英 +華 +葉 +藤 +行 +街 +西 +見 +訁 +語 +谷 +貝 +貴 +車 +軍 +辶 +道 +郎 +郡 +部 +都 +里 +野 +金 +鈴 +镇 +長 +門 +間 +阝 +阿 +陳 +陽 +雄 +青 +面 +風 +食 +香 +馬 +高 +龍 +龸 +fi +fl +！ +（ +） +， +－ +． +／ +： +？ +～ +the +of +and +in +to +was +he +is +as +for +on +with +that +it +his +by +at +from +her +##s +she +you +had +an +were +but +be +this +are +not +my +they +one +which +or +have +him +me +first +all +also +their +has +up +who +out +been +when +after +there +into +new +two +its +##a +time +would +no +what +about +said +we +over +then +other +so +more +##e +can +if +like +back +them +only +some +could +##i +where +just +##ing +during +before +##n +do +##o +made +school +through +than +now +years +most +world +may +between +down +well +three +##d +year +while +will +##ed +##r +##y +later +##t +city +under +around +did +such +being +used +state +people +part +know +against +your +many +second +university +both +national +##er +these +don +known +off +way +until +re +how +even +get +head +... +didn +##ly +team +american +because +de +##l +born +united +film +since +still +long +work +south +us +became +any +high +again +day +family +see +right +man +eyes +house +season +war +states +including +took +life +north +same +each +called +name +much +place +however +go +four +group +another +found +won +area +here +going +10 +away +series +left +home +music +best +make +hand +number +company +several +never +last +john +000 +very +album +take +end +good +too +following +released +game +played +little +began +district +##m +old +want +those +side +held +own +early +county +ll +league +use +west +##u +face +think +##es +2010 +government +##h +march +came +small +general +town +june +##on +line +based +something +##k +september +thought +looked +along +international +2011 +air +july +club +went +january +october +our +august +april +york +12 +few +2012 +2008 +east +show +member +college +2009 +father +public +##us +come +men +five +set +station +church +##c +next +former +november +room +party +located +december +2013 +age +got +2007 +##g +system +let +love +2006 +though +every +2014 +look +song +water +century +without +body +black +night +within +great +women +single +ve +building +large +population +river +named +band +white +started +##an +once +15 +20 +should +18 +2015 +service +top +built +british +open +death +king +moved +local +times +children +february +book +why +11 +door +need +president +order +final +road +wasn +although +due +major +died +village +third +knew +2016 +asked +turned +st +wanted +say +##p +together +received +main +son +served +different +##en +behind +himself +felt +members +power +football +law +voice +play +##in +near +park +history +30 +having +2005 +16 +##man +saw +mother +##al +army +point +front +help +english +street +art +late +hands +games +award +##ia +young +14 +put +published +country +division +across +told +13 +often +ever +french +london +center +six +red +2017 +led +days +include +light +25 +find +tell +among +species +really +according +central +half +2004 +form +original +gave +office +making +enough +lost +full +opened +must +included +live +given +german +player +run +business +woman +community +cup +might +million +land +2000 +court +development +17 +short +round +ii +km +seen +class +story +always +become +sure +research +almost +director +council +la +##2 +career +things +using +island +##z +couldn +car +##is +24 +close +force +##1 +better +free +support +control +field +students +2003 +education +married +##b +nothing +worked +others +record +big +inside +level +anything +continued +give +james +##3 +military +established +non +returned +feel +does +title +written +thing +feet +william +far +co +association +hard +already +2002 +##ra +championship +human +western +100 +##na +department +hall +role +various +production +21 +19 +heart +2001 +living +fire +version +##ers +##f +television +royal +##4 +produced +working +act +case +society +region +present +radio +period +looking +least +total +keep +england +wife +program +per +brother +mind +special +22 +##le +am +works +soon +##6 +political +george +services +taken +created +##7 +further +able +reached +david +union +joined +upon +done +important +social +information +either +##ic +##x +appeared +position +ground +lead +rock +dark +election +23 +board +france +hair +course +arms +site +police +girl +instead +real +sound +##v +words +moment +##te +someone +##8 +summer +project +announced +san +less +wrote +past +followed +##5 +blue +founded +al +finally +india +taking +records +america +##ne +1999 +design +considered +northern +god +stop +battle +toward +european +outside +described +track +today +playing +language +28 +call +26 +heard +professional +low +australia +miles +california +win +yet +green +##ie +trying +blood +##ton +southern +science +maybe +everything +match +square +27 +mouth +video +race +recorded +leave +above +##9 +daughter +points +space +1998 +museum +change +middle +common +##0 +move +tv +post +##ta +lake +seven +tried +elected +closed +ten +paul +minister +##th +months +start +chief +return +canada +person +sea +release +similar +modern +brought +rest +hit +formed +mr +##la +1997 +floor +event +doing +thomas +1996 +robert +care +killed +training +star +week +needed +turn +finished +railway +rather +news +health +sent +example +ran +term +michael +coming +currently +yes +forces +despite +gold +areas +50 +stage +fact +29 +dead +says +popular +2018 +originally +germany +probably +developed +result +pulled +friend +stood +money +running +mi +signed +word +songs +child +eventually +met +tour +average +teams +minutes +festival +current +deep +kind +1995 +decided +usually +eastern +seemed +##ness +episode +bed +added +table +indian +private +charles +route +available +idea +throughout +centre +addition +appointed +style +1994 +books +eight +construction +press +mean +wall +friends +remained +schools +study +##ch +##um +institute +oh +chinese +sometimes +events +possible +1992 +australian +type +brown +forward +talk +process +food +debut +seat +performance +committee +features +character +arts +herself +else +lot +strong +russian +range +hours +peter +arm +##da +morning +dr +sold +##ry +quickly +directed +1993 +guitar +china +##w +31 +list +##ma +performed +media +uk +players +smile +##rs +myself +40 +placed +coach +province +towards +wouldn +leading +whole +boy +official +designed +grand +census +##el +europe +attack +japanese +henry +1991 +##re +##os +cross +getting +alone +action +lower +network +wide +washington +japan +1990 +hospital +believe +changed +sister +##ar +hold +gone +sir +hadn +ship +##ka +studies +academy +shot +rights +below +base +bad +involved +kept +largest +##ist +bank +future +especially +beginning +mark +movement +section +female +magazine +plan +professor +lord +longer +##ian +sat +walked +hill +actually +civil +energy +model +families +size +thus +aircraft +completed +includes +data +captain +##or +fight +vocals +featured +richard +bridge +fourth +1989 +officer +stone +hear +##ism +means +medical +groups +management +self +lips +competition +entire +lived +technology +leaving +federal +tournament +bit +passed +hot +independent +awards +kingdom +mary +spent +fine +doesn +reported +##ling +jack +fall +raised +itself +stay +true +studio +1988 +sports +replaced +paris +systems +saint +leader +theatre +whose +market +capital +parents +spanish +canadian +earth +##ity +cut +degree +writing +bay +christian +awarded +natural +higher +bill +##as +coast +provided +previous +senior +ft +valley +organization +stopped +onto +countries +parts +conference +queen +security +interest +saying +allowed +master +earlier +phone +matter +smith +winning +try +happened +moving +campaign +los +##ley +breath +nearly +mid +1987 +certain +girls +date +italian +african +standing +fell +artist +##ted +shows +deal +mine +industry +1986 +##ng +everyone +republic +provide +collection +library +student +##ville +primary +owned +older +via +heavy +1st +makes +##able +attention +anyone +africa +##ri +stated +length +ended +fingers +command +staff +skin +foreign +opening +governor +okay +medal +kill +sun +cover +job +1985 +introduced +chest +hell +feeling +##ies +success +meet +reason +standard +meeting +novel +1984 +trade +source +buildings +##land +rose +guy +goal +##ur +chapter +native +husband +previously +unit +limited +entered +weeks +producer +operations +mountain +takes +covered +forced +related +roman +complete +successful +key +texas +cold +##ya +channel +1980 +traditional +films +dance +clear +approximately +500 +nine +van +prince +question +active +tracks +ireland +regional +silver +author +personal +sense +operation +##ine +economic +1983 +holding +twenty +isbn +additional +speed +hour +edition +regular +historic +places +whom +shook +movie +km² +secretary +prior +report +chicago +read +foundation +view +engine +scored +1982 +units +ask +airport +property +ready +immediately +lady +month +listed +contract +##de +manager +themselves +lines +##ki +navy +writer +meant +##ts +runs +##ro +practice +championships +singer +glass +commission +required +forest +starting +culture +generally +giving +access +attended +test +couple +stand +catholic +martin +caught +executive +##less +eye +##ey +thinking +chair +quite +shoulder +1979 +hope +decision +plays +defeated +municipality +whether +structure +offered +slowly +pain +ice +direction +##ion +paper +mission +1981 +mostly +200 +noted +individual +managed +nature +lives +plant +##ha +helped +except +studied +computer +figure +relationship +issue +significant +loss +die +smiled +gun +ago +highest +1972 +##am +male +bring +goals +mexico +problem +distance +commercial +completely +location +annual +famous +drive +1976 +neck +1978 +surface +caused +italy +understand +greek +highway +wrong +hotel +comes +appearance +joseph +double +issues +musical +companies +castle +income +review +assembly +bass +initially +parliament +artists +experience +1974 +particular +walk +foot +engineering +talking +window +dropped +##ter +miss +baby +boys +break +1975 +stars +edge +remember +policy +carried +train +stadium +bar +sex +angeles +evidence +##ge +becoming +assistant +soviet +1977 +upper +step +wing +1970 +youth +financial +reach +##ll +actor +numerous +##se +##st +nodded +arrived +##ation +minute +##nt +believed +sorry +complex +beautiful +victory +associated +temple +1968 +1973 +chance +perhaps +metal +##son +1945 +bishop +##et +lee +launched +particularly +tree +le +retired +subject +prize +contains +yeah +theory +empire +##ce +suddenly +waiting +trust +recording +##to +happy +terms +camp +champion +1971 +religious +pass +zealand +names +2nd +port +ancient +tom +corner +represented +watch +legal +anti +justice +cause +watched +brothers +45 +material +changes +simply +response +louis +fast +##ting +answer +60 +historical +1969 +stories +straight +create +feature +increased +rate +administration +virginia +el +activities +cultural +overall +winner +programs +basketball +legs +guard +beyond +cast +doctor +mm +flight +results +remains +cost +effect +winter +##ble +larger +islands +problems +chairman +grew +commander +isn +1967 +pay +failed +selected +hurt +fort +box +regiment +majority +journal +35 +edward +plans +##ke +##ni +shown +pretty +irish +characters +directly +scene +likely +operated +allow +spring +##j +junior +matches +looks +mike +houses +fellow +##tion +beach +marriage +##ham +##ive +rules +oil +65 +florida +expected +nearby +congress +sam +peace +recent +iii +wait +subsequently +cell +##do +variety +serving +agreed +please +poor +joe +pacific +attempt +wood +democratic +piece +prime +##ca +rural +mile +touch +appears +township +1964 +1966 +soldiers +##men +##ized +1965 +pennsylvania +closer +fighting +claimed +score +jones +physical +editor +##ous +filled +genus +specific +sitting +super +mom +##va +therefore +supported +status +fear +cases +store +meaning +wales +minor +spain +tower +focus +vice +frank +follow +parish +separate +golden +horse +fifth +remaining +branch +32 +presented +stared +##id +uses +secret +forms +##co +baseball +exactly +##ck +choice +note +discovered +travel +composed +truth +russia +ball +color +kiss +dad +wind +continue +ring +referred +numbers +digital +greater +##ns +metres +slightly +direct +increase +1960 +responsible +crew +rule +trees +troops +##no +broke +goes +individuals +hundred +weight +creek +sleep +memory +defense +provides +ordered +code +value +jewish +windows +1944 +safe +judge +whatever +corps +realized +growing +pre +##ga +cities +alexander +gaze +lies +spread +scott +letter +showed +situation +mayor +transport +watching +workers +extended +##li +expression +normal +##ment +chart +multiple +border +##ba +host +##ner +daily +mrs +walls +piano +##ko +heat +cannot +##ate +earned +products +drama +era +authority +seasons +join +grade +##io +sign +difficult +machine +1963 +territory +mainly +##wood +stations +squadron +1962 +stepped +iron +19th +##led +serve +appear +sky +speak +broken +charge +knowledge +kilometres +removed +ships +article +campus +simple +##ty +pushed +britain +##ve +leaves +recently +cd +soft +boston +latter +easy +acquired +poland +##sa +quality +officers +presence +planned +nations +mass +broadcast +jean +share +image +influence +wild +offer +emperor +electric +reading +headed +ability +promoted +yellow +ministry +1942 +throat +smaller +politician +##by +latin +spoke +cars +williams +males +lack +pop +80 +##ier +acting +seeing +consists +##ti +estate +1961 +pressure +johnson +newspaper +jr +chris +olympics +online +conditions +beat +elements +walking +vote +##field +needs +carolina +text +featuring +global +block +shirt +levels +francisco +purpose +females +et +dutch +duke +ahead +gas +twice +safety +serious +turning +highly +lieutenant +firm +maria +amount +mixed +daniel +proposed +perfect +agreement +affairs +3rd +seconds +contemporary +paid +1943 +prison +save +kitchen +label +administrative +intended +constructed +academic +nice +teacher +races +1956 +formerly +corporation +ben +nation +issued +shut +1958 +drums +housing +victoria +seems +opera +1959 +graduated +function +von +mentioned +picked +build +recognized +shortly +protection +picture +notable +exchange +elections +1980s +loved +percent +racing +fish +elizabeth +garden +volume +hockey +1941 +beside +settled +##ford +1940 +competed +replied +drew +1948 +actress +marine +scotland +steel +glanced +farm +steve +1957 +risk +tonight +positive +magic +singles +effects +gray +screen +dog +##ja +residents +bus +sides +none +secondary +literature +polish +destroyed +flying +founder +households +1939 +lay +reserve +usa +gallery +##ler +1946 +industrial +younger +approach +appearances +urban +ones +1950 +finish +avenue +powerful +fully +growth +page +honor +jersey +projects +advanced +revealed +basic +90 +infantry +pair +equipment +visit +33 +evening +search +grant +effort +solo +treatment +buried +republican +primarily +bottom +owner +1970s +israel +gives +jim +dream +bob +remain +spot +70 +notes +produce +champions +contact +ed +soul +accepted +ways +del +##ally +losing +split +price +capacity +basis +trial +questions +##ina +1955 +20th +guess +officially +memorial +naval +initial +##ization +whispered +median +engineer +##ful +sydney +##go +columbia +strength +300 +1952 +tears +senate +00 +card +asian +agent +1947 +software +44 +draw +warm +supposed +com +pro +##il +transferred +leaned +##at +candidate +escape +mountains +asia +potential +activity +entertainment +seem +traffic +jackson +murder +36 +slow +product +orchestra +haven +agency +bbc +taught +website +comedy +unable +storm +planning +albums +rugby +environment +scientific +grabbed +protect +##hi +boat +typically +1954 +1953 +damage +principal +divided +dedicated +mount +ohio +##berg +pick +fought +driver +##der +empty +shoulders +sort +thank +berlin +prominent +account +freedom +necessary +efforts +alex +headquarters +follows +alongside +des +simon +andrew +suggested +operating +learning +steps +1949 +sweet +technical +begin +easily +34 +teeth +speaking +settlement +scale +##sh +renamed +ray +max +enemy +semi +joint +compared +##rd +scottish +leadership +analysis +offers +georgia +pieces +captured +animal +deputy +guest +organized +##lin +tony +combined +method +challenge +1960s +huge +wants +battalion +sons +rise +crime +types +facilities +telling +path +1951 +platform +sit +1990s +##lo +tells +assigned +rich +pull +##ot +commonly +alive +##za +letters +concept +conducted +wearing +happen +bought +becomes +holy +gets +ocean +defeat +languages +purchased +coffee +occurred +titled +##q +declared +applied +sciences +concert +sounds +jazz +brain +##me +painting +fleet +tax +nick +##ius +michigan +count +animals +leaders +episodes +##line +content +##den +birth +##it +clubs +64 +palace +critical +refused +fair +leg +laughed +returning +surrounding +participated +formation +lifted +pointed +connected +rome +medicine +laid +taylor +santa +powers +adam +tall +shared +focused +knowing +yards +entrance +falls +##wa +calling +##ad +sources +chosen +beneath +resources +yard +##ite +nominated +silence +zone +defined +##que +gained +thirty +38 +bodies +moon +##ard +adopted +christmas +widely +register +apart +iran +premier +serves +du +unknown +parties +##les +generation +##ff +continues +quick +fields +brigade +quiet +teaching +clothes +impact +weapons +partner +flat +theater +supreme +1938 +37 +relations +##tor +plants +suffered +1936 +wilson +kids +begins +##age +1918 +seats +armed +internet +models +worth +laws +400 +communities +classes +background +knows +thanks +quarter +reaching +humans +carry +killing +format +kong +hong +setting +75 +architecture +disease +railroad +inc +possibly +wish +arthur +thoughts +harry +doors +density +##di +crowd +illinois +stomach +tone +unique +reports +anyway +##ir +liberal +der +vehicle +thick +dry +drug +faced +largely +facility +theme +holds +creation +strange +colonel +##mi +revolution +bell +politics +turns +silent +rail +relief +independence +combat +shape +write +determined +sales +learned +4th +finger +oxford +providing +1937 +heritage +fiction +situated +designated +allowing +distribution +hosted +##est +sight +interview +estimated +reduced +##ria +toronto +footballer +keeping +guys +damn +claim +motion +sport +sixth +stayed +##ze +en +rear +receive +handed +twelve +dress +audience +granted +brazil +##well +spirit +##ated +noticed +etc +olympic +representative +eric +tight +trouble +reviews +drink +vampire +missing +roles +ranked +newly +household +finals +wave +critics +##ee +phase +massachusetts +pilot +unlike +philadelphia +bright +guns +crown +organizations +roof +42 +respectively +clearly +tongue +marked +circle +fox +korea +bronze +brian +expanded +sexual +supply +yourself +inspired +labour +fc +##ah +reference +vision +draft +connection +brand +reasons +1935 +classic +driving +trip +jesus +cells +entry +1920 +neither +trail +claims +atlantic +orders +labor +nose +afraid +identified +intelligence +calls +cancer +attacked +passing +stephen +positions +imperial +grey +jason +39 +sunday +48 +swedish +avoid +extra +uncle +message +covers +allows +surprise +materials +fame +hunter +##ji +1930 +citizens +figures +davis +environmental +confirmed +shit +titles +di +performing +difference +acts +attacks +##ov +existing +votes +opportunity +nor +shop +entirely +trains +opposite +pakistan +##pa +develop +resulted +representatives +actions +reality +pressed +##ish +barely +wine +conversation +faculty +northwest +ends +documentary +nuclear +stock +grace +sets +eat +alternative +##ps +bag +resulting +creating +surprised +cemetery +1919 +drop +finding +sarah +cricket +streets +tradition +ride +1933 +exhibition +target +ear +explained +rain +composer +injury +apartment +municipal +educational +occupied +netherlands +clean +billion +constitution +learn +1914 +maximum +classical +francis +lose +opposition +jose +ontario +bear +core +hills +rolled +ending +drawn +permanent +fun +##tes +##lla +lewis +sites +chamber +ryan +##way +scoring +height +1934 +##house +lyrics +staring +55 +officials +1917 +snow +oldest +##tic +orange +##ger +qualified +interior +apparently +succeeded +thousand +dinner +lights +existence +fans +heavily +41 +greatest +conservative +send +bowl +plus +enter +catch +##un +economy +duty +1929 +speech +authorities +princess +performances +versions +shall +graduate +pictures +effective +remembered +poetry +desk +crossed +starring +starts +passenger +sharp +##ant +acres +ass +weather +falling +rank +fund +supporting +check +adult +publishing +heads +cm +southeast +lane +##burg +application +bc +##ura +les +condition +transfer +prevent +display +ex +regions +earl +federation +cool +relatively +answered +besides +1928 +obtained +portion +##town +mix +##ding +reaction +liked +dean +express +peak +1932 +##tte +counter +religion +chain +rare +miller +convention +aid +lie +vehicles +mobile +perform +squad +wonder +lying +crazy +sword +##ping +attempted +centuries +weren +philosophy +category +##ize +anna +interested +47 +sweden +wolf +frequently +abandoned +kg +literary +alliance +task +entitled +##ay +threw +promotion +factory +tiny +soccer +visited +matt +fm +achieved +52 +defence +internal +persian +43 +methods +##ging +arrested +otherwise +cambridge +programming +villages +elementary +districts +rooms +criminal +conflict +worry +trained +1931 +attempts +waited +signal +bird +truck +subsequent +programme +##ol +ad +49 +communist +details +faith +sector +patrick +carrying +laugh +##ss +controlled +korean +showing +origin +fuel +evil +1927 +##ent +brief +identity +darkness +address +pool +missed +publication +web +planet +ian +anne +wings +invited +##tt +briefly +standards +kissed +##be +ideas +climate +causing +walter +worse +albert +articles +winners +desire +aged +northeast +dangerous +gate +doubt +1922 +wooden +multi +##ky +poet +rising +funding +46 +communications +communication +violence +copies +prepared +ford +investigation +skills +1924 +pulling +electronic +##ak +##ial +##han +containing +ultimately +offices +singing +understanding +restaurant +tomorrow +fashion +christ +ward +da +pope +stands +5th +flow +studios +aired +commissioned +contained +exist +fresh +americans +##per +wrestling +approved +kid +employed +respect +suit +1925 +angel +asking +increasing +frame +angry +selling +1950s +thin +finds +##nd +temperature +statement +ali +explain +inhabitants +towns +extensive +narrow +51 +jane +flowers +images +promise +somewhere +object +fly +closely +##ls +1912 +bureau +cape +1926 +weekly +presidential +legislative +1921 +##ai +##au +launch +founding +##ny +978 +##ring +artillery +strike +un +institutions +roll +writers +landing +chose +kevin +anymore +pp +##ut +attorney +fit +dan +billboard +receiving +agricultural +breaking +sought +dave +admitted +lands +mexican +##bury +charlie +specifically +hole +iv +howard +credit +moscow +roads +accident +1923 +proved +wear +struck +hey +guards +stuff +slid +expansion +1915 +cat +anthony +##kin +melbourne +opposed +sub +southwest +architect +failure +plane +1916 +##ron +map +camera +tank +listen +regarding +wet +introduction +metropolitan +link +ep +fighter +inch +grown +gene +anger +fixed +buy +dvd +khan +domestic +worldwide +chapel +mill +functions +examples +##head +developing +1910 +turkey +hits +pocket +antonio +papers +grow +unless +circuit +18th +concerned +attached +journalist +selection +journey +converted +provincial +painted +hearing +aren +bands +negative +aside +wondered +knight +lap +survey +ma +##ow +noise +billy +##ium +shooting +guide +bedroom +priest +resistance +motor +homes +sounded +giant +##mer +150 +scenes +equal +comic +patients +hidden +solid +actual +bringing +afternoon +touched +funds +wedding +consisted +marie +canal +sr +kim +treaty +turkish +recognition +residence +cathedral +broad +knees +incident +shaped +fired +norwegian +handle +cheek +contest +represent +##pe +representing +beauty +##sen +birds +advantage +emergency +wrapped +drawing +notice +pink +broadcasting +##ong +somehow +bachelor +seventh +collected +registered +establishment +alan +assumed +chemical +personnel +roger +retirement +jeff +portuguese +wore +tied +device +threat +progress +advance +##ised +banks +hired +manchester +nfl +teachers +structures +forever +##bo +tennis +helping +saturday +sale +applications +junction +hip +incorporated +neighborhood +dressed +ceremony +##ds +influenced +hers +visual +stairs +decades +inner +kansas +hung +hoped +gain +scheduled +downtown +engaged +austria +clock +norway +certainly +pale +protected +1913 +victor +employees +plate +putting +surrounded +##ists +finishing +blues +tropical +##ries +minnesota +consider +philippines +accept +54 +retrieved +1900 +concern +anderson +properties +institution +gordon +successfully +vietnam +##dy +backing +outstanding +muslim +crossing +folk +producing +usual +demand +occurs +observed +lawyer +educated +##ana +kelly +string +pleasure +budget +items +quietly +colorado +philip +typical +##worth +derived +600 +survived +asks +mental +##ide +56 +jake +jews +distinguished +ltd +1911 +sri +extremely +53 +athletic +loud +thousands +worried +shadow +transportation +horses +weapon +arena +importance +users +tim +objects +contributed +dragon +douglas +aware +senator +johnny +jordan +sisters +engines +flag +investment +samuel +shock +capable +clark +row +wheel +refers +session +familiar +biggest +wins +hate +maintained +drove +hamilton +request +expressed +injured +underground +churches +walker +wars +tunnel +passes +stupid +agriculture +softly +cabinet +regarded +joining +indiana +##ea +##ms +push +dates +spend +behavior +woods +protein +gently +chase +morgan +mention +burning +wake +combination +occur +mirror +leads +jimmy +indeed +impossible +singapore +paintings +covering +##nes +soldier +locations +attendance +sell +historian +wisconsin +invasion +argued +painter +diego +changing +egypt +##don +experienced +inches +##ku +missouri +vol +grounds +spoken +switzerland +##gan +reform +rolling +ha +forget +massive +resigned +burned +allen +tennessee +locked +values +improved +##mo +wounded +universe +sick +dating +facing +pack +purchase +user +##pur +moments +##ul +merged +anniversary +1908 +coal +brick +understood +causes +dynasty +queensland +establish +stores +crisis +promote +hoping +views +cards +referee +extension +##si +raise +arizona +improve +colonial +formal +charged +##rt +palm +lucky +hide +rescue +faces +95 +feelings +candidates +juan +##ell +goods +6th +courses +weekend +59 +luke +cash +fallen +##om +delivered +affected +installed +carefully +tries +swiss +hollywood +costs +lincoln +responsibility +##he +shore +file +proper +normally +maryland +assistance +jump +constant +offering +friendly +waters +persons +realize +contain +trophy +800 +partnership +factor +58 +musicians +cry +bound +oregon +indicated +hero +houston +medium +##ure +consisting +somewhat +##ara +57 +cycle +##che +beer +moore +frederick +gotten +eleven +worst +weak +approached +arranged +chin +loan +universal +bond +fifteen +pattern +disappeared +##ney +translated +##zed +lip +arab +capture +interests +insurance +##chi +shifted +cave +prix +warning +sections +courts +coat +plot +smell +feed +golf +favorite +maintain +knife +vs +voted +degrees +finance +quebec +opinion +translation +manner +ruled +operate +productions +choose +musician +discovery +confused +tired +separated +stream +techniques +committed +attend +ranking +kings +throw +passengers +measure +horror +fan +mining +sand +danger +salt +calm +decade +dam +require +runner +##ik +rush +associate +greece +##ker +rivers +consecutive +matthew +##ski +sighed +sq +documents +steam +edited +closing +tie +accused +1905 +##ini +islamic +distributed +directors +organisation +bruce +7th +breathing +mad +lit +arrival +concrete +taste +08 +composition +shaking +faster +amateur +adjacent +stating +1906 +twin +flew +##ran +tokyo +publications +##tone +obviously +ridge +storage +1907 +carl +pages +concluded +desert +driven +universities +ages +terminal +sequence +borough +250 +constituency +creative +cousin +economics +dreams +margaret +notably +reduce +montreal +mode +17th +ears +saved +jan +vocal +##ica +1909 +andy +##jo +riding +roughly +threatened +##ise +meters +meanwhile +landed +compete +repeated +grass +czech +regularly +charges +tea +sudden +appeal +##ung +solution +describes +pierre +classification +glad +parking +##ning +belt +physics +99 +rachel +add +hungarian +participate +expedition +damaged +gift +childhood +85 +fifty +##red +mathematics +jumped +letting +defensive +mph +##ux +##gh +testing +##hip +hundreds +shoot +owners +matters +smoke +israeli +kentucky +dancing +mounted +grandfather +emma +designs +profit +argentina +##gs +truly +li +lawrence +cole +begun +detroit +willing +branches +smiling +decide +miami +enjoyed +recordings +##dale +poverty +ethnic +gay +##bi +gary +arabic +09 +accompanied +##one +##ons +fishing +determine +residential +acid +##ary +alice +returns +starred +mail +##ang +jonathan +strategy +##ue +net +forty +cook +businesses +equivalent +commonwealth +distinct +ill +##cy +seriously +##ors +##ped +shift +harris +replace +rio +imagine +formula +ensure +##ber +additionally +scheme +conservation +occasionally +purposes +feels +favor +##and +##ore +1930s +contrast +hanging +hunt +movies +1904 +instruments +victims +danish +christopher +busy +demon +sugar +earliest +colony +studying +balance +duties +##ks +belgium +slipped +carter +05 +visible +stages +iraq +fifa +##im +commune +forming +zero +07 +continuing +talked +counties +legend +bathroom +option +tail +clay +daughters +afterwards +severe +jaw +visitors +##ded +devices +aviation +russell +kate +##vi +entering +subjects +##ino +temporary +swimming +forth +smooth +ghost +audio +bush +operates +rocks +movements +signs +eddie +##tz +ann +voices +honorary +06 +memories +dallas +pure +measures +racial +promised +66 +harvard +ceo +16th +parliamentary +indicate +benefit +flesh +dublin +louisiana +1902 +1901 +patient +sleeping +1903 +membership +coastal +medieval +wanting +element +scholars +rice +62 +limit +survive +makeup +rating +definitely +collaboration +obvious +##tan +boss +ms +baron +birthday +linked +soil +diocese +##lan +ncaa +##mann +offensive +shell +shouldn +waist +##tus +plain +ross +organ +resolution +manufacturing +adding +relative +kennedy +98 +whilst +moth +marketing +gardens +crash +72 +heading +partners +credited +carlos +moves +cable +##zi +marshall +##out +depending +bottle +represents +rejected +responded +existed +04 +jobs +denmark +lock +##ating +treated +graham +routes +talent +commissioner +drugs +secure +tests +reign +restored +photography +##gi +contributions +oklahoma +designer +disc +grin +seattle +robin +paused +atlanta +unusual +##gate +praised +las +laughing +satellite +hungary +visiting +##sky +interesting +factors +deck +poems +norman +##water +stuck +speaker +rifle +domain +premiered +##her +dc +comics +actors +01 +reputation +eliminated +8th +ceiling +prisoners +script +##nce +leather +austin +mississippi +rapidly +admiral +parallel +charlotte +guilty +tools +gender +divisions +fruit +##bs +laboratory +nelson +fantasy +marry +rapid +aunt +tribe +requirements +aspects +suicide +amongst +adams +bone +ukraine +abc +kick +sees +edinburgh +clothing +column +rough +gods +hunting +broadway +gathered +concerns +##ek +spending +ty +12th +snapped +requires +solar +bones +cavalry +##tta +iowa +drinking +waste +index +franklin +charity +thompson +stewart +tip +flash +landscape +friday +enjoy +singh +poem +listening +##back +eighth +fred +differences +adapted +bomb +ukrainian +surgery +corporate +masters +anywhere +##more +waves +odd +sean +portugal +orleans +dick +debate +kent +eating +puerto +cleared +96 +expect +cinema +97 +guitarist +blocks +electrical +agree +involving +depth +dying +panel +struggle +##ged +peninsula +adults +novels +emerged +vienna +metro +debuted +shoes +tamil +songwriter +meets +prove +beating +instance +heaven +scared +sending +marks +artistic +passage +superior +03 +significantly +shopping +##tive +retained +##izing +malaysia +technique +cheeks +##ola +warren +maintenance +destroy +extreme +allied +120 +appearing +##yn +fill +advice +alabama +qualifying +policies +cleveland +hat +battery +smart +authors +10th +soundtrack +acted +dated +lb +glance +equipped +coalition +funny +outer +ambassador +roy +possibility +couples +campbell +dna +loose +ethan +supplies +1898 +gonna +88 +monster +##res +shake +agents +frequency +springs +dogs +practices +61 +gang +plastic +easier +suggests +gulf +blade +exposed +colors +industries +markets +pan +nervous +electoral +charts +legislation +ownership +##idae +mac +appointment +shield +copy +assault +socialist +abbey +monument +license +throne +employment +jay +93 +replacement +charter +cloud +powered +suffering +accounts +oak +connecticut +strongly +wright +colour +crystal +13th +context +welsh +networks +voiced +gabriel +jerry +##cing +forehead +mp +##ens +manage +schedule +totally +remix +##ii +forests +occupation +print +nicholas +brazilian +strategic +vampires +engineers +76 +roots +seek +correct +instrumental +und +alfred +backed +hop +##des +stanley +robinson +traveled +wayne +welcome +austrian +achieve +67 +exit +rates +1899 +strip +whereas +##cs +sing +deeply +adventure +bobby +rick +jamie +careful +components +cap +useful +personality +knee +##shi +pushing +hosts +02 +protest +ca +ottoman +symphony +##sis +63 +boundary +1890 +processes +considering +considerable +tons +##work +##ft +##nia +cooper +trading +dear +conduct +91 +illegal +apple +revolutionary +holiday +definition +harder +##van +jacob +circumstances +destruction +##lle +popularity +grip +classified +liverpool +donald +baltimore +flows +seeking +honour +approval +92 +mechanical +till +happening +statue +critic +increasingly +immediate +describe +commerce +stare +##ster +indonesia +meat +rounds +boats +baker +orthodox +depression +formally +worn +naked +claire +muttered +sentence +11th +emily +document +77 +criticism +wished +vessel +spiritual +bent +virgin +parker +minimum +murray +lunch +danny +printed +compilation +keyboards +false +blow +belonged +68 +raising +78 +cutting +##board +pittsburgh +##up +9th +shadows +81 +hated +indigenous +jon +15th +barry +scholar +ah +##zer +oliver +##gy +stick +susan +meetings +attracted +spell +romantic +##ver +ye +1895 +photo +demanded +customers +##ac +1896 +logan +revival +keys +modified +commanded +jeans +##ious +upset +raw +phil +detective +hiding +resident +vincent +##bly +experiences +diamond +defeating +coverage +lucas +external +parks +franchise +helen +bible +successor +percussion +celebrated +il +lift +profile +clan +romania +##ied +mills +##su +nobody +achievement +shrugged +fault +1897 +rhythm +initiative +breakfast +carbon +700 +69 +lasted +violent +74 +wound +ken +killer +gradually +filmed +°c +dollars +processing +94 +remove +criticized +guests +sang +chemistry +##vin +legislature +disney +##bridge +uniform +escaped +integrated +proposal +purple +denied +liquid +karl +influential +morris +nights +stones +intense +experimental +twisted +71 +84 +##ld +pace +nazi +mitchell +ny +blind +reporter +newspapers +14th +centers +burn +basin +forgotten +surviving +filed +collections +monastery +losses +manual +couch +description +appropriate +merely +tag +missions +sebastian +restoration +replacing +triple +73 +elder +julia +warriors +benjamin +julian +convinced +stronger +amazing +declined +versus +merchant +happens +output +finland +bare +barbara +absence +ignored +dawn +injuries +##port +producers +##ram +82 +luis +##ities +kw +admit +expensive +electricity +nba +exception +symbol +##ving +ladies +shower +sheriff +characteristics +##je +aimed +button +ratio +effectively +summit +angle +jury +bears +foster +vessels +pants +executed +evans +dozen +advertising +kicked +patrol +1889 +competitions +lifetime +principles +athletics +##logy +birmingham +sponsored +89 +rob +nomination +1893 +acoustic +##sm +creature +longest +##tra +credits +harbor +dust +josh +##so +territories +milk +infrastructure +completion +thailand +indians +leon +archbishop +##sy +assist +pitch +blake +arrangement +girlfriend +serbian +operational +hence +sad +scent +fur +dj +sessions +hp +refer +rarely +##ora +exists +1892 +##ten +scientists +dirty +penalty +burst +portrait +seed +79 +pole +limits +rival +1894 +stable +alpha +grave +constitutional +alcohol +arrest +flower +mystery +devil +architectural +relationships +greatly +habitat +##istic +larry +progressive +remote +cotton +##ics +##ok +preserved +reaches +##ming +cited +86 +vast +scholarship +decisions +cbs +joy +teach +1885 +editions +knocked +eve +searching +partly +participation +gap +animated +fate +excellent +##ett +na +87 +alternate +saints +youngest +##ily +climbed +##ita +##tors +suggest +##ct +discussion +staying +choir +lakes +jacket +revenue +nevertheless +peaked +instrument +wondering +annually +managing +neil +1891 +signing +terry +##ice +apply +clinical +brooklyn +aim +catherine +fuck +farmers +figured +ninth +pride +hugh +evolution +ordinary +involvement +comfortable +shouted +tech +encouraged +taiwan +representation +sharing +##lia +##em +panic +exact +cargo +competing +fat +cried +83 +1920s +occasions +pa +cabin +borders +utah +marcus +##isation +badly +muscles +##ance +victorian +transition +warner +bet +permission +##rin +slave +terrible +similarly +shares +seth +uefa +possession +medals +benefits +colleges +lowered +perfectly +mall +transit +##ye +##kar +publisher +##ened +harrison +deaths +elevation +##ae +asleep +machines +sigh +ash +hardly +argument +occasion +parent +leo +decline +1888 +contribution +##ua +concentration +1000 +opportunities +hispanic +guardian +extent +emotions +hips +mason +volumes +bloody +controversy +diameter +steady +mistake +phoenix +identify +violin +##sk +departure +richmond +spin +funeral +enemies +1864 +gear +literally +connor +random +sergeant +grab +confusion +1865 +transmission +informed +op +leaning +sacred +suspended +thinks +gates +portland +luck +agencies +yours +hull +expert +muscle +layer +practical +sculpture +jerusalem +latest +lloyd +statistics +deeper +recommended +warrior +arkansas +mess +supports +greg +eagle +1880 +recovered +rated +concerts +rushed +##ano +stops +eggs +files +premiere +keith +##vo +delhi +turner +pit +affair +belief +paint +##zing +mate +##ach +##ev +victim +##ology +withdrew +bonus +styles +fled +##ud +glasgow +technologies +funded +nbc +adaptation +##ata +portrayed +cooperation +supporters +judges +bernard +justin +hallway +ralph +##ick +graduating +controversial +distant +continental +spider +bite +##ho +recognize +intention +mixing +##ese +egyptian +bow +tourism +suppose +claiming +tiger +dominated +participants +vi +##ru +nurse +partially +tape +##rum +psychology +##rn +essential +touring +duo +voting +civilian +emotional +channels +##king +apparent +hebrew +1887 +tommy +carrier +intersection +beast +hudson +##gar +##zo +lab +nova +bench +discuss +costa +##ered +detailed +behalf +drivers +unfortunately +obtain +##lis +rocky +##dae +siege +friendship +honey +##rian +1861 +amy +hang +posted +governments +collins +respond +wildlife +preferred +operator +##po +laura +pregnant +videos +dennis +suspected +boots +instantly +weird +automatic +businessman +alleged +placing +throwing +ph +mood +1862 +perry +venue +jet +remainder +##lli +##ci +passion +biological +boyfriend +1863 +dirt +buffalo +ron +segment +fa +abuse +##era +genre +thrown +stroke +colored +stress +exercise +displayed +##gen +struggled +##tti +abroad +dramatic +wonderful +thereafter +madrid +component +widespread +##sed +tale +citizen +todd +monday +1886 +vancouver +overseas +forcing +crying +descent +##ris +discussed +substantial +ranks +regime +1870 +provinces +switch +drum +zane +ted +tribes +proof +lp +cream +researchers +volunteer +manor +silk +milan +donated +allies +venture +principle +delivery +enterprise +##ves +##ans +bars +traditionally +witch +reminded +copper +##uk +pete +inter +links +colin +grinned +elsewhere +competitive +frequent +##oy +scream +##hu +tension +texts +submarine +finnish +defending +defend +pat +detail +1884 +affiliated +stuart +themes +villa +periods +tool +belgian +ruling +crimes +answers +folded +licensed +resort +demolished +hans +lucy +1881 +lion +traded +photographs +writes +craig +##fa +trials +generated +beth +noble +debt +percentage +yorkshire +erected +ss +viewed +grades +confidence +ceased +islam +telephone +retail +##ible +chile +m² +roberts +sixteen +##ich +commented +hampshire +innocent +dual +pounds +checked +regulations +afghanistan +sung +rico +liberty +assets +bigger +options +angels +relegated +tribute +wells +attending +leaf +##yan +butler +romanian +forum +monthly +lisa +patterns +gmina +##tory +madison +hurricane +rev +##ians +bristol +##ula +elite +valuable +disaster +democracy +awareness +germans +freyja +##ins +loop +absolutely +paying +populations +maine +sole +prayer +spencer +releases +doorway +bull +##ani +lover +midnight +conclusion +##sson +thirteen +lily +mediterranean +##lt +nhl +proud +sample +##hill +drummer +guinea +##ova +murphy +climb +##ston +instant +attributed +horn +ain +railways +steven +##ao +autumn +ferry +opponent +root +traveling +secured +corridor +stretched +tales +sheet +trinity +cattle +helps +indicates +manhattan +murdered +fitted +1882 +gentle +grandmother +mines +shocked +vegas +produces +##light +caribbean +##ou +belong +continuous +desperate +drunk +historically +trio +waved +raf +dealing +nathan +bat +murmured +interrupted +residing +scientist +pioneer +harold +aaron +##net +delta +attempting +minority +mini +believes +chorus +tend +lots +eyed +indoor +load +shots +updated +jail +##llo +concerning +connecting +wealth +##ved +slaves +arrive +rangers +sufficient +rebuilt +##wick +cardinal +flood +muhammad +whenever +relation +runners +moral +repair +viewers +arriving +revenge +punk +assisted +bath +fairly +breathe +lists +innings +illustrated +whisper +nearest +voters +clinton +ties +ultimate +screamed +beijing +lions +andre +fictional +gathering +comfort +radar +suitable +dismissed +hms +ban +pine +wrist +atmosphere +voivodeship +bid +timber +##ned +##nan +giants +##ane +cameron +recovery +uss +identical +categories +switched +serbia +laughter +noah +ensemble +therapy +peoples +touching +##off +locally +pearl +platforms +everywhere +ballet +tables +lanka +herbert +outdoor +toured +derek +1883 +spaces +contested +swept +1878 +exclusive +slight +connections +##dra +winds +prisoner +collective +bangladesh +tube +publicly +wealthy +thai +##ys +isolated +select +##ric +insisted +pen +fortune +ticket +spotted +reportedly +animation +enforcement +tanks +110 +decides +wider +lowest +owen +##time +nod +hitting +##hn +gregory +furthermore +magazines +fighters +solutions +##ery +pointing +requested +peru +reed +chancellor +knights +mask +worker +eldest +flames +reduction +1860 +volunteers +##tis +reporting +##hl +wire +advisory +endemic +origins +settlers +pursue +knock +consumer +1876 +eu +compound +creatures +mansion +sentenced +ivan +deployed +guitars +frowned +involves +mechanism +kilometers +perspective +shops +maps +terminus +duncan +alien +fist +bridges +##pers +heroes +fed +derby +swallowed +##ros +patent +sara +illness +characterized +adventures +slide +hawaii +jurisdiction +##op +organised +##side +adelaide +walks +biology +se +##ties +rogers +swing +tightly +boundaries +##rie +prepare +implementation +stolen +##sha +certified +colombia +edwards +garage +##mm +recalled +##ball +rage +harm +nigeria +breast +##ren +furniture +pupils +settle +##lus +cuba +balls +client +alaska +21st +linear +thrust +celebration +latino +genetic +terror +##cia +##ening +lightning +fee +witness +lodge +establishing +skull +##ique +earning +hood +##ei +rebellion +wang +sporting +warned +missile +devoted +activist +porch +worship +fourteen +package +1871 +decorated +##shire +housed +##ock +chess +sailed +doctors +oscar +joan +treat +garcia +harbour +jeremy +##ire +traditions +dominant +jacques +##gon +##wan +relocated +1879 +amendment +sized +companion +simultaneously +volleyball +spun +acre +increases +stopping +loves +belongs +affect +drafted +tossed +scout +battles +1875 +filming +shoved +munich +tenure +vertical +romance +pc +##cher +argue +##ical +craft +ranging +www +opens +honest +tyler +yesterday +virtual +##let +muslims +reveal +snake +immigrants +radical +screaming +speakers +firing +saving +belonging +ease +lighting +prefecture +blame +farmer +hungry +grows +rubbed +beam +sur +subsidiary +##cha +armenian +sao +dropping +conventional +##fer +microsoft +reply +qualify +spots +1867 +sweat +festivals +##ken +immigration +physician +discover +exposure +sandy +explanation +isaac +implemented +##fish +hart +initiated +connect +stakes +presents +heights +householder +pleased +tourist +regardless +slip +closest +##ction +surely +sultan +brings +riley +preparation +aboard +slammed +baptist +experiment +ongoing +interstate +organic +playoffs +##ika +1877 +130 +##tar +hindu +error +tours +tier +plenty +arrangements +talks +trapped +excited +sank +ho +athens +1872 +denver +welfare +suburb +athletes +trick +diverse +belly +exclusively +yelled +1868 +##med +conversion +##ette +1874 +internationally +computers +conductor +abilities +sensitive +hello +dispute +measured +globe +rocket +prices +amsterdam +flights +tigers +inn +municipalities +emotion +references +3d +##mus +explains +airlines +manufactured +pm +archaeological +1873 +interpretation +devon +comment +##ites +settlements +kissing +absolute +improvement +suite +impressed +barcelona +sullivan +jefferson +towers +jesse +julie +##tin +##lu +grandson +hi +gauge +regard +rings +interviews +trace +raymond +thumb +departments +burns +serial +bulgarian +scores +demonstrated +##ix +1866 +kyle +alberta +underneath +romanized +##ward +relieved +acquisition +phrase +cliff +reveals +han +cuts +merger +custom +##dar +nee +gilbert +graduation +##nts +assessment +cafe +difficulty +demands +swung +democrat +jennifer +commons +1940s +grove +##yo +completing +focuses +sum +substitute +bearing +stretch +reception +##py +reflected +essentially +destination +pairs +##ched +survival +resource +##bach +promoting +doubles +messages +tear +##down +##fully +parade +florence +harvey +incumbent +partial +framework +900 +pedro +frozen +procedure +olivia +controls +##mic +shelter +personally +temperatures +##od +brisbane +tested +sits +marble +comprehensive +oxygen +leonard +##kov +inaugural +iranian +referring +quarters +attitude +##ivity +mainstream +lined +mars +dakota +norfolk +unsuccessful +##° +explosion +helicopter +congressional +##sing +inspector +bitch +seal +departed +divine +##ters +coaching +examination +punishment +manufacturer +sink +columns +unincorporated +signals +nevada +squeezed +dylan +dining +photos +martial +manuel +eighteen +elevator +brushed +plates +ministers +ivy +congregation +##len +slept +specialized +taxes +curve +restricted +negotiations +likes +statistical +arnold +inspiration +execution +bold +intermediate +significance +margin +ruler +wheels +gothic +intellectual +dependent +listened +eligible +buses +widow +syria +earn +cincinnati +collapsed +recipient +secrets +accessible +philippine +maritime +goddess +clerk +surrender +breaks +playoff +database +##ified +##lon +ideal +beetle +aspect +soap +regulation +strings +expand +anglo +shorter +crosses +retreat +tough +coins +wallace +directions +pressing +##oon +shipping +locomotives +comparison +topics +nephew +##mes +distinction +honors +travelled +sierra +ibn +##over +fortress +sa +recognised +carved +1869 +clients +##dan +intent +##mar +coaches +describing +bread +##ington +beaten +northwestern +##ona +merit +youtube +collapse +challenges +em +historians +objective +submitted +virus +attacking +drake +assume +##ere +diseases +marc +stem +leeds +##cus +##ab +farming +glasses +##lock +visits +nowhere +fellowship +relevant +carries +restaurants +experiments +101 +constantly +bases +targets +shah +tenth +opponents +verse +territorial +##ira +writings +corruption +##hs +instruction +inherited +reverse +emphasis +##vic +employee +arch +keeps +rabbi +watson +payment +uh +##ala +nancy +##tre +venice +fastest +sexy +banned +adrian +properly +ruth +touchdown +dollar +boards +metre +circles +edges +favour +comments +ok +travels +liberation +scattered +firmly +##ular +holland +permitted +diesel +kenya +den +originated +##ral +demons +resumed +dragged +rider +##rus +servant +blinked +extend +torn +##ias +##sey +input +meal +everybody +cylinder +kinds +camps +##fe +bullet +logic +##wn +croatian +evolved +healthy +fool +chocolate +wise +preserve +pradesh +##ess +respective +1850 +##ew +chicken +artificial +gross +corresponding +convicted +cage +caroline +dialogue +##dor +narrative +stranger +mario +br +christianity +failing +trent +commanding +buddhist +1848 +maurice +focusing +yale +bike +altitude +##ering +mouse +revised +##sley +veteran +##ig +pulls +theology +crashed +campaigns +legion +##ability +drag +excellence +customer +cancelled +intensity +excuse +##lar +liga +participating +contributing +printing +##burn +variable +##rk +curious +bin +legacy +renaissance +##my +symptoms +binding +vocalist +dancer +##nie +grammar +gospel +democrats +ya +enters +sc +diplomatic +hitler +##ser +clouds +mathematical +quit +defended +oriented +##heim +fundamental +hardware +impressive +equally +convince +confederate +guilt +chuck +sliding +##ware +magnetic +narrowed +petersburg +bulgaria +otto +phd +skill +##ama +reader +hopes +pitcher +reservoir +hearts +automatically +expecting +mysterious +bennett +extensively +imagined +seeds +monitor +fix +##ative +journalism +struggling +signature +ranch +encounter +photographer +observation +protests +##pin +influences +##hr +calendar +##all +cruz +croatia +locomotive +hughes +naturally +shakespeare +basement +hook +uncredited +faded +theories +approaches +dare +phillips +filling +fury +obama +##ain +efficient +arc +deliver +min +raid +breeding +inducted +leagues +efficiency +axis +montana +eagles +##ked +supplied +instructions +karen +picking +indicating +trap +anchor +practically +christians +tomb +vary +occasional +electronics +lords +readers +newcastle +faint +innovation +collect +situations +engagement +160 +claude +mixture +##feld +peer +tissue +logo +lean +##ration +°f +floors +##ven +architects +reducing +##our +##ments +rope +1859 +ottawa +##har +samples +banking +declaration +proteins +resignation +francois +saudi +advocate +exhibited +armor +twins +divorce +##ras +abraham +reviewed +jo +temporarily +matrix +physically +pulse +curled +##ena +difficulties +bengal +usage +##ban +annie +riders +certificate +##pi +holes +warsaw +distinctive +jessica +##mon +mutual +1857 +customs +circular +eugene +removal +loaded +mere +vulnerable +depicted +generations +dame +heir +enormous +lightly +climbing +pitched +lessons +pilots +nepal +ram +google +preparing +brad +louise +renowned +##₂ +liam +##ably +plaza +shaw +sophie +brilliant +bills +##bar +##nik +fucking +mainland +server +pleasant +seized +veterans +jerked +fail +beta +brush +radiation +stored +warmth +southeastern +nate +sin +raced +berkeley +joke +athlete +designation +trunk +##low +roland +qualification +archives +heels +artwork +receives +judicial +reserves +##bed +woke +installation +abu +floating +fake +lesser +excitement +interface +concentrated +addressed +characteristic +amanda +saxophone +monk +auto +##bus +releasing +egg +dies +interaction +defender +ce +outbreak +glory +loving +##bert +sequel +consciousness +http +awake +ski +enrolled +##ress +handling +rookie +brow +somebody +biography +warfare +amounts +contracts +presentation +fabric +dissolved +challenged +meter +psychological +lt +elevated +rally +accurate +##tha +hospitals +undergraduate +specialist +venezuela +exhibit +shed +nursing +protestant +fluid +structural +footage +jared +consistent +prey +##ska +succession +reflect +exile +lebanon +wiped +suspect +shanghai +resting +integration +preservation +marvel +variant +pirates +sheep +rounded +capita +sailing +colonies +manuscript +deemed +variations +clarke +functional +emerging +boxing +relaxed +curse +azerbaijan +heavyweight +nickname +editorial +rang +grid +tightened +earthquake +flashed +miguel +rushing +##ches +improvements +boxes +brooks +180 +consumption +molecular +felix +societies +repeatedly +variation +aids +civic +graphics +professionals +realm +autonomous +receiver +delayed +workshop +militia +chairs +trump +canyon +##point +harsh +extending +lovely +happiness +##jan +stake +eyebrows +embassy +wellington +hannah +##ella +sony +corners +bishops +swear +cloth +contents +xi +namely +commenced +1854 +stanford +nashville +courage +graphic +commitment +garrison +##bin +hamlet +clearing +rebels +attraction +literacy +cooking +ruins +temples +jenny +humanity +celebrate +hasn +freight +sixty +rebel +bastard +##art +newton +##ada +deer +##ges +##ching +smiles +delaware +singers +##ets +approaching +assists +flame +##ph +boulevard +barrel +planted +##ome +pursuit +##sia +consequences +posts +shallow +invitation +rode +depot +ernest +kane +rod +concepts +preston +topic +chambers +striking +blast +arrives +descendants +montgomery +ranges +worlds +##lay +##ari +span +chaos +praise +##ag +fewer +1855 +sanctuary +mud +fbi +##ions +programmes +maintaining +unity +harper +bore +handsome +closure +tournaments +thunder +nebraska +linda +facade +puts +satisfied +argentine +dale +cork +dome +panama +##yl +1858 +tasks +experts +##ates +feeding +equation +##las +##ida +##tu +engage +bryan +##ax +um +quartet +melody +disbanded +sheffield +blocked +gasped +delay +kisses +maggie +connects +##non +sts +poured +creator +publishers +##we +guided +ellis +extinct +hug +gaining +##ord +complicated +##bility +poll +clenched +investigate +##use +thereby +quantum +spine +cdp +humor +kills +administered +semifinals +##du +encountered +ignore +##bu +commentary +##maker +bother +roosevelt +140 +plains +halfway +flowing +cultures +crack +imprisoned +neighboring +airline +##ses +##view +##mate +##ec +gather +wolves +marathon +transformed +##ill +cruise +organisations +carol +punch +exhibitions +numbered +alarm +ratings +daddy +silently +##stein +queens +colours +impression +guidance +liu +tactical +##rat +marshal +della +arrow +##ings +rested +feared +tender +owns +bitter +advisor +escort +##ides +spare +farms +grants +##ene +dragons +encourage +colleagues +cameras +##und +sucked +pile +spirits +prague +statements +suspension +landmark +fence +torture +recreation +bags +permanently +survivors +pond +spy +predecessor +bombing +coup +##og +protecting +transformation +glow +##lands +##book +dug +priests +andrea +feat +barn +jumping +##chen +##ologist +##con +casualties +stern +auckland +pipe +serie +revealing +ba +##bel +trevor +mercy +spectrum +yang +consist +governing +collaborated +possessed +epic +comprises +blew +shane +##ack +lopez +honored +magical +sacrifice +judgment +perceived +hammer +mtv +baronet +tune +das +missionary +sheets +350 +neutral +oral +threatening +attractive +shade +aims +seminary +##master +estates +1856 +michel +wounds +refugees +manufacturers +##nic +mercury +syndrome +porter +##iya +##din +hamburg +identification +upstairs +purse +widened +pause +cared +breathed +affiliate +santiago +prevented +celtic +fisher +125 +recruited +byzantine +reconstruction +farther +##mp +diet +sake +au +spite +sensation +##ert +blank +separation +105 +##hon +vladimir +armies +anime +##lie +accommodate +orbit +cult +sofia +archive +##ify +##box +founders +sustained +disorder +honours +northeastern +mia +crops +violet +threats +blanket +fires +canton +followers +southwestern +prototype +voyage +assignment +altered +moderate +protocol +pistol +##eo +questioned +brass +lifting +1852 +math +authored +##ual +doug +dimensional +dynamic +##san +1851 +pronounced +grateful +quest +uncomfortable +boom +presidency +stevens +relating +politicians +chen +barrier +quinn +diana +mosque +tribal +cheese +palmer +portions +sometime +chester +treasure +wu +bend +download +millions +reforms +registration +##osa +consequently +monitoring +ate +preliminary +brandon +invented +ps +eaten +exterior +intervention +ports +documented +log +displays +lecture +sally +favourite +##itz +vermont +lo +invisible +isle +breed +##ator +journalists +relay +speaks +backward +explore +midfielder +actively +stefan +procedures +cannon +blond +kenneth +centered +servants +chains +libraries +malcolm +essex +henri +slavery +##hal +facts +fairy +coached +cassie +cats +washed +cop +##fi +announcement +item +2000s +vinyl +activated +marco +frontier +growled +curriculum +##das +loyal +accomplished +leslie +ritual +kenny +##00 +vii +napoleon +hollow +hybrid +jungle +stationed +friedrich +counted +##ulated +platinum +theatrical +seated +col +rubber +glen +1840 +diversity +healing +extends +id +provisions +administrator +columbus +##oe +tributary +te +assured +org +##uous +prestigious +examined +lectures +grammy +ronald +associations +bailey +allan +essays +flute +believing +consultant +proceedings +travelling +1853 +kit +kerala +yugoslavia +buddy +methodist +##ith +burial +centres +batman +##nda +discontinued +bo +dock +stockholm +lungs +severely +##nk +citing +manga +##ugh +steal +mumbai +iraqi +robot +celebrity +bride +broadcasts +abolished +pot +joel +overhead +franz +packed +reconnaissance +johann +acknowledged +introduce +handled +doctorate +developments +drinks +alley +palestine +##nis +##aki +proceeded +recover +bradley +grain +patch +afford +infection +nationalist +legendary +##ath +interchange +virtually +gen +gravity +exploration +amber +vital +wishes +powell +doctrine +elbow +screenplay +##bird +contribute +indonesian +pet +creates +##com +enzyme +kylie +discipline +drops +manila +hunger +##ien +layers +suffer +fever +bits +monica +keyboard +manages +##hood +searched +appeals +##bad +testament +grande +reid +##war +beliefs +congo +##ification +##dia +si +requiring +##via +casey +1849 +regret +streak +rape +depends +syrian +sprint +pound +tourists +upcoming +pub +##xi +tense +##els +practiced +echo +nationwide +guild +motorcycle +liz +##zar +chiefs +desired +elena +bye +precious +absorbed +relatives +booth +pianist +##mal +citizenship +exhausted +wilhelm +##ceae +##hed +noting +quarterback +urge +hectares +##gue +ace +holly +##tal +blonde +davies +parked +sustainable +stepping +twentieth +airfield +galaxy +nest +chip +##nell +tan +shaft +paulo +requirement +##zy +paradise +tobacco +trans +renewed +vietnamese +##cker +##ju +suggesting +catching +holmes +enjoying +md +trips +colt +holder +butterfly +nerve +reformed +cherry +bowling +trailer +carriage +goodbye +appreciate +toy +joshua +interactive +enabled +involve +##kan +collar +determination +bunch +facebook +recall +shorts +superintendent +episcopal +frustration +giovanni +nineteenth +laser +privately +array +circulation +##ovic +armstrong +deals +painful +permit +discrimination +##wi +aires +retiring +cottage +ni +##sta +horizon +ellen +jamaica +ripped +fernando +chapters +playstation +patron +lecturer +navigation +behaviour +genes +georgian +export +solomon +rivals +swift +seventeen +rodriguez +princeton +independently +sox +1847 +arguing +entity +casting +hank +criteria +oakland +geographic +milwaukee +reflection +expanding +conquest +dubbed +##tv +halt +brave +brunswick +doi +arched +curtis +divorced +predominantly +somerset +streams +ugly +zoo +horrible +curved +buenos +fierce +dictionary +vector +theological +unions +handful +stability +chan +punjab +segments +##lly +altar +ignoring +gesture +monsters +pastor +##stone +thighs +unexpected +operators +abruptly +coin +compiled +associates +improving +migration +pin +##ose +compact +collegiate +reserved +##urs +quarterfinals +roster +restore +assembled +hurry +oval +##cies +1846 +flags +martha +##del +victories +sharply +##rated +argues +deadly +neo +drawings +symbols +performer +##iel +griffin +restrictions +editing +andrews +java +journals +arabia +compositions +dee +pierce +removing +hindi +casino +runway +civilians +minds +nasa +hotels +##zation +refuge +rent +retain +potentially +conferences +suburban +conducting +##tto +##tions +##tle +descended +massacre +##cal +ammunition +terrain +fork +souls +counts +chelsea +durham +drives +cab +##bank +perth +realizing +palestinian +finn +simpson +##dal +betty +##ule +moreover +particles +cardinals +tent +evaluation +extraordinary +##oid +inscription +##works +wednesday +chloe +maintains +panels +ashley +trucks +##nation +cluster +sunlight +strikes +zhang +##wing +dialect +canon +##ap +tucked +##ws +collecting +##mas +##can +##sville +maker +quoted +evan +franco +aria +buying +cleaning +eva +closet +provision +apollo +clinic +rat +##ez +necessarily +ac +##gle +##ising +venues +flipped +cent +spreading +trustees +checking +authorized +##sco +disappointed +##ado +notion +duration +trumpet +hesitated +topped +brussels +rolls +theoretical +hint +define +aggressive +repeat +wash +peaceful +optical +width +allegedly +mcdonald +strict +copyright +##illa +investors +mar +jam +witnesses +sounding +miranda +michelle +privacy +hugo +harmony +##pp +valid +lynn +glared +nina +102 +headquartered +diving +boarding +gibson +##ncy +albanian +marsh +routine +dealt +enhanced +er +intelligent +substance +targeted +enlisted +discovers +spinning +observations +pissed +smoking +rebecca +capitol +visa +varied +costume +seemingly +indies +compensation +surgeon +thursday +arsenal +westminster +suburbs +rid +anglican +##ridge +knots +foods +alumni +lighter +fraser +whoever +portal +scandal +##ray +gavin +advised +instructor +flooding +terrorist +##ale +teenage +interim +senses +duck +teen +thesis +abby +eager +overcome +##ile +newport +glenn +rises +shame +##cc +prompted +priority +forgot +bomber +nicolas +protective +360 +cartoon +katherine +breeze +lonely +trusted +henderson +richardson +relax +banner +candy +palms +remarkable +##rio +legends +cricketer +essay +ordained +edmund +rifles +trigger +##uri +##away +sail +alert +1830 +audiences +penn +sussex +siblings +pursued +indianapolis +resist +rosa +consequence +succeed +avoided +1845 +##ulation +inland +##tie +##nna +counsel +profession +chronicle +hurried +##una +eyebrow +eventual +bleeding +innovative +cure +##dom +committees +accounting +con +scope +hardy +heather +tenor +gut +herald +codes +tore +scales +wagon +##oo +luxury +tin +prefer +fountain +triangle +bonds +darling +convoy +dried +traced +beings +troy +accidentally +slam +findings +smelled +joey +lawyers +outcome +steep +bosnia +configuration +shifting +toll +brook +performers +lobby +philosophical +construct +shrine +aggregate +boot +cox +phenomenon +savage +insane +solely +reynolds +lifestyle +##ima +nationally +holdings +consideration +enable +edgar +mo +mama +##tein +fights +relegation +chances +atomic +hub +conjunction +awkward +reactions +currency +finale +kumar +underwent +steering +elaborate +gifts +comprising +melissa +veins +reasonable +sunshine +chi +solve +trails +inhabited +elimination +ethics +huh +ana +molly +consent +apartments +layout +marines +##ces +hunters +bulk +##oma +hometown +##wall +##mont +cracked +reads +neighbouring +withdrawn +admission +wingspan +damned +anthology +lancashire +brands +batting +forgive +cuban +awful +##lyn +104 +dimensions +imagination +##ade +dante +##ship +tracking +desperately +goalkeeper +##yne +groaned +workshops +confident +burton +gerald +milton +circus +uncertain +slope +copenhagen +sophia +fog +philosopher +portraits +accent +cycling +varying +gripped +larvae +garrett +specified +scotia +mature +luther +kurt +rap +##kes +aerial +750 +ferdinand +heated +es +transported +##shan +safely +nonetheless +##orn +##gal +motors +demanding +##sburg +startled +##brook +ally +generate +caps +ghana +stained +demo +mentions +beds +ap +afterward +diary +##bling +utility +##iro +richards +1837 +conspiracy +conscious +shining +footsteps +observer +cyprus +urged +loyalty +developer +probability +olive +upgraded +gym +miracle +insects +graves +1844 +ourselves +hydrogen +amazon +katie +tickets +poets +##pm +planes +##pan +prevention +witnessed +dense +jin +randy +tang +warehouse +monroe +bang +archived +elderly +investigations +alec +granite +mineral +conflicts +controlling +aboriginal +carlo +##zu +mechanics +stan +stark +rhode +skirt +est +##berry +bombs +respected +##horn +imposed +limestone +deny +nominee +memphis +grabbing +disabled +##als +amusement +aa +frankfurt +corn +referendum +varies +slowed +disk +firms +unconscious +incredible +clue +sue +##zhou +twist +##cio +joins +idaho +chad +developers +computing +destroyer +103 +mortal +tucker +kingston +choices +yu +carson +1800 +os +whitney +geneva +pretend +dimension +staged +plateau +maya +##une +freestyle +##bc +rovers +hiv +##ids +tristan +classroom +prospect +##hus +honestly +diploma +lied +thermal +auxiliary +feast +unlikely +iata +##tel +morocco +pounding +treasury +lithuania +considerably +1841 +dish +1812 +geological +matching +stumbled +destroying +marched +brien +advances +cake +nicole +belle +settling +measuring +directing +##mie +tuesday +bassist +capabilities +stunned +fraud +torpedo +##list +##phone +anton +wisdom +surveillance +ruined +##ulate +lawsuit +healthcare +theorem +halls +trend +aka +horizontal +dozens +acquire +lasting +swim +hawk +gorgeous +fees +vicinity +decrease +adoption +tactics +##ography +pakistani +##ole +draws +##hall +willie +burke +heath +algorithm +integral +powder +elliott +brigadier +jackie +tate +varieties +darker +##cho +lately +cigarette +specimens +adds +##ree +##ensis +##inger +exploded +finalist +cia +murders +wilderness +arguments +nicknamed +acceptance +onwards +manufacture +robertson +jets +tampa +enterprises +blog +loudly +composers +nominations +1838 +ai +malta +inquiry +automobile +hosting +viii +rays +tilted +grief +museums +strategies +furious +euro +equality +cohen +poison +surrey +wireless +governed +ridiculous +moses +##esh +##room +vanished +##ito +barnes +attract +morrison +istanbul +##iness +absent +rotation +petition +janet +##logical +satisfaction +custody +deliberately +observatory +comedian +surfaces +pinyin +novelist +strictly +canterbury +oslo +monks +embrace +ibm +jealous +photograph +continent +dorothy +marina +doc +excess +holden +allegations +explaining +stack +avoiding +lance +storyline +majesty +poorly +spike +dos +bradford +raven +travis +classics +proven +voltage +pillow +fists +butt +1842 +interpreted +##car +1839 +gage +telegraph +lens +promising +expelled +casual +collector +zones +##min +silly +nintendo +##kh +##bra +downstairs +chef +suspicious +afl +flies +vacant +uganda +pregnancy +condemned +lutheran +estimates +cheap +decree +saxon +proximity +stripped +idiot +deposits +contrary +presenter +magnus +glacier +im +offense +edwin +##ori +upright +##long +bolt +##ois +toss +geographical +##izes +environments +delicate +marking +abstract +xavier +nails +windsor +plantation +occurring +equity +saskatchewan +fears +drifted +sequences +vegetation +revolt +##stic +1843 +sooner +fusion +opposing +nato +skating +1836 +secretly +ruin +lease +##oc +edit +##nne +flora +anxiety +ruby +##ological +##mia +tel +bout +taxi +emmy +frost +rainbow +compounds +foundations +rainfall +assassination +nightmare +dominican +##win +achievements +deserve +orlando +intact +armenia +##nte +calgary +valentine +106 +marion +proclaimed +theodore +bells +courtyard +thigh +gonzalez +console +troop +minimal +monte +everyday +##ence +##if +supporter +terrorism +buck +openly +presbyterian +activists +carpet +##iers +rubbing +uprising +##yi +cute +conceived +legally +##cht +millennium +cello +velocity +ji +rescued +cardiff +1835 +rex +concentrate +senators +beard +rendered +glowing +battalions +scouts +competitors +sculptor +catalogue +arctic +ion +raja +bicycle +wow +glancing +lawn +##woman +gentleman +lighthouse +publish +predicted +calculated +##val +variants +##gne +strain +##ui +winston +deceased +##nus +touchdowns +brady +caleb +sinking +echoed +crush +hon +blessed +protagonist +hayes +endangered +magnitude +editors +##tine +estimate +responsibilities +##mel +backup +laying +consumed +sealed +zurich +lovers +frustrated +##eau +ahmed +kicking +mit +treasurer +1832 +biblical +refuse +terrified +pump +agrees +genuine +imprisonment +refuses +plymouth +##hen +lou +##nen +tara +trembling +antarctic +ton +learns +##tas +crap +crucial +faction +atop +##borough +wrap +lancaster +odds +hopkins +erik +lyon +##eon +bros +##ode +snap +locality +tips +empress +crowned +cal +acclaimed +chuckled +##ory +clara +sends +mild +towel +##fl +##day +##а +wishing +assuming +interviewed +##bal +##die +interactions +eden +cups +helena +##lf +indie +beck +##fire +batteries +filipino +wizard +parted +##lam +traces +##born +rows +idol +albany +delegates +##ees +##sar +discussions +##ex +notre +instructed +belgrade +highways +suggestion +lauren +possess +orientation +alexandria +abdul +beats +salary +reunion +ludwig +alright +wagner +intimate +pockets +slovenia +hugged +brighton +merchants +cruel +stole +trek +slopes +repairs +enrollment +politically +underlying +promotional +counting +boeing +##bb +isabella +naming +##и +keen +bacteria +listing +separately +belfast +ussr +450 +lithuanian +anybody +ribs +sphere +martinez +cock +embarrassed +proposals +fragments +nationals +##fs +##wski +premises +fin +1500 +alpine +matched +freely +bounded +jace +sleeve +##af +gaming +pier +populated +evident +##like +frances +flooded +##dle +frightened +pour +trainer +framed +visitor +challenging +pig +wickets +##fold +infected +email +##pes +arose +##aw +reward +ecuador +oblast +vale +ch +shuttle +##usa +bach +rankings +forbidden +cornwall +accordance +salem +consumers +bruno +fantastic +toes +machinery +resolved +julius +remembering +propaganda +iceland +bombardment +tide +contacts +wives +##rah +concerto +macdonald +albania +implement +daisy +tapped +sudan +helmet +angela +mistress +##lic +crop +sunk +finest +##craft +hostile +##ute +##tsu +boxer +fr +paths +adjusted +habit +ballot +supervision +soprano +##zen +bullets +wicked +sunset +regiments +disappear +lamp +performs +app +##gia +##oa +rabbit +digging +incidents +entries +##cion +dishes +##oi +introducing +##ati +##fied +freshman +slot +jill +tackles +baroque +backs +##iest +lone +sponsor +destiny +altogether +convert +##aro +consensus +shapes +demonstration +basically +feminist +auction +artifacts +##bing +strongest +twitter +halifax +2019 +allmusic +mighty +smallest +precise +alexandra +viola +##los +##ille +manuscripts +##illo +dancers +ari +managers +monuments +blades +barracks +springfield +maiden +consolidated +electron +##end +berry +airing +wheat +nobel +inclusion +blair +payments +geography +bee +cc +eleanor +react +##hurst +afc +manitoba +##yu +su +lineup +fitness +recreational +investments +airborne +disappointment +##dis +edmonton +viewing +##row +renovation +##cast +infant +bankruptcy +roses +aftermath +pavilion +##yer +carpenter +withdrawal +ladder +##hy +discussing +popped +reliable +agreements +rochester +##abad +curves +bombers +220 +rao +reverend +decreased +choosing +107 +stiff +consulting +naples +crawford +tracy +ka +ribbon +cops +##lee +crushed +deciding +unified +teenager +accepting +flagship +explorer +poles +sanchez +inspection +revived +skilled +induced +exchanged +flee +locals +tragedy +swallow +loading +hanna +demonstrate +##ela +salvador +flown +contestants +civilization +##ines +wanna +rhodes +fletcher +hector +knocking +considers +##ough +nash +mechanisms +sensed +mentally +walt +unclear +##eus +renovated +madame +##cks +crews +governmental +##hin +undertaken +monkey +##ben +##ato +fatal +armored +copa +caves +governance +grasp +perception +certification +froze +damp +tugged +wyoming +##rg +##ero +newman +##lor +nerves +curiosity +graph +115 +##ami +withdraw +tunnels +dull +meredith +moss +exhibits +neighbors +communicate +accuracy +explored +raiders +republicans +secular +kat +superman +penny +criticised +##tch +freed +update +conviction +wade +ham +likewise +delegation +gotta +doll +promises +technological +myth +nationality +resolve +convent +##mark +sharon +dig +sip +coordinator +entrepreneur +fold +##dine +capability +councillor +synonym +blown +swan +cursed +1815 +jonas +haired +sofa +canvas +keeper +rivalry +##hart +rapper +speedway +swords +postal +maxwell +estonia +potter +recurring +##nn +##ave +errors +##oni +cognitive +1834 +##² +claws +nadu +roberto +bce +wrestler +ellie +##ations +infinite +ink +##tia +presumably +finite +staircase +108 +noel +patricia +nacional +##cation +chill +eternal +tu +preventing +prussia +fossil +limbs +##logist +ernst +frog +perez +rene +##ace +pizza +prussian +##ios +##vy +molecules +regulatory +answering +opinions +sworn +lengths +supposedly +hypothesis +upward +habitats +seating +ancestors +drank +yield +hd +synthesis +researcher +modest +##var +mothers +peered +voluntary +homeland +##the +acclaim +##igan +static +valve +luxembourg +alto +carroll +fe +receptor +norton +ambulance +##tian +johnston +catholics +depicting +jointly +elephant +gloria +mentor +badge +ahmad +distinguish +remarked +councils +precisely +allison +advancing +detection +crowded +##10 +cooperative +ankle +mercedes +dagger +surrendered +pollution +commit +subway +jeffrey +lesson +sculptures +provider +##fication +membrane +timothy +rectangular +fiscal +heating +teammate +basket +particle +anonymous +deployment +##ple +missiles +courthouse +proportion +shoe +sec +##ller +complaints +forbes +blacks +abandon +remind +sizes +overwhelming +autobiography +natalie +##awa +risks +contestant +countryside +babies +scorer +invaded +enclosed +proceed +hurling +disorders +##cu +reflecting +continuously +cruiser +graduates +freeway +investigated +ore +deserved +maid +blocking +phillip +jorge +shakes +dove +mann +variables +lacked +burden +accompanying +que +consistently +organizing +provisional +complained +endless +##rm +tubes +juice +georges +krishna +mick +labels +thriller +##uch +laps +arcade +sage +snail +##table +shannon +fi +laurence +seoul +vacation +presenting +hire +churchill +surprisingly +prohibited +savannah +technically +##oli +170 +##lessly +testimony +suited +speeds +toys +romans +mlb +flowering +measurement +talented +kay +settings +charleston +expectations +shattered +achieving +triumph +ceremonies +portsmouth +lanes +mandatory +loser +stretching +cologne +realizes +seventy +cornell +careers +webb +##ulating +americas +budapest +ava +suspicion +##ison +yo +conrad +##hai +sterling +jessie +rector +##az +1831 +transform +organize +loans +christine +volcanic +warrant +slender +summers +subfamily +newer +danced +dynamics +rhine +proceeds +heinrich +gastropod +commands +sings +facilitate +easter +ra +positioned +responses +expense +fruits +yanked +imported +25th +velvet +vic +primitive +tribune +baldwin +neighbourhood +donna +rip +hay +pr +##uro +1814 +espn +welcomed +##aria +qualifier +glare +highland +timing +##cted +shells +eased +geometry +louder +exciting +slovakia +##sion +##iz +##lot +savings +prairie +##ques +marching +rafael +tonnes +##lled +curtain +preceding +shy +heal +greene +worthy +##pot +detachment +bury +sherman +##eck +reinforced +seeks +bottles +contracted +duchess +outfit +walsh +##sc +mickey +##ase +geoffrey +archer +squeeze +dawson +eliminate +invention +##enberg +neal +##eth +stance +dealer +coral +maple +retire +polo +simplified +##ht +1833 +hid +watts +backwards +jules +##oke +genesis +mt +frames +rebounds +burma +woodland +moist +santos +whispers +drained +subspecies +##aa +streaming +ulster +burnt +correspondence +maternal +gerard +denis +stealing +##load +genius +duchy +##oria +inaugurated +momentum +suits +placement +sovereign +clause +thames +##hara +confederation +reservation +sketch +yankees +lets +rotten +charm +hal +verses +ultra +commercially +dot +salon +citation +adopt +winnipeg +mist +allocated +cairo +##boy +jenkins +interference +objectives +##wind +1820 +portfolio +armoured +sectors +##eh +initiatives +##world +integrity +exercises +robe +tap +ab +gazed +##tones +distracted +rulers +111 +favorable +jerome +tended +cart +factories +##eri +diplomat +valued +gravel +charitable +##try +calvin +exploring +chang +shepherd +terrace +pdf +pupil +##ural +reflects +ups +##rch +governors +shelf +depths +##nberg +trailed +crest +tackle +##nian +##ats +hatred +##kai +clare +makers +ethiopia +longtime +detected +embedded +lacking +slapped +rely +thomson +anticipation +iso +morton +successive +agnes +screenwriter +straightened +philippe +playwright +haunted +licence +iris +intentions +sutton +112 +logical +correctly +##weight +branded +licked +tipped +silva +ricky +narrator +requests +##ents +greeted +supernatural +cow +##wald +lung +refusing +employer +strait +gaelic +liner +##piece +zoe +sabha +##mba +driveway +harvest +prints +bates +reluctantly +threshold +algebra +ira +wherever +coupled +240 +assumption +picks +##air +designers +raids +gentlemen +##ean +roller +blowing +leipzig +locks +screw +dressing +strand +##lings +scar +dwarf +depicts +##nu +nods +##mine +differ +boris +##eur +yuan +flip +##gie +mob +invested +questioning +applying +##ture +shout +##sel +gameplay +blamed +illustrations +bothered +weakness +rehabilitation +##of +##zes +envelope +rumors +miners +leicester +subtle +kerry +##ico +ferguson +##fu +premiership +ne +##cat +bengali +prof +catches +remnants +dana +##rily +shouting +presidents +baltic +ought +ghosts +dances +sailors +shirley +fancy +dominic +##bie +madonna +##rick +bark +buttons +gymnasium +ashes +liver +toby +oath +providence +doyle +evangelical +nixon +cement +carnegie +embarked +hatch +surroundings +guarantee +needing +pirate +essence +##bee +filter +crane +hammond +projected +immune +percy +twelfth +##ult +regent +doctoral +damon +mikhail +##ichi +lu +critically +elect +realised +abortion +acute +screening +mythology +steadily +##fc +frown +nottingham +kirk +wa +minneapolis +##rra +module +algeria +mc +nautical +encounters +surprising +statues +availability +shirts +pie +alma +brows +munster +mack +soup +crater +tornado +sanskrit +cedar +explosive +bordered +dixon +planets +stamp +exam +happily +##bble +carriers +kidnapped +##vis +accommodation +emigrated +##met +knockout +correspondent +violation +profits +peaks +lang +specimen +agenda +ancestry +pottery +spelling +equations +obtaining +ki +linking +1825 +debris +asylum +##20 +buddhism +teddy +##ants +gazette +##nger +##sse +dental +eligibility +utc +fathers +averaged +zimbabwe +francesco +coloured +hissed +translator +lynch +mandate +humanities +mackenzie +uniforms +lin +##iana +##gio +asset +mhz +fitting +samantha +genera +wei +rim +beloved +shark +riot +entities +expressions +indo +carmen +slipping +owing +abbot +neighbor +sidney +##av +rats +recommendations +encouraging +squadrons +anticipated +commanders +conquered +##oto +donations +diagnosed +##mond +divide +##iva +guessed +decoration +vernon +auditorium +revelation +conversations +##kers +##power +herzegovina +dash +alike +protested +lateral +herman +accredited +mg +##gent +freeman +mel +fiji +crow +crimson +##rine +livestock +##pped +humanitarian +bored +oz +whip +##lene +##ali +legitimate +alter +grinning +spelled +anxious +oriental +wesley +##nin +##hole +carnival +controller +detect +##ssa +bowed +educator +kosovo +macedonia +##sin +occupy +mastering +stephanie +janeiro +para +unaware +nurses +noon +135 +cam +hopefully +ranger +combine +sociology +polar +rica +##eer +neill +##sman +holocaust +##ip +doubled +lust +1828 +109 +decent +cooling +unveiled +##card +1829 +nsw +homer +chapman +meyer +##gin +dive +mae +reagan +expertise +##gled +darwin +brooke +sided +prosecution +investigating +comprised +petroleum +genres +reluctant +differently +trilogy +johns +vegetables +corpse +highlighted +lounge +pension +unsuccessfully +elegant +aided +ivory +beatles +amelia +cain +dubai +sunny +immigrant +babe +click +##nder +underwater +pepper +combining +mumbled +atlas +horns +accessed +ballad +physicians +homeless +gestured +rpm +freak +louisville +corporations +patriots +prizes +rational +warn +modes +decorative +overnight +din +troubled +phantom +##ort +monarch +sheer +##dorf +generals +guidelines +organs +addresses +##zon +enhance +curling +parishes +cord +##kie +linux +caesar +deutsche +bavaria +##bia +coleman +cyclone +##eria +bacon +petty +##yama +##old +hampton +diagnosis +1824 +throws +complexity +rita +disputed +##₃ +pablo +##sch +marketed +trafficking +##ulus +examine +plague +formats +##oh +vault +faithful +##bourne +webster +##ox +highlights +##ient +##ann +phones +vacuum +sandwich +modeling +##gated +bolivia +clergy +qualities +isabel +##nas +##ars +wears +screams +reunited +annoyed +bra +##ancy +##rate +differential +transmitter +tattoo +container +poker +##och +excessive +resides +cowboys +##tum +augustus +trash +providers +statute +retreated +balcony +reversed +void +storey +preceded +masses +leap +laughs +neighborhoods +wards +schemes +falcon +santo +battlefield +pad +ronnie +thread +lesbian +venus +##dian +beg +sandstone +daylight +punched +gwen +analog +stroked +wwe +acceptable +measurements +dec +toxic +##kel +adequate +surgical +economist +parameters +varsity +##sberg +quantity +ella +##chy +##rton +countess +generating +precision +diamonds +expressway +ga +##ı +1821 +uruguay +talents +galleries +expenses +scanned +colleague +outlets +ryder +lucien +##ila +paramount +##bon +syracuse +dim +fangs +gown +sweep +##sie +toyota +missionaries +websites +##nsis +sentences +adviser +val +trademark +spells +##plane +patience +starter +slim +##borg +toe +incredibly +shoots +elliot +nobility +##wyn +cowboy +endorsed +gardner +tendency +persuaded +organisms +emissions +kazakhstan +amused +boring +chips +themed +##hand +llc +constantinople +chasing +systematic +guatemala +borrowed +erin +carey +##hard +highlands +struggles +1810 +##ifying +##ced +wong +exceptions +develops +enlarged +kindergarten +castro +##ern +##rina +leigh +zombie +juvenile +##most +consul +##nar +sailor +hyde +clarence +intensive +pinned +nasty +useless +jung +clayton +stuffed +exceptional +ix +apostolic +230 +transactions +##dge +exempt +swinging +cove +religions +##ash +shields +dairy +bypass +190 +pursuing +bug +joyce +bombay +chassis +southampton +chat +interact +redesignated +##pen +nascar +pray +salmon +rigid +regained +malaysian +grim +publicity +constituted +capturing +toilet +delegate +purely +tray +drift +loosely +striker +weakened +trinidad +mitch +itv +defines +transmitted +ming +scarlet +nodding +fitzgerald +fu +narrowly +sp +tooth +standings +virtue +##₁ +##wara +##cting +chateau +gloves +lid +##nel +hurting +conservatory +##pel +sinclair +reopened +sympathy +nigerian +strode +advocated +optional +chronic +discharge +##rc +suck +compatible +laurel +stella +shi +fails +wage +dodge +128 +informal +sorts +levi +buddha +villagers +##aka +chronicles +heavier +summoned +gateway +3000 +eleventh +jewelry +translations +accordingly +seas +##ency +fiber +pyramid +cubic +dragging +##ista +caring +##ops +android +contacted +lunar +##dt +kai +lisbon +patted +1826 +sacramento +theft +madagascar +subtropical +disputes +ta +holidays +piper +willow +mare +cane +itunes +newfoundland +benny +companions +dong +raj +observe +roar +charming +plaque +tibetan +fossils +enacted +manning +bubble +tina +tanzania +##eda +##hir +funk +swamp +deputies +cloak +ufc +scenario +par +scratch +metals +anthem +guru +engaging +specially +##boat +dialects +nineteen +cecil +duet +disability +messenger +unofficial +##lies +defunct +eds +moonlight +drainage +surname +puzzle +honda +switching +conservatives +mammals +knox +broadcaster +sidewalk +cope +##ried +benson +princes +peterson +##sal +bedford +sharks +eli +wreck +alberto +gasp +archaeology +lgbt +teaches +securities +madness +compromise +waving +coordination +davidson +visions +leased +possibilities +eighty +jun +fernandez +enthusiasm +assassin +sponsorship +reviewer +kingdoms +estonian +laboratories +##fy +##nal +applies +verb +celebrations +##zzo +rowing +lightweight +sadness +submit +mvp +balanced +dude +##vas +explicitly +metric +magnificent +mound +brett +mohammad +mistakes +irregular +##hing +##ass +sanders +betrayed +shipped +surge +##enburg +reporters +termed +georg +pity +verbal +bulls +abbreviated +enabling +appealed +##are +##atic +sicily +sting +heel +sweetheart +bart +spacecraft +brutal +monarchy +##tter +aberdeen +cameo +diane +##ub +survivor +clyde +##aries +complaint +##makers +clarinet +delicious +chilean +karnataka +coordinates +1818 +panties +##rst +pretending +ar +dramatically +kiev +bella +tends +distances +113 +catalog +launching +instances +telecommunications +portable +lindsay +vatican +##eim +angles +aliens +marker +stint +screens +bolton +##rne +judy +wool +benedict +plasma +europa +spark +imaging +filmmaker +swiftly +##een +contributor +##nor +opted +stamps +apologize +financing +butter +gideon +sophisticated +alignment +avery +chemicals +yearly +speculation +prominence +professionally +##ils +immortal +institutional +inception +wrists +identifying +tribunal +derives +gains +##wo +papal +preference +linguistic +vince +operative +brewery +##ont +unemployment +boyd +##ured +##outs +albeit +prophet +1813 +bi +##rr +##face +##rad +quarterly +asteroid +cleaned +radius +temper +##llen +telugu +jerk +viscount +menu +##ote +glimpse +##aya +yacht +hawaiian +baden +##rl +laptop +readily +##gu +monetary +offshore +scots +watches +##yang +##arian +upgrade +needle +xbox +lea +encyclopedia +flank +fingertips +##pus +delight +teachings +confirm +roth +beaches +midway +winters +##iah +teasing +daytime +beverly +gambling +bonnie +##backs +regulated +clement +hermann +tricks +knot +##shing +##uring +##vre +detached +ecological +owed +specialty +byron +inventor +bats +stays +screened +unesco +midland +trim +affection +##ander +##rry +jess +thoroughly +feedback +##uma +chennai +strained +heartbeat +wrapping +overtime +pleaded +##sworth +mon +leisure +oclc +##tate +##ele +feathers +angelo +thirds +nuts +surveys +clever +gill +commentator +##dos +darren +rides +gibraltar +##nc +##mu +dissolution +dedication +shin +meals +saddle +elvis +reds +chaired +taller +appreciation +functioning +niece +favored +advocacy +robbie +criminals +suffolk +yugoslav +passport +constable +congressman +hastings +vera +##rov +consecrated +sparks +ecclesiastical +confined +##ovich +muller +floyd +nora +1822 +paved +1827 +cumberland +ned +saga +spiral +##flow +appreciated +yi +collaborative +treating +similarities +feminine +finishes +##ib +jade +import +##nse +##hot +champagne +mice +securing +celebrities +helsinki +attributes +##gos +cousins +phases +ache +lucia +gandhi +submission +vicar +spear +shine +tasmania +biting +detention +constitute +tighter +seasonal +##gus +terrestrial +matthews +##oka +effectiveness +parody +philharmonic +##onic +1816 +strangers +encoded +consortium +guaranteed +regards +shifts +tortured +collision +supervisor +inform +broader +insight +theaters +armour +emeritus +blink +incorporates +mapping +##50 +##ein +handball +flexible +##nta +substantially +generous +thief +##own +carr +loses +1793 +prose +ucla +romeo +generic +metallic +realization +damages +mk +commissioners +zach +default +##ther +helicopters +lengthy +stems +spa +partnered +spectators +rogue +indication +penalties +teresa +1801 +sen +##tric +dalton +##wich +irving +photographic +##vey +dell +deaf +peters +excluded +unsure +##vable +patterson +crawled +##zio +resided +whipped +latvia +slower +ecole +pipes +employers +maharashtra +comparable +va +textile +pageant +##gel +alphabet +binary +irrigation +chartered +choked +antoine +offs +waking +supplement +##wen +quantities +demolition +regain +locate +urdu +folks +alt +114 +##mc +scary +andreas +whites +##ava +classrooms +mw +aesthetic +publishes +valleys +guides +cubs +johannes +bryant +conventions +affecting +##itt +drain +awesome +isolation +prosecutor +ambitious +apology +captive +downs +atmospheric +lorenzo +aisle +beef +foul +##onia +kidding +composite +disturbed +illusion +natives +##ffer +emi +rockets +riverside +wartime +painters +adolf +melted +##ail +uncertainty +simulation +hawks +progressed +meantime +builder +spray +breach +unhappy +regina +russians +##urg +determining +##tation +tram +1806 +##quin +aging +##12 +1823 +garion +rented +mister +diaz +terminated +clip +1817 +depend +nervously +disco +owe +defenders +shiva +notorious +disbelief +shiny +worcester +##gation +##yr +trailing +undertook +islander +belarus +limitations +watershed +fuller +overlooking +utilized +raphael +1819 +synthetic +breakdown +klein +##nate +moaned +memoir +lamb +practicing +##erly +cellular +arrows +exotic +##graphy +witches +117 +charted +rey +hut +hierarchy +subdivision +freshwater +giuseppe +aloud +reyes +qatar +marty +sideways +utterly +sexually +jude +prayers +mccarthy +softball +blend +damien +##gging +##metric +wholly +erupted +lebanese +negro +revenues +tasted +comparative +teamed +transaction +labeled +maori +sovereignty +parkway +trauma +gran +malay +121 +advancement +descendant +2020 +buzz +salvation +inventory +symbolic +##making +antarctica +mps +##gas +##bro +mohammed +myanmar +holt +submarines +tones +##lman +locker +patriarch +bangkok +emerson +remarks +predators +kin +afghan +confession +norwich +rental +emerge +advantages +##zel +rca +##hold +shortened +storms +aidan +##matic +autonomy +compliance +##quet +dudley +atp +##osis +1803 +motto +documentation +summary +professors +spectacular +christina +archdiocese +flashing +innocence +remake +##dell +psychic +reef +scare +employ +rs +sticks +meg +gus +leans +##ude +accompany +bergen +tomas +##iko +doom +wages +pools +##nch +##bes +breasts +scholarly +alison +outline +brittany +breakthrough +willis +realistic +##cut +##boro +competitor +##stan +pike +picnic +icon +designing +commercials +washing +villain +skiing +micro +costumes +auburn +halted +executives +##hat +logistics +cycles +vowel +applicable +barrett +exclaimed +eurovision +eternity +ramon +##umi +##lls +modifications +sweeping +disgust +##uck +torch +aviv +ensuring +rude +dusty +sonic +donovan +outskirts +cu +pathway +##band +##gun +##lines +disciplines +acids +cadet +paired +##40 +sketches +##sive +marriages +##⁺ +folding +peers +slovak +implies +admired +##beck +1880s +leopold +instinct +attained +weston +megan +horace +##ination +dorsal +ingredients +evolutionary +##its +complications +deity +lethal +brushing +levy +deserted +institutes +posthumously +delivering +telescope +coronation +motivated +rapids +luc +flicked +pays +volcano +tanner +weighed +##nica +crowds +frankie +gifted +addressing +granddaughter +winding +##rna +constantine +gomez +##front +landscapes +rudolf +anthropology +slate +werewolf +##lio +astronomy +circa +rouge +dreaming +sack +knelt +drowned +naomi +prolific +tracked +freezing +herb +##dium +agony +randall +twisting +wendy +deposit +touches +vein +wheeler +##bbled +##bor +batted +retaining +tire +presently +compare +specification +daemon +nigel +##grave +merry +recommendation +czechoslovakia +sandra +ng +roma +##sts +lambert +inheritance +sheikh +winchester +cries +examining +##yle +comeback +cuisine +nave +##iv +ko +retrieve +tomatoes +barker +polished +defining +irene +lantern +personalities +begging +tract +swore +1809 +175 +##gic +omaha +brotherhood +##rley +haiti +##ots +exeter +##ete +##zia +steele +dumb +pearson +210 +surveyed +elisabeth +trends +##ef +fritz +##rf +premium +bugs +fraction +calmly +viking +##birds +tug +inserted +unusually +##ield +confronted +distress +crashing +brent +turks +resign +##olo +cambodia +gabe +sauce +##kal +evelyn +116 +extant +clusters +quarry +teenagers +luna +##lers +##ister +affiliation +drill +##ashi +panthers +scenic +libya +anita +strengthen +inscriptions +##cated +lace +sued +judith +riots +##uted +mint +##eta +preparations +midst +dub +challenger +##vich +mock +cf +displaced +wicket +breaths +enables +schmidt +analyst +##lum +ag +highlight +automotive +axe +josef +newark +sufficiently +resembles +50th +##pal +flushed +mum +traits +##ante +commodore +incomplete +warming +titular +ceremonial +ethical +118 +celebrating +eighteenth +cao +lima +medalist +mobility +strips +snakes +##city +miniature +zagreb +barton +escapes +umbrella +automated +doubted +differs +cooled +georgetown +dresden +cooked +fade +wyatt +rna +jacobs +carlton +abundant +stereo +boost +madras +inning +##hia +spur +ip +malayalam +begged +osaka +groan +escaping +charging +dose +vista +##aj +bud +papa +communists +advocates +edged +tri +##cent +resemble +peaking +necklace +fried +montenegro +saxony +goose +glances +stuttgart +curator +recruit +grocery +sympathetic +##tting +##fort +127 +lotus +randolph +ancestor +##rand +succeeding +jupiter +1798 +macedonian +##heads +hiking +1808 +handing +fischer +##itive +garbage +node +##pies +prone +singular +papua +inclined +attractions +italia +pouring +motioned +grandma +garnered +jacksonville +corp +ego +ringing +aluminum +##hausen +ordering +##foot +drawer +traders +synagogue +##play +##kawa +resistant +wandering +fragile +fiona +teased +var +hardcore +soaked +jubilee +decisive +exposition +mercer +poster +valencia +hale +kuwait +1811 +##ises +##wr +##eed +tavern +gamma +122 +johan +##uer +airways +amino +gil +##ury +vocational +domains +torres +##sp +generator +folklore +outcomes +##keeper +canberra +shooter +fl +beams +confrontation +##lling +##gram +feb +aligned +forestry +pipeline +jax +motorway +conception +decay +##tos +coffin +##cott +stalin +1805 +escorted +minded +##nam +sitcom +purchasing +twilight +veronica +additions +passive +tensions +straw +123 +frequencies +1804 +refugee +cultivation +##iate +christie +clary +bulletin +crept +disposal +##rich +##zong +processor +crescent +##rol +bmw +emphasized +whale +nazis +aurora +##eng +dwelling +hauled +sponsors +toledo +mega +ideology +theatres +tessa +cerambycidae +saves +turtle +cone +suspects +kara +rusty +yelling +greeks +mozart +shades +cocked +participant +##tro +shire +spit +freeze +necessity +##cos +inmates +nielsen +councillors +loaned +uncommon +omar +peasants +botanical +offspring +daniels +formations +jokes +1794 +pioneers +sigma +licensing +##sus +wheelchair +polite +1807 +liquor +pratt +trustee +##uta +forewings +balloon +##zz +kilometre +camping +explicit +casually +shawn +foolish +teammates +nm +hassan +carrie +judged +satisfy +vanessa +knives +selective +cnn +flowed +##lice +eclipse +stressed +eliza +mathematician +cease +cultivated +##roy +commissions +browns +##ania +destroyers +sheridan +meadow +##rius +minerals +##cial +downstream +clash +gram +memoirs +ventures +baha +seymour +archie +midlands +edith +fare +flynn +invite +canceled +tiles +stabbed +boulder +incorporate +amended +camden +facial +mollusk +unreleased +descriptions +yoga +grabs +550 +raises +ramp +shiver +##rose +coined +pioneering +tunes +qing +warwick +tops +119 +melanie +giles +##rous +wandered +##inal +annexed +nov +30th +unnamed +##ished +organizational +airplane +normandy +stoke +whistle +blessing +violations +chased +holders +shotgun +##ctic +outlet +reactor +##vik +tires +tearing +shores +fortified +mascot +constituencies +nc +columnist +productive +tibet +##rta +lineage +hooked +oct +tapes +judging +cody +##gger +hansen +kashmir +triggered +##eva +solved +cliffs +##tree +resisted +anatomy +protesters +transparent +implied +##iga +injection +mattress +excluding +##mbo +defenses +helpless +devotion +##elli +growl +liberals +weber +phenomena +atoms +plug +##iff +mortality +apprentice +howe +convincing +aaa +swimmer +barber +leone +promptly +sodium +def +nowadays +arise +##oning +gloucester +corrected +dignity +norm +erie +##ders +elders +evacuated +sylvia +compression +##yar +hartford +pose +backpack +reasoning +accepts +24th +wipe +millimetres +marcel +##oda +dodgers +albion +1790 +overwhelmed +aerospace +oaks +1795 +showcase +acknowledge +recovering +nolan +ashe +hurts +geology +fashioned +disappearance +farewell +swollen +shrug +marquis +wimbledon +124 +rue +1792 +commemorate +reduces +experiencing +inevitable +calcutta +intel +##court +murderer +sticking +fisheries +imagery +bloom +280 +brake +##inus +gustav +hesitation +memorable +po +viral +beans +accidents +tunisia +antenna +spilled +consort +treatments +aye +perimeter +##gard +donation +hostage +migrated +banker +addiction +apex +lil +trout +##ously +conscience +##nova +rams +sands +genome +passionate +troubles +##lets +##set +amid +##ibility +##ret +higgins +exceed +vikings +##vie +payne +##zan +muscular +##ste +defendant +sucking +##wal +ibrahim +fuselage +claudia +vfl +europeans +snails +interval +##garh +preparatory +statewide +tasked +lacrosse +viktor +##lation +angola +##hra +flint +implications +employs +teens +patrons +stall +weekends +barriers +scrambled +nucleus +tehran +jenna +parsons +lifelong +robots +displacement +5000 +##bles +precipitation +##gt +knuckles +clutched +1802 +marrying +ecology +marx +accusations +declare +scars +kolkata +mat +meadows +bermuda +skeleton +finalists +vintage +crawl +coordinate +affects +subjected +orchestral +mistaken +##tc +mirrors +dipped +relied +260 +arches +candle +##nick +incorporating +wildly +fond +basilica +owl +fringe +rituals +whispering +stirred +feud +tertiary +slick +goat +honorable +whereby +skip +ricardo +stripes +parachute +adjoining +submerged +synthesizer +##gren +intend +positively +ninety +phi +beaver +partition +fellows +alexis +prohibition +carlisle +bizarre +fraternity +##bre +doubts +icy +cbc +aquatic +sneak +sonny +combines +airports +crude +supervised +spatial +merge +alfonso +##bic +corrupt +scan +undergo +##ams +disabilities +colombian +comparing +dolphins +perkins +##lish +reprinted +unanimous +bounced +hairs +underworld +midwest +semester +bucket +paperback +miniseries +coventry +demise +##leigh +demonstrations +sensor +rotating +yan +##hler +arrange +soils +##idge +hyderabad +labs +##dr +brakes +grandchildren +##nde +negotiated +rover +ferrari +continuation +directorate +augusta +stevenson +counterpart +gore +##rda +nursery +rican +ave +collectively +broadly +pastoral +repertoire +asserted +discovering +nordic +styled +fiba +cunningham +harley +middlesex +survives +tumor +tempo +zack +aiming +lok +urgent +##rade +##nto +devils +##ement +contractor +turin +##wl +##ool +bliss +repaired +simmons +moan +astronomical +cr +negotiate +lyric +1890s +lara +bred +clad +angus +pbs +##ience +engineered +posed +##lk +hernandez +possessions +elbows +psychiatric +strokes +confluence +electorate +lifts +campuses +lava +alps +##ep +##ution +##date +physicist +woody +##page +##ographic +##itis +juliet +reformation +sparhawk +320 +complement +suppressed +jewel +##½ +floated +##kas +continuity +sadly +##ische +inability +melting +scanning +paula +flour +judaism +safer +vague +##lm +solving +curb +##stown +financially +gable +bees +expired +miserable +cassidy +dominion +1789 +cupped +145 +robbery +facto +amos +warden +resume +tallest +marvin +ing +pounded +usd +declaring +gasoline +##aux +darkened +270 +650 +sophomore +##mere +erection +gossip +televised +risen +dial +##eu +pillars +##link +passages +profound +##tina +arabian +ashton +silicon +nail +##ead +##lated +##wer +##hardt +fleming +firearms +ducked +circuits +blows +waterloo +titans +##lina +atom +fireplace +cheshire +financed +activation +algorithms +##zzi +constituent +catcher +cherokee +partnerships +sexuality +platoon +tragic +vivian +guarded +whiskey +meditation +poetic +##late +##nga +##ake +porto +listeners +dominance +kendra +mona +chandler +factions +22nd +salisbury +attitudes +derivative +##ido +##haus +intake +paced +javier +illustrator +barrels +bias +cockpit +burnett +dreamed +ensuing +##anda +receptors +someday +hawkins +mattered +##lal +slavic +1799 +jesuit +cameroon +wasted +tai +wax +lowering +victorious +freaking +outright +hancock +librarian +sensing +bald +calcium +myers +tablet +announcing +barack +shipyard +pharmaceutical +##uan +greenwich +flush +medley +patches +wolfgang +pt +speeches +acquiring +exams +nikolai +##gg +hayden +kannada +##type +reilly +##pt +waitress +abdomen +devastated +capped +pseudonym +pharmacy +fulfill +paraguay +1796 +clicked +##trom +archipelago +syndicated +##hman +lumber +orgasm +rejection +clifford +lorraine +advent +mafia +rodney +brock +##ght +##used +##elia +cassette +chamberlain +despair +mongolia +sensors +developmental +upstream +##eg +##alis +spanning +165 +trombone +basque +seeded +interred +renewable +rhys +leapt +revision +molecule +##ages +chord +vicious +nord +shivered +23rd +arlington +debts +corpus +sunrise +bays +blackburn +centimetres +##uded +shuddered +gm +strangely +gripping +cartoons +isabelle +orbital +##ppa +seals +proving +##lton +refusal +strengthened +bust +assisting +baghdad +batsman +portrayal +mara +pushes +spears +og +##cock +reside +nathaniel +brennan +1776 +confirmation +caucus +##worthy +markings +yemen +nobles +ku +lazy +viewer +catalan +encompasses +sawyer +##fall +sparked +substances +patents +braves +arranger +evacuation +sergio +persuade +dover +tolerance +penguin +cum +jockey +insufficient +townships +occupying +declining +plural +processed +projection +puppet +flanders +introduces +liability +##yon +gymnastics +antwerp +taipei +hobart +candles +jeep +wes +observers +126 +chaplain +bundle +glorious +##hine +hazel +flung +sol +excavations +dumped +stares +sh +bangalore +triangular +icelandic +intervals +expressing +turbine +##vers +songwriting +crafts +##igo +jasmine +ditch +rite +##ways +entertaining +comply +sorrow +wrestlers +basel +emirates +marian +rivera +helpful +##some +caution +downward +networking +##atory +##tered +darted +genocide +emergence +replies +specializing +spokesman +convenient +unlocked +fading +augustine +concentrations +resemblance +elijah +investigator +andhra +##uda +promotes +bean +##rrell +fleeing +wan +simone +announcer +##ame +##bby +lydia +weaver +132 +residency +modification +##fest +stretches +##ast +alternatively +nat +lowe +lacks +##ented +pam +tile +concealed +inferior +abdullah +residences +tissues +vengeance +##ided +moisture +peculiar +groove +zip +bologna +jennings +ninja +oversaw +zombies +pumping +batch +livingston +emerald +installations +1797 +peel +nitrogen +rama +##fying +##star +schooling +strands +responding +werner +##ost +lime +casa +accurately +targeting +##rod +underway +##uru +hemisphere +lester +##yard +occupies +2d +griffith +angrily +reorganized +##owing +courtney +deposited +##dd +##30 +estadio +##ifies +dunn +exiled +##ying +checks +##combe +##о +##fly +successes +unexpectedly +blu +assessed +##flower +##ه +observing +sacked +spiders +kn +##tail +mu +nodes +prosperity +audrey +divisional +155 +broncos +tangled +adjust +feeds +erosion +paolo +surf +directory +snatched +humid +admiralty +screwed +gt +reddish +##nese +modules +trench +lamps +bind +leah +bucks +competes +##nz +##form +transcription +##uc +isles +violently +clutching +pga +cyclist +inflation +flats +ragged +unnecessary +##hian +stubborn +coordinated +harriet +baba +disqualified +330 +insect +wolfe +##fies +reinforcements +rocked +duel +winked +embraced +bricks +##raj +hiatus +defeats +pending +brightly +jealousy +##xton +##hm +##uki +lena +gdp +colorful +##dley +stein +kidney +##shu +underwear +wanderers +##haw +##icus +guardians +m³ +roared +habits +##wise +permits +gp +uranium +punished +disguise +bundesliga +elise +dundee +erotic +partisan +pi +collectors +float +individually +rendering +behavioral +bucharest +ser +hare +valerie +corporal +nutrition +proportional +##isa +immense +##kis +pavement +##zie +##eld +sutherland +crouched +1775 +##lp +suzuki +trades +endurance +operas +crosby +prayed +priory +rory +socially +##urn +gujarat +##pu +walton +cube +pasha +privilege +lennon +floods +thorne +waterfall +nipple +scouting +approve +##lov +minorities +voter +dwight +extensions +assure +ballroom +slap +dripping +privileges +rejoined +confessed +demonstrating +patriotic +yell +investor +##uth +pagan +slumped +squares +##cle +##kins +confront +bert +embarrassment +##aid +aston +urging +sweater +starr +yuri +brains +williamson +commuter +mortar +structured +selfish +exports +##jon +cds +##him +unfinished +##rre +mortgage +destinations +##nagar +canoe +solitary +buchanan +delays +magistrate +fk +##pling +motivation +##lier +##vier +recruiting +assess +##mouth +malik +antique +1791 +pius +rahman +reich +tub +zhou +smashed +airs +galway +xii +conditioning +honduras +discharged +dexter +##pf +lionel +129 +debates +lemon +tiffany +volunteered +dom +dioxide +procession +devi +sic +tremendous +advertisements +colts +transferring +verdict +hanover +decommissioned +utter +relate +pac +racism +##top +beacon +limp +similarity +terra +occurrence +ant +##how +becky +capt +updates +armament +richie +pal +##graph +halloween +mayo +##ssen +##bone +cara +serena +fcc +dolls +obligations +##dling +violated +lafayette +jakarta +exploitation +##ime +infamous +iconic +##lah +##park +kitty +moody +reginald +dread +spill +crystals +olivier +modeled +bluff +equilibrium +separating +notices +ordnance +extinction +onset +cosmic +attachment +sammy +expose +privy +anchored +##bil +abbott +admits +bending +baritone +emmanuel +policeman +vaughan +winged +climax +dresses +denny +polytechnic +mohamed +burmese +authentic +nikki +genetics +grandparents +homestead +gaza +postponed +metacritic +una +##sby +##bat +unstable +dissertation +##rial +##cian +curls +obscure +uncovered +bronx +praying +disappearing +##hoe +prehistoric +coke +turret +mutations +nonprofit +pits +monaco +##ي +##usion +prominently +dispatched +podium +##mir +uci +##uation +133 +fortifications +birthplace +kendall +##lby +##oll +preacher +rack +goodman +##rman +persistent +##ott +countless +jaime +recorder +lexington +persecution +jumps +renewal +wagons +##11 +crushing +##holder +decorations +##lake +abundance +wrath +laundry +£1 +garde +##rp +jeanne +beetles +peasant +##sl +splitting +caste +sergei +##rer +##ema +scripts +##ively +rub +satellites +##vor +inscribed +verlag +scrapped +gale +packages +chick +potato +slogan +kathleen +arabs +##culture +counterparts +reminiscent +choral +##tead +rand +retains +bushes +dane +accomplish +courtesy +closes +##oth +slaughter +hague +krakow +lawson +tailed +elias +ginger +##ttes +canopy +betrayal +rebuilding +turf +##hof +frowning +allegiance +brigades +kicks +rebuild +polls +alias +nationalism +td +rowan +audition +bowie +fortunately +recognizes +harp +dillon +horrified +##oro +renault +##tics +ropes +##α +presumed +rewarded +infrared +wiping +accelerated +illustration +##rid +presses +practitioners +badminton +##iard +detained +##tera +recognizing +relates +misery +##sies +##tly +reproduction +piercing +potatoes +thornton +esther +manners +hbo +##aan +ours +bullshit +ernie +perennial +sensitivity +illuminated +rupert +##jin +##iss +##ear +rfc +nassau +##dock +staggered +socialism +##haven +appointments +nonsense +prestige +sharma +haul +##tical +solidarity +gps +##ook +##rata +igor +pedestrian +##uit +baxter +tenants +wires +medication +unlimited +guiding +impacts +diabetes +##rama +sasha +pas +clive +extraction +131 +continually +constraints +##bilities +sonata +hunted +sixteenth +chu +planting +quote +mayer +pretended +abs +spat +##hua +ceramic +##cci +curtains +pigs +pitching +##dad +latvian +sore +dayton +##sted +##qi +patrols +slice +playground +##nted +shone +stool +apparatus +inadequate +mates +treason +##ija +desires +##liga +##croft +somalia +laurent +mir +leonardo +oracle +grape +obliged +chevrolet +thirteenth +stunning +enthusiastic +##ede +accounted +concludes +currents +basil +##kovic +drought +##rica +mai +##aire +shove +posting +##shed +pilgrimage +humorous +packing +fry +pencil +wines +smells +144 +marilyn +aching +newest +clung +bon +neighbours +sanctioned +##pie +mug +##stock +drowning +##mma +hydraulic +##vil +hiring +reminder +lilly +investigators +##ncies +sour +##eous +compulsory +packet +##rion +##graphic +##elle +cannes +##inate +depressed +##rit +heroic +importantly +theresa +##tled +conway +saturn +marginal +rae +##xia +corresponds +royce +pact +jasper +explosives +packaging +aluminium +##ttered +denotes +rhythmic +spans +assignments +hereditary +outlined +originating +sundays +lad +reissued +greeting +beatrice +##dic +pillar +marcos +plots +handbook +alcoholic +judiciary +avant +slides +extract +masculine +blur +##eum +##force +homage +trembled +owens +hymn +trey +omega +signaling +socks +accumulated +reacted +attic +theo +lining +angie +distraction +primera +talbot +##key +1200 +ti +creativity +billed +##hey +deacon +eduardo +identifies +proposition +dizzy +gunner +hogan +##yam +##pping +##hol +ja +##chan +jensen +reconstructed +##berger +clearance +darius +##nier +abe +harlem +plea +dei +circled +emotionally +notation +fascist +neville +exceeded +upwards +viable +ducks +##fo +workforce +racer +limiting +shri +##lson +possesses +1600 +kerr +moths +devastating +laden +disturbing +locking +##cture +gal +fearing +accreditation +flavor +aide +1870s +mountainous +##baum +melt +##ures +motel +texture +servers +soda +##mb +herd +##nium +erect +puzzled +hum +peggy +examinations +gould +testified +geoff +ren +devised +sacks +##law +denial +posters +grunted +cesar +tutor +ec +gerry +offerings +byrne +falcons +combinations +ct +incoming +pardon +rocking +26th +avengers +flared +mankind +seller +uttar +loch +nadia +stroking +exposing +##hd +fertile +ancestral +instituted +##has +noises +prophecy +taxation +eminent +vivid +pol +##bol +dart +indirect +multimedia +notebook +upside +displaying +adrenaline +referenced +geometric +##iving +progression +##ddy +blunt +announce +##far +implementing +##lav +aggression +liaison +cooler +cares +headache +plantations +gorge +dots +impulse +thickness +ashamed +averaging +kathy +obligation +precursor +137 +fowler +symmetry +thee +225 +hears +##rai +undergoing +ads +butcher +bowler +##lip +cigarettes +subscription +goodness +##ically +browne +##hos +##tech +kyoto +donor +##erty +damaging +friction +drifting +expeditions +hardened +prostitution +152 +fauna +blankets +claw +tossing +snarled +butterflies +recruits +investigative +coated +healed +138 +communal +hai +xiii +academics +boone +psychologist +restless +lahore +stephens +mba +brendan +foreigners +printer +##pc +ached +explode +27th +deed +scratched +dared +##pole +cardiac +1780 +okinawa +proto +commando +compelled +oddly +electrons +##base +replica +thanksgiving +##rist +sheila +deliberate +stafford +tidal +representations +hercules +ou +##path +##iated +kidnapping +lenses +##tling +deficit +samoa +mouths +consuming +computational +maze +granting +smirk +razor +fixture +ideals +inviting +aiden +nominal +##vs +issuing +julio +pitt +ramsey +docks +##oss +exhaust +##owed +bavarian +draped +anterior +mating +ethiopian +explores +noticing +##nton +discarded +convenience +hoffman +endowment +beasts +cartridge +mormon +paternal +probe +sleeves +interfere +lump +deadline +##rail +jenks +bulldogs +scrap +alternating +justified +reproductive +nam +seize +descending +secretariat +kirby +coupe +grouped +smash +panther +sedan +tapping +##18 +lola +cheer +germanic +unfortunate +##eter +unrelated +##fan +subordinate +##sdale +suzanne +advertisement +##ility +horsepower +##lda +cautiously +discourse +luigi +##mans +##fields +noun +prevalent +mao +schneider +everett +surround +governorate +kira +##avia +westward +##take +misty +rails +sustainability +134 +unused +##rating +packs +toast +unwilling +regulate +thy +suffrage +nile +awe +assam +definitions +travelers +affordable +##rb +conferred +sells +undefeated +beneficial +torso +basal +repeating +remixes +##pass +bahrain +cables +fang +##itated +excavated +numbering +statutory +##rey +deluxe +##lian +forested +ramirez +derbyshire +zeus +slamming +transfers +astronomer +banana +lottery +berg +histories +bamboo +##uchi +resurrection +posterior +bowls +vaguely +##thi +thou +preserving +tensed +offence +##inas +meyrick +callum +ridden +watt +langdon +tying +lowland +snorted +daring +truman +##hale +##girl +aura +overly +filing +weighing +goa +infections +philanthropist +saunders +eponymous +##owski +latitude +perspectives +reviewing +mets +commandant +radial +##kha +flashlight +reliability +koch +vowels +amazed +ada +elaine +supper +##rth +##encies +predator +debated +soviets +cola +##boards +##nah +compartment +crooked +arbitrary +fourteenth +##ctive +havana +majors +steelers +clips +profitable +ambush +exited +packers +##tile +nude +cracks +fungi +##е +limb +trousers +josie +shelby +tens +frederic +##ος +definite +smoothly +constellation +insult +baton +discs +lingering +##nco +conclusions +lent +staging +becker +grandpa +shaky +##tron +einstein +obstacles +sk +adverse +elle +economically +##moto +mccartney +thor +dismissal +motions +readings +nostrils +treatise +##pace +squeezing +evidently +prolonged +1783 +venezuelan +je +marguerite +beirut +takeover +shareholders +##vent +denise +digit +airplay +norse +##bbling +imaginary +pills +hubert +blaze +vacated +eliminating +##ello +vine +mansfield +##tty +retrospective +barrow +borne +clutch +bail +forensic +weaving +##nett +##witz +desktop +citadel +promotions +worrying +dorset +ieee +subdivided +##iating +manned +expeditionary +pickup +synod +chuckle +185 +barney +##rz +##ffin +functionality +karachi +litigation +meanings +uc +lick +turbo +anders +##ffed +execute +curl +oppose +ankles +typhoon +##د +##ache +##asia +linguistics +compassion +pressures +grazing +perfection +##iting +immunity +monopoly +muddy +backgrounds +136 +namibia +francesca +monitors +attracting +stunt +tuition +##ии +vegetable +##mates +##quent +mgm +jen +complexes +forts +##ond +cellar +bites +seventeenth +royals +flemish +failures +mast +charities +##cular +peruvian +capitals +macmillan +ipswich +outward +frigate +postgraduate +folds +employing +##ouse +concurrently +fiery +##tai +contingent +nightmares +monumental +nicaragua +##kowski +lizard +mal +fielding +gig +reject +##pad +harding +##ipe +coastline +##cin +##nos +beethoven +humphrey +innovations +##tam +##nge +norris +doris +solicitor +huang +obey +141 +##lc +niagara +##tton +shelves +aug +bourbon +curry +nightclub +specifications +hilton +##ndo +centennial +dispersed +worm +neglected +briggs +sm +font +kuala +uneasy +plc +##nstein +##bound +##aking +##burgh +awaiting +pronunciation +##bbed +##quest +eh +optimal +zhu +raped +greens +presided +brenda +worries +##life +venetian +marxist +turnout +##lius +refined +braced +sins +grasped +sunderland +nickel +speculated +lowell +cyrillic +communism +fundraising +resembling +colonists +mutant +freddie +usc +##mos +gratitude +##run +mural +##lous +chemist +wi +reminds +28th +steals +tess +pietro +##ingen +promoter +ri +microphone +honoured +rai +sant +##qui +feather +##nson +burlington +kurdish +terrorists +deborah +sickness +##wed +##eet +hazard +irritated +desperation +veil +clarity +##rik +jewels +xv +##gged +##ows +##cup +berkshire +unfair +mysteries +orchid +winced +exhaustion +renovations +stranded +obe +infinity +##nies +adapt +redevelopment +thanked +registry +olga +domingo +noir +tudor +ole +##atus +commenting +behaviors +##ais +crisp +pauline +probable +stirling +wigan +##bian +paralympics +panting +surpassed +##rew +luca +barred +pony +famed +##sters +cassandra +waiter +carolyn +exported +##orted +andres +destructive +deeds +jonah +castles +vacancy +suv +##glass +1788 +orchard +yep +famine +belarusian +sprang +##forth +skinny +##mis +administrators +rotterdam +zambia +zhao +boiler +discoveries +##ride +##physics +lucius +disappointing +outreach +spoon +##frame +qualifications +unanimously +enjoys +regency +##iidae +stade +realism +veterinary +rodgers +dump +alain +chestnut +castile +censorship +rumble +gibbs +##itor +communion +reggae +inactivated +logs +loads +##houses +homosexual +##iano +ale +informs +##cas +phrases +plaster +linebacker +ambrose +kaiser +fascinated +850 +limerick +recruitment +forge +mastered +##nding +leinster +rooted +threaten +##strom +borneo +##hes +suggestions +scholarships +propeller +documentaries +patronage +coats +constructing +invest +neurons +comet +entirety +shouts +identities +annoying +unchanged +wary +##antly +##ogy +neat +oversight +##kos +phillies +replay +constance +##kka +incarnation +humble +skies +minus +##acy +smithsonian +##chel +guerrilla +jar +cadets +##plate +surplus +audit +##aru +cracking +joanna +louisa +pacing +##lights +intentionally +##iri +diner +nwa +imprint +australians +tong +unprecedented +bunker +naive +specialists +ark +nichols +railing +leaked +pedal +##uka +shrub +longing +roofs +v8 +captains +neural +tuned +##ntal +##jet +emission +medina +frantic +codex +definitive +sid +abolition +intensified +stocks +enrique +sustain +genoa +oxide +##written +clues +cha +##gers +tributaries +fragment +venom +##rity +##ente +##sca +muffled +vain +sire +laos +##ingly +##hana +hastily +snapping +surfaced +sentiment +motive +##oft +contests +approximate +mesa +luckily +dinosaur +exchanges +propelled +accord +bourne +relieve +tow +masks +offended +##ues +cynthia +##mmer +rains +bartender +zinc +reviewers +lois +##sai +legged +arrogant +rafe +rosie +comprise +handicap +blockade +inlet +lagoon +copied +drilling +shelley +petals +##inian +mandarin +obsolete +##inated +onward +arguably +productivity +cindy +praising +seldom +busch +discusses +raleigh +shortage +ranged +stanton +encouragement +firstly +conceded +overs +temporal +##uke +cbe +##bos +woo +certainty +pumps +##pton +stalked +##uli +lizzie +periodic +thieves +weaker +##night +gases +shoving +chooses +wc +##chemical +prompting +weights +##kill +robust +flanked +sticky +hu +tuberculosis +##eb +##eal +christchurch +resembled +wallet +reese +inappropriate +pictured +distract +fixing +fiddle +giggled +burger +heirs +hairy +mechanic +torque +apache +obsessed +chiefly +cheng +logging +##tag +extracted +meaningful +numb +##vsky +gloucestershire +reminding +##bay +unite +##lit +breeds +diminished +clown +glove +1860s +##ن +##ug +archibald +focal +freelance +sliced +depiction +##yk +organism +switches +sights +stray +crawling +##ril +lever +leningrad +interpretations +loops +anytime +reel +alicia +delighted +##ech +inhaled +xiv +suitcase +bernie +vega +licenses +northampton +exclusion +induction +monasteries +racecourse +homosexuality +##right +##sfield +##rky +dimitri +michele +alternatives +ions +commentators +genuinely +objected +pork +hospitality +fencing +stephan +warships +peripheral +wit +drunken +wrinkled +quentin +spends +departing +chung +numerical +spokesperson +##zone +johannesburg +caliber +killers +##udge +assumes +neatly +demographic +abigail +bloc +##vel +mounting +##lain +bentley +slightest +xu +recipients +##jk +merlin +##writer +seniors +prisons +blinking +hindwings +flickered +kappa +##hel +80s +strengthening +appealing +brewing +gypsy +mali +lashes +hulk +unpleasant +harassment +bio +treaties +predict +instrumentation +pulp +troupe +boiling +mantle +##ffe +ins +##vn +dividing +handles +verbs +##onal +coconut +senegal +340 +thorough +gum +momentarily +##sto +cocaine +panicked +destined +##turing +teatro +denying +weary +captained +mans +##hawks +##code +wakefield +bollywood +thankfully +##16 +cyril +##wu +amendments +##bahn +consultation +stud +reflections +kindness +1787 +internally +##ovo +tex +mosaic +distribute +paddy +seeming +143 +##hic +piers +##15 +##mura +##verse +popularly +winger +kang +sentinel +mccoy +##anza +covenant +##bag +verge +fireworks +suppress +thrilled +dominate +##jar +swansea +##60 +142 +reconciliation +##ndi +stiffened +cue +dorian +##uf +damascus +amor +ida +foremost +##aga +porsche +unseen +dir +##had +##azi +stony +lexi +melodies +##nko +angular +integer +podcast +ants +inherent +jaws +justify +persona +##olved +josephine +##nr +##ressed +customary +flashes +gala +cyrus +glaring +backyard +ariel +physiology +greenland +html +stir +avon +atletico +finch +methodology +ked +##lent +mas +catholicism +townsend +branding +quincy +fits +containers +1777 +ashore +aragon +##19 +forearm +poisoning +##sd +adopting +conquer +grinding +amnesty +keller +finances +evaluate +forged +lankan +instincts +##uto +guam +bosnian +photographed +workplace +desirable +protector +##dog +allocation +intently +encourages +willy +##sten +bodyguard +electro +brighter +##ν +bihar +##chev +lasts +opener +amphibious +sal +verde +arte +##cope +captivity +vocabulary +yields +##tted +agreeing +desmond +pioneered +##chus +strap +campaigned +railroads +##ович +emblem +##dre +stormed +501 +##ulous +marijuana +northumberland +##gn +##nath +bowen +landmarks +beaumont +##qua +danube +##bler +attorneys +th +ge +flyers +critique +villains +cass +mutation +acc +##0s +colombo +mckay +motif +sampling +concluding +syndicate +##rell +neon +stables +ds +warnings +clint +mourning +wilkinson +##tated +merrill +leopard +evenings +exhaled +emil +sonia +ezra +discrete +stove +farrell +fifteenth +prescribed +superhero +##rier +worms +helm +wren +##duction +##hc +expo +##rator +hq +unfamiliar +antony +prevents +acceleration +fiercely +mari +painfully +calculations +cheaper +ign +clifton +irvine +davenport +mozambique +##np +pierced +##evich +wonders +##wig +##cate +##iling +crusade +ware +##uel +enzymes +reasonably +mls +##coe +mater +ambition +bunny +eliot +kernel +##fin +asphalt +headmaster +torah +aden +lush +pins +waived +##care +##yas +joao +substrate +enforce +##grad +##ules +alvarez +selections +epidemic +tempted +##bit +bremen +translates +ensured +waterfront +29th +forrest +manny +malone +kramer +reigning +cookies +simpler +absorption +205 +engraved +##ffy +evaluated +1778 +haze +146 +comforting +crossover +##abe +thorn +##rift +##imo +##pop +suppression +fatigue +cutter +##tr +201 +wurttemberg +##orf +enforced +hovering +proprietary +gb +samurai +syllable +ascent +lacey +tick +lars +tractor +merchandise +rep +bouncing +defendants +##yre +huntington +##ground +##oko +standardized +##hor +##hima +assassinated +nu +predecessors +rainy +liar +assurance +lyrical +##uga +secondly +flattened +ios +parameter +undercover +##mity +bordeaux +punish +ridges +markers +exodus +inactive +hesitate +debbie +nyc +pledge +savoy +nagar +offset +organist +##tium +hesse +marin +converting +##iver +diagram +propulsion +pu +validity +reverted +supportive +##dc +ministries +clans +responds +proclamation +##inae +##ø +##rea +ein +pleading +patriot +sf +birch +islanders +strauss +hates +##dh +brandenburg +concession +rd +##ob +1900s +killings +textbook +antiquity +cinematography +wharf +embarrassing +setup +creed +farmland +inequality +centred +signatures +fallon +370 +##ingham +##uts +ceylon +gazing +directive +laurie +##tern +globally +##uated +##dent +allah +excavation +threads +##cross +148 +frantically +icc +utilize +determines +respiratory +thoughtful +receptions +##dicate +merging +chandra +seine +147 +builders +builds +diagnostic +dev +visibility +goddamn +analyses +dhaka +cho +proves +chancel +concurrent +curiously +canadians +pumped +restoring +1850s +turtles +jaguar +sinister +spinal +traction +declan +vows +1784 +glowed +capitalism +swirling +install +universidad +##lder +##oat +soloist +##genic +##oor +coincidence +beginnings +nissan +dip +resorts +caucasus +combustion +infectious +##eno +pigeon +serpent +##itating +conclude +masked +salad +jew +##gr +surreal +toni +##wc +harmonica +151 +##gins +##etic +##coat +fishermen +intending +bravery +##wave +klaus +titan +wembley +taiwanese +ransom +40th +incorrect +hussein +eyelids +jp +cooke +dramas +utilities +##etta +##print +eisenhower +principally +granada +lana +##rak +openings +concord +##bl +bethany +connie +morality +sega +##mons +##nard +earnings +##kara +##cine +wii +communes +##rel +coma +composing +softened +severed +grapes +##17 +nguyen +analyzed +warlord +hubbard +heavenly +behave +slovenian +##hit +##ony +hailed +filmmakers +trance +caldwell +skye +unrest +coward +likelihood +##aging +bern +sci +taliban +honolulu +propose +##wang +1700 +browser +imagining +cobra +contributes +dukes +instinctively +conan +violinist +##ores +accessories +gradual +##amp +quotes +sioux +##dating +undertake +intercepted +sparkling +compressed +139 +fungus +tombs +haley +imposing +rests +degradation +lincolnshire +retailers +wetlands +tulsa +distributor +dungeon +nun +greenhouse +convey +atlantis +aft +exits +oman +dresser +lyons +##sti +joking +eddy +judgement +omitted +digits +##cts +##game +juniors +##rae +cents +stricken +une +##ngo +wizards +weir +breton +nan +technician +fibers +liking +royalty +##cca +154 +persia +terribly +magician +##rable +##unt +vance +cafeteria +booker +camille +warmer +##static +consume +cavern +gaps +compass +contemporaries +foyer +soothing +graveyard +maj +plunged +blush +##wear +cascade +demonstrates +ordinance +##nov +boyle +##lana +rockefeller +shaken +banjo +izzy +##ense +breathless +vines +##32 +##eman +alterations +chromosome +dwellings +feudal +mole +153 +catalonia +relics +tenant +mandated +##fm +fridge +hats +honesty +patented +raul +heap +cruisers +accusing +enlightenment +infants +wherein +chatham +contractors +zen +affinity +hc +osborne +piston +156 +traps +maturity +##rana +lagos +##zal +peering +##nay +attendant +dealers +protocols +subset +prospects +biographical +##cre +artery +##zers +insignia +nuns +endured +##eration +recommend +schwartz +serbs +berger +cromwell +crossroads +##ctor +enduring +clasped +grounded +##bine +marseille +twitched +abel +choke +https +catalyst +moldova +italians +##tist +disastrous +wee +##oured +##nti +wwf +nope +##piration +##asa +expresses +thumbs +167 +##nza +coca +1781 +cheating +##ption +skipped +sensory +heidelberg +spies +satan +dangers +semifinal +202 +bohemia +whitish +confusing +shipbuilding +relies +surgeons +landings +ravi +baku +moor +suffix +alejandro +##yana +litre +upheld +##unk +rajasthan +##rek +coaster +insists +posture +scenarios +etienne +favoured +appoint +transgender +elephants +poked +greenwood +defences +fulfilled +militant +somali +1758 +chalk +potent +##ucci +migrants +wink +assistants +nos +restriction +activism +niger +##ario +colon +shaun +##sat +daphne +##erated +swam +congregations +reprise +considerations +magnet +playable +xvi +##р +overthrow +tobias +knob +chavez +coding +##mers +propped +katrina +orient +newcomer +##suke +temperate +##pool +farmhouse +interrogation +##vd +committing +##vert +forthcoming +strawberry +joaquin +macau +ponds +shocking +siberia +##cellular +chant +contributors +##nant +##ologists +sped +absorb +hail +1782 +spared +##hore +barbados +karate +opus +originates +saul +##xie +evergreen +leaped +##rock +correlation +exaggerated +weekday +unification +bump +tracing +brig +afb +pathways +utilizing +##ners +mod +mb +disturbance +kneeling +##stad +##guchi +100th +pune +##thy +decreasing +168 +manipulation +miriam +academia +ecosystem +occupational +rbi +##lem +rift +##14 +rotary +stacked +incorporation +awakening +generators +guerrero +racist +##omy +cyber +derivatives +culminated +allie +annals +panzer +sainte +wikipedia +pops +zu +austro +##vate +algerian +politely +nicholson +mornings +educate +tastes +thrill +dartmouth +##gating +db +##jee +regan +differing +concentrating +choreography +divinity +##media +pledged +alexandre +routing +gregor +madeline +##idal +apocalypse +##hora +gunfire +culminating +elves +fined +liang +lam +programmed +tar +guessing +transparency +gabrielle +##gna +cancellation +flexibility +##lining +accession +shea +stronghold +nets +specializes +##rgan +abused +hasan +sgt +ling +exceeding +##₄ +admiration +supermarket +##ark +photographers +specialised +tilt +resonance +hmm +perfume +380 +sami +threatens +garland +botany +guarding +boiled +greet +puppy +russo +supplier +wilmington +vibrant +vijay +##bius +paralympic +grumbled +paige +faa +licking +margins +hurricanes +##gong +fest +grenade +ripping +##uz +counseling +weigh +##sian +needles +wiltshire +edison +costly +##not +fulton +tramway +redesigned +staffordshire +cache +gasping +watkins +sleepy +candidacy +##group +monkeys +timeline +throbbing +##bid +##sos +berth +uzbekistan +vanderbilt +bothering +overturned +ballots +gem +##iger +sunglasses +subscribers +hooker +compelling +ang +exceptionally +saloon +stab +##rdi +carla +terrifying +rom +##vision +coil +##oids +satisfying +vendors +31st +mackay +deities +overlooked +ambient +bahamas +felipe +olympia +whirled +botanist +advertised +tugging +##dden +disciples +morales +unionist +rites +foley +morse +motives +creepy +##₀ +soo +##sz +bargain +highness +frightening +turnpike +tory +reorganization +##cer +depict +biographer +##walk +unopposed +manifesto +##gles +institut +emile +accidental +kapoor +##dam +kilkenny +cortex +lively +##13 +romanesque +jain +shan +cannons +##ood +##ske +petrol +echoing +amalgamated +disappears +cautious +proposes +sanctions +trenton +##ر +flotilla +aus +contempt +tor +canary +cote +theirs +##hun +conceptual +deleted +fascinating +paso +blazing +elf +honourable +hutchinson +##eiro +##outh +##zin +surveyor +tee +amidst +wooded +reissue +intro +##ono +cobb +shelters +newsletter +hanson +brace +encoding +confiscated +dem +caravan +marino +scroll +melodic +cows +imam +##adi +##aneous +northward +searches +biodiversity +cora +310 +roaring +##bers +connell +theologian +halo +compose +pathetic +unmarried +dynamo +##oot +az +calculation +toulouse +deserves +humour +nr +forgiveness +tam +undergone +martyr +pamela +myths +whore +counselor +hicks +290 +heavens +battleship +electromagnetic +##bbs +stellar +establishments +presley +hopped +##chin +temptation +90s +wills +nas +##yuan +nhs +##nya +seminars +##yev +adaptations +gong +asher +lex +indicator +sikh +tobago +cites +goin +##yte +satirical +##gies +characterised +correspond +bubbles +lure +participates +##vid +eruption +skate +therapeutic +1785 +canals +wholesale +defaulted +sac +460 +petit +##zzled +virgil +leak +ravens +256 +portraying +##yx +ghetto +creators +dams +portray +vicente +##rington +fae +namesake +bounty +##arium +joachim +##ota +##iser +aforementioned +axle +snout +depended +dismantled +reuben +480 +##ibly +gallagher +##lau +##pd +earnest +##ieu +##iary +inflicted +objections +##llar +asa +gritted +##athy +jericho +##sea +##was +flick +underside +ceramics +undead +substituted +195 +eastward +undoubtedly +wheeled +chimney +##iche +guinness +cb +##ager +siding +##bell +traitor +baptiste +disguised +inauguration +149 +tipperary +choreographer +perched +warmed +stationary +eco +##ike +##ntes +bacterial +##aurus +flores +phosphate +##core +attacker +invaders +alvin +intersects +a1 +indirectly +immigrated +businessmen +cornelius +valves +narrated +pill +sober +ul +nationale +monastic +applicants +scenery +##jack +161 +motifs +constitutes +cpu +##osh +jurisdictions +sd +tuning +irritation +woven +##uddin +fertility +gao +##erie +antagonist +impatient +glacial +hides +boarded +denominations +interception +##jas +cookie +nicola +##tee +algebraic +marquess +bahn +parole +buyers +bait +turbines +paperwork +bestowed +natasha +renee +oceans +purchases +157 +vaccine +215 +##tock +fixtures +playhouse +integrate +jai +oswald +intellectuals +##cky +booked +nests +mortimer +##isi +obsession +sept +##gler +##sum +440 +scrutiny +simultaneous +squinted +##shin +collects +oven +shankar +penned +remarkably +##я +slips +luggage +spectral +1786 +collaborations +louie +consolidation +##ailed +##ivating +420 +hoover +blackpool +harness +ignition +vest +tails +belmont +mongol +skinner +##nae +visually +mage +derry +##tism +##unce +stevie +transitional +##rdy +redskins +drying +prep +prospective +##21 +annoyance +oversee +##loaded +fills +##books +##iki +announces +fda +scowled +respects +prasad +mystic +tucson +##vale +revue +springer +bankrupt +1772 +aristotle +salvatore +habsburg +##geny +dal +natal +nut +pod +chewing +darts +moroccan +walkover +rosario +lenin +punjabi +##ße +grossed +scattering +wired +invasive +hui +polynomial +corridors +wakes +gina +portrays +##cratic +arid +retreating +erich +irwin +sniper +##dha +linen +lindsey +maneuver +butch +shutting +socio +bounce +commemorative +postseason +jeremiah +pines +275 +mystical +beads +bp +abbas +furnace +bidding +consulted +assaulted +empirical +rubble +enclosure +sob +weakly +cancel +polly +yielded +##emann +curly +prediction +battered +70s +vhs +jacqueline +render +sails +barked +detailing +grayson +riga +sloane +raging +##yah +herbs +bravo +##athlon +alloy +giggle +imminent +suffers +assumptions +waltz +##itate +accomplishments +##ited +bathing +remixed +deception +prefix +##emia +deepest +##tier +##eis +balkan +frogs +##rong +slab +##pate +philosophers +peterborough +grains +imports +dickinson +rwanda +##atics +1774 +dirk +lan +tablets +##rove +clone +##rice +caretaker +hostilities +mclean +##gre +regimental +treasures +norms +impose +tsar +tango +diplomacy +variously +complain +192 +recognise +arrests +1779 +celestial +pulitzer +##dus +bing +libretto +##moor +adele +splash +##rite +expectation +lds +confronts +##izer +spontaneous +harmful +wedge +entrepreneurs +buyer +##ope +bilingual +translate +rugged +conner +circulated +uae +eaton +##gra +##zzle +lingered +lockheed +vishnu +reelection +alonso +##oom +joints +yankee +headline +cooperate +heinz +laureate +invading +##sford +echoes +scandinavian +##dham +hugging +vitamin +salute +micah +hind +trader +##sper +radioactive +##ndra +militants +poisoned +ratified +remark +campeonato +deprived +wander +prop +##dong +outlook +##tani +##rix +##eye +chiang +darcy +##oping +mandolin +spice +statesman +babylon +182 +walled +forgetting +afro +##cap +158 +giorgio +buffer +##polis +planetary +##gis +overlap +terminals +kinda +centenary +##bir +arising +manipulate +elm +ke +1770 +ak +##tad +chrysler +mapped +moose +pomeranian +quad +macarthur +assemblies +shoreline +recalls +stratford +##rted +noticeable +##evic +imp +##rita +##sque +accustomed +supplying +tents +disgusted +vogue +sipped +filters +khz +reno +selecting +luftwaffe +mcmahon +tyne +masterpiece +carriages +collided +dunes +exercised +flare +remembers +muzzle +##mobile +heck +##rson +burgess +lunged +middleton +boycott +bilateral +##sity +hazardous +lumpur +multiplayer +spotlight +jackets +goldman +liege +porcelain +rag +waterford +benz +attracts +hopeful +battling +ottomans +kensington +baked +hymns +cheyenne +lattice +levine +borrow +polymer +clashes +michaels +monitored +commitments +denounced +##25 +##von +cavity +##oney +hobby +akin +##holders +futures +intricate +cornish +patty +##oned +illegally +dolphin +##lag +barlow +yellowish +maddie +apologized +luton +plagued +##puram +nana +##rds +sway +fanny +łodz +##rino +psi +suspicions +hanged +##eding +initiate +charlton +##por +nak +competent +235 +analytical +annex +wardrobe +reservations +##rma +sect +162 +fairfax +hedge +piled +buckingham +uneven +bauer +simplicity +snyder +interpret +accountability +donors +moderately +byrd +continents +##cite +##max +disciple +hr +jamaican +ping +nominees +##uss +mongolian +diver +attackers +eagerly +ideological +pillows +miracles +apartheid +revolver +sulfur +clinics +moran +163 +##enko +ile +katy +rhetoric +##icated +chronology +recycling +##hrer +elongated +mughal +pascal +profiles +vibration +databases +domination +##fare +##rant +matthias +digest +rehearsal +polling +weiss +initiation +reeves +clinging +flourished +impress +ngo +##hoff +##ume +buckley +symposium +rhythms +weed +emphasize +transforming +##taking +##gence +##yman +accountant +analyze +flicker +foil +priesthood +voluntarily +decreases +##80 +##hya +slater +sv +charting +mcgill +##lde +moreno +##iu +besieged +zur +robes +##phic +admitting +api +deported +turmoil +peyton +earthquakes +##ares +nationalists +beau +clair +brethren +interrupt +welch +curated +galerie +requesting +164 +##ested +impending +steward +viper +##vina +complaining +beautifully +brandy +foam +nl +1660 +##cake +alessandro +punches +laced +explanations +##lim +attribute +clit +reggie +discomfort +##cards +smoothed +whales +##cene +adler +countered +duffy +disciplinary +widening +recipe +reliance +conducts +goats +gradient +preaching +##shaw +matilda +quasi +striped +meridian +cannabis +cordoba +certificates +##agh +##tering +graffiti +hangs +pilgrims +repeats +##ych +revive +urine +etat +##hawk +fueled +belts +fuzzy +susceptible +##hang +mauritius +salle +sincere +beers +hooks +##cki +arbitration +entrusted +advise +sniffed +seminar +junk +donnell +processors +principality +strapped +celia +mendoza +everton +fortunes +prejudice +starving +reassigned +steamer +##lund +tuck +evenly +foreman +##ffen +dans +375 +envisioned +slit +##xy +baseman +liberia +rosemary +##weed +electrified +periodically +potassium +stride +contexts +sperm +slade +mariners +influx +bianca +subcommittee +##rane +spilling +icao +estuary +##nock +delivers +iphone +##ulata +isa +mira +bohemian +dessert +##sbury +welcoming +proudly +slowing +##chs +musee +ascension +russ +##vian +waits +##psy +africans +exploit +##morphic +gov +eccentric +crab +peck +##ull +entrances +formidable +marketplace +groom +bolted +metabolism +patton +robbins +courier +payload +endure +##ifier +andes +refrigerator +##pr +ornate +##uca +ruthless +illegitimate +masonry +strasbourg +bikes +adobe +##³ +apples +quintet +willingly +niche +bakery +corpses +energetic +##cliffe +##sser +##ards +177 +centimeters +centro +fuscous +cretaceous +rancho +##yde +andrei +telecom +tottenham +oasis +ordination +vulnerability +presiding +corey +cp +penguins +sims +##pis +malawi +piss +##48 +correction +##cked +##ffle +##ryn +countdown +detectives +psychiatrist +psychedelic +dinosaurs +blouse +##get +choi +vowed +##oz +randomly +##pol +49ers +scrub +blanche +bruins +dusseldorf +##using +unwanted +##ums +212 +dominique +elevations +headlights +om +laguna +##oga +1750 +famously +ignorance +shrewsbury +##aine +ajax +breuning +che +confederacy +greco +overhaul +##screen +paz +skirts +disagreement +cruelty +jagged +phoebe +shifter +hovered +viruses +##wes +mandy +##lined +##gc +landlord +squirrel +dashed +##ι +ornamental +gag +wally +grange +literal +spurs +undisclosed +proceeding +yin +##text +billie +orphan +spanned +humidity +indy +weighted +presentations +explosions +lucian +##tary +vaughn +hindus +##anga +##hell +psycho +171 +daytona +protects +efficiently +rematch +sly +tandem +##oya +rebranded +impaired +hee +metropolis +peach +godfrey +diaspora +ethnicity +prosperous +gleaming +dar +grossing +playback +##rden +stripe +pistols +##tain +births +labelled +##cating +172 +rudy +alba +##onne +aquarium +hostility +##gb +##tase +shudder +sumatra +hardest +lakers +consonant +creeping +demos +homicide +capsule +zeke +liberties +expulsion +pueblo +##comb +trait +transporting +##ddin +##neck +##yna +depart +gregg +mold +ledge +hangar +oldham +playboy +termination +analysts +gmbh +romero +##itic +insist +cradle +filthy +brightness +slash +shootout +deposed +bordering +##truct +isis +microwave +tumbled +sheltered +cathy +werewolves +messy +andersen +convex +clapped +clinched +satire +wasting +edo +vc +rufus +##jak +mont +##etti +poznan +##keeping +restructuring +transverse +##rland +azerbaijani +slovene +gestures +roommate +choking +shear +##quist +vanguard +oblivious +##hiro +disagreed +baptism +##lich +coliseum +##aceae +salvage +societe +cory +locke +relocation +relying +versailles +ahl +swelling +##elo +cheerful +##word +##edes +gin +sarajevo +obstacle +diverted +##nac +messed +thoroughbred +fluttered +utrecht +chewed +acquaintance +assassins +dispatch +mirza +##wart +nike +salzburg +swell +yen +##gee +idle +ligue +samson +##nds +##igh +playful +spawned +##cise +tease +##case +burgundy +##bot +stirring +skeptical +interceptions +marathi +##dies +bedrooms +aroused +pinch +##lik +preferences +tattoos +buster +digitally +projecting +rust +##ital +kitten +priorities +addison +pseudo +##guard +dusk +icons +sermon +##psis +##iba +bt +##lift +##xt +ju +truce +rink +##dah +##wy +defects +psychiatry +offences +calculate +glucose +##iful +##rized +##unda +francaise +##hari +richest +warwickshire +carly +1763 +purity +redemption +lending +##cious +muse +bruises +cerebral +aero +carving +##name +preface +terminology +invade +monty +##int +anarchist +blurred +##iled +rossi +treats +guts +shu +foothills +ballads +undertaking +premise +cecilia +affiliates +blasted +conditional +wilder +minors +drone +rudolph +buffy +swallowing +horton +attested +##hop +rutherford +howell +primetime +livery +penal +##bis +minimize +hydro +wrecked +wrought +palazzo +##gling +cans +vernacular +friedman +nobleman +shale +walnut +danielle +##ection +##tley +sears +##kumar +chords +lend +flipping +streamed +por +dracula +gallons +sacrifices +gamble +orphanage +##iman +mckenzie +##gible +boxers +daly +##balls +##ان +208 +##ific +##rative +##iq +exploited +slated +##uity +circling +hillary +pinched +goldberg +provost +campaigning +lim +piles +ironically +jong +mohan +successors +usaf +##tem +##ught +autobiographical +haute +preserves +##ending +acquitted +comparisons +203 +hydroelectric +gangs +cypriot +torpedoes +rushes +chrome +derive +bumps +instability +fiat +pets +##mbe +silas +dye +reckless +settler +##itation +info +heats +##writing +176 +canonical +maltese +fins +mushroom +stacy +aspen +avid +##kur +##loading +vickers +gaston +hillside +statutes +wilde +gail +kung +sabine +comfortably +motorcycles +##rgo +169 +pneumonia +fetch +##sonic +axel +faintly +parallels +##oop +mclaren +spouse +compton +interdisciplinary +miner +##eni +181 +clamped +##chal +##llah +separates +versa +##mler +scarborough +labrador +##lity +##osing +rutgers +hurdles +como +166 +burt +divers +##100 +wichita +cade +coincided +##erson +bruised +mla +##pper +vineyard +##ili +##brush +notch +mentioning +jase +hearted +kits +doe +##acle +pomerania +##ady +ronan +seizure +pavel +problematic +##zaki +domenico +##ulin +catering +penelope +dependence +parental +emilio +ministerial +atkinson +##bolic +clarkson +chargers +colby +grill +peeked +arises +summon +##aged +fools +##grapher +faculties +qaeda +##vial +garner +refurbished +##hwa +geelong +disasters +nudged +bs +shareholder +lori +algae +reinstated +rot +##ades +##nous +invites +stainless +183 +inclusive +##itude +diocesan +til +##icz +denomination +##xa +benton +floral +registers +##ider +##erman +##kell +absurd +brunei +guangzhou +hitter +retaliation +##uled +##eve +blanc +nh +consistency +contamination +##eres +##rner +dire +palermo +broadcasters +diaries +inspire +vols +brewer +tightening +ky +mixtape +hormone +##tok +stokes +##color +##dly +##ssi +pg +##ometer +##lington +sanitation +##tility +intercontinental +apps +##adt +¹⁄₂ +cylinders +economies +favourable +unison +croix +gertrude +odyssey +vanity +dangling +##logists +upgrades +dice +middleweight +practitioner +##ight +206 +henrik +parlor +orion +angered +lac +python +blurted +##rri +sensual +intends +swings +angled +##phs +husky +attain +peerage +precinct +textiles +cheltenham +shuffled +dai +confess +tasting +bhutan +##riation +tyrone +segregation +abrupt +ruiz +##rish +smirked +blackwell +confidential +browning +amounted +##put +vase +scarce +fabulous +raided +staple +guyana +unemployed +glider +shay +##tow +carmine +troll +intervene +squash +superstar +##uce +cylindrical +len +roadway +researched +handy +##rium +##jana +meta +lao +declares +##rring +##tadt +##elin +##kova +willem +shrubs +napoleonic +realms +skater +qi +volkswagen +##ł +tad +hara +archaeologist +awkwardly +eerie +##kind +wiley +##heimer +##24 +titus +organizers +cfl +crusaders +lama +usb +vent +enraged +thankful +occupants +maximilian +##gaard +possessing +textbooks +##oran +collaborator +quaker +##ulo +avalanche +mono +silky +straits +isaiah +mustang +surged +resolutions +potomac +descend +cl +kilograms +plato +strains +saturdays +##olin +bernstein +##ype +holstein +ponytail +##watch +belize +conversely +heroine +perpetual +##ylus +charcoal +piedmont +glee +negotiating +backdrop +prologue +##jah +##mmy +pasadena +climbs +ramos +sunni +##holm +##tner +##tri +anand +deficiency +hertfordshire +stout +##avi +aperture +orioles +##irs +doncaster +intrigued +bombed +coating +otis +##mat +cocktail +##jit +##eto +amir +arousal +sar +##proof +##act +##ories +dixie +pots +##bow +whereabouts +159 +##fted +drains +bullying +cottages +scripture +coherent +fore +poe +appetite +##uration +sampled +##ators +##dp +derrick +rotor +jays +peacock +installment +##rro +advisors +##coming +rodeo +scotch +##mot +##db +##fen +##vant +ensued +rodrigo +dictatorship +martyrs +twenties +##н +towed +incidence +marta +rainforest +sai +scaled +##cles +oceanic +qualifiers +symphonic +mcbride +dislike +generalized +aubrey +colonization +##iation +##lion +##ssing +disliked +lublin +salesman +##ulates +spherical +whatsoever +sweating +avalon +contention +punt +severity +alderman +atari +##dina +##grant +##rop +scarf +seville +vertices +annexation +fairfield +fascination +inspiring +launches +palatinate +regretted +##rca +feral +##iom +elk +nap +olsen +reddy +yong +##leader +##iae +garment +transports +feng +gracie +outrage +viceroy +insides +##esis +breakup +grady +organizer +softer +grimaced +222 +murals +galicia +arranging +vectors +##rsten +bas +##sb +##cens +sloan +##eka +bitten +ara +fender +nausea +bumped +kris +banquet +comrades +detector +persisted +##llan +adjustment +endowed +cinemas +##shot +sellers +##uman +peek +epa +kindly +neglect +simpsons +talon +mausoleum +runaway +hangul +lookout +##cic +rewards +coughed +acquainted +chloride +##ald +quicker +accordion +neolithic +##qa +artemis +coefficient +lenny +pandora +tx +##xed +ecstasy +litter +segunda +chairperson +gemma +hiss +rumor +vow +nasal +antioch +compensate +patiently +transformers +##eded +judo +morrow +penis +posthumous +philips +bandits +husbands +denote +flaming +##any +##phones +langley +yorker +1760 +walters +##uo +##kle +gubernatorial +fatty +samsung +leroy +outlaw +##nine +unpublished +poole +jakob +##ᵢ +##ₙ +crete +distorted +superiority +##dhi +intercept +crust +mig +claus +crashes +positioning +188 +stallion +301 +frontal +armistice +##estinal +elton +aj +encompassing +camel +commemorated +malaria +woodward +calf +cigar +penetrate +##oso +willard +##rno +##uche +illustrate +amusing +convergence +noteworthy +##lma +##rva +journeys +realise +manfred +##sable +410 +##vocation +hearings +fiance +##posed +educators +provoked +adjusting +##cturing +modular +stockton +paterson +vlad +rejects +electors +selena +maureen +##tres +uber +##rce +swirled +##num +proportions +nanny +pawn +naturalist +parma +apostles +awoke +ethel +wen +##bey +monsoon +overview +##inating +mccain +rendition +risky +adorned +##ih +equestrian +germain +nj +conspicuous +confirming +##yoshi +shivering +##imeter +milestone +rumours +flinched +bounds +smacked +token +##bei +lectured +automobiles +##shore +impacted +##iable +nouns +nero +##leaf +ismail +prostitute +trams +##lace +bridget +sud +stimulus +impressions +reins +revolves +##oud +##gned +giro +honeymoon +##swell +criterion +##sms +##uil +libyan +prefers +##osition +211 +preview +sucks +accusation +bursts +metaphor +diffusion +tolerate +faye +betting +cinematographer +liturgical +specials +bitterly +humboldt +##ckle +flux +rattled +##itzer +archaeologists +odor +authorised +marshes +discretion +##ов +alarmed +archaic +inverse +##leton +explorers +##pine +drummond +tsunami +woodlands +##minate +##tland +booklet +insanity +owning +insert +crafted +calculus +##tore +receivers +##bt +stung +##eca +##nched +prevailing +travellers +eyeing +lila +graphs +##borne +178 +julien +##won +morale +adaptive +therapist +erica +cw +libertarian +bowman +pitches +vita +##ional +crook +##ads +##entation +caledonia +mutiny +##sible +1840s +automation +##ß +flock +##pia +ironic +pathology +##imus +remarried +##22 +joker +withstand +energies +##att +shropshire +hostages +madeleine +tentatively +conflicting +mateo +recipes +euros +ol +mercenaries +nico +##ndon +albuquerque +augmented +mythical +bel +freud +##child +cough +##lica +365 +freddy +lillian +genetically +nuremberg +calder +209 +bonn +outdoors +paste +suns +urgency +vin +restraint +tyson +##cera +##selle +barrage +bethlehem +kahn +##par +mounts +nippon +barony +happier +ryu +makeshift +sheldon +blushed +castillo +barking +listener +taped +bethel +fluent +headlines +pornography +rum +disclosure +sighing +mace +doubling +gunther +manly +##plex +rt +interventions +physiological +forwards +emerges +##tooth +##gny +compliment +rib +recession +visibly +barge +faults +connector +exquisite +prefect +##rlin +patio +##cured +elevators +brandt +italics +pena +173 +wasp +satin +ea +botswana +graceful +respectable +##jima +##rter +##oic +franciscan +generates +##dl +alfredo +disgusting +##olate +##iously +sherwood +warns +cod +promo +cheryl +sino +##ة +##escu +twitch +##zhi +brownish +thom +ortiz +##dron +densely +##beat +carmel +reinforce +##bana +187 +anastasia +downhill +vertex +contaminated +remembrance +harmonic +homework +##sol +fiancee +gears +olds +angelica +loft +ramsay +quiz +colliery +sevens +##cape +autism +##hil +walkway +##boats +ruben +abnormal +ounce +khmer +##bbe +zachary +bedside +morphology +punching +##olar +sparrow +convinces +##35 +hewitt +queer +remastered +rods +mabel +solemn +notified +lyricist +symmetric +##xide +174 +encore +passports +wildcats +##uni +baja +##pac +mildly +##ease +bleed +commodity +mounds +glossy +orchestras +##omo +damian +prelude +ambitions +##vet +awhile +remotely +##aud +asserts +imply +##iques +distinctly +modelling +remedy +##dded +windshield +dani +xiao +##endra +audible +powerplant +1300 +invalid +elemental +acquisitions +##hala +immaculate +libby +plata +smuggling +ventilation +denoted +minh +##morphism +430 +differed +dion +kelley +lore +mocking +sabbath +spikes +hygiene +drown +runoff +stylized +tally +liberated +aux +interpreter +righteous +aba +siren +reaper +pearce +millie +##cier +##yra +gaius +##iso +captures +##ttering +dorm +claudio +##sic +benches +knighted +blackness +##ored +discount +fumble +oxidation +routed +##ς +novak +perpendicular +spoiled +fracture +splits +##urt +pads +topology +##cats +axes +fortunate +offenders +protestants +esteem +221 +broadband +convened +frankly +hound +prototypes +isil +facilitated +keel +##sher +sahara +awaited +bubba +orb +prosecutors +186 +hem +520 +##xing +relaxing +remnant +romney +sorted +slalom +stefano +ulrich +##active +exemption +folder +pauses +foliage +hitchcock +epithet +204 +criticisms +##aca +ballistic +brody +hinduism +chaotic +youths +equals +##pala +pts +thicker +analogous +capitalist +improvised +overseeing +sinatra +ascended +beverage +##tl +straightforward +##kon +curran +##west +bois +325 +induce +surveying +emperors +sax +unpopular +##kk +cartoonist +fused +##mble +unto +##yuki +localities +##cko +##ln +darlington +slain +academie +lobbying +sediment +puzzles +##grass +defiance +dickens +manifest +tongues +alumnus +arbor +coincide +184 +appalachian +mustafa +examiner +cabaret +traumatic +yves +bracelet +draining +heroin +magnum +baths +odessa +consonants +mitsubishi +##gua +kellan +vaudeville +##fr +joked +null +straps +probation +##ław +ceded +interfaces +##pas +##zawa +blinding +viet +224 +rothschild +museo +640 +huddersfield +##vr +tactic +##storm +brackets +dazed +incorrectly +##vu +reg +glazed +fearful +manifold +benefited +irony +##sun +stumbling +##rte +willingness +balkans +mei +wraps +##aba +injected +##lea +gu +syed +harmless +##hammer +bray +takeoff +poppy +timor +cardboard +astronaut +purdue +weeping +southbound +cursing +stalls +diagonal +##neer +lamar +bryce +comte +weekdays +harrington +##uba +negatively +##see +lays +grouping +##cken +##henko +affirmed +halle +modernist +##lai +hodges +smelling +aristocratic +baptized +dismiss +justification +oilers +##now +coupling +qin +snack +healer +##qing +gardener +layla +battled +formulated +stephenson +gravitational +##gill +##jun +1768 +granny +coordinating +suites +##cd +##ioned +monarchs +##cote +##hips +sep +blended +apr +barrister +deposition +fia +mina +policemen +paranoid +##pressed +churchyard +covert +crumpled +creep +abandoning +tr +transmit +conceal +barr +understands +readiness +spire +##cology +##enia +##erry +610 +startling +unlock +vida +bowled +slots +##nat +##islav +spaced +trusting +admire +rig +##ink +slack +##70 +mv +207 +casualty +##wei +classmates +##odes +##rar +##rked +amherst +furnished +evolve +foundry +menace +mead +##lein +flu +wesleyan +##kled +monterey +webber +##vos +wil +##mith +##на +bartholomew +justices +restrained +##cke +amenities +191 +mediated +sewage +trenches +ml +mainz +##thus +1800s +##cula +##inski +caine +bonding +213 +converts +spheres +superseded +marianne +crypt +sweaty +ensign +historia +##br +spruce +##post +##ask +forks +thoughtfully +yukon +pamphlet +ames +##uter +karma +##yya +bryn +negotiation +sighs +incapable +##mbre +##ntial +actresses +taft +##mill +luce +prevailed +##amine +1773 +motionless +envoy +testify +investing +sculpted +instructors +provence +kali +cullen +horseback +##while +goodwin +##jos +gaa +norte +##ldon +modify +wavelength +abd +214 +skinned +sprinter +forecast +scheduling +marries +squared +tentative +##chman +boer +##isch +bolts +swap +fisherman +assyrian +impatiently +guthrie +martins +murdoch +194 +tanya +nicely +dolly +lacy +med +##45 +syn +decks +fashionable +millionaire +##ust +surfing +##ml +##ision +heaved +tammy +consulate +attendees +routinely +197 +fuse +saxophonist +backseat +malaya +##lord +scowl +tau +##ishly +193 +sighted +steaming +##rks +303 +911 +##holes +##hong +ching +##wife +bless +conserved +jurassic +stacey +unix +zion +chunk +rigorous +blaine +198 +peabody +slayer +dismay +brewers +nz +##jer +det +##glia +glover +postwar +int +penetration +sylvester +imitation +vertically +airlift +heiress +knoxville +viva +##uin +390 +macon +##rim +##fighter +##gonal +janice +##orescence +##wari +marius +belongings +leicestershire +196 +blanco +inverted +preseason +sanity +sobbing +##due +##elt +##dled +collingwood +regeneration +flickering +shortest +##mount +##osi +feminism +##lat +sherlock +cabinets +fumbled +northbound +precedent +snaps +##mme +researching +##akes +guillaume +insights +manipulated +vapor +neighbour +sap +gangster +frey +f1 +stalking +scarcely +callie +barnett +tendencies +audi +doomed +assessing +slung +panchayat +ambiguous +bartlett +##etto +distributing +violating +wolverhampton +##hetic +swami +histoire +##urus +liable +pounder +groin +hussain +larsen +popping +surprises +##atter +vie +curt +##station +mute +relocate +musicals +authorization +richter +##sef +immortality +tna +bombings +##press +deteriorated +yiddish +##acious +robbed +colchester +cs +pmid +ao +verified +balancing +apostle +swayed +recognizable +oxfordshire +retention +nottinghamshire +contender +judd +invitational +shrimp +uhf +##icient +cleaner +longitudinal +tanker +##mur +acronym +broker +koppen +sundance +suppliers +##gil +4000 +clipped +fuels +petite +##anne +landslide +helene +diversion +populous +landowners +auspices +melville +quantitative +##xes +ferries +nicky +##llus +doo +haunting +roche +carver +downed +unavailable +##pathy +approximation +hiroshima +##hue +garfield +valle +comparatively +keyboardist +traveler +##eit +congestion +calculating +subsidiaries +##bate +serb +modernization +fairies +deepened +ville +averages +##lore +inflammatory +tonga +##itch +co₂ +squads +##hea +gigantic +serum +enjoyment +retailer +verona +35th +cis +##phobic +magna +technicians +##vati +arithmetic +##sport +levin +##dation +amtrak +chow +sienna +##eyer +backstage +entrepreneurship +##otic +learnt +tao +##udy +worcestershire +formulation +baggage +hesitant +bali +sabotage +##kari +barren +enhancing +murmur +pl +freshly +putnam +syntax +aces +medicines +resentment +bandwidth +##sier +grins +chili +guido +##sei +framing +implying +gareth +lissa +genevieve +pertaining +admissions +geo +thorpe +proliferation +sato +bela +analyzing +parting +##gor +awakened +##isman +huddled +secrecy +##kling +hush +gentry +540 +dungeons +##ego +coasts +##utz +sacrificed +##chule +landowner +mutually +prevalence +programmer +adolescent +disrupted +seaside +gee +trusts +vamp +georgie +##nesian +##iol +schedules +sindh +##market +etched +hm +sparse +bey +beaux +scratching +gliding +unidentified +216 +collaborating +gems +jesuits +oro +accumulation +shaping +mbe +anal +##xin +231 +enthusiasts +newscast +##egan +janata +dewey +parkinson +179 +ankara +biennial +towering +dd +inconsistent +950 +##chet +thriving +terminate +cabins +furiously +eats +advocating +donkey +marley +muster +phyllis +leiden +##user +grassland +glittering +iucn +loneliness +217 +memorandum +armenians +##ddle +popularized +rhodesia +60s +lame +##illon +sans +bikini +header +orbits +##xx +##finger +##ulator +sharif +spines +biotechnology +strolled +naughty +yates +##wire +fremantle +milo +##mour +abducted +removes +##atin +humming +wonderland +##chrome +##ester +hume +pivotal +##rates +armand +grams +believers +elector +rte +apron +bis +scraped +##yria +endorsement +initials +##llation +eps +dotted +hints +buzzing +emigration +nearer +##tom +indicators +##ulu +coarse +neutron +protectorate +##uze +directional +exploits +pains +loire +1830s +proponents +guggenheim +rabbits +ritchie +305 +hectare +inputs +hutton +##raz +verify +##ako +boilers +longitude +##lev +skeletal +yer +emilia +citrus +compromised +##gau +pokemon +prescription +paragraph +eduard +cadillac +attire +categorized +kenyan +weddings +charley +##bourg +entertain +monmouth +##lles +nutrients +davey +mesh +incentive +practised +ecosystems +kemp +subdued +overheard +##rya +bodily +maxim +##nius +apprenticeship +ursula +##fight +lodged +rug +silesian +unconstitutional +patel +inspected +coyote +unbeaten +##hak +34th +disruption +convict +parcel +##cl +##nham +collier +implicated +mallory +##iac +##lab +susannah +winkler +##rber +shia +phelps +sediments +graphical +robotic +##sner +adulthood +mart +smoked +##isto +kathryn +clarified +##aran +divides +convictions +oppression +pausing +burying +##mt +federico +mathias +eileen +##tana +kite +hunched +##acies +189 +##atz +disadvantage +liza +kinetic +greedy +paradox +yokohama +dowager +trunks +ventured +##gement +gupta +vilnius +olaf +##thest +crimean +hopper +##ej +progressively +arturo +mouthed +arrondissement +##fusion +rubin +simulcast +oceania +##orum +##stra +##rred +busiest +intensely +navigator +cary +##vine +##hini +##bies +fife +rowe +rowland +posing +insurgents +shafts +lawsuits +activate +conor +inward +culturally +garlic +265 +##eering +eclectic +##hui +##kee +##nl +furrowed +vargas +meteorological +rendezvous +##aus +culinary +commencement +##dition +quota +##notes +mommy +salaries +overlapping +mule +##iology +##mology +sums +wentworth +##isk +##zione +mainline +subgroup +##illy +hack +plaintiff +verdi +bulb +differentiation +engagements +multinational +supplemented +bertrand +caller +regis +##naire +##sler +##arts +##imated +blossom +propagation +kilometer +viaduct +vineyards +##uate +beckett +optimization +golfer +songwriters +seminal +semitic +thud +volatile +evolving +ridley +##wley +trivial +distributions +scandinavia +jiang +##ject +wrestled +insistence +##dio +emphasizes +napkin +##ods +adjunct +rhyme +##ricted +##eti +hopeless +surrounds +tremble +32nd +smoky +##ntly +oils +medicinal +padded +steer +wilkes +219 +255 +concessions +hue +uniquely +blinded +landon +yahoo +##lane +hendrix +commemorating +dex +specify +chicks +##ggio +intercity +1400 +morley +##torm +highlighting +##oting +pang +oblique +stalled +##liner +flirting +newborn +1769 +bishopric +shaved +232 +currie +##ush +dharma +spartan +##ooped +favorites +smug +novella +sirens +abusive +creations +espana +##lage +paradigm +semiconductor +sheen +##rdo +##yen +##zak +nrl +renew +##pose +##tur +adjutant +marches +norma +##enity +ineffective +weimar +grunt +##gat +lordship +plotting +expenditure +infringement +lbs +refrain +av +mimi +mistakenly +postmaster +1771 +##bara +ras +motorsports +tito +199 +subjective +##zza +bully +stew +##kaya +prescott +1a +##raphic +##zam +bids +styling +paranormal +reeve +sneaking +exploding +katz +akbar +migrant +syllables +indefinitely +##ogical +destroys +replaces +applause +##phine +pest +##fide +218 +articulated +bertie +##thing +##cars +##ptic +courtroom +crowley +aesthetics +cummings +tehsil +hormones +titanic +dangerously +##ibe +stadion +jaenelle +auguste +ciudad +##chu +mysore +partisans +##sio +lucan +philipp +##aly +debating +henley +interiors +##rano +##tious +homecoming +beyonce +usher +henrietta +prepares +weeds +##oman +ely +plucked +##pire +##dable +luxurious +##aq +artifact +password +pasture +juno +maddy +minsk +##dder +##ologies +##rone +assessments +martian +royalist +1765 +examines +##mani +##rge +nino +223 +parry +scooped +relativity +##eli +##uting +##cao +congregational +noisy +traverse +##agawa +strikeouts +nickelodeon +obituary +transylvania +binds +depictions +polk +trolley +##yed +##lard +breeders +##under +dryly +hokkaido +1762 +strengths +stacks +bonaparte +connectivity +neared +prostitutes +stamped +anaheim +gutierrez +sinai +##zzling +bram +fresno +madhya +##86 +proton +##lena +##llum +##phon +reelected +wanda +##anus +##lb +ample +distinguishing +##yler +grasping +sermons +tomato +bland +stimulation +avenues +##eux +spreads +scarlett +fern +pentagon +assert +baird +chesapeake +ir +calmed +distortion +fatalities +##olis +correctional +pricing +##astic +##gina +prom +dammit +ying +collaborate +##chia +welterweight +33rd +pointer +substitution +bonded +umpire +communicating +multitude +paddle +##obe +federally +intimacy +##insky +betray +ssr +##lett +##lean +##lves +##therapy +airbus +##tery +functioned +ud +bearer +biomedical +netflix +##hire +##nca +condom +brink +ik +##nical +macy +##bet +flap +gma +experimented +jelly +lavender +##icles +##ulia +munro +##mian +##tial +rye +##rle +60th +gigs +hottest +rotated +predictions +fuji +bu +##erence +##omi +barangay +##fulness +##sas +clocks +##rwood +##liness +cereal +roe +wight +decker +uttered +babu +onion +xml +forcibly +##df +petra +sarcasm +hartley +peeled +storytelling +##42 +##xley +##ysis +##ffa +fibre +kiel +auditor +fig +harald +greenville +##berries +geographically +nell +quartz +##athic +cemeteries +##lr +crossings +nah +holloway +reptiles +chun +sichuan +snowy +660 +corrections +##ivo +zheng +ambassadors +blacksmith +fielded +fluids +hardcover +turnover +medications +melvin +academies +##erton +ro +roach +absorbing +spaniards +colton +##founded +outsider +espionage +kelsey +245 +edible +##ulf +dora +establishes +##sham +##tries +contracting +##tania +cinematic +costello +nesting +##uron +connolly +duff +##nology +mma +##mata +fergus +sexes +gi +optics +spectator +woodstock +banning +##hee +##fle +differentiate +outfielder +refinery +226 +312 +gerhard +horde +lair +drastically +##udi +landfall +##cheng +motorsport +odi +##achi +predominant +quay +skins +##ental +edna +harshly +complementary +murdering +##aves +wreckage +##90 +ono +outstretched +lennox +munitions +galen +reconcile +470 +scalp +bicycles +gillespie +questionable +rosenberg +guillermo +hostel +jarvis +kabul +volvo +opium +yd +##twined +abuses +decca +outpost +##cino +sensible +neutrality +##64 +ponce +anchorage +atkins +turrets +inadvertently +disagree +libre +vodka +reassuring +weighs +##yal +glide +jumper +ceilings +repertory +outs +stain +##bial +envy +##ucible +smashing +heightened +policing +hyun +mixes +lai +prima +##ples +celeste +##bina +lucrative +intervened +kc +manually +##rned +stature +staffed +bun +bastards +nairobi +priced +##auer +thatcher +##kia +tripped +comune +##ogan +##pled +brasil +incentives +emanuel +hereford +musica +##kim +benedictine +biennale +##lani +eureka +gardiner +rb +knocks +sha +##ael +##elled +##onate +efficacy +ventura +masonic +sanford +maize +leverage +##feit +capacities +santana +##aur +novelty +vanilla +##cter +##tour +benin +##oir +##rain +neptune +drafting +tallinn +##cable +humiliation +##boarding +schleswig +fabian +bernardo +liturgy +spectacle +sweeney +pont +routledge +##tment +cosmos +ut +hilt +sleek +universally +##eville +##gawa +typed +##dry +favors +allegheny +glaciers +##rly +recalling +aziz +##log +parasite +requiem +auf +##berto +##llin +illumination +##breaker +##issa +festivities +bows +govern +vibe +vp +333 +sprawled +larson +pilgrim +bwf +leaping +##rts +##ssel +alexei +greyhound +hoarse +##dler +##oration +seneca +##cule +gaping +##ulously +##pura +cinnamon +##gens +##rricular +craven +fantasies +houghton +engined +reigned +dictator +supervising +##oris +bogota +commentaries +unnatural +fingernails +spirituality +tighten +##tm +canadiens +protesting +intentional +cheers +sparta +##ytic +##iere +##zine +widen +belgarath +controllers +dodd +iaaf +navarre +##ication +defect +squire +steiner +whisky +##mins +560 +inevitably +tome +##gold +chew +##uid +##lid +elastic +##aby +streaked +alliances +jailed +regal +##ined +##phy +czechoslovak +narration +absently +##uld +bluegrass +guangdong +quran +criticizing +hose +hari +##liest +##owa +skier +streaks +deploy +##lom +raft +bose +dialed +huff +##eira +haifa +simplest +bursting +endings +ib +sultanate +##titled +franks +whitman +ensures +sven +##ggs +collaborators +forster +organising +ui +banished +napier +injustice +teller +layered +thump +##otti +roc +battleships +evidenced +fugitive +sadie +robotics +##roud +equatorial +geologist +##iza +yielding +##bron +##sr +internationale +mecca +##diment +sbs +skyline +toad +uploaded +reflective +undrafted +lal +leafs +bayern +##dai +lakshmi +shortlisted +##stick +##wicz +camouflage +donate +af +christi +lau +##acio +disclosed +nemesis +1761 +assemble +straining +northamptonshire +tal +##asi +bernardino +premature +heidi +42nd +coefficients +galactic +reproduce +buzzed +sensations +zionist +monsieur +myrtle +##eme +archery +strangled +musically +viewpoint +antiquities +bei +trailers +seahawks +cured +pee +preferring +tasmanian +lange +sul +##mail +##working +colder +overland +lucivar +massey +gatherings +haitian +##smith +disapproval +flaws +##cco +##enbach +1766 +npr +##icular +boroughs +creole +forums +techno +1755 +dent +abdominal +streetcar +##eson +##stream +procurement +gemini +predictable +##tya +acheron +christoph +feeder +fronts +vendor +bernhard +jammu +tumors +slang +##uber +goaltender +twists +curving +manson +vuelta +mer +peanut +confessions +pouch +unpredictable +allowance +theodor +vascular +##factory +bala +authenticity +metabolic +coughing +nanjing +##cea +pembroke +##bard +splendid +36th +ff +hourly +##ahu +elmer +handel +##ivate +awarding +thrusting +dl +experimentation +##hesion +##46 +caressed +entertained +steak +##rangle +biologist +orphans +baroness +oyster +stepfather +##dridge +mirage +reefs +speeding +##31 +barons +1764 +227 +inhabit +preached +repealed +##tral +honoring +boogie +captives +administer +johanna +##imate +gel +suspiciously +1767 +sobs +##dington +backbone +hayward +garry +##folding +##nesia +maxi +##oof +##ppe +ellison +galileo +##stand +crimea +frenzy +amour +bumper +matrices +natalia +baking +garth +palestinians +##grove +smack +conveyed +ensembles +gardening +##manship +##rup +##stituting +1640 +harvesting +topography +jing +shifters +dormitory +##carriage +##lston +ist +skulls +##stadt +dolores +jewellery +sarawak +##wai +##zier +fences +christy +confinement +tumbling +credibility +fir +stench +##bria +##plication +##nged +##sam +virtues +##belt +marjorie +pba +##eem +##made +celebrates +schooner +agitated +barley +fulfilling +anthropologist +##pro +restrict +novi +regulating +##nent +padres +##rani +##hesive +loyola +tabitha +milky +olson +proprietor +crambidae +guarantees +intercollegiate +ljubljana +hilda +##sko +ignorant +hooded +##lts +sardinia +##lidae +##vation +frontman +privileged +witchcraft +##gp +jammed +laude +poking +##than +bracket +amazement +yunnan +##erus +maharaja +linnaeus +264 +commissioning +milano +peacefully +##logies +akira +rani +regulator +##36 +grasses +##rance +luzon +crows +compiler +gretchen +seaman +edouard +tab +buccaneers +ellington +hamlets +whig +socialists +##anto +directorial +easton +mythological +##kr +##vary +rhineland +semantic +taut +dune +inventions +succeeds +##iter +replication +branched +##pired +jul +prosecuted +kangaroo +penetrated +##avian +middlesbrough +doses +bleak +madam +predatory +relentless +##vili +reluctance +##vir +hailey +crore +silvery +1759 +monstrous +swimmers +transmissions +hawthorn +informing +##eral +toilets +caracas +crouch +kb +##sett +295 +cartel +hadley +##aling +alexia +yvonne +##biology +cinderella +eton +superb +blizzard +stabbing +industrialist +maximus +##gm +##orus +groves +maud +clade +oversized +comedic +##bella +rosen +nomadic +fulham +montane +beverages +galaxies +redundant +swarm +##rot +##folia +##llis +buckinghamshire +fen +bearings +bahadur +##rom +gilles +phased +dynamite +faber +benoit +vip +##ount +##wd +booking +fractured +tailored +anya +spices +westwood +cairns +auditions +inflammation +steamed +##rocity +##acion +##urne +skyla +thereof +watford +torment +archdeacon +transforms +lulu +demeanor +fucked +serge +##sor +mckenna +minas +entertainer +##icide +caress +originate +residue +##sty +1740 +##ilised +##org +beech +##wana +subsidies +##ghton +emptied +gladstone +ru +firefighters +voodoo +##rcle +het +nightingale +tamara +edmond +ingredient +weaknesses +silhouette +285 +compatibility +withdrawing +hampson +##mona +anguish +giggling +##mber +bookstore +##jiang +southernmost +tilting +##vance +bai +economical +rf +briefcase +dreadful +hinted +projections +shattering +totaling +##rogate +analogue +indicted +periodical +fullback +##dman +haynes +##tenberg +##ffs +##ishment +1745 +thirst +stumble +penang +vigorous +##ddling +##kor +##lium +octave +##ove +##enstein +##inen +##ones +siberian +##uti +cbn +repeal +swaying +##vington +khalid +tanaka +unicorn +otago +plastered +lobe +riddle +##rella +perch +##ishing +croydon +filtered +graeme +tripoli +##ossa +crocodile +##chers +sufi +mined +##tung +inferno +lsu +##phi +swelled +utilizes +£2 +cale +periodicals +styx +hike +informally +coop +lund +##tidae +ala +hen +qui +transformations +disposed +sheath +chickens +##cade +fitzroy +sas +silesia +unacceptable +odisha +1650 +sabrina +pe +spokane +ratios +athena +massage +shen +dilemma +##drum +##riz +##hul +corona +doubtful +niall +##pha +##bino +fines +cite +acknowledging +bangor +ballard +bathurst +##resh +huron +mustered +alzheimer +garments +kinase +tyre +warship +##cp +flashback +pulmonary +braun +cheat +kamal +cyclists +constructions +grenades +ndp +traveller +excuses +stomped +signalling +trimmed +futsal +mosques +relevance +##wine +wta +##23 +##vah +##lter +hoc +##riding +optimistic +##´s +deco +sim +interacting +rejecting +moniker +waterways +##ieri +##oku +mayors +gdansk +outnumbered +pearls +##ended +##hampton +fairs +totals +dominating +262 +notions +stairway +compiling +pursed +commodities +grease +yeast +##jong +carthage +griffiths +residual +amc +contraction +laird +sapphire +##marine +##ivated +amalgamation +dissolve +inclination +lyle +packaged +altitudes +suez +canons +graded +lurched +narrowing +boasts +guise +wed +enrico +##ovsky +rower +scarred +bree +cub +iberian +protagonists +bargaining +proposing +trainers +voyages +vans +fishes +##aea +##ivist +##verance +encryption +artworks +kazan +sabre +cleopatra +hepburn +rotting +supremacy +mecklenburg +##brate +burrows +hazards +outgoing +flair +organizes +##ctions +scorpion +##usions +boo +234 +chevalier +dunedin +slapping +##34 +ineligible +pensions +##38 +##omic +manufactures +emails +bismarck +238 +weakening +blackish +ding +mcgee +quo +##rling +northernmost +xx +manpower +greed +sampson +clicking +##ange +##horpe +##inations +##roving +torre +##eptive +##moral +symbolism +38th +asshole +meritorious +outfits +splashed +biographies +sprung +astros +##tale +302 +737 +filly +raoul +nw +tokugawa +linden +clubhouse +##apa +tracts +romano +##pio +putin +tags +##note +chained +dickson +gunshot +moe +gunn +rashid +##tails +zipper +##bas +##nea +contrasted +##ply +##udes +plum +pharaoh +##pile +aw +comedies +ingrid +sandwiches +subdivisions +1100 +mariana +nokia +kamen +hz +delaney +veto +herring +##words +possessive +outlines +##roup +siemens +stairwell +rc +gallantry +messiah +palais +yells +233 +zeppelin +##dm +bolivar +##cede +smackdown +mckinley +##mora +##yt +muted +geologic +finely +unitary +avatar +hamas +maynard +rees +bog +contrasting +##rut +liv +chico +disposition +pixel +##erate +becca +dmitry +yeshiva +narratives +##lva +##ulton +mercenary +sharpe +tempered +navigate +stealth +amassed +keynes +##lini +untouched +##rrie +havoc +lithium +##fighting +abyss +graf +southward +wolverine +balloons +implements +ngos +transitions +##icum +ambushed +concacaf +dormant +economists +##dim +costing +csi +rana +universite +boulders +verity +##llon +collin +mellon +misses +cypress +fluorescent +lifeless +spence +##ulla +crewe +shepard +pak +revelations +##م +jolly +gibbons +paw +##dro +##quel +freeing +##test +shack +fries +palatine +##51 +##hiko +accompaniment +cruising +recycled +##aver +erwin +sorting +synthesizers +dyke +realities +sg +strides +enslaved +wetland +##ghan +competence +gunpowder +grassy +maroon +reactors +objection +##oms +carlson +gearbox +macintosh +radios +shelton +##sho +clergyman +prakash +254 +mongols +trophies +oricon +228 +stimuli +twenty20 +cantonese +cortes +mirrored +##saurus +bhp +cristina +melancholy +##lating +enjoyable +nuevo +##wny +downfall +schumacher +##ind +banging +lausanne +rumbled +paramilitary +reflex +ax +amplitude +migratory +##gall +##ups +midi +barnard +lastly +sherry +##hp +##nall +keystone +##kra +carleton +slippery +##53 +coloring +foe +socket +otter +##rgos +mats +##tose +consultants +bafta +bison +topping +##km +490 +primal +abandonment +transplant +atoll +hideous +mort +pained +reproduced +tae +howling +##turn +unlawful +billionaire +hotter +poised +lansing +##chang +dinamo +retro +messing +nfc +domesday +##mina +blitz +timed +##athing +##kley +ascending +gesturing +##izations +signaled +tis +chinatown +mermaid +savanna +jameson +##aint +catalina +##pet +##hers +cochrane +cy +chatting +##kus +alerted +computation +mused +noelle +majestic +mohawk +campo +octagonal +##sant +##hend +241 +aspiring +##mart +comprehend +iona +paralyzed +shimmering +swindon +rhone +##eley +reputed +configurations +pitchfork +agitation +francais +gillian +lipstick +##ilo +outsiders +pontifical +resisting +bitterness +sewer +rockies +##edd +##ucher +misleading +1756 +exiting +galloway +##nging +risked +##heart +246 +commemoration +schultz +##rka +integrating +##rsa +poses +shrieked +##weiler +guineas +gladys +jerking +owls +goldsmith +nightly +penetrating +##unced +lia +##33 +ignited +betsy +##aring +##thorpe +follower +vigorously +##rave +coded +kiran +knit +zoology +tbilisi +##28 +##bered +repository +govt +deciduous +dino +growling +##bba +enhancement +unleashed +chanting +pussy +biochemistry +##eric +kettle +repression +toxicity +nrhp +##arth +##kko +##bush +ernesto +commended +outspoken +242 +mca +parchment +sms +kristen +##aton +bisexual +raked +glamour +navajo +a2 +conditioned +showcased +##hma +spacious +youthful +##esa +usl +appliances +junta +brest +layne +conglomerate +enchanted +chao +loosened +picasso +circulating +inspect +montevideo +##centric +##kti +piazza +spurred +##aith +bari +freedoms +poultry +stamford +lieu +##ect +indigo +sarcastic +bahia +stump +attach +dvds +frankenstein +lille +approx +scriptures +pollen +##script +nmi +overseen +##ivism +tides +proponent +newmarket +inherit +milling +##erland +centralized +##rou +distributors +credentials +drawers +abbreviation +##lco +##xon +downing +uncomfortably +ripe +##oes +erase +franchises +##ever +populace +##bery +##khar +decomposition +pleas +##tet +daryl +sabah +##stle +##wide +fearless +genie +lesions +annette +##ogist +oboe +appendix +nair +dripped +petitioned +maclean +mosquito +parrot +rpg +hampered +1648 +operatic +reservoirs +##tham +irrelevant +jolt +summarized +##fp +medallion +##taff +##− +clawed +harlow +narrower +goddard +marcia +bodied +fremont +suarez +altering +tempest +mussolini +porn +##isms +sweetly +oversees +walkers +solitude +grimly +shrines +hk +ich +supervisors +hostess +dietrich +legitimacy +brushes +expressive +##yp +dissipated +##rse +localized +systemic +##nikov +gettysburg +##js +##uaries +dialogues +muttering +251 +housekeeper +sicilian +discouraged +##frey +beamed +kaladin +halftime +kidnap +##amo +##llet +1754 +synonymous +depleted +instituto +insulin +reprised +##opsis +clashed +##ctric +interrupting +radcliffe +insisting +medici +1715 +ejected +playfully +turbulent +##47 +starvation +##rini +shipment +rebellious +petersen +verification +merits +##rified +cakes +##charged +1757 +milford +shortages +spying +fidelity +##aker +emitted +storylines +harvested +seismic +##iform +cheung +kilda +theoretically +barbie +lynx +##rgy +##tius +goblin +mata +poisonous +##nburg +reactive +residues +obedience +##евич +conjecture +##rac +401 +hating +sixties +kicker +moaning +motown +##bha +emancipation +neoclassical +##hering +consoles +ebert +professorship +##tures +sustaining +assaults +obeyed +affluent +incurred +tornadoes +##eber +##zow +emphasizing +highlanders +cheated +helmets +##ctus +internship +terence +bony +executions +legislators +berries +peninsular +tinged +##aco +1689 +amplifier +corvette +ribbons +lavish +pennant +##lander +worthless +##chfield +##forms +mariano +pyrenees +expenditures +##icides +chesterfield +mandir +tailor +39th +sergey +nestled +willed +aristocracy +devotees +goodnight +raaf +rumored +weaponry +remy +appropriations +harcourt +burr +riaa +##lence +limitation +unnoticed +guo +soaking +swamps +##tica +collapsing +tatiana +descriptive +brigham +psalm +##chment +maddox +##lization +patti +caliph +##aja +akron +injuring +serra +##ganj +basins +##sari +astonished +launcher +##church +hilary +wilkins +sewing +##sf +stinging +##fia +##ncia +underwood +startup +##ition +compilations +vibrations +embankment +jurist +##nity +bard +juventus +groundwater +kern +palaces +helium +boca +cramped +marissa +soto +##worm +jae +princely +##ggy +faso +bazaar +warmly +##voking +229 +pairing +##lite +##grate +##nets +wien +freaked +ulysses +rebirth +##alia +##rent +mummy +guzman +jimenez +stilled +##nitz +trajectory +tha +woken +archival +professions +##pts +##pta +hilly +shadowy +shrink +##bolt +norwood +glued +migrate +stereotypes +devoid +##pheus +625 +evacuate +horrors +infancy +gotham +knowles +optic +downloaded +sachs +kingsley +parramatta +darryl +mor +##onale +shady +commence +confesses +kan +##meter +##placed +marlborough +roundabout +regents +frigates +io +##imating +gothenburg +revoked +carvings +clockwise +convertible +intruder +##sche +banged +##ogo +vicky +bourgeois +##mony +dupont +footing +##gum +pd +##real +buckle +yun +penthouse +sane +720 +serviced +stakeholders +neumann +bb +##eers +comb +##gam +catchment +pinning +rallies +typing +##elles +forefront +freiburg +sweetie +giacomo +widowed +goodwill +worshipped +aspirations +midday +##vat +fishery +##trick +bournemouth +turk +243 +hearth +ethanol +guadalajara +murmurs +sl +##uge +afforded +scripted +##hta +wah +##jn +coroner +translucent +252 +memorials +puck +progresses +clumsy +##race +315 +candace +recounted +##27 +##slin +##uve +filtering +##mac +howl +strata +heron +leveled +##ays +dubious +##oja +##т +##wheel +citations +exhibiting +##laya +##mics +##pods +turkic +##lberg +injunction +##ennial +##mit +antibodies +##44 +organise +##rigues +cardiovascular +cushion +inverness +##zquez +dia +cocoa +sibling +##tman +##roid +expanse +feasible +tunisian +algiers +##relli +rus +bloomberg +dso +westphalia +bro +tacoma +281 +downloads +##ours +konrad +duran +##hdi +continuum +jett +compares +legislator +secession +##nable +##gues +##zuka +translating +reacher +##gley +##ła +aleppo +##agi +tc +orchards +trapping +linguist +versatile +drumming +postage +calhoun +superiors +##mx +barefoot +leary +##cis +ignacio +alfa +kaplan +##rogen +bratislava +mori +##vot +disturb +haas +313 +cartridges +gilmore +radiated +salford +tunic +hades +##ulsive +archeological +delilah +magistrates +auditioned +brewster +charters +empowerment +blogs +cappella +dynasties +iroquois +whipping +##krishna +raceway +truths +myra +weaken +judah +mcgregor +##horse +mic +refueling +37th +burnley +bosses +markus +premio +query +##gga +dunbar +##economic +darkest +lyndon +sealing +commendation +reappeared +##mun +addicted +ezio +slaughtered +satisfactory +shuffle +##eves +##thic +##uj +fortification +warrington +##otto +resurrected +fargo +mane +##utable +##lei +##space +foreword +ox +##aris +##vern +abrams +hua +##mento +sakura +##alo +uv +sentimental +##skaya +midfield +##eses +sturdy +scrolls +macleod +##kyu +entropy +##lance +mitochondrial +cicero +excelled +thinner +convoys +perceive +##oslav +##urable +systematically +grind +burkina +287 +##tagram +ops +##aman +guantanamo +##cloth +##tite +forcefully +wavy +##jou +pointless +##linger +##tze +layton +portico +superficial +clerical +outlaws +##hism +burials +muir +##inn +creditors +hauling +rattle +##leg +calais +monde +archers +reclaimed +dwell +wexford +hellenic +falsely +remorse +##tek +dough +furnishings +##uttered +gabon +neurological +novice +##igraphy +contemplated +pulpit +nightstand +saratoga +##istan +documenting +pulsing +taluk +##firmed +busted +marital +##rien +disagreements +wasps +##yes +hodge +mcdonnell +mimic +fran +pendant +dhabi +musa +##nington +congratulations +argent +darrell +concussion +losers +regrets +thessaloniki +reversal +donaldson +hardwood +thence +achilles +ritter +##eran +demonic +jurgen +prophets +goethe +eki +classmate +buff +##cking +yank +irrational +##inging +perished +seductive +qur +sourced +##crat +##typic +mustard +ravine +barre +horizontally +characterization +phylogenetic +boise +##dit +##runner +##tower +brutally +intercourse +seduce +##bbing +fay +ferris +ogden +amar +nik +unarmed +##inator +evaluating +kyrgyzstan +sweetness +##lford +##oki +mccormick +meiji +notoriety +stimulate +disrupt +figuring +instructional +mcgrath +##zoo +groundbreaking +##lto +flinch +khorasan +agrarian +bengals +mixer +radiating +##sov +ingram +pitchers +nad +tariff +##cript +tata +##codes +##emi +##ungen +appellate +lehigh +##bled +##giri +brawl +duct +texans +##ciation +##ropolis +skipper +speculative +vomit +doctrines +stresses +253 +davy +graders +whitehead +jozef +timely +cumulative +haryana +paints +appropriately +boon +cactus +##ales +##pid +dow +legions +##pit +perceptions +1730 +picturesque +##yse +periphery +rune +wr +##aha +celtics +sentencing +whoa +##erin +confirms +variance +425 +moines +mathews +spade +rave +m1 +fronted +fx +blending +alleging +reared +##gl +237 +##paper +grassroots +eroded +##free +##physical +directs +ordeal +##sław +accelerate +hacker +rooftop +##inia +lev +buys +cebu +devote +##lce +specialising +##ulsion +choreographed +repetition +warehouses +##ryl +paisley +tuscany +analogy +sorcerer +hash +huts +shards +descends +exclude +nix +chaplin +gaga +ito +vane +##drich +causeway +misconduct +limo +orchestrated +glands +jana +##kot +u2 +##mple +##sons +branching +contrasts +scoop +longed +##virus +chattanooga +##75 +syrup +cornerstone +##tized +##mind +##iaceae +careless +precedence +frescoes +##uet +chilled +consult +modelled +snatch +peat +##thermal +caucasian +humane +relaxation +spins +temperance +##lbert +occupations +lambda +hybrids +moons +mp3 +##oese +247 +rolf +societal +yerevan +ness +##ssler +befriended +mechanized +nominate +trough +boasted +cues +seater +##hom +bends +##tangle +conductors +emptiness +##lmer +eurasian +adriatic +tian +##cie +anxiously +lark +propellers +chichester +jock +ev +2a +##holding +credible +recounts +tori +loyalist +abduction +##hoot +##redo +nepali +##mite +ventral +tempting +##ango +##crats +steered +##wice +javelin +dipping +laborers +prentice +looming +titanium +##ː +badges +emir +tensor +##ntation +egyptians +rash +denies +hawthorne +lombard +showers +wehrmacht +dietary +trojan +##reus +welles +executing +horseshoe +lifeboat +##lak +elsa +infirmary +nearing +roberta +boyer +mutter +trillion +joanne +##fine +##oked +sinks +vortex +uruguayan +clasp +sirius +##block +accelerator +prohibit +sunken +byu +chronological +diplomats +ochreous +510 +symmetrical +1644 +maia +##tology +salts +reigns +atrocities +##ия +hess +bared +issn +##vyn +cater +saturated +##cycle +##isse +sable +voyager +dyer +yusuf +##inge +fountains +wolff +##39 +##nni +engraving +rollins +atheist +ominous +##ault +herr +chariot +martina +strung +##fell +##farlane +horrific +sahib +gazes +saetan +erased +ptolemy +##olic +flushing +lauderdale +analytic +##ices +530 +navarro +beak +gorilla +herrera +broom +guadalupe +raiding +sykes +311 +bsc +deliveries +1720 +invasions +carmichael +tajikistan +thematic +ecumenical +sentiments +onstage +##rians +##brand +##sume +catastrophic +flanks +molten +##arns +waller +aimee +terminating +##icing +alternately +##oche +nehru +printers +outraged +##eving +empires +template +banners +repetitive +za +##oise +vegetarian +##tell +guiana +opt +cavendish +lucknow +synthesized +##hani +##mada +finalized +##ctable +fictitious +mayoral +unreliable +##enham +embracing +peppers +rbis +##chio +##neo +inhibition +slashed +togo +orderly +embroidered +safari +salty +236 +barron +benito +totaled +##dak +pubs +simulated +caden +devin +tolkien +momma +welding +sesame +##ept +gottingen +hardness +630 +shaman +temeraire +620 +adequately +pediatric +##kit +ck +assertion +radicals +composure +cadence +seafood +beaufort +lazarus +mani +warily +cunning +kurdistan +249 +cantata +##kir +ares +##41 +##clusive +nape +townland +geared +insulted +flutter +boating +violate +draper +dumping +malmo +##hh +##romatic +firearm +alta +bono +obscured +##clave +exceeds +panorama +unbelievable +##train +preschool +##essed +disconnected +installing +rescuing +secretaries +accessibility +##castle +##drive +##ifice +##film +bouts +slug +waterway +mindanao +##buro +##ratic +halves +##ل +calming +liter +maternity +adorable +bragg +electrification +mcc +##dote +roxy +schizophrenia +##body +munoz +kaye +whaling +239 +mil +tingling +tolerant +##ago +unconventional +volcanoes +##finder +deportivo +##llie +robson +kaufman +neuroscience +wai +deportation +masovian +scraping +converse +##bh +hacking +bulge +##oun +administratively +yao +580 +amp +mammoth +booster +claremont +hooper +nomenclature +pursuits +mclaughlin +melinda +##sul +catfish +barclay +substrates +taxa +zee +originals +kimberly +packets +padma +##ality +borrowing +ostensibly +solvent +##bri +##genesis +##mist +lukas +shreveport +veracruz +##ь +##lou +##wives +cheney +tt +anatolia +hobbs +##zyn +cyclic +radiant +alistair +greenish +siena +dat +independents +##bation +conform +pieter +hyper +applicant +bradshaw +spores +telangana +vinci +inexpensive +nuclei +322 +jang +nme +soho +spd +##ign +cradled +receptionist +pow +##43 +##rika +fascism +##ifer +experimenting +##ading +##iec +##region +345 +jocelyn +maris +stair +nocturnal +toro +constabulary +elgin +##kker +msc +##giving +##schen +##rase +doherty +doping +sarcastically +batter +maneuvers +##cano +##apple +##gai +##git +intrinsic +##nst +##stor +1753 +showtime +cafes +gasps +lviv +ushered +##thed +fours +restart +astonishment +transmitting +flyer +shrugs +##sau +intriguing +cones +dictated +mushrooms +medial +##kovsky +##elman +escorting +gaped +##26 +godfather +##door +##sell +djs +recaptured +timetable +vila +1710 +3a +aerodrome +mortals +scientology +##orne +angelina +mag +convection +unpaid +insertion +intermittent +lego +##nated +endeavor +kota +pereira +##lz +304 +bwv +glamorgan +insults +agatha +fey +##cend +fleetwood +mahogany +protruding +steamship +zeta +##arty +mcguire +suspense +##sphere +advising +urges +##wala +hurriedly +meteor +gilded +inline +arroyo +stalker +##oge +excitedly +revered +##cure +earle +introductory +##break +##ilde +mutants +puff +pulses +reinforcement +##haling +curses +lizards +stalk +correlated +##fixed +fallout +macquarie +##unas +bearded +denton +heaving +802 +##ocation +winery +assign +dortmund +##lkirk +everest +invariant +charismatic +susie +##elling +bled +lesley +telegram +sumner +bk +##ogen +##к +wilcox +needy +colbert +duval +##iferous +##mbled +allotted +attends +imperative +##hita +replacements +hawker +##inda +insurgency +##zee +##eke +casts +##yla +680 +ives +transitioned +##pack +##powering +authoritative +baylor +flex +cringed +plaintiffs +woodrow +##skie +drastic +ape +aroma +unfolded +commotion +nt +preoccupied +theta +routines +lasers +privatization +wand +domino +ek +clenching +nsa +strategically +showered +bile +handkerchief +pere +storing +christophe +insulting +316 +nakamura +romani +asiatic +magdalena +palma +cruises +stripping +405 +konstantin +soaring +##berman +colloquially +forerunner +havilland +incarcerated +parasites +sincerity +##utus +disks +plank +saigon +##ining +corbin +homo +ornaments +powerhouse +##tlement +chong +fastened +feasibility +idf +morphological +usable +##nish +##zuki +aqueduct +jaguars +keepers +##flies +aleksandr +faust +assigns +ewing +bacterium +hurled +tricky +hungarians +integers +wallis +321 +yamaha +##isha +hushed +oblivion +aviator +evangelist +friars +##eller +monograph +ode +##nary +airplanes +labourers +charms +##nee +1661 +hagen +tnt +rudder +fiesta +transcript +dorothea +ska +inhibitor +maccabi +retorted +raining +encompassed +clauses +menacing +1642 +lineman +##gist +vamps +##ape +##dick +gloom +##rera +dealings +easing +seekers +##nut +##pment +helens +unmanned +##anu +##isson +basics +##amy +##ckman +adjustments +1688 +brutality +horne +##zell +sui +##55 +##mable +aggregator +##thal +rhino +##drick +##vira +counters +zoom +##01 +##rting +mn +montenegrin +packard +##unciation +##♭ +##kki +reclaim +scholastic +thugs +pulsed +##icia +syriac +quan +saddam +banda +kobe +blaming +buddies +dissent +##lusion +##usia +corbett +jaya +delle +erratic +lexie +##hesis +435 +amiga +hermes +##pressing +##leen +chapels +gospels +jamal +##uating +compute +revolving +warp +##sso +##thes +armory +##eras +##gol +antrim +loki +##kow +##asian +##good +##zano +braid +handwriting +subdistrict +funky +pantheon +##iculate +concurrency +estimation +improper +juliana +##his +newcomers +johnstone +staten +communicated +##oco +##alle +sausage +stormy +##stered +##tters +superfamily +##grade +acidic +collateral +tabloid +##oped +##rza +bladder +austen +##ellant +mcgraw +##hay +hannibal +mein +aquino +lucifer +wo +badger +boar +cher +christensen +greenberg +interruption +##kken +jem +244 +mocked +bottoms +cambridgeshire +##lide +sprawling +##bbly +eastwood +ghent +synth +##buck +advisers +##bah +nominally +hapoel +qu +daggers +estranged +fabricated +towels +vinnie +wcw +misunderstanding +anglia +nothin +unmistakable +##dust +##lova +chilly +marquette +truss +##edge +##erine +reece +##lty +##chemist +##connected +272 +308 +41st +bash +raion +waterfalls +##ump +##main +labyrinth +queue +theorist +##istle +bharatiya +flexed +soundtracks +rooney +leftist +patrolling +wharton +plainly +alleviate +eastman +schuster +topographic +engages +immensely +unbearable +fairchild +1620 +dona +lurking +parisian +oliveira +ia +indictment +hahn +bangladeshi +##aster +vivo +##uming +##ential +antonia +expects +indoors +kildare +harlan +##logue +##ogenic +##sities +forgiven +##wat +childish +tavi +##mide +##orra +plausible +grimm +successively +scooted +##bola +##dget +##rith +spartans +emery +flatly +azure +epilogue +##wark +flourish +##iny +##tracted +##overs +##oshi +bestseller +distressed +receipt +spitting +hermit +topological +##cot +drilled +subunit +francs +##layer +eel +##fk +##itas +octopus +footprint +petitions +ufo +##say +##foil +interfering +leaking +palo +##metry +thistle +valiant +##pic +narayan +mcpherson +##fast +gonzales +##ym +##enne +dustin +novgorod +solos +##zman +doin +##raph +##patient +##meyer +soluble +ashland +cuffs +carole +pendleton +whistling +vassal +##river +deviation +revisited +constituents +rallied +rotate +loomed +##eil +##nting +amateurs +augsburg +auschwitz +crowns +skeletons +##cona +bonnet +257 +dummy +globalization +simeon +sleeper +mandal +differentiated +##crow +##mare +milne +bundled +exasperated +talmud +owes +segregated +##feng +##uary +dentist +piracy +props +##rang +devlin +##torium +malicious +paws +##laid +dependency +##ergy +##fers +##enna +258 +pistons +rourke +jed +grammatical +tres +maha +wig +512 +ghostly +jayne +##achal +##creen +##ilis +##lins +##rence +designate +##with +arrogance +cambodian +clones +showdown +throttle +twain +##ception +lobes +metz +nagoya +335 +braking +##furt +385 +roaming +##minster +amin +crippled +##37 +##llary +indifferent +hoffmann +idols +intimidating +1751 +261 +influenza +memo +onions +1748 +bandage +consciously +##landa +##rage +clandestine +observes +swiped +tangle +##ener +##jected +##trum +##bill +##lta +hugs +congresses +josiah +spirited +##dek +humanist +managerial +filmmaking +inmate +rhymes +debuting +grimsby +ur +##laze +duplicate +vigor +##tf +republished +bolshevik +refurbishment +antibiotics +martini +methane +newscasts +royale +horizons +levant +iain +visas +##ischen +paler +##around +manifestation +snuck +alf +chop +futile +pedestal +rehab +##kat +bmg +kerman +res +fairbanks +jarrett +abstraction +saharan +##zek +1746 +procedural +clearer +kincaid +sash +luciano +##ffey +crunch +helmut +##vara +revolutionaries +##tute +creamy +leach +##mmon +1747 +permitting +nes +plight +wendell +##lese +contra +ts +clancy +ipa +mach +staples +autopsy +disturbances +nueva +karin +pontiac +##uding +proxy +venerable +haunt +leto +bergman +expands +##helm +wal +##pipe +canning +celine +cords +obesity +##enary +intrusion +planner +##phate +reasoned +sequencing +307 +harrow +##chon +##dora +marred +mcintyre +repay +tarzan +darting +248 +harrisburg +margarita +repulsed +##hur +##lding +belinda +hamburger +novo +compliant +runways +bingham +registrar +skyscraper +ic +cuthbert +improvisation +livelihood +##corp +##elial +admiring +##dened +sporadic +believer +casablanca +popcorn +##29 +asha +shovel +##bek +##dice +coiled +tangible +##dez +casper +elsie +resin +tenderness +rectory +##ivision +avail +sonar +##mori +boutique +##dier +guerre +bathed +upbringing +vaulted +sandals +blessings +##naut +##utnant +1680 +306 +foxes +pia +corrosion +hesitantly +confederates +crystalline +footprints +shapiro +tirana +valentin +drones +45th +microscope +shipments +texted +inquisition +wry +guernsey +unauthorized +resigning +760 +ripple +schubert +stu +reassure +felony +##ardo +brittle +koreans +##havan +##ives +dun +implicit +tyres +##aldi +##lth +magnolia +##ehan +##puri +##poulos +aggressively +fei +gr +familiarity +##poo +indicative +##trust +fundamentally +jimmie +overrun +395 +anchors +moans +##opus +britannia +armagh +##ggle +purposely +seizing +##vao +bewildered +mundane +avoidance +cosmopolitan +geometridae +quartermaster +caf +415 +chatter +engulfed +gleam +purge +##icate +juliette +jurisprudence +guerra +revisions +##bn +casimir +brew +##jm +1749 +clapton +cloudy +conde +hermitage +278 +simulations +torches +vincenzo +matteo +##rill +hidalgo +booming +westbound +accomplishment +tentacles +unaffected +##sius +annabelle +flopped +sloping +##litz +dreamer +interceptor +vu +##loh +consecration +copying +messaging +breaker +climates +hospitalized +1752 +torino +afternoons +winfield +witnessing +##teacher +breakers +choirs +sawmill +coldly +##ege +sipping +haste +uninhabited +conical +bibliography +pamphlets +severn +edict +##oca +deux +illnesses +grips +##pl +rehearsals +sis +thinkers +tame +##keepers +1690 +acacia +reformer +##osed +##rys +shuffling +##iring +##shima +eastbound +ionic +rhea +flees +littered +##oum +rocker +vomiting +groaning +champ +overwhelmingly +civilizations +paces +sloop +adoptive +##tish +skaters +##vres +aiding +mango +##joy +nikola +shriek +##ignon +pharmaceuticals +##mg +tuna +calvert +gustavo +stocked +yearbook +##urai +##mana +computed +subsp +riff +hanoi +kelvin +hamid +moors +pastures +summons +jihad +nectar +##ctors +bayou +untitled +pleasing +vastly +republics +intellect +##η +##ulio +##tou +crumbling +stylistic +sb +##ی +consolation +frequented +h₂o +walden +widows +##iens +404 +##ignment +chunks +improves +288 +grit +recited +##dev +snarl +sociological +##arte +##gul +inquired +##held +bruise +clube +consultancy +homogeneous +hornets +multiplication +pasta +prick +savior +##grin +##kou +##phile +yoon +##gara +grimes +vanishing +cheering +reacting +bn +distillery +##quisite +##vity +coe +dockyard +massif +##jord +escorts +voss +##valent +byte +chopped +hawke +illusions +workings +floats +##koto +##vac +kv +annapolis +madden +##onus +alvaro +noctuidae +##cum +##scopic +avenge +steamboat +forte +illustrates +erika +##trip +570 +dew +nationalities +bran +manifested +thirsty +diversified +muscled +reborn +##standing +arson +##lessness +##dran +##logram +##boys +##kushima +##vious +willoughby +##phobia +286 +alsace +dashboard +yuki +##chai +granville +myspace +publicized +tricked +##gang +adjective +##ater +relic +reorganisation +enthusiastically +indications +saxe +##lassified +consolidate +iec +padua +helplessly +ramps +renaming +regulars +pedestrians +accents +convicts +inaccurate +lowers +mana +##pati +barrie +bjp +outta +someplace +berwick +flanking +invoked +marrow +sparsely +excerpts +clothed +rei +##ginal +wept +##straße +##vish +alexa +excel +##ptive +membranes +aquitaine +creeks +cutler +sheppard +implementations +ns +##dur +fragrance +budge +concordia +magnesium +marcelo +##antes +gladly +vibrating +##rral +##ggles +montrose +##omba +lew +seamus +1630 +cocky +##ament +##uen +bjorn +##rrick +fielder +fluttering +##lase +methyl +kimberley +mcdowell +reductions +barbed +##jic +##tonic +aeronautical +condensed +distracting +##promising +huffed +##cala +##sle +claudius +invincible +missy +pious +balthazar +ci +##lang +butte +combo +orson +##dication +myriad +1707 +silenced +##fed +##rh +coco +netball +yourselves +##oza +clarify +heller +peg +durban +etudes +offender +roast +blackmail +curvature +##woods +vile +309 +illicit +suriname +##linson +overture +1685 +bubbling +gymnast +tucking +##mming +##ouin +maldives +##bala +gurney +##dda +##eased +##oides +backside +pinto +jars +racehorse +tending +##rdial +baronetcy +wiener +duly +##rke +barbarian +cupping +flawed +##thesis +bertha +pleistocene +puddle +swearing +##nob +##tically +fleeting +prostate +amulet +educating +##mined +##iti +##tler +75th +jens +respondents +analytics +cavaliers +papacy +raju +##iente +##ulum +##tip +funnel +271 +disneyland +##lley +sociologist +##iam +2500 +faulkner +louvre +menon +##dson +276 +##ower +afterlife +mannheim +peptide +referees +comedians +meaningless +##anger +##laise +fabrics +hurley +renal +sleeps +##bour +##icle +breakout +kristin +roadside +animator +clover +disdain +unsafe +redesign +##urity +firth +barnsley +portage +reset +narrows +268 +commandos +expansive +speechless +tubular +##lux +essendon +eyelashes +smashwords +##yad +##bang +##claim +craved +sprinted +chet +somme +astor +wrocław +orton +266 +bane +##erving +##uing +mischief +##amps +##sund +scaling +terre +##xious +impairment +offenses +undermine +moi +soy +contiguous +arcadia +inuit +seam +##tops +macbeth +rebelled +##icative +##iot +590 +elaborated +frs +uniformed +##dberg +259 +powerless +priscilla +stimulated +980 +qc +arboretum +frustrating +trieste +bullock +##nified +enriched +glistening +intern +##adia +locus +nouvelle +ollie +ike +lash +starboard +ee +tapestry +headlined +hove +rigged +##vite +pollock +##yme +thrive +clustered +cas +roi +gleamed +olympiad +##lino +pressured +regimes +##hosis +##lick +ripley +##ophone +kickoff +gallon +rockwell +##arable +crusader +glue +revolutions +scrambling +1714 +grover +##jure +englishman +aztec +263 +contemplating +coven +ipad +preach +triumphant +tufts +##esian +rotational +##phus +328 +falkland +##brates +strewn +clarissa +rejoin +environmentally +glint +banded +drenched +moat +albanians +johor +rr +maestro +malley +nouveau +shaded +taxonomy +v6 +adhere +bunk +airfields +##ritan +1741 +encompass +remington +tran +##erative +amelie +mazda +friar +morals +passions +##zai +breadth +vis +##hae +argus +burnham +caressing +insider +rudd +##imov +##mini +##rso +italianate +murderous +textual +wainwright +armada +bam +weave +timer +##taken +##nh +fra +##crest +ardent +salazar +taps +tunis +##ntino +allegro +gland +philanthropic +##chester +implication +##optera +esq +judas +noticeably +wynn +##dara +inched +indexed +crises +villiers +bandit +royalties +patterned +cupboard +interspersed +accessory +isla +kendrick +entourage +stitches +##esthesia +headwaters +##ior +interlude +distraught +draught +1727 +##basket +biased +sy +transient +triad +subgenus +adapting +kidd +shortstop +##umatic +dimly +spiked +mcleod +reprint +nellie +pretoria +windmill +##cek +singled +##mps +273 +reunite +##orous +747 +bankers +outlying +##omp +##ports +##tream +apologies +cosmetics +patsy +##deh +##ocks +##yson +bender +nantes +serene +##nad +lucha +mmm +323 +##cius +##gli +cmll +coinage +nestor +juarez +##rook +smeared +sprayed +twitching +sterile +irina +embodied +juveniles +enveloped +miscellaneous +cancers +dq +gulped +luisa +crested +swat +donegal +ref +##anov +##acker +hearst +mercantile +##lika +doorbell +ua +vicki +##alla +##som +bilbao +psychologists +stryker +sw +horsemen +turkmenistan +wits +##national +anson +mathew +screenings +##umb +rihanna +##agne +##nessy +aisles +##iani +##osphere +hines +kenton +saskatoon +tasha +truncated +##champ +##itan +mildred +advises +fredrik +interpreting +inhibitors +##athi +spectroscopy +##hab +##kong +karim +panda +##oia +##nail +##vc +conqueror +kgb +leukemia +##dity +arrivals +cheered +pisa +phosphorus +shielded +##riated +mammal +unitarian +urgently +chopin +sanitary +##mission +spicy +drugged +hinges +##tort +tipping +trier +impoverished +westchester +##caster +267 +epoch +nonstop +##gman +##khov +aromatic +centrally +cerro +##tively +##vio +billions +modulation +sedimentary +283 +facilitating +outrageous +goldstein +##eak +##kt +ld +maitland +penultimate +pollard +##dance +fleets +spaceship +vertebrae +##nig +alcoholism +als +recital +##bham +##ference +##omics +m2 +##bm +trois +##tropical +##в +commemorates +##meric +marge +##raction +1643 +670 +cosmetic +ravaged +##ige +catastrophe +eng +##shida +albrecht +arterial +bellamy +decor +harmon +##rde +bulbs +synchronized +vito +easiest +shetland +shielding +wnba +##glers +##ssar +##riam +brianna +cumbria +##aceous +##rard +cores +thayer +##nsk +brood +hilltop +luminous +carts +keynote +larkin +logos +##cta +##ا +##mund +##quay +lilith +tinted +277 +wrestle +mobilization +##uses +sequential +siam +bloomfield +takahashi +274 +##ieving +presenters +ringo +blazed +witty +##oven +##ignant +devastation +haydn +harmed +newt +therese +##peed +gershwin +molina +rabbis +sudanese +001 +innate +restarted +##sack +##fus +slices +wb +##shah +enroll +hypothetical +hysterical +1743 +fabio +indefinite +warped +##hg +exchanging +525 +unsuitable +##sboro +gallo +1603 +bret +cobalt +homemade +##hunter +mx +operatives +##dhar +terraces +durable +latch +pens +whorls +##ctuated +##eaux +billing +ligament +succumbed +##gly +regulators +spawn +##brick +##stead +filmfare +rochelle +##nzo +1725 +circumstance +saber +supplements +##nsky +##tson +crowe +wellesley +carrot +##9th +##movable +primate +drury +sincerely +topical +##mad +##rao +callahan +kyiv +smarter +tits +undo +##yeh +announcements +anthologies +barrio +nebula +##islaus +##shaft +##tyn +bodyguards +2021 +assassinate +barns +emmett +scully +##mah +##yd +##eland +##tino +##itarian +demoted +gorman +lashed +prized +adventist +writ +##gui +alla +invertebrates +##ausen +1641 +amman +1742 +align +healy +redistribution +##gf +##rize +insulation +##drop +adherents +hezbollah +vitro +ferns +yanking +269 +php +registering +uppsala +cheerleading +confines +mischievous +tully +##ross +49th +docked +roam +stipulated +pumpkin +##bry +prompt +##ezer +blindly +shuddering +craftsmen +frail +scented +katharine +scramble +shaggy +sponge +helix +zaragoza +279 +##52 +43rd +backlash +fontaine +seizures +posse +cowan +nonfiction +telenovela +wwii +hammered +undone +##gpur +encircled +irs +##ivation +artefacts +oneself +searing +smallpox +##belle +##osaurus +shandong +breached +upland +blushing +rankin +infinitely +psyche +tolerated +docking +evicted +##col +unmarked +##lving +gnome +lettering +litres +musique +##oint +benevolent +##jal +blackened +##anna +mccall +racers +tingle +##ocene +##orestation +introductions +radically +292 +##hiff +##باد +1610 +1739 +munchen +plead +##nka +condo +scissors +##sight +##tens +apprehension +##cey +##yin +hallmark +watering +formulas +sequels +##llas +aggravated +bae +commencing +##building +enfield +prohibits +marne +vedic +civilized +euclidean +jagger +beforehand +blasts +dumont +##arney +##nem +740 +conversions +hierarchical +rios +simulator +##dya +##lellan +hedges +oleg +thrusts +shadowed +darby +maximize +1744 +gregorian +##nded +##routed +sham +unspecified +##hog +emory +factual +##smo +##tp +fooled +##rger +ortega +wellness +marlon +##oton +##urance +casket +keating +ley +enclave +##ayan +char +influencing +jia +##chenko +412 +ammonia +erebidae +incompatible +violins +cornered +##arat +grooves +astronauts +columbian +rampant +fabrication +kyushu +mahmud +vanish +##dern +mesopotamia +##lete +ict +##rgen +caspian +kenji +pitted +##vered +999 +grimace +roanoke +tchaikovsky +twinned +##analysis +##awan +xinjiang +arias +clemson +kazakh +sizable +1662 +##khand +##vard +plunge +tatum +vittorio +##nden +cholera +##dana +##oper +bracing +indifference +projectile +superliga +##chee +realises +upgrading +299 +porte +retribution +##vies +nk +stil +##resses +ama +bureaucracy +blackberry +bosch +testosterone +collapses +greer +##pathic +ioc +fifties +malls +##erved +bao +baskets +adolescents +siegfried +##osity +##tosis +mantra +detecting +existent +fledgling +##cchi +dissatisfied +gan +telecommunication +mingled +sobbed +6000 +controversies +outdated +taxis +##raus +fright +slams +##lham +##fect +##tten +detectors +fetal +tanned +##uw +fray +goth +olympian +skipping +mandates +scratches +sheng +unspoken +hyundai +tracey +hotspur +restrictive +##buch +americana +mundo +##bari +burroughs +diva +vulcan +##6th +distinctions +thumping +##ngen +mikey +sheds +fide +rescues +springsteen +vested +valuation +##ece +##ely +pinnacle +rake +sylvie +##edo +almond +quivering +##irus +alteration +faltered +##wad +51st +hydra +ticked +##kato +recommends +##dicated +antigua +arjun +stagecoach +wilfred +trickle +pronouns +##pon +aryan +nighttime +##anian +gall +pea +stitch +##hei +leung +milos +##dini +eritrea +nexus +starved +snowfall +kant +parasitic +cot +discus +hana +strikers +appleton +kitchens +##erina +##partisan +##itha +##vius +disclose +metis +##channel +1701 +tesla +##vera +fitch +1735 +blooded +##tila +decimal +##tang +##bai +cyclones +eun +bottled +peas +pensacola +basha +bolivian +crabs +boil +lanterns +partridge +roofed +1645 +necks +##phila +opined +patting +##kla +##lland +chuckles +volta +whereupon +##nche +devout +euroleague +suicidal +##dee +inherently +involuntary +knitting +nasser +##hide +puppets +colourful +courageous +southend +stills +miraculous +hodgson +richer +rochdale +ethernet +greta +uniting +prism +umm +##haya +##itical +##utation +deterioration +pointe +prowess +##ropriation +lids +scranton +billings +subcontinent +##koff +##scope +brute +kellogg +psalms +degraded +##vez +stanisław +##ructured +ferreira +pun +astonishing +gunnar +##yat +arya +prc +gottfried +##tight +excursion +##ographer +dina +##quil +##nare +huffington +illustrious +wilbur +gundam +verandah +##zard +naacp +##odle +constructive +fjord +kade +##naud +generosity +thrilling +baseline +cayman +frankish +plastics +accommodations +zoological +##fting +cedric +qb +motorized +##dome +##otted +squealed +tackled +canucks +budgets +situ +asthma +dail +gabled +grasslands +whimpered +writhing +judgments +##65 +minnie +pv +##carbon +bananas +grille +domes +monique +odin +maguire +markham +tierney +##estra +##chua +libel +poke +speedy +atrium +laval +notwithstanding +##edly +fai +kala +##sur +robb +##sma +listings +luz +supplementary +tianjin +##acing +enzo +jd +ric +scanner +croats +transcribed +##49 +arden +cv +##hair +##raphy +##lver +##uy +357 +seventies +staggering +alam +horticultural +hs +regression +timbers +blasting +##ounded +montagu +manipulating +##cit +catalytic +1550 +troopers +##meo +condemnation +fitzpatrick +##oire +##roved +inexperienced +1670 +castes +##lative +outing +314 +dubois +flicking +quarrel +ste +learners +1625 +iq +whistled +##class +282 +classify +tariffs +temperament +355 +folly +liszt +##yles +immersed +jordanian +ceasefire +apparel +extras +maru +fished +##bio +harta +stockport +assortment +craftsman +paralysis +transmitters +##cola +blindness +##wk +fatally +proficiency +solemnly +##orno +repairing +amore +groceries +ultraviolet +##chase +schoolhouse +##tua +resurgence +nailed +##otype +##× +ruse +saliva +diagrams +##tructing +albans +rann +thirties +1b +antennas +hilarious +cougars +paddington +stats +##eger +breakaway +ipod +reza +authorship +prohibiting +scoffed +##etz +##ttle +conscription +defected +trondheim +##fires +ivanov +keenan +##adan +##ciful +##fb +##slow +locating +##ials +##tford +cadiz +basalt +blankly +interned +rags +rattling +##tick +carpathian +reassured +sync +bum +guildford +iss +staunch +##onga +astronomers +sera +sofie +emergencies +susquehanna +##heard +duc +mastery +vh1 +williamsburg +bayer +buckled +craving +##khan +##rdes +bloomington +##write +alton +barbecue +##bians +justine +##hri +##ndt +delightful +smartphone +newtown +photon +retrieval +peugeot +hissing +##monium +##orough +flavors +lighted +relaunched +tainted +##games +##lysis +anarchy +microscopic +hopping +adept +evade +evie +##beau +inhibit +sinn +adjustable +hurst +intuition +wilton +cisco +44th +lawful +lowlands +stockings +thierry +##dalen +##hila +##nai +fates +prank +tb +maison +lobbied +provocative +1724 +4a +utopia +##qual +carbonate +gujarati +purcell +##rford +curtiss +##mei +overgrown +arenas +mediation +swallows +##rnik +respectful +turnbull +##hedron +##hope +alyssa +ozone +##ʻi +ami +gestapo +johansson +snooker +canteen +cuff +declines +empathy +stigma +##ags +##iner +##raine +taxpayers +gui +volga +##wright +##copic +lifespan +overcame +tattooed +enactment +giggles +##ador +##camp +barrington +bribe +obligatory +orbiting +peng +##enas +elusive +sucker +##vating +cong +hardship +empowered +anticipating +estrada +cryptic +greasy +detainees +planck +sudbury +plaid +dod +marriott +kayla +##ears +##vb +##zd +mortally +##hein +cognition +radha +319 +liechtenstein +meade +richly +argyle +harpsichord +liberalism +trumpets +lauded +tyrant +salsa +tiled +lear +promoters +reused +slicing +trident +##chuk +##gami +##lka +cantor +checkpoint +##points +gaul +leger +mammalian +##tov +##aar +##schaft +doha +frenchman +nirvana +##vino +delgado +headlining +##eron +##iography +jug +tko +1649 +naga +intersections +##jia +benfica +nawab +##suka +ashford +gulp +##deck +##vill +##rug +brentford +frazier +pleasures +dunne +potsdam +shenzhen +dentistry +##tec +flanagan +##dorff +##hear +chorale +dinah +prem +quezon +##rogated +relinquished +sutra +terri +##pani +flaps +##rissa +poly +##rnet +homme +aback +##eki +linger +womb +##kson +##lewood +doorstep +orthodoxy +threaded +westfield +##rval +dioceses +fridays +subsided +##gata +loyalists +##biotic +##ettes +letterman +lunatic +prelate +tenderly +invariably +souza +thug +winslow +##otide +furlongs +gogh +jeopardy +##runa +pegasus +##umble +humiliated +standalone +tagged +##roller +freshmen +klan +##bright +attaining +initiating +transatlantic +logged +viz +##uance +1723 +combatants +intervening +stephane +chieftain +despised +grazed +317 +cdc +galveston +godzilla +macro +simulate +##planes +parades +##esses +960 +##ductive +##unes +equator +overdose +##cans +##hosh +##lifting +joshi +epstein +sonora +treacherous +aquatics +manchu +responsive +##sation +supervisory +##christ +##llins +##ibar +##balance +##uso +kimball +karlsruhe +mab +##emy +ignores +phonetic +reuters +spaghetti +820 +almighty +danzig +rumbling +tombstone +designations +lured +outset +##felt +supermarkets +##wt +grupo +kei +kraft +susanna +##blood +comprehension +genealogy +##aghan +##verted +redding +##ythe +1722 +bowing +##pore +##roi +lest +sharpened +fulbright +valkyrie +sikhs +##unds +swans +bouquet +merritt +##tage +##venting +commuted +redhead +clerks +leasing +cesare +dea +hazy +##vances +fledged +greenfield +servicemen +##gical +armando +blackout +dt +sagged +downloadable +intra +potion +pods +##4th +##mism +xp +attendants +gambia +stale +##ntine +plump +asteroids +rediscovered +buds +flea +hive +##neas +1737 +classifications +debuts +##eles +olympus +scala +##eurs +##gno +##mute +hummed +sigismund +visuals +wiggled +await +pilasters +clench +sulfate +##ances +bellevue +enigma +trainee +snort +##sw +clouded +denim +##rank +##rder +churning +hartman +lodges +riches +sima +##missible +accountable +socrates +regulates +mueller +##cr +1702 +avoids +solids +himalayas +nutrient +pup +##jevic +squat +fades +nec +##lates +##pina +##rona +##ου +privateer +tequila +##gative +##mpton +apt +hornet +immortals +##dou +asturias +cleansing +dario +##rries +##anta +etymology +servicing +zhejiang +##venor +##nx +horned +erasmus +rayon +relocating +£10 +##bags +escalated +promenade +stubble +2010s +artisans +axial +liquids +mora +sho +yoo +##tsky +bundles +oldies +##nally +notification +bastion +##ths +sparkle +##lved +1728 +leash +pathogen +highs +##hmi +immature +880 +gonzaga +ignatius +mansions +monterrey +sweets +bryson +##loe +polled +regatta +brightest +pei +rosy +squid +hatfield +payroll +addict +meath +cornerback +heaviest +lodging +##mage +capcom +rippled +##sily +barnet +mayhem +ymca +snuggled +rousseau +##cute +blanchard +284 +fragmented +leighton +chromosomes +risking +##md +##strel +##utter +corinne +coyotes +cynical +hiroshi +yeomanry +##ractive +ebook +grading +mandela +plume +agustin +magdalene +##rkin +bea +femme +trafford +##coll +##lun +##tance +52nd +fourier +upton +##mental +camilla +gust +iihf +islamabad +longevity +##kala +feldman +netting +##rization +endeavour +foraging +mfa +orr +##open +greyish +contradiction +graz +##ruff +handicapped +marlene +tweed +oaxaca +spp +campos +miocene +pri +configured +cooks +pluto +cozy +pornographic +##entes +70th +fairness +glided +jonny +lynne +rounding +sired +##emon +##nist +remade +uncover +##mack +complied +lei +newsweek +##jured +##parts +##enting +##pg +293 +finer +guerrillas +athenian +deng +disused +stepmother +accuse +gingerly +seduction +521 +confronting +##walker +##going +gora +nostalgia +sabres +virginity +wrenched +##minated +syndication +wielding +eyre +##56 +##gnon +##igny +behaved +taxpayer +sweeps +##growth +childless +gallant +##ywood +amplified +geraldine +scrape +##ffi +babylonian +fresco +##rdan +##kney +##position +1718 +restricting +tack +fukuoka +osborn +selector +partnering +##dlow +318 +gnu +kia +tak +whitley +gables +##54 +##mania +mri +softness +immersion +##bots +##evsky +1713 +chilling +insignificant +pcs +##uis +elites +lina +purported +supplemental +teaming +##americana +##dding +##inton +proficient +rouen +##nage +##rret +niccolo +selects +##bread +fluffy +1621 +gruff +knotted +mukherjee +polgara +thrash +nicholls +secluded +smoothing +thru +corsica +loaf +whitaker +inquiries +##rrier +##kam +indochina +289 +marlins +myles +peking +##tea +extracts +pastry +superhuman +connacht +vogel +##ditional +##het +##udged +##lash +gloss +quarries +refit +teaser +##alic +##gaon +20s +materialized +sling +camped +pickering +tung +tracker +pursuant +##cide +cranes +soc +##cini +##typical +##viere +anhalt +overboard +workout +chores +fares +orphaned +stains +##logie +fenton +surpassing +joyah +triggers +##itte +grandmaster +##lass +##lists +clapping +fraudulent +ledger +nagasaki +##cor +##nosis +##tsa +eucalyptus +tun +##icio +##rney +##tara +dax +heroism +ina +wrexham +onboard +unsigned +##dates +moshe +galley +winnie +droplets +exiles +praises +watered +noodles +##aia +fein +adi +leland +multicultural +stink +bingo +comets +erskine +modernized +canned +constraint +domestically +chemotherapy +featherweight +stifled +##mum +darkly +irresistible +refreshing +hasty +isolate +##oys +kitchener +planners +##wehr +cages +yarn +implant +toulon +elects +childbirth +yue +##lind +##lone +cn +rightful +sportsman +junctions +remodeled +specifies +##rgh +291 +##oons +complimented +##urgent +lister +ot +##logic +bequeathed +cheekbones +fontana +gabby +##dial +amadeus +corrugated +maverick +resented +triangles +##hered +##usly +nazareth +tyrol +1675 +assent +poorer +sectional +aegean +##cous +296 +nylon +ghanaian +##egorical +##weig +cushions +forbid +fusiliers +obstruction +somerville +##scia +dime +earrings +elliptical +leyte +oder +polymers +timmy +atm +midtown +piloted +settles +continual +externally +mayfield +##uh +enrichment +henson +keane +persians +1733 +benji +braden +pep +324 +##efe +contenders +pepsi +valet +##isches +298 +##asse +##earing +goofy +stroll +##amen +authoritarian +occurrences +adversary +ahmedabad +tangent +toppled +dorchester +1672 +modernism +marxism +islamist +charlemagne +exponential +racks +unicode +brunette +mbc +pic +skirmish +##bund +##lad +##powered +##yst +hoisted +messina +shatter +##ctum +jedi +vantage +##music +##neil +clemens +mahmoud +corrupted +authentication +lowry +nils +##washed +omnibus +wounding +jillian +##itors +##opped +serialized +narcotics +handheld +##arm +##plicity +intersecting +stimulating +##onis +crate +fellowships +hemingway +casinos +climatic +fordham +copeland +drip +beatty +leaflets +robber +brothel +madeira +##hedral +sphinx +ultrasound +##vana +valor +forbade +leonid +villas +##aldo +duane +marquez +##cytes +disadvantaged +forearms +kawasaki +reacts +consular +lax +uncles +uphold +##hopper +concepcion +dorsey +lass +##izan +arching +passageway +1708 +researches +tia +internationals +##graphs +##opers +distinguishes +javanese +divert +##uven +plotted +##listic +##rwin +##erik +##tify +affirmative +signifies +validation +##bson +kari +felicity +georgina +zulu +##eros +##rained +##rath +overcoming +##dot +argyll +##rbin +1734 +chiba +ratification +windy +earls +parapet +##marks +hunan +pristine +astrid +punta +##gart +brodie +##kota +##oder +malaga +minerva +rouse +##phonic +bellowed +pagoda +portals +reclamation +##gur +##odies +##⁄₄ +parentheses +quoting +allergic +palette +showcases +benefactor +heartland +nonlinear +##tness +bladed +cheerfully +scans +##ety +##hone +1666 +girlfriends +pedersen +hiram +sous +##liche +##nator +1683 +##nery +##orio +##umen +bobo +primaries +smiley +##cb +unearthed +uniformly +fis +metadata +1635 +ind +##oted +recoil +##titles +##tura +##ια +406 +hilbert +jamestown +mcmillan +tulane +seychelles +##frid +antics +coli +fated +stucco +##grants +1654 +bulky +accolades +arrays +caledonian +carnage +optimism +puebla +##tative +##cave +enforcing +rotherham +seo +dunlop +aeronautics +chimed +incline +zoning +archduke +hellenistic +##oses +##sions +candi +thong +##ople +magnate +rustic +##rsk +projective +slant +##offs +danes +hollis +vocalists +##ammed +congenital +contend +gesellschaft +##ocating +##pressive +douglass +quieter +##cm +##kshi +howled +salim +spontaneously +townsville +buena +southport +##bold +kato +1638 +faerie +stiffly +##vus +##rled +297 +flawless +realising +taboo +##7th +bytes +straightening +356 +jena +##hid +##rmin +cartwright +berber +bertram +soloists +411 +noses +417 +coping +fission +hardin +inca +##cen +1717 +mobilized +vhf +##raf +biscuits +curate +##85 +##anial +331 +gaunt +neighbourhoods +1540 +##abas +blanca +bypassed +sockets +behold +coincidentally +##bane +nara +shave +splinter +terrific +##arion +##erian +commonplace +juris +redwood +waistband +boxed +caitlin +fingerprints +jennie +naturalized +##ired +balfour +craters +jody +bungalow +hugely +quilt +glitter +pigeons +undertaker +bulging +constrained +goo +##sil +##akh +assimilation +reworked +##person +persuasion +##pants +felicia +##cliff +##ulent +1732 +explodes +##dun +##inium +##zic +lyman +vulture +hog +overlook +begs +northwards +ow +spoil +##urer +fatima +favorably +accumulate +sargent +sorority +corresponded +dispersal +kochi +toned +##imi +##lita +internacional +newfound +##agger +##lynn +##rigue +booths +peanuts +##eborg +medicare +muriel +nur +##uram +crates +millennia +pajamas +worsened +##breakers +jimi +vanuatu +yawned +##udeau +carousel +##hony +hurdle +##ccus +##mounted +##pod +rv +##eche +airship +ambiguity +compulsion +recapture +##claiming +arthritis +##osomal +1667 +asserting +ngc +sniffing +dade +discontent +glendale +ported +##amina +defamation +rammed +##scent +fling +livingstone +##fleet +875 +##ppy +apocalyptic +comrade +lcd +##lowe +cessna +eine +persecuted +subsistence +demi +hoop +reliefs +710 +coptic +progressing +stemmed +perpetrators +1665 +priestess +##nio +dobson +ebony +rooster +itf +tortricidae +##bbon +##jian +cleanup +##jean +##øy +1721 +eighties +taxonomic +holiness +##hearted +##spar +antilles +showcasing +stabilized +##nb +gia +mascara +michelangelo +dawned +##uria +##vinsky +extinguished +fitz +grotesque +£100 +##fera +##loid +##mous +barges +neue +throbbed +cipher +johnnie +##a1 +##mpt +outburst +##swick +spearheaded +administrations +c1 +heartbreak +pixels +pleasantly +##enay +lombardy +plush +##nsed +bobbie +##hly +reapers +tremor +xiang +minogue +substantive +hitch +barak +##wyl +kwan +##encia +910 +obscene +elegance +indus +surfer +bribery +conserve +##hyllum +##masters +horatio +##fat +apes +rebound +psychotic +##pour +iteration +##mium +##vani +botanic +horribly +antiques +dispose +paxton +##hli +##wg +timeless +1704 +disregard +engraver +hounds +##bau +##version +looted +uno +facilitates +groans +masjid +rutland +antibody +disqualification +decatur +footballers +quake +slacks +48th +rein +scribe +stabilize +commits +exemplary +tho +##hort +##chison +pantry +traversed +##hiti +disrepair +identifiable +vibrated +baccalaureate +##nnis +csa +interviewing +##iensis +##raße +greaves +wealthiest +343 +classed +jogged +£5 +##58 +##atal +illuminating +knicks +respecting +##uno +scrubbed +##iji +##dles +kruger +moods +growls +raider +silvia +chefs +kam +vr +cree +percival +##terol +gunter +counterattack +defiant +henan +ze +##rasia +##riety +equivalence +submissions +##fra +##thor +bautista +mechanically +##heater +cornice +herbal +templar +##mering +outputs +ruining +ligand +renumbered +extravagant +mika +blockbuster +eta +insurrection +##ilia +darkening +ferocious +pianos +strife +kinship +##aer +melee +##anor +##iste +##may +##oue +decidedly +weep +##jad +##missive +##ppel +354 +puget +unease +##gnant +1629 +hammering +kassel +ob +wessex +##lga +bromwich +egan +paranoia +utilization +##atable +##idad +contradictory +provoke +##ols +##ouring +##tangled +knesset +##very +##lette +plumbing +##sden +##¹ +greensboro +occult +sniff +338 +zev +beaming +gamer +haggard +mahal +##olt +##pins +mendes +utmost +briefing +gunnery +##gut +##pher +##zh +##rok +1679 +khalifa +sonya +##boot +principals +urbana +wiring +##liffe +##minating +##rrado +dahl +nyu +skepticism +np +townspeople +ithaca +lobster +somethin +##fur +##arina +##−1 +freighter +zimmerman +biceps +contractual +##herton +amend +hurrying +subconscious +##anal +336 +meng +clermont +spawning +##eia +##lub +dignitaries +impetus +snacks +spotting +twigs +##bilis +##cz +##ouk +libertadores +nic +skylar +##aina +##firm +gustave +asean +##anum +dieter +legislatures +flirt +bromley +trolls +umar +##bbies +##tyle +blah +parc +bridgeport +crank +negligence +##nction +46th +constantin +molded +bandages +seriousness +00pm +siegel +carpets +compartments +upbeat +statehood +##dner +##edging +marko +730 +platt +##hane +paving +##iy +1738 +abbess +impatience +limousine +nbl +##talk +441 +lucille +mojo +nightfall +robbers +##nais +karel +brisk +calves +replicate +ascribed +telescopes +##olf +intimidated +##reen +ballast +specialization +##sit +aerodynamic +caliphate +rainer +visionary +##arded +epsilon +##aday +##onte +aggregation +auditory +boosted +reunification +kathmandu +loco +robyn +402 +acknowledges +appointing +humanoid +newell +redeveloped +restraints +##tained +barbarians +chopper +1609 +italiana +##lez +##lho +investigates +wrestlemania +##anies +##bib +690 +##falls +creaked +dragoons +gravely +minions +stupidity +volley +##harat +##week +musik +##eries +##uously +fungal +massimo +semantics +malvern +##ahl +##pee +discourage +embryo +imperialism +1910s +profoundly +##ddled +jiangsu +sparkled +stat +##holz +sweatshirt +tobin +##iction +sneered +##cheon +##oit +brit +causal +smyth +##neuve +diffuse +perrin +silvio +##ipes +##recht +detonated +iqbal +selma +##nism +##zumi +roasted +##riders +tay +##ados +##mament +##mut +##rud +840 +completes +nipples +cfa +flavour +hirsch +##laus +calderon +sneakers +moravian +##ksha +1622 +rq +294 +##imeters +bodo +##isance +##pre +##ronia +anatomical +excerpt +##lke +dh +kunst +##tablished +##scoe +biomass +panted +unharmed +gael +housemates +montpellier +##59 +coa +rodents +tonic +hickory +singleton +##taro +451 +1719 +aldo +breaststroke +dempsey +och +rocco +##cuit +merton +dissemination +midsummer +serials +##idi +haji +polynomials +##rdon +gs +enoch +prematurely +shutter +taunton +£3 +##grating +##inates +archangel +harassed +##asco +326 +archway +dazzling +##ecin +1736 +sumo +wat +##kovich +1086 +honneur +##ently +##nostic +##ttal +##idon +1605 +403 +1716 +blogger +rents +##gnan +hires +##ikh +##dant +howie +##rons +handler +retracted +shocks +1632 +arun +duluth +kepler +trumpeter +##lary +peeking +seasoned +trooper +##mara +laszlo +##iciencies +##rti +heterosexual +##inatory +##ssion +indira +jogging +##inga +##lism +beit +dissatisfaction +malice +##ately +nedra +peeling +##rgeon +47th +stadiums +475 +vertigo +##ains +iced +restroom +##plify +##tub +illustrating +pear +##chner +##sibility +inorganic +rappers +receipts +watery +##kura +lucinda +##oulos +reintroduced +##8th +##tched +gracefully +saxons +nutritional +wastewater +rained +favourites +bedrock +fisted +hallways +likeness +upscale +##lateral +1580 +blinds +prequel +##pps +##tama +deter +humiliating +restraining +tn +vents +1659 +laundering +recess +rosary +tractors +coulter +federer +##ifiers +##plin +persistence +##quitable +geschichte +pendulum +quakers +##beam +bassett +pictorial +buffet +koln +##sitor +drills +reciprocal +shooters +##57 +##cton +##tees +converge +pip +dmitri +donnelly +yamamoto +aqua +azores +demographics +hypnotic +spitfire +suspend +wryly +roderick +##rran +sebastien +##asurable +mavericks +##fles +##200 +himalayan +prodigy +##iance +transvaal +demonstrators +handcuffs +dodged +mcnamara +sublime +1726 +crazed +##efined +##till +ivo +pondered +reconciled +shrill +sava +##duk +bal +cad +heresy +jaipur +goran +##nished +341 +lux +shelly +whitehall +##hre +israelis +peacekeeping +##wled +1703 +demetrius +ousted +##arians +##zos +beale +anwar +backstroke +raged +shrinking +cremated +##yck +benign +towing +wadi +darmstadt +landfill +parana +soothe +colleen +sidewalks +mayfair +tumble +hepatitis +ferrer +superstructure +##gingly +##urse +##wee +anthropological +translators +##mies +closeness +hooves +##pw +mondays +##roll +##vita +landscaping +##urized +purification +sock +thorns +thwarted +jalan +tiberius +##taka +saline +##rito +confidently +khyber +sculptors +##ij +brahms +hammersmith +inspectors +battista +fivb +fragmentation +hackney +##uls +arresting +exercising +antoinette +bedfordshire +##zily +dyed +##hema +1656 +racetrack +variability +##tique +1655 +austrians +deteriorating +madman +theorists +aix +lehman +weathered +1731 +decreed +eruptions +1729 +flaw +quinlan +sorbonne +flutes +nunez +1711 +adored +downwards +fable +rasped +1712 +moritz +mouthful +renegade +shivers +stunts +dysfunction +restrain +translit +327 +pancakes +##avio +##cision +##tray +351 +vial +##lden +bain +##maid +##oxide +chihuahua +malacca +vimes +##rba +##rnier +1664 +donnie +plaques +##ually +337 +bangs +floppy +huntsville +loretta +nikolay +##otte +eater +handgun +ubiquitous +##hett +eras +zodiac +1634 +##omorphic +1820s +##zog +cochran +##bula +##lithic +warring +##rada +dalai +excused +blazers +mcconnell +reeling +bot +este +##abi +geese +hoax +taxon +##bla +guitarists +##icon +condemning +hunts +inversion +moffat +taekwondo +##lvis +1624 +stammered +##rest +##rzy +sousa +fundraiser +marylebone +navigable +uptown +cabbage +daniela +salman +shitty +whimper +##kian +##utive +programmers +protections +rm +##rmi +##rued +forceful +##enes +fuss +##tao +##wash +brat +oppressive +reykjavik +spartak +ticking +##inkles +##kiewicz +adolph +horst +maui +protege +straighten +cpc +landau +concourse +clements +resultant +##ando +imaginative +joo +reactivated +##rem +##ffled +##uising +consultative +##guide +flop +kaitlyn +mergers +parenting +somber +##vron +supervise +vidhan +##imum +courtship +exemplified +harmonies +medallist +refining +##rrow +##ка +amara +##hum +780 +goalscorer +sited +overshadowed +rohan +displeasure +secretive +multiplied +osman +##orth +engravings +padre +##kali +##veda +miniatures +mis +##yala +clap +pali +rook +##cana +1692 +57th +antennae +astro +oskar +1628 +bulldog +crotch +hackett +yucatan +##sure +amplifiers +brno +ferrara +migrating +##gree +thanking +turing +##eza +mccann +ting +andersson +onslaught +gaines +ganga +incense +standardization +##mation +sentai +scuba +stuffing +turquoise +waivers +alloys +##vitt +regaining +vaults +##clops +##gizing +digger +furry +memorabilia +probing +##iad +payton +rec +deutschland +filippo +opaque +seamen +zenith +afrikaans +##filtration +disciplined +inspirational +##merie +banco +confuse +grafton +tod +##dgets +championed +simi +anomaly +biplane +##ceptive +electrode +##para +1697 +cleavage +crossbow +swirl +informant +##lars +##osta +afi +bonfire +spec +##oux +lakeside +slump +##culus +##lais +##qvist +##rrigan +1016 +facades +borg +inwardly +cervical +xl +pointedly +050 +stabilization +##odon +chests +1699 +hacked +ctv +orthogonal +suzy +##lastic +gaulle +jacobite +rearview +##cam +##erted +ashby +##drik +##igate +##mise +##zbek +affectionately +canine +disperse +latham +##istles +##ivar +spielberg +##orin +##idium +ezekiel +cid +##sg +durga +middletown +##cina +customized +frontiers +harden +##etano +##zzy +1604 +bolsheviks +##66 +coloration +yoko +##bedo +briefs +slabs +debra +liquidation +plumage +##oin +blossoms +dementia +subsidy +1611 +proctor +relational +jerseys +parochial +ter +##ici +esa +peshawar +cavalier +loren +cpi +idiots +shamrock +1646 +dutton +malabar +mustache +##endez +##ocytes +referencing +terminates +marche +yarmouth +##sop +acton +mated +seton +subtly +baptised +beige +extremes +jolted +kristina +telecast +##actic +safeguard +waldo +##baldi +##bular +endeavors +sloppy +subterranean +##ensburg +##itung +delicately +pigment +tq +##scu +1626 +##ound +collisions +coveted +herds +##personal +##meister +##nberger +chopra +##ricting +abnormalities +defective +galician +lucie +##dilly +alligator +likened +##genase +burundi +clears +complexion +derelict +deafening +diablo +fingered +champaign +dogg +enlist +isotope +labeling +mrna +##erre +brilliance +marvelous +##ayo +1652 +crawley +ether +footed +dwellers +deserts +hamish +rubs +warlock +skimmed +##lizer +870 +buick +embark +heraldic +irregularities +##ajan +kiara +##kulam +##ieg +antigen +kowalski +##lge +oakley +visitation +##mbit +vt +##suit +1570 +murderers +##miento +##rites +chimneys +##sling +condemn +custer +exchequer +havre +##ghi +fluctuations +##rations +dfb +hendricks +vaccines +##tarian +nietzsche +biking +juicy +##duced +brooding +scrolling +selangor +##ragan +352 +annum +boomed +seminole +sugarcane +##dna +departmental +dismissing +innsbruck +arteries +ashok +batavia +daze +kun +overtook +##rga +##tlan +beheaded +gaddafi +holm +electronically +faulty +galilee +fractures +kobayashi +##lized +gunmen +magma +aramaic +mala +eastenders +inference +messengers +bf +##qu +407 +bathrooms +##vere +1658 +flashbacks +ideally +misunderstood +##jali +##weather +mendez +##grounds +505 +uncanny +##iii +1709 +friendships +##nbc +sacrament +accommodated +reiterated +logistical +pebbles +thumped +##escence +administering +decrees +drafts +##flight +##cased +##tula +futuristic +picket +intimidation +winthrop +##fahan +interfered +339 +afar +francoise +morally +uta +cochin +croft +dwarfs +##bruck +##dents +##nami +biker +##hner +##meral +nano +##isen +##ometric +##pres +##ан +brightened +meek +parcels +securely +gunners +##jhl +##zko +agile +hysteria +##lten +##rcus +bukit +champs +chevy +cuckoo +leith +sadler +theologians +welded +##section +1663 +jj +plurality +xander +##rooms +##formed +shredded +temps +intimately +pau +tormented +##lok +##stellar +1618 +charred +ems +essen +##mmel +alarms +spraying +ascot +blooms +twinkle +##abia +##apes +internment +obsidian +##chaft +snoop +##dav +##ooping +malibu +##tension +quiver +##itia +hays +mcintosh +travers +walsall +##ffie +1623 +beverley +schwarz +plunging +structurally +m3 +rosenthal +vikram +##tsk +770 +ghz +##onda +##tiv +chalmers +groningen +pew +reckon +unicef +##rvis +55th +##gni +1651 +sulawesi +avila +cai +metaphysical +screwing +turbulence +##mberg +augusto +samba +56th +baffled +momentary +toxin +##urian +##wani +aachen +condoms +dali +steppe +##3d +##app +##oed +##year +adolescence +dauphin +electrically +inaccessible +microscopy +nikita +##ega +atv +##cel +##enter +##oles +##oteric +##ы +accountants +punishments +wrongly +bribes +adventurous +clinch +flinders +southland +##hem +##kata +gough +##ciency +lads +soared +##ה +undergoes +deformation +outlawed +rubbish +##arus +##mussen +##nidae +##rzburg +arcs +##ingdon +##tituted +1695 +wheelbase +wheeling +bombardier +campground +zebra +##lices +##oj +##bain +lullaby +##ecure +donetsk +wylie +grenada +##arding +##ης +squinting +eireann +opposes +##andra +maximal +runes +##broken +##cuting +##iface +##ror +##rosis +additive +britney +adultery +triggering +##drome +detrimental +aarhus +containment +jc +swapped +vichy +##ioms +madly +##oric +##rag +brant +##ckey +##trix +1560 +1612 +broughton +rustling +##stems +##uder +asbestos +mentoring +##nivorous +finley +leaps +##isan +apical +pry +slits +substitutes +##dict +intuitive +fantasia +insistent +unreasonable +##igen +##vna +domed +hannover +margot +ponder +##zziness +impromptu +jian +lc +rampage +stemming +##eft +andrey +gerais +whichever +amnesia +appropriated +anzac +clicks +modifying +ultimatum +cambrian +maids +verve +yellowstone +##mbs +conservatoire +##scribe +adherence +dinners +spectra +imperfect +mysteriously +sidekick +tatar +tuba +##aks +##ifolia +distrust +##athan +##zle +c2 +ronin +zac +##pse +celaena +instrumentalist +scents +skopje +##mbling +comical +compensated +vidal +condor +intersect +jingle +wavelengths +##urrent +mcqueen +##izzly +carp +weasel +422 +kanye +militias +postdoctoral +eugen +gunslinger +##ɛ +faux +hospice +##for +appalled +derivation +dwarves +##elis +dilapidated +##folk +astoria +philology +##lwyn +##otho +##saka +inducing +philanthropy +##bf +##itative +geek +markedly +sql +##yce +bessie +indices +rn +##flict +495 +frowns +resolving +weightlifting +tugs +cleric +contentious +1653 +mania +rms +##miya +##reate +##ruck +##tucket +bien +eels +marek +##ayton +##cence +discreet +unofficially +##ife +leaks +##bber +1705 +332 +dung +compressor +hillsborough +pandit +shillings +distal +##skin +381 +##tat +##you +nosed +##nir +mangrove +undeveloped +##idia +textures +##inho +##500 +##rise +ae +irritating +nay +amazingly +bancroft +apologetic +compassionate +kata +symphonies +##lovic +airspace +##lch +930 +gifford +precautions +fulfillment +sevilla +vulgar +martinique +##urities +looting +piccolo +tidy +##dermott +quadrant +armchair +incomes +mathematicians +stampede +nilsson +##inking +##scan +foo +quarterfinal +##ostal +shang +shouldered +squirrels +##owe +344 +vinegar +##bner +##rchy +##systems +delaying +##trics +ars +dwyer +rhapsody +sponsoring +##gration +bipolar +cinder +starters +##olio +##urst +421 +signage +##nty +aground +figurative +mons +acquaintances +duets +erroneously +soyuz +elliptic +recreated +##cultural +##quette +##ssed +##tma +##zcz +moderator +scares +##itaire +##stones +##udence +juniper +sighting +##just +##nsen +britten +calabria +ry +bop +cramer +forsyth +stillness +##л +airmen +gathers +unfit +##umber +##upt +taunting +##rip +seeker +streamlined +##bution +holster +schumann +tread +vox +##gano +##onzo +strive +dil +reforming +covent +newbury +predicting +##orro +decorate +tre +##puted +andover +ie +asahi +dept +dunkirk +gills +##tori +buren +huskies +##stis +##stov +abstracts +bets +loosen +##opa +1682 +yearning +##glio +##sir +berman +effortlessly +enamel +napoli +persist +##peration +##uez +attache +elisa +b1 +invitations +##kic +accelerating +reindeer +boardwalk +clutches +nelly +polka +starbucks +##kei +adamant +huey +lough +unbroken +adventurer +embroidery +inspecting +stanza +##ducted +naia +taluka +##pone +##roids +chases +deprivation +florian +##jing +##ppet +earthly +##lib +##ssee +colossal +foreigner +vet +freaks +patrice +rosewood +triassic +upstate +##pkins +dominates +ata +chants +ks +vo +##400 +##bley +##raya +##rmed +555 +agra +infiltrate +##ailing +##ilation +##tzer +##uppe +##werk +binoculars +enthusiast +fujian +squeak +##avs +abolitionist +almeida +boredom +hampstead +marsden +rations +##ands +inflated +334 +bonuses +rosalie +patna +##rco +329 +detachments +penitentiary +54th +flourishing +woolf +##dion +##etched +papyrus +##lster +##nsor +##toy +bobbed +dismounted +endelle +inhuman +motorola +tbs +wince +wreath +##ticus +hideout +inspections +sanjay +disgrace +infused +pudding +stalks +##urbed +arsenic +leases +##hyl +##rrard +collarbone +##waite +##wil +dowry +##bant +##edance +genealogical +nitrate +salamanca +scandals +thyroid +necessitated +##! +##" +### +##$ +##% +##& +##' +##( +##) +##* +##+ +##, +##- +##. +##/ +##: +##; +##< +##= +##> +##? +##@ +##[ +##\ +##] +##^ +##_ +##` +##{ +##| +##} +##~ +##¡ +##¢ +##£ +##¤ +##¥ +##¦ +##§ +##¨ +##© +##ª +##« +##¬ +##® +##± +##´ +##µ +##¶ +##· +##º +##» +##¼ +##¾ +##¿ +##æ +##ð +##÷ +##þ +##đ +##ħ +##ŋ +##œ +##ƒ +##ɐ +##ɑ +##ɒ +##ɔ +##ɕ +##ə +##ɡ +##ɣ +##ɨ +##ɪ +##ɫ +##ɬ +##ɯ +##ɲ +##ɴ +##ɹ +##ɾ +##ʀ +##ʁ +##ʂ +##ʃ +##ʉ +##ʊ +##ʋ +##ʌ +##ʎ +##ʐ +##ʑ +##ʒ +##ʔ +##ʰ +##ʲ +##ʳ +##ʷ +##ʸ +##ʻ +##ʼ +##ʾ +##ʿ +##ˈ +##ˡ +##ˢ +##ˣ +##ˤ +##β +##γ +##δ +##ε +##ζ +##θ +##κ +##λ +##μ +##ξ +##ο +##π +##ρ +##σ +##τ +##υ +##φ +##χ +##ψ +##ω +##б +##г +##д +##ж +##з +##м +##п +##с +##у +##ф +##х +##ц +##ч +##ш +##щ +##ъ +##э +##ю +##ђ +##є +##і +##ј +##љ +##њ +##ћ +##ӏ +##ա +##բ +##գ +##դ +##ե +##թ +##ի +##լ +##կ +##հ +##մ +##յ +##ն +##ո +##պ +##ս +##վ +##տ +##ր +##ւ +##ք +##־ +##א +##ב +##ג +##ד +##ו +##ז +##ח +##ט +##י +##ך +##כ +##ל +##ם +##מ +##ן +##נ +##ס +##ע +##ף +##פ +##ץ +##צ +##ק +##ר +##ש +##ת +##، +##ء +##ب +##ت +##ث +##ج +##ح +##خ +##ذ +##ز +##س +##ش +##ص +##ض +##ط +##ظ +##ع +##غ +##ـ +##ف +##ق +##ك +##و +##ى +##ٹ +##پ +##چ +##ک +##گ +##ں +##ھ +##ہ +##ے +##अ +##आ +##उ +##ए +##क +##ख +##ग +##च +##ज +##ट +##ड +##ण +##त +##थ +##द +##ध +##न +##प +##ब +##भ +##म +##य +##र +##ल +##व +##श +##ष +##स +##ह +##ा +##ि +##ी +##ो +##। +##॥ +##ং +##অ +##আ +##ই +##উ +##এ +##ও +##ক +##খ +##গ +##চ +##ছ +##জ +##ট +##ড +##ণ +##ত +##থ +##দ +##ধ +##ন +##প +##ব +##ভ +##ম +##য +##র +##ল +##শ +##ষ +##স +##হ +##া +##ি +##ী +##ে +##க +##ச +##ட +##த +##ந +##ன +##ப +##ம +##ய +##ர +##ல +##ள +##வ +##ா +##ி +##ு +##ே +##ை +##ನ +##ರ +##ಾ +##ක +##ය +##ර +##ල +##ව +##ා +##ก +##ง +##ต +##ท +##น +##พ +##ม +##ย +##ร +##ล +##ว +##ส +##อ +##า +##เ +##་ +##། +##ག +##ང +##ད +##ན +##པ +##བ +##མ +##འ +##ར +##ལ +##ས +##မ +##ა +##ბ +##გ +##დ +##ე +##ვ +##თ +##ი +##კ +##ლ +##მ +##ნ +##ო +##რ +##ს +##ტ +##უ +##ᄀ +##ᄂ +##ᄃ +##ᄅ +##ᄆ +##ᄇ +##ᄉ +##ᄊ +##ᄋ +##ᄌ +##ᄎ +##ᄏ +##ᄐ +##ᄑ +##ᄒ +##ᅡ +##ᅢ +##ᅥ +##ᅦ +##ᅧ +##ᅩ +##ᅪ +##ᅭ +##ᅮ +##ᅯ +##ᅲ +##ᅳ +##ᅴ +##ᅵ +##ᆨ +##ᆫ +##ᆯ +##ᆷ +##ᆸ +##ᆼ +##ᴬ +##ᴮ +##ᴰ +##ᴵ +##ᴺ +##ᵀ +##ᵃ +##ᵇ +##ᵈ +##ᵉ +##ᵍ +##ᵏ +##ᵐ +##ᵒ +##ᵖ +##ᵗ +##ᵘ +##ᵣ +##ᵤ +##ᵥ +##ᶜ +##ᶠ +##‐ +##‑ +##‒ +##– +##— +##― +##‖ +##‘ +##’ +##‚ +##“ +##” +##„ +##† +##‡ +##• +##… +##‰ +##′ +##″ +##› +##‿ +##⁄ +##⁰ +##ⁱ +##⁴ +##⁵ +##⁶ +##⁷ +##⁸ +##⁹ +##⁻ +##ⁿ +##₅ +##₆ +##₇ +##₈ +##₉ +##₊ +##₍ +##₎ +##ₐ +##ₑ +##ₒ +##ₓ +##ₕ +##ₖ +##ₗ +##ₘ +##ₚ +##ₛ +##ₜ +##₤ +##₩ +##€ +##₱ +##₹ +##ℓ +##№ +##ℝ +##™ +##⅓ +##⅔ +##← +##↑ +##→ +##↓ +##↔ +##↦ +##⇄ +##⇌ +##⇒ +##∂ +##∅ +##∆ +##∇ +##∈ +##∗ +##∘ +##√ +##∞ +##∧ +##∨ +##∩ +##∪ +##≈ +##≡ +##≤ +##≥ +##⊂ +##⊆ +##⊕ +##⊗ +##⋅ +##─ +##│ +##■ +##▪ +##● +##★ +##☆ +##☉ +##♠ +##♣ +##♥ +##♦ +##♯ +##⟨ +##⟩ +##ⱼ +##⺩ +##⺼ +##⽥ +##、 +##。 +##〈 +##〉 +##《 +##》 +##「 +##」 +##『 +##』 +##〜 +##あ +##い +##う +##え +##お +##か +##き +##く +##け +##こ +##さ +##し +##す +##せ +##そ +##た +##ち +##っ +##つ +##て +##と +##な +##に +##ぬ +##ね +##の +##は +##ひ +##ふ +##へ +##ほ +##ま +##み +##む +##め +##も +##や +##ゆ +##よ +##ら +##り +##る +##れ +##ろ +##を +##ん +##ァ +##ア +##ィ +##イ +##ウ +##ェ +##エ +##オ +##カ +##キ +##ク +##ケ +##コ +##サ +##シ +##ス +##セ +##タ +##チ +##ッ +##ツ +##テ +##ト +##ナ +##ニ +##ノ +##ハ +##ヒ +##フ +##ヘ +##ホ +##マ +##ミ +##ム +##メ +##モ +##ャ +##ュ +##ョ +##ラ +##リ +##ル +##レ +##ロ +##ワ +##ン +##・ +##ー +##一 +##三 +##上 +##下 +##不 +##世 +##中 +##主 +##久 +##之 +##也 +##事 +##二 +##五 +##井 +##京 +##人 +##亻 +##仁 +##介 +##代 +##仮 +##伊 +##会 +##佐 +##侍 +##保 +##信 +##健 +##元 +##光 +##八 +##公 +##内 +##出 +##分 +##前 +##劉 +##力 +##加 +##勝 +##北 +##区 +##十 +##千 +##南 +##博 +##原 +##口 +##古 +##史 +##司 +##合 +##吉 +##同 +##名 +##和 +##囗 +##四 +##国 +##國 +##土 +##地 +##坂 +##城 +##堂 +##場 +##士 +##夏 +##外 +##大 +##天 +##太 +##夫 +##奈 +##女 +##子 +##学 +##宀 +##宇 +##安 +##宗 +##定 +##宣 +##宮 +##家 +##宿 +##寺 +##將 +##小 +##尚 +##山 +##岡 +##島 +##崎 +##川 +##州 +##巿 +##帝 +##平 +##年 +##幸 +##广 +##弘 +##張 +##彳 +##後 +##御 +##德 +##心 +##忄 +##志 +##忠 +##愛 +##成 +##我 +##戦 +##戸 +##手 +##扌 +##政 +##文 +##新 +##方 +##日 +##明 +##星 +##春 +##昭 +##智 +##曲 +##書 +##月 +##有 +##朝 +##木 +##本 +##李 +##村 +##東 +##松 +##林 +##森 +##楊 +##樹 +##橋 +##歌 +##止 +##正 +##武 +##比 +##氏 +##民 +##水 +##氵 +##氷 +##永 +##江 +##沢 +##河 +##治 +##法 +##海 +##清 +##漢 +##瀬 +##火 +##版 +##犬 +##王 +##生 +##田 +##男 +##疒 +##発 +##白 +##的 +##皇 +##目 +##相 +##省 +##真 +##石 +##示 +##社 +##神 +##福 +##禾 +##秀 +##秋 +##空 +##立 +##章 +##竹 +##糹 +##美 +##義 +##耳 +##良 +##艹 +##花 +##英 +##華 +##葉 +##藤 +##行 +##街 +##西 +##見 +##訁 +##語 +##谷 +##貝 +##貴 +##車 +##軍 +##辶 +##道 +##郎 +##郡 +##部 +##都 +##里 +##野 +##金 +##鈴 +##镇 +##長 +##門 +##間 +##阝 +##阿 +##陳 +##陽 +##雄 +##青 +##面 +##風 +##食 +##香 +##馬 +##高 +##龍 +##龸 +##fi +##fl +##！ +##（ +##） +##， +##－ +##． +##／ +##： +##？ +##～