Update README.md

Author: aneubeck

crates/bpe/README.md

@@ -41,6 +41,16 @@ Note: the tik-token library uses a combination of 1) and 3) where sections are d
 
 ## Properties of BPE
 
+### Definition: Byte Pair Encoding
+
+The byte pair encoding is defined by an ordered list of tokens where the position in the list corresponds to the token id.
+Thereby each multi-byte token must have been constructed from exactly two previous tokens in the list.
+
+The encoding starts with all bytes being converted into their one-byte token ids.
+Then, one scans over the preliminary encoding and determines the smallest token id by which any pair of neighboring token ids could be replaced.
+The left most of them is replaced with that token id.
+The process continues until no further replacement is possible.
+
 ### Definition: Valid Encoding Sequence
 
 An encoding sequence `e_0..e_n` is said to be valid if decoding the sequence and reencoding it with BPE produces the very same sequence.