Significantly sped up filtering and minor bug fixes

MANIFEST.in

@@ -1,4 +1,4 @@
 include README.md
 include LICENSE
 include docs/*
-include tests/*
+include test/*

docs/source/api.rst

@@ -106,6 +106,18 @@ By default (for DLGs), Ringtail will store the best-scored (lowest energy) bindi
     rtc.add_results_from_files( file_path = "path2"
                                 max_poses = 5)
 
+Iteratively appending to a database
+------------------------------------
+When results are added to the database, there is a final step where some tables are indexed, and some database properties saved. If you are adding data iteratively through e.g., a for-loop and adding some number at files at once, it is time-consuming (and not necessary) to do this every iteration. Instead, you can invoke the keyword ``finalize=False``, and run the finalization method separately at the end:
+
+.. code-block:: python
+
+    for folder in enumerate("path_with_many_folders"):
+        rtc.add_results_from_files( file_path = folder,
+                                    finalize = False)
+
+    rtc.finalize_write()
+
 Filtering
 **********
 
@@ -157,16 +169,17 @@ The ``max_miss`` keywords allows the user to filter by given interactions exclud
 
 Ligand filters 
 ===============
-Several filters pertaining to the SMARTS structure of the ligand can be used. For example, the ``ligand_substruct_pos`` keyword may be used to filter for a specific ligand substructure (specified with a SMARTS string) to be placed within some distance of a given cartesian coordinate. The format for this option is ``"<SMARTS pattern: str>" <index of atom in SMARTS: int> <cutoff distance: float> <target x coord: float> <target y coord: float> <target z coord: float>``.
-ligand_name: Specify ligand name(s). Will combine name filters with 'OR'.
-ligand_substruct: SMARTS pattern(s) for substructure matching.
-ligand_substruct_pos: SMARTS pattern(s) for substructure matching, e.g., ['[Oh]C', 0, 1.2, -5.5, 10.0, 15.5] -> [smart_string, index_of_positioned_atom, cutoff_distance, x, y, z].
-ligand_max_atoms: Maximum number of heavy atoms a ligand may have.
-ligand_operator: Logical join operator for multiple SMARTS.
+Several filters pertaining to the SMARTS structure of the ligand can be used. For example, ligands can be filtered for presence of certain substrctures specified by their SMARTS string using ``ligand_substruct``, as well as their ligand name contaning a specific phrase ``ligand_name``. The ligand name search will include any ligand names that contain the specified phrase, and does not look for exact matches only. Use the keyword ``ligand_operator`` to determine if the ligand filters should be evaluated as this ``OR`` that (default), or combined with ``AND``. ``ligand_max_atoms`` can be used to specify maximum number of heavy atoms a ligand may have.
+
+.. code-block:: python
+
+    rtc.filter(ligand_substruct=["[Oh]C", "C=O"], ligand_name="cool_ligand",ligand_operator="AND", ligand_max_atoms=5)
+
+The ``ligand_substruct_pos`` option may be used to filter for a specific ligand substructure to be placed within some distance of a given cartesian coordinate. The format for this option using the API is as a list of the six elements: ``["<SMARTS pattern: str>"," <index of atom in SMARTS: int>, <cutoff distance: float>, <target x coord: float>, <target y coord: float>, <target z coord: float>]``. If seachring for more than one ``ligand_substruct_pos`` make the value a list of lists. 
 
 .. code-block:: python
 
-    rtc.filter(ligand_substruct=["[Oh]C"], ligand_substruct_pos=["[Oh]C", 0, 1.2, -5.5, 10.0, 15.5])
+    rtc.filter(ligand_name="_1", ligand_substruct_pos=[["C=O", 1, 10, 102, 106, 154], ['[C][Oh]', 1, 10, 102, 106, 154]])
 
 
 Clustering

docs/source/changes.rst

@@ -3,7 +3,18 @@
 Changes in Ringtail
 ######################
 
-Changes in 2.0.0: fully developed API
+Changes in 2.1.0: enhanced filtering speed
+******************************************
+Enhancements to the code base
+==============================
+* The format of the queries produced to filter the database have been completely rewritten, reducing filtering time by at least a factor of 10 compared to 1.1.0. Extra indices were added to three of the tables to support the faster filtering speeds. 
+
+Bug fixes
+===========
+* The use of the keywords `--ligand_name`, `--ligand_substruct`, and `--ligand_substruct_pos` had ambiguous behavior where if they were invoked more than once, only the last filter value would be used (as opposed to concatenating the values). They now will work by supplying multiple values to one keyword, as well as one or more values to two or more keywords. Further, `ligand_substruct_pos` now takes input as one string (`"[C][Oh] 1 1.5 -20 42 -7.1"`)as opposed to one string and five numbers (`"[C][Oh]"" 1 1.5 -20 42 -7.1`).
+* `--ligand_max_atoms` counted all atoms in the ligand, including hydrogens. With bug fix it counts only heavy atoms(not hydrogens). 
+
+Changes in 2.x: fully developed API
 ***************************************
 
 Changes in keywords used for the command line tool
@@ -24,18 +35,19 @@ Enhancements to the codebase
 ==============================
 * Fully developed API can use python for scripting exclusively (see :ref:`API <api>` page for full description)
 * Can add docking results directly without using file system (for vina only as output comes as a string). 
-* The Ringtail log is now written to a logging file in addition to STDOUT
+* The Ringtail log is now written to a logging file in addition to STDOUT if log level is det to "DEBUG". 
 
 Changes to code behavior
 =========================
 * Interaction tables: one new table has been added (`Interactions`) which references the interaction id from `Interaction_indices`, while the table `Interaction_bitvectors` has been discontinued.
-* A new method to update an existing database 1.1.0 (or 1.0.0) to 2.0.0 is included. However, if the existing database was created with the duplicate handling option, there is a chance of inconsistent behavior of anything involving interactions as the Pose_ID was not used as an explicit foreign key in db v1.0.0 and v1.1.0 (see Bug fixes below).
+* A new method to update an existing database 1.1.0 (or 1.0.0) to 2.0 is included. However, if the existing database was created with the duplicate handling option, there is a chance of inconsistent behavior of anything involving interactions as the Pose_ID was not used as an explicit foreign key in db v1.0.0 and v1.1.0 (see Bug fixes below).
 
 Bug fixes
 ===========
-* The option `duplicate_handling` could previously only be applied during database creation and produced inconsistent table behavior. Option can now be applied at any time results are added to a database, and will create internally consistent tables. **Please note: if you have created tables in the past and invoking the keyword `duplicate_handling` you may have errors in the "Interaction_bitvectors" table (<2.0.0). These errors cannot be recovered, and we recommend you re-make the database with Ringtail 2.0.0.**
+* The option `duplicate_handling` could previously only be applied during database creation and produced inconsistent table behavior. Option can now be applied at any time results are added to a database, and will create internally consistent tables. **Please note: if you have created tables in the past and invoking the keyword `duplicate_handling` you may have errors in the "Interaction_bitvectors" table (<2.0). These errors cannot be recovered, and we recommend you re-make the database with Ringtail 2.0.**
 * Writing SDFs from filtering bookmarks: will check that bookmark exists and has data before writing, and will now produce SDFs for any bookmarks existing bookmarks. If the bookmark results from a filtering where `max_miss` &lt; 0 it will note if the non-union bookmark is used, and if the base name for such bookmarks is provided it will default to the `basename_union` bookmark for writing the SDFs.
 * Output from filtering using `max_miss` and `output_all_poses=False`(default) now producing expected behavior of outputting only one pose per ligand. Filtering for interactions `max_miss` allows any given pose for a ligand to miss `max_miss` interactions and still be considered to pass the filter. Previously, in the resulting `union` bookmark and `output_log` text file some ligands would present with more than one pose, although the option to `output_all_poses` was `False` (and thus the expectation would be one pose outputted per ligand). This would give the wrong count for how many ligands passed a filter, as some were counted more than once. 
+* The use of the keywords `--ligand_name`, `--ligand_substruct`, and `--ligand_substruct_pos` had ambiguous behavior where if they were invoked more than once, only the last filter value would be used (as opposed to concatenating the values). They now will work by supplying multiple values to one keyword, as well as one or more values to two or more keywords. Further, `ligand_substruct_pos` now takes input as one string (`"[C][Oh] 1 1.5 -20 42 -7.1"`)as opposed to one string and five numbers (`"[C][Oh]"" 1 1.5 -20 42 -7.1`).
 
 Changes in 1.1.0: enhanced database performance
 ***********************************************