Updated docs

Author: hugegreenbug

_images/BackendClassHierarchy.png

@@ -0,0 +1,21 @@
+Backends
+========================
+
+Backends have front-end and back-end functions. Backends connect users to DSI Core middleware (front-end), and Backends allow DSI Middleware data structures to read and write to persistent external storage (back-end). Backends are modular to support user contribution. Backend contributors are encouraged to offer custom Backend abstract classes and Backend implementations. A contributed Backend abstract class may extend another Backend to inherit the properties of the parent. In order to be compatible with DSI Core middleware, Backends should create an interface to Python built-in data structures or data structures from the Python ``collections`` library. Backend extensions will be accepted conditional to the extention of ``backends/tests`` to demonstrate new Backend capability. We can not accept pull requests that are not tested.
+
+
+.. image:: BackendClassHierarchy.png
+
+.. automodule:: dsi.backends.filesystem
+   :members:
+
+.. automodule:: dsi.backends.sqlite
+   :members:
+
+.. automodule:: dsi.backends.gufi
+   :members:
+
+.. automodule:: dsi.backends.parquet
+   :members:
+
+

_images/DriverClassHierarchy.png

@@ -2,27 +2,27 @@ Core
 ===================
 The DSI Core middleware defines the Terminal concept. An instantiated Terminal is the human/machine DSI interface. The person setting up a Core Terminal only needs to know how they want to ask questions, and what metadata they want to ask questions about. If they don’t see an option to ask questions the way they like, or they don’t see the metadata they want to ask questions about, then they should ask a Driver Contributor or a Plugin Contributor, respectively.
 
-A Core Terminal is a home for Plugins, and an interface for Drivers. A Core Terminal is instantiated with a set of default Plugins and Drivers, but they must be loaded before a user query is attempted. Here's an example of how you might work with DSI using an interactive Python interpreter for your data science workflows::
+A Core Terminal is a home for Plugins (Readers/Writers), and an interface for Backends. A Core Terminal is instantiated with a set of default Plugins and Backends, but they must be loaded before a user query is attempted. Here's an example of how you might work with DSI using an interactive Python interpreter for your data science workflows::
 
 >>> from dsi.core import Terminal
 >>> a=Terminal()
 >>> a.list_available_modules('plugin')
 >>> # ['Bueno', 'Hostname', 'SystemKernel']
->>> a.load_module('plugin','Bueno','consumer',filename='./data/bueno.data')
->>> # Bueno plugin consumer loaded successfully.
->>> a.load_module('plugin','Hostname','producer')
->>> # Hostname plugin producer loaded successfully.
+>>> a.load_module('plugin','Bueno','reader',filename='./data/bueno.data')
+>>> # Bueno plugin reader loaded successfully.
+>>> a.load_module('plugin','Hostname','writer')
+>>> # Hostname plugin writer loaded successfully.
 >>> a.list_loaded_modules()
->>> # {'producer': [<dsi.plugins.env.Hostname object at 0x7f21232474d0>], 
->>> #  'consumer': [<dsi.plugins.env.Bueno object at 0x7f2123247410>], 
+>>> # {'writer': [<dsi.plugins.env.Hostname object at 0x7f21232474d0>], 
+>>> #  'reader': [<dsi.plugins.env.Bueno object at 0x7f2123247410>], 
 >>> #  'front-end': [], 
 >>> #   'back-end': []}
 
 
-At this point, you might decide that you are ready to collect data for inspection. It is possible to utilize DSI Drivers to load additional metadata to supplement your Plugin metadata, but you can also sample Plugin data and search it directly. 
+At this point, you might decide that you are ready to collect data for inspection. It is possible to utilize DSI Backends to load additional metadata to supplement your Plugin metadata, but you can also sample Plugin data and search it directly. 
 
 
-The process of transforming a set of Plugin producers and consumers into a querable format is called transloading. A DSI Core Terminal has a ``transload()`` method which may be called to execute all Plugins at once::
+The process of transforming a set of Plugin writers and readers into a querable format is called transloading. A DSI Core Terminal has a ``transload()`` method which may be called to execute all Plugins at once::
 
 >>> a.transload()
 >>> a.active_metadata
@@ -36,32 +36,32 @@ Once a Core Terminal has been transloaded, no further Plugins may be added. Howe
 >>> a.active_metadata
 >>> # OrderedDict([('uid', [1000, 1000, 1000, 1000]), ('effective_gid', [1000, 1000, 1000...
 
-If you perform data science tasks using Python, it is not necessary to create a DSI Core Terminal front-end because the data is already in a Python data structure. If your data science tasks can be completed in one session, it is not required to interact with DSI Drivers. However, if you do want to save your work, you can load a DSI Driver with a back-end function::
+If you perform data science tasks using Python, it is not necessary to create a DSI Core Terminal front-end because the data is already in a Python data structure. If your data science tasks can be completed in one session, it is not required to interact with DSI Backends. However, if you do want to save your work, you can load a DSI Backend with a back-end function::
 
->>> a.list_available_modules('driver')
+>>> a.list_available_modules('backend')
 >>> # ['Gufi', 'Sqlite', 'Parquet']
->>> a.load_module('driver','Parquet','back-end',filename='parquet.data')
->>> # Parquet driver back-end loaded successfully.
+>>> a.load_module('backend','Parquet','back-end',filename='parquet.data')
+>>> # Parquet backend loaded successfully.
 >>> a.list_loaded_modules()
->>> # {'producer': [<dsi.plugins.env.Hostname object at 0x7f21232474d0>], 
->>> #  'consumer': [<dsi.plugins.env.Bueno object at 0x7f2123247410>], 
+>>> # {'writer': [<dsi.plugins.env.Hostname object at 0x7f21232474d0>], 
+>>> #  'reader': [<dsi.plugins.env.Bueno object at 0x7f2123247410>], 
 >>> #  'front-end': [], 
->>> #   'back-end': [<dsi.drivers.parquet.Parquet object at 0x7f212325a110>]}
+>>> #   'back-end': [<dsi.backends.parquet.Parquet object at 0x7f212325a110>]}
 >>> a.artifact_handler(interaction_type='put')
 
 The contents of the active DSI Core Terminal metadata storage will be saved to a Parquet object at the path you provided at module loading time.
 
 It is possible that you prefer to perform data science tasks using a higher level abstraction than Python itself. This is the purpose of the DSI Driver front-end functionality. Unlike Plugins, Drivers can be added after the initial ``transload()`` operation has been performed::
 
->>> a.load_module('driver','Parquet','front-end',filename='parquet.data')
->>> # Parquet driver front-end loaded successfully.
+>>> a.load_module('backend','Parquet','front-end',filename='parquet.data')
+>>> # Parquet backend front-end loaded successfully.
 >>> a.list_loaded_modules()
->>> # {'producer': [<dsi.plugins.env.Hostname object at 0x7fce3c612b50>], 
->>> # 'consumer': [<dsi.plugins.env.Bueno object at 0x7fce3c622110>], 
->>> # 'front-end': [<dsi.drivers.parquet.Parquet object at 0x7fce3c622290>], 
->>> # 'back-end': [<dsi.drivers.parquet.Parquet object at 0x7fce3c622650>]}
+>>> # {'writer': [<dsi.plugins.env.Hostname object at 0x7fce3c612b50>], 
+>>> # 'reader': [<dsi.plugins.env.Bueno object at 0x7fce3c622110>], 
+>>> # 'front-end': [<dsi.backends.parquet.Parquet object at 0x7fce3c622290>], 
+>>> # 'back-end': [<dsi.backends.parquet.Parquet object at 0x7fce3c622650>]}
 
-Any front-end may be used, but in this case the Parquet driver has a front-end implementation which builds a jupyter notebook from scratch that loads your metadata collection into a Pandas Dataframe. The Parquet front-end will then launch the Jupyter Notebook to support an interactive data science workflow::
+Any front-end may be used, but in this case the Parquet backend has a front-end implementation which builds a jupyter notebook from scratch that loads your metadata collection into a Pandas Dataframe. The Parquet front-end will then launch the Jupyter Notebook to support an interactive data science workflow::
 
 >>> a.artifact_handler(interaction_type='inspect')
 >>> # Writing Jupyter notebook...
@@ -74,13 +74,13 @@ You can then close your Jupyter notebook, ``transload()`` additionally to increa
 
 Although this demonstration only used one Plugin per Plugin functionality, any number of plugins can be added to collect an arbitrary amount of queriable metadata::
 
->>> a.load_module('plugin','SystemKernel','producer')
->>> # SystemKernel plugin producer loaded successfully
+>>> a.load_module('plugin','SystemKernel','writer')
+>>> # SystemKernel plugin writer loaded successfully
 >>> a.list_loaded_modules()
->>> # {'producer': [<dsi.plugins.env.Hostname object at 0x7fce3c612b50>, <dsi.plugins.env.SystemKernel object at 0x7fce68519250>], 
->>> # 'consumer': [<dsi.plugins.env.Bueno object at 0x7fce3c622110>], 
->>> # 'front-end': [<dsi.drivers.parquet.Parquet object at 0x7fce3c622290>], 
->>> # 'back-end': [<dsi.drivers.parquet.Parquet object at 0x7fce3c622650>]}
+>>> # {'writer': [<dsi.plugins.env.Hostname object at 0x7fce3c612b50>, <dsi.plugins.env.SystemKernel object at 0x7fce68519250>], 
+>>> # 'reader': [<dsi.plugins.env.Bueno object at 0x7fce3c622110>], 
+>>> # 'front-end': [<dsi.backends.parquet.Parquet object at 0x7fce3c622290>], 
+>>> # 'back-end': [<dsi.backends.parquet.Parquet object at 0x7fce3c622650>]}
 
 .. automodule:: dsi.core
       :members:

_images/PluginClassHierarchy.png

@@ -15,8 +15,8 @@ The Data Science Infrastructure Project (DSI)
    installation
    core
    plugins
-   drivers
-   permissions
+   backends
+
 
 Indices and tables
 ==================

_sources/backends.rst.txt

@@ -19,21 +19,21 @@ The DSI system is composed of three fundamental parts:
 
 DSI Core Middleware
 -------------------
-DSI's core middleware is focused on delivering user-queries on unified metadata which are distributed across many files and security domains. DSI currently supports Linux, and is tested on RedHat- and Debian-based distributions. The DSI Core middleware is a home for DSI Plugins and an interface for DSI Drivers.
+DSI's core middleware is focused on delivering user-queries on unified metadata which are distributed across many files and security domains. DSI currently supports Linux, and is tested on RedHat- and Debian-based distributions. The DSI Core middleware is a home for DSI Plugins and an interface for DSI Backends.
 
 Plugin Abstract Classes
 -----------------------
 Plugins transform an arbitrary data source into a format that is compatible with our middleware. We call the parsed and queriable attributes "metadata" (data about the data). Metadata share the same security profile as the source data.
 
-Plugins can operate as data consumers or data producers. A simple data consumer might parse an application's output file and place it into a middleware compatible data structure: Python built-ins and members of the popular Python ``collection`` module. A simple data producer might execute an application to supplement existing data and queriable metadata.
+Plugins can operate as data readers or data writers. A simple data reader might parse an application's output file and place it into a middleware compatible data structure: Python built-ins and members of the popular Python ``collection`` module. A simple data writer might execute an application to supplement existing data and queriable metadata.
 
 Plugins are defined by a base abstract class, and support child abstract classes which inherit the properties of their ancestors.
 
 .. image:: PluginClassHierarchy.png
 
-Driver Abstract Classes
------------------------
-Drivers are an interface between the User and the Core, or an interface between the Core and a storage medium. Drivers can operate as Front-ends or Back-ends, and a Driver contributor can choose to implement one or both. Driver front-ends are built to deliver an experience which is compatible with a User Story. A simple supporting User Story is a need to query metadata by SQL query. Because the set of queriable metadata are spread across filesystems and security domains, a supporting Driver Back-end is required to assemble query results and present them to the DSI core middleware for transformation and return, creating an experience which is compatible with the User Story.
+Backend Abstract Classes
+------------------------
+Backends are an interface between the User and the Core, or an interface between the Core and a storage medium. Backends can operate as Front-ends or Back-ends, and a Backend contributor can choose to implement one or both. Backend front-ends are built to deliver an experience which is compatible with a User Story. A simple supporting User Story is a need to query metadata by SQL query. Because the set of queriable metadata are spread across filesystems and security domains, a supporting Backend Back-end is required to assemble query results and present them to the DSI core middleware for transformation and return, creating an experience which is compatible with the User Story.
 
 .. image:: user_story.png
    :scale: 50%

_sources/core.rst.txt

@@ -1,9 +1,12 @@
 Plugins
 ===================
-Plugins connect data-producing applications to DSI middleware. Plugins have "producer" or "consumer" functions. A Plugin consumer function deals with existing data files or input streams. A Plugin producer deals with generating new data. Plugins are modular to support user contribution. Plugin contributors are encouraged to offer custom Plugin abstract classes and Plugin implementations. A contributed Plugin abstract class may extend another plugin to inherit the properties of the parent. In order to be compatible with DSI middleware, Plugins should produce data in Python built-in data structures or data structures sourced from the Python ``collections`` library. Plugin extensions will be accepted conditional to the extention of ``plugins/tests`` to demonstrate the new Plugin capability. We can not accept pull requests that are not tested.
+Plugins connect data-producing applications to DSI middleware. Plugins have "writers" or "readers" functions. A Plugin reader function deals with existing data files or input streams. A Plugin writer deals with generating new data. Plugins are modular to support user contribution. Plugin contributors are encouraged to offer custom Plugin abstract classes and Plugin implementations. A contributed Plugin abstract class may extend another plugin to inherit the properties of the parent. In order to be compatible with DSI middleware, Plugins should produce data in Python built-in data structures or data structures sourced from the Python ``collections`` library. Plugin extensions will be accepted conditional to the extention of ``plugins/tests`` to demonstrate the new Plugin capability. We can not accept pull requests that are not tested.
 
 .. image:: PluginClassHierarchy.png
 
+.. automodule:: dsi.plugins.plugin
+   :members:
+
 .. automodule:: dsi.plugins.metadata
    :members:
 

_sources/drivers.rst.txt

@@ -1,9 +1,20 @@
-/* Compatability shim for jQuery and underscores.js.
+/*
+ * _sphinx_javascript_frameworks_compat.js
+ * ~~~~~~~~~~
+ *
+ * Compatability shim for jQuery and underscores.js.
+ *
+ * WILL BE REMOVED IN Sphinx 6.0
+ * xref RemovedInSphinx60Warning
  *
- * Copyright Sphinx contributors
- * Released under the two clause BSD licence
  */
 
+/**
+ * select a different prefix for underscore
+ */
+$u = _.noConflict();
+
+
 /**
  * small helper function to urldecode strings
  *