Skip to content

Commit f23e8a1

Browse files
committed
Update to v0.9.6
1 parent 54790cf commit f23e8a1

File tree

150 files changed

+13570
-14321
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

150 files changed

+13570
-14321
lines changed

atomate/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
__version__ = "0.9.5"
1+
__version__ = "0.9.6"

docs/_sources/advanced_stores.rst.txt

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
2+
.. title:: Advanced Storage Stratagies
3+
.. _advanced_storage:
4+
5+
==================
6+
Advanced Storage Stratagies
7+
==================
8+
9+
Storing data greater than 16 Mb
10+
============
11+
12+
Introduction
13+
------------
14+
15+
As the analysis capabilities of the high-throughput computational material science grows, more and more data is required to perform the analysis.
16+
While the atomic structure and metadata about the calculations can be stored within the 16 Mb limit of MongoDB's data framework the storage requirement for field data like the charge density is often orders of magnitude larger than that limit.
17+
As such, alternative methods are required to handle the storage of specific fields in the parsed task document.
18+
For these fields, with larger storage requirements, the data itself will be removed from the task document and replaced with and ``fs_id`` to reference where the data is stored.
19+
This results in a task document with a much smaller size, which will still be uploaded onto the MongoDB specified in your ``DB_FILE``
20+
21+
Two approaches are currently available in atomate to handle the storage of large data chunk.
22+
The user can implement a GridFS-based chunking and storage procedure as we have done in ``VaspCalcDb``.
23+
But the recommended method for large object storage is to use ``maggma`` stores implemented in the ``CalcDb`` class.
24+
Currently, only the Amazon S3 store is implemented.
25+
26+
Please read the documentation for ``maggma`` for more details _maggma: https://materialsproject.github.io/maggma
27+
28+
Configuration
29+
------------
30+
31+
To storing the larger items to an AWS S3 bucket via the ``maggma`` API, the user needs to have the ``maggma_store`` keyword in present in the ``db.json`` file used by atomate.
32+
33+
.. code-block:: json
34+
35+
{
36+
"host": "<<HOSTNAME>>",
37+
"port": <<PORT>>,
38+
"database": "<<DB_NAME>>",
39+
"collection": "tasks",
40+
"admin_user": "<<ADMIN_USERNAME>>",
41+
"admin_password": "<<ADMIN_PASSWORD>>",
42+
"readonly_user": "<<READ_ONLY_PASSWORD>>",
43+
"readonly_password": "<<READ_ONLY_PASSWORD>>",
44+
"aliases": {}
45+
"maggma_store": {
46+
"bucket" : "<<BUCKET_NAME>>",
47+
"s3_profile" : "<<S3_PROFILE_NAME>>",
48+
"compress" : true,
49+
"endpoint_url" : "<<S3_URL>>"
50+
}
51+
}
52+
53+
Where ``<<BUCKET_NAME>>`` is S3 bucket where the data will be stored, ``<<S3_PROFILE_NAME>>`` is the name of the S3 profile from the ``$HOME/.aws`` folder.
54+
Note, this AWS profile needs to be available anywhere the ``VaspCalcDb.insert_task`` is called (i.e. on the computing resource where the database upload of the tasks takes place).
55+
56+
Usage
57+
-----------
58+
59+
Example: store the charge density
60+
61+
To parse a completed calculation directory. We need to instantiate the ``drone`` with the ``parse_aeccar`` or ``parse_chgcar`` flag.
62+
63+
.. code-block:: python
64+
65+
calc_dir = "<<DIR>>/launcher_2019-09-03-11-46-20-683785"
66+
drone = VaspDrone(parse_chgcar=True, parse_aeccar=True)
67+
doc = drone.assimilate(calc_dir)
68+
task_id = mmdb.insert_task(doc)
69+
70+
Some workflows like the ``StaticWF`` will pass the parsing flags like ``parse_chgcar`` to the drone directly.
71+
72+
To access the data using the task_id we can call
73+
74+
.. code-block:: python
75+
76+
chgcar = mmdb.get_chgcar(task_id)
77+
78+
Similar functionalities exist for the band structure and DOS.
79+
Please refer to the documentation of ``VaspCalcDb`` for more details.

docs/_sources/atomate.common.firetasks.rst.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ Subpackages
55
-----------
66

77
.. toctree::
8+
:maxdepth: 4
89

910
atomate.common.firetasks.tests
1011

@@ -35,7 +36,6 @@ atomate.common.firetasks.run\_calc module
3536
:undoc-members:
3637
:show-inheritance:
3738

38-
3939
Module contents
4040
---------------
4141

docs/_sources/atomate.common.firetasks.tests.rst.txt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@ atomate.common.firetasks.tests.test\_parse\_outputs module
2020
:undoc-members:
2121
:show-inheritance:
2222

23-
2423
Module contents
2524
---------------
2625

docs/_sources/atomate.common.rst.txt

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,21 @@ Subpackages
55
-----------
66

77
.. toctree::
8+
:maxdepth: 4
89

910
atomate.common.firetasks
11+
atomate.common.tests
12+
13+
Submodules
14+
----------
15+
16+
atomate.common.powerups module
17+
------------------------------
18+
19+
.. automodule:: atomate.common.powerups
20+
:members:
21+
:undoc-members:
22+
:show-inheritance:
1023

1124
Module contents
1225
---------------
Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
atomate.common.tests package
2+
============================
3+
4+
Submodules
5+
----------
6+
7+
atomate.common.tests.test\_powerups module
8+
------------------------------------------
9+
10+
.. automodule:: atomate.common.tests.test_powerups
11+
:members:
12+
:undoc-members:
13+
:show-inheritance:
14+
15+
Module contents
16+
---------------
17+
18+
.. automodule:: atomate.common.tests
19+
:members:
20+
:undoc-members:
21+
:show-inheritance:

docs/_sources/atomate.feff.firetasks.rst.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ Subpackages
55
-----------
66

77
.. toctree::
8+
:maxdepth: 4
89

910
atomate.feff.firetasks.tests
1011

@@ -43,7 +44,6 @@ atomate.feff.firetasks.write\_inputs module
4344
:undoc-members:
4445
:show-inheritance:
4546

46-
4747
Module contents
4848
---------------
4949

docs/_sources/atomate.feff.firetasks.tests.rst.txt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@ atomate.feff.firetasks.tests.test\_tasks module
1212
:undoc-members:
1313
:show-inheritance:
1414

15-
1615
Module contents
1716
---------------
1817

docs/_sources/atomate.feff.fireworks.rst.txt

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ Subpackages
55
-----------
66

77
.. toctree::
8+
:maxdepth: 4
89

910
atomate.feff.fireworks.tests
1011

@@ -19,7 +20,6 @@ atomate.feff.fireworks.core module
1920
:undoc-members:
2021
:show-inheritance:
2122

22-
2323
Module contents
2424
---------------
2525

docs/_sources/atomate.feff.fireworks.tests.rst.txt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,6 @@ atomate.feff.fireworks.tests.test\_fireworks module
1212
:undoc-members:
1313
:show-inheritance:
1414

15-
1615
Module contents
1716
---------------
1817

0 commit comments

Comments
 (0)