Harmony integration

.github/workflows/unit_test.yml

@@ -21,7 +21,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.9", "3.12"] #NOTE: min and max Python versions supported by icepyx
+        python-version: ["3.10", "3.12"] #NOTE: min and max Python versions supported by icepyx
 
     steps:
       - uses: "actions/checkout@v4"
@@ -33,9 +33,12 @@ jobs:
           python-version: "${{ matrix.python-version }}"
 
       - name: "Run tests"
+        env:
+          EARTHDATA_PASSWORD: "${{ secrets.EARTHDATA_PASSWORD }}"
+          EARTHDATA_USERNAME: ${{ secrets.EARTHDATA_USERNAME }}
+          NSIDC_LOGIN: "${{ secrets.EARTHDATA_PASSWORD }}" # remove this
         run: |
-          pytest icepyx/ --verbose --cov app \
-            --ignore=icepyx/tests/integration
+          pytest icepyx/tests/unit --verbose --cov app
 
       - name: "Upload coverage report"
         uses: "codecov/[email protected]"

.pre-commit-config.yaml

@@ -26,14 +26,14 @@ repos:
           - tomli
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.7.2
+    rev: v0.8.1
     hooks:
       - id: ruff
         args: ["--fix", "--show-fixes"]
       - id: ruff-format
 
   - repo: https://github.com/abravalheri/validate-pyproject
-    rev: v0.22
+    rev: v0.23
     hooks:
       - id: validate-pyproject
 

HARMONY_MIGRATION_NOTES.md → doc/source/HARMONY_MIGRATION_NOTES.md

@@ -81,7 +81,7 @@ The type annotations will help with this process!
    Since we want the user to be able to provide short_name and version, implementing the
    concept ID as a `@cached_property` on `Query` which asks CMR for the concept ID makes
    sense to me.
-* Broken assumption: Harmony features are equivilent to NSIDC's ECS-based
+* Broken assumption: Harmony features are equivalent to NSIDC's ECS-based
   ordering system. As mentioned above, Harmony will not support variable
   subsetting, reprojection, or reformatting for IS2 collections on day 1. In the
   future, these features may be implemented in Harmony. For now, we need to
@@ -165,7 +165,7 @@ See this thread on EOSDIS Slack for more details:
 
 ## Remaining tasks
 
-Remaining tasks for "take2" development incldue:
+Remaining tasks for "take2" development include:
 
 * Implement support for full-granule orders via `earthaccess`
 * Check user inputs against supported harmony services. E.g., see `is2ref`

doc/source/example_notebooks/IS2_cloud_data_access.ipynb

@@ -131,7 +131,7 @@
    },
    "outputs": [],
    "source": [
-    "reg.order_vars.avail()"
+    "reg.order_vars"
    ]
   },
   {
@@ -384,7 +384,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.10"
+   "version": "3.11.11"
   }
  },
  "nbformat": 4,

doc/source/example_notebooks/IS2_data_access.ipynb

@@ -624,7 +624,7 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "icepyx",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
    "name": "python3"
   },
@@ -638,7 +638,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.10"
+   "version": "3.11.11"
   }
  },
  "nbformat": 4,

doc/source/example_notebooks/IS2_data_access2-subsetting.ipynb

@@ -105,7 +105,7 @@
    },
    "outputs": [],
    "source": [
-    "region_a.show_custom_options(dictview=True)"
+    "region_a.show_custom_options()"
    ]
   },
   {
@@ -153,73 +153,55 @@
   },
   {
    "cell_type": "markdown",
-   "metadata": {
-    "user_expressions": []
-   },
+   "metadata": {},
    "source": [
-    "### Determine what variables are available for your data product\n",
-    "There are multiple ways to get a complete list of available variables.\n",
-    "To increase readability, some display options (2 and 3, below) show the 200+ variable + path combinations as a dictionary where the keys are variable names and the values are the paths to that variable.\n",
+    "## _Why not just download all the data and subset locally? What if I need more variables/granules?_\n",
     "\n",
-    "1. `region_a.order_vars.avail`, a list of all valid path+variable strings\n",
-    "2. `region_a.show_custom_options(dictview=True)`, all available subsetting options\n",
-    "3. `region_a.order_vars.parse_var_list(region_a.order_vars.avail)`, a dictionary of variable:paths key:value pairs"
+    "_Taking advantage of the NSIDC subsetter is a great way to reduce your download size and thus your download time and the amount of storage required, especially if you're storing your data locally during analysis. By downloading your data using icepyx, it is easy to go back and get additional data with the same, similar, or different parameters (e.g. you can keep the same spatial and temporal bounds but change the variable list). Related tools (e.g. [`captoolkit`](https://github.com/fspaolo/captoolkit)) will let you easily merge files if you're uncomfortable merging them during read-in for processing._"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "tags": []
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "region_a.order_vars.avail()"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {
-    "user_expressions": []
-   },
-   "source": [
-    "By passing the boolean `options=True` to the `avail` method, you can obtain lists of unique possible variable inputs (var_list inputs) and path subdirectory inputs (keyword_list and beam_list inputs) for your data product. These can be helpful for building your wanted variable list."
+    "short_name = 'ATL06'\n",
+    "spatial_extent = './supporting_files/simple_test_poly.gpkg'\n",
+    "date_range = ['2019-10-01','2019-10-05']"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "tags": []
-   },
-   "outputs": [],
-   "source": [
-    "region_a.order_vars.avail(options=True)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
    "metadata": {},
+   "outputs": [],
    "source": [
-    "## _Why not just download all the data and subset locally? What if I need more variables/granules?_\n",
+    "region_a = ipx.Query(short_name, spatial_extent\n",
+    ", \n",
+    "   cycles=['03','04','05','06'], tracks=['0849','0902'])\n",
     "\n",
-    "_Taking advantage of the NSIDC subsetter is a great way to reduce your download size and thus your download time and the amount of storage required, especially if you're storing your data locally during analysis. By downloading your data using icepyx, it is easy to go back and get additional data with the same, similar, or different parameters (e.g. you can keep the same spatial and temporal bounds but change the variable list). Related tools (e.g. [`captoolkit`](https://github.com/fspaolo/captoolkit)) will let you easily merge files if you're uncomfortable merging them during read-in for processing._"
+    "print(region_a.product)\n",
+    "print(region_a.product_version)\n",
+    "print(region_a.cycles)\n",
+    "print(region_a.tracks)\n",
+    "print(region_a.spatial_extent)"
    ]
   },
   {
-   "cell_type": "markdown",
+   "cell_type": "code",
+   "execution_count": null,
    "metadata": {},
+   "outputs": [],
    "source": [
-    "### Building the default wanted variable list"
+    "region_a.visualize_spatial_extent()"
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
    "source": [
-    "region_a.order_vars.wanted"
+    "We can still print a list of available granules for our query"
    ]
   },
   {
@@ -228,8 +210,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "region_a.order_vars.append(defaults=True)\n",
-    "pprint(region_a.order_vars.wanted)"
+    "region_a.avail_granules(cloud=True)"
    ]
   },
   {
@@ -247,17 +228,15 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "region_a.subsetparams(Coverage=region_a.order_vars.wanted)"
+    "order = region_a.order_granules(subset=True) \n",
+    "order"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Or, you can put the `Coverage` parameter directly into `order_granules`:\n",
-    "`region_a.order_granules(Coverage=region_a.order_vars.wanted)`\n",
-    "\n",
-    "However, then you cannot view your subset parameters (`region_a.subsetparams`) prior to submitting your order."
+    "### Checking an order status"
    ]
   },
   {
@@ -266,8 +245,14 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "region_a.order_granules()# <-- you do not need to include the 'Coverage' kwarg to\n",
-    "                             # order if you have already included it in a call to subsetparams"
+    "order.status()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Downloading subsetted granules"
    ]
   },
   {
@@ -276,8 +261,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "region_a.download_granules('/home/jovyan/icepyx/dev-notebooks/vardata') # <-- you do not need to include the 'Coverage' kwarg to\n",
-    "                             # download if you have already submitted it with your order"
+    "files = order.download_granules(\"./data\")"
    ]
   },
   {
@@ -300,22 +284,13 @@
     "Compare the available variables associated with the full product relative to those in your downloaded data file."
    ]
   },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# put the full filepath to a data file here. You can get this in JupyterHub by navigating to the file,\n",
-    "# right clicking, and selecting copy path. Then you can paste the path in the quotes below.\n",
-    "fn = ''"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
     "## Check the downloaded data\n",
+    "\n",
+    "### Note: this needs to be updated\n",
     "Get all `latitude` variables in your downloaded file:"
    ]
   },
@@ -341,22 +316,6 @@
     "    if vn==varname: print(tvar) "
    ]
   },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Compare to the variable paths available in the original data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "region_a.order_vars.parse_var_list(region_a.order_vars.avail)[0][varname]"
-   ]
-  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -369,9 +328,9 @@
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "icepyx-dev",
+   "display_name": "Python 3 (ipykernel)",
    "language": "python",
-   "name": "icepyx-dev"
+   "name": "python3"
   },
   "language_info": {
    "codemirror_mode": {
@@ -383,7 +342,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.11.4"
+   "version": "3.11.11"
   }
  },
  "nbformat": 4,

icepyx/__init__.py

@@ -1,5 +1,7 @@
 from _icepyx_version import version as __version__
-from icepyx.core.query import GenQuery, Query
+
+from icepyx.core.base_query import BaseQuery, GenQuery
+from icepyx.core.query import Query
 from icepyx.core.read import Read
 from icepyx.core.variables import Variables
 from icepyx.quest.quest import Quest

icepyx/core/APIformatting.py

@@ -426,6 +426,9 @@ def build_params(self, **kwargs) -> None:
         else:
             self._check_valid_keys()
 
+        if "concept_id" in kwargs:
+            self._fmted_keys.update({"concept_id": kwargs["concept_id"]})
+
         if self.partype == "required":
             if not self._reqtype:
                 raise TypeGuardException