3.1.0 release merge Develop -> Main

README.md

@@ -1,8 +1,8 @@
 # Metacat: Data Preservation and Discovery System
 
-Version: 3.0.0 Release
+Version: 3.1.0 Release
 - See [Release Notes](RELEASE-NOTES.md)
-- DOI: [doi:10.18739/A2VX0650Z](https://doi.org/10.18739/A2VX0650Z)
+- DOI: [doi:10.18739/A2HT2GD6X](https://doi.org/10.18739/A2HT2GD6X)
 
 Send feedback and bugs to: 
 
@@ -43,7 +43,7 @@ for the next release.
 
 Cite this software as:
 
-> Jing Tao, Matthew Brooke, Dou Mok, Matthew B. Jones. 2024. Metacat: Data Preservation and Discovery System (3.0.0). Arctic Data Center. Software. [doi:10.18739/A2VX0650Z](https://doi.org/10.18739/A2VX0650Z)
+> Jing Tao, Matthew Brooke, Dou Mok, Matthew B. Jones. 2024. Metacat: Data Preservation and Discovery System (3.1.0). Arctic Data Center. Software. [doi:10.18739/A2HT2GD6X](https://doi.org/10.18739/A2HT2GD6X)
 
 ## Current Contributors
 

RELEASE-NOTES.md

@@ -1,7 +1,73 @@
 # Metacat Release Notes
 
+
+## Release Notes for Metacat 3.1.0
+
+**Release date: 2024-12-16**
+
+> **IMPORTANT** If you are upgrading from version 2.19, please see the [Upgrade
+> Notes](#upgrade-notes-2190-to-300-or-above) below.
+
+The Metacat version 3.1.0 release introduces significant performance enhancements for managing large
+datasets. This was achieved by integrating with the new Dataone hash-based storage library,
+[`hashstore-java`](https://github.com/DataONEorg/hashstore-java), which necessitated substantial
+changes to disk-based file-storage methods.
+
+### New Features & Enhancements:
+* Metacat uses the DataONE [`hashstore-java`](https://github.com/DataONEorg/hashstore-java)
+  hash-based storage library.
+* Existing stored files are seamlessly migrated to the new storage system.
+* **Setup no longer requires an auth token in order to index private datasets**, thus simplifying
+  installation.
+* Upgraded datacite version to 4.5 from 4.3 for EZID DOI service ([Issue
+  #2005](https://github.com/NCEAS/metacat/issues/2005))
+* Docker changes:
+  * Updated base image to `tomcat:9.0.96-jre17-temurin-jammy` (from `9.0.82`)
+  * Run container as tomcat user & group, with uid 59997 and gid 59997
+
+### Upgrades and Bug Fixes:
+* Bump com.rabbitmq:amqp-client from 5.14.2 to 5.21.0
+  ([Issue #1922](https://github.com/NCEAS/metacat/issues/1922))
+* Add indexes to the access_log table ([Issue #1965](https://github.com/NCEAS/metacat/issues/1965))
+* Optimize the query to get log record ([Issue #1976](https://github.com/NCEAS/metacat/issues/1976))
+
+## Release Notes for helm chart 2.1.0
+
+**Release date: 2024-12-16**
+
+### New Features & Enhancements:
+* Increase Application version to 3.1.0
+* Increase MetacatUI subchart version to 1.0.1 (MetacatUI app version 2.31.0)
+* Increase dataone-indexer subchart version to 3.1.0
+  * Setup no longer requires an auth token in order to index private datasets
+* DataONE Member Node setup simplified, and no longer requires admin access to the nginx controller;
+  all [X-Proxy-Header setup now takes place in the current release
+  namespace](https://github.com/NCEAS/metacat/pull/2027/files)
+* Helm chart README updated with new setup for v3.1.0, and new [checklist added, for Metacat K8s
+  Installation and Migration/Upgrade
+  Steps](https://github.com/NCEAS/metacat/blob/main/helm/admin/MetacatQuickRef.md)
+* Example PV and PVC creation files added in the [`helm/admin` directory](./helm/admin).
+* Changes to UID & GID defaults:
+  * Metacat `podSecurityContext` defaults updated to `runAsUser: 59997`,
+    `runAsGroup: 59997` and `fsGroup: 59997`.
+  * Dataone-indexer subchart `podSecurityContext` defaults updated to `fsGroup: 59997`.
+  * Postgresql subchart `containerSecurityContext` defaults updated to `runAsUser: 59996`, and
+    `podSecurityContext` defaults updated to `runAsNonRoot: true`
+  * (NOTE you will need to chmod any existing `metacat` directory to `59997:59997`
+    and chmod any existing `postgresql` directory to `59996:59996`)
+* RabbitMQ subchart `consumer_timeout` set to `144000000`, to prevent the indexer from timing out
+  when indexing large datasets
+* Solr and Zookeeper subchart default Persistent Volume size increased to 100Gi.
+
+
 ## Release Notes for Helm Chart 1.1.1
 
+Setting up a Token and Optional CA certificate for Indexer Access
+
+**IMPORTANT:** In order for Metacat 3.0.0 to function correctly, the
+dataone-indexer needs a valid authentication token, to enable
+indexing for private datasets, via calls to metacat's DataONE API.
+
 Release date: 2024-07-25
 
 > Note the helm chart is a beta feature. It has been tested, and we believe it to be working well,
@@ -75,10 +141,10 @@ Also in this release:
   we recommend caution with this early release. If you try it, [we'd love to hear your
   feedback](https://www.dataone.org/contact/)!
 
-### Upgrade Notes (2.19.0 to 3.0.0):
+### Upgrade Notes (2.19.0 to 3.0.0 or above):
 
 - Starting Requirements:
-  - Your existing Metacat installation must already have been successfully upgraded to [version v2.19.0](https://github.com/NCEAS/metacat/releases/tag/2.19.0) before you can begin upgrading to version 3.0.0.
+  - Your existing Metacat installation must already have been successfully upgraded to [version v2.19.0](https://github.com/NCEAS/metacat/releases/tag/2.19.0) before you can begin upgrading to version 3.0.0 or above.
     - If not, please upgrade to v2.19.0 first, before proceeding.
   - You must have Java 17 installed
     - If it is not installed, please install it and set it as the default version
@@ -93,15 +159,11 @@ Also in this release:
     - Stop Tomcat - ex. `sudo systemctl stop tomcat9`
     - Stop solr - ex. `sudo systemctl stop solr`
 - Download/upgrade your solr version to 9.5.0
-  - Solr upgrade is not supported for 3.0.0 with old cores, you must start with a new core (new solr-home)
-    - Please use a new Solr home and reindex all objects due to incompatibility between old data and the new Solr schema.
-    - Example below to reindex all objects
-      ```sh
-      # curl -X PUT -H "Authorization: Bearer $TOKEN" https://<your-host>/<your-context>/d1/mn/v2/index?all=true
-      # where $TOKEN is an environment variable containing your indexer token (see overview of major changes above).
-      # example:
-      curl -X PUT -H "Authorization: Bearer $TOKEN" https://knb.ecoinformatics.org/knb/d1/mn/v2/index?all=true
-      ```
+  > NOTE: Solr upgrade from older cores is not supported. Instead, you must choose a new value for
+  > `solr-home` in the property configuration page, pointing to a new, empty directory. Metacat will
+  > then create the new solr core without any data. Finally, you will need to reindex all objects
+  > ***after the upgrade process is done***, due to incompatibility between old data and the new Solr
+  > schema. ([Instructions below](#you-are-now-ready-to-install-metacat-300).)
   - Ensure that `/etc/default/solr.in.sh` is group writable
     - ex. `sudo chmod g+w /etc/default/solr.in.sh`
   - In `solr.in.sh`, be sure to delete the old solr home add a new solr path:
@@ -114,12 +176,28 @@ Also in this release:
    sudo apt install rabbitmq-server
    sudo systemctl restart rabbitmq-server
    ```
-- You are now ready to install Metacat 3.0.0
-  - Additional notes:
-    - `metacat.properties` no longer contains custom settings, and should not be edited.
-      - Please first re-configure Metacat through the Metacat Admin UI after upgrading.
-      - If you have custom properties that are not available for configuration in the Metacat Admin UI, these can be added to `metacat-site.properties`.
-    - The database upgrade process may require several minutes or longer to complete.
+
+#### You are now ready to install Metacat 3.0.0!
+Additional notes:
+- `metacat.properties` no longer contains custom settings, and should not be edited.
+  - Please first re-configure Metacat through the Metacat Admin UI after upgrading.
+  - If you have custom properties that are not available for configuration in the Metacat Admin UI, these can be added to `metacat-site.properties`.
+- The database upgrade process may require several minutes or longer to complete.
+- **Solr Re-Index:** as mentioned above, Solr upgrade of old cores is not supported. You should have a new, empty
+  solr core available, and can now reindex all objects:
+  - first, a metacat administrator should obtain a temporary auth token by [logging into the KNB
+    website with their ORCID](https://knb.ecoinformatics.org), and navigating to "My Profile" ->
+    "Settings" -> "Authentication Token".
+    > Note - each token is only valid for 24 hours!
+  - then issue the command to reindex all:
+    ```sh
+    $ curl -X PUT -H "Authorization: Bearer $TOKEN" https://<your-host>/<your-context>/d1/mn/v2/index?all=true
+    # where $TOKEN is an environment variable containing your auth token (see overview of major changes above).
+    #
+    # example:
+    # curl -X PUT -H "Authorization: Bearer $TOKEN" https://knb.ecoinformatics.org/knb/d1/mn/v2/index?all=true
+    ```
+
 
 ### New Features & Enhancements:
 

build.properties

@@ -2,11 +2,11 @@
 
 #Version of this build.  This needs to be a dotted numeric version.  For
 #instance 1.9.1 is okay.  1.9.1_rc1 is not.
-metacat.version=3.0.0
+metacat.version=3.1.0
 
 # Release tag for the version of metacatui to bundle with this release
 # Find the latest version here: https://nceas.github.io/metacatui/install/
-metacatui.tag=2.28.0
+metacatui.tag=2.31.0
 
 # java source code version and target version for java compilation
 java.target=17

build.xml

@@ -75,6 +75,10 @@
         <sequential>
             <git command="clone">
                 <args>
+                    <arg value="-b"/>
+                    <arg value="@{refname}"/>
+                    <arg value="--depth"/>
+                    <arg value="1"/>
                     <arg value="@{repository}"/>
                     <arg value="@{dest}"/>
                 </args>

docker/Dockerfile

@@ -2,10 +2,10 @@
 ARG ENVIRONMENT="prod"
 
 # The first stage is used to select the base image for production, based on the JRE
-FROM tomcat:9.0.82-jre17-temurin-jammy AS prod-base
+FROM tomcat:9.0.96-jre17-temurin-jammy AS prod-base
 
 # The second stage is used for testing, if needed, based on the JDK
-FROM tomcat:9.0.82-jdk17-temurin-jammy AS test-base
+FROM tomcat:9.0.96-jdk17-temurin-jammy AS test-base
 
 RUN    printf '\n* * * * * * * * * TEST BUILD * * * * * * * * *'      \
     && printf '\n* * * * * * * * * TEST BUILD * * * * * * * * *'      \
@@ -36,13 +36,13 @@ ENV USRBINDIR=/usr/local/bin
 ENV PATH=${USRBINDIR}:$PATH
 ENV DEVTOOLS=${DEVTOOLS}
 
-# Add a user with name 'metacat'
-RUN groupadd -g 1000 metacat
-RUN useradd -u 1000 -g 1000 metacat
+# Add a user with name 'tomcat'
+RUN groupadd -g 59997 tomcat
+RUN useradd -u 59997 -g 59997 tomcat
 
 ## ADD auto-inflates the .tar.gz
 ADD "$DISTBIN" /tmp/
-ADD "$DISTSRC" /home/metacat/metacat-source/
+ADD "$DISTSRC" /home/tomcat/metacat-source/
 
 RUN apt-get update && apt-get install -y --no-install-recommends \
         python3-bcrypt    \
@@ -58,9 +58,9 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
         vim               \
     &&  rm -rf /var/lib/apt/lists/                                        \
     &&  mv /tmp/metacat.war ${TC_HOME}/webapps                            \
-    &&  chown -R metacat ${TC_HOME}                                       \
-    &&  mkdir -p /var/metacat  && chown metacat:metacat /var/metacat      \
-    &&  mkdir -p /home/metacat && chown -R metacat:metacat /home/metacat  \
+    &&  chown -R tomcat ${TC_HOME}                                       \
+    &&  mkdir -p /var/metacat  && chown tomcat:tomcat /var/metacat      \
+    &&  mkdir -p /home/tomcat && chown -R tomcat:tomcat /home/tomcat  \
     ## Tomcat Mods to Extend Allowed Character Set
         ## - Encoded forward slashes (%2f):
     &&  echo "org.apache.tomcat.util.buf.UDecoder.ALLOW_ENCODED_SLASH=true"     \
@@ -70,18 +70,18 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
                                       >> ${TC_HOME}/conf/catalina.properties
 
 ## Redirect file-based logging to stdout for access logs
-COPY --chown=metacat:metacat tomcat.k8s.server.xml ${TC_HOME}/conf/server.xml
+COPY --chown=tomcat:tomcat tomcat.k8s.server.xml ${TC_HOME}/conf/server.xml
 
 ## Redirect file-based logging for all other logs
-COPY --chown=metacat:metacat tomcat.k8s.logging.properties ${TC_HOME}/conf/logging.properties
+COPY --chown=tomcat:tomcat tomcat.k8s.logging.properties ${TC_HOME}/conf/logging.properties
 
-COPY --chown=metacat:metacat apply_context.py     ${USRBINDIR}/
-COPY --chown=metacat:metacat docker-entrypoint.sh ${USRBINDIR}/
+COPY --chown=tomcat:tomcat apply_context.py     ${USRBINDIR}/
+COPY --chown=tomcat:tomcat docker-entrypoint.sh ${USRBINDIR}/
 
-#Run Container as 'metacat'
-USER 1000
+#Run Container as 'tomcat'
+USER 59997:59997
 
-#Set working directory to tomcat's bin dir, to match legacy metacat relative paths
+#Set working directory to tomcat's bin dir, to match legacy relative paths
 WORKDIR "$TC_HOME"/bin
 
 EXPOSE 8080

docker/README.md

@@ -22,7 +22,7 @@ If you wish to build your own version locally, this can be accomplished by first
 metacat distribution associated with a given version, and then building the docker image based on
 that. Starting in the root directory of the "metacat" repo:
 
-```console
+```shell
 $ ant clean distbinmc
 
 # ... a very long build process ensues, resulting in a tar.gz file
@@ -35,14 +35,14 @@ $ ./build.sh  -t 3.0.0-no-mcui  -v 3.0.0
 
 $ docker image ls
 
-REPOSITORY               TAG              IMAGE ID        CREATED           SIZE
-ghcr.io/nceas/metacat    3.0.0-no-mcui    8da92210dfc4    1 minute ago      1.09GB
+REPOSITORY               TAG             IMAGE ID        CREATED           SIZE
+ghcr.io/nceas/metacat    3.0.0-no-mcui   8da92210dfc4    1 minute ago      1.09GB
 ```
 
 You can then [publish the image to GitHub Container Repository (GHCR)](#publish-the-image-to-ghcr).
 
 Finally, the image can then be deployed in a Kubernetes environment - [see the helm chart](..
-/helm/README.md). Don't forget to change the image tag in your values.yaml file, to match the one 
+/helm/README.md). Don't forget to change the image tag in your values.yaml file, to match the one
 you used when building the image. In the above example, it would be `3.0.0-no-mcui`; e.g:
 
 ```yaml
@@ -58,7 +58,7 @@ inside the container.
 
 To build the image, starting in the root directory of the "metacat" repo: 
 
-```console
+```shell
 $ ant fulldist
 
 # ... a very long build process ensues, resulting in two tar.gz files
@@ -99,9 +99,9 @@ available via the GitHub Container Registry (ghcr.io) so that it can be pulled b
 image can be pushed to the registry after logging in with a GitHub Personal Access Token (PAT).
 
 Commands for pushing the built image are shown below (note this example assumes the tag is `TEST`, 
-although it could also be `DEVELOP`, or a version number, such as `3.0.0`):
+although it could also be `DEVELOP`, or a version number, such as `3.0.0` or `3.0.0-no-mcui`):
 
-```console
+```shell
 # first log in:
 GITHUB_PAT="your-own-secret-GitHub-Personal-Access-Token-goes-here"
 echo $GITHUB_PAT | docker login ghcr.io -u <your-username> --password-stdin

docker/build.sh

@@ -6,13 +6,17 @@ set -e
 TAG=""
 DEFAULT_TAG="DEVELOP"
 TEST_TAG="TEST"
-MC_VERSION=""
-DEFAULT_MC_VERSION="3.0.0"
 DEVTOOLS=false
 ENVIRONMENT='prod'
 DEV_BUILD_OPTS=""
 DISTBIN=""
 DISTSRC="./README.md"
+DEFAULT_MC_VERSION=$(xmllint --xpath \
+        "string(/*[local-name()='project']/*[local-name()='version'])" ../pom.xml)
+if [ -z $DEFAULT_MC_VERSION ]; then
+    echo "Failed to populate \$DEFAULT_MC_VERSION from ../pom.xml. Exiting..."
+    exit 1
+fi
 
 # Function to display usage
 usage() {
@@ -53,15 +57,15 @@ testmode-info() {
 # Parse command-line arguments
 while getopts ":t:v:-:" opt; do
     case "$opt" in
-        t) TAG="$OPTARG" ;;
-        v) MC_VERSION="$OPTARG" ;;
-        -)
-            case "${OPTARG}" in
-                devtools) DEVTOOLS=true ;;
-                *) usage ;;
-            esac
-            ;;
-        ?) usage ;;
+    t) TAG="$OPTARG" ;;
+    v) MC_VERSION="$OPTARG" ;;
+    -)
+        case "${OPTARG}" in
+        devtools) DEVTOOLS=true ;;
+        *) usage ;;
+        esac
+        ;;
+    ?) usage ;;
     esac
 done
 
@@ -136,10 +140,10 @@ if [[ -n $DEV_BUILD_OPTS ]]; then
     echo "  BUILDING OPTIONS:  $DEV_BUILD_OPTS"
 fi
 
-docker image build $DEV_BUILD_OPTS       \
-    --tag ghcr.io/nceas/metacat:"$TAG"   \
+docker image build $DEV_BUILD_OPTS \
+    --tag ghcr.io/nceas/metacat:"$TAG" \
     --build-arg MC_VERSION="$MC_VERSION" \
     --build-arg ENVIRONMENT="$ENVIRONMENT" \
-    --build-arg DISTBIN="$DISTBIN"       \
-    --build-arg DISTSRC="$DISTSRC"       \
-    --build-arg DEVTOOLS="$DEVTOOLS"  .
+    --build-arg DISTBIN="$DISTBIN" \
+    --build-arg DISTSRC="$DISTSRC" \
+    --build-arg DEVTOOLS="$DEVTOOLS" .