Release 2.25 (#219)

* Control versions of indirect dependencies from hadoop (which is needed for parquet at least for now).

* Include specific version of protobuf-java for indirect dependencies.

* Documentation and dependency updates related to the 2.25 release.

* A few more changes to documentation and some changes related to dependency updates (mostly changing test strings to match, but also fixing a deprecated lucene method).

* Add some new files/directories to the exclude list for the build.

* Update docs with links to where the new war file will be.

* Switch back to unpack (instead of unpackWhenChanged) for maven-download-plugin.

Author: ChrisJohnNOAA

DEPLOY_INSTALL.md

@@ -27,7 +27,7 @@ ERDDAP™ can run on any server that supports Java and Tomcat (and other applica
 
     ERDDAP™ works with Java from other sources, but we recommend Adoptium because it is the main, community-supported, free (as in beer and speech) version of Java 21 that offers Long Term Support (free upgrades for many years past the initial release). For security reasons, please update your ERDDAP's version of Java periodically as new versions of Java 21 become available from Adoptium.
 
-    ERDDAP™ has been tested and used extensively with Java 17 and 21, not other versions. For various reasons, we don't test with nor support other versions of Java.  
+    ERDDAP™ has been tested and used extensively with 21, not other versions. For various reasons, we don't test with nor support other versions of Java.  
 
 
 2.  [Set up](#tomcat) [Tomcat](https://tomcat.apache.org).  
@@ -37,7 +37,7 @@ ERDDAP™ can run on any server that supports Java and Tomcat (and other applica
 
 
     *   Download Tomcat and unpack it on your server or PC.  
-        For security reasons, it is almost always best to use the latest version of Tomcat 10 (version 9 and below are not acceptable) which is designed to work with Java 17 or newer. Below, the Tomcat directory will be referred to as _tomcat_.
+        For security reasons, it is almost always best to use the latest version of Tomcat 10 (version 9 and below are not acceptable) which is designed to work with Java 21 or newer. Below, the Tomcat directory will be referred to as _tomcat_.
 
         Warning! If you already have a Tomcat running some other web application (especially THREDDS), we recommend that you install ERDDAP™ in [a second Tomcat](#secondTomcat), because ERDDAP™ needs different Tomcat settings and shouldn't have to contend with other applications for memory.
 
@@ -173,15 +173,16 @@ ERDDAP™ can run on any server that supports Java and Tomcat (and other applica
         *   Or, you can join the [ERDDAP™ Google Group / Mailing List](#ERDDAPMailingList) and post your question there.  
 
 3.  [Set up the _tomcat_/content/erddap configuration files.](#erddapContent)  
-    On Linux, Mac, and Windows, download [erddapContent.zip](https://github.com/ERDDAP/erddap/releases/download/v2.23/erddapContent.zip) (version 2.23, 19810 bytes, MD5=1E26F62E7A06191EE6868C40B9A29362, dated 2023-02-27) and unzip it into _tomcat_, creating _tomcat_/content/erddap .
+    On Linux, Mac, and Windows, download [erddapContent.zip](https://github.com/ERDDAP/erddapContent/releases/download/content1.0.0/erddapContent.zip) (version 1.0.0, 20333 bytes, MD5=2B8D2A5AE5ED73E3A42B529C168C60B5, dated 2024-10-14) and unzip it into _tomcat_, creating _tomcat_/content/erddap .
 
     [Other Directory:](#erddapContentDirectory) For Red Hat Enterprise Linux (RHEL) or for other situations where you aren't allowed to modify the Tomcat directory or where you want/need to put the ERDDAP™ content directory in some other location for some other reason (for example, if you use Jetty instead of Tomcat), unzip erddapContent.zip into the desired directory (to which only user=tomcat has access) and set the erddapContentDirectory system property (e.g., erddapContentDirectory=~tomcat/content/erddap) so ERDDAP™ can find this new content directory.
 
     \[Some previous versions are also available:  
     [2.17](https://github.com/ERDDAP/erddap/releases/download/v2.17/erddapContent.zip) (19,792 bytes, MD5=8F892616BAEEF2DF0F4BB036DCB4AD7C, dated 2022-02-16)  
     [2.18](https://github.com/ERDDAP/erddap/releases/download/v2.18/erddapContent.zip) (19,792 bytes, MD5=8F892616BAEEF2DF0F4BB036DCB4AD7C, dated 2022-02-16)  
     [2.21](https://github.com/ERDDAP/erddap/releases/download/v2.21/erddapContent.zip) (19,810 bytes, MD5=1E26F62E7A06191EE6868C40B9A29362, dated 2022-10-09)  
-    [2.22](https://github.com/ERDDAP/erddap/releases/download/v2.22/erddapContent.zip) (19,810 bytes, MD5=1E26F62E7A06191EE6868C40B9A29362, dated 2022-12-08)  
+    [2.22](https://github.com/ERDDAP/erddap/releases/download/v2.22/erddapContent.zip) (19,810 bytes, MD5=1E26F62E7A06191EE6868C40B9A29362, dated 2022-12-08)
+    [2.23](https://github.com/ERDDAP/erddap/releases/download/v2.23/erddapContent.zip) (19,810 bytes, MD5=1E26F62E7A06191EE6868C40B9A29362, dated 2023-02-27)
     and unzip it into _tomcat_, creating _tomcat_/content/erddap . \]
 
     Then,
@@ -210,16 +211,19 @@ ERDDAP™ can run on any server that supports Java and Tomcat (and other applica
     ERDDAP™ will not work correctly if the setup.xml or datasets.xml file isn't a well-formed XML file. So, after you edit these files, it is a good idea to verify that the result is well-formed XML by pasting the XML text into an XML checker like [xmlvalidation(https://www.xmlvalidation.com/).  
 
 4.  [Install the erddap.war file.](#erddap.war)  
-    On Linux, Mac, and Windows, download [erddap.war](https://github.com/ERDDAP/erddap/releases/download/v2.24/erddap.war) into _tomcat_/webapps .  
-    (version 2.24, 568,748,187 bytes, MD5=970fbee172e28b0b8a07756eecbc898e, dated 2024-06-07)
+    On Linux, Mac, and Windows, download [erddap.war](https://github.com/ERDDAP/erddap/releases/download/v2.25/erddap.war) into _tomcat_/webapps .  
+    (version 2.25, 592,429,675 bytes, MD5=0D93F045A3F38018117C0BB5BA419C99, dated 2024-10-17) 
+    
     The .war file is big because it contains high resolution coastline, boundary, and elevation data needed to create maps.
 
     \[Some previous versions are also available.  
     [2.17](https://github.com/ERDDAP/erddap/releases/download/v2.17/erddap.war) (551,068,245 bytes, MD5=5FEA912B5D42E50EAB9591F773EA848D, dated 2022-02-16)  
     [2.18](https://github.com/ERDDAP/erddap/releases/download/v2.18/erddap.war) (551,069,844 bytes, MD5=461325E97E7577EC671DD50246CCFB8B, dated 2022-02-23)  
     [2.21](https://github.com/ERDDAP/erddap/releases/download/v2.21/erddap.war) (568,644,411 bytes, MD5=F2CFF805893146E932E498FDDBD519B6, dated 2022-10-09)  
     [2.22](https://github.com/ERDDAP/erddap/releases/download/v2.22/erddap.war) (567,742,765 bytes, MD5=2B33354F633294213AE2AFDDCF4DA6D0, dated 2022-12-08)
-    [2.23](https://github.com/ERDDAP/erddap/releases/download/v2.23/erddap.war)(version 2.23, 572,124,953 bytes, MD5=D843A043C506725EBD6F8EFDCCA8FD5F, dated 2023-03-03) \]
+    [2.23](https://github.com/ERDDAP/erddap/releases/download/v2.23/erddap.war) (572,124,953 bytes, MD5=D843A043C506725EBD6F8EFDCCA8FD5F, dated 2023-03-03)
+    [2.24](https://github.com/ERDDAP/erddap/releases/download/v2.24/erddap.war) (568,748,187 bytes, MD5=970fbee172e28b0b8a07756eecbc898e, dated 2024-06-07)
+    \]
 
 5.  [Use ProxyPass](#ProxyPass) so users don't have to put the port number, e.g., :8080, in the URL.  
     On Linux computers, if Tomcat is running in Apache, please modify the Apache httpd.conf file (usually in /etc/httpd/conf/ ) to allow HTTP traffic to/from ERDDAP™ without requiring the port number, e.g., :8080, in the URL. As the root user:

DEPLOY_UPDATE.md

@@ -3,10 +3,10 @@
 
 1.  Make the changes listed in [Changes](https://erddap.github.io/changes.html) in the section entitled "Things ERDDAP™ Administrators Need to Know and Do" for all of the ERDDAP™ versions since the version you were using.  
 
-2.  If you are upgrading from ERDDAP™ version 2.18 or below, you need to switch to Java 17 (or newer) and the related Tomcat 10. See the regular ERDDAP™ installation instructions for [Java](#java) and [Tomcat](#tomcat). You'll also have to copy your _tomcat_/content/erddap directory from your old Tomcat installation to your new Tomcat installation.  
+2.  If you are upgrading from ERDDAP™ version 2.18 or below, you need to switch to Java 21 (or newer) and the related Tomcat 10. See the regular ERDDAP™ installation instructions for [Java](#java) and [Tomcat](#tomcat). You'll also have to copy your _tomcat_/content/erddap directory from your old Tomcat installation to your new Tomcat installation.  
 
-3.  Download [erddap.war](https://github.com/ERDDAP/erddap/releases/download/v2.24/erddap.war) into _tomcat_/webapps .  
-    (version 2.24, 568,748,187 bytes, MD5=970fbee172e28b0b8a07756eecbc898e, dated 2024-06-07)
+3.  Download [erddap.war](https://github.com/ERDDAP/erddap/releases/download/v2.25/erddap.war) into _tomcat_/webapps .  
+    (version 2.25, 592,429,675 bytes, MD5=0D93F045A3F38018117C0BB5BA419C99, dated 2024-10-17)
 
 4.  [messages.xml](#messages.xml)
     *   Common: If you are upgrading from ERDDAP™ version 1.46 (or above) and you just use the standard messages, the new standard messages.xml will be installed automatically (amongst the .class files via erddap.war).  

PROGRAMMER_GUIDE.md

@@ -17,7 +17,7 @@ ERDDAP™ uses Maven to load code dependencies as well as some static reference
 
   - [ref\_files.zip](https://github.com/ERDDAP/ERDDAPRefFiles/releases/download/1.0.0/ref_files.zip) and unzip it into /WEB-INF/ref/ .
 
-  - [erddapContent.zip](https://github.com/ERDDAP/erddap/releases/download/v2.24/erddapContent.zip) (version 2.24, 19810 bytes, MD5=1E26F62E7A06191EE6868C40B9A29362, dated 2024-06-07) and unzip it into _tomcat_, creating _tomcat_/content/erddap .
+  - [erddapContent.zip](https://github.com/ERDDAP/erddapContent/releases/download/content1.0.0/erddapContent.zip) (version 1.0.0, 20333 bytes, MD5=2B8D2A5AE5ED73E3A42B529C168C60B5, dated 2024-10-14) and unzip it into _tomcat_, creating _tomcat_/content/erddap .
 
 NOTE: Maven caches downloads but will unzip the downloaded archives on each execution, which takes time. To skip downloading
 and unzipping archives, you may specify the `skipResourceDownload` property to Maven (e.g. `mvn -DskipResourceDownload package`).

WEB-INF/classes/gov/noaa/pfel/erddap/Erddap.java

@@ -70,6 +70,7 @@
 import java.util.regex.Pattern;
 import java.util.zip.ZipOutputStream;
 import org.apache.lucene.document.Document;
+import org.apache.lucene.index.StoredFields;
 import org.apache.lucene.index.Term;
 import org.apache.lucene.search.BooleanClause;
 import org.apache.lucene.search.BooleanQuery;
@@ -16118,6 +16119,7 @@ public StringArray getSearchDatasetIDs(
                   + (System.currentTimeMillis() - luceneTime)
                   + "ms");
         luceneTime = System.currentTimeMillis();
+        StoredFields storedFields = indexSearcher.storedFields();
         for (int i = 0; i < nHits; i++) {
           // 3 ways to find datasetID:
 
@@ -16134,7 +16136,7 @@ public StringArray getSearchDatasetIDs(
           Integer docNI = Integer.valueOf(docN);
           String tDatasetID = EDStatic.luceneDocNToDatasetID.get(docNI);
           if (tDatasetID == null) { // not yet in luceneDocNToDatasetID
-            Document doc = indexSearcher.doc(docN);
+            Document doc = storedFields.document(docN);
             if (doc == null) // perhaps just removed from index
             continue;
             tDatasetID = doc.get("datasetID");

WEB-INF/classes/gov/noaa/pfel/erddap/util/EDStatic.java

@@ -195,7 +195,8 @@ public class EDStatic {
    * 2.21 released on 2022-10-09 <br>
    * 2.22 released on 2022-12-08 <br>
    * 2.23 released on 2023-02-27 <br>
-   * 2.24 released on 2024-06-07
+   * 2.24 released on 2024-06-07 <br>
+   * 2.25 released on 2024-10-16 <br>
    *
    * <p>For main branch releases, this will be a floating point number with 2 decimal digits, with
    * no additional text. !!! In general, people other than the main ERDDAP developer (Bob) should
@@ -205,7 +206,7 @@ public class EDStatic {
    * anything following it. A request to http.../erddap/version will return just the number (as
    * text). A request to http.../erddap/version_string will return the full string.
    */
-  public static String erddapVersion = "2.24"; // see comment above
+  public static String erddapVersion = "2.25"; // see comment above
 
   /**
    * This is almost always false. During development, Bob sets this to true. No one else needs to.
@@ -1806,7 +1807,7 @@ public static int convertToPublicSourceUrlFromSlashPo(String tFrom) {
   private static Table gdxAcronymsTable;
 
   private static HashMap<String, String> gdxAcronymsHashMap, gdxVariableNamesHashMap;
-  public static boolean useSharedWatchService = false;
+  public static boolean useSharedWatchService = true;
 
   /**
    * This static block reads this class's static String values from contentDirectory, which must
@@ -2276,7 +2277,7 @@ public static int convertToPublicSourceUrlFromSlashPo(String tFrom) {
       variablesMustHaveIoosCategory =
           getSetupEVBoolean(setup, ev, "variablesMustHaveIoosCategory", true);
       warName = getSetupEVString(setup, ev, "warName", "erddap");
-      useSharedWatchService = getSetupEVBoolean(setup, ev, "useSharedWatchService", false);
+      useSharedWatchService = getSetupEVBoolean(setup, ev, "useSharedWatchService", true);
 
       // use Lucence?
       if (searchEngine.equals("lucene")) {

download/AccessToPrivateDatasets.html

@@ -194,7 +194,7 @@ <h2><a class="selfLink" id="contact" href="#contact" rel="bookmark">Contact</a><
 
 <br>&nbsp;
 <hr>
-<p>ERDDAP, Version 2.21
+<p>ERDDAP, Version 2.25
 <br><a rel="bookmark" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html">Disclaimers</a> | 
     <a rel="bookmark" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html#privacyPolicy">Privacy Policy</a>
 <p>&nbsp;

download/EDDTableFromEML.html

@@ -606,7 +606,7 @@ <h2><a class="selfLink" id="contact" href="#contact" rel="bookmark">Contact</a><
 
 <br>&nbsp;
 <hr>
-<p>ERDDAP, Version 2.24
+<p>ERDDAP, Version 2.25
 <br><a rel="bookmark" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html">Disclaimers</a> | 
     <a rel="bookmark" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html#privacyPolicy">Privacy Policy</a>
 

download/NCCSV.html

@@ -943,7 +943,7 @@ <h2><a class="selfLink" id="contact" href="#contact" rel="bookmark">Contact</a><
 
 <br>&nbsp;
 <hr>
-<p>ERDDAP, Version 2.24
+<p>ERDDAP, Version 2.25
 <br><a rel="bookmark" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html">Disclaimers</a> | 
     <a rel="bookmark" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html#privacyPolicy">Privacy Policy</a>
 </div>

download/NCCSV_1.00.html

@@ -859,7 +859,7 @@ <h2><a class="selfLink" id="contact" href="#contact" rel="bookmark">Contact</a><
 
 <br>&nbsp;
 <hr>
-<p>ERDDAP, Version 2.24
+<p>ERDDAP, Version 2.25
 <br><a rel="bookmark" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html">Disclaimers</a> | 
     <a rel="bookmark" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html#privacyPolicy">Privacy Policy</a>
 </div>

download/NCCSV_1.10.html

@@ -930,7 +930,7 @@ <h2><a class="selfLink" id="contact" href="#contact" rel="bookmark">Contact</a><
 
 <br>&nbsp;
 <hr>
-<p>ERDDAP, Version 2.24
+<p>ERDDAP, Version 2.25
 <br><a rel="bookmark" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html">Disclaimers</a> | 
     <a rel="bookmark" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html#privacyPolicy">Privacy Policy</a>
 </div>

download/changes.html

@@ -70,7 +70,6 @@ <h1 style="text-align:center;">ERDDAP™ Changes</h1>
 
 <!-- ****************************************************************************** -->
 
-<!-- TODO Make sure github.io site is updated with all the latest documentation for release -->
 <h2><a class="selfLink" id="changes2.25" href="#changes2.25" rel="bookmark">Changes</a>
   in ERDDAP™ version 2.25 (released TBD)</h2>
 <ul>
@@ -81,6 +80,15 @@ <h2><a class="selfLink" id="changes2.25" href="#changes2.25" rel="bookmark">Chan
     </ul>
   <li><strong>Things ERDDAP™ Administrators Need to Know and Do:</strong>
     <ul>
+      <li>We recommend upgradding to Java 21 or newer. This is the LTS version and has been available for over a year. Development
+        of ERDDAP™ version 2.25 was done with Java 21.
+        <br>&nbsp;
+      <li>The SharedWatchService is now the default. If you need to disable it, please contact chris.john at noaa.gov to let me know,
+        so I can improve it in future versions and add:
+        <br><kbd>
+          &lt;useSharedWatchService&gt;false&lt;/useSharedWatchService&gt;
+        </kbd> to your setup.xml.
+        <br>&nbsp;
       <li>The ERDDAP™ servlet will now start at server startup. Which means datasets will begin loading immediately instead of
         waiting until a request is made.
         <br>&nbsp;
@@ -149,9 +157,9 @@ <h2><a class="selfLink" id="changes2.24" href="#changes2.24" rel="bookmark">Chan
         threads used to watch for changes. It does mean all datasets get updated together instead of each dataset
         having its own update frequency. Most likely this will mean more frequent updates for most datasets.
 
-        <br>To enable this add <kbd>
-          <useSharedWatchService>true</useSharedWatchService>
-        </kbd> to your setup.xml.
+        <br><p>To enable this add <kbd>
+          &lt;useSharedWatchService&gt;true&lt;/useSharedWatchService&gt;
+        </kbd> to your setup.xml.</p>
 
         <br>Please do try this and report back how it works for you to chris.john at noaa.gov.
         <br>&nbsp;
@@ -161,6 +169,18 @@ <h2><a class="selfLink" id="changes2.24" href="#changes2.24" rel="bookmark">Chan
       <li>Some small changes and bug fixes.
         <br>&nbsp;
     </ul>
+    <li><strong>Improvements for ERDDAP™ developers:</strong>
+      <ul>
+        <li>Support for local development using Docker.
+        Thanks Matt Hopson and Roje.
+        <br>&nbsp;
+        <li>Support for local development using Jetty and documentation improvements.
+          Thanks Micah Wengren.
+          <br>&nbsp;
+        <li>Changes to tests to reduce issues cross platform.
+          Thanks Shane St. Savage.
+          <br>&nbsp;
+      </ul>
 </ul>
 
 <!-- ****************************************************************************** -->
@@ -7137,7 +7157,7 @@ <h2><a class="selfLink" id="contact" href="#contact" rel="bookmark">Contact</a><
 
 <br>&nbsp;
 <hr>
-<p>ERDDAP, Version 2.24
+<p>ERDDAP, Version 2.25
 <br><a href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html">Disclaimers</a> | 
     <a href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html#privacyPolicy">Privacy Policy</a>
 

download/grids.html

@@ -1248,7 +1248,7 @@ <h3><a class="selfLink" id="RemoteReplicationOfDatasets_Solutions" href="#Remote
 
 <br>&nbsp;
 <hr>
-<p>ERDDAP, Version 2.24
+<p>ERDDAP, Version 2.25
 <br><a rel="help" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html">Disclaimers</a> | 
     <a rel="help" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html#privacyPolicy">Privacy Policy</a>
 

download/setup.html

@@ -4302,7 +4302,7 @@ <h2><a class="selfLink" id="contact" href="#contact" rel="bookmark">Contact</a><
 
 <br>&nbsp;
 <hr>
-<p>ERDDAP, Version 2.24
+<p>ERDDAP, Version 2.25
 <br><a rel="bookmark" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html">Disclaimers</a> | 
     <a rel="bookmark" href="https://coastwatch.pfeg.noaa.gov/erddap/legal.html#privacyPolicy">Privacy Policy</a>