Taxanomic Infrastructure

Table of Contents

• Grails 3 nsl-domain-plugin
  • Getting Started
  • Build
  • Install
  • Considerations
  • Where to make your changes.
  • What version number
  • Notes
  • What is this?
  • Shard ID ranges
• Publishing the plugin (maven repo)
  • Snapshots
  • Release
• Create a clean Database
  • Known Issues

This is a grails plugin for the NSL project that maps the Domain objects of NSL. It is used to generate and update the schema for the NSL.

Grails 3 nsl-domain-plugin

Getting Started

Build

1. To build the plugin you need a database called nslimport

2. To install the plugin you need grails3 and Java 8

~/IdeaProjects/NSL/nsl-domain-plugin-g3$ grails --version
| Grails Version: 5.2.5
| JVM Version: 1.8.0_232

-- Create role and database from sql prompt
CREATE ROLE nsldev with LOGIN PASSWORD 'nsldev';
CREATE DATABASE nslimport WITH OWNER nsldev;

Install

1. With that in place, you can now run ./gradlew build from the project root

2. Once that succeeds, you can install the plugin using the command ./gradlew publishToMavenLocal

3. This installs the plugin in the maven directory as below

mo@local:~/nsl-domain-plugin-g3$ ls -l ~/.m2/repository/au/org/biodiversity/grails/plugins/nsl-domain-plugin-g3
total 32
drwxrwxr-x. 2 mo mo 4096 Jun 17  2019 1.17-SNAPSHOT
drwxrwxr-x. 2 mo mo 4096 Jul 25  2019 1.18-SNAPSHOT
drwxrwxr-x. 2 mo mo 4096 Aug  9  2019 1.19-SNAPSHOT
drwxrwxr-x. 2 mo mo 4096 Aug 14  2019 1.19.SNAPSHOT
drwxrwxr-x. 2 mo mo 4096 Aug 27  2019 3.1.19-SNAPSHOT
drwxrwxr-x. 2 mo mo 4096 Sep  2  2019 3.1.20-SNAPSHOT
drwxrwxr-x. 2 mo mo 4096 Jun 29 11:48 3.1.21-SNAPSHOT  # Current version
-rw-rw-r--. 1 mo mo  553 Nov 23 09:48 maven-metadata-local.xml

If you run into issues, checkout our known issues section

Considerations

When making schema changes we have to consider two cases:

• creating the schema for a new database

• modifying the schema in an existing database

and this means your changes need to appear in an update script and the nsl-ddl.sql file.

Where to make your changes.

Adding/Changing a view, trigger, index or default data

If you are adding a view, trigger, index or default data: 1. first add your changes to an appropriate file in the /src/main/resources/sql/views directory 1. run grails generate-nsl-ddl to regenerate the nsl-ddl.sql file with your changes included 1. Add your change to the latest update-to-nn.sql file in the src/main/resources/sql directory

Remember the update-to-nn script is updating a database from the previous version, so you must take into account existing views and not loose data.

Adding or changing tables (domain objects)

If you are changing a table or tables you must make those changes in the Grails domain classes which will generate the appropriate sql when the grails generate-nsl-ddl command is run. This keeps the grails domain classes,sql and DB schema in sync.

1. make your changes to the grails domain classes in /grails-app/domain/au/org/biodiversity/nsl

2. preferably test your changes using gradle install locally

3. run gradle clean to remove the old ddl build directory

4. run grails generate-nsl-ddl to regenerate the /src/main/resources/nsl-ddl.sql file with your changes included

5. diff nsl-ddl.sql to extract changes to put into the latest update-to-nn.sql file in the /src/main/resources/sql directory

What version number

Make sure the version number you are working on is not the currently released version, if necessary create a new version and update the default version number in sql/views/other-setup.sql as well as grails-app/services/au/org/biodiversity/nsl/NslDomainService.groovy.

Notes

The simplest way to make the update sql is the re-generate the DDL from grails using the script:

grails generate-nsl-ddl

Warning

You should run grails clean before doing grails generate-nsl-ddl or you may end up with the hibernate generated ddl in there twice.

and diff the result to see what needs to be changed.

When starting off with a fresh NSL shard it’s probably best to execute the nsl-ddl.sql file against your database (traditionally called nsl).

When writing update-to-nn.sql scripts we try to make the update script idempotent, so that if it is run twice (either deliberately or accidentaly) it results in the same thing. Note data loss is possible it the scripts are run again after a while.

What is this?

This is part of the NSL (National Species List) Services Application developed at the Australian National Botanic Gardens in association with CHAH, the ALA and the IBIS team.

This code is Licensed under the Apache 2 License and we welcome contributions from the taxonomic community.

Please use Github issues for any bug reports.

Shard ID ranges

For Australian NSL infrastructure we are splitting the ID ranges used across different known shards. This is not essential but it does mean we have a means to discriminate data sources on IDs if somehow we manage to loose track.

The current allocated ranges are:

• Vascular (APNI) 1000 - 10,000,000 + 50,000,001 - 60,000,000

• Moss 10,000,001 - 20,000,000

• Algae 20,000,001 - 30,000,000

• Lichen 30,000,001 - 40,000,000

• DAWR 40,000,001 - 50,000,000 (Aust. Dept. of Agriculture and Water Resources)

• Fungi 60,000,001 - 70,000,000

• AFD 70,000,001 - 80,000,000

There is a confluence page for the NSL infrastructure as well at https://www.anbg.gov.au/ibis25/pages/viewpage.action?spaceKey=NSL&title=NSL+Project+2.0

You need to set the shard sequence rang in the nsl-ddl.sql script before running it. e.g.

create sequence nsl_global_seq minvalue 80000001 maxvalue 90000000;

Publishing the plugin (maven repo)

Snapshots

This repository is automatically built by Travis CI (https://travis-ci.org/bio-org-au/nsl-domain-plugin-g3) and deployed to 'OJO' (http://oss.jfrog.org), which is an artifact (maven) repository for open source projects.

We publish the plugin to a oss.jfrog.org. To do this we need to set the BINTRAY_USER and BINTRAY_KEY environment variables and the call ./gradlew publish to publish to 'OJO'.

To use the snapshot you need to include the snapshot repository by adding this to your gradle repositories block:

build.gradle

repositories {
    maven {
            url "https://oss.jfrog.org/artifactory/oss-snapshot-local"
        }
}

to consume the snapshot add the following to the dependencies:

compile "au.org.biodiversity.grails.plugins:nsl-domain-plugin-g3:3.1.19-SNAPSHOT"

Release

To release the plugin we merge master into the release branch and remove the SNAPSHOT status from the plugin version. Then run the bintrayUpload gradle command to upload the released plugin to bintray https://bintray.com/beta/#/bio-org-au/plugins/nsl-domain-plugin-g3?tab=overview

Note	No releases have been pushed yet for G3. We will add a separate travis.yml in the release branch to build the release.

Create a clean Database

• Create a clean DB

• use psql to run the nsl-ddl.sql command (in src/main/resources/sql)

creatdb mynsl && psql -f nsl-ddl.sql mynsl

Known Issues

You might get the below error when you run grails install

Symptom: No profile found for name [web-plugin]

$ grails install --stacktrace
| Error Error occurred running Grails CLI: No profile found for name [web-plugin]. (NOTE: Stack trace has been filtered. Use --verbose to see entire trace.)
java.lang.IllegalStateException: No profile found for name [web-plugin].
	at org.grails.cli.GrailsCli.initializeProfile(GrailsCli.groovy:518)
	at org.grails.cli.GrailsCli.initializeApplication(GrailsCli.groovy:306)
	at org.grails.cli.GrailsCli.execute(GrailsCli.groovy:269)
	at org.grails.cli.GrailsCli.main(GrailsCli.groovy:159)
| Error Error occurred running Grails CLI: No profile found for name [web-plugin].

Fix: Run gradle clean task

$ ./gradlew clean
user null
:clean

$ ./gradlew install
user null
:generatePomFileForMavenPublication
:compileAstJava NO-SOURCE
:compileAstGroovy NO-SOURCE
:processAstResources NO-SOURCE
:astClasses UP-TO-DATE
:compileJava NO-SOURCE
:configScript
:compileGroovy
:copyAstClasses NO-SOURCE
:assetPluginPackage
:copyCommands
:copyTemplates NO-SOURCE
:processResources
:classes
:compileWebappGroovyPages NO-SOURCE
:compileGroovyPages
:jar
:groovydoc
Trying to override old definition of task fileScanner
[ant:groovydoc] Resource org/codehaus/groovy/tools/groovy.ico not found so skipped
:javadoc
:javadocJar
:sourcesJar
:publishMavenPublicationToMavenLocal
:publishToMavenLocal
:install

BUILD SUCCESSFUL

Total time: 16.441 secs