TEKSI Modules continuous integration and release management

[ponceta]

TEKSI Modules continuous integration and release management

Teksi modules component should be up to date, functionnal and correctly documented and translated.

This can be insured by the continuous integration and release management and should be pretty similar in all TEKSI modules.

[sjib]

General information

• Applicant: Arnaud Poncet / Stefan Burckhardt
• Developer: to be defined
• Service provider group referent: to be defined
• Developments referent: to be defined

Users and administrators needs

Providers want to easly administer TEKSI modules for their users:

1. Setup and initialisation
2. Upgrade and maintenance
3. INTERLIS Import of data
4. INTERLIS Export of data

The Release manager (currently the technical secretary) wants to easily test and create new releases for the TEKSI Modules
A release is created if either:

• the datamodel is adapted
• the project file is adapted
• the plugin is adapted
• the QGIS Minimal version needs to be adapted (to discuss)
• the Postgres Minimal version needs to be adapted (to discuss)

5. Creation of delta files for the different datamodels for the TEKSI Modules
6. Creation and testing of a adapted project file
7. Creation of project file translations (currently de, fr)
8. Adaption of demo data to fit new datamodel and project
9. Creation and testing of a TEKSI module plugin
10. Creation and testing of the TMMT plugin

Graphical identity / Logo

TEKSI Modules Management Tool (TMMT)
[Illustrations if applicable.]

Description of users and administrators feature requirements

A. Administraton of TEKSI Modules (by (TEKSI) providers) for their clients :
To easily administer TEKSI modules for their users, the (TEKSI) providers need to be able to do

1. Setup and initialisation

• The providers want an easy way of setting up a new database for a specific module, as it has been developed with the qgep datamodel tool where they choose a datamodel version for the module, select or define the configuration and initialize the selected data model version on postgres:

  • Selection of Module (Water, Wastewater, Distance Heating etc.)
  • Selection of corresponding TEKSI data model version
  • Selection or definition of connection configuration

• The providers then want to download a project file version that fits this data model version and configuration.

• They would also like to choose the language of the project (new feature)

• They want to easily define and/or check the prefix (new feature)

• They want to easily set default values for fk_dataowner (Datenherr), fk_provider (Datenlieferant), fk_owner (Eigentuemer), fk_operator (Betreiber) (integrate new functionality)

• They want to easily manage user and access rights to a database cluster and / or specific database (integrate new functionality for defaults, documentation on how to use the default roles and setup specific user, select authentication methode (e.g. configure pg_conf / pgpass or other login options from QGIS), define new functionality to support these steps)

2. Upgrade and maintenance

• The providers want an easy way to upgrade an existing database to a new data model version e.g. 2024.0 to 2024.1
• They should be able to easily create a backup (documentation and/or sample script)
• They should be able to easily able to restore that backup (documentation and/or sample script)
• They should be able to check, if an upgrade is possible on that backup
• They should then run the upgrade on that backup
• They should then run some integrity checks on the upgraded version
• If everything goes well they should perform the same upgrade on the master database

3. INTERLIS Import of data

• The administrators should be able to easily import data provided in the corresponding INTERLIS model version for the module
• Selection option for location of log files
• The import process automatically analyses the MODEL of the xtf data.
• If the MODEL is supported, the process continues, else a warning is provided to the user.
• The import process is first checking the validity of the INTERLIS xtf file
• New feature option (to discuss): Allow to skip checking (MANDATORY constraints, Relation constraints), if first attempt failed, with clear warning to the user, that this could cause other problems in a special "Old data migration mode"
• If ok it then performs an import into the intermediary schema created with ModelBaker
• It then provides a wizard to make one of the following choices (general selection buttons (new feature)):

  • Update and append (Default)

  • Update (only updates all existing data in the database)

  • **Append"" (Only append new data)

    And two general selection buttons

    • Deselect all data in all classes
    • Select all data in all classes (Default)
      And keep the feature to select / deselect:
    • Select/Deselect all data in one specific class (needs improvement in speed and implementation - see Optimize Import for large dataset wastewater#206 (comment)
• It then allows to make the final import into TEKSI with the selected options
• Post processes (like re-enabling trigger, update fk_main_cover, update fk_main_node, etc.) are applied and communicated to the user
• The user is informed whether the import has been successful
• and a link to the log files is provided
• Configuration for additional models is standardized and can be easily integrated e.g. for additional classes, extensions such as Protection_tubes, Remote control cables
• Documentation about this additional configuration is provided in the Developers Guide

4. INTERLIS Export of data

• The administrators should be able to easily export data provided in the corresponding INTERLIS models version for the module:
  • Selection of Export model
  • Option to only export selected features
  • Selection of Orientation
  • Selection options for Label Export
  • Selection option for location of log files
• Starting of Export process by the user
• If ok it then performs an import into the intermediary schema created with ModelBaker and provides a log for that step
• The export process then writes the xtf File based on the selected model and provides a log for that step
• The export process is lastly checking the validity of the INTERLIS xtf file and provides a log for that step
• The user is informed whether a valid export files has been created
• and a link to the log files is provided
• Configuration for additional models is standardized and can be easily integrated e.g. for additional classes, extensions such as Protection_tubes, Remote control cables
• Documentation about this additional configuration is provided in the Developers Guide

B. Code infrastructure and Release management
The Release Manger needs to have a code infrastructure (currently github) that allows to easily test and create new releases for the TEKSI Modules.

Current modules:

• TEKSI wastewater & GEP (TWW)

Planned modules:

• TEKSI Management Tool (TMMT)
• TEKSI Drinking Water (TDW)
• TEKSI Distance Heating (TDH)
• ...

Release Processes:
5. Creation of delta files for the different datamodels for the TEKSI Modules

• Create Concept and definition of control of data model evolution:
  • delta files based on experience with qgep / qwat have been a good concept for application data models
  • To discuss if this is also the good concept for major module data model changes
  • Setup a discussion meeting with Technical secretary, opengis and SJiB
  • review of suggested solutions by Technical group
• and integration of definitions in the Developers Guide
• Define adaptions to PUM (psycopg3, tests, transactions
• Implementation of needed adaptions
• Testing of changelogs with test datasets for each module

6. Creation and testing of a adapted project file

• Define how to do this
• Use QGIS trackable plugin to structure qgs file and easier comparison of differences.
• ...

7. Creation of project file translations (currently de, fr)

• Review and adapt workflows on github
• Add module specific dictionaries - check if paid version for Transifex is needed / helpful
• Document process for support of translation by TEKSI Members
• ...

8. Adaption of demo data to fit new datamodel and project

• Develop concept for versioned demo data managment (versioning)
• Check utilization of historization module for QGIS
• Check use MUTATION class in VSA-DSS data model
• Implement workflows and tests on github
• Discuss and define options for demo data sets: Check on copyright, synergies with VSA demo dataset, real world data versus generic data
• Check to transform of existing data like Community A. (former qgep demo data), other small community from members, ... to comply with copyright and data security,
• ...

9. Creation and testing of a TEKSI module plugin

• Implement workflows and tests on github - up to now only artifacts of dumps for data model
• Creation of upgrade workflow for each TEKSI module plugin
• Creation and testing of a pre-release for a module - for easy testing by selected members / developer
• Creation and testing of a Release for a module - for publication in QGIS plugin repository
• Traduction of packages (TEKSI Modules)
• Publication of a TEKSI module plugin in the QGIS plugin repository

10. Creation and testing of the TMMT plugin

• Listens to the last releases of the modules (new feature, currently it is hard-coded)
• Option to show only releases, pre-releases, dev-releases
• Implement workflows and tests on github
• Creation of upgrade workflow for the TMMT plugin
• Creation and testing of a pre-release for the TMMT plugin - for easy testing by selected members / developer
• Creation and testing of a Release for the TMMT plugin - for publication in QGIS plugin repository
• Traduction of packages (TMMT)
• Publication of a TEKSI module plugin in the QGIS plugin repository

General requirements

For the TEKSI Management Module (former qgep datamodel tool) see teksi/TMMT#1

Specific requirements

Necessary features:

Illustrations of the features and the requirements

Schemes, screen captures, or any other illustrations.

Graphical User Interface (GUI)

(https://teksi.github.io/wastewater/_images/datamodel-tool.png)
Data model Tool - should be adapted to become the TEKSI Modules Management Tool (TMMT). A new option to select the module is needed

Technical analysis and specifications

Feasibility study

Feasibility, options, alternatives and technical specificities

Before the implementation of part B start

Risks and constraints

Creation and publication of new releases of TEKSI modules has been a partly workflow supported process that is only vaguely documented and depends highly on the 2-3 people and has a high dependency on the know how of the developers that set it up in the last 10 years. If these people disappear the release process is blocked and publication and maintenance of all TEKSI modules is blocked.

As there are also a lot of manuel steps and an undefined amount of time needed for the publication of a new release, publication of new releases have been only once two two times a year. TEKSI has defined a roadmap and would like to publish three releases per year with a major release in Spring and two minor release four and eight months later to allow wider testing of new features and functionalities.

Therefore is it highly recommended to standardize and document these processes and make them more indepentant of specific people.

Mutualisation

Mutualisation and possible impacts with other projects (QGIS, QWAT, QGEP)

This project is a key element in the ongoing mutualisation of the former modules qgep and qwat and lay a solid foundation for the installation and maintenance of all (future) TEKSI modules. As specified in the Road map, TEKSI will invest a lot of efforts in the development of the existing wastewater and water modules and has started also first developments for new topics (Distance heating). Therefore standardization of the installation, upgrade and import / export processes over all existing and future TEKSI modules is a key task to keep or even reduce maintenance costs for users, service providers and the TEKSI code base. A main goal is to heavily support the TEKSI service providers in their daily work on installing and maintaining the modules and databases of their clients.

With this project the vision of

Let's install a TEKSI module (for demonstration purposes) in less than ten minutes!

should become realistic.

Specification and offers ( Developer )

Not shared on Github.

Specifications

Tenders

Not shared on Github.

Timeline / deadlines

Add a detailed timeline

• 13.05.2024 : First draft

• 14.05.2024 : Presentation to Technical Group

• Search of project manager

• Finalize draft to become valid project description

• Draft of Requirements document

• 29.05.2024 Presentation to TEKSI board with proposed project manager

• Finalize Definition of Requirements document

• June 2024 Call for offer

• June 2024 Launch crowdsourcing

• Define timeplan with code sprints and phases for development, testing, integration, release

• Confirm offer

• Start development Phase 1

• ..

• Warranty period : to be adressed

Related softwares versions

• QGIS version: 3.34.6 (LTR)
• Database model version: TWW 2024.0 and newer (to discuss - include also qgep and qwat?)
• INTERLIS Model version:

Public issues links :

• …

Helpdesk issues links :

• …

Realisation dates

• Tender reception date :
• Tender validation date :

Documentation

Documentation links

• Roadmap
• Features development list
• Developer Guide
• Contributor Guide

• User manuals : TEKSI Wastewater / TEKSI Distance Heating / qwat

Development and integration

Link and comments from the developer about the product

• Deliverable :
• Recommendations and comments :
• Future prospects :

Conclusion

Conclusions from applicant, users or admins group

• Conclusion:
• Future prospects:

[sjib]

Decide Datamodel generation: see https://github.com/orgs/teksi/discussions/92

[sjib]

Integrate conceptional phase to add additional INTERLIS models - Distant heating, but also Protection_tubes and Control_cables that have an extra model, but are integrated database wise into the water, wastewater, distant heating, etc. modules.
Discuss whether INTERLIS import / export should also be moved to TMMT plugin and how configuration files will be structured when adding more models.

[sjib]

Another point with INTERLIS import / export is the command line tool mode of it - see https://github.com/QGEP/qgepqwat2ili that should be supported again fully also with TEKSI

[tproduit]

Proposition: Description of users and administrators feature requirements

There are three modules

• A: Code infrastructure - release management
• B: Administraton of TEKSI Modules (TMMT - Teksi Management Modules Tool)
• C: Import and export of Interlis files (TIT - Teksi Interlis Tool)

A. Code infrastructure and Release management

The Release Manger needs to have a code infrastructure (currently github) that allows to easily test and create new releases for the TEKSI Modules.

Business Modules:

• TEKSI wastewater & GEP (TWW)
• TEKSI Drinking Water (TDW - planned)
• TEKSI Distance Heating (TDH - planned)

Management modules:

• TEKSI Management Tool (TMMT - planned)
• TEKSI Interlis Tool (TIT - planned)

Release Processes:

1. Creation of delta files for the different datamodels for the TEKSI Modules

• Create concept and definition of control of data model evolution:

  • delta files based on experience with qgep / qwat have been a good concept for application data models.
    • delta files could be generated automatically (by comparing two models)
  • To discuss if this is also the good concept for major module data model changes
  • Setup a discussion meeting with Technical secretary, opengis and SJiB
  • review of suggested solutions by Technical group

• Integration of definitions in the Developers Guide

• Define adaptions to PUM (psycopg3, tests, transactions)

• Implementation of needed adaptions

• Testing of changelogs with test datasets for each module

2. Creation and testing of a adapted project file

• Define how to do this
• Use QGIS trackable plugin to structure .qgs file and easier comparison of differences.
• ...

3. Creation of project file translations (currently de, fr)

• Review and adapt workflows on github
• Add module specific dictionaries - check if paid version for Transifex is needed / helpful
• Document process for support of translation by TEKSI Members
• ...

4. Adaption of demo data to fit new datamodel and project

• Develop concept for versioned demo data managment (versioning)
• Check utilization of historization module for QGIS
• Check use MUTATION class in VSA-DSS data model
• Implement workflows and tests on github
• Discuss and define options for demo data sets: Check on copyright, synergies with VSA demo dataset, real world data versus generic data
• Check to transform of existing data like Community A. (former qgep demo data), other small community from members, ... to comply with copyright and data security,
• ...

5. Creation and testing of a business TEKSI module plugin

• Implement workflows and tests on github - up to now only artifacts of dumps for data model
• Creation of upgrade workflow for each TEKSI module plugin
• Creation and testing of a pre-release for a module - for easy testing by selected members / developer
• Creation and testing of a Release for a module - for publication in QGIS plugin repository
• Traduction of packages (TEKSI Modules)

6. Publication of a business TEKSI module plugin in the QGIS plugin repository

7. Testing and release of management plugins

• Define if a release of the managment plugins is necessary

TMMT plugin

Test of the functionalities:

• Must enable the installation of the new release (releases, pre-releases, dev-releases)
• Must enable the upgrade to the new release

TIT plugin

Test of the functionalities:

• import of interlis files
• export of interlis files

B. Administraton of TEKSI Modules (TMMT - Teksi Management Modules Tool)

To easily administer TEKSI modules for their users, the (TEKSI) providers need to be able to do

1. Setup and initialisation

• The providers want an easy way of setting up a new database for a specific module, as it has been developed with the qgep datamodel tool where they choose a datamodel version for the module, select or define the configuration and initialize the selected data model version on postgres:

  • Selection of Module (Water, Wastewater, Distance Heating etc.)
    • Listens to the last releases of the modules (new feature, currently it is hard-coded)
  • Selection of corresponding TEKSI data model version
    • Option to show only releases, pre-releases, dev-releases
  • Selection or definition of connection configuration
  • Selection of a mode (production, demo (include data), test-dev)

• The providers then want to download a project file version that fits this data model version and configuration.

• They would also like to choose the language of the project (new feature)

• They want to easily define and/or check the prefix (new feature)

• They want to easily set default values for fk_dataowner (Datenherr), fk_provider (Datenlieferant), fk_owner (Eigentuemer), fk_operator (Betreiber) (integrate new functionality)

• They want to easily manage user and access rights to a database cluster and / or specific database (integrate new functionality for defaults, documentation on how to use the default roles and setup specific user, select authentication methode (e.g. configure pg_conf / pgpass or other login options from QGIS), define new functionality to support these steps)

  • Define a matrix of typical users (admin, project manager, geomatician) and rights (read, edit, schemas, tables, views)

2. Upgrade and maintenance

• The providers want an easy way to upgrade an existing database to a new data model version e.g. 2024.0 to 2024.1. There are two modes, the first perfom the upgrade directly. The seconds allows more (PUM) options:
  • They should be able to easily create a backup (documentation and/or sample script)
  • They should be able to easily able to restore that backup (documentation and/or sample script)
  • They should be able to check, if an upgrade is possible on that backup
  • They should then run the upgrade on that backup
  • They should then run some integrity checks on the upgraded version
  • If everything goes well they should perform the same upgrade on the master database

3. TMMT Release management

• Implement workflows and tests on github
• Creation of upgrade workflow for the TMMT plugin
• Creation and testing of a pre-release for the TMMT plugin - for easy testing by selected members / developer
• Creation and testing of a Release for the TMMT plugin - for publication in QGIS plugin repository
• Traduction of packages (TMMT)
• Publication in the QGIS plugin repository

C.Import and export of Interlis files (TIT - Teksi Interlis Tool)

Determine which part is common to all modules and which part is specific to one module.

General:

• Determine which module is loaded in the QGIS project
• Get Interlis model (Git, local)
• Modelbacker generate the schema

Specific to a module:

• Transfert configuration file
• Translation from schema TXX to interlis schema

Open questions:

• Should this tool support aggregated format? How (views, queries)?:
  • SDEE mini
  • SIRE

1. INTERLIS Import of data

• The administrators should be able to easily import data provided in the corresponding INTERLIS model version for the module

  • Selection option for location of log files
  • The import process automatically analyses the MODEL of the xtf data.
  • If the MODEL is supported, the process continues, else a warning is provided to the user.
  • The import process is first checking the validity of the INTERLIS xtf file
  • New feature option (to discuss): Allow to skip checking (MANDATORY constraints, Relation constraints), if first attempt failed, with clear warning to the user, that this could cause other problems in a special "Old data migration mode"
  • If ok it then performs an import into the intermediary schema created with ModelBaker

• It then provides a wizard to make one of the following choices (general selection buttons (new feature)):

  • Update and append (Default)

  • Update (only updates all existing data in the database)

    • Append (Only append new data)

  • And two general selection buttons

    • Deselect all data in all classes
    • Select all data in all classes (Default)
    • And keep the feature to select / deselect:
    • Select/Deselect all data in one specific class (needs improvement in speed and implementation - see

    Optimize Import for large dataset wastewater#206 (comment)

• It then allows to make the final import into TEKSI with the selected options

• Post processes (like re-enabling trigger, update fk_main_cover, update fk_main_node, etc.) are applied and communicated to the user

• The user is informed whether the import has been successful

• and a link to the log files is provided

• Configuration for additional models is standardized and can be easily integrated e.g. for additional classes, extensions such as Protection_tubes, Remote control cables

• Documentation about this additional configuration is provided in the Developers Guide

2. INTERLIS Export of data

• The administrators should be able to easily export data provided in the corresponding INTERLIS models version for the module:

  • Selection of Export model
  • Option to only export selected features
  • Selection of Orientation
  • Selection options for Label Export
  • Selection option for location of log files

• Starting of Export process by the user

• If ok it then performs an import into the intermediary schema created with ModelBaker and provides a log for that step

• The export process then writes the xtf File based on the selected model and provides a log for that step

• The export process is lastly checking the validity of the INTERLIS xtf file and provides a log for that step

• The user is informed whether a valid export files has been created

• and a link to the log files is provided

• Configuration for additional models is standardized and can be easily integrated e.g. for additional classes, * extensions such as Protection_tubes, Remote control cables

• Documentation about this additional configuration is provided in the Developers Guide

3. TIT Release management

• Implement workflows and tests on github
• Creation of upgrade workflow for the TIT plugin
• Creation and testing of a pre-release for the TIT plugin - for easy testing by selected members / developer
• Creation and testing of a Release for the TIT plugin - for publication in QGIS plugin repository
• Traduction of packages (TIT)
• Publication in the QGIS plugin repository