THINGS TO DO AND STANDARD ERROR MESSAGES DURING DATA INTEGRATION 2024

[elfunesto]

STEPS

Before starting

copy the following markdown file, store it somewhere on your computer
https://github.com/ices-eg/wg_WGEEL/tree/master/Misc/dc_2024/00_template.md
rename it to e.g VA.md if your country is Vatican. You can open it with a text editor.
Adapt this file to your country.
In this file you will take notes of what's happening during data integration, like problems changes or failure to import. Don't forget to do it, we need this to check the data call at the end.

Get your data from accession, save all in the same folder, at the end you will put the final excel files and the md. on the sharepoint of the working group e.g. data/data_integration_part1/VA

Data integration

(1) Enter password, enter the main assessor (you) and the secondary assessor (the person who is helping you) ... this is for our log.
(2) Load data, don't make a mistake on the annex type, this is the most common error
During this load you will have a message on possible errors, fix them before reloading again. Check with support room if you have doubts.
(3) Check duplicate
(4) Get all new / duplicated / deleted files, edit qal_id if necessary before integration, save them in your working folder.
(5) Load all data, don't forget to load the data and note the messages received from the shiny during upload in your markdown document.
(6) Do a little dance before starting again with another annex.

Things that you have to correct before data integration

Annex 1-3 and 9

Symptoms :

error "Join columns in x must be present in the data. ✖ Problem with fi_id_cou."

In the heading of individual metric sheets (update, delete, new),
please replace fi_idcou by fi_id_cou (add underscore),
replace is_female(1=female,0=male) by is_female_(1=female,0=male) (add underscore),
replace anguillicola_presence(1=present,0=absent) by anguillicola_presence_(1=present,0=absent) (add underscore).

From thursday on :
replace gr_last_update with gr_lastupdate
replace fi_last_update with fi_lastupdate

For those sheets, if you want to update data ASK US FIRST, as some values may have been dropped from the template.

Annex 10 (Biomass)

In the heading of updated data replace the heading biom_per_T with biom_perc_T
Symptoms : you get a warning about column names during step0 data check

Can't subset column that don't exist. Column biom_perc_T doesn't exist

Annex 11 (Mortality rates)

The existing data has an eel_dta_code column, this is an error, it shouldn't be copied to updated or deleted data.
In sheet deleted data replace heading eel_type_name with eel_typ_name

other common errors

problem in column names :...19 & ...20 file = 0.xlsx

You have data in extra columns, select those columns and delete extra data

problem in column name biom_perc_F&biom_perc_T ...

You have loaded the biomass file while other type (e.g. catch and landings) is selected

problems when updating data : during integration step2 the data is duplicated

This might happen if you change the year, or the stage, of the habitat. This is a bad idea as there might be somewhere in the database some data with this particular year, stage and habitat.