Routine

Variables update

  1. Clone the project

The code is available on GitLab : https://gitlab.com/ifb-elixirfr/covid19/EMERGEN-DB

git clone https://gitlab.com/ifb-elixirfr/covid19/EMERGEN-DB.git
  1. Add the updated trame file to the folder static-apps/version.

Danger

It’s very important to respect the name rule. This is because the download button in the application will retrieve the last file using the file name and the version number using a regular expression. You must therefore be particularly careful with this name. The name structure is YYYY-MM-DD_ACRONYME-PF_semaineXX_emergen_typage_vY.Y.Y.xlsx whare you replace the Y.Y.Y by the version number. Fot example, when writing this documentation the last trame file had the name: YYYY-MM-DD_ACRONYME-PF_semaineXX_emergen_typage_v3.0.6.xlsx

  1. Test : The documentation for testing the code is available here: Code quality

  2. Repo update

git push origin NOM_BRANCHE

Than do a une merge request

  1. Connect to the server hosting the application

  2. Update the code

sudo git pull origin master
  1. Image update

sudo docker-compose -f docker-compose.prod.yml pull web
sudo docker-compose -f docker-compose.prod.yml up -d
sudo docker-compose -f docker-compose.prod.yml exec web python manage.py collectstatic --no-input --clear
sudo docker system prune -a -f

And ONLY in case column descrition changes:

docker-compose -f docker-compose.prod.yml exec web python manage.py load_description

Extremely rare case : the connexion to the registry is active, but this connexion may be lost. You can use the following command to reconnecter:

sudo docker login registry.gitlab.com

8. Notify users - In the admin page, in the AUTHENTIFICATION ET AUTORISATION, click on Utilisateurs. Select all the users, then in actions : Send email.

Users update

Frequency : Variable (important at the beginning of the project than decreasing)

This part is detailed here : Page d’administration

Note

As a reminder, the file must be in CSV, Semicolon separator and Unicode encoding (UTF-8)

Description update

Frequency : Rare

Uniquement lors de l’ajout de nouvelle colonne ou de la modification des descriptions.

This part is detailed here : Page d’administration

Annotation update

Fréquence : Rare

This is currently under developement for an attempt at automatic indexing.

This part is detailed here : Page d’administration

Static files

Fréquence : Rare

L’architecture de l’application impose de mettre à jour de temps en temps les fichiers statiques. Par exemple, lors de l’ajout d’un nouveau fichier trame.

Les étapes sont :

  1. Connect to your server and go to the application folder

  2. Updating the git (as static files pass through Gitlab, the aim is to to have a repo that’s always up to date)

sudo git pull origin master
  1. Static files update

sudo docker-compose -f docker-compose.prod.yml exec web python manage.py collectstatic --no-input --clear

The documentation

Fréquence : Rare

The documentation is written in sphinx and all the explanations needed to manage it are here: Gestion de la documentation

Documentation is automatically generated when the image is created in the Gitlab CI.

Shut down and restart the application

Fréquence : Rare

sudo docker-compose -f docker-compose.prod.yml down
sudo docker-compose -f docker-compose.prod.yml up -d

Extremely rare case:

sudo systemctl start docker

Compilation of the Go scripts

docker-compose -f docker-compose.prod.yml exec web sh
cd scripts
wget https://go.dev/dl/go1.17.8.linux-amd64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.17.8.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin

The « scripts » folder contains 5 folder, each containing a specific Go script.

  1. export_sequences_fasta_resultID: export all the results with a header build from the uniqueID.

cd export_sequences_fasta_resultID
go build export_sequences_fasta_resultID.go
  1. export_sequences_fasta_seqID: export all the results with a header build from the sequence id.

cd export_sequences_fasta_seqID
go export_sequences_fasta_seqID.go

3) export_sequences_tsv: export all the results in a tsv file. .. code-block:: bash

cd export_sequences_tsv go export_sequences_tsv.go

  1. export_metadata_sequences: export all the results and their sequences.

cd export_metadata_sequences
go export_metadata_sequences.go
  1. export_metadata: export all the results

cd export_metadata
go export_metadata.go

EMERGEN-DB dashboards generation on the IFB cluster

Moving into the right project

cd /shared/projects/emergen_ifb/emergen-external-reports

Creation/update of the Singularity image

singularity build rtools4emergen.sif docker://registry.gitlab.com/ifb-elixirfr/covid19/rtools4emergen:latest

Start the job with slurm manually

sbatch generate_dashboard_report.sh

where generate_dashboard_report.sh

#!/bin/bash
module load singularity
cd /shared/projects/emergen_ifb/emergen-external-reports
# singularity build rtools4emergen.sif docker://registry.gitlab.com/ifb-elixirfr/covid19/rtools4emergen:latest
singularity run rtools4emergen.sif bash -c "Rscript --vanilla generate_dashboard_report.R XXXXXXXXXXXXX"
curl -X 'POST' \
'https://emergen-db.france-bioinformatique.fr/fr/api/external/report/upload/' \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-H 'Authorization: Token XXXXXXXXXXXXX' \
-F 'category=Dashboard' \
-F 'external_report=@dashboard.html' \
-F 'public=false' \
-F 'teamNames=IFB' \
-F 'type_file=HTML' \
-F 'update=true'

XXXXXXXXXXX is your Token in EMERGEN-DB and generate_dashboard_report.R

#!/usr/bin/env Rscript
args = commandArgs(trailingOnly=TRUE)
if (length(args)==0) {
stop("At least one argument must be supplied (emergen_token).n", call.=FALSE)
}
library(Rtools4Emergen)
emergenToken=args[1]
generate_dashboard_report(emergenToken = emergenToken)

The documentation is also available: Doc Dashboard