Routine¶
Variables update¶
Clone the project
The code is available on GitLab : https://gitlab.com/ifb-elixirfr/covid19/EMERGEN-DB
git clone https://gitlab.com/ifb-elixirfr/covid19/EMERGEN-DB.git
Add the updated trame file to the folder
static-apps/version
.
Danger
It’s very important to respect the name rule. This is because the download
button in the application will retrieve the last file using the file name
and the version number using a regular expression. You must therefore be
particularly careful with this name. The name structure is
YYYY-MM-DD_ACRONYME-PF_semaineXX_emergen_typage_vY.Y.Y.xlsx
whare you replace the Y.Y.Y by the version number. Fot example, when writing
this documentation the last trame file had the name:
YYYY-MM-DD_ACRONYME-PF_semaineXX_emergen_typage_v3.0.6.xlsx
Test : The documentation for testing the code is available here: Code quality
Repo update
git push origin NOM_BRANCHE
Than do a une merge request
Connect to the server hosting the application
Update the code
sudo git pull origin master
Image update
sudo docker-compose -f docker-compose.prod.yml pull web
sudo docker-compose -f docker-compose.prod.yml up -d
sudo docker-compose -f docker-compose.prod.yml exec web python manage.py collectstatic --no-input --clear
sudo docker system prune -a -f
And ONLY in case column descrition changes:
docker-compose -f docker-compose.prod.yml exec web python manage.py load_description
Extremely rare case : the connexion to the registry is active, but this connexion may be lost. You can use the following command to reconnecter:
sudo docker login registry.gitlab.com
8. Notify users - In the admin page, in the AUTHENTIFICATION ET AUTORISATION
, click on Utilisateurs
.
Select all the users, then in actions : Send email
.
Users update¶
Frequency : Variable (important at the beginning of the project than decreasing)
This part is detailed here : Page d’administration
Note
As a reminder, the file must be in CSV, Semicolon separator and Unicode encoding (UTF-8)
Description update¶
Frequency : Rare
Uniquement lors de l’ajout de nouvelle colonne ou de la modification des descriptions.
This part is detailed here : Page d’administration
Annotation update¶
Fréquence : Rare
This is currently under developement for an attempt at automatic indexing.
This part is detailed here : Page d’administration
Static files¶
Fréquence : Rare
L’architecture de l’application impose de mettre à jour de temps en temps les fichiers statiques. Par exemple, lors de l’ajout d’un nouveau fichier trame.
Les étapes sont :
Connect to your server and go to the application folder
Updating the git (as static files pass through Gitlab, the aim is to to have a repo that’s always up to date)
sudo git pull origin master
Static files update
sudo docker-compose -f docker-compose.prod.yml exec web python manage.py collectstatic --no-input --clear
The documentation¶
Fréquence : Rare
The documentation is written in sphinx and all the explanations needed to manage it are here: Gestion de la documentation
Documentation is automatically generated when the image is created in the Gitlab CI.
Shut down and restart the application¶
Fréquence : Rare
sudo docker-compose -f docker-compose.prod.yml down
sudo docker-compose -f docker-compose.prod.yml up -d
Extremely rare case:
sudo systemctl start docker
Compilation of the Go scripts¶
docker-compose -f docker-compose.prod.yml exec web sh
cd scripts
wget https://go.dev/dl/go1.17.8.linux-amd64.tar.gz
rm -rf /usr/local/go && tar -C /usr/local -xzf go1.17.8.linux-amd64.tar.gz
export PATH=$PATH:/usr/local/go/bin
The « scripts » folder contains 5 folder, each containing a specific Go script.
export_sequences_fasta_resultID: export all the results with a header build from the uniqueID.
cd export_sequences_fasta_resultID
go build export_sequences_fasta_resultID.go
export_sequences_fasta_seqID: export all the results with a header build from the sequence id.
cd export_sequences_fasta_seqID
go export_sequences_fasta_seqID.go
3) export_sequences_tsv: export all the results in a tsv file. .. code-block:: bash
cd export_sequences_tsv go export_sequences_tsv.go
export_metadata_sequences: export all the results and their sequences.
cd export_metadata_sequences
go export_metadata_sequences.go
export_metadata: export all the results
cd export_metadata
go export_metadata.go
EMERGEN-DB dashboards generation on the IFB cluster¶
Moving into the right project
cd /shared/projects/emergen_ifb/emergen-external-reports
Creation/update of the Singularity image
singularity build rtools4emergen.sif docker://registry.gitlab.com/ifb-elixirfr/covid19/rtools4emergen:latest
Start the job with slurm manually
sbatch generate_dashboard_report.sh
where generate_dashboard_report.sh
#!/bin/bash
module load singularity
cd /shared/projects/emergen_ifb/emergen-external-reports
# singularity build rtools4emergen.sif docker://registry.gitlab.com/ifb-elixirfr/covid19/rtools4emergen:latest
singularity run rtools4emergen.sif bash -c "Rscript --vanilla generate_dashboard_report.R XXXXXXXXXXXXX"
curl -X 'POST' \
'https://emergen-db.france-bioinformatique.fr/fr/api/external/report/upload/' \
-H 'accept: application/json' \
-H 'Content-Type: multipart/form-data' \
-H 'Authorization: Token XXXXXXXXXXXXX' \
-F 'category=Dashboard' \
-F 'external_report=@dashboard.html' \
-F 'public=false' \
-F 'teamNames=IFB' \
-F 'type_file=HTML' \
-F 'update=true'
XXXXXXXXXXX
is your Token in EMERGEN-DB and generate_dashboard_report.R
#!/usr/bin/env Rscript
args = commandArgs(trailingOnly=TRUE)
if (length(args)==0) {
stop("At least one argument must be supplied (emergen_token).n", call.=FALSE)
}
library(Rtools4Emergen)
emergenToken=args[1]
generate_dashboard_report(emergenToken = emergenToken)
The documentation is also available: Doc Dashboard