1.2 Reproducible bioinformatics: project layout, environments and git¶

Theme: Tools & technology

Over the next four days you will run two workflows, use four Python packages, make dozens of plots and write a gene dossier. By Friday you need to be able to say exactly which data, code and software versions produced each figure. The easiest way to manage that is to set up a few habits on day one.

This module covers three of them: a standard project layout, software environments with pinned versions, and version control with git and GitHub. At the end you fork the course repository and make your first commit to your own dossier folder. You will commit to it every day.

The ideas here draw on Vince Buffalo’s Bioinformatics Data Skills [1] and Wilson et al.'s “Good enough practices in scientific computing” [2]. Both are worth reading in full.

Learning objectives¶

At the end of this module you will be able to:

• Organise a bioinformatics project so that raw data are protected and results can be regenerated.

• Explain why software environments and version pinning matter, and manage a project environment with pixi.

• Explain the git concepts repository, commit, branch, remote, fork and pull request.

• Fork the course repository, and commit and push your own work to it.

Setup¶

We only need tools that are already in Colab and Codespaces (the shell, curl, git and pandas); the environments section installs pixi if it is missing. All files are written to a course-work/ folder next to this notebook.

Why reproducibility is worth the effort¶

A result is reproducible if someone else, with your data and code, gets the same answer. That “someone” is usually you, three months later, when a reviewer asks you to change one parameter. Common ways this goes wrong:

• You cannot tell which of results_final.tsv and results_final_v2_USE_THIS.tsv went into the figure.

• A raw data file was edited by hand (or overwritten), and the original is gone.

• A package update changed a default, and the same code now gives a different answer.

• The analysis ran on a laptop that no longer exists.

None of the fixes needs special skills. Both of the sources above make the same point: aim for practices that are “good enough” and that you will actually keep up, not perfect ones.

Project layout¶

Principles¶

1. One project, one folder, one git repository. Everything needed to understand the project lives under one directory.

2. Raw data are read-only. Put downloaded or sequenced data in data/raw/, never edit it, and record where each file came from, when, and its checksum. Make the files read-only so you cannot overwrite them by accident.

3. Results can be thrown away. Everything in results/ is produced by code from the raw data. If you could not delete results/ and rebuild it, something is missing from your code.

4. Code is separate from data and results. Reusable code goes in scripts/ (or workflow/ for a Snakemake workflow). Exploratory notebooks go in notebooks/, numbered in the order they run.

5. Write things down. A README.md at the top says what the project is and how to run it. Keep a dated log of what you did and why; Buffalo suggests a README in each data directory describing where the files came from.

6. Use relative paths (data/raw/samples.tsv, not /Users/sanjay/Desktop/...) so the project works on another computer.

7. Name files for computers and people. No spaces, lower case, dates as YYYY-MM-DD so they sort correctly, zero-padded numbers (01-, 02-, ... 10-).

8. Prefer plain text (TSV, CSV, FASTA, YAML) to spreadsheets for anything a program reads.

A layout for your gene dossier¶

my-dossier/
├── README.md            # what this is, how to reproduce it
├── pixi.toml, pixi.lock  # the software you used (Module 1.2, below)
├── data/
│   ├── README.md        # where each raw file came from, when, checksums
│   └── raw/             # read-only
├── notebooks/           # 01-busia-de.ipynb, 02-anoexpress.ipynb, ...
├── scripts/             # reusable functions and scripts
├── results/             # generated; safe to delete
└── docs/                # dossier text, figures for your talk

RNA-Seq-Pop and AmpSeeker use the standard Snakemake layout (config/, workflow/, resources/, results/), which follows the same ideas. You will see it in Module 1.4.

Practical: set up a project and protect the raw data¶

We create the layout, then download a small real file: the ENA run table for the Busia RNA-seq project (PRJNA748581). It lists every sequencing run with its FASTQ checksums.

Now record where the file came from, with its checksum, and make it read-only.

The run table itself shows why checksums matter. ENA gives an MD5 for every FASTQ file, so after downloading ~113 GB of reads you can check that nothing was corrupted on the way.

Software environments¶

Why environments?¶

Your results depend on your data, your code and the exact versions of all the software the code uses. Packages change defaults, fix bugs and drop functions. Two tools may also need incompatible versions of the same library. An environment is an isolated set of packages for one project, and an environment file is a recipe to rebuild it anywhere.

Tools¶

Conda packages come from community channels: conda-forge for general software and bioconda for bioinformatics tools [3]. pixi and conda use the same packages; they differ in how they manage environments.

Levels of pinning¶

• Unpinned (pandas): you get whatever is newest today. Not reproducible.

• Pinned (pandas=2.2.2): the packages you asked for are fixed, but their dependencies can still change.

• Locked (pixi.lock, conda-lock, uv.lock): every package, including dependencies, is fixed for each platform.

Pinning is a trade-off: pinned environments are reproducible but get old. Update on purpose, rerun your analysis, and commit the new files.

Why we recommend pixi¶

• One environment per project, in the project. No need to remember which named conda environment goes with which folder; pixi run always uses the right one.

• Locked by default. Every pixi add updates pixi.lock, so a collaborator (or you, next year) gets exactly the same versions with pixi install.

• Conda and PyPI together. Many of our tools (malariagen_data, anoexpress, anoprimer) are only on PyPI; pixi resolves them together with conda packages instead of layering pip on top.

• Tasks. Common commands are written down in pixi.toml, so pixi run build means the same thing for everyone.

The everyday commands:

pixi init my-dossier                        # create my-dossier/pixi.toml
cd my-dossier
pixi add python=3.11 pandas=2.2.2           # conda-forge packages
pixi add --pypi malariagen_data==15.9.0     # a PyPI-only package
pixi task add plots "python scripts/make_plots.py"
pixi run plots                              # run a task inside the environment
pixi shell                                  # or start a shell in the environment
pixi install                                # recreate the environment from pixi.lock

Commit pixi.toml and pixi.lock; never commit the .pixi/ folder (pixi adds it to .gitignore for you).

Example: this course’s own environment¶

The course repository has a pixi.toml at its root. It builds the course book and runs the notebooks outside Colab. Notice the comments: malariagen_data needs exactly pandas 2.2.2 and numpy 2.0, so those are pinned on the conda side too. Constraints like these are the kind of thing that silently breaks environments that are not locked.

# Environment for building the course book and running the course notebooks locally.
# Colab users do not need this: each notebook pins its own packages with %pip.
# Usage:  pixi install       # create the environment from pixi.lock
#         pixi run start     # live preview of the book
#         pixi run build     # build the HTML book
#         pixi run lab       # open the notebooks in JupyterLab

[workspace]
name = "anopheles-omics-training"
description = "Anopheles Omics: from gene expression to field assays"
channels = ["conda-forge"]
platforms = ["linux-64", "osx-64", "osx-arm64"]

[tasks]
build = "jupyter book build --html"
start = "jupyter book start"
lab = "jupyter lab"

[dependencies]
python = "3.11.*"
jupyter-book = ">=2.1,<3"
jupyterlab = ">=4,<5"
pandas = "==2.2.2"     # malariagen_data 15.9.0 requires exactly this
plotly = ">=5.24,<6"     # anoexpress needs plotly 5
numpy = "2.0.*"          # malariagen_data needs numpy >=2.0.2,<2.1
packaging = ">=24,<26"   # snakemake 9.27 needs packaging <26
graphviz = "*"        # the `dot` program, for Snakemake DAGs in module 1.3
pip = "*"

# Not on conda-forge, so installed from PyPI
[pypi-dependencies]
malariagen_data = "==15.9.0"
anoexpress = "==0.3.3"
anoprimer = "==2.0.8"
jupyterquiz = "==2.9.6.4"
snakemake = "==9.27.0"   # module 1.3

After cloning the repository, pixi install recreates the environment from pixi.lock, pixi run lab opens the notebooks, and pixi run build builds the book.

pixi.toml and environment.yml¶

You will still see conda environment.yml files everywhere, including inside Snakemake workflows. The two formats describe the same thing:

For comparison, a conda environment.yml:

name: my-dossier
channels:
  - conda-forge
  - bioconda
dependencies:
  - python=3.11
  - pandas=2.2.2
  - samtools=1.21
  - pip
  - pip:
      - malariagen_data==15.9.0

Why workflows still use conda¶

Snakemake can give each rule its own environment, defined by a small environment.yml file next to the rule, and build them for you with --use-conda (or --software-deployment-method conda). RNA-Seq-Pop and AmpSeeker do this, because their tools (kallisto, freebayes, R packages) would conflict in one environment. So in the Codespace you use pixi to install Snakemake itself, and Snakemake uses conda to build the per-rule environments. You will see this in Module 1.3.

Every Colab notebook in this course starts with pinned %pip install lines for the same reasons: Colab has no pixi or conda by default.

Practical: a pixi project¶

This installs pixi (a single program) if it is not already available, creates a pixi project inside my-project, adds two packages and a task, and runs it. Adding packages downloads Python and pandas, so it takes a minute or two.

pixi run versions used the project’s own Python and pandas, not the ones in this notebook. For comparison, here is the notebook’s own environment, recorded the pip way:

Version control with git and GitHub¶

Why?¶

Git records a history of your project. Every saved version (a commit) has a message saying what changed and why, and you can compare or return to any earlier version. That removes the need for analysis_v2_final.py. GitHub hosts git repositories online, so your work is backed up, shareable, and can be linked to a paper [4].

Concepts¶

A history with one branch might look like this:

And this is how the course repository, your fork and your codespace relate:

What not to commit¶

• Large data. GitHub rejects files over 100 MB, and git is slow with big binary files. Keep raw data in data/raw/ but list it in .gitignore; the data README says how to get it back. Share large data through ENA, SRA or Zenodo instead (Module 5.2).

• Secrets: passwords, API keys and access tokens. Once pushed, assume they are public.

• Anything that can be regenerated quickly, such as most of results/. Small final tables and figures for your dossier are fine to commit.

Practical 1: git in a practice repository¶

This runs in Colab, Codespaces or on your own computer. We turn my-project into a git repository and make two commits. The git config lines set your name for this repository only.

Practical 2: fork the course repo and start your dossier¶

This part uses your own GitHub account, so you do it in the browser and a terminal rather than in this notebook. If you do not have a GitHub account, create one at https://github.com first.

1. Fork. Go to https://github.com/sanjaynagi/anopheles-omics-training and click Fork. You now have https://github.com/<your-username>/anopheles-omics-training.

2. Open a terminal on your fork. We recommend GitHub Codespaces, which you also use for RNA-Seq-Pop in Module 1.4. On your fork, click Code → Codespaces → Create codespace on main. Codespaces is already logged in to your GitHub account, so git push works without a password.

3. Make your dossier folder and commit it. In the codespace terminal:

git remote -v                     # origin should point to YOUR fork
mkdir -p my-dossier
cp assessment/gene-dossier-template.md my-dossier/dossier.md
printf "# Gene dossier\n\nMy work for the malaria vector software course.\n" > my-dossier/README.md
git status
git add my-dossier
git commit -m "Start my gene dossier"
git push

Refresh your fork on GitHub: the my-dossier/ folder should be there.

Other options.

• github.dev: on your fork, press . to open a browser editor. You can create and edit files and commit from the Source Control panel. There is no terminal, but it is enough for editing your dossier.

• Colab: you can !git clone your fork into Colab to read it, but pushing needs a personal access token. Never type a token into a notebook cell that you save or share. For pushing, use Codespaces or github.dev.

4. Keep it going. At the end of each day, commit your dossier notes and figures with a message that says what you did (e.g. Day 2: AnoExpress results for CYP6P3). When the course repo is updated, use Sync fork on GitHub and then git pull in your codespace.

Exercises¶

Exercise 1¶

Make git ignore data/raw/ and results/ in my-project, then commit the .gitignore file. (pixi init has already created a .gitignore containing .pixi/, so add to it rather than replacing it.) Use git status before and after to check it worked.

Exercise 2¶

Using pixi, set up a project that uses Python 3.11, pandas, kallisto (from bioconda) and the anoexpress package (from PyPI), with every package pinned, and a task called kallisto-version that prints kallisto’s version. Why must bioconda be added as a channel? And why might you need to pin pandas to 2.2.2 and numpy to 2.0?

Exercise 3¶

In your own words, what is the difference between (a) a fork and a clone, and (b) a commit and a push?

Quiz¶

Check your understanding. The quiz runs in Colab and in the course book.

Summary¶

• Keep each project in one folder with a standard layout. Raw data are read-only and documented; results can be deleted and rebuilt from code.

• Environments isolate the software for each project. We recommend pixi: pixi.toml says what you asked for, pixi.lock fixes every version, and tasks record how to run things. Workflows still use conda for their per-rule environments.

• Git records the history of your project; GitHub backs it up and lets you share it. Commit small, meaningful changes with clear messages, and never commit large data or secrets.

• You have forked the course repo and started my-dossier/. Commit to it every day.

Well done!¶

Next, in Module 1.3, you see how workflow managers apply the same ideas to whole analyses.

References¶

1. Buffalo V (2015). Bioinformatics Data Skills: Reproducible and Robust Research with Open Source Tools. O’Reilly Media. ISBN 978-1-4493-6737-4. Chapters 1–2 (project organisation) and 5 (git).

2. Wilson G et al. (2017). Good enough practices in scientific computing. PLoS Computational Biology. Wilson et al. (2017)

3. Grüning B et al. (2018). Bioconda: sustainable and comprehensive software distribution for the life sciences. Nature Methods. undefined et al. (2018)

4. Perez-Riverol Y et al. (2016). Ten simple rules for taking advantage of Git and GitHub. PLoS Computational Biology. Perez-Riverol et al. (2016)

5. Noble WS (2009). A quick guide to organizing computational biology projects. PLoS Computational Biology. Noble (2009)

pixi documentation: https://pixi.sh