DeBasher

Introduction

DeBasher is a software package implementing a flow-based programming extension for Bash. In the flow-based programming paradigm, networks of black box processes with well defined data exchange connections are executed. DeBasher is characterized by a strong orientation towards the definition of software components or modules.

DeBasher can be used to execute general programs. However, DeBasher is particularly appropriate for the execution of pipelines, also called workflows. Workflow execution is important in different fields. One particular example would be the field of Bioinformatics.

DeBasher runs two types of programs. A general program processes a batch of inputs: each of its processes runs once, as soon as its inputs are ready, and ends. A resident program processes a stream: its processes are long-lived Python classes with a state of their own, which exchange messages through FIFOs, react to what is written into the program from outside, and survive the crash of any of them. Programs of both types can be written by hand, as Bash modules, or built, run and observed from a browser with the DeBasher web interface.

The DeBasher software package was created and is currently maintained by Daniel Ortiz-Martínez. Daniel is an experienced machine learning researcher and also holds an MSc degree in Bioinformatics.

General Features

The package incorporates the following features:

Definition of complex programs composed of several processes.
Framework to define programs and processes using Bash shell scripting or other programming languages (Python, R, Perl and Groovy).
Concurrent execution of processes depending on detailed process dependencies.
Reexecution of only those processes that are pending or whose execution failed.
A built-in scheduler for a single machine, and integration with workload schedulers such as Slurm.
Support for Conda environments and Docker containers.
Reproducible workflows.
Generation of graphical representations of programs.
A web interface to build, run and observe programs from a browser.

Advanced Features

DeBasher also incorporates features that are not usual in other similar tools:

Implementation of interactive workflows.
Support for complex workflows that include cycles.
Ability to implement stateful processes.
Possibility to combine workflows at runtime.
Efficient execution of workflows composed of thousands of processes by resorting to static scheduling.
Resident programs, whose processes keep their state and are fault tolerant:
- a process that crashes is relaunched and returns to the state it would have had without the crash, while the others keep running;
- consistent snapshots of the state of every process, taken at any moment;
- orderly stop of a whole program, and resumption later from where it stopped;
- failures detected and reported, never silent;
- processes that observe the outside world, such as a directory where files arrive, and processes that launch a general program for each request they receive.

Web Interface

The DeBasher web interface lets you build a program without writing its module by hand: each process is a box on a canvas and each connection an edge between two of them. The web interface translates the drawing into a DeBasher module, runs it with the tools of DeBasher, colors each process by its status as the run goes on, and lets you inspect what each process left and talk to the FIFOs of a running program. It supports both general and resident programs, and it can also import a module written by hand.

Distribution Details

DeBasher has been coded using Bash shell-scripting and Python; its web interface is written in TypeScript, with React, and served by a Python server built on FastAPI. DeBasher is developed on Linux, and is tested on Linux, on macOS and on Windows through WSL2. See the "Documentation and Support" section of these instructions if you experience problems during compilation.

It is released under the GNU Lesser General Public License (LGPL), version 3.

Installation

Requirements

Bash 4.3 or newer, which runs every script of DeBasher.
Python 3.
The usual command line tools of a Linux system: those of GNU coreutils (realpath and timeout among them), grep, sed, awk and find, and flock, from util-linux.
Graphviz, whose dot command draws the graphs of programs.
The autotools, to configure the package: autoconf 2.71 or newer and automake; and make, to build it.
For the web interface, which is optional: Node.js 22.12 or newer, with npm, to build it, and a Python virtual environment for its server (see below). Without npm, configure leaves the web interface out, as it does when given --disable-frontend.

Some features need further software (see below).

Linux

On Ubuntu (or Debian), the required packages are installed with:

 $ sudo apt install autoconf automake make git graphviz python3 python3-venv

Bash, coreutils and util-linux are part of every installation. The Node.js package of the distribution is usually older than 22.12; a newer one is installed from nodejs.org or with nvm. Other distributions have packages of the same names, or close to them.

Windows

DeBasher runs inside WSL2, the Linux system of Windows, and is tested there with Ubuntu 24.04. WSL2 is installed, with an Ubuntu distribution, from a PowerShell opened as administrator:

 > wsl --install

Then, in a terminal of that Ubuntu, follow the instructions for Linux above. Keep the sources of DeBasher, your programs and their output directories in the file system of Linux (under your home directory there), not in the drives of Windows that WSL2 mounts under /mnt/c: DeBasher creates FIFOs and locks files, which those drives may not support, and reaching them from Linux is much slower. The web interface, started inside WSL2, is reached from a browser of Windows at http://localhost:8000/, since WSL2 forwards the ports of localhost to Windows.

macOS

The steps below install everything that DeBasher needs, and make the command line tools that it finds behave as those of Linux do. They are the steps with which DeBasher is tested on macOS 15, on Apple silicon. If something fails, please report it on the issue tracker.

Install the command line tools of Xcode, which provide git and make:
```
 $ xcode-select --install
```
Install Homebrew, and with it the rest of the requirements:
```
 $ brew install bash coreutils findutils gnu-sed grep gawk flock \
     autoconf automake graphviz python node
```
macOS ships Bash 3.2 and has no flock, and its sed, grep, find and the tools of coreutils are those of BSD, which differ in details from the GNU ones that DeBasher is tested with.
Put the Bash of Homebrew and the GNU tools first in the PATH, under their usual names (Homebrew installs the GNU tools with a g prefix, gsed or gtimeout, and keeps the usual names in gnubin directories), for example in ~/.zprofile:
```
BREW="$(brew --prefix)"
export PATH="$BREW/opt/coreutils/libexec/gnubin:$BREW/opt/findutils/libexec/gnubin:$BREW/opt/gnu-sed/libexec/gnubin:$BREW/opt/grep/libexec/gnubin:$BREW/bin:$PATH"
```
configure takes the first bash of the PATH, which every script of DeBasher then runs with, and refuses one older than 4.3; bash --version shows which one comes first.
Build and install the package as described below, and run make installcheck, which runs the examples of DeBasher and says whether they work.

Mirror taps (the --mirror option of define_fifo_opt) rely on how Linux and macOS treat a FIFO opened for both reading and writing. On macOS, a mirror tap whose reader is gone takes a second longer to stop.

Building and Installing

Once the requirements are available, the package is built and installed with the following steps:

Obtain the package using git:
```
$ git clone https://github.com/daormar/debasher.git
```
Or download it in a zip file
cd to the directory containing the package's source code and type ./reconf.
Type ./configure to configure the package.
Type make to compile the package.
Type make install to install the programs and any data files and documentation.
Optionally, type make installcheck to check the installation: it runs the example programs with the installed tools.
You can remove the program binaries and object files from the source code directory by typing make clean.

By default the files are installed under the /usr/local directory (or similar, depending on the OS you use); however, since Step 5 requires root privileges, another directory can be specified during Step 3 by typing:

 $ ./configure --prefix=<absolute-installation-path>

For example, if user1 wants to install the DeBasher package in the directory /home/user1/debasher, the sequence of commands to execute should be the following:

 $ ./reconf
 $ ./configure --prefix=/home/user1/debasher
 $ make
 $ make install

The installation directory can be the same directory where the DeBasher package was decompressed.

Starting the Web Interface

The server of the web interface needs the Python packages listed in share/debasher/api/requirements.txt, under the installation directory, which make install does not install. They are installed once, in a virtual environment:

 $ python3 -m venv ~/debasher-venv
 $ ~/debasher-venv/bin/pip install -r <installation-path>/share/debasher/api/requirements.txt

Then, with that environment activated, debasher_webui starts the server, and the web interface is at http://127.0.0.1:8000/:

 $ source ~/debasher-venv/bin/activate
 $ debasher_webui

The server has no authentication and runs every program as the user who started it, so by default it listens only on the local machine; to reach it from another machine, forward its port over SSH.

Third Party Software

Slurm

DeBasher can be configured to use Slurm as a workload scheduler. Slurm is particularly indicated to execute large pipelines or to execute pipelines in high performance computing environments.

Conda

DeBasher provides support for automated installation of Conda packages. Such packages are organized in environments and (optionally) used within DeBasher software modules.

Docker

DeBasher also provides support for Docker containers, favoring reproducibility of results.

Current Status

The DeBasher package is under development. Below there is a list of features that will be incorporated in a near future:

Support for additional workload schedulers.

Sample Projects Using DeBasher

Geno-DeBasher: Geno-DeBasher is a bioinformatics software project implementing DeBasher modules related to genome variant calling.

Documentation and Support

Project documentation includes:

These instructions.
Technical documentation: how to execute and implement programs, resident programs, the web interface, the tools of DeBasher and its API, with examples.
A journal paper.

DeBasher has been described in a journal article. This article incorporates a supplementary file with multiple DeBasher code examples.

If you need additional help, you can:

use the github issue tracker.
send an e-mail to the author.

Citation

If you use DeBasher in your research, please cite the following publication:

Ortiz-Martínez, D. DeBasher: a flow-based programming bash extension for the implementation of complex and interactive workflows with stateful processes. BMC Bioinformatics 26, 106 (2025). https://doi.org/10.1186/s12859-025-06108-1