Usage: CLI

twb cli is a command-line app for preprocessing, repeated-event grouping, deduplication, labeling, and filtering of Twitcher CSV files.

See Installation and Reference/annotation data pages for the instructions to install the app and obtain and arrange the reference genome and annotation data.

The current dedup_cli.py (the module twb cli runs) supports three main modes:

  • Full post-processing mode — enrich raw Twitcher CSV files using the same backend pipeline as the web viewer, including configurable repeated-event handling.
  • Dedup-only mode — run the lightweight historical exact-column deduplication utility.
  • Filter-only mode — apply web-style filters, exported web filter presets, raw filter JSON, or inline CLI filters to an already processed CSV.

Full preprocessing supports three repeated-event modes: group (default), collapse, and off. These modes are separate from the legacy --dedup-only utility.

twb is registered as a console script in pyproject.toml:

[project.scripts]
twb = "backend.cli:main"

Contents

Running twb cli under different platforms

Singularity

Pull and convert the image:

singularity pull twb.sif docker://aryashta/twb:latest

Test that the image works:

ref=./relative/path/to/hg38.no_alt.fa.gz
singularity exec -B $PWD:$PWD twb.sif twb cli -r $ref -o output.csv input.csv

Tip! It may be convenient to write a wrapper for the Singularity call and thus avoid writing a long command every time. That can be done by going to the directory of the twb.sif file and executing the commands:

cat > twb-cli << EOF
#!/bin/bash
singularity exec -B \$PWD:\$PWD $(realpath twb.sif) twb cli \$*
EOF

chmod +x twb-cli

The resulting wrapper file twb-cli can now be moved to somewhere on the execution path (e.g. to /usr/local/bin/) and then called directly:

twb-cli -r $ref -o output.csv input.csv

Note that the file system mapping used in the wrapper only exposes the current working directory and its subdirectories for the TWB container. If you get “File not found” error, check that the files are real files and not links to somewhere else in the file system.

Docker

Pull and tag the image:

docker pull aryashta/twb:latest

docker tag aryashta/twb:latest twb

Test that the image works:

docker run --rm -v $PWD:$PWD -w $PWD twb \
  twb cli input.csv -o output.csv -r $ref

Tip! It may be convenient to write a wrapper for the Docker call and tus avoid writing a long command every time. That can be done by executing the commands:

cat > twb-cli << EOF
#!/bin/bash
docker run --rm -v \$PWD:\$PWD -w \$PWD aryashta/twb:latest twb cli \$*
EOF

chmod +x twb-cli

The resulting wrapper file twb-cli can now be moved to somewhere on the execution path (e.g. to /usr/local/bin/) and then called directly:

twb-cli -r $ref -o output.csv input.csv

Note that the file system mapping used in the wrapper only exposes the current working directory and its subdirectories for the TWB container. If you get “File not found” error, check that the files are real files and not links to somewhere else in the file system.

Local / pip install

# if using virtual environment, activate it first
source twb_env/bin/activate  

twb cli --help

Available command-line parameters

The current dedup_cli.py includes the following short options:

-h                   --help
-o                   --output
-r                   --reference / --ref
-g                   --gff
-m                   --mask
-O                   --filter-only
-d                   --dedup-only
                     --duplicate-mode {group,collapse,off}
-n                   --no-dedup (deprecated alias for --duplicate-mode off)
-k                   --key
-K                   --keep
-R                   --report-json
                     --report-only
-I                   --keep-intermediate
-p                   --filter-preset-json
-P                   --filter-preset
-J                   --filter-json
-f                   --filter
-A                   --filter-mode
-x                   --missing-filter-column
-L                   --list-columns
-q                   --query
                     --query-json

Note: the current CLI has short aliases for common arguments, but it does not currently define convenience shortcut filters such as --likely-tsm or --min-final-score. Use -f/--filter for column-specific filtering.

CLI: inspect available columns

Use -L / --list-columns to print the CSV header and exit. This mode does not require -o:

twb cli output.processed.csv -L

This is useful before writing filters, because processed CSVs can contain many annotation columns.

CLI: full post-processing mode

Use full post-processing when you want the enriched CSV that the browser produces after upload.

twb cli input.csv \
  -o output.processed.csv \
  -r ref/genomes/hg38/hg38.no_alt.fa.gz \
  -g ref/annotations/hg38/Homo_sapiens.GRCh38.115.ucsc.sorted.gff3.gz \
  -m ref/masks/hg38/hg38.no_alt.masking.v2.sorted.gff3.gz \
  -R postprocess_report.json

Required for full post-processing:

-r, --reference, --ref

Optional:

-g, --gff
-m, --mask
-R, --report-json
-I, --keep-intermediate

If --gff is omitted, gene-context annotation is skipped. If --mask is omitted, repeat-mask annotation is skipped.

Repeated-event handling in full preprocessing

Use --duplicate-mode to control how observations with the same repeated-event signature are handled. The grouping signature uses the same cluster_region, normalized inner CIGAR, and 10 bp of expanded CIGAR context on both sides of the template-switch block.

Group and retain all observations, default

twb cli input.csv \
  -o output.grouped.csv \
  -r ref/genomes/hg38/hg38.no_alt.fa.gz \
  --duplicate-mode group

group preserves every observation, adds repeated-event metadata, and marks one representative observation per group. This mode is recommended for trios, families, and multi-sample datasets.

Collapse repeated groups

twb cli input.csv \
  -o output.collapsed.csv \
  -r ref/genomes/hg38/hg38.no_alt.fa.gz \
  --duplicate-mode collapse

collapse keeps one representative observation per repeated-event group. This is the legacy-style behavior for users who want one output row per grouped event.

Disable repeated-event grouping

twb cli input.csv \
  -o output.ungrouped.csv \
  -r ref/genomes/hg38/hg38.no_alt.fa.gz \
  --duplicate-mode off

off preserves and processes every row independently. -n/--no-dedup remains available as a deprecated alias for --duplicate-mode off.

The legacy -d/--dedup-only command is different: it performs lightweight exact-column deduplication and does not run the full annotation pipeline or the new CIGAR-context grouping logic.

Full post-processing followed by filtering

The current CLI can apply filters after full preprocessing by adding filter arguments to the full preprocessing command. The processed output is copied to the requested output path and then filtered in place:

twb cli input.csv \
  -o output.processed.likely.csv \
  -r ref/genomes/hg38/hg38.no_alt.fa.gz \
  -g ref/annotations/hg38/Homo_sapiens.GRCh38.115.ucsc.sorted.gff3.gz \
  -m ref/masks/hg38/hg38.no_alt.masking.v2.sorted.gff3.gz \
  -f tsm_confidence_label:eq:"Likely TSM" \
  -f tsm_final_score:gte:6 \
  -R output.processed.likely.report.json

For reproducibility, a two-step workflow is often clearer:

# 1. Produce an enriched processed CSV
twb cli input.csv \
  -o output.processed.csv \
  -r ref/genomes/hg38/hg38.no_alt.fa.gz \
  -g ref/annotations/hg38/Homo_sapiens.GRCh38.115.ucsc.sorted.gff3.gz \
  -m ref/masks/hg38/hg38.no_alt.masking.v2.sorted.gff3.gz

# 2. Apply filters to the processed CSV
twb cli output.processed.csv \
  -O \
  -f tsm_confidence_label:eq:"Likely TSM" \
  -o output.processed.filtered.csv \
  -R output.processed.filtered.report.json

CLI: label sample_name values

Twitcher output uses sample_name as the per-row sample identifier. The sample name inferred from the input data may not be ideal for the downstream analyses, and twb cli allows modifying it.

Non-interactive inline labeling

twb cli output.processed.csv \
  -O \
  --sample-label 'NA12878=Mother' \
  --sample-label 'NA12891=Father' \
  --sample-label 'NA12892=Child' \
  -o output.labeled.csv

Non-interactive JSON labeling

twb cli output.processed.csv \
  -O \
  --sample-label-json trio-labels.json \
  -o output.labeled.csv

The JSON file maps exact sample_name values to display labels:

{
  "NA12878": "Mother",
  "NA12891": "Father",
  "NA12892": "Child"
}

Interactive labeling

twb cli output.processed.csv \
  -O \
  --sample-label-interactive \
  --sample-label-output-json trio-labels.json \
  -o output.labeled.csv

Handling unmapped sample_name values

sample_label is always populated: full post-processing (process_csv_job, used by both the web viewer and this CLI’s full preprocessing mode) automatically defaults sample_label to the raw sample_name for any row without an explicit label. --unmapped-sample-label controls what --sample-label/--sample-label-json/--sample-label-interactive do with identifiers not present in the supplied mapping, on top of that baseline:

sample-name  Copy the original sample_name into sample_label. This is the default.
empty        Leave sample_label blank (opt back into the old behavior).
error        Stop if any non-empty sample_name has no label.

Mapping precedence for sample labels

  1. Existing non-empty sample_label values in the input CSV.
  2. Values loaded from --sample-label-json.
  3. Repeated --sample-label arguments.
  4. Changes made by --sample-label-interactive.

Later sources override earlier sources for the same exact sample_name.

Renaming a sample label permanently (web UI)

The web UI’s rename button (pencil icon on a sample badge) calls POST /rename_identity_label, which rewrites sample_label for every matching row directly in the processed CSV on disk, atomically. The rename therefore survives a reload and is reflected in exports — it is not just a browser-local override. The raw sample_name column is never modified by a rename, and no repeated-event regrouping is needed or triggered: duplicate_group_id never depends on the identity or label columns (see ADR 0001, ADR 0002).

Filter observations by sample_label

After labels are assigned, sample_label can be used like any other column:

twb cli output.processed.csv \
  -O \
  --sample-label-json trio-labels.json \
  -f 'sample_label:eq:Child' \
  -o child-events.csv

CLI: repeated-event group filters

Observation filters inspect one row at a time. Group filters inspect all observations belonging to the same duplicate_group_id. Group filtering is intended for trio, family, and multi-sample questions such as:

  • present in Child, Mother, and Father;
  • present in Child and Mother but absent from Father;
  • present in any parent;
  • present in exactly a selected set of labeled samples.

Group presence and absence are evaluated before observation-level filtering, using the complete retained group. This prevents a normal row filter from making an existing Father observation appear absent.

Group filtering requires output created with:

--duplicate-mode group

Do not use pedigree group filters on collapse-mode output because non-representative observations have already been removed.

Group filter operators

Supported group-filter operators are:

group_has_all   Every selected value must occur somewhere in the group.
group_has_any   At least one selected value must occur somewhere in the group.
group_has_none  None of the selected values may occur in the group.
group_exact     The observed value set must exactly equal the selected set.

The syntax is:

--group-filter 'COLUMN:OPERATOR:VALUE1,VALUE2'

Present in Child, Mother, and Father

twb cli output.processed.csv \
  -O \
  --read-label-json trio-labels.json \
  --group-filter 'read_label:group_has_all:Child,Mother,Father' \
  -o complete-trio-events.csv

Present in Child and Mother, but absent from Father

Separate group filters are combined with AND by default:

twb cli output.processed.csv \
  -O \
  --read-label-json trio-labels.json \
  --group-filter 'read_label:group_has_all:Child,Mother' \
  --group-filter 'read_label:group_has_none:Father' \
  --group-filter-mode all \
  -o child-mother-not-father.csv

Present in either parent

twb cli output.processed.csv \
  -O \
  --read-label-json trio-labels.json \
  --group-filter 'read_label:group_has_any:Mother,Father' \
  -o present-in-parent.csv

Exact labeled composition

This matches groups containing Child and Mother, but excludes groups that also contain Father or another label:

twb cli output.processed.csv \
  -O \
  --read-label-json trio-labels.json \
  --group-filter 'read_label:group_exact:Child,Mother' \
  -o exact-child-mother.csv

Combine group and observation filters

The following first selects groups present in the complete trio, then retains only Child observations classified as Likely TSM:

twb cli output.processed.csv \
  -O \
  --read-label-json trio-labels.json \
  --group-filter 'read_label:group_has_all:Child,Mother,Father' \
  -A all \
  -f 'read_label:eq:Child' \
  -f 'tsm_confidence_label:eq:Likely TSM' \
  -o trio-child-likely-tsm.csv

Combine separate group filters with AND or OR

Use:

--group-filter-mode all

to require every separate group filter, logical AND. This is the default.

Use:

--group-filter-mode any

to accept a group when at least one separate group filter matches, logical OR.

Group filters from JSON

Use --group-filter-json with a raw JSON list:

[
  {
    "col": "read_label",
    "op": "group_has_all",
    "val": ["Child", "Mother"]
  },
  {
    "col": "read_label",
    "op": "group_has_none",
    "val": ["Father"]
  }
]

Run:

twb cli output.processed.csv \
  -O \
  --read-label-json trio-labels.json \
  --group-filter-json pedigree-filters.json \
  --group-filter-mode all \
  -o child-mother-not-father.csv

CLI: filter-only mode

Use -O / --filter-only when you already have a processed CSV and only want to apply filters.

Filter-only mode does not run:

  • preprocessing
  • deduplication
  • complexity annotation
  • TSM scoring
  • gene annotation
  • repeat-mask annotation

Filter-only mode requires at least one filter source:

-f / --filter
-J / --filter-json
-p / --filter-preset-json

Inline filters with -f/--filter

The most direct syntax is:

-f COL:OP:VALUE

The option can be repeated.

twb cli output.processed.csv \
  -O \
  -f tsm_confidence_label:eq:"Likely TSM" \
  -o likely_tsm.csv

Multiple filters are combined with AND by default:

twb cli output.processed.csv \
  -O \
  -f tsm_confidence_label:eq:"Likely TSM" \
  -f tsm_final_score:gte:6 \
  -f tsm_evidence_score:gte:8 \
  -o high_confidence_tsm.csv

Use -A any / --filter-mode any to combine filters with OR:

twb cli output.processed.csv \
  -O \
  -A any \
  -f is_microsatellite:eq:1 \
  -f gene_context:eq:exonic \
  -o microsatellite_or_exonic.csv

For empty and notempty, the value can be omitted:

twb cli output.processed.csv \
  -O \
  -f gene_names:notempty \
  -o gene_overlapping_events.csv

Common inline filter examples

Likely TSM events:

twb cli output.processed.csv -O \
  -f tsm_confidence_label:eq:"Likely TSM" \
  -o likely_tsm.csv

Repeat-associated TSM-like events:

twb cli output.processed.csv -O \
  -f tsm_confidence_label:eq:"Repeat-associated TSM-like" \
  -o repeat_associated_tsm_like.csv

High final score:

twb cli output.processed.csv -O \
  -f tsm_final_score:gte:6 \
  -o min_final_score_6.csv

Microsatellite-associated events:

twb cli output.processed.csv -O \
  -f is_microsatellite:eq:1 \
  -o microsatellite_events.csv

Insertion-containing events, using the quantitative inserted-length column:

twb cli output.processed.csv -O \
  -f tsm_inserted_len:gt:0 \
  -o insertion_events.csv

Events with indel footprint:

twb cli output.processed.csv -O \
  -f tsm_indel_len:gt:0 \
  -o indel_events.csv

Palindrome-capable events:

twb cli output.processed.csv -O \
  -f tsm_palindrome_capable:eq:1 \
  -o palindrome_capable_events.csv

Events overlapping at least one of several pipe-separated gene names, OR within one filter:

twb cli output.processed.csv -O \
  -f gene_names:has_any:LINC00681,LINC03019 \
  -o selected_gene_events.csv

Events whose same gene_names cell contains all selected names, AND within one filter:

twb cli output.processed.csv -O \
  -f gene_names:has_all:BRCA1,TP53 \
  -o brca1_and_tp53_events.csv

For has_all, all requested values must occur in the same observation row. Values split across different observations do not satisfy the filter.

Events with selected repeat text in repeat_types_raw:

twb cli output.processed.csv -O \
  -f repeat_types_raw:contains:LINE \
  -o line_repeat_events.csv

Four-point model examples:

# Inverted repeat class
twb cli output.processed.csv -O \
  -f tsm_four_point_class:contains:"Inverted repeat" \
  -o inverted_repeat_events.csv

# Observed four-point groups only
twb cli output.processed.csv -O \
  -f tsm_four_point_observed:eq:1 \
  -o observed_four_point_events.csv

# Group B novel four-point events
twb cli output.processed.csv -O \
  -f tsm_four_point_group:eq:B \
  -o group_b_four_point_events.csv

Supported filter operators

Filter objects use the same frontend-style shape used by the web viewer:

{"col": "column_name", "op": "eq", "val": "value"}

Supported operators:

eq
neq
contains
not_contains
starts_with
ends_with
gt
lt
gte
lte
between
empty
notempty
in
notin
has_any
has_all
has_none
region_overlaps
region_within
region_contains
region_distance_lt
cigar_has_insertion
cigar_has_deletion
cigar_has_mismatch
cigar_insertion_count_gt
cigar_deletion_count_gt
cigar_mismatch_count_gt

Operator notes:

  • contains and not_contains perform case-insensitive substring matching (present/absent).
  • starts_with and ends_with perform case-insensitive prefix/suffix matching.
  • eq and neq are numeric-aware, so a filter value of 1 can match CSV values such as 1 and 1.0.
  • gt, lt, gte, and lte convert the column to numeric values.
  • between is an inclusive numeric range check: through -f, write LOW,HIGH, e.g. 5,10. Through --query/--query-json, use a two-element list: [5, 10].
  • in and notin accept comma-separated values when used through -f.
  • has_any, has_all, and has_none are intended for pipe-separated fields such as gene_names or gene_feature_types; through -f, provide comma-separated wanted values.
  • has_any:A,B matches a cell containing at least one selected value, equivalent to A OR B within that filter.
  • has_all:A,B matches only when the same cell contains every selected value, equivalent to A AND B within that filter.
  • has_none:A,B matches when the cell contains none of the selected values.
  • region_overlaps, region_within, and region_contains operate on genomic-region columns such as cluster_region or ts_1_4_region (format chrom:start-end); a row’s region must be on the same chromosome as the filter value to match.
  • region_distance_lt also operates on genomic-region columns, but takes a compound value: through -f, write REGION,DISTANCE (region, then a comma, then the maximum distance in bp), e.g. chr1:1000-2000,500. Through --query/--query-json, use {"region": "chr1:1000-2000", "distance": 500} instead.
  • cigar_has_insertion, cigar_has_deletion, and cigar_has_mismatch operate on CIGAR columns such as ts_cigar_ctx and take no value, like empty/notempty: col:cigar_has_insertion. They match when the CIGAR string contains at least one I, D, or X operation respectively.
  • cigar_insertion_count_gt, cigar_deletion_count_gt, and cigar_mismatch_count_gt also operate on CIGAR columns, and take a single numeric threshold, e.g. col:cigar_insertion_count_gt:2. They match when the total summed length of that CIGAR operation exceeds the threshold.

Region operator examples

Events whose ts_1_4_region overlaps a region of interest:

twb cli output.processed.csv -O \
  -f ts_1_4_region:region_overlaps:chr1:100000-200000 \
  -o overlapping_events.csv

Events entirely within a region:

twb cli output.processed.csv -O \
  -f ts_1_4_region:region_within:chr1:100000-200000 \
  -o within_region_events.csv

Events whose region fully contains a smaller region of interest:

twb cli output.processed.csv -O \
  -f ts_1_4_region:region_contains:chr1:150000-150100 \
  -o containing_events.csv

Events within 500 bp of a region of interest:

twb cli output.processed.csv -O \
  -f ts_1_4_region:region_distance_lt:chr1:150000-150100,500 \
  -o nearby_events.csv

Events excluding an exact region (neq) and excluding a chromosome (not_contains):

twb cli output.processed.csv -O \
  -f ts_1_4_region:neq:chr1:100000-200000 \
  -f ts_1_4_region:not_contains:chrY \
  -o filtered_events.csv

Numeric-range and CIGAR operator examples

Events with a final score in an inclusive range (between):

twb cli output.processed.csv -O \
  -f tsm_final_score:between:5,10 \
  -o mid_confidence_events.csv

Events whose alignment CIGAR contains at least one insertion:

twb cli output.processed.csv -O \
  -f ts_cigar_ctx:cigar_has_insertion \
  -o insertion_cigar_events.csv

Events with more than 2 bp of deletion in the CIGAR:

twb cli output.processed.csv -O \
  -f ts_cigar_ctx:cigar_deletion_count_gt:2 \
  -o large_deletion_events.csv

Events with an insertion in the CIGAR, combined with a score range:

twb cli output.processed.csv -O \
  -A all \
  -f ts_cigar_ctx:cigar_has_insertion \
  -f tsm_final_score:between:5,10 \
  -o insertion_mid_confidence_events.csv

-f’s flat COL:OP:VALUE syntax has no negation, so there is no built-in “does not have a mismatch” form for cigar_has_mismatch. To exclude mismatch-containing events instead, use --query/--query-json with a negated group around cigar_has_mismatch (see “CLI: advanced query filtering” below).

Filter combination mode: AND versus OR

-A/--filter-mode controls how separate filter expressions are combined.

Default mode is all, meaning every separate filter must match, logical AND:

twb cli output.processed.csv -O \
  -A all \
  -f tsm_final_score:gte:6 \
  -f tsm_inserted_len:gt:0 \
  -o high_score_insertions.csv

Use any when at least one separate filter may match, logical OR:

twb cli output.processed.csv -O \
  -A any \
  -f is_microsatellite:eq:1 \
  -f repeat_overlap:eq:1 \
  -o microsatellite_or_repeat_overlap.csv

This is separate from multi-value behavior inside one filter:

  • in:A,B and has_any:A,B use OR among their values.
  • has_all:A,B uses AND among its values and requires both values in the same cell.
  • --filter-mode all or --filter-mode any combines separate -f filters.

For example, the following requires one observation to contain both BRCA1 and TP53, and also requires an exonic gene context:

twb cli output.processed.csv -O \
  -A all \
  -f gene_names:has_all:BRCA1,TP53 \
  -f gene_context:eq:exonic \
  -o exonic_brca1_tp53_events.csv

Missing filter columns

By default, missing filter columns are skipped and recorded in the report:

twb cli output.processed.csv -O \
  -f missing_column:eq:1 \
  -o filtered.csv \
  -R filtered.report.json

Use -x error / --missing-filter-column error to fail instead:

twb cli output.processed.csv -O \
  -x error \
  -f missing_column:eq:1 \
  -o filtered.csv

Filter-only with exported web preset JSON

The web UI can export filter presets to JSON. Use -p / --filter-preset-json to load an exported preset file, and -P / --filter-preset to choose the preset by name or id:

twb cli output.processed.csv \
  -O \
  -p twitcher-filter-presets.json \
  -P "Perfect 30bp identity" \
  -o output.processed.filtered.csv \
  -R output.processed.filtered.report.json

If the preset JSON contains exactly one preset, -P/--filter-preset can be omitted:

twb cli output.processed.csv \
  -O \
  -p one-preset.json \
  -o output.processed.filtered.csv

If the preset JSON contains multiple presets, -P/--filter-preset is required.

Filter-only with raw filter JSON

Use -J / --filter-json to provide a raw list of filters without the preset wrapper.

Example filters.json:

[
  {
    "col": "identity_30bp_upstream",
    "op": "eq",
    "val": "1"
  },
  {
    "col": "identity_30bp_downstream",
    "op": "eq",
    "val": "1"
  }
]

Run:

twb cli output.processed.csv \
  -O \
  -J filters.json \
  -o output.processed.filtered.csv \
  -R output.processed.filtered.report.json

Combining preset/JSON filters with inline filters

The CLI appends inline -f filters to filters loaded from -p/--filter-preset-json or -J/--filter-json:

twb cli output.processed.csv \
  -O \
  -p twitcher-filter-presets.json \
  -P "Perfect 30bp identity" \
  -f tsm_final_score:gte:6 \
  -o perfect_identity_score6.csv

CLI: advanced query filtering

Use --query or --query-json when flat -f filters are not expressive enough. The query engine accepts a JSON AST with nested AND/OR groups and negation, using the same structure as the web viewer’s advanced filter.

This is useful for conditions such as:

  • (score ≥ 6 AND label = Likely TSM) OR gene_context = exonic
  • NOT (is_microsatellite = 1 AND repeat_overlap = 1)

Query AST format

{
  "type": "group",
  "combinator": "and",
  "negated": false,
  "children": [
    {
      "type": "rule",
      "column": "tsm_confidence_label",
      "operator": "eq",
      "value": "Likely TSM"
    },
    {
      "type": "rule",
      "column": "tsm_final_score",
      "operator": "gte",
      "value": "6"
    }
  ]
}

Supported combinators: and, or. Set "negated": true on any group to invert it. Supported operators inside rules: eq, neq, contains, not_contains, starts_with, ends_with, gt, lt, gte, lte, empty, notempty, in, notin, has_any, has_all, has_none, region_overlaps, region_within, region_contains, region_distance_lt.

The region operators apply to genomic-region columns such as cluster_region or ts_1_4_region (format chrom:start-end). region_distance_lt takes a compound value, {"region": "chrom:start-end", "distance": <bp>}:

{
  "type": "rule",
  "column": "ts_1_4_region",
  "operator": "region_distance_lt",
  "value": {"region": "chr1:150000-150100", "distance": 500}
}

Inline query from the CLI

Pass a JSON string directly with --query:

twb cli output.processed.csv \
  -O \
  --query '{"type":"group","combinator":"and","negated":false,"children":[{"type":"rule","column":"tsm_confidence_label","operator":"eq","value":"Likely TSM"},{"type":"rule","column":"tsm_final_score","operator":"gte","value":"6"}]}' \
  -o likely_tsm_score6.csv

Query from a file

Save the AST to a JSON file and use --query-json:

twb cli output.processed.csv \
  -O \
  --query-json my_query.json \
  -o filtered.csv \
  -R filtered.report.json

Nested OR group example

Keep rows where either the label is Likely TSM, or the gene context is exonic:

{
  "type": "group",
  "combinator": "or",
  "negated": false,
  "children": [
    {
      "type": "rule",
      "column": "tsm_confidence_label",
      "operator": "eq",
      "value": "Likely TSM"
    },
    {
      "type": "rule",
      "column": "gene_context",
      "operator": "eq",
      "value": "exonic"
    }
  ]
}

Combining query filtering with observation filters

Query filters are applied first, then observation -f filters. Use --query-json for the nested part and -f for flat additions:

twb cli output.processed.csv \
  -O \
  --query-json nested_query.json \
  -f gene_names:notempty \
  -o filtered_with_genes.csv

Note: --query and --query-json are mutually exclusive. Use one or the other.

CLI: filter reports

When -R/--report-json is provided, reports include a filtering section:

{
  "mode": "filter_only",
  "input_csv": "output.processed.csv",
  "output_csv": "output.processed.filtered.csv",
  "filtering": {
    "preset_id": "preset_1780492344850_ybn021t5lg",
    "preset_name": "Perfect 30bp identity",
    "num_filters": 2,
    "filter_mode": "all",
    "missing_filter_column_policy": "skip",
    "filters_applied": 2,
    "rows_before_filter": 24454,
    "rows_after_filter": 4321,
    "rows_removed_by_filter": 20133,
    "missing_filter_columns": [],
    "filter_steps": [
      {
        "filter": {
          "col": "identity_30bp_upstream",
          "op": "eq",
          "val": "1"
        },
        "status": "applied",
        "rows_before_this_filter": 24454,
        "rows_matching_this_filter_alone": 18000,
        "rows_after_this_filter": 18000
      }
    ]
  }
}

filter_steps is useful for debugging filters or presets that unexpectedly remove too many rows.

CLI: dedup-only mode

Use -d / --dedup-only when you only want the old lightweight deduplication behavior.

twb cli input.csv \
  -o output.dedup.csv \
  -d \
  -R dedup_report.json

By default, dedup-only mode uses:

cluster_region + ts_cigar_ctx

If ts_cigar_ctx is missing, the CLI falls back to ts_cigar.

Override the deduplication key:

twb cli input.csv \
  -o output.dedup.csv \
  -d \
  -k cluster_region,ts_cigar_ctx \
  -K first

Available -K/--keep values:

first
last

Dedup-only output can also be filtered in the same command:

twb cli input.csv \
  -o output.dedup.filtered.csv \
  -d \
  -f is_microsatellite:eq:1

CLI: report-only mode

Report-only mode applies to deduplication reports:

twb cli input.csv \
  -o unused.csv \
  -d \
  --report-only \
  -R dedup_report.json

CLI: efficiency tips

List columns before writing filters. On any CSV you haven’t seen before, run -L first. It prints the header and exits without requiring -o:

twb cli output.processed.csv -L

Preprocess once, filter many times. Full preprocessing (FASTA alignment, TSM scoring, gene annotation) is slow. Once you have an enriched CSV, each -O filter-only run is fast. The two-step pattern is always clearer and more reproducible than combining preprocessing and filtering in one command.

Always pass -R report.json. The filter_steps array in the report shows the row count before and after each individual filter. When a filter removes more rows than expected, filter_steps tells you exactly which one is responsible.

Save interactive labels for reuse. After --read-label-interactive, add --read-label-output-json trio-labels.json. The saved JSON can be replayed with --read-label-json in any future run without re-entering labels manually.

Extend presets with inline -f filters. Inline -f arguments are appended to filters loaded from -p / --filter-preset-json. You can tighten an exported web preset with one or two extra CLI filters without editing the JSON file:

twb cli output.processed.csv \
  -O \
  -p my-preset.json \
  -P "Perfect 30bp identity" \
  -f tsm_final_score:gte:6 \
  -o strict_filtered.csv

Keep --duplicate-mode group if you plan to use pedigree filters later. The collapse mode removes non-representative observations, which breaks --group-filter logic. Process with group (the default), then use -O --group-filter in a separate filter-only step.

Use --query/--query-json for nested logic. Flat -f filters can only combine with a single AND or OR. For nested conditions such as (A AND B) OR C, use the query AST. See the advanced query filtering section above.

Avoid -x error / --missing-filter-column error in exploratory runs. The default skip behaviour mirrors the web viewer and lets you reuse preset JSON files on CSVs that may be missing optional columns. Switch to error only in production pipelines where a missing column should be a hard failure.