CSV columns

Two kinds of columns exist in this file:

A quick naming note before the tables: ts_ means different things in different places, because the raw Twitcher schema, the gene-annotation code, and the edit-footprint code all independently chose ts_ as a shorthand for “template switch.”

Prefix meaning	Example	Comes from
Raw Twitcher template-switch model	`ts_num`, `ts_cigar`, `ts_1_2`	Twitcher itself
Annotation of the `ts_1_4_region` field specifically	`ts1_4_gene_context`	`gene_annotation.py`
Edits inside the TS CIGAR block (outer + inner)	`ts_outer_insertion_bp`, `ts_inner_insertion_bp`, `ts_total_insertion_bp`	`edit_delta.py`

Only the middle one was renamed (ts_ → ts1_4_) in this pass, since it was genuinely ambiguous with the raw Twitcher fields. The edit_delta.py family keeps ts_outer_/ts_inner_/ts_total_ because it’s always used alongside fw_* and reads unambiguously in context.

1. Twitcher raw output columns

Column	Meaning
`id`	Event/observation identifier.
`cluster_id`	ID of the cluster this event belongs to.
`cluster_region`	Genomic region of the event cluster (`chrom:start-end`).
`ref_ctx_region`	Reference-context region used for the alignment window.
`alt_ctx_region`	Alternate-context region used for the alignment window.
`ts_1_4_region`	Genomic region spanning points 1→4 of the template-switch model.
`ts_1_4_region_maximal`	`ts_1_4_region` widened to its maximal genomic extent per switch block, using breakpoint-uncertainty shifts. Computed by the workbench, not emitted by Twitcher. Used as part of the repeated-event grouping key (see ADR 0002).
`read_label`	Human-facing label for the read. Defaults to the raw `read_id` when no explicit label has been assigned (never blank after full processing). Renaming via the web UI or `POST /rename_identity_label` permanently rewrites this column in the processed CSV; `read_id` itself is never modified. See CONTEXT.md “label column”, ADR 0003.
`read_id`	Raw read identifier from the source BAM/FASTQ. Old-version Twitcher output only.
`sample_name`	Raw sample identifier from new-version Twitcher output, taking the role of `read_id`. Detection: if `sample_name` is present, it is the identity column (wins over `read_id` if both are present). Never modified by labeling or renaming — see CONTEXT.md “identity column”.
`sample_label`	Human-facing label for the sample, analogous to `read_label`. Defaults to the raw `sample_name` when no explicit label has been assigned (never blank after full processing). Renaming via the web UI or `POST /rename_identity_label` permanently rewrites this column in the processed CSV; `sample_name` itself is never modified. See CONTEXT.md “label column”, ADR 0003.
`fw_cigar_ctx`	Forward-only (non-TS) alignment CIGAR, over the full context window.
`fw_mi_ctx`	Forward alignment’s inserted-sequence annotation (used to recover the literal inserted bases).
`ts_cigar_ctx`	Full template-switch alignment CIGAR, including `[TS...]` block(s), over the context window.
`ts_num`	Number of template-switch blocks Twitcher found for this event.
`ts_1_2`	Twitcher’s raw point-1→point-2 segment value(s) (pipe-separated if multiple switches).
`ts_2_3`	Twitcher’s raw point-2→point-3 (copied/templated) segment value(s).
`ts_2_3_max`	Twitcher’s own raw maximal (shift-widened) templated-segment length, pipe-separated per switch block like `ts_2_3`. Used as part of the repeated-event grouping key (see ADR 0002).
`ts_1_4`	Twitcher’s raw point-1→point-4 segment value(s).
`fw_cost_ctx`	Alignment cost of the forward-only model, over the context window.
`ts_cost_ctx`	Alignment cost of the template-switch model, over the context window.
`fw_cost`	Alignment cost of the forward-only model (raw, non-context).
`ts_cost`	Alignment cost of the template-switch model (raw, non-context).
`fw_cigar`	Forward-only CIGAR, non-context fallback.
`ts_cigar`	Template-switch CIGAR, non-context fallback.
`ref_cluster_offset`	Offset of the event within the reference cluster window.
`alt_cluster_offset`	Offset of the event within the alt cluster window.
`ts_start_left_shift` / `ts_start_right_shift`	Breakpoint-uncertainty shift at the start of the switch (pipe-separated per switch block). Unsigned magnitude — direction is carried by the column name (`left`/`right`), never by a sign on the value.
`ts_end_left_shift` / `ts_end_right_shift`	Breakpoint-uncertainty shift at the end of the switch. Same unsigned-magnitude convention as the start-shift columns above.
`ts_inner_alignment_cigar`	CIGAR of the alignment inside the `[TS...]` block (the templated/copied segment vs. its donor).

1a. Flanking-identity and TS-length columns (computed by `preprocess.py`)

Derived purely from ts_cigar_ctx (via identity.py’s flank_identity() / split_ts_blocks()) and ts_inner_alignment_cigar / ts_start_left_shift / ts_end_right_shift — no reference FASTA needed.

Column	Meaning
`identity_30bp_upstream` / `identity_30bp_downstream`	Strict sequence identity (0–1) of the 30 bp of aligned CIGAR immediately adjacent to the `[TS...]` block, upstream/downstream respectively. The primary “backbone” flank-quality check used by `tsm_scoring.py`.
`identity_20bp_upstream` / `identity_20bp_downstream`	Same strict-identity calculation, but over a tighter 20 bp window. A shorter-range companion to the 30 bp columns above, giving more sensitivity right at the breakpoint. (Renamed from `identity_100bp_upstream`/`identity_100bp_downstream` and the window shortened from 100 bp to 20 bp — see Changelog.)
`ts_minimum_length` / `ts_minimum_length_sum`	Per-event minimum templated-segment length in bp (min across switch blocks / sum across switch blocks), computed from `ts_inner_alignment_cigar` adjusted by the start/end breakpoint-uncertainty shifts. `None` if any shift is invalid (-1 or missing).
`ts_maximum_length` / `ts_maximum_length_sum`	Same, but the maximum (un-shifted) templated-segment length rather than the shift-adjusted minimum.

2. Complexity / microsatellite columns

Computed in the complexity block of jobs.py, using complexity.py. Unchanged by this refactor.

Column	Meaning
`is_low_complexity`	1 if the sequence context has low Shannon entropy (below `LOW_COMPLEXITY_ENTROPY1`, default 0.60).
`is_microsatellite`	1 if a tandem repeat run of at least `MS_MIN_RUN_BP` (default 12 bp) was found.
`ms_motif`	The repeating motif of the strongest microsatellite found (e.g. `"AT"`).
`ms_run_bp`	Length in bp of that microsatellite run.
`complexity_region_source`	Which region column (`cluster_region`, `ts_1_4_region`, etc.) the complexity window was taken from. Internal/hidden column.
`ref_seq_len`	Length of the reference sequence window fetched for complexity analysis. Internal/hidden column.
`entropy_1mer_norm`	Normalized single-base (1-mer) Shannon entropy of the sequence window, 0–1.
`entropy_2mer_norm`	Normalized 2-mer Shannon entropy.
`lz_ratio`	Lempel-Ziv compression ratio of the sequence window (lower = more repetitive).

3. TSM interpretation columns

Computed by tsm_four_point.py (fragment-order classification) and tsm_scoring.py (confidence scoring). This is the section this refactor focused on, per your request — see the Changelog at the bottom for exactly what changed and why.

3a. Four-point model classification (`tsm_four_point.py`, unchanged)

Column	Meaning
`tsm_four_point_order`	Canonical fragment order (e.g. `"1-4-3-2"`) describing how the four breakpoints of the template switch relate. Mirror orders are collapsed to their canonical form.
`tsm_four_point_class`	Human-readable biological label for that order, e.g. `"Inverted repeat"`, `"Two inverted repeats with inverted spacer"`.
`tsm_four_point_observed`	1 if this order has been observed in real human data per the taxonomy in `tsm_four_point.py`; 0 if it’s theoretically possible but unobserved (Group C).
`tsm_is_mirror`	1 if the raw classified order was the mirror-direction of its canonical order.
`tsm_intra_strand`	1 if every switch block in the event stayed on the same strand (no inter-strand flip).
`tsm_num_jumps`	Number of template-switch jump blocks in the event (usually 1 or 2).

3b. Confidence scoring (`tsm_scoring.py`)

Column	Meaning
`tsm_confidence_label`	Overall call: `Likely TSM`, `Possible TSM`, `Ambiguous TSM-like`, `Repeat-associated TSM-like`, `Repeat/slippage-like`, or `Unlikely TSM`.
`tsm_evidence_score`	Sum of positive evidence points accumulated during scoring.
`tsm_specificity_penalty`	Sum of penalty points (repeat confounders, poor flanking identity, etc.).
`tsm_final_score`	`tsm_evidence_score − tsm_specificity_penalty`. Primary sortable confidence number.
`tsm_mechanism_hint`	Best-guess mechanistic label, e.g. `"Inverted-copy template switch (inter-strand)"`.
`tsm_reason`	Pipe-separated list of every scoring factor that fired for this row, in plain English — the “show your work” column.
`tsm_model_improvement`	`fw_cost_ctx − ts_cost_ctx` (or the non-context `fw_cost − ts_cost` if `_ctx` costs are unavailable). The log-likelihood-style gain from using the template-switch model instead of a plain forward alignment. Positive and large = strong evidence for a real switch; negative = the TS model is worse than forward-only, which vetoes `Likely TSM`.
`tsm_copied_segment_bp`	Length in bp of the longest copied/templated segment for this event, taken from `ts_maximum_length`/`ts_minimum_length`/`ts_2_3` (whichever is available).
`tsm_copied_segment_bp_total`	Sum of copied-segment lengths across all switch blocks in a multi-switch event.
`tsm_copy_identity`	Sequence identity (0–1) between the copied segment and its donor, from `ts_inner_alignment_cigar`.
`tsm_inter_strand`	1 if any switch block in the event involves a reverse-complement (inter-strand) copy.
`tsm_jump_distance_max`	Largest absolute jump distance (bp) across the event’s switch block(s).
`tsm_repeat_confounder`	1 if the repeat-context penalty pushed the event into “confounded” territory (repeat confounder level ≥ 2).
`tsm_repeat_level`	0 (clean) to 3 (severe) repeat-confounder severity, combining microsatellite/satellite/tandem/low-complexity/RepeatMasker signals.
`tsm_repeat_flags`	Semicolon-separated list of which specific repeat signals fired (e.g. `"satellite repeat; low-complexity context"`).
`tsm_breakpoint_shift_max_bp`	Largest absolute value across the four `ts__shift` columns — how uncertain the breakpoint placement is. (Renamed from `tsm_uncertainty_length_max`; the old name didn’t say what was uncertain.)*
`tsm_palindrome_capable`	1 if the inserted sequence (from `fw_mi_ctx`) is a palindrome or has a reverse-complement seed match — a hairpin-compatible insertion. (Was computed internally for scoring but never actually written to the output row before this fix — see Changelog.)

3c. Forward-alignment / TS edit-footprint columns (`edit_delta.py`, unchanged)

These describe the raw edit content (insertions/deletions/mismatches) of the forward alignment vs. the template-switch alignment, and the delta between them. label below is one of insertion, deletion, mismatch.

Column pattern	Meaning
`fw_{label}_events` / `fw_{label}_bp`	Count / total bp of that edit type in the forward-only CIGAR (`fw_cigar_ctx`).
`ts_outer_{label}_events` / `ts_outer_{label}_bp`	Same, but in the CIGAR outside the `[TS...]` block(s) of `ts_cigar_ctx`.
`ts_inner_{label}_events` / `ts_inner_{label}_bp`	Same, but in `ts_inner_alignment_cigar` (inside the switch, vs. the donor).
`ts_total_{label}_events` / `ts_total_{label}_bp`	`ts_outer + ts_inner` — total TS-model edits of that type.
`delta_total_{label}_events` / `delta_total_{label}_bp`	`fw − ts_total` — how many fewer edits the TS model needs vs. forward-only. This is the per-type version of `tsm_model_improvement`’s intuition, expressed in raw edit counts instead of alignment cost.
`fw_edit_events_total` / `fw_edit_bp_total`	Sum across insertion+deletion+mismatch, forward-only.
`ts_outer_edit_events_total` / `ts_outer_edit_bp_total`	Sum across types, TS-outer.
`ts_inner_edit_events_total` / `ts_inner_edit_bp_total`	Sum across types, TS-inner.
`ts_total_edit_events` / `ts_total_edit_bp`	Sum across types, TS total.
`delta_total_edit_events` / `delta_total_edit_bp`	Sum across types, delta.

fw_insertion_bp / fw_deletion_bp / fw_mismatch_bp / fw_edit_bp_total are also what tsm_scoring.py uses internally for its indel-footprint scoring criteria — there is no separate tsm_-namespaced copy of these numbers anymore (see Changelog).

4. Gene context columns

Computed by gene_annotation.py, with gene_priority_score / gene_priority_label / gene_display_names computed in tsm_scoring.py from the columns below (kept here, not in the TSM section, because they summarise gene annotation rather than TSM evidence).

Three parallel families exist because gene context can be evaluated from three different genomic windows: the combined “best pick,” the cluster_region, and the ts_1_4_region.

Column	Meaning
`gene_context`	Combined pick. `exonic` / `intronic` / `intergenic` / `unknown`, prioritizing an exonic hit from either `cluster_region` or `ts_1_4_region`. This is the column most filters/sorts should use.
`gene_context_source`	Which region (`cluster_region` or `ts_1_4_region`) the combined pick came from.
`gene_names`	Pipe-separated gene symbols overlapping the combined pick’s region(s).
`gene_feature_types`	Pipe-separated GFF feature types (`gene`, `exon`, `CDS`, …) found.
`distance_to_exon`	Distance in bp to the nearest exon (0 if overlapping). From the combined pick’s primary region.
`nearest_exon`	`chrom:start-end` label of that nearest exon.
`gene_priority_score`	0 (intergenic) to 4 (CDS/protein-coding exon) clinical/functional relevance score, derived from `gene_context`/`gene_feature_types`/`distance_to_exon`. (Renamed from `tsm_gene_priority` — it’s a gene-impact summary, not TSM evidence.)
`gene_priority_label`	Human-readable version, e.g. `"CDS (protein-coding exon)"`, `"Intronic/UTR (protein-coding gene)"`, `"Intergenic"`. (Renamed from `tsm_gene_priority_label`.)
`gene_display_names`	Cleaned, deduplicated gene symbol list (Ensembl IDs and transcript-isoform names filtered out) for display. (Renamed from `tsm_gene_display`.)
`cluster_gene_context` / `cluster_overlaps_gene`* / `cluster_overlaps_exon`* / `cluster_gene_names` / `cluster_gene_feature_types` / `cluster_distance_to_exon` / `cluster_nearest_exon`	Same fields, but evaluated only against `cluster_region`.
`ts1_4_gene_context` / `ts1_4_overlaps_gene`* / `ts1_4_overlaps_exon`* / `ts1_4_gene_names` / `ts1_4_gene_feature_types` / `ts1_4_distance_to_exon` / `ts1_4_nearest_exon`	Same fields, but evaluated only against `ts_1_4_region`. (Renamed from `ts_gene_context` etc. — the `ts_` prefix collided with raw Twitcher `ts_` fields; see the note at the top of this document.)

* overlaps_gene/overlaps_exon and their cluster_/ts1_4_ variants are computed internally but dropped from the final CSV by reorder_processed_columns() — gene_context already encodes the same information more usefully (exonic implies overlaps_exon, etc.), so keeping the raw booleans around is redundant. They still exist as intermediate values if you’re calling gene_annotation.py directly.

5. Repeat context columns

Computed by repeat_analysis.py + the repeat block in jobs.py, from a repeat-mask GFF/GFF3 (e.g. RepeatMasker output). Unchanged by this refactor.

Column	Meaning
`repeat_overlap`	1 if the event overlaps any repeat-mask feature. Broadest signal.
`repeat_mask_overlap`	Same as `repeat_overlap` but computed specifically from the mask-file lookup (kept separate from `repeat_overlap` for auditing).
`repeat_locus_overlap`	1 if the specific locus (not just the general region) overlaps a repeat.
`repeat_ts_region_overlap`	1 if `ts_1_4_region` specifically overlaps a repeat.
`repeat_sequence_complexity`	Repeat-derived sequence-complexity flag (separate signal from `is_low_complexity`, which is entropy-based).
`repeat_event_sources`	Which region(s) the repeat lookup was run against.
`repeat_types_raw`	Raw repeat type/name strings from the mask file, semicolon or pipe joined.
`repeat_alu`	1 if an Alu (SINE) element overlaps.
`repeat_line`	1 if a LINE element overlaps.
`repeat_sine`	1 if a SINE element overlaps (Alu is a subtype of SINE; `repeat_alu` is broken out separately because Alu-specific confounding differs from other SINEs).
`repeat_ltr`	1 if an LTR retrotransposon overlaps.
`repeat_dna`	1 if a DNA transposon overlaps.
`repeat_satellite`	1 if a satellite repeat overlaps. Strongest slippage confounder.
`repeat_tandem`	1 if a tandem repeat element overlaps.
`repeat_low_complexity`	1 if RepeatMasker’s own low-complexity annotation overlaps (distinct from the entropy-based `is_low_complexity`).

6. Repeated-event grouping columns

Computed in jobs.py using preprocess.py’s grouping key: preferred ts_1_4_region_maximal + ts_2_3_max (falls back to exact ts_1_4_region + ts_2_3 when the maximal-extent fields are unavailable) identifies the same underlying event, observed multiple times. See ADR 0001, ADR 0002.

Column	Meaning
`duplicate_group_id`	Stable hash ID (`dgrp_...`) for the repeated-event group this row belongs to.
`duplicate_group_count`	Total number of observations (rows) in this group.
`duplicate_group_rank`	This row’s 1-indexed position within its group (order of appearance).
`duplicate_group_is_representative`	1 for the first-seen row in the group (used to pick one row per group in “collapse” mode).
`duplicate_group_mode`	Which duplicate-handling mode produced this CSV: `group`, `collapse`, or `off`.
`dup_total_count`	Backward-compatible alias for `duplicate_group_count`.
`dup_removed_count`	Backward-compatible: in `collapse` mode, how many observations were merged into this representative row (0 in `group`/`off` mode).

Changelog

identity_100bp_upstream/identity_100bp_downstream → identity_20bp_upstream/identity_20bp_downstream. The longer-range flanking-identity window was renamed and shortened from 100 bp to 20 bp (the underlying flank_identity() call now uses n=20 instead of n=100). The backend (preprocess.py) now only ever produces the 20 bp columns going forward. The frontend and the popup-note builders in igv.py/jbrowse_tracks.py still recognize the legacy identity_100bp_* columns as a fallback, so CSVs processed before this change continue to display correctly — but new processing runs never write identity_100bp_* again. tsm_scoring.py’s “longer-range context quality” evidence/penalty block was updated to read the new 20 bp columns with its thresholds otherwise unchanged.
tsm_uncertainty_length_max → tsm_breakpoint_shift_max_bp.
ts_ → ts1_4_ for the gene-annotation columns evaluated against ts_1_4_region specifically (see the naming note at the top of this document).
tsm_gene_priority/tsm_gene_priority_label/tsm_gene_display → gene_priority_score/gene_priority_label/gene_display_names.
tsm_palindrome_capable is now actually written to the output row (it was computed internally for scoring but previously dropped before this fix).
The tsm_-namespaced copies of fw_insertion_bp/fw_deletion_bp/ fw_mismatch_bp/fw_edit_bp_total were removed; tsm_scoring.py reads the fw_* columns directly instead.