What a run writes

Folder layout

A dyco detect-remove run fills --output-dir like this:

1_lag_detection/    STEP 1 -- detect (diagnostics + results)
    plots/          per-chunk PWB diagnostic figures      (--save-plots)
    plots_summary/  batch-level overview figures          (--save-plots)
    detect_and_remove_tlag_decisions.txt         start here: the lag applied to
                                                 each output file, and why
    detect_and_remove_tlag_summary.csv           one row per chunk
    detect_and_remove_tlag_summary_columns.md    data dictionary for that CSV
    detect_and_remove_tlag_checkpoint.csv        phase-1 snapshot
    detect_and_remove_tlag_remove_checkpoint.csv phase-2 snapshot

2_lag_removed/      STEP 2 -- remove (the deliverable)
    Lag-corrected chunk files, and only those.

run_settings.txt    every setting used, each with a one-line explanation
detect_remove_tui_settings.yaml   reloadable by `dyco tui`
log.txt             plain-text console log
README.txt          this layout, written into the folder itself

2_lag_removed/ is the one to hand to your flux software. The subdirectory names come from --detect-subdir and --data-subdir if you want different ones.

pwb-batch and apply-batch write their own log.txt beside their results. pwb-batch’s results file is called tlag_results.csv, which is what apply-batch --results-csv expects.

Note

Lag is expressed in seconds. The covariance-maximization method removed in v3 counted records instead, so older notes, plots and result files may mean something different by “lag”.

Which lag was applied, and why

detect_and_remove_tlag_decisions.txt answers both, one block per written file:

CH-CHA_202110080730.csv
  CH4     +2.20 s (+44 rec)  detected here and reliable (S1): HDI 0.40 s < 0.50 s
  N2O     +1.85 s (+37 rec)  nothing had been detected yet at this point in the run,
                             so the lag was filled 2 period(s) back from
                             CH-CHA_202110080900.csv, the first period that did

CH-CHA_202110080800.csv
  CH4     +2.20 s (+44 rec)  carried 1 period(s) forward from CH-CHA_202110080730.csv:
                             HDI 2.70 s, wider than the 1.00 s prefilter
  N2O     +1.85 s (+37 rec)  borrowed from CH4: HDI 9.15 s, wider than the 1.00 s
                             prefilter, and no detection of its own within
                             2 period(s) either side

The thresholds behind the decisions head the file, periods that produced no output file are listed at the foot, and a tally closes it. The same sentences are in the summary CSV as {gas}_lag_reason, if you would rather join them to the numbers.

The summary CSV

detect_and_remove_tlag_summary.csv has one row per chunk: the detected lag, its HDI, the reliability flag, the PWBOPT columns and the records actually applied. Its schema mirrors tlag_results.csv from pwb-batch, plus the columns this pipeline adds at the removal step.

Every column is described in detect_and_remove_tlag_summary_columns.md, written next to the CSV by the same run. That file is generated from the run’s own settings — it names your gases, your sampling rate and the specific lag column that was applied — so it is always more accurate than a list in these docs could be. Read it first.

The parts worth knowing before you open either file:

{gas}_lag_applied_s

The lag that was actually removed, in seconds. Derived from the record shift the pipeline made, so it describes the files on disk rather than what was asked for — and it is always a whole number of records, which is why it can sit up to half a record from the requested lag. Six other lag columns exist per gas; they are the working steps that led to this one.

{gas}_lag_reason

Why that value was chosen, in words. The same text as in the decisions report above.

{gas}_tlag_s

The raw per-chunk detection, before PWBOPT. This is not necessarily what was removed. A wide-HDI chunk’s raw lag can be spurious, which is the entire reason PWBOPT exists.

{gas}_hdi_range_s

The width of the 95% interval, and the number every PWBOPT decision turns on. Below --hdi-thresh the detection is accepted outright (S1). {gas}_hdi_lo_s and {gas}_hdi_hi_s are the bounds it was computed from, and {gas}_is_reliable is the same test as a boolean.

{gas}_n_valid

How many records of this gas the chunk actually holds. 0 means the analyser was offline for the whole period: nothing was detected, nothing was carried or borrowed in, and the column was written through untouched.

{gas}_best_combination

Which of the four pre-whitening combinations won: cw and wc are the gas against vertical wind, ct and tc the gas against sonic temperature. Strong fluxes usually win on cw/wc. A trace gas that keeps falling back to the temperature pair is telling you its own signal against W is too weak to work with.

{gas}_tlag_final_pf_s

The PWBOPT-optimised, pre-filtered, gap-filled lag — the default value of --lag-column-template, and so by default the column that was requested for removal. The data dictionary flags whichever column your run used.

{gas}_lag_source

Where the period’s lag came from: own (the gas detected it here, or carried it from one of its own nearby periods), from:CO2 (borrowed from a donor gas), median (the last-resort median of rejected detections), no_data (the gas was missing for the whole period, so no lag was needed and none was applied), or none. See PWBOPT.

{gas}_carry_periods

How far that lag travelled: 0 if it was detected in this very period, n if it came from n periods away, empty if it came from somewhere other than the gas’s own carry. own alone cannot distinguish a fresh detection from an inherited one; this can.

{gas}_applied_records

The shift in records, which is {gas}_lag_applied_s * hz.

status and {gas}_status

Two different things. The row-level status is the chunk’s fate — only ok rows produced an output file, while skipped:short, skipped:duplicate and error rows are reported for traceability and write nothing. The per-gas {gas}_status is that gas’s phase-2 outcome within an otherwise fine chunk: ok, skipped:lag_nan (no finite PWBOPT lag to apply), or pending (the chunk never reached phase 2).

Note

Rows that wrote no file carry no lag. For any row whose status is not ok, the final, applied and carry columns are left empty and {gas}_lag_source reads none. There is no data there to align, so a number in those columns would suggest a correction that never happened. The detection columns are untouched.

Tip

A run where many periods show median in {gas}_lag_source is telling you that gas could not locate its own lag and had no donor. Give it one with @lagfrom=, and set --max-carry alongside it — without a carry limit the gas keeps reaching every period with its own lag and the donor is never consulted.

Diagnostic plots

--save-plots writes two kinds of figure. Both are off by default, because they cost time and disk on a long run.

One figure per chunk and gas, in plots/

Three panels, left to right (PreWhiteningBootstrap.plot):

1. The pre-whitened cross-correlation, grey stems with the smoothed line over them, a Bartlett significance band, and a red marker at the detected lag. If the peak does not clear the band, there was nothing to find in that period.

2. The raw cross-covariance, same layout, with the same lag marked. This is the curve the old covariance-maximization method worked on, so comparing the two panels shows what pre-whitening bought.

3. The bootstrap lag distribution for the winning combination: a histogram of the peak found in each resample, the 95% interval shaded, and the mode marked. A tight cluster is a reliable lag; a spread-out or multi-peaked one is what a wide interval looks like. The panel title names the combination that won.

Batch overviews, in plots_summary/

Five panels per gas, plus one cross-gas comparison figure (PwbBatchDetection.plot_summary):

1. Detected lags over the run, coloured by S1/S2/S3 flag.

2. The final gap-filled lags, with the S1/S2 detections that anchor them drawn as filled markers and the applied lag as open circles. This is the panel that shows carry and borrowing at a glance.

3. Interval width per period, against the S1 threshold and the pre-filter threshold as lines.

4. Flag counts per period, standard rule beside pre-filtered.

5. A histogram of every detected lag, with the mode marked.

The comparison figure puts all gases on one scatter with a density curve per gas, which is where a systematic offset between two gases in the same tube shows up.

log.txt and run_settings.txt

Every CLI writes a log.txt to its output folder: the run header, the per-file and per-chunk lines, and the finish time. The animated progress display is deliberately kept out of it, so the log stays readable as plain text.

run_settings.txt is written at the start of the run and records every setting with a one-line explanation of what it does. Between it and detect_remove_tui_settings.yaml, a finished run carries enough to reproduce itself.

Feeding the output to flux software

Important

Downstream flux processing must run with time-lag maximization disabled. The lag has already been removed from the files in 2_lag_removed/.