dyco detect-remove

The main command. Read raw files, cut them into averaging-period chunks, rotate each chunk in memory, detect its time lag, remove that lag from the unrotated data, and write the result.

Also available as the standalone dyco-detect-remove.

A complete command

This is the bundled CH-LAE example. It processes examples/data/CH-LAE_202507251300.csv.gz, one hour of 20 Hz data, gzipped, with a 3-row header, and writes two 30-minute lag-corrected chunks:

dyco detect-remove --input-dir examples/data --output-dir ./dyco_out --file-pattern "*.csv.gz" --col-u "U_[HS50-B]" --col-v "V_[HS50-B]" --col-w "W_[HS50-B]" --col-tsonic "T_SONIC_[HS50-B]" --scalar "CO2:CO2_DRY_[IRGA72-A]" --scalar "H2O:H2O_DRY_[IRGA72-A]@lag=30;uws=30" --hz 20 --chunk-seconds 1800 --lag-max 10 --lws 0 --uws 10 --n-bootstrap 100 --skiprows 0 --extra-rows 2 --sep "," --start-time-regex "(\d{12})" --start-time-format "%Y%m%d%H%M" --chunk-name-template "CH-LAE_{starttime}{suffix}" --n-workers 4 --random-state 42

Reading it in groups:

Flags	What they say
--input-dir --output-dir --file-pattern	Where the raw files are, where results go, which of them to take.
--col-u/v/w --col-tsonic	The four wind columns. T_SONIC is required, because PWB tries it as an alternative reference.
--scalar LABEL:column	One per gas, repeated. LABEL becomes the prefix in the results (co2_tlag_s). @lag=30;uws=30 gives H₂O its own wider window, since sorption on the tube walls delays it beyond the dry gases.
--hz --chunk-seconds	Sampling rate, and the averaging period each file is cut into.
--lag-max --lws --uws	Search window in seconds. 0 to 10 keeps only positive lags, because a closed-path tube delay cannot be negative.
--skiprows --extra-rows --sep	The file format. See Input file formats.
--start-time-regex --start-time-format --chunk-name-template	Read the start time out of the filename so each output chunk can be named for its own wall-clock time.
--n-workers --random-state	Parallelism, and a seed that makes the bootstrap reproducible.

Per-gas search windows

Gases with different inlet geometry need different search windows. Each gas can have its own:

dyco detect-remove --scalar "CH4:ch4" --scalar "H2O:h2o@lag=30;uws=25" --lws 0 --uws 5

A positive-only window keeps only physical tube delays (a closed-path delay is always > 0). A long-inlet gas such as H₂O can use a wider window than the dry gases in the same run, which matters because EddyPro applies a single lag setting to all gases downstream. Keep the expected lag near the middle of the window; detections pinned to a boundary are unreliable and are discarded.

Important

Downstream flux processing must run with time-lag maximization disabled. The lag has already been removed.

All options

Per-file PWB pipeline: read raw -> rotate wind (in memory) -> detect time lag -> remove lag from the unrotated raw data -> write. Alias: uv run dyco-detect-remove

usage: dyco detect-remove [-h] --input-dir INPUT_DIR --output-dir OUTPUT_DIR
                          [--file-pattern FILE_PATTERN] --col-u COL_U --col-v
                          COL_V --col-w COL_W --col-tsonic COL_TSONIC --scalar
                          LABEL:column[@lag=..;uws=..] [--hz HZ]
                          [--lag-max LAG_MAX] [--n-bootstrap N_BOOTSTRAP]
                          [--block-length BLOCK_LENGTH]
                          [--output-suffix OUTPUT_SUFFIX] [--wdt WDT]
                          [--lws LWS] [--uws UWS]
                          [--random-state RANDOM_STATE]
                          [--chunk-seconds CHUNK_SECONDS]
                          [--min-chunk-seconds MIN_CHUNK_SECONDS]
                          [--chunk-name-template CHUNK_NAME_TEMPLATE]
                          [--start-time-regex START_TIME_REGEX]
                          [--start-time-format START_TIME_FORMAT]
                          [--skiprows SKIPROWS] [--extra-rows EXTRA_ROWS]
                          [--sep SEP] [--lineterm LINETERM]
                          [--na-values NA_VALUES [NA_VALUES ...]]
                          [--na-rep NA_REP] [--hdi-thresh HDI_THRESH]
                          [--dev-thresh DEV_THRESH]
                          [--hdi-prefilter HDI_PREFILTER]
                          [--max-carry MAX_CARRY]
                          [--lag-column-template LAG_COLUMN_TEMPLATE]
                          [--detect-subdir DETECT_SUBDIR]
                          [--data-subdir DATA_SUBDIR] [--save-plots]
                          [--n-workers N_WORKERS] [--strict]

Named Arguments

--input-dir

Directory containing raw (unrotated) EC files.

--output-dir

Output directory for lag-corrected (still unrotated) files.

--file-pattern

Glob pattern for input files.

Default: '*.csv'

--col-u

Column name for U (horizontal x).

--col-v

Column name for V (horizontal y).

--col-w

Column name for W (vertical).

--col-tsonic

Column name for sonic temperature.

--scalar

Gas label and column name in the raw file, e.g. “CH4:CH4_DRY_[LGR-A]”. Repeat for each gas. Append an optional per-gas time-lag window after “@” as “;”-separated key=value pairs (seconds): lag (lag_max), block, lws, uws – e.g. “H2O:h2o@lag=30;uws=25” gives a long-inlet gas a wider window than the dry gases. “lagfrom=LABEL” makes this gas borrow another’s lag for periods it has none of its own. Donate only between gases that behave alike: H2O sticks to the tube wall and lags longer than the flow, so it must NOT be the donor for CO2, CH4 or N2O even when it is the gas that detects best.

--hz

Sampling frequency in Hz.

Default: 20

--lag-max

CCF search half-width [s].

Default: 10.0

--n-bootstrap

Number of block-bootstrap replicates (paper: 99).

Default: 99

--block-length

Bootstrap block length [s] (paper: L = 20 s).

Default: 20.0

--output-suffix

Extension the written chunks carry, format and compression together: .csv, .csv.gz, .dat.zip, .txt. The leading dot is required. Naming the compression alone (.gz) keeps the input text format in front of it. auto (default) reuses the extension of the input file. The template placeholder {suffix} expands to this.

Default: 'auto'

--wdt

Width [records] of the centred rolling mean applied to each bootstrap CCF before its peak is taken. 5 follows RFlux; the paper uses hz/2+1 (11 at 20 Hz, 6 at 10 Hz).

Default: 5

--lws

Optional lower limit [s] of an asymmetric lag search window applied to all gases (per-gas “@lws=” overrides).

--uws

Optional upper limit [s] of the asymmetric lag search window applied to all gases (per-gas “@uws=” overrides).

--random-state

Base seed for reproducible bootstrap. Each file, chunk and gas gets a derived seed.

--chunk-seconds

Chunk length [s] (default: 1800 = 30 min). One output file per chunk per input file.

Default: 1800.0

--min-chunk-seconds

Chunks shorter than this [s] are skipped (default 300 s = 5 min). PWB needs enough records to fit the block-bootstrap.

Default: 300.0

--chunk-name-template

Template for chunk output filenames. Placeholders: {stem}, {suffix}, {index}, {starttime} (last one requires –start-time-regex).

Default: '{stem}_chunk{index:02d}{suffix}'

--start-time-regex

Regex extracting the start timestamp of the input file from its name. Concatenated capture groups (or the whole match) parsed via –start-time-format.

--start-time-format

strftime/strptime format for –start-time-regex and for the {starttime} placeholder in the chunk-name template.

Default: '%Y%m%d-%H%M'

--skiprows

Lines BEFORE the column-name row (raw CSV with header on line 1: 0; EddyPro rotated: 9).

Default: 0

--extra-rows

Extra rows AFTER the header but BEFORE data (e.g. units + instrument-source rows: 2 for the typical raw EC CSV).

Default: 2

--sep

Field separator. Default ‘,’ (CSV). Use ‘s+’ for whitespace or ‘t’ for TSV.

Default: ','

--lineterm

Line terminator for the output file. Default ‘auto’ reproduces the input file’s convention (CRLF for typical Windows EC logger files, LF for Unix). Override with ‘rn’ or ‘n’ to force one.

Default: 'auto'

--na-values

Strings to treat as NaN on read.

Default: ['-9999', '-9999.0', '-9999.0000000000000']

--na-rep

Value written for NaN on output.

Default: '-9999'

--hdi-thresh

S1 HDI threshold [s]: chunks with HDI range below this are flagged S1_optimal (reliable).

Default: 0.5

--dev-thresh

S2 deviation threshold [s]: uncertain chunks are accepted if within this distance of the preceding optimal lag.

Default: 0.5

--hdi-prefilter

Pre-filter [s]: lags with HDI range above this are set to NaN before PWBOPT (pre-filtered variant). Set to 0 to disable.

Default: 1.0

--max-carry

Longest carry, in averaging periods, that PWBOPT S3 may use: a period with no usable detection takes the nearest optimal lag only if it is within this many periods, otherwise the lag expires and falls through to a donor gas or the median. Default: unlimited, which is the published behaviour.

--lag-column-template

Which PWBOPT lag column to actually remove in phase 2. Use {prefix} for the lowercased scalar label. Default {prefix}_tlag_final_pf_s (pre-filtered, gap-filled “best” lag; matches dyco-apply-batch). Use {prefix}_tlag_final_s for the non-pre-filtered PWBOPT lag.

Default: '{prefix}_tlag_final_pf_s'

--detect-subdir

Subfolder of –output-dir for step-1 (detect) diagnostics: plots/ and plots_summary/ (default: 1_lag_detection).

Default: '1_lag_detection'

--data-subdir

Subfolder of –output-dir for step-2 (remove) output: the lag-corrected chunk files (default: 2_lag_removed). Kept separate from the summary CSV / log / plots so it can be used directly as the input directory for the next flux step.

Default: '2_lag_removed'

--save-plots

Save the 3-panel PWB diagnostic figure per chunk per scalar into <output-dir>/<detect-subdir>/plots/, and the batch-overview figures into <output-dir>/<detect-subdir>/plots_summary/.

Default: False

--n-workers

Parallel worker processes. Default: os.cpu_count(). Set to 1 for sequential in-process execution (useful for debugging).

--strict

Re-raise exceptions on the first failure instead of capturing them per file.

Default: False