sp_validation.sacc_io

SACC_IO.

Name:

sacc_io.py

Description:

Read/write the standard SACC data-product layout for the weak-lensing validation package. One file describes each catalogue version:

• {version}.sacc — NZ tracers, reporting-grid ξ±, pseudo-Cℓ (EE/BB/EB) with bandpower windows, COSEBIs, pure E/B, ρ/τ PSF diagnostics, and the fine ξ± integration input for COSEBIs / pure-EB (grid='integration' tagged points). The covariance is assembled block-diagonally from the per-statistic covariances (zero cross-blocks, never materialized): the analysis blocks first, then a dense per-pair integration-ξ block (the analytic integration-binning covariance when it exists — it feeds derived-statistic error propagation — or the TreeCorr varxip/varxim diagonal as degraded fallback). assemble_covariance hands sacc a list of blocks, which stores a BlockDiagonalCovariance (one FITS table per block, Σ block² on disk rather than a dense N²) — cost scales with the integration grid’s size, a parameter set by the caller, not baked into the layout.

Insertion order is load-bearing. A Sacc is a flat list of data points in the order add_data_point was called, and row/column i of the covariance refers to the i-th inserted point — there is no other linkage between a point and its covariance entry. SACC preserves that order bitwise through FITS save/load, so writers define the covariance layout by their insertion sequence. Writers below insert in the canonical order — ξ+ then ξ−, Cℓ (ee, bb, eb), COSEBIs (all Eₙ then all Bₙ), pure E/B (xip_E, xim_E, xip_B, xim_B, xip_amb, xim_amb — matching b_modes._EB_KEYS), ρ, then τ — but readers never assume global order: they resolve indices through Sacc.indices(dtype, tracers, **tags).

Tag filters are plain keyword arguments to indices / get_data_points / get_tag; the tags={...} form silently selects nothing and must never be used.

Tomographic ξ ordering: each add_xi call inserts one tracer pair as [xip; xim], so a multi-pair vector is pair-major — [pair_0 xip; pair_0 xim; pair_1 xip; …] — not type-major ([all xip; all xim]). A tomographic ξ covariance with cross-pair correlations is therefore supplied to assemble_covariance as ONE contiguous block spanning the consecutive add_xi calls, ordered pair-by-pair to match insertion. Writers must call add_xi in the same pair order the covariance was built in. Converters that need a type-major layout (e.g. the DES 2pt-FITS convention) permute explicitly via s.indices rather than assuming global order.

Optionality: a file’s contents are flexible about which components of

a statistic it actually has. add_pseudo_cl requires EE (the bandpower window’s reference series) but BB and EB are optional — an analysis that never computed EB simply omits it. add_cosebis requires Eₙ but Bₙ is optional. add_pure_eb requires xip_E/xim_E but the B and ambiguous-mode blocks are each optional, independently (B and amb are unrelated computations). Everything else a writer takes — θ/ℓ grids, tracer/bin identifiers, the value array for a component you ARE adding — is structurally necessary and has no default; supplying it partially would desynchronise the covariance layout, so it is refused rather than degraded. Readers mirror this split: a composite reader (get_pseudo_cl, get_cosebis, get_pure_eb) returns None (or omits the key) for a component the file doesn’t carry, but a selection naming that component explicitly (s.indices, _mean, extract) still fails loud on no match — silence is reserved for “this file doesn’t have that optional piece”, never for “you asked for something specific and it isn’t there”.

source_name(i)

SACC tracer name for source redshift bin i (0-based).

Kept as the single definition of the source_{i} naming contract — external consumers address tracers through it rather than the f-string.

new_sacc(nz, metadata=None)

Create a Sacc with the survey’s NZ (and PSF) tracers.

Parameters:

• nz (dict or sequence) – Redshift distributions, one per source bin. Either a mapping {i: (z, nz)} keyed by 0-based bin index, or a sequence of (z, nz) array pairs (bin index = position). Tracers are named source_{i}.

• metadata (dict, optional) – Key/value pairs stored on s.metadata.

Returns:

Sacc holding the source_{i} NZ tracers and the psf_stars Misc tracer (needed by ρ/τ diagnostics).

Return type:

sacc.Sacc

_pair(bins)

Resolve a (i, j) bin pair to the (source_i, source_j) names.

The pair is normalised to i <= j: shear-shear statistics are symmetric in the tracer pair, and SACC stores each pair under one ordering, so (1, 0) must address the same points as (0, 1). Kept as a helper: every add_*/get_* function relies on this normalisation.

_check_ascending(name, values)

Require values to be strictly ascending; else raise ValueError.

Insertion order is the covariance (and bandpower-window) order, and readers return points in insertion order, so an out-of-order grid would silently desynchronise a data vector from its covariance. Enforce monotonicity at write time instead.

_add_theta_series(s, dtype, tracers, theta, values, **tags)

Insert one theta-tagged series, one point per (theta, value) pair.

add_xi(s, bins, theta, xip, xim, *, grid, theta_nom=None, npairs=None, weight=None)

Add a real-space shear 2PCF (ξ+ then ξ−) for one tracer pair.

Parameters:

• s (sacc.Sacc) – Target, mutated in place.

• bins (tuple of int) – Source bin pair (i, j).

• theta (array_like) – Angular separations (arcmin) — TreeCorr meanr.

• xip (array_like) – ξ+ and ξ− at theta.

• xim (array_like) – ξ+ and ξ− at theta.

• grid ({'reporting', 'integration'}) – Stored as the grid tag on every point. The reporting (analysis) ξ± and the fine COSEBIs/pure-EB integration input share the same data type and tracer pair, so this tag is the only thing that disambiguates them in the file.

• theta_nom (array_like, optional) – Nominal bin centres — TreeCorr rnom — stored as theta_nom.

• npairs (array_like, optional) – TreeCorr pair counts and weights, stored per point.

• weight (array_like, optional) – TreeCorr pair counts and weights, stored per point.

add_pseudo_cl(s, bins, ell_eff, cl_ee, cl_bb=None, cl_eb=None, *, window_ells, window_weights, grid='reporting')

Add pseudo-Cℓ EE (required) plus whichever of BB/EB were computed.

Parameters:

• s (sacc.Sacc) – Target, mutated in place.

• bins (tuple of int) – Source bin pair (i, j).

• ell_eff (array_like) – Effective multipole of each bandpower.

• cl_ee (array_like) – EE bandpowers at ell_eff.

• cl_bb (array_like, optional) – BB and/or EB bandpowers at ell_eff. Each defaults to None and is then simply not written — EB in particular is often not computed at all.

• cl_eb (array_like, optional) – BB and/or EB bandpowers at ell_eff. Each defaults to None and is then simply not written — EB in particular is often not computed at all.

• window_ells (array_like) – Multipoles spanned by the bandpower window matrix (shape (nell,)).

• window_weights (array_like) – Window matrix W of shape (nell, nbp) — one column per bandpower — from NaMaster get_bandpower_windows. One sacc.BandpowerWindow is built and shared across every component written.

• grid (str, optional) – Stored as the grid tag on every point (default 'reporting'), joining merge’s ℓ-consistency group; variant ℓ binnings belong under different tag values.

add_cosebis(s, bins, En, scale_cut, Bn=None)

Add COSEBIs Eₙ (required) and Bₙ (optional) for one scale cut.

Parameters:

• s (sacc.Sacc) – Target, mutated in place.

• bins (tuple of int) – Source bin pair (i, j).

• En (array_like) – E-mode COSEBI amplitudes, one per logarithmic mode n (1-based).

• scale_cut (tuple of float) – (theta_min, theta_max) in arcmin, stored on every point as the theta_min/theta_max tags; multiple cuts coexist in one file, told apart by these tags.

• Bn (array_like, optional) – B-mode COSEBI amplitudes at the same n. Defaults to None and is then simply not written. The [En; Bn] layout, when both are present, matches the COSEBI covariance.

add_pure_eb(s, bins, theta, xip_E, xim_E, xip_B=None, xim_B=None, xip_amb=None, xim_amb=None, *, grid='reporting')

Add pure E-mode (required) plus whichever of B/ambiguous were computed.

Blocks are inserted in PURE_KEYS order (xip_E, xim_E, xip_B, xim_B, xip_amb, xim_amb), matching b_modes._EB_KEYS and the pure-EB covariance layout — whichever subset is present.

Parameters:

• s (sacc.Sacc) – Target, mutated in place.

• bins (tuple of int) – Source bin pair (i, j).

• theta (array_like) – Angular separations (arcmin), shared by every block written.

• xip_E (array_like) – The pure E-mode correlation functions at theta.

• xim_E (array_like) – The pure E-mode correlation functions at theta.

• xip_B (array_like, optional) – The pure B-mode correlation functions. Both default to None; the pair is written together or not at all — supply both or neither.

• xim_B (array_like, optional) – The pure B-mode correlation functions. Both default to None; the pair is written together or not at all — supply both or neither.

• xip_amb (array_like, optional) – The ambiguous-mode correlation functions. Both default to None; same both-or-neither rule as B.

• xim_amb (array_like, optional) – The ambiguous-mode correlation functions. Both default to None; same both-or-neither rule as B.

• grid (str, optional) – Stored as the grid tag on every point (default 'reporting'), joining ξ’s theta-consistency group in merge’s guard.

add_rho(s, k, theta, rho_p, rho_m, *, grid='reporting')

Add a ρ_k PSF statistic (ρ+ then ρ−) on the psf_stars tracer.

Parameters:

• s (sacc.Sacc) – Target, mutated in place.

• k (int) – ρ index (0…5).

• theta (array_like) – Angular separations (arcmin).

• rho_p (array_like) – ρ_k+ and ρ_k− at theta.

• rho_m (array_like) – ρ_k+ and ρ_k− at theta.

• grid (str, optional) – Stored as the grid tag on every point (default 'reporting'), joining ξ’s theta-consistency group in merge’s guard.

add_tau(s, bins, k, theta, tau_p, tau_m, *, grid='reporting')

Add a τ_k PSF-leakage statistic (τ+ then τ−).

Parameters:

• s (sacc.Sacc) – Target, mutated in place.

• bins (tuple of int) – Source bin i and PSF; the τ tracers are (source_i, psf_stars). Only bins[0] is used.

• k (int) – τ index (0, 2 or 5).

• theta (array_like) – Angular separations (arcmin).

• tau_p (array_like) – τ_k+ and τ_k− at theta.

• tau_m (array_like) – τ_k+ and τ_k− at theta.

• grid (str, optional) – Stored as the grid tag on every point (default 'reporting'), joining ξ’s theta-consistency group in merge’s guard.

assemble_covariance(s, blocks)

Assemble a BlockDiagonalCovariance from per-statistic blocks.

Each block is validated against the current insertion order: its indices must be contiguous and ascending, the blocks must tile 0…len(s.mean) exactly (no gap, no overlap), and each block must be square with a size matching its index span. Any violation raises ValueError naming the mismatch. Cross-blocks are zero and implicit — never materialized — because the blocks are passed to add_covariance as a list, which sacc.BaseCovariance.make turns into a BlockDiagonalCovariance (one FITS table per block, Σ block² on disk rather than a dense N² file).

Parameters:

• s (sacc.Sacc) – Target, mutated in place via add_covariance.

• blocks (sequence) – Ordered (selector, cov) pairs (or a mapping of the same). Each selector is either an index array, or a (data_type, tracers) / (data_type, tracers, tags) tuple resolved through s.indices; cov is the block’s dense covariance.

Returns:

s, with the assembled BlockDiagonalCovariance attached.

Return type:

sacc.Sacc

_resolve_indices(s, selector)

Resolve a covariance-block selector to a sorted index array.

add_diagonal_covariance(s, variances)

Attach a DiagonalCovariance from a 1-D variance array.

The 1-D array is passed straight to add_covariance (never np.diag), which is what makes SACC store a DiagonalCovariance.

Parameters:

• s (sacc.Sacc) – Target, mutated in place.

• variances (array_like) – Per-point variances, len == len(s.mean).

Returns:

s, with the DiagonalCovariance attached.

Return type:

sacc.Sacc

get_nz(s, i)

Return (z, nz) for source bin i.

_get_pm(s, dtype_p, dtype_m, tracers, **tags)

Return (theta, plus, minus) for a +/− data-type pair.

get_xi(s, bins, *, grid)

Return (theta, xip, xim) for one tracer pair and grid.

get_pseudo_cl(s, bins)

Return (ell_eff, cl_ee, cl_bb, cl_eb, window) for one tracer pair.

cl_bb/cl_eb come back None if the file doesn’t carry that component (add_pseudo_cl makes both optional). window is the shared sacc.BandpowerWindow recovered via get_bandpower_windows; its columns are in the same insertion order as the returned ell_eff/cl arrays, so window column j corresponds to ell_eff[j].

get_cosebis(s, bins, scale_cut=None)

Return (n, En, Bn) for one tracer pair.

Bn comes back None if the file doesn’t carry it (add_cosebis makes it optional).

Parameters:

scale_cut (tuple of float, optional) – (theta_min, theta_max) to select when several cuts share the file.

get_pure_eb(s, bins)

Return (theta, {key: array}) for whichever pure-EB blocks exist.

xip_E/xim_E are always present (add_pure_eb requires them); the dict holds whichever of the B and ambiguous-mode keys (out of PURE_KEYS: xip_E, xim_E, xip_B, xim_B, xip_amb, xim_amb) the file actually carries — a key absent from the file is simply absent from the dict, not mapped to None.

get_rho(s, k)

Return (theta, rho_p, rho_m) for ρ index k.

get_tau(s, bins, k)

Return (theta, tau_p, tau_m) for τ index k and source bin.

_indices(s, data_type, tracers, **tag_filters)

Sacc.indices that fails loud instead of selecting nothing.

Sacc.indices returns an empty array (warning only) when a selection matches no point — e.g. a typo’d tag value, or a float tag filter that is not bitwise-identical to the stored one. Every reader here funnels through this guard so an unmatched selection raises instead of propagating empty arrays downstream.

_mean(s, data_type, tracers, **tag_filters)

Mean values for a selection, in s.indices (insertion) order.

Never re-sort: insertion order is the covariance and bandpower-window order, so returning in s.indices order keeps every reader aligned with the covariance for any file (and ascending for canonically-written files, which the writers enforce).

_tag(s, data_type, tracers, tag, **tag_filters)

Values of tag for a selection, in insertion order.

_mean_optional(s, data_type, tracers, **tag_filters)

Mean values for a selection, or None if the file has none.

Used by composite readers (get_pseudo_cl, get_cosebis, get_pure_eb) for the components a writer made optional (BB/EB, COSEBI Bₙ, pure B/ambiguous): absence is a legitimate “this file doesn’t have that piece”, not a typo to fail loud on — unlike _mean/ _indices, used for selections that name a component explicitly.

extract(s, data_type=None, tracers=None, **tag_filters)

Extract a sub-Sacc (points + aligned covariance sub-block).

A copy is made and everything not matching the selection is removed, so the covariance sub-block comes out correctly aligned and the original is untouched.

Parameters:

• s (sacc.Sacc) – Source, left unmodified.

• data_type (str, optional) – Data type to keep.

• tracers (tuple, optional) – Tracer pair to keep, as SACC tracer names (e.g. ("source_0", "source_0") or ("source_0", "psf_stars")) — not integer bin indices. This differs deliberately from the add_* / get_* interface, whose bins argument takes integer pairs: extract is the generic selection escape hatch, mirroring Sacc.keep_selection and addressing non-source tracers uniformly.

• **tag_filters – Tag filters (plain kwargs, e.g. grid='integration').

Returns:

New Sacc holding only the selected points.

Return type:

sacc.Sacc

merge(saccs)

Merge several per-statistic Sacc objects into one file’s worth.

A thin wrapper around sacc.concatenate_data_sets: data points concatenate in input order, tracers shared by several inputs (the source_i NZ tracers, psf_stars) are stored once, and the covariance combines block-diagonally in the same order — the library requires either all inputs to carry a covariance or none, and raises otherwise (cross-statistic covariance assembly beyond block-diagonal is out of scope here; see assemble_covariance).

Metadata must be consistent: keys present in several inputs must carry equal values (a type: data file cannot merge with a type: mock file), and the union lands on the result. This deliberately replaces the library’s clash behaviour, which mangles clashing keys by appending labels.

Grid consistency follows tagging semantics: the grid tag declares which binning a set of points lives on, so all same-length theta (or ell) arrays under one tag value must be bitwise identical — sacc itself never validates angles across data types/tracers, and a grid that differs only at floating-point level chokes CosmoSIS downstream instead of failing loud here. Different lengths within a tag group pass (scale-cut subsets are legitimate); grids under different tag values are unconstrained (reporting vs integration differ by design); θ and ℓ are separate domains, each checked against itself only. ℓ series sharing a bitwise-equal grid must also share the bandpower window (series without windows skip that check).

Parameters:

saccs (sequence of sacc.Sacc) – The per-statistic data sets, in the insertion order the merged file should have. Inputs are left unmodified.

Returns:

The merged data set.

Return type:

sacc.Sacc

Raises:

ValueError – If metadata conflicts, a shared tracer differs across inputs, two same-length theta or ell arrays under the same grid tag value are not bitwise identical, or two ℓ series sharing a grid carry different bandpower windows.

_grid_groups(s, angle)

Nested map grid-tag-value -> (data_type, tracers) -> point indices.

One entry per (data_type, tracers) series carrying an angle ('theta' or 'ell') tag, in each series’ own insertion order (never re-sorted), nested under the grid tag value it lives on (None for untagged series) — the shape merge’s consistency guard checks within each tag value. Indices (not angle values) are kept so the ℓ check can also recover each series’ bandpower window.

_check_grid_consistency(s, angle)

Raise unless same-length angle arrays under one grid tag match.

Consistency follows tagging semantics: the grid tag declares which binning a series lives on, so all same-length theta (or ell) arrays sharing a tag value must be bitwise identical — sacc never validates angles across data types/tracers, and a grid diverging at floating-point level chokes CosmoSIS downstream instead of failing loud here. Different lengths within a tag value pass (scale-cut subsets are legitimate); series under different tag values are unconstrained (reporting vs integration differ by design). θ and ℓ are separate domains, each checked against itself only. For ℓ, two series on a bitwise-equal grid must also share the bandpower window (equal window ells and weight matrix); series without windows (foreign files) skip the window check.

update_statistic(s, sub)

Overwrite the values of s’s points that match sub’s, in place.

The merge-back half of the extract → conceal → merge blinding flow (PRD #241 §4): each point of sub is matched to exactly one point of s by (data_type, tracers, tags), and that point’s value is replaced. Nothing else changes — insertion order, tags, windows and the covariance are untouched (blinding shifts the mean only), so sub’s own covariance (e.g. the sub-block extract attaches) is deliberately not consulted. A sub point with no match, or with several, raises ValueError.

Parameters:

• s (sacc.Sacc) – Target, mutated in place.

• sub (sacc.Sacc) – The replacement block, e.g. extract(s, ...) after concealment.

save(s, path, *, type)

Write s to path (FITS), overwriting any existing file.

Parameters:

• s (sacc.Sacc) – Data set to write; its metadata is stamped in place.

• type ({'data', 'mock'}) – Provenance of the underlying catalogue, stored as the required type metadata tag (PRD #241 §4, “Mocks vs data”). The caller — the pipeline computing the data vector — knows whether its input catalogue is a mock; there is deliberately no default. load refuses type='data' files that are not blinded.

load(path, *, allow_unblinded=False)

Load a Sacc from path (FITS), failing closed on unblinded data.

Every sacc_io file carries a type: data|mock metadata tag (stamped by save); blinded files are additionally stamped concealed=True by Smokescreen. A type='data' file without that stamp is real, unblinded data, and loading it raises — skipping the blind can never silently expose the measured vector (PRD #241 §4). Mocks load freely, blinded or not.

Parameters:

• path (str) – File to load.

• allow_unblinded (bool, optional) – Escape hatch for the two legitimate consumers of unblinded data: the blinding step itself (which must read the true vector to conceal it) and the unblinding/verification tooling. Nothing else — no analysis, plotting or inference code — may pass True.

Returns:

The loaded data set.

Return type:

sacc.Sacc