sp_validation.grammar

Column grammars: the v2 schema, and the maps that present a table in it.

ShapePipe products name their columns in one of two grammars. v2 is the grammar the rest of sp_validation reads (HSM_T_PSF, NGMIX_G1_NOSHEAR, MASK_4_Stars, …; see docs/ngmix_psf_column_migration.md). v1 is the grammar of the catalogues released up to and including v1.6.x (SIGMA_PSF_HSM, the 2-vector NGMIX_ELL_NOSHEAR, …).

This module is the single place that knows the difference. adapt hands a table back as a lazy view presenting v2 names and units, so downstream code reads one grammar and never branches on the generation. Two rule families apply:

• The v1 shape-measurement columns, applied when the table is v1 (see detect_generation):

  • E{1,2}_{PSF,STAR}_HSM and FLAG_{PSF,STAR}_HSM are renamed;

  • SIGMA_{PSF,STAR}_HSM (sigma) is presented as HSM_T_{PSF,STAR} holding T = 2 sigma^2, via cs_util.size.sigma_to_T;

  • the 2-vector ngmix columns NGMIX_ELL[_ERR|_PSFo]_{shear} (or their flattened _0/_1 forms, as in the comprehensive HDF5) are split into NGMIX_G{1,2}[_ERR|_PSF_ORIG]_{shear};

  • NGMIX_T_PSFo_{shear}, NGMIX_Tpsf_{shear} and NGMIX_MOM_FAIL become NGMIX_T_PSF_ORIG_{shear}, NGMIX_T_PSF_RECONV_{shear} and NGMIX_MCAL_TYPES_FAIL, except that NGMIX_T_PSF_RECONV_NOSHEAR is read from NGMIX_Tpsf_1P when the table has it (see below).

• The UNIONS mask-bit columns, applied to any table. Bit b of the healsparse mask product is the boolean MASK_{b}_{label} (MASK_1_Faint_star_halos, MASK_4_Stars, MASK_2048_z2, …; mask_column, labels from MASK_LABELS). Two other spellings of the same bit are presented under that name: {b}_{label}, the data_ext dataset of every released v1 comprehensive HDF5, and MASK_n{b}, written by pre-release ShapePipe v2 runs. These columns belong to the mask product, not to a ShapePipe generation, so no spelling marks or requires one.

IMAFLAGS_ISO is not mapped: its v1 bits (2 halo, 4 border, 16 Messier, 32 NGC, 128 spike) differ in meaning from the mask bits of the same value, so it passes through under its own name for the v1 mask configs that cut on it.

The mapping is by name, deliberately, including where the v1 values mean something different from their v2 namesakes: v1 NGMIX_*_PSFo_* hold the reconvolved-PSF alias rather than a fit to the original PSF, and NGMIX_MOM_FAIL counts moments-guess failures rather than failed metacal types. Those values are presented unchanged.

One v1 value is corrected rather than renamed. ShapePipe v1 wrote a wrong no-shear reconvolved-PSF size: NGMIX_Tpsf_NOSHEAR differs by ~2% for nearly every object from the reconvolution kernel metacal applied, which the sheared types’ NGMIX_Tpsf_{1P,1M,2P,2M} record (they agree to ~1e-5). The v1.4.6.3 release cut on and wrote the 1P value, so NGMIX_T_PSF_RECONV_NOSHEAR reads NGMIX_Tpsf_1P whenever the table has it, and NGMIX_Tpsf_NOSHEAR otherwise (a cut catalogue such as v1.4.6.3’s, whose no-shear column already holds the 1P value). ShapePipe v2 reuses one reconvolved PSF across metacal types, so its columns need no correction.

A catalogue in another naming convention can carry a column_map, {v2 name: name in the file}, which adapt applies as renames ahead of the rules above: an entry overrides the rule producing the same v2 name and no other (a column it renames still feeds the other rules that read it), and the v1 detection runs on the columns the map does not rename. An entry may hold one * on both sides, matching any stem: {"NGMIX_*": "MYFIT_*"} presents every MYFIT_<stem> column as NGMIX_<stem>. Explicit entries take precedence over pattern entries.

Every other column passes through under its own name; source names that have a v2 equivalent are hidden.

adapt also joins row-aligned tables into one view, as the comprehensive HDF5 splits one catalogue over its data and data_ext datasets.

SHEARS = ('NOSHEAR', '1P', '1M', '2P', '2M')

Metacal shear types carried by the ngmix columns.

MASK_LABELS = {1: 'Faint_star_halos', 2: 'Bright_star_halos', 4: 'Stars', 8: 'Manual', 16: 'u', 32: 'g', 64: 'r', 128: 'i', 256: 'z', 512: 'Tile_RA_DEC_cut', 1024: 'Maximask', 2048: 'z2'}

Bits of the UNIONS healsparse mask product and what each flags. The column for bit b is MASK_{b}_{label} (mask_column).

mask_column(bit)

Return the column holding mask bit bit: MASK_{bit}_{label}.

Raises:

KeyError – if bit is not a bit of the mask product

class Rule(v2: str, source: str, kind: str = 'rename', arg: int | None = None, fallback: str | None = None)

Bases: object

One v2 column presented from a source column.

kind is "rename" (same values), "sigma_to_T" (T = 2 sigma^2) or "component" (element arg of a 2-vector, stored either as one vector column source or as the flattened {source}_{arg}). fallback names a column read instead when source is absent.

v2: str

source: str

kind: str = 'rename'

arg: int | None = None

fallback: str | None = None

sources()

Return the source names this rule can read from, in preference order.

apply(column)

Return the v2 column computed from source column.

_v1_rules()

V1_RULES = (Rule(v2='HSM_G1_PSF', source='E1_PSF_HSM', kind='rename', arg=None, fallback=None), Rule(v2='HSM_G2_PSF', source='E2_PSF_HSM', kind='rename', arg=None, fallback=None), Rule(v2='HSM_T_PSF', source='SIGMA_PSF_HSM', kind='sigma_to_T', arg=None, fallback=None), Rule(v2='HSM_FLAG_PSF', source='FLAG_PSF_HSM', kind='rename', arg=None, fallback=None), Rule(v2='HSM_G1_STAR', source='E1_STAR_HSM', kind='rename', arg=None, fallback=None), Rule(v2='HSM_G2_STAR', source='E2_STAR_HSM', kind='rename', arg=None, fallback=None), Rule(v2='HSM_T_STAR', source='SIGMA_STAR_HSM', kind='sigma_to_T', arg=None, fallback=None), Rule(v2='HSM_FLAG_STAR', source='FLAG_STAR_HSM', kind='rename', arg=None, fallback=None), Rule(v2='NGMIX_G1_NOSHEAR', source='NGMIX_ELL_NOSHEAR', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G1_ERR_NOSHEAR', source='NGMIX_ELL_ERR_NOSHEAR', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G1_PSF_ORIG_NOSHEAR', source='NGMIX_ELL_PSFo_NOSHEAR', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G2_NOSHEAR', source='NGMIX_ELL_NOSHEAR', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_G2_ERR_NOSHEAR', source='NGMIX_ELL_ERR_NOSHEAR', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_G2_PSF_ORIG_NOSHEAR', source='NGMIX_ELL_PSFo_NOSHEAR', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_T_PSF_ORIG_NOSHEAR', source='NGMIX_T_PSFo_NOSHEAR', kind='rename', arg=None, fallback=None), Rule(v2='NGMIX_T_PSF_RECONV_NOSHEAR', source='NGMIX_Tpsf_1P', kind='rename', arg=None, fallback='NGMIX_Tpsf_NOSHEAR'), Rule(v2='NGMIX_G1_1P', source='NGMIX_ELL_1P', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G1_ERR_1P', source='NGMIX_ELL_ERR_1P', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G1_PSF_ORIG_1P', source='NGMIX_ELL_PSFo_1P', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G2_1P', source='NGMIX_ELL_1P', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_G2_ERR_1P', source='NGMIX_ELL_ERR_1P', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_G2_PSF_ORIG_1P', source='NGMIX_ELL_PSFo_1P', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_T_PSF_ORIG_1P', source='NGMIX_T_PSFo_1P', kind='rename', arg=None, fallback=None), Rule(v2='NGMIX_T_PSF_RECONV_1P', source='NGMIX_Tpsf_1P', kind='rename', arg=None, fallback=None), Rule(v2='NGMIX_G1_1M', source='NGMIX_ELL_1M', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G1_ERR_1M', source='NGMIX_ELL_ERR_1M', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G1_PSF_ORIG_1M', source='NGMIX_ELL_PSFo_1M', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G2_1M', source='NGMIX_ELL_1M', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_G2_ERR_1M', source='NGMIX_ELL_ERR_1M', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_G2_PSF_ORIG_1M', source='NGMIX_ELL_PSFo_1M', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_T_PSF_ORIG_1M', source='NGMIX_T_PSFo_1M', kind='rename', arg=None, fallback=None), Rule(v2='NGMIX_T_PSF_RECONV_1M', source='NGMIX_Tpsf_1M', kind='rename', arg=None, fallback=None), Rule(v2='NGMIX_G1_2P', source='NGMIX_ELL_2P', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G1_ERR_2P', source='NGMIX_ELL_ERR_2P', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G1_PSF_ORIG_2P', source='NGMIX_ELL_PSFo_2P', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G2_2P', source='NGMIX_ELL_2P', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_G2_ERR_2P', source='NGMIX_ELL_ERR_2P', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_G2_PSF_ORIG_2P', source='NGMIX_ELL_PSFo_2P', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_T_PSF_ORIG_2P', source='NGMIX_T_PSFo_2P', kind='rename', arg=None, fallback=None), Rule(v2='NGMIX_T_PSF_RECONV_2P', source='NGMIX_Tpsf_2P', kind='rename', arg=None, fallback=None), Rule(v2='NGMIX_G1_2M', source='NGMIX_ELL_2M', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G1_ERR_2M', source='NGMIX_ELL_ERR_2M', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G1_PSF_ORIG_2M', source='NGMIX_ELL_PSFo_2M', kind='component', arg=0, fallback=None), Rule(v2='NGMIX_G2_2M', source='NGMIX_ELL_2M', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_G2_ERR_2M', source='NGMIX_ELL_ERR_2M', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_G2_PSF_ORIG_2M', source='NGMIX_ELL_PSFo_2M', kind='component', arg=1, fallback=None), Rule(v2='NGMIX_T_PSF_ORIG_2M', source='NGMIX_T_PSFo_2M', kind='rename', arg=None, fallback=None), Rule(v2='NGMIX_T_PSF_RECONV_2M', source='NGMIX_Tpsf_2M', kind='rename', arg=None, fallback=None), Rule(v2='NGMIX_MCAL_TYPES_FAIL', source='NGMIX_MOM_FAIL', kind='rename', arg=None, fallback=None))

The v1 -> v2 map of the shape-measurement columns, applied to v1 tables.

MASK_RULES = (Rule(v2='MASK_1_Faint_star_halos', source='1_Faint_star_halos', kind='rename', arg=None, fallback='MASK_n1'), Rule(v2='MASK_2_Bright_star_halos', source='2_Bright_star_halos', kind='rename', arg=None, fallback='MASK_n2'), Rule(v2='MASK_4_Stars', source='4_Stars', kind='rename', arg=None, fallback='MASK_n4'), Rule(v2='MASK_8_Manual', source='8_Manual', kind='rename', arg=None, fallback='MASK_n8'), Rule(v2='MASK_16_u', source='16_u', kind='rename', arg=None, fallback='MASK_n16'), Rule(v2='MASK_32_g', source='32_g', kind='rename', arg=None, fallback='MASK_n32'), Rule(v2='MASK_64_r', source='64_r', kind='rename', arg=None, fallback='MASK_n64'), Rule(v2='MASK_128_i', source='128_i', kind='rename', arg=None, fallback='MASK_n128'), Rule(v2='MASK_256_z', source='256_z', kind='rename', arg=None, fallback='MASK_n256'), Rule(v2='MASK_512_Tile_RA_DEC_cut', source='512_Tile_RA_DEC_cut', kind='rename', arg=None, fallback='MASK_n512'), Rule(v2='MASK_1024_Maximask', source='1024_Maximask', kind='rename', arg=None, fallback='MASK_n1024'), Rule(v2='MASK_2048_z2', source='2048_z2', kind='rename', arg=None, fallback='MASK_n2048'))

{b}_{label} (v1 data_ext) or MASK_n{b} (pre-release v2) -> MASK_{b}_{label}, applied to every table.

V1_MARKERS = frozenset({'E1_PSF_HSM', 'E1_STAR_HSM', 'E2_PSF_HSM', 'E2_STAR_HSM', 'FLAG_PSF_HSM', 'FLAG_STAR_HSM', 'NGMIX_ELL_1M', 'NGMIX_ELL_1M_0', 'NGMIX_ELL_1M_1', 'NGMIX_ELL_1P', 'NGMIX_ELL_1P_0', 'NGMIX_ELL_1P_1', 'NGMIX_ELL_2M', 'NGMIX_ELL_2M_0', 'NGMIX_ELL_2M_1', 'NGMIX_ELL_2P', 'NGMIX_ELL_2P_0', 'NGMIX_ELL_2P_1', 'NGMIX_ELL_ERR_1M', 'NGMIX_ELL_ERR_1M_0', 'NGMIX_ELL_ERR_1M_1', 'NGMIX_ELL_ERR_1P', 'NGMIX_ELL_ERR_1P_0', 'NGMIX_ELL_ERR_1P_1', 'NGMIX_ELL_ERR_2M', 'NGMIX_ELL_ERR_2M_0', 'NGMIX_ELL_ERR_2M_1', 'NGMIX_ELL_ERR_2P', 'NGMIX_ELL_ERR_2P_0', 'NGMIX_ELL_ERR_2P_1', 'NGMIX_ELL_ERR_NOSHEAR', 'NGMIX_ELL_ERR_NOSHEAR_0', 'NGMIX_ELL_ERR_NOSHEAR_1', 'NGMIX_ELL_NOSHEAR', 'NGMIX_ELL_NOSHEAR_0', 'NGMIX_ELL_NOSHEAR_1', 'NGMIX_ELL_PSFo_1M', 'NGMIX_ELL_PSFo_1M_0', 'NGMIX_ELL_PSFo_1M_1', 'NGMIX_ELL_PSFo_1P', 'NGMIX_ELL_PSFo_1P_0', 'NGMIX_ELL_PSFo_1P_1', 'NGMIX_ELL_PSFo_2M', 'NGMIX_ELL_PSFo_2M_0', 'NGMIX_ELL_PSFo_2M_1', 'NGMIX_ELL_PSFo_2P', 'NGMIX_ELL_PSFo_2P_0', 'NGMIX_ELL_PSFo_2P_1', 'NGMIX_ELL_PSFo_NOSHEAR', 'NGMIX_ELL_PSFo_NOSHEAR_0', 'NGMIX_ELL_PSFo_NOSHEAR_1', 'NGMIX_MOM_FAIL', 'NGMIX_T_PSFo_1M', 'NGMIX_T_PSFo_1P', 'NGMIX_T_PSFo_2M', 'NGMIX_T_PSFo_2P', 'NGMIX_T_PSFo_NOSHEAR', 'NGMIX_Tpsf_1M', 'NGMIX_Tpsf_1P', 'NGMIX_Tpsf_2M', 'NGMIX_Tpsf_2P', 'NGMIX_Tpsf_NOSHEAR', 'SIGMA_PSF_HSM', 'SIGMA_STAR_HSM'})

Names whose presence marks a table as v1 / as v2.

column_names(table)

Return the column names of a structured array, FITS_rec or h5py Dataset.

detect_generation(names)

Return "v1", "v2" or None from the column names present.

None means grammar-neutral: no shape-measurement column either grammar names (e.g. a cut catalogue carrying only RA, Dec, e1, e2, w, or a table of mask columns). Mask columns mark no generation.

Raises:

ValueError – if columns of both grammars are present.

_map_rules(names, column_map)

Return the rename rules column_map yields for a table’s names.

Entries whose file column is absent yield nothing, so one map serves tables that carry only some of its columns.

_resolve(names, column_map=None)

Return (presented names, {v2 name: (rule, source name)}) for names.

v2_names(names, column_map=None)

Return the column names a table with columns names presents.

Header-only counterpart of adapt.

BLOCK_BYTES = 268435456

Bytes of whole HDF5 rows read per block when selecting rows of a dataset.

_read_window(table, name, start, stop)

Read rows start:stop of column name, and only those.

_take(table, rows, fields=None)

Return rows rows of table as an in-memory structured array.

rows is None (all rows), a range or an integer index array; fields restricts the output to those columns. An h5py Dataset is read in one pass: whole rows in blocks of BLOCK_BYTES over the span of the selection, skipping blocks that hold no selected row, so each byte of the file is read at most once however many columns are wanted.

_row_key(rows)

Return rows (a range or index array) as a numpy row index.

class V2View(bases, rows=None, dtype=None, column_map=None)

Bases: object

Lazy v2-grammar view of one table, or of row-aligned tables joined.

Wraps numpy structured arrays, FITS_recs or h5py Datasets without reading them. view[name] returns one column as an array (computed on access for derived columns); a list of names returns a structured array of those columns; view[()], view[...] and any other row key (slice, boolean mask, index array) select rows and return a view over them.

Selecting rows of a view over h5py Datasets reads the selected rows of every column into memory in one pass per dataset, as indexing the Dataset itself would, and wraps them in a view over the in-memory arrays: reading many columns of a selection one at a time would otherwise re-read the whole span of rows for each. Over in-memory tables, selection stays lazy.

Column names are case-sensitive, unlike a FITS_rec’s.

property names

Presented column names.

keys()

Presented column names.

property dtype

Numpy dtype of the presented columns (found without reading rows).

property shape

to_structured(names=None)

Materialise the presented columns as a numpy structured array.

Over h5py Datasets, each dataset is read once, for all the columns requested of it.

_select(key)

Base rows of key applied to this view’s rows.

_read(name)

_column(name)

_field_dtype(name)

Dtype (with any subarray shape) _column(name) returns.

adapt(table, *tables, column_map=None)

Present table (joined with any further tables) in the v2 grammar.

A single table with nothing to rename is returned unchanged, as is a V2View given no column_map; otherwise the result is a V2View. Each table is anything whose dtype.names lists its columns and whose table[name] reads one: a numpy structured array, FITS_rec, h5py Dataset or V2View. Joined tables must have equal lengths and disjoint column names. column_map ({v2 name: name in the table}, see the module docstring) renames a foreign convention’s columns; over a V2View it renames the columns that view presents.

materialise(table, names=None)

Return adapt(table) as an in-memory numpy structured array.

names restricts the output to those v2-grammar columns.