sp_validation.io

IO.

Name:

io.py

Description:

Input and output: output directories and statistics files, and the catalogue reader (read_catalogue, Catalogue), the one place that knows how a catalogue’s rows are stored on disk.

Author:

Martin Kilbinger

make_out_dirs(output_dir, plot_dir, plot_subdirs, verbose=False)

Make output directories.

Create output directories and subdirs

Parameters:

• plot_dir (string) – directory name

• plot_subdirs (array of string) – subdirectory names

• verbose (bool, optional, default=False) – verbose output if True

open_stats_file(directory, file_name)

Open statistics file.

Open output file for statistics

Parameters:

• directory (string) – directory

• file_name (string) – file name

print_stats(msg, stats_file, verbose=False)

Print stats.

Print message to stats file.

Parameters:

• msg (string) – message

• stats_file (file handler) – statistics output file

• verbose (bool, optional, default=False) – print message to stdout if True

print_ratio(msg, numerator, denominator, stats_file, verbose=False)

Print Ratio.

pretty-print ratio of two numbers

msgstring

message

numeratorfloat

ratio numerator

denominatorfloat

ratio denominator

stats_filefile handler

output staistic file

verbosebool, optional, default=False

verbose output if True

write_binned_quantity(quantity, key, bin_edges, extra_key='quantity')

read_binned_quantity(filename)

N_UNITS_ATTRS = {'n_exposures': 'exposure', 'n_tiles': 'tile'}

Root attributes in which a chunked ShapePipe product declares its chunk count, with the name of one chunk (for error messages).

class Catalogue(path, hdu=None, column_map=None)

Bases: object

An open catalogue file, presented as row chunks in the v2 grammar.

The container is detected from the file contents (h5py.is_hdf5), not from its extension. Supported layouts:

• FITS: one table HDU, hdu or by default the first table HDU;

• HDF5 comprehensive catalogue: a root data dataset, joined column-wise with data_ext when present;

• HDF5 single dataset at the root, of any name;

• HDF5 group of per-unit row chunks (one dataset per tile or exposure, as ShapePipe writes campaign galaxy and star catalogues), found by find_dataset_group, concatenated in dataset-name order, and checked against a declared n_tiles or n_exposures count.

Nothing is read on opening: chunks maps each chunk name to a lazy table. Use as a context manager, or close when done; tables read into memory stay valid after closing.

Parameters:

• path (str or os.PathLike) – catalogue file

• hdu (int, optional) – FITS HDU to read; ignored for HDF5. Default is None, the first table HDU

• column_map (dict, optional) – {v2 name: name in the file} for a catalogue in another naming convention, applied by grammar.adapt (patterns such as {"NGMIX_*": "MYFIT_*"} allowed). Default is None

close()

Close the underlying file; a no-op once closed, and for FITS.

dtype(columns=None)

Return the structured dtype read(columns) produces.

Each column’s dtype is promoted across every chunk (group_dtype).

Raises:

KeyError – if a chunk lacks one of columns, naming the chunk

table()

Return the catalogue as one table, lazily when it is not chunked.

An unchunked catalogue is returned as its lazy table (a FITS_rec, an h5py Dataset or a grammar.V2View), valid while the file is open; a chunked one is read into memory (read).

iter_chunks(columns=None, verbose=False)

Yield the row chunks one at a time, in memory, restricted to columns.

A caller filling a preallocated array (len and dtype give its shape) so never holds more than one chunk on top of its output.

read(columns=None, key_column=None, verbose=False)

Read the catalogue into one numpy structured array.

Chunks are copied one at a time into a preallocated output, so peak memory is the output plus one chunk.

Parameters:

• columns (list of str, optional) – v2-grammar columns to read; default is None (all)

• key_column (str, optional) – for a chunked catalogue, name of an integer column to add, holding for each row the leading integer of its chunk’s name. ShapePipe star catalogues name each chunk by its exposure number ("2086324", or "2110000p" with the CFIS processed-exposure suffix), which this keeps (key_column="EXPID"). An unchunked catalogue has no chunk names and gets no column. Default is None

• verbose (bool, optional) – report progress if True

Returns:

catalogue data in the v2 grammar

Return type:

numpy.ndarray

Raises:

• KeyError – if a chunk lacks one of columns

• ValueError – if key_column collides with a column, or a chunk name does not begin with an integer

_load(key)

Return chunk key in memory, or lazily when the file is unchunked.

A chunk (one tile or exposure) is small, so it is read whole, which is faster than column by column; an unchunked table is read per column.

read_catalogue(path, columns=None, hdu=None, column_map=None, key_column=None, verbose=False)

Read a catalogue into memory, in the v2 column grammar.

Any container Catalogue supports; columns restricts what is read. A catalogue in another naming convention keeps its own column names, unless column_map renames them. See Catalogue and Catalogue.read for the parameters.

Returns:

catalogue data

Return type:

numpy.ndarray or astropy.io.fits.FITS_rec

open_catalogue(path, hdu=None, column_map=None)

Return a catalogue as one table, reading no data up front.

A FITS table is memory-mapped and an unchunked HDF5 table read column by column on access (its file stays open while the table is referenced), so a consumer pays only for the columns it indexes, derived v2 columns included. A chunked HDF5 catalogue has no single on-disk table and is read into memory. Parameters as for Catalogue.

Returns:

catalogue data in the v2 grammar

Return type:

astropy.io.fits.FITS_rec, h5py.Dataset, grammar.V2View or numpy.ndarray

open_entry(entry)

Return the catalogue a config block names (e.g. a cat_config shear).

The block’s path, optional hdu and optional column_map go to open_catalogue.

_first_table_hdu(hdus, path)

_hdf5_parts(hdf5_file, path)

Return ({chunk name: tuple of row-aligned datasets}, chunked) of a file.

_chunk_numbers(keys, key_column, dtype, path)

Return {chunk name: its leading integer}, for Catalogue.read.

_check_columns(dtype, columns, path, chunk=None)

Raise a clear error if requested columns are absent from a chunk.

find_dataset_group(hdf5_file)

Find Dataset Group.

Descend from the root of an open HDF5 file to the single group whose members are the per-unit datasets (one per tile, or one per exposure).

This makes the reader independent of how deeply the products nest that group: it walks down as long as the current node holds exactly one sub-group, and stops as soon as the members are datasets. It therefore reads both the nested patches/<campaign>/<tile-ID> layout (the “patches” key is a ShapePipe-side compatibility shim, not a concept) and a flat tiles/<tile-ID> or exposures/<exp> layout.

Parameters:

hdf5_file (h5py.File or h5py.Group) – open input file

Returns:

group whose members are the per-unit datasets

Return type:

h5py.Group

Raises:

ValueError – if the file is empty, or a level holds more than one sub-group

promote_dtypes(dtype_a, dtype_b)

Promote Dtypes.

Return a scalar dtype that holds both input dtypes without truncation or overflow.

Parameters:

• dtype_a (numpy.dtype) – first input dtype

• dtype_b (numpy.dtype) – second input dtype

Returns:

promoted dtype

Return type:

numpy.dtype

group_dtype(tables, columns=None)

Group Dtype.

Build the structured output dtype of a group of row-chunk tables, promoting each column across every table. A campaign that was partially reprocessed can carry e.g. S7 tile IDs in one tile and S12 in another, or f4 next to f8; taking the dtype of the first table alone would silently truncate the others.

Note that ShapePipe currently writes TILE_ID as f8 (the tile 183.307 arrives as the float 183.307), not as a string, so the string-promotion branch above is for a future string-valued TILE_ID and is not exercised by today’s products.

TODO: whether TILE_ID should be a string is an open schema decision. A float cannot represent the ID exactly and cannot be compared for equality safely; changing it is a breaking product change, so it is left as-is here and this reader deliberately handles both.

Parameters:

• tables (list or dict) – row-chunk tables (h5py Datasets, or their grammar.adapt views), optionally keyed by dataset name for error messages

• columns (list of str, optional) – columns to keep; default is None (all columns of the first table)

Returns:

structured output dtype

Return type:

numpy.dtype

Raises:

KeyError – if a table lacks one of the columns, naming the tables that do

check_n_units(hdf5_file, group, file_path)

Check Number of Units.

Compare the number of datasets found against the count a ShapePipe product declares in a root attribute (N_UNITS_ATTRS: n_tiles for galaxy catalogues, n_exposures for star catalogues), to catch a catalogue truncated by an interrupted merge job or file transfer.

A missing attribute is not an error: older products carry none.

Parameters:

• hdf5_file (h5py.File) – open input file

• group (h5py.Group) – group holding the per-unit datasets

• file_path (str) – input file path, for the error message

Raises:

ValueError – if the number of datasets differs from the declared count