mccd.mccd_utils

MCCD UTILS.

These functions include several functions needed by the MCCD.

Authors: Tobias Liaudat <tobias.liaudat@cea.fr> Samuel Farrens <samuel.farrens@cea.fr>

Notes

I added two functions from Pysap astro plugin and from ModOpt.signal.wavelet so that I could replicate the functionv get_mr_transform() from ModOpt without the need of having sparse2d installed as an exectubale by using Pysap python bindings of the aforementioned C++ package. Thanks Samuel Farrens (main developer of ModOpt and Pysap) for the help.

find_ccd_idx(ccd_id, ccd_list)[source]: Find the index of an element in a list.

class Loc2Glob(x_gap=70, y_gap=425, x_npix=2048, y_npix=4612, ccd_tot=40)[source]

Bases: object

Change from local to global coordinates.

Class to pass from local coordinates to global coordinates under CFIS (CFHT) MegaCam instrument. The geometrical informcation of the instrument is encoded in this function.

Parameters

x_gap (int) – Gap between the CCDs on the horizontal direction. Default to 70 (CFIS value).
y_gap (int) – Gap between the CCDs on the vertical direction. Default to 425 (CFIS value).
x_npix (int) – Number of pixels on one CCD on the horizontal direction. Default to 2048 (CFIS value).
y_npix (int) – Number of pixels on one CCD on the vertical direction. Default to 4612 (CFIS value).
ccd_tot (int) – Total number of CCDs. Default to 40 (CFIS value).

Notes

MegaCams geometry. Watch out with the conventions ba,ab that means where is the local coordinate system origin for each CCD. For more info look MegaCam’s instrument webpage.

Examples

python

>>> 'COMMENT (North on top, East to the left)',
    'COMMENT    --------------------------',
    'COMMENT    ba ba ba ba ba ba ba ba ba',
    'COMMENT    00 01 02 03 04 05 06 07 08',
    'COMMENT --------------------------------',
    'COMMENT ba ba ba ba ba ba ba ba ba ba ba',
    'COMMENT 36 09 10 11 12 13 14 15 16 17 37',
    'COMMENT --------------*-----------------',
    'COMMENT 38 18 19 20 21 22 23 24 25 26 39',
    'COMMENT ab ab ab ab ab ab ab ab ab ab ab',
    'COMMENT --------------------------------',
    'COMMENT    27 28 29 30 31 32 33 34 35',
    'COMMENT    ab ab ab ab ab ab ab ab ab',
    'COMMENT    __________________________'

loc2glob_img_coord(ccd_n, x_coor, y_coor)[source]

Go from the local to the global img (pixel) coordinate system.

Global system with (0,0) in the intersection of ccds [12,13,21,22].

Parameters

ccd_n (int) – CCD number of the considered positions.
x_coor (float) – Local coordinate system hotizontal value.
y_coor (float) – Local coordinate system vertical value.

Returns

glob_x_coor (float) – Horizontal position in global coordinate system.
glob_y_coor (float) – Vertical position in global coordinate system.

flip_coord(ccd_n, x_coor, y_coor)[source]

Change of coordinate convention.

So that all of them are coherent on the global coordinate system. So that the origin is on the south-west corner. Positive: South to North ; West to East.

x_coord_range()[source]: Return range of the x coordinate.

y_coord_range()[source]: Return range of the y coordinate.

shift_coord(ccd_n)[source]

Provide the shifting.

It is needed to go from the local coordinate system origin to the global coordinate system origin.

class Loc2Glob_EUCLID_sim(x_gap=0, y_gap=0, x_npix=4096, y_npix=4096, ccd_tot=36)[source]

Bases: object

Change from local to global coordinates.

Class to pass from local coordinates to global coordinates under a simulation of Euclid’s VIS instrument. The geometrical informcation of the instrument is encoded in this function.

Parameters

x_gap (int) – Gap between the CCDs on the horizontal direction.
y_gap (int) – Gap between the CCDs on the vertical direction.
x_npix (int (float)) – Number of pixels on one CCD on the horizontal direction.
y_npix (int (float)) – Number of pixels on one CCD on the vertical direction.
ccd_tot (int) – Total number of CCDs. Default to 36 (Euclid value).

Notes

The global origin is in the middle of the focal plane. This is the bottom left corner of the CCD 15.

python

>>> 'COMMENT   ------------------------',
    'COMMENT    00  01  02  03  04  05 ',
    'COMMENT   ------------------------',
    'COMMENT    06  07  08  09  10  11 ',
    'COMMENT   ------------^-----------',
    'COMMENT    12  13  14 |15  16  17 ',
    'COMMENT   ------------*->---------',
    'COMMENT    18  19  20  21  22  23 ',
    'COMMENT   ------------------------',
    'COMMENT    24  25  26  27  28  29 ',
    'COMMENT   ------------------------',
    'COMMENT    30  31  32  33  34  35 ',
    'COMMENT   ________________________'

loc2glob_img_coord(ccd_n, x_coor, y_coor)[source]

Go from the local to the global img (pixel) coordinate system.

Global system with (0,0) in the bottom left corner of the CCD 15.

Parameters

ccd_n (int) – CCD number of the considered positions.
x_coor (float) – Local coordinate system hotizontal value.
y_coor (float) – Local coordinate system vertical value.

Returns

glob_x_coor (float) – Horizontal position in global coordinate system.
glob_y_coor (float) – Vertical position in global coordinate system.

glob2loc_img_coord(x_coor, y_coor)[source]: Go from global to local coordinates.

flip_coord(ccd_n, x_coor, y_coor)[source]

Change of coordinate convention.

There is no fliping for Euclid.

x_coord_range()[source]: Return range of the x coordinate.

y_coord_range()[source]: Return range of the y coordinate.

shift_coord(ccd_n)[source]

Provide the shifting.

It is needed to go from the local coordinate system origin to the global coordinate system origin.

class Glob2CCD(loc2glob, with_gaps=True)[source]

Bases: object

Get the CCD ID number from the global coordinate position.

The Loc2Glob() object as input is the one that defines the instrument’s geometry.

Parameters

loc2glob (Loc2Glob object) – Object with the desired focal plane geometry.
with_gaps (bool) – If add the gaps to the CCD area.

build_all_edges()[source]: Build the edges for all the CCDs in the focal plane.

build_edge(ccd_n)[source]: Build the edges of the ccd_n in global coordinates.

is_inside(x, y, edge_x, edge_y)[source]

Is the position inside the edges.

Return True if the position is within the rectangle defined by the edges.

Parameters

x (float) – Horizontal position in global coordinate system.
y (float) – Vertical position in global coordinate system.
edge_x (np.ndarray) – Edge defined as np.array([min_x, max_x]).
edge_y (np.ndarray) – Edge defined as np.array([min_y, max_y]).

get_ccd_n(x, y)[source]

Returns the CCD number from the position (x, y).

Returns None if the position is not found.

class MccdInputs(separator='-', coord_x_descriptor='XWIN_IMAGE', coord_y_descriptor='YWIN_IMAGE', mask_thresh=- 100000.0, loc2glob=None, fits_tb_pos=2)[source]

Bases: object

Handle inputs for the MCCD algorithm.

This method was set up to work with a catalog outputed by SExtractor but it should not be restricted to that. The input files are generally created one per exposure per CCD. Methods like PSFEx that build one model per exposure per CCD do not have any preprocessing to do. The MCCD method that uses all the stars in one exposure to build the model needs to have only one file that will contain all the information gathered in one exposure. This is the main reason of this class, to be able to handle the different CCD files from a particular exposure and integrate them in a single file following the MCCD conventions, ie changing the local coordinate system to the global one. The expected files are in fits format. An filename example would be: star_selection-1234567-04.fits Where the exposure ID is: 1234567 Where the CCD ID is: 04

Parameters

separator (str) – String separating the file names. Between the file pattern, the exposure ID and the CCD ID. Default is '-'.
coord_x_descriptor (str) – Name of the fits column to be used as the horizontal position coordinate. Default is 'XWIN_IMAGE'.
coord_y_descriptor (str) – Name of the fits column to be used as the vertical position coordinate. Default is 'YWIN_IMAGE'.
mask_thresh (float) – Threshold to be used when constructing the masks. SExtractor has the particular way of masking pixels by setting pixels to the value -1e30. Values beneath the threshold will be considered masked. Default is -1e5.
loc2glob (object) – The object that allows to do the coordinate conversion from local to global. It is specific for each instrument’s focal plane geometry. If is None it defaults to the CFIS MegaCam instrument.
fits_tb_pos (int) – Position in the fits file of the useful table. Default is 2 that’s the case for the CFIS data.

static handle_mask(stars, thresh, apply_to_stars=False)[source]

Handle and generate masks.

Reads SExtracted star stamps, generates MCCD-compatible masks (that is, binary weights), and replaces bad pixels with 0s - they will not be used by MCCD, but the ridiculous numerical values can otherwise still lead to problems because of convolutions.

Parameters

stars (numpy.ndarray) – Stars to be masked.
thresh (float) – Threshold that will define the mask. Values below the threhsold will be considered masked. Default is -1e5.
apply_to_stars (bool) – Boolean to define if the stars should be thresholded with thresh on top of the mask construction.

parse_path(path)[source]

Parse one path and extract info.

Parameters: path (str) – Path to the file to be parsed in order to extract the exposure ID and the CCD ID.

parse_folder(folder_path, pattern)[source]

Parse a folder that match a specfic pattern.

Separate into a list of np.array each one for each starcat_id present in the folder. Each np.array in the list contains (starcat_id, ccd_n, path).

Parameters

folder_path (str) – Path to the folder to be preprocessed.
pattern (str) – pattern to match on the files found in folder_path.

parse_pipeline_input_list(input_list, element_position=0)[source]

Parse a pipeline input file list.

Separate into a list of np.array for each starcat_id present in the folder.

Parameters

input_list (list of str) – List containing the path to all the files to be processed.
element_position – The element to consider on the input list. If there are multiple elements that share the name pattern and the CCD ID, ie train/test, the element_position determines which one will be used. If there is only one element it should be 0. Default is 0.

Notes

The starcat_list that is saved contains in each element (starcat_id, ccd_n, path).

prep_mccd_inputs(starcat_array)[source]

Prepare the inputs for mccd algorithm.

Taks done:

Translate from local to global coordinate system.
Apply mask to stars.
Normalize star values.
Modify the star format.

Parameters

starcat_array (numpy.ndarray) – Array with (starcat_id, ccd_n, path) for every file in one starcat_id (exposure ID).

Returns

star_list (list) – List containing the masked star stamps.
position_list (list) – List containing the positions in the global coordinate system.
mask_list (list) – List containing the masks corresponding to the star stamps.
ccd_list (list) – List containing the CCD ids for the stars.
SNR_list (list) – List containing the estimated SNR values for the stars. Will be None if there are no SNR values available.
RA_list (list) – List containing the RA coordinate for the stars. Will be None if there are no RA coordinates available.
DEC_list (list) – List containing the DEC coordinate for the stars. Will be None if there are no DEC coordinates available.

preprocess_data(folder_path, pattern)[source]

Preprocess the data.

Parameters

folder_path (str) – Path to the folder containing the data files.
pattern (str) – Pattern to match with the files in folder_path.

Returns

catalog_ids – List of exposure IDs preprocessed.

Return type

list

proprocess_pipeline_data(input_list, element_position)[source]

Preprocess Shapepipe pipeline’s input file list.

Also return the list of ids.

Parameters

input_list (list of str) – List containing the path to all the files to be processed.
element_position – The element to consider on the input list. If there are multiple elements that share the name pattern and the CCD ID, ie train/test, the element_position determines which one will be used. If there is only one element it should be 0. Default is 0.

Returns

catalog_ids – List of exposure IDs preprocessed.

Return type

list

get_inputs(catalog_id)[source]

Get the MCCD inputs from a specific exposure id.

Returns several lists corresponding to the outputs of the function prep_mccd_inputs.

Parameters: catalog_id (int) – Catalog id (exposure id) to be processed.

get_catalog_ids()[source]: Get preprocessed catalog ids.

static outlier_rejection(star_list, pos_list, mask_list, ccd_list, SNR_list=None, RA_list=None, DEC_list=None, shape_std_max=5.0, print_fun=None)[source]

Outlier star rejection method.

Notes

It is based on the measurements from Galsim’s HSM adaptive moments. The method calculates the 2nd order moments from the stars in the exposure. If there are stars that have an aberrant value in one of the stats, e1, e2 or R2 we discard the star. An aberrant value is defined as a value that is more than shape_std_max sigmas away from the mean.

It inputs all the lists that will be used as a method and returns the same lists without the stars that where considered as outliers.

print_fun is a function that prints details about the stars being removed.

random_indexes(n_tot, train_per=0.8, min_n_train=20)[source]

Generate random indexes to separate datasets.

Separate datasets into train and test sets following some criteria.

Parameters

n_tot (int) – Total number of elements.
train_per (float) – Percentage of those elements to be used for train. Default is 0.8.
min_n_train (float) – Minimum number of elements that should be used for training. Default is 20.

Returns

train_idx (numpy.ndarray) – Array of random indexes used for the training.
test_idx (numpy.ndarray) – Array of random indexes used for the testing.

save_to_fits(dictionary, output_path)[source]

Save dictionary of numpy.ndarray into a fits file.

output_path should be the path + new_name to save the fits and should include .fits extension.

Parameters

dictionary (dict) – Dictionary to be saved to the fits file.
output_path (str) – Should be the concatenation of the path to the folder and the name of the new fits file and should include .fits extension.

Notes

It is important that all the numpy.ndarrays share the first dimension as we are saving a table.

save_fits(dictionary, train_bool, cat_id, output_path)[source]

Save fits file.

Save a dictionary into a fits file format.

Parameters

dictionary (dict) – Dictionary containing the data to be saved.
train_bool (bool) – Bool to determine if it will be a training catalog or a testing catalog. Changes the name pattern used for the file.
cat_id (int or str) – Catalog id (exposure id) to be added in the file name.
output_path (str) – Path to folder to save the new file.

return_loc_neighbors(new_pos, obs_pos, vals, n_neighbors)[source]

Find nearest neighbors locally in one CCD.

Parameters

new_pos (numpy.ndarray) – Array containing the new target position (x,y).
obs_pos (numpy.ndarray) – Array containing the positions of the training stars.
vals (numpy.ndarray) – Values that will be used to interpolate and need to be extracted.
n_neighbors (int) – Number of closest neighbors to return.

Returns

nbs (numpy.ndarray) – Values from vals of the closest n_neighbors from new_pos.
pos (numpy.ndarray) – Positions of the closest n_neighbors from new_pos.

return_glob_neighbors(new_pos, obs_pos_list, val_list, n_neighbors)[source]

Find nearest neighbors locally in all the available CCDs.

Parameters

new_pos (numpy.ndarray) – Array containing the new target position (x,y).
obs_pos_list (list) – List containing the positions of the training stars from all the CCDs.
val_list (list) – List containing the values from all the CCDs that will be used to interpolate and need to be extracted.
n_neighbors (int) – Number of closest neighbors to return.

Returns

values (numpy.ndarray) – Values from vals of the closest n_neighbors from new_pos.
positions (numpy.ndarray) – Positions of the closest n_neighbors from new_pos.

interpolation_Pi(position_list, d_comp_glob, loc2glob=None)[source]

Create polynomial interpolation Pi matrix.

Create a Pi matrix list that will be used for the interpolation of the global model.

Parameters

position_list (list) – List containing the all the positions of the training stars.
d_comp_glob (int) – Maximum polynomial degree to be used for the Pi matrix construction.
loc2glob (object) – The object that allows to do the coordinate conversion from local to global. It is specific for each instrument’s focal plane geometry. If is None it defaults to the CFIS MegaCam instrument.

Returns

interp_Pi – List containing all the Pi polynomial matrices, one for each CCD (object in the position list).

Return type

list

trim_filter(filter_array)[source]

Trim the filters to the minimal size.

This method will get rid of the extra zero coefficients in the filter. :param filter_array: The filter to be trimmed :type filter_array: numpy.ndarray

Returns: Trimmed filter
Return type: numpy.ndarray

Notes

Function copied from ModOpt.signal.wavelet as we need it to replicate ModOpt’s get_mr_filters() but using the wavelet transforms from Pysap.

get_mr_filters(data_shape, opt, n_scales=3, coarse=False, trim=False)[source]

Get transform filters.

This method obtains wavelet filters.

Parameters

data_shape (tuple) – 2D data shape
opt (str) – Name of wavelet transform (in Pysap convention, see Notes)
n_scales (int, optional) – Number of transform scales. Default is 4.
coarse (bool, optional) – Option to keep coarse scale. Default is False.
trim (bool, optional) – Option to trim the filters down to their minimal size Default is False.

Returns

Return type

numpy.ndarray 3D array of wavelet filters.

Notes

Function copied from Pysap package’s astro plugin. Added the trim_filter() functionality from the ModOpt package. The name of the wavelet transform must be in Pysap convention that differs from the sparse2d input arguments. To see the available transforms in Pysap, you need to import the python module (import pysap) and then call pysap.AVAILABLE_TRANSFORMS.

mccd v1.2.4

mccd.mccd_utils

Enter search term