shapepipe.pipeline.job_handler¶

JOB HANDLER.

This module defines a class for handling pipeline jobs.

Author: Samuel Farrens <samuel.farrens@cea.fr>

class JobHandler(module, filehd, config, log, job_type='parallel', parallel_mode='smp', batch_size=None, backend=None, timeout=None, verbose=True)[source]¶

Bases: object

Job Handler.

This class handles the submition of jobs to workers distributed among a specified number of CPUs.

Parameters

module (str) – Module name
filehd (FileHandler) – File handler instance
config (CustomParser) – Configuaration parser instance
log (logging.Logger) – Logging instance
job_type (str, optional) – Job type, the default is ‘parallel’
parallel_mode (str, optional) – Parallisation mode, default is ‘smp’
batch_size (int, optional) – Number of jobs to submitted simultaneously, the default is None
backend (str, optional) – Joblib backend, the default is None (which corresponds to ‘loky’)
timeout (int, optional) – Timeout limit for a given job in seconds, the default is None
verbose (bool, optional) – Verbose setting, default is True

property config¶

Set Config.

This method defines the configuation parser instance

Raises: TypeError – For incorrect input type

property log¶

Set Log.

This method defines the logging instance

Raises: TypeError – For incorrect input type

property job_type¶

Set Job Type.

This method defines the job type

Raises: TypeError – For incorrect input type

property parallel_mode¶

Set Parallel Mode.

This method defines the mode of parallelisation.

Raises: TypeError – For incorrect input type

property batch_size¶

Set Batch Size.

This method defines the job batch size.

Raises: ValueError – For invalid batch size value

property backend¶

Set Backend.

This method defines the joblib backend. The default is ‘loky’.

Raises: ValueError – For invalid backend value

property timeout¶

Set Timeout Limit.

This method defines the timeout limit for all jobs.

Raises

TypeError – For incorrect input type
ValueError – For invalid timeout limit value

finish_up()[source]¶

Finish Up.

Finish up JobHandler session.

submit_jobs()[source]¶

Submit Jobs.

Submit jobs in serial or parallel.

static hms2sec(time_str)[source]¶

Convert HMS to Seconds.

Convert a string from hours, minutes and seconds to seconds.

Parameters: time_str (str) – Time string
Returns: Time in seconds
Return type: int

Notes

Time strings should take the form ‘HH:MM:SS’.

_log_job_parameters()[source]¶

Log Job Parameters.

This method logs the job handler instance parameters.

_log_num_processes()[source]¶

Log Number of Processes.

This method logs the number of processes detected for a given module.

_distribute_smp_jobs()[source]¶

Distribute SMP Jobs.

This method distributes the jobs to the workers using SMP.

submit_serial_job()[source]¶

Submit Serial Job.

Submit a single serial job with access to all processes.

_check_for_errors()[source]¶

Check for Errors.

This method checks the worker dictionaries for errors and exceptions.

_check_exception_status()[source]¶

Check Exception Status.

This method checks the worker dictionaries for exceptions raised by Python and logs the instances.

_check_stderr_status()[source]¶

Check STDERR Status.

This method checks the worker dictionaries for errors raised by stderr and logs the instances.

_check_missed_processes()[source]¶

Check Missed Processes.

This method checks the file handler for processes that were not submitted.

clean_up()[source]¶

Finish.

Finish job handler instance.

ShapePipe v1.0

shapepipe.pipeline.job_handler

shapepipe.pipeline.job_handler¶