shapepipe.pipeline.job_handler

shapepipe.pipeline.job_handler

JOB HANDLER.

This module defines a class for handling pipeline jobs.

Author

Samuel Farrens <samuel.farrens@cea.fr>

class JobHandler(module, filehd, config, log, job_type='parallel', parallel_mode='smp', batch_size=None, backend=None, timeout=None, verbose=True)[source]

Bases: object

Job Handler.

This class handles the submition of jobs to workers distributed among a specified number of CPUs.

Parameters
  • module (str) – Module name

  • filehd (FileHandler) – File handler instance

  • config (CustomParser) – Configuaration parser instance

  • log (logging.Logger) – Logging instance

  • job_type (str, optional) – Job type, the default is ‘parallel’

  • parallel_mode (str, optional) – Parallisation mode, default is ‘smp’

  • batch_size (int, optional) – Number of jobs to submitted simultaneously, the default is None

  • backend (str, optional) – Joblib backend, the default is None (which corresponds to ‘loky’)

  • timeout (int, optional) – Timeout limit for a given job in seconds, the default is None

  • verbose (bool, optional) – Verbose setting, default is True

property config

Set Config.

This method defines the configuation parser instance

Raises

TypeError – For incorrect input type

property log

Set Log.

This method defines the logging instance

Raises

TypeError – For incorrect input type

property job_type

Set Job Type.

This method defines the job type

Raises

TypeError – For incorrect input type

property parallel_mode

Set Parallel Mode.

This method defines the mode of parallelisation.

Raises

TypeError – For incorrect input type

property batch_size

Set Batch Size.

This method defines the job batch size.

Raises

ValueError – For invalid batch size value

property backend

Set Backend.

This method defines the joblib backend. The default is ‘loky’.

Raises

ValueError – For invalid backend value

property timeout

Set Timeout Limit.

This method defines the timeout limit for all jobs.

Raises
finish_up()[source]

Finish Up.

Finish up JobHandler session.

submit_jobs()[source]

Submit Jobs.

Submit jobs in serial or parallel.

static hms2sec(time_str)[source]

Convert HMS to Seconds.

Convert a string from hours, minutes and seconds to seconds.

Parameters

time_str (str) – Time string

Returns

Time in seconds

Return type

int

Notes

Time strings should take the form ‘HH:MM:SS’.

_log_job_parameters()[source]

Log Job Parameters.

This method logs the job handler instance parameters.

_log_num_processes()[source]

Log Number of Processes.

This method logs the number of processes detected for a given module.

_distribute_smp_jobs()[source]

Distribute SMP Jobs.

This method distributes the jobs to the workers using SMP.

submit_serial_job()[source]

Submit Serial Job.

Submit a single serial job with access to all processes.

_check_for_errors()[source]

Check for Errors.

This method checks the worker dictionaries for errors and exceptions.

_check_exception_status()[source]

Check Exception Status.

This method checks the worker dictionaries for exceptions raised by Python and logs the instances.

_check_stderr_status()[source]

Check STDERR Status.

This method checks the worker dictionaries for errors raised by stderr and logs the instances.

_check_missed_processes()[source]

Check Missed Processes.

This method checks the file handler for processes that were not submitted.

clean_up()[source]

Finish.

Finish job handler instance.