shapepipe.pipeline.worker_handler

shapepipe.pipeline.worker_handler

WORKER HANDLER.

This module defines a class for handling pipeline wokers.

Author

Samuel Farrens <samuel.farrens@cea.fr>

class WorkerHandler(verbose=True)[source]

Bases: object

Worker Handler.

This class defines the worker to process a given job.

worker(process, job_name, w_log_name, run_dirs, config, module_config_sec, timeout, module_runner)[source]

Worker.

This method defines a worker.

Parameters
  • process (numpy.ndarray) – File(s) to be processed

  • w_log_name (str) – Worker log name

  • module_runner (function) – Module runner

  • run_dirs (dict) – Run directories

  • config (CustomParser) – Configuaration parser instance

  • module_config_sec (str) – Configuration file section name

  • timeout (int) – Timeout limit in seconds

Returns

Worker dictionary

Return type

dict

static _set_job_name(num)[source]

Set Job Name.

This method creates a job name for a given process number.

Parameters

num (int) – Process number

Returns

Job name

Return type

str

_prepare_worker(process, job_name, timeout, module)[source]

Prepare Worker.

This method defines a worker instance dictionary.

Parameters
  • Process (str) – File to be processed

  • config (CustomParser) – Configuaration parser instance

  • timeout (int) – Timeout limit in seconds

  • module (str) – Module runner name

_create_worker_log()[source]

Create Worker Log.

This method prepares a logging instance for the worker and logs the worker parameters.

_run_worker()[source]

Run Worker.

This method runs the worker with a given timeout limit and catches the corresponding errors.

_worker_execution()[source]

Worker Execution.

This method executes a worker job and logs the results.

_run_module()[source]

Run Module.

This method runs a module script.

Raises

RuntimeError – For non-existent module runner

_log_stdout()[source]

Log STDOUT.

This method logs the stdout and stderr output of the job.