pulsar.client.staging package

Submodules

pulsar.client.staging.down module

Code run on the client side for unstaging complete Pulsar jobs.

pulsar.client.staging.down.finish_job(client, cleanup_job, job_completed_normally, client_outputs, pulsar_outputs)

Process for “un-staging” a complete Pulsar job.

This function is responsible for downloading results from remote server and cleaning up Pulsar staging directory (if needed.)

pulsar.client.staging.up module

pulsar.client.staging.up.submit_job(client, client_job_description, job_config=None)

Module contents

class pulsar.client.staging.ClientInput(path, input_type, object_store_ref=None)

Bases: object

action_source

class pulsar.client.staging.ClientInputs(client_inputs)

Bases: object

Abstraction describing input datasets for a job.

static for_simple_input_paths(input_files)

class pulsar.client.staging.ClientJobDescription(command_line, tool=None, config_files=None, input_files=None, client_inputs=None, client_outputs=None, working_directory=None, metadata_directory=None, dependencies_description=None, env=[], arbitrary_files=None, job_directory_files=None, tool_directory_required_files=None, rewrite_paths=True, touch_outputs=None, container=None, remote_pulsar_app_config=None, guest_ports=None)

Bases: object

A description of how client views job - command_line, inputs, etc..

Parameters

command_line : str

The local command line to execute, this will be rewritten for the remote server.

config_files : list

List of Galaxy ‘configfile’s produced for this job. These will be rewritten and sent to remote server.

input_files : list

List of input files used by job. These will be transferred and references rewritten.

client_outputs : ClientOutputs

Description of outputs produced by job (at least output files along with optional version string and working directory outputs.

tool_dir : str

Directory containing tool to execute (if a wrapper is used, it will be transferred to remote server).

working_directory : str

Local path created by Galaxy for running this job (job_wrapper.tool_working_directory).

metadata_directory : str

Local path created by Galaxy for running this job (job_wrapper.working_directory).

dependencies_description : list

galaxy.tools.deps.dependencies.DependencyDescription object describing tool dependency context for remote depenency resolution.

env: list

List of dict object describing environment variables to populate.

version_file : str

Path to version file expected on the client server

arbitrary_files : dict()

Additional non-input, non-tool, non-config, non-working directory files to transfer before staging job. This is most likely data indices but can be anything. For now these are copied into staging working directory but this will be reworked to find a better, more robust location.

rewrite_paths : boolean

Indicates whether paths should be rewritten in job inputs (command_line and config files) while staging files).

input_files

output_files

tool_dependencies

version_file

class pulsar.client.staging.ClientOutputs(working_directory=None, output_files=[], work_dir_outputs=None, version_file=None, dynamic_outputs=None, metadata_directory=None, job_directory=None, dynamic_file_sources=None)

Bases: object

Abstraction describing the output datasets EXPECTED by the Galaxy job runner client.

dynamic_match(filename)

static from_dict(config_dict)

to_dict()

class pulsar.client.staging.DynamicFileSourceType

Bases: str, enum.Enum

An enumeration.

galaxy = 'galaxy'

legacy_galaxy = 'legacy_galaxy'

class pulsar.client.staging.PulsarOutputs(working_directory_contents, output_directory_contents, metadata_directory_contents, job_directory_contents, remote_separator='/', realized_dynamic_file_sources=None)

Bases: object

Abstraction describing the output files PRODUCED by the remote Pulsar server.

static from_status_response(complete_response)

has_output_file(output_file)

output_extras(output_file)

Returns dict mapping local path to remote name.