pulsar.client.staging package

Submodules

pulsar.client.staging.down module

Code run on the client side for unstaging complete Pulsar jobs.

pulsar.client.staging.down.finish_job(client, cleanup_job, job_completed_normally, client_outputs, pulsar_outputs)[source]

Process for “un-staging” a complete Pulsar job.

This function is responsible for downloading results from remote server and cleaning up Pulsar staging directory (if needed.)

pulsar.client.staging.up module

pulsar.client.staging.up.submit_job(client, client_job_description, job_config=None)[source]

Module contents

class pulsar.client.staging.ClientInput(path, input_type, object_store_ref=None)[source]

Bases: object

property action_source
class pulsar.client.staging.ClientInputs(client_inputs)[source]

Bases: object

Abstraction describing input datasets for a job.

static for_simple_input_paths(input_files)[source]
class pulsar.client.staging.ClientJobDescription(command_line, tool=None, config_files=None, input_files=None, client_inputs=None, client_outputs=None, working_directory=None, metadata_directory=None, dependencies_description=None, env=[], arbitrary_files=None, job_directory_files=None, tool_directory_required_files=None, rewrite_paths=True, touch_outputs=None, container=None, remote_pulsar_app_config=None, guest_ports=None)[source]

Bases: object

A description of how client views job - command_line, inputs, etc..

Parameters

command_linestr

The local command line to execute, this will be rewritten for the remote server.

config_fileslist

List of Galaxy ‘configfile’s produced for this job. These will be rewritten and sent to remote server.

input_fileslist

List of input files used by job. These will be transferred and references rewritten.

client_outputsClientOutputs

Description of outputs produced by job (at least output files along with optional version string and working directory outputs.

tool_dirstr

Directory containing tool to execute (if a wrapper is used, it will be transferred to remote server).

working_directorystr

Local path created by Galaxy for running this job (job_wrapper.tool_working_directory).

metadata_directorystr

Local path created by Galaxy for running this job (job_wrapper.working_directory).

dependencies_descriptionlist

galaxy.tools.deps.dependencies.DependencyDescription object describing tool dependency context for remote depenency resolution.

env: list

List of dict object describing environment variables to populate.

version_filestr

Path to version file expected on the client server

arbitrary_filesdict()

Additional non-input, non-tool, non-config, non-working directory files to transfer before staging job. This is most likely data indices but can be anything. For now these are copied into staging working directory but this will be reworked to find a better, more robust location.

rewrite_pathsboolean

Indicates whether paths should be rewritten in job inputs (command_line and config files) while staging files).

property input_files
property output_files
property tool_dependencies
property version_file
class pulsar.client.staging.ClientOutputs(working_directory=None, output_files=[], work_dir_outputs=None, version_file=None, dynamic_outputs=None, metadata_directory=None, job_directory=None, dynamic_file_sources=None)[source]

Bases: object

Abstraction describing the output datasets EXPECTED by the Galaxy job runner client.

dynamic_match(filename)[source]
static from_dict(config_dict)[source]
to_dict()[source]
class pulsar.client.staging.DynamicFileSourceType(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)[source]

Bases: str, Enum

galaxy = 'galaxy'
legacy_galaxy = 'legacy_galaxy'
class pulsar.client.staging.PulsarOutputs(working_directory_contents, output_directory_contents, metadata_directory_contents, job_directory_contents, remote_separator='/', realized_dynamic_file_sources=None)[source]

Bases: object

Abstraction describing the output files PRODUCED by the remote Pulsar server.

static from_status_response(complete_response)[source]
has_output_file(output_file)[source]
output_extras(output_file)[source]

Returns dict mapping local path to remote name.