pipeline.h.tasks.exportdata package

Submodules

pipeline.h.tasks.exportdata.aqua module

Prototype pipeline AQUA report generator

Definitions

Metrics are physical quantities, e.g. phase rms improvement resulting from WVR calibration., % data flagged, etc

Scores are numbers between 0.0 and 1.0 derived from metrics. Not all metrics currently derived in the pipeline are scored.

Structure

The report contains

A project structure section. A QA summary section. A per stage QA section. A per topic QA section.

Issues with the Original Schema / Current Pipeline Design

The per ASDM dimension was ignored.

The multiple metrics / scores per stage and / or per ASDM dimension was ignored.

For stages with single scores / metrics and multiple ASDMs the current report generator selects the MS with the worst metric and reports that value. This metric by definition corresponds to the lowest score.

Stages which generate multiple scores / metrics and multiple ASDMs are currently dealt with on an ad hoc basis.

The scores and metrics are noew stored with the stage results.

Metrics may have units information. They may be encoded as CASA quanta strings if appropriate.

Future Technical Solutions

Suggestions Add a toAqua method to the base results class which returns a list of metrics for export. Pass these to the QA classes for scoring.

Add the euivalent of a toAqua registration method similar to what is done with QA handlers already

class pipeline.h.tasks.exportdata.aqua.AquaXmlGenerator

Bases: object

Class to create the XML for an AQUA pipeline report.

get_calibration_topic(context, topic_results)

Get the XML for the calibration topic.

Parameters

• context – pipeline context

• topic_results – list of Results for this topic

Returns

XML for calibration topic

Return type

xml.etree.cElementTree.Element

get_dataset_topic(context, topic_results)

Get the XML for the dataset topic.

Parameters

• context – pipeline context

• topic_results – list of Results for this topic

Returns

XML for dataset topic

Return type

xml.etree.cElementTree.Element

get_flagging_topic(context, topic_results)

Get the XML for the flagging topic.

Parameters

• context – pipeline context

• topic_results – list of Results for this topic

Returns

XML for flagging topic

Return type

xml.etree.cElementTree.Element

get_imaging_topic(context, topic_results)

Get the XML for the imaging topic.

Parameters

• context – pipeline context

• topic_results – list of Results for this topic

Returns

XML for imaging topic

Return type

xml.etree.cElementTree.Element

get_per_stage_qa(context, all_results)

Get the XML for all stages.

Parameters

• context – pipeline context

• all_results – all Results for this pipeline run

Returns

XML for all stages

Return type

xml.etree.cElementTree.Element

get_project_structure(context)

Get the project structure element.

Given the current data flow it is unclear how the report generator generator will acquire the entity id of the original processing request.

The processing procedure name is known but not yet passed to the pipeline processing request.

Parameters

context – pipeline context

Returns

XML for project structure

Return type

xml.etree.cElementTree.Element

get_qa_summary(context)

Get the AQUA summary XML element.

Parameters

context – pipeline context

Returns

XML summarising execution

Return type

xml.etree.cElementTree.Element

get_report_xml(context)

Create and return the AQUA report XML for the results stored in a context.

Parameters

context – pipeline context to parse

Returns

root XML Element of AQUA report

Return type

xml.etree.cElementTree.Element

get_topics_qa(context, all_results)

Get the XML for all results, divided into sections by topic.

Parameters

• context – pipeline context

• all_results – all Results for this pipeline run

Returns

XML for topics

Return type

xml.etree.cElementTree.Element

class pipeline.h.tasks.exportdata.aqua.GenericMetricXmlGenerator

Bases: pipeline.h.tasks.exportdata.aqua.MetricXmlGenerator

Metric XML Generator that processes any score it is given, formatting the metric value to 3dp.

handles(_)

Returns True if this class can generate XML for the given metric.

Parameters

metric_name – name of metric

Returns

True if metric handled by this class

class pipeline.h.tasks.exportdata.aqua.LowestScoreMetricXmlGenerator(metric_name, formatters=None)

Bases: pipeline.h.tasks.exportdata.aqua.MetricXmlGenerator

Metric XML Generator that only returns XML for the lowest QA score that it handles.

filter(qa_scores)

Reduce a list of entries to those entries that require XML to be generated.

Parameters

qa_scores – list of QAScores

Returns

list of QAScores

class pipeline.h.tasks.exportdata.aqua.MetricXmlGenerator(metric_name, formatters=None)

Bases: object

Creates a AQUA report XML element for QA scores.

filter(qa_scores: List[pipeline.infrastructure.pipelineqa.QAScore]) → List[pipeline.infrastructure.pipelineqa.QAScore]

Reduce a list of entries to those entries that require XML to be generated.

Parameters

qa_scores – list of QAScores

Returns

list of QAScores

handles(metric_name: str) → bool

Returns True if this class can generate XML for the given metric.

Parameters

metric_name – name of metric

Returns

True if metric handled by this class

to_xml(qa_score: pipeline.infrastructure.pipelineqa.QAScore) → xml.etree.ElementTree.ElementTree

Return the XML representation of a QA score and associated metric.

Parameters

qa_score – QA score to convert

Returns

XML element

Return type

xml.etree.ElementTree.Element

pipeline.h.tasks.exportdata.aqua.export_to_disk(report, filename)

Convert an XML document to a nicely formatted XML string and save it in a file.

pipeline.h.tasks.exportdata.aqua.register_aqua_metric(fn)

Register a ‘QA metric to XML’ conversion function.

This function can also be used as a decorator.

Parameters

fn – function to register

Returns

pipeline.h.tasks.exportdata.aqua.sensitivity_xml_for_stages(context, results, name='')

Get the XML for all sensitivities reported by all tasks.

Parameters

• context – pipeline context

• results – all results for the imaging topic

• name – the name of per stage tag (optional)

Returns

XML for sensitivities

Return type

xml.etree.cElementTree.Element

pipeline.h.tasks.exportdata.aqua.vis_to_asdm(vispath)

Get the expected ASDM name from the path of a measurement set.

Parameters

vispath – path to convert

Returns

expected name of ASDM for MS

pipeline.h.tasks.exportdata.aqua.xml_for_sensitivity(d)

Return the XML representation for a sensitivity dictionary.

Parameters

d – sensitivity dict

Returns

XML element

Return type

xml.etree.cElementTree.Element

pipeline.h.tasks.exportdata.aqua.xml_for_sensitivity_stage(context, stage_results, exporter, name)

Translate the sensitivity dictionaries contained in a task result to XML.

Parameters

• context – pipeline context

• stage_results – hifa_preimagecheck result

• exporter – function that returns a list of sensitivity dicts from the result

• name – the name of per stage tag (optional)

Returns

XML for all sensitivities reported by the result stage

Return type

xml.etree.cElementTree.Element

pipeline.h.tasks.exportdata.aqua.xml_generator_for_metric(qa_label, value_spec)

Return a function that converts a matching QAScore to XML.

Parameters

• qa_label – QA metric label to match

• value_spec – string format spec for how to format metric value

Returns

function

pipeline.h.tasks.exportdata.exportdata module

The exportdata module provides base classes for preparing data products on disk for upload to the archive.

To test these classes, register some data with the pipeline using ImportData, then execute:

import pipeline vis = [ ‘<MS name>’ ]

# Create a pipeline context and register some data context = pipeline.Pipeline().context inputs = pipeline.tasks.ImportData.Inputs(context, vis=vis) task = pipeline.tasks.ImportData(inputs) results = task.execute(dry_run=False) results.accept(context)

# Run some other pipeline tasks, e.g flagging, calibration, # and imaging in a similar manner

# Execute the export data task. The details of # what gets exported depends on what tasks were run # previously but may include the following # TBD inputs = pipeline.tasks.exportdata.Exportdata.Inputs(context,

vis, output_dir, sessions, pprfile, products_dir)

task = pipeline.tasks.exportdata.ExportData(inputs)

results = task.execute(dry_run = True)

class pipeline.h.tasks.exportdata.exportdata.ExportData(inputs)

Bases: pipeline.infrastructure.basetask.StandardTaskTemplate

ExportData is the base class for exporting data to the products subdirectory. It performs the following operations:

• Saves the pipeline processing request in an XML file

• Saves the final flags per ASDM in a compressed / tarred CASA flag versions file

• Saves the final calibration apply list per ASDM in a text file

• Saves the final set of caltables per session in a compressed / tarred file containing CASA tables

• Saves the final web log in a compressed / tarred file

• Saves the final CASA command log in a text file

• Saves the final pipeline script in a Python file

• Saves the final pipeline restore script in a Python file

• Saves the images in FITS cubes one per target and spectral window

Inputs

alias of ExportDataInputs

NameBuilder

alias of PipelineProductNameBuiler

analyse(results)

Analyse the results of the export data operation.

This method does not perform any analysis, so the results object is returned exactly as-is, with no data massaging or results items added.

Return type

:class:~`ExportDataResults`

get_oussid(context)

Determine the ous prefix

get_recipename(context)

Get the recipe name

is_multi_vis_task = True

prepare()

Prepare and execute an export data job appropriate to the task inputs.

class pipeline.h.tasks.exportdata.exportdata.ExportDataInputs(context, output_dir=None, session=None, vis=None, exportmses=None, pprfile=None, calintents=None, calimages=None, targetimages=None, products_dir=None, imaging_products_only=None)

Bases: pipeline.infrastructure.vdp.StandardInputs

ExportDataInputs manages the inputs for the ExportData task.

context

the (Context) holding all pipeline state

output_dir

the directory containing the output of the pipeline

session

a string or list of strings containing the sessions(s) associated with each vis. Default to a single session containing all vis. Vis without a matching session are assigned to the last session in the list.

vis

a string or list of strings containing the MS name(s) on which to operate

pprfile

the pipeline processing request.

calintents

the list of calintents defining the calibrator source images to be saved. Defaults to all calibrator intents.

calimages

the list of calibrator source images to be saved. Defaults to all calibrator images matching calintents. If defined overrides calintents and the calibrator images in the context.

targetimages

the list of target source images to be saved. Defaults to all target images. If defined overrides the list of target images in the context.

products_dir

the directory where the data productions will be written

calimages

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

calintents

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

exportcalprods

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

exportmses

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

imaging_products_only

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

pprfile

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

products_dir

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

session

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

targetimages

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

class pipeline.h.tasks.exportdata.exportdata.ExportDataResults(pprequest='', sessiondict=None, msvisdict=None, calvisdict=None, calimages=None, targetimages=None, weblog='', pipescript='', restorescript='', commandslog='', manifest='')

Bases: pipeline.infrastructure.basetask.Results

class pipeline.h.tasks.exportdata.exportdata.PipelineProductNameBuiler

Bases: object

classmethod aqua_report(basename, project_structure=None, ousstatus_entity_id=None, output_dir=None)

classmethod auxiliary_products(basename, ousstatus_entity_id=None, output_dir=None)

classmethod calapply_list(vis, aux_product=False, output_dir=None)

classmethod caltables(ousstatus_entity_id=None, session_name=None, aux_product=False, output_dir=None)

classmethod casa_script(basename, project_structure=None, ousstatus_entity_id=None, output_dir=None)

classmethod manifest(basename, ousstatus_entity_id, output_dir=None)

classmethod weblog(project_structure=None, ousstatus_entity_id=None, output_dir=None)

class pipeline.h.tasks.exportdata.exportdata.StdFileProducts(ppr_file, weblog_file, casa_commands_file, casa_pipescript, casa_restore_script)

Bases: tuple

property casa_commands_file

Alias for field number 2

property casa_pipescript

Alias for field number 3

property casa_restore_script

Alias for field number 4

property ppr_file

Alias for field number 0

property weblog_file

Alias for field number 1

pipeline.h.tasks.exportdata.qa module

class pipeline.h.tasks.exportdata.qa.ExportDataListQAHandler

Bases: pipeline.infrastructure.pipelineqa.QAPlugin

QA handler for a list containing ExportDataResults.

child_cls

alias of pipeline.h.tasks.exportdata.exportdata.ExportDataResults

handle(context, result)

result_cls

alias of collections.abc.Iterable

class pipeline.h.tasks.exportdata.qa.ExportDataQAHandler

Bases: pipeline.infrastructure.pipelineqa.QAPlugin

child_cls = None

handle(context, result)

result_cls

alias of pipeline.h.tasks.exportdata.exportdata.ExportDataResults

Module contents