pipeline.h.tasks.common package

Submodules

pipeline.h.tasks.common.arrayflaggerbase module

class pipeline.h.tasks.common.arrayflaggerbase.FlagCmd(filename=None, rulename=None, ruleaxis=None, spw=None, antenna=None, intent=None, pol=None, time=None, field=None, axisnames=None, flagcoords=None, channel_axis=None, reason=None, extendfields=None, antenna_id_to_name=None)[source]

Bases: object

Create a flagcmd.

Added detailed docs here.

property flagchannels

Return list of channels flagged.

match(spectrum)[source]

Return True if the FlagCmd operates on this SpectrumResult.

match_image(image)[source]

Return True if the FlagCmd operates on this ImageResult.

pipeline.h.tasks.common.arrayflaggerbase.channel_ranges(channels)[source]

Given a list of channels will return a list of ranges that describe them.

pipeline.h.tasks.common.arrayflaggerbase.consolidate_flagcmd_channels(flagcmds, antenna_id_to_name=None)[source]

Method to consolidate multiple flagcmds that specify a single channel into fewer flagcmds with channel ranges.

pipeline.h.tasks.common.arrayflaggerbase.median_and_mad(data)[source]

Return the median and MAD of the numpy data array.

pipeline.h.tasks.common.atmutil module

class pipeline.h.tasks.common.atmutil.AtmType[source]

Bases: object

midLatitudeSummer = 2
midLatitudeWinter = 3
subarcticSummer = 4
subarcticWinter = 5
tropical = 1
pipeline.h.tasks.common.atmutil.calc_airmass(elevation=45.0)[source]

elevation: elevation [deg]

pipeline.h.tasks.common.atmutil.calc_transmission(airmass, dry_opacity, wet_opacity)[source]
pipeline.h.tasks.common.atmutil.get_dry_opacity(at)[source]

at: atmosphere tool

pipeline.h.tasks.common.atmutil.get_median_elevation(vis, antenna_id)[source]
pipeline.h.tasks.common.atmutil.get_spw_spec(vis, spw_id)[source]

vis: MS name spw_id: spw id

return: center frequency [GHz], number of channels, and resolution [GHz]

pipeline.h.tasks.common.atmutil.get_transmission(vis, antenna_id=0, spw_id=0, doplot=False)[source]

calculate atmospheric transmission assuming PWV=1mm.

vis – MS name antenna_id – antenna ID spw_id – spw ID

Returns

(frequency array [GHz], atm transmission)

pipeline.h.tasks.common.atmutil.get_wet_opacity(at)[source]

at: atmosphere tool

pipeline.h.tasks.common.atmutil.init_at(at, humidity=20.0, temperature=270.0, pressure=560.0, atmtype=3, altitude=5000.0, fcenter=100.0, nchan=4096, resolution=0.001)[source]

at: atmosphere tool humidity: relative humidity [%] temperature: temperature [K] pressure: pressure [mbar] atmtype: AtmType enum altitude: altitude [m] fcenter: center frequency [GHz] nchan: number of channels resolution: channel width [GHz]

pipeline.h.tasks.common.atmutil.plot(frequency, dry_opacity, wet_opacity, transmission)[source]
pipeline.h.tasks.common.atmutil.test(pwv=1.0, elevation=45.0)[source]

pwv: water vapor content [mm] elevation: elevation [deg]

pipeline.h.tasks.common.calibrationtableaccess module

class pipeline.h.tasks.common.calibrationtableaccess.CalibrationTableData(name)[source]

Bases: object

CalibrationTableData holds the name of a CASA calibration table and its contents.

addrow(columns, *coldata)[source]
property rows
setvis(vis)[source]
class pipeline.h.tasks.common.calibrationtableaccess.CalibrationTableDataFiller[source]

Bases: object

CalibrationTableDataFiller contains methods to fill a CalibrationTableData object.

static getcal(caltable, columns=[])[source]
class pipeline.h.tasks.common.calibrationtableaccess.CalibrationTableRow(columns, *coldata)[source]

Bases: object

CalibrationTableRow represents a row in a CASA calibration table.

get(field)[source]

pipeline.h.tasks.common.commonfluxresults module

class pipeline.h.tasks.common.commonfluxresults.FluxCalibrationResults(vis, resantenna=None, uvrange=None, measurements=None)[source]

Bases: pipeline.infrastructure.basetask.Results

TODO: refactor this results structure. We’re falling into the dictionaries of dictionaries trap again..

merge_with_context(context)[source]

Merge these results with the given context.

This method will be called during the execution of accept(). For calibration tasks, a typical implementation will register caltables with the pipeline callibrary.

At this point the result is deemed safe to merge, so no further checks on the context need be performed.

Parameters

context (Context) – the target Context

pipeline.h.tasks.common.commonhelpermethods module

pipeline.h.tasks.common.commonhelpermethods.get_antenna_names(ms)[source]

Get antenna names.

pipeline.h.tasks.common.commonhelpermethods.get_corr_axis(ms, spwid)[source]

Get names of polarizations

pipeline.h.tasks.common.commonhelpermethods.get_corr_products(ms, spwid)[source]

Get names of corr products stored in ms.

pipeline.h.tasks.common.commonresultobjects module

class pipeline.h.tasks.common.commonresultobjects.ImageResult(filename, data, datatype, axes, flag=None, nodata=None, intent=None, field_id=None, field_name=None, spw=None, pol=None, ant=None, units=None, time=None)[source]

Bases: pipeline.h.tasks.common.commonresultobjects.ResultBase

setflags(axisname=None, indices=None)[source]

Set the flag array for specified axis values.

class pipeline.h.tasks.common.commonresultobjects.QaResult[source]

Bases: object

addview(description, viewresult)[source]
descriptions()[source]
first(description)[source]
flagcmds()[source]
last(description)[source]
class pipeline.h.tasks.common.commonresultobjects.ResultAxis(name, units, data, channel_width=None)[source]

Bases: object

class pipeline.h.tasks.common.commonresultobjects.ResultBase[source]

Bases: object

property description
property fieldname
class pipeline.h.tasks.common.commonresultobjects.SpectrumResult(data, datatype, data_mad=None, axis=None, flag=None, nodata=None, noisychannels=None, filename=None, intent=None, field_id=None, field_name=None, spw=None, pol=None, ant=None, units=None, time=None, normalise=False)[source]

Bases: pipeline.h.tasks.common.commonresultobjects.ResultBase

pipeline.h.tasks.common.flaggableviewresults module

class pipeline.h.tasks.common.flaggableviewresults.FlaggableViewResults[source]

Bases: object

add_flag_reason_plane(flag_reason_plane, flag_reason_key)[source]
addflags(flags)[source]
addview(description, viewresult)[source]
descriptions()[source]
first(description)[source]
flagcmds()[source]
flagged()[source]
importfrom(result)[source]
last(description)[source]
sort_flagcmds()[source]

pipeline.h.tasks.common.flagging_renderer_utils module

Created on 25 Nov 2014

@author: sjw

pipeline.h.tasks.common.flagging_renderer_utils.FlagTotal

alias of pipeline.h.tasks.common.flagging_renderer_utils.FlagSummary

pipeline.h.tasks.common.flagging_renderer_utils.adjust_non_science_totals(flagtotals, non_science_agents=None)[source]

Return a copy of the FlagSummary dictionaries, with totals reduced to account for flagging performed by non-science flagging agents.

The incoming flagtotals report how much data was flagged per agent per data selection. These flagtotals are divided into two groups: those whose agent should be considered ‘non-science’ (and are indicated as such in the non_science_agents argument) and the remainder. The total number of rows flagged due to non-science agents is calculated and subtracted from the total for each of the remainder agents.

pipeline.h.tasks.common.flagging_renderer_utils.flags_by_intent(ms, summaries, intents=None)[source]
pipeline.h.tasks.common.flagging_renderer_utils.flags_by_science_spws(ms, summaries)[source]
pipeline.h.tasks.common.flagging_renderer_utils.flags_for_result(result, context: pipeline.infrastructure.launcher.Context, intents_to_summarise: Optional[List[str]] = None, non_science_agents: Optional[List[str]] = None)[source]
pipeline.h.tasks.common.flagging_renderer_utils.intents_to_summarise(context: pipeline.infrastructure.launcher.Context) → List[str][source]

pipeline.h.tasks.common.manifest module

class pipeline.h.tasks.common.manifest.PipelineManifest(ouss_id)[source]

Bases: object

Class for creating the pipeline data product manifest

static add_antennapos_file(ous, antennapos_file)[source]

Add the antenna positions file to the OUS element Remove at some point

static add_aqua_report(ous, aqua_report)[source]

Add the AQUA report to the OUS element

static add_asdm(session, asdm_name, flags_file, calapply_file)[source]

Add an ASDM element to a SESSION element

static add_asdm_imlist(session, asdm_name, ms_file, flags_file, calapply_file, imagelist, imtype)[source]

Add an ASDM element to a SESSION element

static add_aux_products_file(ous, auxproducts_file)[source]

Add the auxiliary products file. Is one enough ?

static add_auxasdm(session, asdm_name, calapply_file)[source]

Add an ASDM element to a SESSION element

static add_auxcaltables(session, caltables_file)[source]
static add_caltables(session, caltables_file)[source]
static add_casa_cmdlog(ous, casa_cmdlog)[source]

Add the CASA commands log to the OUS element

static add_casa_version(ous, casa_version)[source]

Set the CASA version

static add_cont_file(ous, cont_file)[source]

Add the continuum frequency ranges file to the OUS element Remove at some point

static add_environment_info(ous)[source]
static add_execution_node(root, host_details)[source]
static add_flux_file(ous, flux_file)[source]

Add the flux file to the OUS element Remove at some point.

static add_images(ous, imagelist, imtype)[source]

Add a list of images to the OUS element. Note that this does not have to be an ous element, e.d. an asdm element will do

static add_pipeline_version(ous, pipeline_version)[source]

Set the pipeline version

static add_pipescript(ous, pipescript)[source]

Add the pipeline processing script to the OUS element

static add_pprfile(ous, ppr_file)[source]

Add the pipeline processing request file to the OUS element

static add_procedure_name(ous, procedure_name)[source]

Set the procedure name

static add_restorescript(ous, restorescript)[source]

Add the pipeline restore script to the OUS element

static add_weblog(ous, weblog)[source]

Add the weblog to the OUS element

static get_applycals(ous)[source]

Get a list of the final applycal instructions

static get_asdm(session, asdm_name)[source]

Get an ASDM element in a SESSION element and return it

static get_caltables(ous)[source]
static get_casa_version(ous)[source]

Get the CASA version

static get_execution_nodes(ous)[source]
static get_final_flagversions(ous)[source]

Get a list of the final flag versions

get_ous()[source]

Currently this assumes there is only one ous as is the case for member ous processing

static get_pipeline_version(ous)[source]

Get the pipeline version

static get_session(ous, session_name)[source]

Get a SESSION element in an OUS element and return it

import_xml(xmlfile)[source]

Import the manifest from an existing manifest file

set_ous(ous_name)[source]

Set an OUS element and return it

static set_session(ous, session_name)[source]

Set a SESSION element in an OUS element and return it

write(filename)[source]

Convert the document to a nicely formatted XML string and save it in a file

pipeline.h.tasks.common.ozone module

pipeline.h.tasks.common.ozone.get_ozone_channels_for_spw(ms, spwid)[source]

For given spectral window in measurement set, compute an array of booleans where each element corresponds to a channel in the spectral window and signifies whether that channel falls inside a known atmospheric ozone line.

Parameters

  • ms (MeasurementSet) – measurement set to inspect

  • spwid (int) – spectral window id to inspect

Returns

Numpy array of booleans, representing mask of channels where channels corresponding to ozone lines are set to True.

Return type

numpy.array[boolean]

pipeline.h.tasks.common.sensitivity module

class pipeline.h.tasks.common.sensitivity.Sensitivity(*args, **kwargs)[source]

Bases: dict

Sensitivity template definition.

pipeline.h.tasks.common.viewflaggers module

class pipeline.h.tasks.common.viewflaggers.MatrixFlagger(inputs)[source]

Bases: pipeline.infrastructure.basetask.StandardTaskTemplate

Inputs

alias of MatrixFlaggerInputs

analyse(result)[source]

Determine the best parameters by analysing the given jobs before returning any final jobs to execute.

Parameters

jobs (a list ofJobRequest) – the job requests generated by prepare()

Return type

Result

flag_reason_index = {'bad antenna': 9, 'bad quadrant': 8, 'high outlier': 5, 'low outlier': 6, 'max abs': 1, 'min abs': 2, 'nmedian': 3, 'outlier': 4, 'too many entirely flagged': 10, 'too many flags': 7}
flag_reason_key = {1: 'max abs', 2: 'min abs', 3: 'nmedian', 4: 'outlier', 5: 'high outlier', 6: 'low outlier', 7: 'too many flags', 8: 'bad quadrant', 9: 'bad antenna', 10: 'too many entirely flagged'}
flag_view(view, rules)[source]
generate_flags(matrix, rules)[source]

Calculate the statistics of a matrix and flag the data according to a list of specified rules.

Keyword arguments: matrix - ImageResult object containing data to be flagged. rules - Rules to be applied.

static make_flag_rules(flag_hilo=False, fhl_limit=5.0, fhl_minsample=5, flag_hi=False, fhi_limit=5.0, fhi_minsample=5, flag_lo=False, flo_limit=5.0, flo_minsample=5, flag_tmf1=False, tmf1_axis='Time', tmf1_limit=1.0, tmf1_excess_limit=10000000, flag_tmf2=False, tmf2_axis='Time', tmf2_limit=1.0, tmf2_excess_limit=10000000, flag_tmef1=False, tmef1_axis='Antenna1', tmef1_limit=1.0, flag_nmedian=False, fnm_lo_limit=0.7, fnm_hi_limit=1.3, flag_maxabs=False, fmax_limit=0.1, flag_minabs=False, fmin_limit=0.0, flag_bad_quadrant=False, fbq_hilo_limit=7.0, fbq_antenna_frac_limit=0.5, fbq_baseline_frac_limit=0.5, flag_bad_antenna=False, fba_lo_limit=7.0, fba_frac_limit=0.05, fba_number_limit=3, fba_minsample=5)[source]

Generate a list of flagging rules from a set of flagging parameters. Added detailed docs here.

prepare()[source]

Prepare job requests for execution.

Parameters

parameters – the parameters to pass through to the subclass. Refer to the implementing subclass for specific information on what these parameters are.

Return type

a class implementing Result

set_flags(flags, summarize_before=False, summarize_after=False)[source]
class pipeline.h.tasks.common.viewflaggers.MatrixFlaggerInputs(context, output_dir=None, vis=None, datatask=None, viewtask=None, flagsettertask=None, rules=None, niter=None, extendfields=None, extendbaseband=None, iter_datatask=None, use_antenna_names=None, prepend=None, skip_fully_flagged=None)[source]

Bases: pipeline.infrastructure.vdp.StandardInputs

prepend

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

  1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

  2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

  3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

skip_fully_flagged

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

  1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

  2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

  3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

use_antenna_names

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

  1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

  2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

  3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

class pipeline.h.tasks.common.viewflaggers.MatrixFlaggerResults(vis=None)[source]

Bases: pipeline.infrastructure.basetask.Results, pipeline.h.tasks.common.flaggableviewresults.FlaggableViewResults

merge_with_context(context)[source]

Merge these results with the given context.

This method will be called during the execution of accept(). For calibration tasks, a typical implementation will register caltables with the pipeline callibrary.

At this point the result is deemed safe to merge, so no further checks on the context need be performed.

Parameters

context (Context) – the target Context

class pipeline.h.tasks.common.viewflaggers.VectorFlagger(inputs)[source]

Bases: pipeline.infrastructure.basetask.StandardTaskTemplate

Inputs

alias of VectorFlaggerInputs

analyse(result)[source]

Determine the best parameters by analysing the given jobs before returning any final jobs to execute.

Parameters

jobs (a list ofJobRequest) – the job requests generated by prepare()

Return type

Result

flag_view(view)[source]
generate_flags(vector)[source]

Calculate the statistics of a vector and flag the data according to a list of specified rules.

Keyword arguments: vector - SpectrumResult object containing data to be flagged. rules - Rules to be applied.

static make_flag_rules(flag_edges=False, edge_limit=2.0, flag_minabs=False, fmin_limit=0.0, flag_nmedian=False, fnm_lo_limit=0.7, fnm_hi_limit=1.3, flag_hilo=False, fhl_limit=5.0, fhl_minsample=5, flag_sharps=False, sharps_limit=0.05, flag_diffmad=False, diffmad_limit=10, diffmad_nchan_limit=4, flag_tmf=None, tmf_frac_limit=0.1, tmf_nchan_limit=4)[source]

Generate a list of flagging rules from a set of flagging parameters. Added detailed docs here.

prepare()[source]

Prepare job requests for execution.

Parameters

parameters – the parameters to pass through to the subclass. Refer to the implementing subclass for specific information on what these parameters are.

Return type

a class implementing Result

set_flags(flags, summarize_before=False, summarize_after=False)[source]
class pipeline.h.tasks.common.viewflaggers.VectorFlaggerInputs(context, output_dir=None, vis=None, datatask=None, viewtask=None, flagsettertask=None, rules=None, niter=None, iter_datatask=None, use_antenna_names=None, prepend=None)[source]

Bases: pipeline.infrastructure.vdp.StandardInputs

prepend

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

  1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

  2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

  3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

use_antenna_names

VisDependentProperty is a Python data descriptor that standardises the behaviour of pipeline Inputs properties and lets them create default values more easily.

On reading a VisDependentProperty (ie. using the dot prefix: inputs.solint), one of two things happens:

  1. If a NullMarker is found - signifying that no user input has been provided - and a ‘getter’ function has been defined, the getter function will be called to provide a default value for that measurement set.

  2. If a user has overridden the value (eg. inputs.solint = 123), that value will be retrieved.

  3. The value, either the default from step 1 or user-provided from step 2, is run through the optional postprocess function, which gives a final opportunity to change the value depending on the state/value of other properties.

A VisDependentProperty can be made read-only by specifying ‘readonly=True’ when creating the instance.

A VisDependentProperty can be hidden from the containing Inputs string representation by setting ‘hidden=True’ when creating the instance. This will hide the property from the web log and CLI getInputs calls.

Each VisDependentProperty has a set of values it considers equivalent to null. When the user sets the VDP value to one of these null values, the VDP machinery converts this to a private NullObject marker that signifies the property is now unset, resulting in the default value being returned next time the property is read. Developers can specify which values should be converted to NullObject by specifying null_input at creation time, e.g.,

solint = @VisDependentProperty(default=5, null_input=[None, ‘’, ‘RESET’, -1])

class pipeline.h.tasks.common.viewflaggers.VectorFlaggerResults(vis=None)[source]

Bases: pipeline.infrastructure.basetask.Results, pipeline.h.tasks.common.flaggableviewresults.FlaggableViewResults

merge_with_context(context)[source]

Merge these results with the given context.

This method will be called during the execution of accept(). For calibration tasks, a typical implementation will register caltables with the pipeline callibrary.

At this point the result is deemed safe to merge, so no further checks on the context need be performed.

Parameters

context (Context) – the target Context

Module contents