pipeline.hifa.tasks.applycal package

Submodules

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa module

class pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.AntennaFit(ant, pol, amp, phase)

Bases: tuple

property amp

Alias for field number 2

property ant

Alias for field number 0

property phase

Alias for field number 3

property pol

Alias for field number 1

class pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.LinearFitParameters(slope, intercept)

Bases: tuple

property intercept

Alias for field number 1

property slope

Alias for field number 0

class pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.Outlier(vis, intent, scan, spw, ant, pol, num_sigma, reason)

Bases: tuple

property ant

Alias for field number 4

property intent

Alias for field number 1

property num_sigma

Alias for field number 6

property pol

Alias for field number 5

property reason

Alias for field number 7

property scan

Alias for field number 2

property spw

Alias for field number 3

property vis

Alias for field number 0

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.PHASE_REF_FN(all_fits, accessor)

Get the median best fit from a list of best fits.

The accessor argument should be a function that, when given the list of all fits, returns fits of the desired type (amp.slope, phase.intercept, etc.)

Parameters

• all_fits – mixed list of best fits

• accessor – function to filter best fits

Type

list of AntennaFit instances

Returns

median value and uncertainty of fits of selected type

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.ValueAndUncertainty

alias of pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.FitAndError

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.calc_vk(wrapper)

Return a NumPy array containing time-averaged visibilities for each baseline in the input MSWrapper.

Parameters

wrapper – MSWrapper to process

Returns

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.consolidate_data_selections(outliers)

Consolidate a list of Outliers into a smaller set of equivalent Outliers by consolidating their data selection arguments.

This function works by merging Outliers that have the same list of reasons.

Parameters

outliers – an iterable of Outliers

Returns

an equivalent consolidated list of Outliers

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.data_selection_contains(proposed, ds_args)

Return True if one data selection is contained within another.

Parameters

• proposed – data selection 1

• ds_args – data selection 2

Returns

True if data selection 2 is contained within data selection 1

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.fit_angular_model(angular_model, nu, angdata, angsigma)

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.get_amp_fit(amp_model_fn, frequencies, visibilities, sigma)

Fit a linear amplitude vs frequency model to a set of time-averaged visibilities.

Parameters

• amp_model_fn – the amplitude linear model to optimise

• frequencies – numpy array of channel frequencies

• visibilities – numpy array of time-averaged visibilities

• sigma – numpy array of uncertainies in time-averaged visibilities

Returns

tuple of best fit params, uncertainty tuple

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.get_angular_linear_function(midpoint, x_scale)

Angular linear model to fit phases only.

Parameters

• midpoint –

• x_scale –

Returns

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.get_best_fits_per_ant(wrapper)

Calculate and return the best amp/phase vs freq fits for data in the input MSWrapper.

This function calculates an independent best fit per polarisation per antenna, returning a list of AntennaFit objects that characterise the fit parameters and fit uncertainties per fit.

Parameters

wrapper – MSWrapper to process

Returns

a list of AntennaFit objects

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.get_chi2_ang_model(angular_model, nu, omega, phi, angdata, angsigma)

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.get_linear_function(midpoint, x_scale)

Return a scaled linear function (a function of slope and intercept).

Parameters

• midpoint –

• x_scale –

Returns

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.get_median_fit(all_fits, accessor)

Get the median best fit from a list of best fits.

The accessor argument should be a function that, when given the list of all fits, returns fits of the desired type (amp.slope, phase.intercept, etc.)

Parameters

• all_fits – mixed list of best fits

• accessor – function to filter best fits

Type

list of AntennaFit instances

Returns

median value and uncertainty of fits of selected type

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.get_phase_fit(amp_model_fn, ang_model_fn, frequencies, visibilities, sigma)

Fit a linear model for phase vs frequency to a set of time-averaged visibilities.

Parameters

• amp_model_fn – model function for amplitude

• ang_model_fn – model function for phase angle

• frequencies – numpy array of channel frequencies

• visibilities – numpy array of time-averaged visibilities

• sigma – numpy array of uncertainies in time-averaged visibilities

Returns

tuple of best fit params, uncertainty tuple

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.robust_stats(a)

Return median and estimate standard deviation of numpy array A using median statistics

Parameters

a –

Returns

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.score_X_vs_freq_fits(all_fits, attr, ref_value_fn, outlier_fn, sigma_threshold)

Score a set of best fits, comparing the fits identified by the ‘attr’ attribute against a reference value calculated by the ref_value_fn, marking outliers that deviate by more than sigma_threshold from this reference value as outliers, to be returned as Outlier objects created by the outlier_fn.

Parameters

• all_fits – a list of fit parameters

• attr – identifier of the fits to consider, e.g., ‘amp.slope’

• ref_value_fn – a function that takes a list of fits and returns a value to be used as a reference value in fit comparisons

• outlier_fn – a function returning Outlier objects

• sigma_threshold – the nsigma threshold to be considered an outlier

Returns

list of Outlier objects

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.score_all(all_fits, outlier_fn, flag_all: bool = False)

Compare and score the calculated best fits based on how they deviate from a reference value.

Setting the test argument flag_all to True sets all fits as outliers. This is useful for testing the QA score roll-up and summary functions in the QA plugin.

Parameters

• all_fits –

• outlier_fn –

• flag_all – True if all fits should be classed as outliers

Returns

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.score_all_scans(ms, intent: str, flag_all: bool = False) → List[pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.Outlier]

Calculate best fits for amplitude vs frequency and phase vs frequency for time-averaged visibilities, score each fit by comparison against a reference value, and return outliers.

Outliers are returned as a list of Outlier objects.

By default, outliers are measured against a PWG-defined threshold for each fit type. Set flag_all to True to make classify all fits as outliers. This is useful when testing QA plugin score roll-up functionality.

Parameters

• ms – MeasurementSet to process

• intent – data intent to process

• flag_all – (optional) True if all fits should be classified as outliers

Returns

outliers that deviate from a reference fit

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.score_amp_intercept(all_fits, outlier_fn, sigma_threshold)

Score the intercept of the best fit against the intercept of the median best fit, marking fits that deviate by sigma_threshold from the median dispersion as outliers. Identified outliers are returned as a list of Outlier object returned by the outlier_fn.

The outlier_fn argument should be a function that returns Outlier objects. In practice, this function should be a partially-applied Outlier constructor that requires a more required arguments to be supplied for an Outlier instance to be created.

Parameters

• all_fits – list of all AntennaFit best fit parameters for all metrics

• outlier_fn – a function returning Outlier objects

• sigma_threshold – the nsigma threshold to be considered an outlier

Returns

list of Outlier objects

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.score_amp_slope(all_fits, outlier_fn, sigma_threshold)

For all amplitude vs frequency fits, score the slope of the fit against the slope of the median best fit, marking fits that deviate by sigma_threshold from the median dispersion as outliers. Identified outliers are returned as a list of Outlier object returned by the outlier_fn.

The outlier_fn argument should be a function that returns Outlier objects. In practice, this function should be a partially-applied Outlier constructor that requires a more required arguments to be supplied for an Outlier instance to be created.

Parameters

• all_fits – list of all AntennaFit best fit parameters for all metrics

• outlier_fn – a function returning Outlier objects

• sigma_threshold – the nsigma threshold to be considered an outlier

Returns

list of Outlier objects

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.score_fits(all_fits, reference_value_fn, accessor, outlier_fn, sigma_threshold)

Score a list of best fit parameters against a reference value, identifying outliers as fits that deviate by more than sigma_threshold * std dev from the reference value.

Parameters

• all_fits – list of AntennaFits

• reference_value_fn – function that returns a reference ValueAndUncertainty from a list of these objects

• accessor – function that returns one LinearFitParameters from an AntennaFit

• outlier_fn – function that returns an Outlier instance

• sigma_threshold – threshold nsigma deviation for comparisons

Returns

list of Outliers

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.score_phase_intercept(all_fits, outlier_fn, sigma_threshold)

For all phase vs frequency fits, score the intercept of the fit against the intercept of the median best fit, marking fits that deviate by sigma_threshold from the median dispersion as outliers. Identified outliers are returned as a list of Outlier object returned by the outlier_fn.

The outlier_fn argument should be a function that returns Outlier objects. In practice, this function should be a partially-applied Outlier constructor that requires a more required arguments to be supplied for an Outlier instance to be created.

Parameters

• all_fits – list of all AntennaFit best fit parameters for all metrics

• outlier_fn – a function returning Outlier objects

• sigma_threshold – the nsigma threshold to be considered an outlier

Returns

list of Outlier objects

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.score_phase_slope(all_fits, outlier_fn, sigma_threshold)

For all phase vs frequency fits, score the slope of the fit against the slope of the median best fit, marking fits that deviate by sigma_threshold from the median dispersion as outliers. Identified outliers are returned as a list of Outlier object returned by the outlier_fn.

The outlier_fn argument should be a function that returns Outlier objects. In practice, this function should be a partially-applied Outlier constructor that requires a more required arguments to be supplied for an Outlier instance to be created.

Parameters

• all_fits – list of all AntennaFit best fit parameters for all metrics

• outlier_fn – a function returning Outlier objects

• sigma_threshold – the nsigma threshold to be considered an outlier

Returns

list of Outlier objects

pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.to_linear_fit_parameters(fit, err)

Convert tuples from the best fit evaluation into a LinearFitParameters namedtuple.

Parameters

• fit – 2-tuple of (slope, intercept) best-fit parameters

• err – 2-tuple of uncertainty in (slope, intercept) parameters

Returns

pipeline.hifa.tasks.applycal.mswrapper module

class pipeline.hifa.tasks.applycal.mswrapper.MSWrapper(scan, spw, filename, data, corr_axis, freq_axis)

Bases: object

MSWrapper is a wrapper around a NumPy array populated with measurement set data for a specified scan and spectral window. The MSWrapper can be filtered on various criteria, e.g, spw, scan, antenna, etc., to narrow the data to a particular data selection.

The static method MSWrapper.create_from_ms should be used to instantiate MSWrapper objects.

static create_from_ms(filename, scan, spw)

Create a new MSWrapper for the specified scan and spw.

Reading in raw measurement set data can be a very memory-intensive process, so data selection is deliberately limited to one scan and one spw at a time.

Parameters

• filename – measurement set filename

• scan – integer scan ID

• spw – integer spw ID

Returns

filter(antenna1=None, antenna2=None, **kwargs)

Return a new MSWrapper containing rows matching the column selection criteria.

Data for rows meeting all the column criteria will be funnelled into the new MSWrapper return object. A boolean AND is effectively performed: e.g., antenna1=3 AND antenna2=5 will only return rows for one baseline.

Data can be filtered on any column listed in the wrapper.data.dtype.

xor_filter(antenna1=None, antenna2=None, **kwargs)

Return a new MSWrapper containing rows matching the column selection criteria.

Data for rows meeting any column criteria will be funnelled into the new MSWrapper return object. A boolean AND is effectively performed: e.g., antenna1=3 AND antenna2=5 will only return rows for one baseline.

Data can be filtered on any column listed in the wrapper.data.dtype.

DANGER! DANGER! DANGER!

NOTE! This class has only been tested for baseline selection! Using this method for other use cases could be dangerous. Use at your own risk!

pipeline.hifa.tasks.applycal.mswrapper.get_dtype(data, column_name)

Get the numpy data type for a CASA caltable column.

Parameters

• tb – CASA table tool with caltable open.

• column_name – name of column to process

Returns

3-tuple of column name, NumPy dtype, column shape

pipeline.hifa.tasks.applycal.qa module

QA plugins for the ALMA applycal task.

This module demonstrates how to target QAScore messages at particular sections of the web log.

class pipeline.hifa.tasks.applycal.qa.ALMAApplycalListQAHandler

Bases: pipeline.infrastructure.pipelineqa.QAPlugin

QA plugin to process lists of ALMA applycal results.

This plugin is required to overwrite how representative is set. We want the representative score to be a headline banner score pulled from the per-EB representative scores, not a lower priority score pulled from the general score pool. The latter would be done by default, hence we overwrite the handle() implementation.

child_cls

alias of pipeline.h.tasks.applycal.applycal.ApplycalResults

generating_task

alias of pipeline.hif.tasks.applycal.ifapplycal.IFApplycal

handle(context, result)

result_cls

alias of collections.abc.Iterable

class pipeline.hifa.tasks.applycal.qa.ALMAApplycalQAHandler

Bases: pipeline.infrastructure.pipelineqa.QAPlugin

QA plugin to handle an applycal result for a single ALMA EB.

This plugin does outlier detection and QA score reprocessing to generate messages targeted at different sections of the web log.

child_cls = None

generating_task

alias of pipeline.hif.tasks.applycal.ifapplycal.IFApplycal

handle(context, result: pipeline.h.tasks.applycal.applycal.ApplycalResults)

result_cls

alias of pipeline.h.tasks.applycal.applycal.ApplycalResults

class pipeline.hifa.tasks.applycal.qa.DataSelection(vis, intent, scan, spw, ant, pol)

Bases: tuple

property ant

Alias for field number 4

property intent

Alias for field number 1

property pol

Alias for field number 5

property scan

Alias for field number 2

property spw

Alias for field number 3

property vis

Alias for field number 0

class pipeline.hifa.tasks.applycal.qa.PIPE356Switches(calculate_metrics, export_outliers, export_messages, include_scores, outlier_score, flag_all)

Bases: tuple

property calculate_metrics

Alias for field number 0

property export_messages

Alias for field number 2

property export_outliers

Alias for field number 1

property flag_all

Alias for field number 5

property include_scores

Alias for field number 3

property outlier_score

Alias for field number 4

class pipeline.hifa.tasks.applycal.qa.QAMessage(ms, outlier, reason)

Bases: object

QAMessage constructs a user-friendly QA message for an Outlier.

The QAMessage instance has two attributes, full_message and short_message, that are of interest. full_message holds the text to be used when the message is the first to be printed. short_message holds the text to be used when this message is to be appended to the text of other QAMessages. Naturally, this assumes the the calling code only concatenates messages that originate from the same reason.

pipeline.hifa.tasks.applycal.qa.combine_scores(all_scores: List[pipeline.infrastructure.pipelineqa.QAScore], hierarchy_base: str, discard: List[str], ms: pipeline.domain.measurementset.MeasurementSet, location: pipeline.infrastructure.pipelineqa.WebLogLocation) → List[pipeline.infrastructure.pipelineqa.QAScore]

Combine and summarise a list of QA scores.

QA scores that share a base metric type and/or differ only in the data selection dimensions given in discard are aggregated and summarised together.

pipeline.hifa.tasks.applycal.qa.compress_data_selections(to_merge: Dict[pipeline.hifa.tasks.applycal.qa.DataSelection, List[pipeline.infrastructure.pipelineqa.QAScore]], attrs_to_merge: Reversible[str]) → Dict[pipeline.hifa.tasks.applycal.qa.DataSelection, List[pipeline.infrastructure.pipelineqa.QAScore]]

Combine adjacent data selections to give a new data structure that expresses the same data selection but in a more compressed form.

A data selection applies over various dimensions: spw, field, pol, etc.. This function merges data selections hierarchically, identifying adjacent data selections per data selection dimension given in attrs_to_merge, and concatenating those adjacent dimension indices together.

pipeline.hifa.tasks.applycal.qa.discard_dimension(to_merge: Dict[pipeline.hifa.tasks.applycal.qa.DataSelection, List[pipeline.infrastructure.pipelineqa.QAScore]], attrs: Iterable[str]) → Dict[pipeline.hifa.tasks.applycal.qa.DataSelection, List[pipeline.infrastructure.pipelineqa.QAScore]]

Aggregate QA scores held in one or more DataSelection dimensions, discarding data selection indices for those dimensions.

This function discards DataSelection dimensions. Say four QA scores were registered, one each to spws 16, 18, 20, and 22. Calling this function with attrs=[‘spw’] would combine those scores into a single DataSelection with spw=’’, i.e., spw data selection is left unspecified.

pipeline.hifa.tasks.applycal.qa.get_qa_scores(ms: pipeline.domain.measurementset.MeasurementSet, export_outliers: bool, outlier_score: float, flag_all: bool)

Calculate amp/phase vs freq outliers for an EB and convert to QA scores.

This is the key entry point for applycal QA metric calculation. It delegates to the detailed metric implementation in ampphase_vs_freq_qa.py to detect outliers, converting the outlier descriptions to normalised QA scores.

pipeline.hifa.tasks.applycal.qa.in_casa_format(data_selections: Dict[pipeline.hifa.tasks.applycal.qa.DataSelection, List[pipeline.infrastructure.pipelineqa.QAScore]]) → Dict[pipeline.hifa.tasks.applycal.qa.DataSelection, List[pipeline.infrastructure.pipelineqa.QAScore]]

Restate data selections in concise CASA format.

pipeline.hifa.tasks.applycal.qa.map_data_selection_to_scores(scores: Iterable[pipeline.infrastructure.pipelineqa.QAScore]) → Dict[pipeline.hifa.tasks.applycal.qa.DataSelection, List[pipeline.infrastructure.pipelineqa.QAScore]]

Expand QAScores to a dict-based data structure that maps data selections to the QA scores applicable to that selection.

Parameters

scores – scores to decompose

pipeline.hifa.tasks.applycal.qa.outliers_to_qa_scores(ms: pipeline.domain.measurementset.MeasurementSet, outliers: List[pipeline.hifa.tasks.applycal.ampphase_vs_freq_qa.Outlier], outlier_score: float) → List[pipeline.infrastructure.pipelineqa.QAScore]

Convert a list of consolidated Outliers into a list of equivalent QAScores.

The MeasurementSet argument is required to convert antenna IDs to antenna names.

All generated QAScores will be assigned the numeric score given in outlier_score.

Parameters

• ms – MeasurementSet domain object for the DataSelections

• outliers – list of Outliers

• outlier_score – score to assign to generated QAScores

Returns

pipeline.hifa.tasks.applycal.qa.summarise_scores(all_scores: List[pipeline.infrastructure.pipelineqa.QAScore], ms: pipeline.domain.measurementset.MeasurementSet) → Dict[pipeline.infrastructure.pipelineqa.WebLogLocation, List[pipeline.infrastructure.pipelineqa.QAScore]]

Process a list of QAscores, replacing the detailed and highly specific input scores with compressed representations intended for display in the web log accordion, and even more generalised summaries intended for display as warning banners.

pipeline.hifa.tasks.applycal.qa.take_min_as_representative(to_merge: Dict[pipeline.hifa.tasks.applycal.qa.DataSelection, List[pipeline.infrastructure.pipelineqa.QAScore]]) → Dict[pipeline.hifa.tasks.applycal.qa.DataSelection, List[pipeline.infrastructure.pipelineqa.QAScore]]

Filter out all but the worst score per data selection.

Note that this function also rewrites QAScore.applies_to to match the data selection the QA score applies to.

This function operates on a dict that maps DataSelections to list of QA scores. For each list, it discards all but the worst score as determined by the metric. The selection will be biased towards metrics whose value distribution tends higher than other metrics, but it seem the best we can do.

pipeline.hifa.tasks.applycal.qa.to_data_selection(tds: pipeline.infrastructure.pipelineqa.TargetDataSelection) → pipeline.hifa.tasks.applycal.qa.DataSelection

Convert a pipeline QA TargetDataSelection object to a DataSelection tuple.

Module contents