pipeline.hsd.tasks.common package

Submodules

pipeline.hsd.tasks.common.common module

This module defines common single dish Results and task container Classes.

Classes and Methods:

absolute_path: A method to convert a path to an absolute path. SingleDishResults: Common single dish Results class. ParameterContainerJob: A container class to store task with parameters.

class pipeline.hsd.tasks.common.common.ParameterContainerJob(task: pipeline.infrastructure.basetask.StandardTaskTemplate, **parameters)

Bases: object

A container class to store task class object with parameter values.

Variables

• task – A task class object to execute.

• parameters – A disctionary of task parameters (key: parameter name, value: parameter value) to invoke task.

execute(dry_run: bool = True) → pipeline.infrastructure.basetask.Results

Invoke task using given parameters.

Parameters

dry_run – Run task if False, or only display commands if True.

Retruns:

A Result object of the task.

class pipeline.hsd.tasks.common.common.SingleDishResults(task: Optional[pipeline.infrastructure.basetask.StandardTaskTemplate] = None, success: Optional[bool] = None, outcome: Any = None)

Bases: pipeline.infrastructure.basetask.Results

Single Dish Results class.

This class inherits a common Results class, pipeline.infrastructre.basetask.Results. See the documentation of the super class for details. The class is usually used as a superclass of task specific Results class.

Variables

• task – A task class associated with this class.

• success – A boolean to indicate if the task execution was successful (True) or not (False).

• outcome – Outcome of the task.

• error – A set of strings to store error messages.

merge_with_context(context: pipeline.infrastructure.launcher.Context)

Merge these results with the given context.

See the documenetation of super class for more details.

pipeline.hsd.tasks.common.common.absolute_path(name: str) → str

Return an absolute path of a given file.

pipeline.hsd.tasks.common.compress module

class pipeline.hsd.tasks.common.compress.CompressedIter(obj)

Bases: object

class pipeline.hsd.tasks.common.compress.CompressedList

Bases: list

class pipeline.hsd.tasks.common.compress.CompressedObj(obj, protocol=4, compresslevel=9)

Bases: object

decompress()

pipeline.hsd.tasks.common.compress.compress_object(obj, protocol=4, compresslevel=9)

pipeline.hsd.tasks.common.compress.decompress_object(obj)

pipeline.hsd.tasks.common.direction_utils module

pipeline.hsd.tasks.common.direction_utils.direction_convert(direction, mepoch, mposition, outframe)

pipeline.hsd.tasks.common.direction_utils.direction_offset(direction, reference)

pipeline.hsd.tasks.common.direction_utils.direction_recover(ra, dec, org_direction)

pipeline.hsd.tasks.common.direction_utils.direction_shift(direction, reference, origin)

pipeline.hsd.tasks.common.display module

class pipeline.hsd.tasks.common.display.CustomDateFormatter(fmt, tz=None)

Bases: matplotlib.dates.DateFormatter

Customized date formatter that puts YYYY/MM/DD under usual tick labels at the beginning of tick and when date is changed.

class pipeline.hsd.tasks.common.display.PlotObjectHandler

Bases: object

axhline(*args, **kwargs)

axvspan(*args, **kwargs)

clear()

plot(*args, **kwargs)

text(*args, **kwargs)

class pipeline.hsd.tasks.common.display.SDCalibrationDisplay(inputs)

Bases: object

Inputs

alias of SingleDishDisplayInputs

abstract doplot(result, stage_dir)

plot()

class pipeline.hsd.tasks.common.display.SDImageDisplay(inputs)

Bases: object

Inputs

alias of SDImageDisplayInputs

property data

property edge

property id_spectral

property id_stokes

init()

property mask

property num_valid_spectrum

property rms

class pipeline.hsd.tasks.common.display.SDImageDisplayInputs(context, result)

Bases: pipeline.hsd.tasks.common.display.SingleDishDisplayInputs

property antenna

property antennaid_list

property contamination_plot

property fieldid_list

property imagename

property msid_list

property reduction_group

Retruns ReductionGroupDesc instance corresponding to the reduction group associated to the image

property source

property spw

property spwid_list

property stage_dir

property stage_number

property vis

class pipeline.hsd.tasks.common.display.SDSparseMapPlotter(nh, nv, step, brightnessunit, clearpanel=True, figure_id=None)

Bases: object

property TickSize

add_channel_axis(frequency)

property direction_reference

done()

property nh

property nv

plot(map_data, averaged_data, frequency, fit_result=None, figfile=None)

set_atm_transmission(transmission, frequency)

set_channel_axis()

set_deviation_mask(mask)

set_global_scaling()

setup_labels_absolute(ralist, declist)

setup_labels_relative(refpix_list, refval_list, increment_list)

setup_lines(lines_averaged, lines_map=None)

setup_reference_level(level=0.0)

unset_atm_transmission()

unset_channel_axis()

unset_global_scaling()

class pipeline.hsd.tasks.common.display.SingleDishDisplayInputs(context, result)

Bases: object

property isnro

class pipeline.hsd.tasks.common.display.SparseMapAxesManager(nh, nv, brightnessunit, ticksize, clearpanel=True, figure_id=None)

Bases: pipeline.infrastructure.displays.pointing.MapAxesManagerBase

static MATPLOTLIB_FIGURE_ID()

property axes_atm

property axes_chan

property axes_integsp

property axes_spmap

setup_labels(label_ra, label_dec)

class pipeline.hsd.tasks.common.display.SpectralImage(imagename)

Bases: object

property beam_size

property brightnessunit

direction_axis(idx, unit='deg')

property nchan

property npol

property nx

property ny

spectral_axis(unit='GHz')

to_velocity(frequency, freq_unit='GHz')

class pipeline.hsd.tasks.common.display.TimeAxesManager

Bases: object

init(start_time=None, end_time=None)

pipeline.hsd.tasks.common.display.ch_to_freq(ch, frequency)

pipeline.hsd.tasks.common.display.drop_edge(array)

pipeline.hsd.tasks.common.display.form3(n)

pipeline.hsd.tasks.common.display.form4(n)

pipeline.hsd.tasks.common.display.get_base_frame(table)

pipeline.hsd.tasks.common.display.get_base_frequency(table, freqid, nchan)

pipeline.hsd.tasks.common.display.mjd_to_datedict(val, unit='d')

pipeline.hsd.tasks.common.display.mjd_to_datestring(val, fmt='%Y/%m/%d')

pipeline.hsd.tasks.common.display.mjd_to_datetime(val)

pipeline.hsd.tasks.common.display.mjd_to_plotval(mjd_list)

pipeline.hsd.tasks.common.display.utc_formatter(fmt='%H:%M')

pipeline.hsd.tasks.common.display.utc_locator(start_time=None, end_time=None)

pipeline.hsd.tasks.common.rasterutil module

class pipeline.hsd.tasks.common.rasterutil.MetaDataSet(timestamp, dtrow, field, antenna, ra, dec, srctype, pflag)

Bases: tuple

property antenna

Alias for field number 3

property dec

Alias for field number 5

property dtrow

Alias for field number 1

property field

Alias for field number 2

property pflag

Alias for field number 7

property ra

Alias for field number 4

property srctype

Alias for field number 6

property timestamp

Alias for field number 0

pipeline.hsd.tasks.common.rasterutil.anim_gen(ra, dec, idx_generator, dist_list, cmap)

Generator for generate_animation.

Parameters

• ra (ndarray) – list of RA

• dec (ndarray) – list of Dec

• idx_generator (generator) – generator yielding start and end indices indicating raster row

• dist_list (ndarray) – distance list

• cmap (ListedColorMap) – color map

Yield

position, color, and boolean flag to clear existing plot

Return type

tuple

pipeline.hsd.tasks.common.rasterutil.animate(i)

Generate plot corresponding to single frame

Parameters

i (tuple) – position, color, and boolean flag to clear existing plot

Returns

lines for this frame

Return type

Lines2D

pipeline.hsd.tasks.common.rasterutil.distance(x0, y0, x1, y1)

Compute distance between two points (x0, y0) and (x1, y1).

Parameters

• x0 (float) – x-coordinate value for point 0

• y0 (float) – y-coordinate value for point 0

• x1 (float) – x-coordinate value for point 1

• y1 (float) – y-coordinate value for point 1

Returns

distance between two points

Return type

float

pipeline.hsd.tasks.common.rasterutil.filter_data(metadata, field_id, antenna_id, onsource=True)

Filter metadata.

Parameters

• metadata (MetaDataSet) – input metadata

• field_id (int) – field id

• antenna_id (int) – antenna id

• onsource (bool, optional) – take ON_SOURCE data only, defaults to True

Raises

RuntimeError – filter causes empty result

Returns

filtered metadata

Return type

MetaDataSet

pipeline.hsd.tasks.common.rasterutil.find_most_frequent(v)

Return the most frequent number (mode) in v.

Parameters

v (ndarray) – data

Returns

most frequent number (mode)

Return type

int

pipeline.hsd.tasks.common.rasterutil.find_raster_gap(timestamp, ra, dec, time_gap=None)

Find gaps between individual raster map. Returned list should be used in combination with gap_gen. Here is an example to plot RA/DEC data per raster map:

import maplotlib.pyplot as plt gap = find_raster_gap(timestamp, ra, dec) for s, e in gap_gen(gap):

plt.plot(ra[s:e], dec[s:e], ‘.’)

Parameters

• timestamp (ndarray) – list of time stamp

• ra (ndarray) – list of RA

• dec (ndarray) – list of Dec

• time_gap (ndarray, optional) – list of index of time gaps, defaults to None

Returns

list of index indicating boundary between raster maps

Return type

ndarray

pipeline.hsd.tasks.common.rasterutil.find_time_gap(timestamp)

Find time gap. Condition for gap is

• time interval > 3 * median(time interval) for small gap

• time gap > 3 * median(time gap) for large gap

Parameters

timestamp (ndarray) – list of timestamp. no duplication. must be sorted in ascending order.

Returns

list of small and large time gaps

Return type

two-tuple of lists

pipeline.hsd.tasks.common.rasterutil.flag_incomplete_raster(meta, raster_gap, nd_raster, nd_row)

flag incomplete raster map N: number of data per raster map M: number of data per raster row MN: median of N => typical number of data per raster map MM: median of M => typical number of data per raster row logic:

• if N[x] < MN + MM then flag whole data in raster map x

• if N[x] > MN + MM then flag whole data in raster map x and later

Parameters

• meta (MetaDataSet) – metadata object

• raster_gap (list) – gap list

• nd_raster (int) – typical number of data per raster map (MN)

• nd_row (int) – typical number of data per raster row (MM)

Returns

list of index for raster map

Return type

list

pipeline.hsd.tasks.common.rasterutil.flag_raster_map(metadata)

Return list of index to be flagged by flagging heuristics for raster scan

Parameters

meta (MetaDataSet) – metadata

Returns

per-antenna list of index to be flagged

Return type

list

pipeline.hsd.tasks.common.rasterutil.flag_raster_map_per_field(metadata, field_id)

Flag raster map based on two flagging heuristics for given field id.

Parameters

• metadata (MetaDataSet) – metadata

• field_id (int) – field id to process

Returns

per-antenna list of data ids to be flagged

Return type

list of ndarray

pipeline.hsd.tasks.common.rasterutil.flag_worm_eaten_raster(meta, raster_gap, nd_row)

flag raster map if number of continuous flagged data exceeds upper limit given by nd_row M: number of data per raster row MM: median of M => typical number of data per raster row L: maximum length of continuous flagged data logic:

• if L[x] > MM then flag whole data in raster map x

Parameters

• meta (MetaDataSet) – metadata object

• raster_gap (list) – gap list

• nd_row (int) – typical number of data per raster row (MM)

Returns

list of index for raster map

Return type

list

pipeline.hsd.tasks.common.rasterutil.from_context(context_dir)

read DataTable located in the context directory.

NOTE: only one DataTable will be loaded for multi-EB run

Parameters

context_dir (str) – path to the pipeline context directory

Returns

metadata

Return type

MetaDataSet

pipeline.hsd.tasks.common.rasterutil.gap_gen(gaplist, length=None)

Generate range of data (start and end indices) from given gap list. Return values, s and e, can be used to arr[s:e] to extract the data from the original array, arr.

Parameters

• gaplist (list) – list of indices indicating gap

• length (int, optional) – total number of data, defaults to None

Yield

start and end indices

Return type

two tuple of integers

pipeline.hsd.tasks.common.rasterutil.generate_animation(ra, dec, gaplist, figfile='movie.gif')

Generate animation GIF file to illustrate observing pattern.

Parameters

• ra (ndarray) – list of RA

• dec (ndarray) – list of Dec

• gaplist (list) – list of gaps between raster rows

• figfile (str, optional) – output file name, defaults to ‘movie.gif’

pipeline.hsd.tasks.common.rasterutil.get_angle(dx, dy, aspect_ratio=1)

Compute tangential angle taking into account aspect ratio.

Parameters

• dx (float) – length along x-axis

• dy (float) – length along y-axis

• aspect_ratio (float, optional) – aspect_ratio, defaults to 1

Returns

tangential angle

Return type

float

pipeline.hsd.tasks.common.rasterutil.get_aspect(ax)

Compute aspect ratio of matplotlib figure.

Parameters

ax (Axes) – Axes object to examine

Returns

aspect ratio

Return type

float

pipeline.hsd.tasks.common.rasterutil.get_raster_distance(ra, dec, gaplist)

Compute list of distances between raster rows. Origin of the distance is the first raster row. Representative position of each raster row is its midpoint (mean position).

Parameters

• ra (ndarray) – list of RA

• dec (ndarray) – list of Dec

• gaplist (list) – list of indices indicating gaps between raster rows

Returns

list of distances between raster rows

Return type

ndarray

pipeline.hsd.tasks.common.rasterutil.get_raster_flag_list(flagged1, flagged2, raster_gap, ndata)

Merge flag result and convert raster id to list of data index.

Parameters

• flagged1 (list) – list of flagged raster id

• flagged2 (list) – list of flagged raster id

• raster_gap (list) – list of gaps between raster maps

• ndata (int) – total number of data points

Returns

list of data ids to be flagged

Return type

ndarray

pipeline.hsd.tasks.common.rasterutil.get_science_target_fields(metadata)

Get list of field ids for science targets.

Parameters

metadata (MetaDataSet) – metadata

Returns

list of field ids for science targets

Return type

ndarray

pipeline.hsd.tasks.common.rasterutil.read_datatable(datatable)

extract necessary data from datatable instance.

Parameters

datatable (DataTableImpl) – datatable instance

Returns

metadata

Return type

MetaDataSet

pipeline.hsd.tasks.common.rasterutil.read_readonly_data(table)

pipeline.hsd.tasks.common.rasterutil.read_readwrite_data(table)

pipeline.hsd.tasks.common.rasterutil.squeeze_data(metadata)

Make timestamp in input metadata unique.

Parameters

metadata (MetaDataSet) – input metadata

Returns

metadata without duplication of timestamp

Return type

MetaDataSet

pipeline.hsd.tasks.common.utils module

class pipeline.hsd.tasks.common.utils.EchoDictionary

Bases: dict

class pipeline.hsd.tasks.common.utils.OnDemandStringParseLogger(logger)

Bases: object

PRIORITY_MAP = {'warn': 'warning'}

critical(msg_template, *args, **kwargs)

debug(msg_template, *args, **kwargs)

error(msg_template, *args, **kwargs)

info(msg_template, *args, **kwargs)

static parse(msg_template, *args, **kwargs)

todo(msg_template, *args, **kwargs)

trace(msg_template, *args, **kwargs)

warn(msg_template, *args, **kwargs)

class pipeline.hsd.tasks.common.utils.ProgressTimer(length=80, maxCount=80, LogLevel='info')

Bases: object

count(increment=1)

class pipeline.hsd.tasks.common.utils.RGAccumulator

Bases: object

append(field_id, antenna_id, spw_id, pol_ids=None, grid_table=None, channelmap_range=None)

get_antenna_id_list()

get_channelmap_range_list()

get_field_id_list()

get_grid_table_list()

get_pol_ids_list()

get_process_list(withpol=False)

get_spw_id_list()

iterate_all()

iterate_id()

class pipeline.hsd.tasks.common.utils.SpwDetailedView(spwid, name, num_channels, ref_frequency, min_frequency, max_frequency)

Bases: object

class pipeline.hsd.tasks.common.utils.SpwSimpleView(spwid, name)

Bases: object

pipeline.hsd.tasks.common.utils.TableSelector(name, query)

pipeline.hsd.tasks.common.utils.asdm_name(scantable_object)

Return ASDM name that target scantable belongs to. Assumptions are:

• scantable is generated from MS

• MS is generated from ASDM

• MS name is <uid>.ms

pipeline.hsd.tasks.common.utils.asdm_name_from_ms(ms_domain)

Return ASDM name that target ms originates from. Assumptions are:

• MS is generated from ASDM

• MS name is <uid>.ms

pipeline.hsd.tasks.common.utils.create_parallel_job(task_cls, task_args, context)

pipeline.hsd.tasks.common.utils.create_serial_job(task_cls, task_args, context)

pipeline.hsd.tasks.common.utils.get_brightness_unit(vis, defaultunit='Jy/beam')

pipeline.hsd.tasks.common.utils.get_datacolumn_name(vis)

pipeline.hsd.tasks.common.utils.get_index_list_for_ms(datatable, vis_list, antennaid_list, fieldid_list, spwid_list, srctype=None)

pipeline.hsd.tasks.common.utils.get_index_list_for_ms2(datatable, group_desc, member_list, srctype=None)

pipeline.hsd.tasks.common.utils.get_index_list_for_ms3(datatable_dict, group_desc, member_list, srctype=None)

pipeline.hsd.tasks.common.utils.get_mask_from_flagtra(flagtra)

Convert FLAGTRA (unsigned char) to a mask array (1=valid, 0=flagged)

pipeline.hsd.tasks.common.utils.get_parent_ms_idx(context, msname)

Returns index of corresponding ms in context The method maps both work_data and original MS to a proper index The return value is -1 if no match found.

pipeline.hsd.tasks.common.utils.get_parent_ms_name(context, msname)

Returns name of corresponding parent ms in context The method maps both work_data and original MS to a proper index The return value is “” if no match found.

pipeline.hsd.tasks.common.utils.get_restfrequency(vis, spwid, source_id)

pipeline.hsd.tasks.common.utils.get_spw_names(vis)

pipeline.hsd.tasks.common.utils.get_spw_properties(vis)

pipeline.hsd.tasks.common.utils.get_valid_ms_members(group_desc, msname_filter, ant_selection, field_selection, spw_selection)

pipeline.hsd.tasks.common.utils.get_valid_ms_members2(group_desc, ms_filter, ant_selection, field_selection, spw_selection)

pipeline.hsd.tasks.common.utils.is_nro(context)

pipeline.hsd.tasks.common.utils.iterate_group_member(group_desc, member_id_list)

pipeline.hsd.tasks.common.utils.make_ddid_map(vis)

pipeline.hsd.tasks.common.utils.make_polid_map(srcvis, dstvis)

pipeline.hsd.tasks.common.utils.make_row_map(src_ms, derived_vis, src_tb=None, derived_tb=None)

Make row mapping between source MS and associating MS

src_ms: measurement set domain object for source MS derived_vis: name of the MS that derives from source MS

returns: row mapping dictionary

pipeline.hsd.tasks.common.utils.make_row_map_for_baselined_ms(ms, table_container=None)

Make row mapping between calibrated MS and baselined MS. Return value is a dictionary whose key is row number for calibrated MS and its corresponding value is the one for baselined MS.

ms: measurement set domain object

returns: row mapping dictionary

pipeline.hsd.tasks.common.utils.make_spwid_map(srcvis, dstvis)

pipeline.hsd.tasks.common.utils.mjd_to_datestring(t, unit='sec')

MJD —> date string

t: MJD unit: sec or day

pipeline.hsd.tasks.common.utils.parseEdge(edge)

pipeline.hsd.tasks.common.utils.profiler(func)

pipeline.hsd.tasks.common.utils.require_virtual_spw_id_handling(observing_run)

Judge if spw ids vary across EBs. Return True if ids vary.

observing_run – domain.ObservingRun instance

pipeline.hsd.tasks.common.utils.sort_fields(context)

pipeline.hsd.tasks.common.utils.to_bool(s)

pipeline.hsd.tasks.common.utils.to_list(s)

pipeline.hsd.tasks.common.utils.to_numeric(s)

Module contents