pipeline.extern package

Subpackages

• pipeline.extern.logutils package
  • Submodules
  • pipeline.extern.logutils.adapter module
  • pipeline.extern.logutils.colorize module
  • pipeline.extern.logutils.dictconfig module
  • pipeline.extern.logutils.http module
  • pipeline.extern.logutils.queue module
  • pipeline.extern.logutils.redis module
  • pipeline.extern.logutils.testing module
  • Module contents

Submodules

pipeline.extern.XmlObjectifier module

This module is used to create native Python objects representing an XML document rendering the elements into a hierarchical tree. Name spaces can optionally be mapped into the element names by specifying ‘mapNameSpaces = 1’. Leading characters can be omitted in the name space mapping using the ‘skipChars’ argument.

Characters that are not allowed in Python names (‘.’, ‘-‘, ‘/’, ‘:’) are mapped to ‘_’.

The resulting Python object can be modified and serialized into XML again using the ‘writexml’ method.

Example usage:

import XmlObjectifier xmlObject = XmlObjectifier.XmlObject(xmlString = <XML string>,

skipChars = <string>)

or

xmlObject = XmlObjectifier.XmlObject(fileName = ‘<file name>’,

skipChars = <string>)

This example XML document:

<?xml version=”1.0” encoding=”ISO-8859-1”?> <!– edited with XMLSPY v5 U (http://www.xmlspy.com) by D. Muders (MPIfR) –> <TelCalResult xmlns=”Alma/TelCalResult”

xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”Alma/TelCalResult/TelCalResult-single.xsd”>

<TelCalResultEntity entityId=”2718281828” entityIdEncrypted=””

entityVersion=”0.1”/>

<TelCalResultRef entityId=”2718281827”/> <SchedBlockRef entityId=”31415926534”/> <TelCalResultDetail>

<ResultKind>FocusOffset</ResultKind> <ScanID>1789</ScanID> <NumAntennas>2</NumAntennas> <AntennaID>64</AntennaID> <AntennaID>42</AntennaID> <FocusOffset>-0.25</FocusOffset> <FocusOffset>-0.34</FocusOffset>

</TelCalResultDetail>

</TelCalResult>

can then be navigated in the Python object like this:

#!/usr/bin/env python3

import XmlObjectifier

def printInfo(focusResult):

scanID = focusResult.TelCalResult.TelCalResultDetail.ScanID.getValue() kind = focusResult.TelCalResult.TelCalResultDetail.ResultKind.getValue() numAnts = focusResult.TelCalResult.TelCalResultDetail.NumAntennas.getValue()

print(‘This is a %s result entity. Scan ID: %d. Number of antennas: %d

‘ % (kind, scanID, numAnts))

for ant in range(numAnts):

antID = focusResult.TelCalResult.TelCalResultDetail.AntennaID[ant].getValue() focusOffset = focusResult.TelCalResult.TelCalResultDetail.FocusOffset[ant].getValue() print(‘Antenna #%d focus offset: %.1f’ % (antID, focusOffset))

# Objectify XML focusResult = XmlObjectifier.XmlObject(fileName = ‘FocusResult.xml’)

# Print object summary print(‘Original focus result: ‘) printInfo(focusResult)

# Optionally modify elements focusResult.TelCalResult.TelCalResultDetail.ScanID.setValue(1790) focusResult.TelCalResult.TelCalResultDetail.AntennaID[0].setValue(24) focusResult.TelCalResult.TelCalResultDetail.FocusOffset[0].setValue(0.3) focusResult.TelCalResult.TelCalResultDetail.AntennaID[1].setValue(25) focusResult.TelCalResult.TelCalResultDetail.FocusOffset[1].setValue(0.5)

# Print object summary print(‘

New focus result: ‘) printInfo(focusResult)

# Write XML to a new file f = open(‘FocusResultNew.xml’, ‘w+’) focusResult.writexml(f, ‘’) f.close()

class pipeline.extern.XmlObjectifier.XmlElement(element, mapNameSpaces, nameSpaceMapping, skipChars)

Bases: xml.dom.minidom.Element

Creates an object representing an XML tag/element with all of its content.

childNodes

getAttribute(name)

Overwrites the inherited method and returns a value of the right type.

getValue()

Returns the included TEXT, if present.

namespaceURI

nextSibling

nodeName

ownerDocument

parentNode

prefix

previousSibling

setValue(value)

Sets the included TEXT.

tagName

class pipeline.extern.XmlObjectifier.XmlObject(xmlString=None, fileName=None, skipChars='', mapNameSpaces=0)

Bases: xml.dom.minidom.Document

Creates an object representing the XML document wich is to be objectified. The XML string passed to the constructor is preferred over any specified XML file.

Optionally the name space mapping can be turned on by passing mapNameSpaces = 1.

Leading characters in the name space definitions can be skipped in the mapping by passing the optional “skipChars” argument.

childNodes

doctype

exception pipeline.extern.XmlObjectifier.XmlObjectifierError(msg, code=None)

Bases: Exception

pipeline.extern.XmlObjectifier.castType(value)

pipeline.extern.adopted module

pipeline.extern.adopted.MAD(a, c=0.6745, axis=0)

Median Absolute Deviation along given axis of an array:

median(abs(a - median(a))) / c

c = 0.6745 is the constant to convert from MAD to std; it is used by default

pipeline.extern.adopted.call_qa_time(arg, form='', prec=0, showform=False)

This is a wrapper for qa.time(), which in casa 4.0.0 returns a list of strings instead of just a scalar string. - Todd Hunter

pipeline.extern.adopted.getMJD()

Returns the current MJD. See also getCurrentMJDSec(). -Todd

pipeline.extern.adopted.getMedianPWV(vis='.', myTimes=[0, 999999999999], asdm='', verbose=False)

Extracts the PWV measurements from the WVR on all antennas for the specified time range. The time range is input as a two-element list of MJD seconds (default = all times). First, it tries to find the ASDM_CALWVR table in the ms. If that fails, it then tries to find the ASDM_CALATMOSPHERE table in the ms. If that fails, it then tried to find the CalWVR.xml in the specified ASDM, or failing that, an ASDM of the same name (-.ms). If neither of these exist, then it tries to find CalWVR.xml in the present working directory. If it still fails, it looks for CalWVR.xml in the .ms directory. Thus, you only need to copy this xml file from the ASDM into your ms, rather than the entire ASDM. Returns the median and standard deviation in millimeters. Returns: The median PWV, and the median absolute deviation (scaled to match rms) For further help and examples, see https://safe.nrao.edu/wiki/bin/view/ALMA/GetMedianPWV – Todd Hunter

pipeline.extern.adopted.readPWVFromASDM_CALATMOSPHERE(vis)

Reads the PWV via the water column of the ASDM_CALATMOSPHERE table. - Todd Hunter

pipeline.extern.adopted.readpwv(asdm)

This function assembles the dictionary returned by readwvr() into arrays containing the PWV measurements written by TelCal into the ASDM. Units are in meters. – Todd Hunter

pipeline.extern.adopted.readwvr(sdmfile, verbose=False)

This function reads the CalWVR.xml table from the ASDM and returns a dictionary containing: ‘start’, ‘end’, ‘startmjd’, ‘endmjd’, ‘startmjdsec’, ‘endmjdsec’, ‘timerange’, ‘antenna’, ‘water’, ‘duration’. ‘water’ is the zenith PWV in meters. This function is called by readpwv(). – Todd Hunter

pipeline.extern.asizeof module

This module exposes 9 functions and 2 classes to obtain lengths (in items) and sizes (in bytes) of Python objects for Python 2.6 and later, including Python 3+ 2.

Public Functions 1

Function asizeof calculates the combined (approximate) size in bytes of one or several Python objects.

Function asizesof returns a tuple containing the (approximate) size in bytes for each given Python object separately.

Function asized returns for each object an instance of class Asized containing all the size information of the object and a tuple with the referents 4.

Functions basicsize and itemsize return the basic respectively itesize of the given object, both in bytes. For objects as array.array, numpy.array, numpy.matrix, etc. where the item size varies depending on the instance-specific data type, function itemsize returns that item size.

Function flatsize returns the flat size of a Python object in bytes defined as the basic size plus the item size times the length of the given object.

Function alen 3 returns the length of an object like standard function len but extended for several types. E.g. the alen of a multi-precision int (or long) is the number of digits 6. The length of most mutable sequence objects includes an estimate of the over-allocation and therefore, the alen value may differ from the standard len result. For objects like array.array, numpy.array, numpy.matrix, etc. function alen returns the proper number of items.

Function refs returns (a generator for) the referents 4 of the given object.

Certain classes are known to be sub-classes of or to behave as dict objects. Function adict can be used to install other class objects to be treated like dict.

Public Classes 1

Class Asizer may be used to accumulate the results of several sizing calls. After creating an Asizer instance, use methods asizeof and asizesof as needed to size any number of additional objects and accumulate the sizes.

Call methods exclude_refs and/or exclude_types to exclude references to respectively instances or types of certain objects.

Use one of the print_… methods to report the statistics.

An instance of class Asized is returned for each object sized with the asized function or method.

Duplicate Objects

Any duplicate, given objects are sized only once and the size is included in the combined total only once. But functions asizesof and asized will return a size value respectively an Asized instance for each given object, including duplicates.

Definitions 5

The length of an objects like dict, list, set, str, tuple, etc. is defined as the number of items held in or allocated by the object. Held items are references to other objects, called the referents.

The size of an object is defined as the sum of the flat size of the object plus the sizes of any referents 4. Referents are visited recursively up to the specified detail level. However, the size of objects referenced multiple times is included only once in the total size.

The flat size of an object is defined as the basic size of the object plus the item size times the number of allocated items, references to referents. The flat size does include the size for the references to the referents, but not the size of the referents themselves.

The flat size returned by function flatsize equals the result of function asizeof with options code=True, ignored=False, limit=0 and option align set to the same value.

The accurate flat size for an object is obtained from function sys.getsizeof() where available. Otherwise, the length and size of sequence objects as dicts, lists, sets, etc. is based on an estimate for the number of allocated items. As a result, the reported length and size may differ substantially from the actual length and size.

The basic and item size are obtained from the __basicsize__ respectively __itemsize__ attributes of the (type of the) object. Where necessary (e.g. sequence objects), a zero __itemsize__ is replaced by the size of a corresponding C type.

The overhead for Python’s garbage collector (GC) is included in the basic size of (GC managed) objects as well as the space needed for refcounts (used only in certain Python builds).

Optionally, size values can be aligned to any power of 2 multiple.

Size of (byte)code

The (byte)code size of objects like classes, functions, methods, modules, etc. can be included by setting option code=True.

Iterators are handled like sequences: iterated object(s) are sized like referents 4 but only up to the specified level or recursion limit (and only if function gc.get_referents() returns the referent object of iterators).

Generators are sized as (byte)code only, but the generated objects are never sized.

Old- and New-style Classes

All old- and new-style class, instance and type objects, are handled uniformly such that (a) instance objects are distinguished from class objects and (b) instances of different old-style classes can be dealt with separately.

Class and type objects are represented as <class ....* def> respectively <type ... def> where the * indicates an old-style class and the ... def suffix marks the definition object. Instances of classes are shown as <class module.name*> without the ... def suffix. The * after the name indicates an instance of an old-style class.

Ignored Objects

To avoid excessive sizes, several object types are ignored 5 by default, e.g. built-in functions, built-in types and classes 7, function globals and module referents. However, any instances thereof and module objects will be sized when passed as given objects. Ignored object types are included unless option ignored is set accordingly.

In addition, many __...__ attributes of callable objects are ignored 5, except crucial ones, e.g. class attributes __dict__, __doc__, __name__ and __slots__. For more details, see the type-specific _..._refs() and _len_...() functions below.

Footnotes

1(1,2)

The functions and classes in this module are not thread-safe.

2

Earlier editions of this module supported Python versions down to Python 2.2. To use Python 2.5 or older, try module asizeof from project Pympler 0.3.x.

3

Former function leng, class attribute leng and keyword argument leng have all been renamed to alen. However, function leng is still available for backward compatibility.

4(1,2,3,4)

The referents of an object are the objects referenced by that object. For example, the referents of a list are the objects held in the list, the referents of a dict are the key and value objects in the dict, etc.

5(1,2,3)

These definitions and other assumptions are rather arbitrary and may need corrections or adjustments.

6

See Python source file .../Include/longinterp.h for the C typedef of digit used in multi-precision int (or long) objects. The C sizeof(digit) in bytes can be obtained in Python from the int (or long) __itemsize__ attribute. Function alen determines the number of digits of an int (or long) object.

7

Type``s and ``class``es are considered built-in if the ``__module__ of the type or class is listed in the private _builtin_modules.

class pipeline.extern.asizeof.Asized(size, flat, refs=(), name=None)

Bases: object

Stores the results of an asized object in the following 4 attributes:

size – total size of the object (including referents)

flat – flat size of the object

name – name or repr of the object

refs – tuple containing an Asized instance for each referent

flat

format(format='%(name)s size=%(size)d flat=%(flat)d', detail=- 1, order_by='size', indent='')

Formats the size information of the object and of all sized referents as a string.

format=’%(name)s…’ – specifies the format string per instance, valid interpolation parameters are ‘name’, ‘size’ and ‘flat’

detail=-1 – detail level up to which referents are printed (-1 for unlimited)

order_by=’size’ – sort order of referents, valid choices are ‘name’, ‘size’ or ‘flat’

indent=’’ – optional indentation

get(name, dflt=None)

Returns the named referent (or dflt if not found).

name

refs

size

class pipeline.extern.asizeof.Asizer(**opts)

Bases: object

Sizer state and options.

asized(*objs, **opts)

Sizes each object and returns an Asized instance with size information and referents up to the given detail level (and with modified options, see method set).

If only one object is given, the return value is the Asized instance for that object.

asizeof(*objs, **opts)

Returns the combined size of the given objects (with modified options, see method set).

asizesof(*objs, **opts)

Returns the individual sizes of the given objects (with modified options, see method set).

property duplicate

Number of duplicate objects seen so far.

exclude_refs(*objs)

Excludes any references to the specified objects from sizing.

While any references to the given objects are excluded, the objects will be sized if specified as positional arguments in subsequent calls to methods asizeof and asizesof.

exclude_types(*objs)

Exclude the specified object instances and types from sizing.

All instances and types of the given objects are excluded, even objects specified as positional arguments in subsequent calls to methods asizeof and asizesof.

property missed

Number of objects missed due to errors.

print_profiles(w=0, cutoff=0, **print3opts)

Prints the profiles above cutoff percentage.

The available options and defaults are:

w=0 – indentation for each line

cutoff=0 – minimum percentage printed

print3opts – print keyword arguments, like Python 3.+

print_stats(objs=(), opts={}, sized=(), sizes=(), stats=3.0, **print3opts)

Prints the statistics.

The available options and defaults are:

w=0 – indentation for each line

objs=() – optional, list of objects

opts={} – optional, dict of options used

sized=() – optional, tuple of Asized instances returned

sizes=() – optional, tuple of sizes returned

stats=0.0 – print stats, see function asizeof

print3opts – print keyword arguments, like Python 3.+

print_summary(w=0, objs=(), **print3opts)

Prints the summary statistics.

The available options and defaults are:

w=0 – indentation for each line

objs=() – optional, list of objects

print3opts – print keyword arguments, like Python 3.+

print_typedefs(w=0, **print3opts)

Prints the types and dict tables.

The available options and defaults are:

w=0 – indentation for each line

print3opts – print keyword arguments, like Python 3.+

reset(align=8, clip=80, code=False, derive=False, detail=0, ignored=True, infer=False, limit=100, stats=0, stream=None)

Resets the options, state, etc.

The available options and defaults are:

align=8 – size alignment

clip=80 – clip repr() strings

code=False – incl. (byte)code size

derive=False – derive from super type

detail=0 – Asized refs level

ignored=True – ignore certain types

infer=False – try to infer types

limit=100 – recursion limit

stats=0.0 – print statistics, see function asizeof

stream=None – output stream for printing

See function asizeof for a description of the options.

set(align=None, code=None, detail=None, limit=None, stats=None)

Sets some options. See also method reset.

Any options not set remain unchanged from the previous setting. The available options are:

align – size alignment

code – incl. (byte)code size

detail – Asized refs level

limit – recursion limit

stats – print statistics, see function asizeof

property total

Total size (in bytes) accumulated so far.

pipeline.extern.asizeof.adict(*classes)

Installs one or more classes to be handled as dict.

pipeline.extern.asizeof.alen(obj, **opts)

Returns the length of an object (in items).

See function basicsize for a description of the options.

pipeline.extern.asizeof.asized(*objs, **opts)

Returns a tuple containing an Asized instance for each object passed as positional argument.

The available options and defaults are:

align=8 – size alignment

clip=80 – clip repr() strings

code=False – incl. (byte)code size

derive=False – derive from super type

detail=0 – Asized refs level

ignored=True – ignore certain types

infer=False – try to infer types

limit=100 – recursion limit

stats=0.0 – print statistics, see function asizeof

If only one object is given, the return value is the Asized instance for that object. Otherwise, the length of the returned tuple matches the number of given objects.

Set detail to the desired referents level and limit to the maximum recursion depth.

See function asizeof for descriptions of the other options.

pipeline.extern.asizeof.asizeof(*objs, **opts)

Returns the combined size (in bytes) of all objects passed as positional arguments.

The available options and defaults are:

align=8 – size alignment

all=False – all current objects

clip=80 – clip repr() strings

code=False – incl. (byte)code size

derive=False – derive from super type

ignored=True – ignore certain types

infer=False – try to infer types

limit=100 – recursion limit

stats=0.0 – print statistics

Set align to a power of 2 to align sizes. Any value less than 2 avoids size alignment.

All current module, global and stack objects are sized if all is True and if no positional arguments are supplied.

A positive clip value truncates all repr() strings to at most clip characters.

The (byte)code size of callable objects like functions, methods, classes, etc. is included only if code is True.

If derive is True, new types are handled like an existing (super) type provided there is one and only of those.

By default certain base types like object, super, etc. are ignored. Set ignored to False to include those.

If infer is True, new types are inferred from attributes (only implemented for dict types on callable attributes as get, has_key, items, keys and values).

Set limit to a positive value to accumulate the sizes of the referents of each object, recursively up to the limit. Using limit=0 returns the sum of the flat[4] sizes of the given objects. High limit values may cause runtime errors and miss objects for sizing.

A positive value for stats prints up to 8 statistics, (1) a summary of the number of objects sized and seen, (2) a simple profile of the sized objects by type and (3+) up to 6 tables showing the static, dynamic, derived, ignored, inferred and dict types used, found respectively installed. The fractional part of the stats value (x 100) is the cutoff percentage for simple profiles.

See this module documentation for the definition of flat size.

pipeline.extern.asizeof.asizesof(*objs, **opts)

Returns a tuple containing the size (in bytes) of all objects passed as positional argments.

The available options and defaults are:

align=8 – size alignment

clip=80 – clip repr() strings

code=False – incl. (byte)code size

derive=False – derive from super type

ignored=True – ignore certain types

infer=False – try to infer types

limit=100 – recursion limit

stats=0.0 – print statistics

See function asizeof for a description of the options.

The length of the returned tuple equals the number of given objects.

pipeline.extern.asizeof.basicsize(obj, **opts)

Returns the basic size of an object (in bytes).

The available options and defaults are:

derive=False – derive type from super type

infer=False – try to infer types

save=False – save the type definition if new

pipeline.extern.asizeof.flatsize(obj, align=0, **opts)

Returns the flat size of an object (in bytes), optionally aligned to a given power of 2.

See function basicsize for a description of all other options.

See this module documentation for the definition of flat size.

pipeline.extern.asizeof.itemsize(obj, **opts)

Returns the item size of an object (in bytes).

See function basicsize for a description of the options.

pipeline.extern.asizeof.refs(obj, **opts)

Returns (a generator for) specific referents of an object.

See function basicsize for a description of the options.

pipeline.extern.findContinuum module

This file implements the algorithm to determine continuum channel ranges to use from an ALMA image cube (dirty or clean) in CASA format. All dependencies on what were originally analysisUtils functions have been pasted into this file for convenience. This function is meant to be run inside CASA. Simple usage:

import findContinuum as fc fc.findContinuum(‘my_dirty_cube.image’)

This file can be found in a typical pipeline distribution directory, e.g.: /lustre/naasc/sciops/comm/rindebet/pipeline/branches/trunk/pipeline/extern As of March 7, 2019 (version 3.36), it is compatible with both python 2 and 3.

Code changes for Pipeline2020: (as of August 9, 2020) 0) fix for PIPE-554 (bug found in cube imaging of calibrators by ARI-L project) 1) fix for PIPE-525 (divide by zero in a 130-target MOUS) 2) new feature PIPE-702 (expand mask if mom8fc image has emission outside it) 3) Add warning messages for too little bandwidth or too little spread. 4) if outdir is specified and does not exist, stop and print error message. 5) Modified the is_binary() function to work in both python 2 and 3. 6) Add LSRK frequency ranges in a new final line of the .dat file 7) Increase to 2 significant digits on sigmaFC in the png name 8) Allow manually specified narrow value to be used everywhere instead of ‘auto’ 9) Print Final selection message to casa log 10) fix bug in convertSelectionIntoChannelList (only relevant if avoidance used) 11) allow single value in convertSelection 12) fix bug in avoidance range when gaps were trimmed (not relevant to PL) 13) remove warning about no beam in cube header 14) Add totalChannels & medianWidthOfRanges parameters+logic to pickAutoTrimChannels() 15) Insure that channel ranges are in increasing order if a second range is added to a solo range 16) add imstatListit=False to all imstat calls 17) add minor ticks to lower x-axis (channel number) of the plot 18) added 21 new control parameters to findContinuum (default values listed) for a total of 98:

• returnWarnings=False (PL 2020 might set this to True if Dirk has time)

• sigmaFindContinuumMode=’auto’ (split from sigmaFindContinuum)

• returnSigmaFindContinuum=False

• enableRejectNarrowInnerWindows=True (to be able to disable it manually)

• avoidExtremaInNoiseCalcForJointMask=False

• buildMom8fc=True

• momentdir=’’

• amendMaskIterations=’auto’

• skipchan=1

• checkIfMaskWouldHaveBeenAmended=False

• fontsize=10

• thresholdForSame=0.01 (to control the new 4-letter codes)

• keepIntermediatePngs=True

• enableOnlyExtraMask=True

• useMomentDiff=True

• smallBandwidthFraction=0.05

• smallSpreadFraction=0.33

• skipAmendMask=False (for manual usage)

• useAnnulus=True

• cubeSigmaThreshold=7.5

• npixThreshold=7 (for onlyExtraMask)

19. Added 27 new functions:

• round_half_up (to support the same rounding in python 3 as 2)

• byteDecode (to convert bytes to string for python 3)

• imageSNR

• imageSNRAnnulus

• plotChannelSelections

• compute4LetterCodeAndUpdateLegend

• compute4LetterCode

• cubeNoiseLevel

• updateChannelRangesOnPlot

• computeNpixCubeMedian

• computeNpixMom8Median

• computeNpixMom8MedianBadAtm

• replaceLineFullRangesWithNoise

• gatherWarnings

• tooLittleBandwidth

• tooLittleSpread

• computeSpread

• amendMaskYesOrNo

• extraMaskYesOrNo

• onlyExtraMaskYesOrNo

• invertChannelRanges

• robustMADofContinuumRanges

• findWidestContiguousListInChannelRange

• imagePercentileNoMask (not used by PL because avoidExtremaInNoiseCalcForJointMask=False)

• combineContDat (not used by pipeline)

• getSpwFromPipelineImageName (not used by pipeline)

• getFieldnameFromPipelineImageName (not used by pipeline)

-Todd Hunter

pipeline.extern.findContinuum.CalcAtmTransmissionForImage(img, imageInfo, chanInfo='', airmass=1.5, pwv=- 1, spectralaxis=- 1, value='transmission', P=- 1, H=- 1, T=- 1, altitude=- 1)

This function is called by atmosphereVariation. Supported telescopes are VLA and ALMA (needed for default weather and PWV) img: name of CASA image value: ‘transmission’ or ‘tsky’ chanInfo: a list containing nchan, firstFreqHz, lastFreqHz, channelWidthHz pwv: in mm P: in mbar H: in percent T: in Kelvin Returns: 2 arrays: frequencies (in GHz) and values (Kelvin, or transmission: 0..1)

pipeline.extern.findContinuum.ExpandYLimitsForLegend()

Called by runFindContinuum. Make room for legend text at bottom and top of existing plot.

pipeline.extern.findContinuum.MAD(a, c=0.6745, axis=0)

This function is called by removeInitialQuadraticIfNeeded, findContinuumChannels, meanSpectrum, computeStatisticalSpectrumFromMask, plotStatisticalSpectrumFromMask and runFindContinuum. Median Absolute Deviation along given axis of an array:

median(abs(a - median(a))) / c

c = 0.6745 is the constant to convert from MAD to std

pipeline.extern.findContinuum.aboveBelow(avgSpectrumNansReplaced, threshold)

This function is called by runFindContinuum. Given an array of values (i.e. a spectrum) and a threshold value, this function computes and returns 6 items: * the number of channels above that threshold (group 1) * the number of channels below that threshold (group 2) * the ratio of these numbers (i.e., #ChanAboveThreshold / #ChanBelowThreshold)) * sum of the intensities in group (1) * sum of the intensities in group (2) * the ratio of these sums (i.e., FluxAboveThreshold / FluxBelowThreshold)

(This value is not currently used by the calling function.) Example with 24 channels where 3 have positive spikes, and rest are zero:

3 2 3 channelsAbove = 3

+threshold=+1 …………………… sumAbove = (3-1) + (2-1) + (3-1) = 5

zero ————————— channelsBelow = 21

-threshold=-1 …………………… sumBelow = 21*(1-0) = 21

pipeline.extern.findContinuum.allContinuumSelected(selection, nchan, fraction=0.925)

Returns True if only one range is selected and that range covers sufficient bandwidth (default fraction = 92.5%) nchan: number of channels in the spectrum fraction: default = 0.925 which is 92.5%

pipeline.extern.findContinuum.amendMaskYesOrNo(badAtmosphere, median, momSNR, momSNRCube, Npix, NpixBadAtm, NpixCube, TenEventSigma, MADMomOutside, MADCubeOutside, momLevel, momLevelBadAtm, cubeLevel, NpixCube2, fractionNegativePixels, Npix2=None, Npix2BadAtm=None, verbose=True)

Used in Pipeline2020 going forward. median: meidan of mom8fc img if fc(useMomentDiff=False), or momDiff (mom8-mom0scaled) if useMomentDiff=True momSNR: SNR of mom8fc img if fc(useMomentDiff=False), or momDiff (mom8-mom0scaled) if useMomentDiff=True etc. badAtmosphere: either a boolean or a string (‘goodAtm’ or ‘badAtm’)

pipeline.extern.findContinuum.atmosphereVariation(img, imageInfo, chanInfo, airmass=1.5, pwv=- 1, removeSlope=True)

This function is called by findContinuum. Computes the absolute and percentage variation in atmospheric transmission and sky temperature across an image cube. Returns 5 values: max(Trans)-min(Trans), and as percentage of mean,

Max(Tsky)-min(Tsky), and as percentage of mean, standard devation of Tsky

pipeline.extern.findContinuum.avgOverCube(pixels, useAbsoluteValue=False, threshold=None, median=False, mask='', usermask='', useIAGetProfile=True)

This function was used by Cycle 4 and 5 pipeline from meanSpectrum(), but will not be used in the Cycle 6 default heuristic of ‘mom0mom8jointMask’. Computes the average spectrum across a multi-dimensional array read from an image cube, ignoring any NaN values. Inputs: pixels: a 3D array (with degenerate Stokes axis dropped) threshold: value in Jy median: if True, compute the median instead of the mean mask: use pixels inside this spatial mask (i.e. with value==1 or True) to compute

the average (the spectral mask is taken as the union of all channels) This is the mask located inside the image cube specified in findContinuum(img=’’)

usermask: this array results from the mask specified in findContinuum(mask=’’) parameter If threshold is specified, then it only includes pixels with an intensity above that value. Returns: * average spectrum * percentage of pixels not masked Note: This function assumes that the spectral axis is the final axis.

If it is not, there is no single setting of the axis parameter that can make it work.

pipeline.extern.findContinuum.buildMeanSpectrumFilename(img, meanSpectrumMethod, peakFilterFWHM, amendMaskIterationName='')

This function is called by findContinuum and runFindContinuum. Creates the name of the meanSpectrumFile to search for and/or create. Returns: a string

pipeline.extern.findContinuum.byteDecode(buffer)

pipeline.extern.findContinuum.casaVersionCompare(comparitor, versionString)

This function is called by meanSpectrum. Uses cu.compare_version in CASA >=5, otherwise uses string comparison No longer works in CASA 6.

pipeline.extern.findContinuum.casalogPost(mystring, debug=True)

Generates an INFO message prepended with the version number of findContinuum.

pipeline.extern.findContinuum.centralArcsecArgumentMismatch(centralArcsec, meanSpectrumFile, iteration)

This function is called by checkForMismatch. Determines if the requested centralArcsec value does not match what was used to generate the specified meanSpectrumFile. Returns: True or False

pipeline.extern.findContinuum.channelSelectionRangesToIndexArray(selection, separator=';')

This function is called by runFindContinuum. Convert a channel selection range string to integer array of indices. e.g.: ‘3~8;10~11’ -> [3,4,5,6,7,8,10,11]

pipeline.extern.findContinuum.channelsInLargestGroup(selection)

This function is called by runFindContinuum. Returns the number of channels in the largest group of channels in a CASA selection string.

pipeline.extern.findContinuum.checkForMismatch(meanSpectrumFile, img, mask, useThresholdWithMask, fitsTable, iteration, centralArcsec)

This function is called by runFindContinuum. Returns True if there is a mismatch between the pre-existing meanSpectrumFile and the requested parameters.

pipeline.extern.findContinuum.checkForTriangularWavePattern(img, triangleFraction=0.83, pad=20)

+++++++ This function is not used for meanSpectrumMethod == ‘mom0mom8jointMask’. Fit and remove linear slopes to each half of the spectrum, then comparse the MAD of the residual to the MAD of the original spectrum pad: fraction of total channels to ignore on each edge (e.g. 20: 1/20th) triangleFraction: MAD of dual-linfit residual must be less than this fraction*originalMAD Returns: * Boolean: whether triangular pattern meets the threshold * value: either False (if slope test failed) or a float (the ratio of the MADs)

pipeline.extern.findContinuum.combineContDat(contdatlist, outputfile='cont.dat', fieldname='', spw='')

Takes a list of *.dat files created by findContinuum for a single field/spw and pulls the line that contains LSRK frequency ranges from each one, and writes it to a new cont.dat file suitable for the pipeline that contains an entry for each spw of that field. contdatlist: a list or comma-delimited string of the .dat files, or of

the cube names (to which ‘_findContinuum.dat’ will be appended)

fieldname: use this value if specified, otherwise parse from filename spw: integer or string integer, use this value if specified, otherwise parse

from the filename

Note: It assumes that none of the files originate from an ALLcont spw, so it will never write the “ALL” line that the pipeline triggers on. But this feature could be added by calling the following function if there is only one range of channels read:

allContinuumSelected(selection, nchan, fraction=ALL_CONTINUUM_CRITERION)

We would only need to know nchan.

pipeline.extern.findContinuum.compute4LetterCode(mom8fcPeak, mom8fcPeak0, mom8fcPeakOutside, mom8fcPeakOutside0, mom8fcSum, mom8fcSum0, mom8fcMAD, mom8fcMAD0, threshold=0.01, returnRatios=False)

Used in Pipeline2020 going forward. threshold: to use for Peak, PeakOutside, and Sum; double this value is used for MAD (i.e. 2%)

pipeline.extern.findContinuum.compute4LetterCodeAndUpdateLegend(mom8fcPeak, mom8fcPeak0, mom8fcPeakOutside, mom8fcPeakOutside0, mom8fcSum, mom8fcSum0, mom8fcMAD, mom8fcMAD0, thresholdForSame, areaString, labelDescs, fontsize, ax1, amendMaskDecision, extraMaskDecision, extraMaskDecision2, autoLowerDecision, intersectionOfSelections, useMomentDiff, warnings, channelRanges, sigmaUsedToAmendMask, skipDecisionCode=False, replace=False, addedSelection=False)

Used in Pipeline2020 going forward. Add a variable length code and a 4-letter code to the end of the last line of the upper legend. useMomentDiff: specified in the call to findContinuum() Returns: labelDescs, areaString, mycode

pipeline.extern.findContinuum.computeBandwidth(selection, channelWidth, loc=- 1)

This function is called by runFindContinuum and findContinuum. selection: a string of format: ‘5~6;9~20’ channelWidth: in Hz Returns: bandwidth in GHz

pipeline.extern.findContinuum.computeMadSpectrum(img, box='')

+++++++ This function is not used for meanSpectrumMethod == ‘mom0mom8jointMask’.

Only used by checkForTriangularWavePattern.

Uses the imstat task to compute an array containing the MAD spectrum of a cube.

pipeline.extern.findContinuum.computeMedianCorrectionFactor(baselineMode, percentile)

This function is called by findContinuumChannels and medianCorrected. Computes the effect due to the fact that taking the median of the N%ile lowest points of a Gaussian noise datastream will be lower than the true median of all the points. This is (TrueMedian-ObservedMedian)/ObservedMAD

pipeline.extern.findContinuum.computeNpixCubeMedian(mom8fcMedian, cubeLevel, MADCubeOutside, mom8fc, jointMaskTest)

Used in Pipeline2020 going forward. jointMaskTest: can be a mask, a mask statement or compound mask statement

pipeline.extern.findContinuum.computeNpixMom8Median(mom8fcMedian, mom8level, mom8fcMAD, mom8fc, jointMaskTest, verbose=True)

Used in Pipeline2020 going forward. mom8fcMedian: the median of the moment8 image being used mom8level: a sigma value to use mom8fcMAD: a scaled MAD jointMaskTest: can be a mask, a mask statement or compound mask statement

pipeline.extern.findContinuum.computeNpixMom8MedianBadAtm(mom8fcMedian, mom8levelBadAtm, mom8fcMAD, mom8fc, jointMaskTest)

Used in Pipeline2020 going forward. jointMaskTest: can be a mask, a mask statement or compound mask statement

pipeline.extern.findContinuum.computeSpread(selection, channelWidth)

Used in Pipeline2020 going forward.

pipeline.extern.findContinuum.computeStatisticalSpectrumFromMask(cube, jointmask, pbimg=None, imageInfo=None, statistic='mean', normalizeByMAD=False, projectCode='', higherAnnulusLevel=None, lowerAnnulusLevel=None, outdir='', jointMaskForNormalize=None)

New heuristic for Cycle 6 pipeline. It is called by meanSpectrumFromMom0Mom8JointMask Uses ia.getprofile to compute the mean spectrum of a cube within a masked area. jointmask: a 2D or 3D mask image indicating which pixels shall be used

or an expression with < or > along with a 2D or 3D mask image

jointMaskForNormalize: a 2D or 3D mask image to be used when normalizeByMAD==True pbimg: the path to the primary beam of this cube, or a 2D primary beam with an expression statistic: passed to ia.getprofile via the ‘function’ parameter normalizeByMAD: if True, then create the inverse of the jointmask and

normalize the spectrum by the spectrum of ‘xmadm’ (scaled MAD) computed on the inverse mask

Returns: three arrays: channels, freqs(Hz), intensities, and a Boolean

which says if normalization was applied

pipeline.extern.findContinuum.convertChannelListIntoSelection(channels, trim=0, separator=';')

This function is called by findContinuumChannels. Converts a list of channels into casa selection string syntax. channels: a list of channels trim: the number of channels to trim off of each edge of a block of channels

pipeline.extern.findContinuum.convertColonDelimitersToHMSDMS(mystring, s=True, usePeriodsForDeclination=False)

This function is called by rad2radec. Converts HH:MM:SS.SSS, +DD:MM:SS.SSS to HHhMMmSS.SSSs, +DDdMMmSS.SSSs

or HH:MM:SS.SSS +DD:MM:SS.SSS to HHhMMmSS.SSSs +DDdMMmSS.SSSs or HH:MM:SS.SSS, +DD:MM:SS.SSS to HHhMMmSS.SSSs, +DD.MM.SS.SSS or HH:MM:SS.SSS +DD:MM:SS.SSS to HHhMMmSS.SSSs +DD.MM.SS.SSS

s: whether or not to include the trailing ‘s’ in both axes

pipeline.extern.findContinuum.convertSelectionIntoChannelList(selection)

Converts an arbitrary CASA selection string into an array of channel numbers ‘0~1;4~6’ – > [0,4,5] Ignores any initial spw number that preceeds a colon. Will fail if there is more than one colon.

pipeline.extern.findContinuum.countChannels(channels)

This function is called by runFindContinuum. Counts the number of channels in a CASA channel selection string. If multiple spws are found, then it returns a dictionary of counts: e.g. “1:5~20;30~40” yields 27; or ‘6~30’ yields 25

“1:5~20;30~40,2:6~30” yields {1:27, 2:25}

pipeline.extern.findContinuum.countChannelsInRanges(channels, separator=';')

This function is called by runFindContinuum. Counts the number of channels in one spw of a CASA channel selection string and return a list of numbers. e.g. “5~20;30~40” yields [16,11]

pipeline.extern.findContinuum.countPixelsAboveZero(img, useImstat=True, value=0)

This function is called by meanSpectrumFromMom0Mom8JointMask. Returns number of pixels > a specified threshold. value: threshold (default=0) Total pixels: spatial * spectral * Stokes Todd Hunter

pipeline.extern.findContinuum.countUnmaskedPixels(img, useImstat=True)

This function is called by meanSpectrumFromMom0Mom8JointMask. Returns number of unmasked pixels in an multi-dimensional image, i.e. where the internal mask is True. Total pixels: spatial * spectral * Stokes Todd Hunter

pipeline.extern.findContinuum.createNoisyQuadratic(n=1000, nsigma=3)

Function meant simply to test removeStatContQuadratic. n: number of points in synthetic noise spectrum nsigma: passed to removeStatContQuadratic

pipeline.extern.findContinuum.create_casa_quantity(myqatool, value, unit)

This function is called by CalcAtmTransmissionForImage, frames, and lsrkToRest. A wrapper to handle the changing ways in which casa quantities are invoked. Todd Hunter

pipeline.extern.findContinuum.cubeLSRKToTopo(img, imageInfo, freqrange='', prec=4, verbose=False, nchan=None, f0=None, f1=None, chanwidth=None, vis='')

This function is called by CalcAtmTransmissionForImage and writeContDat. Reads the date of observation, central RA and Dec, and observatory from an image cube and then calls lsrkToTopo to return the specified frequency range in TOPO. freqrange: desired range of frequencies (empty string or list = whole cube)

floating point list of two frequencies, or a delimited string (delimiter = ‘,’, ‘~’ or space)

prec: in fractions of Hz (only used to display the value when verbose=True) vis: read date of observation from the specified measurement set (instead of from img)

pipeline.extern.findContinuum.cubeNoiseLevel(cube, pbcube='', mask='', percentile=50, chans='')

Used in Pipeline2020 going forward. Computes median of the per-channel scaled MAD within the standard noise annulus and outside the specified mask chans: channel selection string (passed to imstat) mask: either mask image to use (e.g. the string passed to imstat will be ‘“mask” == 0’)

or a proper mask string containing >, < or ==

percentile: compute this percentile value of the per-channel scaled MAD

(examples: 50 = median, 90 = close to the highest value) if it is a list, then a list of values is returned

Returns: median of MAD, median

or [list of MADs], median

pipeline.extern.findContinuum.drawYlabel(img, typeOfMeanSpectrum, meanSpectrumMethod, meanSpectrumThreshold, peakFilterFWHM, fontsize, mask, useThresholdWithMask, normalized=False)

This function is called by runFindContinuum. Draws a descriptive y-axis label based on the origin and type of mean spectrum used. Returns: None

pipeline.extern.findContinuum.extraMaskYesOrNo(badAtmosphere, median, momSNR, momSNRCube, NpixMom, NpixMomBadAtm, NpixCubeAnywhere, TenEventSigma, MADMomOutside, MADCubeOutside, momDiffLevel, momDiffLevelBadAtm, cubeLevel, verbose=True)

Used in Pipeline2020 going forward. Also used for autoLower decision logic. badAtmosphere: either a boolean or a string (‘goodAtm’ or ‘badAtm’) momSNR: SNR outside the mask NpixMom: unused now NpixMomBadAtm: unused now NpixCubeAnywhere: unused now

pipeline.extern.findContinuum.findContinuum(img='', pbcube=None, psfcube=None, minbeamfrac=0.3, spw='', transition='', baselineModeA='min', baselineModeB='min', sigmaCube=3, nBaselineChannels=0.19, sigmaFindContinuum='auto', sigmaFindContinuumMode='auto', verbose=False, png='', pngBasename=False, nanBufferChannels=1, source='', useAbsoluteValue=True, trimChannels='auto', percentile=20, continuumThreshold=None, narrow='auto', separator=';', overwrite=True, titleText='', maxTrim=20, maxTrimFraction=1.0, meanSpectrumFile='', centralArcsec='auto', alternateDirectory='.', plotAtmosphere='transmission', airmass=1.5, pwv=1.0, channelFractionForSlopeRemoval=0.75, mask='', invert=False, meanSpectrumMethod='mom0mom8jointMask', peakFilterFWHM=10, skyTempThreshold=1.2, skyTransmissionThreshold=0.08, maxGroupsForSkyThreshold=5, minBandwidthFractionForSkyThreshold=0.2, regressionTest=False, quadraticFit=True, triangleFraction=0.83, maxMemory=- 1, tdmSkyTempThreshold=0.65, negativeThresholdFactor=1.15, vis='', singleContinuum=False, applyMaskToMask=False, plotBaselinePoints=True, dropBaselineChannels=2.0, madRatioUpperLimit=1.5, madRatioLowerLimit=1.15, projectCode='', useIAGetProfile=True, useThresholdWithMask=False, dpi=150, normalizeByMAD='auto', returnSnrs=False, overwriteMoments=False, minPeakOverMadForSFCAdjustment=[19, 19], minPixelsInJointMask=3, userJointMask='', signalRatioTier1=0.967, snrThreshold=23, mom0minsnr=5, mom8minsnr=4, overwriteMasks=True, rmStatContQuadratic=False, quadraticNsigma=1.8, bidirectionalMaskPhase2=False, makeMovie=False, returnAllContinuumBoolean=True, outdir='', allowBluePruning=True, avoidance='', returnSigmaFindContinuum=False, enableRejectNarrowInnerWindows=True, avoidExtremaInNoiseCalcForJointMask=False, buildMom8fc=True, momentdir='', amendMaskIterations='auto', skipchan=1, checkIfMaskWouldHaveBeenAmended=False, fontsize=10, thresholdForSame=0.01, keepIntermediatePngs=True, returnWarnings=False, enableOnlyExtraMask=True, useMomentDiff=True, smallBandwidthFraction=0.05, smallSpreadFraction=0.33, skipAmendMask=False, useAnnulus=True, cubeSigmaThreshold=7.5, npixThreshold=7)

This function calls functions to: 1) compute a representative ‘mean’ spectrum of a dirty cube 2) find the continuum channels 3) plot the results and writes a text file with channel ranges and frequency ranges It calls runFindContinuum up to 2 times. If meanSpectrumMethod != ‘mom0mom8jointMask’, then it runs it a second time with a reduced field size if it finds only one range of continuum channels.

Please note that the channel ranges do not necessarily map back to the visibility spectra in the measurement set(s). In order to produce channel ranges that are guaranteed to map back correctly, then you must supply a list of all desired measurement sets to the vis parameter. In this case, the final line in the .dat file created will contain the channel list in the correct frequency frame and order.

• A channel selection string suitable for the spw parameter of clean.

• The name of the final png produced.

• The aggregate bandwidth in continuum in GHz.

If returnAllContinuumBoolean = True, then it also returns a Boolean desribing

if it thinks the whole spectrum is continuum (True) or not (False) Cycle 7 pipeline usage should set returnAllContinuumBoolean = True

If returnWarnings = True, then it also returns a list of warning strings,

which may be empty [], for tooLittleBandwidth and/or tooLittleSpread

If returnSnrs = True, then it also returns two more lists: mom0snrs and mom8snrs,

and the max baseline (floating point meters) inferred from the psf image

if returnSigmaFindContinuum = True (new in PL2020), then it also returns 7 more things:

1. the final value used for sigmaFindContinuum

2. the Boolean intersectionOfSelections

3. number of central ranges that were (or would have been) dropped, summed over all runs of findContinuumChannels by the final run of runFindContinuum.

4. the result of amendMaskYesOrNo (if it was run; False otherwise)

5. the result of extraMaskYesOrNo (if it was run; False otherwise)

6. the result of extraMaskYesOrNo test 2 (if it was run; False otherwise)

7. the name of the finalMomDiff image (if amendMask=True) else the final mom8fc image

8. the path code (for amendMask)

9. the 4-letter code (for amendMask)

img: the image cube to operate upon (as opposed to specifying a

pre-existing meanSpectrumFile from a previous run)

–or–

meanSpectrumFile: an alternative pre-existing ASCII text file to use

for the mean spectrum, instead of operating on the img to create one.

Note: if you specify meanSpectrumFile and vis, you must also specify img, but

only its metadata will be used.

Parameters relevant only to meanSpectrumFile: * invert: if True, then invert the sign of intensity and add the minimum

spw: the spw name or number (integer or string integer) to put in the x-axis label;

also, it will be used to select the spw for which to generate topo channels for the *.dat file if vis is also specified; if vis is specified and spw is not, then it will search the name of the image for an spw number and use it for generating the topo channel list.

transition: the name of the spectral transition (for the plot title) avoidance: a CASA channel selection string to avoid selecting (for manual use) meanSpectrumMethod: ‘auto’, ‘mom0mom8jointMask’, ‘peakOverMad’, ‘peakOverRms’,

‘meanAboveThreshold’, ‘meanAboveThresholdOverRms’, ‘meanAboveThresholdOverMad’,

where ‘peak’ refers to the peak in a spatially-smoothed version of cube ‘auto’ invokes ‘meanAboveThreshold’ unless sky temperature range

(max-min) from the atmospheric model is more than skyTempThreshold in which case it invokes ‘peakOverMad’ instead

ALMA Cycle 5 used ‘auto’, while ALMA Cycle 6 will use ‘mom0mom8jointMask’

• minPeakOverMadForSFCAdjustment: a list of two values, first is for spws with

  channels > 2500, second is for spws with fewer channels

• pbcube: the primary beam response for img, but only used by

  meanSpectrumMethod=’mom0mom8jointMask’. If not specified, it will be searched for by changing file suffix from .residual to .pb

• psfcube: the psf response for img, but only used by

  meanSpectrumMethod=’mom0mom8jointMask’. If not specified, it will be searched for by changing file suffix from .residual to .psf, if still not found, then pruneMask will default to 6 pixels

• minbeamfrac: only used if psfcube is specified and meanSpectrumMethod=’mom0mom8jointMask’

  floating point value or ‘auto’ for 0.3 for 12m and 0.1 for 7m (maxBaseline<60m) Set to zero to prevent pruning of the joint mask entirely; otherwise pruneMask will use np.max([4,minbeamfrac*pixelsPerBeam])

• minPixelsInJointMask: if fewer than these pixels are found (after pruning), then

  use all pixels above pb=0.3 (or equivalent higher value on mitigated images). This option was essentially obseleted by the addition of pruneMask with the knowledge of beamsize from psfcube. To avoid having any effect, it should be set to < minbeamfrac*minPixelsPerBeam, which is typically 0.5*7=3.5; so use 3 or less.

• userJointMask: if specified, use this joint mask instead of computing one

  (this option is meant for test purposes only, typically *.mask2_bi)

• normalizeByMAD: can be True, False or ‘auto’ (default);

  if ‘auto’, then if atmospheric transmission varies enough across the spw,

  set to True, otherwise set to False.

  If True, then request normalization of the resulting mean spectrum by the

  xmadm spectrum using an outer annulus and outside the joint mask. The nominal annulus is the 0.2-0.3 pb response, but the range will automatically be scaled upward with minimum value in the pb cube, in order to preserve the fractional number of pixels contained in a single field’s 0.2-0.3 annulus.

• mom0minsnr: sets the threshold for the mom0 image (i.e. median + mom0minsnr*scaledMAD)

• mom8minsnr: sets the threshold for the mom8 image (i.e. median + mom8minsnr*scaledMAD)

• snrThreshold: if SNR is higher than this, then run a phase 2 mask calculation

• bidirectionalMaskPhase2: True=extend mask to negative values beyond threshold; False=Cycle6 behavior

  (True was tried but gives worse results in some cases)

• rmStatContQuadratic: if True, then fit and remove a quadratic using the subset of channels that lack signal

• makeMovie: if True, then make a png for each quadratic fit as channels are

  removed one-by-one (in the rmStatContQuadratic option)

• quadraticNsigma: the stopping threshold to use when ignoring outliers prior to

  fitting the quadratic (in the rmStatContQuadratic option). Tto more quickly test different values, be sure to set the meanSpectrumFile option and overwrite=False)

• allowBluePruning: use continuum range pruning heuristic added in Cycle 6 (default=True)

• amendMaskIterations (new in PL2020): if ‘auto’, then choose 0 for TDM and 3 for FDM;

  if 0, then do not attempt any amendMask (i.e. mimic the Cycle 7 pipeline release) if >=1, then when finished the initial findContinuum trial, create a mom8fc image and mom0fc image scaled by nchan and chanwidth in kms, and subtract them to create momDiff image. If the momDiff image has sufficient signal remaining in it, then expand the joint mask, and repeat the findContinuum process with this expanded mask as the userJointMask; if >=2, and there is still sufficient signal in the new momDiff image from the second run of findContinuum, then check if further analysis of the extra portion of the mask is necessary. If so, first check if intersecting the ranges of the first two rounds eliminates the excess, otherwise create a new spectrum from it and pass it into a third round of findContinuum with this as the meanSpectrum file. If >=3, and there is stil sufficient signal in the new momDiff, image then set sigmaFindContinuumMode=’autolower’ and sigmaFindContinuum=most recent value, and run a final round of findContinuum.

• skipAmendMask: if True, then skip the test for amend mask, and move on to onlyExtraMask

• useAnnulus: if amendMaskIterations>0, sets whether median/MAD in imageSNR be assessed in

  annulus but outside the jointmask (True) or simply outside the joint mask (False)

• enableOnlyExtraMask: if False, then do not do the extraMask or onlyExtraMask iterations

• useMomentDiff: if False, use the mom8fc image to assess AmendMask decision instead of momDiff

• checkIfMaskWouldHaveBeenAmended (new in PL2020): if True, then if amendMaskIterations=0,

  then it will still check to see if the mask would have been amended and return the result (assuming that returnSigmaFindContinuum==True) – for test purposes only

• skyTempThreshold: rms value in Kelvin (for FDM spectra<1000MHz), above which

  the meanSpectrumMethod is changed in ‘auto’ mode to peakOverMad; and above which normalizeByMAD is set True in meanSpectrumMethod=’mom0mom8jointMask’ when normalizeByMAD=’auto’ mode

• tdmSkyTempThreshold: rms value in Kelvin (for TDM and 1875MHz FDM spectra)

• skyTransmissionThreshold: threshold on (max-min)/mean, above which the

  meanSpectrumMethod is changed in ‘auto’ mode to peakOverMad; and above which normalizeByMAD is set True in meanSpectrumMethod=’mom0mom8jointMask’ when normalizeByMAD=’auto’ mode

• overwriteMoments: if True, then overwrite any existing moment0 or moment8 image

• overwriteMasks: if True, then overwrite any existing moment0 or moment8 mask image

• rmStatContQuadratic: if True, then fit and remove quadratic after removing outliers

• outdir: directory to write the .mom0 and .mom8 images, .png, .dat and meanSpectrum

  (default is same directory as the cube)

• avoidExtremaInNoiseCalcForJointMask (new in PL2020): experimental Boolean to avoid

  pixels <5%ile and >95%ile in the chauvenet imstat of mom0 and mom8 images

• triangleFraction: remove a triangle shape if the MAD of the dual-linfit residual

  is less than this fraction*originalMAD; set this parameter to zero to turn it off.

• maxMemory: behave as if the machine has this many GB of memory

• nanBufferChannels: when removing or replacing NaNs, do this many extra channels

  beyond their extent

• channelFractionForSlopeRemoval: if this many channels are initially selected,

  then fit and remove a linear slope and re-identify continuum channels Set to 1 to turn off.

• quadraticFit: if True, fit quadratic polynomial to the noise regions when

  deemed appropriate; Otherwise, fit only a linear slope

• centralArcsec: radius of central box within which to compute the mean spectrum

  default=’auto’ means start with whole field, then reduce to 1/10 if only one window is found (unless mask is specified); or ‘fwhm’ for the central square with the same radius as the PB FWHM; or a floating point value in arcseconds

• mask: a mask image preferably equal in shape to the parent image that is used to determine

  the region to compute the noise (outside the mask, i.e. mask=0) and (in the ‘meanAboveThresholdMethod’) the mean spectrum (inside the mask, i.e. mask=1). The spatial union of all masked pixels in all channels is used as the mask for each channel. Option ‘auto’ will look for the <filename>.mask that matches the <filename>.image and if it finds it, it will use it; otherwise it will use no mask. If the mask does not match in shape but is multi-channel, it will be regridded to match and written out as <filename>.mask.regrid. If it matches spatially but is single-channel, this channel will be used for all. To convert a .crtf file to a mask, use: makemask(mode=’copy’, inpimage=img, inpmask=’chariklo.crtf’, output=img+’.mask’)

• applyMaskToMask: if True, apply the mask inside the user mask image to set its masked pixels to 0

• peakFilterFWHM: value (in pixels) to presmooth each plane with a Gaussian kernel of

  this width. Set to 1 to not do any smoothing. Default = 10 which is typically about 2 synthesized beams.

• useAbsoluteValue: passed to meanSpectrum, then avgOverCube – take absolute value of

  the cube before producing mean spectrum

• useIAGetProfile: if True, then for baselineMode=’min’, use ia.getprofile instead of

  ia.getregion and subsequent arithmetic (faster, less memory exhaustive)

• continuumThreshold: if specified, begin by using only pixels above this intensity level

• sigmaCube: multiply this value by the MAD to get the threshold above

  which a voxel is included in the mean spectrum.

• baselineModeA: ‘min’ or ‘edge’, method to define the baseline in meanSpectrum()

  • percentile: control parameter for ‘min’

  Parameters relevant to ‘edge’: * nBaselineChannels: if integer, then the number of channels to use in:

  1. computing the mean spectrum with the ‘meanAboveThreshold’ methods.

  2. computing the MAD of the lowest/highest channels in findContinuumChannels if float, then the fraction of channels to use (i.e. the percentile) default = 0.19, which is 24 channels (i.e. 12 on each side) of a TDM window

• keepIntermediatePngs: if True, keep changing the png name to avoid overwriting them; only

  the final one is returned

• thresholdForSame: fractional value used to determine if the residual in the mom8fc image

  improved (‘L’), worsened (‘H’) or stayed the same (‘S’)

• cubeSigmaThreshold: minimum cube SNR for onlyExtraMask to trigger

• npixThreshold: minimum momDiff pixels for onlyExtraMask to trigger

baselineModeB: ‘min’ (default) or ‘edge’, method to define the baseline

Parameters only relevant if baselineModeB=’min’: * dropBaselineChannels: percentage of extreme values to drop

nBaselineChannels: if integer, then the number of channels to use in computing

standard deviation/MAD of the baseline channels (i.e. the blue points in the plot) if float, then it is the fraction of channels to use (i.e. the percentile)

sigmaFindContinuum: starting value for sigmaFindContinuum; passed to findContinuumChannels,

‘auto’ starts with value: TDM: singleContinuum: 9, meanAboveThreshold: 4.5, peakOverMAD: 6.5,

mom0mom8jointMask: 7.2

FDM: meanAboveThreshold: 3.5, peakOverMAD: 6.0,

mom0mom8jointMask: nchan<750: 4.2 (strong lines) or 4.5 (weak lines)

nchan>=759: 2.6 (strong lines or 3.2 (weak lines)

and adjusts it depending on results

sigmaFindContinuumMode (new in PL2020): ‘auto’, ‘autolower’, ‘fixed’ trimChannels: after doing best job of finding continuum, remove this many

channels from each edge of each block of channels found (for margin of safety) If it is a float between 0..1, then trim this fraction of channels in each group (rounding up). If it is ‘auto’ (default), use 0.1 but not more than maxTrim channels, and not more than maxTrimFraction

narrow: the minimum number of channels that a group of channels must have to survive

if 0<narrow<1, then it is interpreted as the fraction of all

channels within identified blocks

if ‘auto’, then use int(ceil(log10(nchan)))

maxTrim: in trimChannels=’auto’, this is the max channels to trim per group for

TDM spws; it is automatically scaled upward for FDM spws.

maxTrimFraction: in trimChannels=’auto’, the max fraction of channels to trim per group singleContinuum: if True, treat the cube as having come from a Single_Continuum setup;

For testing purpose. This option is overridden by the contents of vis (if specified).

negativeThresholdFactor: scale the nominal negative threshold by this factor (to

adjust the sensitivity to absorption features: smaller values=more sensitive)

signalRatioTier1: threshold for signalRatio, above which we desensitize the level to

consider line emission in order to avoid small differences in noise levels from affecting the result (e.g. that occur between serial and parallel tclean cubes) signalRatio=1 means: no lines seen, while closer to zero: more lines seen

enableRejectNarrowInnerWindows (new in PL2020): if True, then remove any inner groups

of channels that are narrower than both edge windows (when there are 3-15 groups)

verbose: if True, then print additional information during processing momentdir (new in PL2020): alternative directory to look for (and write) the moment 0&8 images png: the name of the png to produce (‘’ yields default name) pngBasename: if True, then remove the directory from img name before generating png name source: the name of the source, to be shown in the title of the spectrum.

if None, then use the filename, up to the first underscore.

overwrite: if True, or ASCII file does not exist, then recalculate the mean spectrum

writing it to <img>.meanSpectrum

if False, then read the mean spectrum from the existing ASCII file

separator: the character to use to separate groups of channels in the string returned titleText: default is img name and transition and the control parameter values plotAtmosphere: ‘’, ‘tsky’, or ‘transmission’ airmass: for plot of atmospheric transmission skipchan (newly-exposed in PL2020): used in making the spectral plot, the number of highest

and lowest intensity channels to avoid when constructing the y-axis range

pwv: in mm (for plot of atmospheric transmission) vis: comma-delimited list or python list of measurement sets to use to convert channel

ranges to topocentric frequency ranges (for use in uvcontfit or uvcontsub)

plotBaselinePoints: if True, then plot the baseline-defining points as black dots fontsize (newly-exposed in PL2020): size to use for channel ranges, axis labels and plot legend dpi: dots per inch to use in writing the png (106 produces 861x649 pixels)

150 produces 1218x918

projectCode: a string to prepend to the title of the plot (useful for regression testing

where you can put the page number from the full PDF when running only a subset)

buildMom8fc (new in PL2020): if True, then when finished, generate “mom8fc” and “mom0fc”

images using selected channels

smallBandwidthFraction: threshold for returning a warning for too little bandwidth found smallSpreadFraction: threshold for returning a warning for too little bandwidth spread found

pipeline.extern.findContinuum.findContinuumChannels(spectrum, nBaselineChannels=16, sigmaFindContinuum=3, nanmin=None, baselineMode='min', trimChannels='auto', narrow='auto', verbose=False, maxTrim=20, maxTrimFraction=1.0, separator=';', peakOverMad=0, fitResult=None, maxGroupsForMaxTrimAdjustment=3, lowHighBaselineThreshold=1.5, lineSNRThreshold=20, negativeThresholdFactor=1.15, dropBaselineChannels=2.0, madRatioUpperLimit=1.5, madRatioLowerLimit=1.15, projectCode='', fCCiteration=0, signalRatioTier1=0.965, signalRatioTier2=0.94, sigmaFindContinuumMode='auto', enableRejectNarrowInnerWindows=True)

This function is called by runFindContinuum. Trys to find continuum channels in a spectrum, based on a threshold or some number of edge channels and their median and standard deviation. Inputs: spectrum: a one-dimensional array of intensity values nBaselineChannels: number of channels over which to compute standard deviation/MAD sigmaFindContinuum: value to multiply the standard deviation by then add

to median to get threshold. Default = 3.

narrow: the minimum number of channels in a contiguous block to accept

if 0<narrow<1, then it is interpreted as the fraction of all

channels within identified blocks

nanmin: the value that NaNs were replaced by in previous steps baselineMode: ‘min’ or ‘edge’, ‘edge’ will use nBaselineChannels/2 from each edge trimChannels: if integer, use that number of channels. If float between

0 and 1, then use that fraction of channels in each contiguous list (rounding up). If ‘auto’, then use 0.1 but not more than maxTrim channels. The ‘auto’ mode can also cause trimChannels to be set to 13 or 6 in certain circumstances (when moderately strong lines are present).

maxTrim: if trimChannels=’auto’, this is the max channels to trim per group for TDM spws; it is automatically scaled upward for FDM spws. maxTrimFraction: in trimChannels=’auto’, the max fraction of channels to trim per group separator: the character to use to separate groups of channels in the string returned negativeThresholdFactor: scale the nominal negative threshold by this factor (to adjust

sensitivity to absorption features: smaller values=more sensitive)

dropBaselineChannels: percentage of extreme values to drop in baseline mode ‘min’ madRatioUpperLimit, madRatioLowerLimit: if ratio of MADs is between these values, then

apply dropBaselineChannels when defining the MAD of the baseline range

fitResult: coefficients from linear or quadratic fit, only relevant when meanSpectrumMethod is

not ‘mom0mom8jointMask’

signalRatioTier1: threshold for signalRatio, above which we desensitize the level to

consider line emission in order to avoid small differences in noise levels from affecting the result (e.g. that occur between serial and parallel tclean cubes) signalRatio=1 means: no lines seen, while closer to zero: more lines seen

signalRatioTier2: second threshold for signalRatio, used for FDM spws (nchan>192) and

for cases of peakOverMad < 5. Should be < signalRatioTier1.

sigmaFindContinuumMode: if ‘auto’, then do not desensitize with signalRatioTier1

Returns: 1 list of channels to use (separated by the specified separator) 2 list converted to ms channel selection syntax 3 positive threshold used 4 median of the baseline-defining channels 5 number of groups found 6 correctionFactor used 7 inferred true median 8 scaled MAD of the baseline-defining channels 9 correction factor applied to get the inferred true median 10 negative threshold used 11 lineStrength factor 12 singleChannelPeaksAboveSFC: how many cases where only a single channel exceeds the threshold 13 allGroupsAboveSFC 14 spectralDiff (percentage of median) 15 value of trimChannels parameter 16 Boolean describing whether the low values were used as the baseline 17 value of the narrow parameter 18 tuple containing the channel numbers of the baseline channels, and their respective y-axis values 19 value of madRatio: MAD/MAD_after_dropping_extreme_channels

This value will be larger if there are many tall single channel peaks.

20 number of ranges it dropped (or would have dropped if it was enabled)

pipeline.extern.findContinuum.findOuterAnnulusForPBCube(pbcube, imstatListit=False, imstatVerbose=False)

Given a PB cube, finds the minimum sensitivity value, then computes the corresponding higher value to form an annulus. Returns 0.2-0.3 for normal images that have not been mitigated. The factor of 1.15 below will effectively mimic a corresponding higher range. For example: CASA <247>: au.gaussianBeamResponse(au.gaussianBeamOffset(0.5)/1.15, fwhm=1) Out[247]: 0.59207684245045789 CASA <248>: au.gaussianBeamResponse(au.gaussianBeamOffset(0.8)/1.15, fwhm=1) Out[248]: 0.8447381483786558

pipeline.extern.findContinuum.findSpectralAxis(img)

This function is called by computeStatisticalSpectrumFromMask, getImageInfo, findContinuum and numberOfChannelsInCube. Finds the spectral axis number of an image tool instance, or an image. img: string or iatool instance

pipeline.extern.findContinuum.findWidestContiguousListInChannelRange(channels, channelRange, continuumChannels)

Given a list of channels, find the longest contiguous list in the first half or second half example: CASA <43>: fc.findWidestContiguousListInOtherHalf([2,3,7,8,9,12,13,14,15], True, 20) Out[43]: [12, 13, 14, 15] channels: list of blue points channel numbers channelRange: range of channels to look for widest set of blue points continuumChannels: current selection of continuuum channels

pipeline.extern.findContinuum.findWidestContiguousListInOtherHalf(channels, lowerHalf, nchan)

Given a list of channels, find the longest contiguous list in the first half or second half example: CASA <43>: fc.findWidestContiguousListInOtherHalf([2,3,7,8,9,12,13,14,15], True, 20) Out[43]: [12, 13, 14, 15] channels: list of blue points channel numbers lowerHalf: a Boolean (True for lower half, False for upper half) nchan: total channels in spectrum

pipeline.extern.findContinuum.flattenMask(mask, outfile='', overwrite=True)

Takes a multi-channel CASA image mask and propagates all pixels to a new single plane image mask outfile: default name = <mask>.flattened - Todd Hunter

pipeline.extern.findContinuum.frames(velocity=286.7, datestring='2005/11/01/00:00:00', ra='05:35:28.105', dec='-069.16.10.99', equinox='J2000', observatory='ALMA', prec=4, verbose=True, myme='', myqa='', restFreq=345.79599, veltype='optical')

This function is called by restToTopo. Converts an optical velocity into barycentric, LSRK and TOPO frames. Converts a radio LSRK velocity into TOPO frame. Inputs: velocity: in km/s datestring: “YYYY/MM/DD/HH:MM:SS” ra: “05:35:28.105” dec: “-069.16.10.99” equinox: “J2000” observatory: “ALMA” prec: precision to display (digits to the right of the decimal point) veltype: ‘radio’ or ‘optical’ restFreq: in Hz, GHz or a string with units Returns: * TOPO frequency in Hz * difference between LSRK-TOPO in km/sec * difference between LSRK-TOPO in Hz

pipeline.extern.findContinuum.gatherWarnings(selection, chanInfo, smallBandwidthFraction, smallSpreadFraction)

Used in Pipeline2020 going forward. Return list of warning strings (often empty).

pipeline.extern.findContinuum.gaussianBeamOffset(response=0.5, fwhm=1.0)

++++ This function is used by pipeline in computeStatisticalSpectrumFromMask(). Computes the radius at which the response of a Gaussian beam drops to the specified level. For the inverse function, see gaussianBeamResponse.

pipeline.extern.findContinuum.gaussianBeamResponse(radius, fwhm)

++++ This function is used by pipeline in computeStatisticalSpectrumFromMask(). Computes the gain at the specified radial offset from the center of a Gaussian beam. For the inverse function, see gaussianBeamOffset. Required inputs: radius: in arcseconds fwhm: in arcseconds

pipeline.extern.findContinuum.getDateObs(img, myia=None)

This function is called by cubeLSRKToTopo. Returns string of format: ‘2014/05/22/08:47:05’ suitable for lsrkToTopo

pipeline.extern.findContinuum.getEquinox(img, myia=None)

This function is called by cubeLSRKToTopo.

pipeline.extern.findContinuum.getFieldnameFromPipelineImageName(img, verbose=False)

Extracts the field name from the pipeline image file name. -Todd Hunter

pipeline.extern.findContinuum.getFreqType(img)

This function is called by runFindContinuum and cubeLSRKToTopo.

pipeline.extern.findContinuum.getImageInfo(img, returnBeamAreaPerChannel=False)

This function is called by findContinuum and meanSpectrum. Extract the beam and pixel information from a CASA image. This was copied from getFitsBeam in analysisUtils.py. Returns: A list of 12 things: bmaj, bmin, bpa, cdelt1, cdelt2,

naxis1, naxis2, frequency, shape, crval1, crval2, maxBaseline

Beam angles are in arcseconds (bpa in degrees), crvals are in radians Frequency is in GHz and is the central frequency MaxBaseline is in meters, inferred from freq and beamsize (will be zero for .residual images)

pipeline.extern.findContinuum.getMaxpixpos(img)

This function is called by findContinuum, but only in Cycle 4+5 heuristic.

pipeline.extern.findContinuum.getMemorySize()

This function is called by findContinuum and runFindContinuum.

pipeline.extern.findContinuum.getObservationStart(vis, obsid=- 1, verbose=False)

Reads the start time of the observation from the OBSERVATION table (using tb tool) and reports it in MJD seconds. obsid: if -1, return the start time of the earliest obsID -Todd Hunter

pipeline.extern.findContinuum.getObservationStartDate(vis, obsid=0, delimiter='-', measuresToolFormat=True)

Uses the tb tool to read the start time of the observation and reports the date. See also getObservationStartDay. Returns: ‘2013-01-31 07:36:01 UT’ measuresToolFormat: if True, then return ‘2013/01/31/07:36:01’, suitable for lsrkToTopo -Todd Hunter

pipeline.extern.findContinuum.getObservatoryName(vis)

+++++++ This function is not used by pipeline, because it is only used by getScienceSpws. Returns the observatory name in the specified ms.

pipeline.extern.findContinuum.getScienceSpws(vis, intent='OBSERVE_TARGET#ON_SOURCE', returnString=True, tdm=True, fdm=True, mymsmd=None, sqld=False)

+++++++ This function is not used by pipeline, because it is only used by isSingleContinuum. Return a list of the each spw with the specified intent. For ALMA data, it ignores channel-averaged and SQLD spws. returnString: if True, it returns: ‘1,2,3’

if False, it returns: [1,2,3]

pipeline.extern.findContinuum.getSpwFromPipelineImageName(img, verbose=False)

Extracts the spw ID from the pipeline image file name. -Todd Hunter

pipeline.extern.findContinuum.getTelescope(img, myia=None)

This function is called by CalcAtmTransmissionForImage and cubeLSRKToTopo.

pipeline.extern.findContinuum.getcube(i, filename='cubelist.txt')

++++ This function is not used by pipeline. Translates a PDF page number to a cube name, for regression purposes, by reading the specified file. filename: a file containing 2-column lines like: ‘1 mycube.residual’

pipeline.extern.findContinuum.grep(filename, arg)

This function is called only by maskArgumentMismatch and centralArcsecArgumentMismatch. Runs grep for string <arg> in a subprocess and reports stdout and stderr.

pipeline.extern.findContinuum.help(match='', debug=False)

Print an alphabetized list of all the defined functions at the top level in this python file. match: limit the list to those functions containing this string (case insensitive) – Todd Hunter

pipeline.extern.findContinuum.imagePercentileNoMask(img, percentiles)

percentiles: a single value or a list Returns: a single value or a list

pipeline.extern.findContinuum.imageSNR(img, axes=[], mask='', maskWithAnnulus='', useAnnulus=False, verbose=False, applyMaskToAll=False, returnAllStats=True, returnFractionalDifference=False, usepb='')

Uses imstat (which automatically applies the internal mask, i.e avoids the black edges of the image) to compute (max-median)/scaledMAD axes: passed to imstat (use [0,1,2] to get a vector SNR per channel) img: a 2D image or a cube mask: additional mask image (or expression) to apply when computing everything but max maskWithAnnulus: additional mask image (or expression) to use for median and MAD

if useAnnulus==True (but always take the smaller of the 2 possible MADs)

applyMaskToAll: also apply the provided mask when computing the max (default=False=anywhere) returnAllStats: if True, then also return the peak, median and scaledMAD where the

peak has not had the median subtracted (in contrast to the SNR)

returnFractionalDifference: if True, then also return the fraction of negative pixels usepb: (optional) name of a single-plane pb image to include in the mask (at >0.23 level) -ToddHunter

pipeline.extern.findContinuum.imageSNRAnnulus(mymomDiff, jointMaskTest, jointMaskTestWithAnnulus, applyMaskToAll=True)

pipeline.extern.findContinuum.invertChannelRanges(invertstring, nchan=0, startchan=0, vis='', spw='', separator=';')

Takes a CASA channel selection string and inverts it. -Todd Hunter

pipeline.extern.findContinuum.isSingleContinuum(vis, spw='', source='', intent='OBSERVE_TARGET', verbose=False, mymsmd=None)

+++++++ This function is not used by pipeline, rather it is an expected input parameter. Checks whether an spw was defined as single continuum in the OT by looking at the transition name in the SOURCE table of a measurement set. vis: a single measurement set, a comma-delimited list, or a python list

(only the first one will be used)

Optional parameters: spw: can be integer ID or string integer ID; if not specified, then it

will use the first science spw

source: passed to transition(); if not specified, it will use the first one intent: if source is blank then use first one with matching intent and spw mymsmd: an existing instance of the msmd tool

pipeline.extern.findContinuum.is_binary(filename)

This function is called by runFindContinuum. Return true if the given filename appears to be binary. Works in python 2.7 and 3

pipeline.extern.findContinuum.linfit(x, y, yerror, pinit=[0, 0])

This function is called by atmosphereVariation in Cycle 4+5+6 and by runFindContinuum in Cycle 4+5. Basic linear function fitter with error bars in y-axis data points. Uses scipy.optimize.leastsq(). Accepts either lists or arrays. .. rubric:: Example

lf = au.linfit() lf.linfit(x, y, yerror, pinit)

Input:

x, y: x and y axis values yerror: uncertainty in the y-axis values (vector or scalar) pinit contains the initial guess of [slope, intercept]

Output:

The fit result as: [slope, y-intercept]

pipeline.extern.findContinuum.locatePBCube(cube)

pipeline.extern.findContinuum.lsrkToRest(lsrkFrequency, velocityLSRK, datestring, ra, dec, equinox='J2000', observatory='ALMA', prec=4, verbose=True)

This function is called by lsrkToTopo. Converts an LSRK frequency, LSRK velocity, and observing date/direction to the corresponding frequency in the rest frame. Inputs: lsrkFrequency: floating point value in Hz or GHz, or a string with units velocityLSRK: floating point value in km/s datestring: “YYYY/MM/DD/HH:MM:SS” (format = image header keyword ‘date-obs’) ra: string “HH:MM:SS.SSSS” dec: string “DD.MM.SS.SSSS” or “DD:MM:SS.SSSS” (colons will be replaced with .) prec: only used to display the value when verbose=True Returns: the Rest frequency in Hz

pipeline.extern.findContinuum.lsrkToTopo(lsrkFrequency, datestring, ra, dec, equinox='J2000', observatory='ALMA', prec=4, verbose=False)

This function is called by cubeLSRKToTopo. Converts an LSRKfrequency and observing date/direction to the corresponding frequency in the TOPO frame. Inputs: lsrkFrequency: floating point value in Hz or GHz, or a string with units datestring: “YYYY/MM/DD/HH:MM:SS” (format = image header keyword ‘date-obs’) ra: string “HH:MM:SS.SSSS” dec: string “DD.MM.SS.SSSS” or “DD:MM:SS.SSSS” (colons will be replaced with .) prec: only used to display the value when verbose=True Returns: the TOPO frequency in Hz

pipeline.extern.findContinuum.makeUvcontsub(files='*.dat', fitorder=1, useFrequency=False)

+++++++ This function is not used anywhere else in this file. Takes a list of output files from findContinuum and builds an spw selection for uvcontsub, then prints the resulting commands to the screen. files: a list of files, either a comma-delimited string, a python list, or a wildcard string fitorder: passed through to the uvcontsub useFrequency: if True, then produce selection string in frequency; otherwise use channels

Note: frequencies in the findContinuum .dat file are topo, which is what uvcontsub wants.

pipeline.extern.findContinuum.maskArgumentMismatch(mask, meanSpectrumFile, useThresholdWithMask)

This function is called by checkForMismatch. Determines if the requested mask does not match what was used to generate the specified meanSpectrumFile. Returns: True or False

pipeline.extern.findContinuum.maxLengthOfLists(lists)

This function is called by findContinuumChannels. lists: a list if lists Returns: an integer that is the length of the longest list

pipeline.extern.findContinuum.meanSpectrum(img, nBaselineChannels=16, sigmaCube=3, verbose=False, nanBufferChannels=2, useAbsoluteValue=False, baselineMode='edge', percentile=20, continuumThreshold=None, meanSpectrumFile='', centralArcsec=- 1, imageInfo=[], chanInfo=[], mask='', meanSpectrumMethod='peakOverRms', peakFilterFWHM=15, iteration=0, applyMaskToMask=False, useIAGetProfile=True, useThresholdWithMask=False, overwrite=False)

This function is not used by Cycle 6 pipeline, but remains as manual user option. Computes a threshold and then uses it to compute the average spectrum across a CASA image cube, via the specified method. Inputs: nBaselineChannels: number of channels to use as the baseline in

the ‘meanAboveThreshold’ methods.

baselineMode: how to select the channels to use as the baseline:

‘edge’: use an equal number of channels on each edge of the spw

‘min’: use the percentile channels with the lowest absolute intensity

sigmaCube: multiply this value by the rms to get the threshold above which a pixel

is included in the mean spectrum

nanBufferChannels: when removing or replacing NaNs, do this many extra channels

beyond their actual extent

useAbsoluteValue: passed to avgOverCube percentile: used with baselineMode=’min’ continuumThreshold: if specified, only use pixels above this intensity level meanSpectrumFile: name of ASCII file to produce to speed up future runs centralArcsec: default=-1 means whole field, or a floating point value in arcsec mask: a mask image (e.g. from clean); restrict calculations to pixels with mask=1 chanInfo: the list returned by numberOfChannelsInCube meanSpectrumMethod: ‘peakOverMad’, ‘peakOverRms’, ‘meanAboveThreshold’,

‘meanAboveThresholdOverRms’, or ‘meanAboveThresholdOverMad’

• ‘meanAboveThreshold’: uses a selection of baseline channels to compute the

  rms to be used as a threshold value (similar to constructing a moment map).

• ‘meanAboveThresholdOverRms/Mad’: scale spectrum by RMS or MAD

• ‘peakOverRms/Mad’ computes the ratio of the peak in a spatially-smoothed

  version of cube to the RMS or MAD. Smoothing is set by peakFilterFWHM.

peakFilterFWHM: value used by ‘peakOverRms’ and ‘peakOverMad’ to presmooth

each plane with a Gaussian kernel of this width (in pixels) Set to 1 to not do any smoothing.

useIAGetProfile: if True, then for peakOverMad, or meanAboveThreshold with

baselineMode=’min’, then use ia.getprofile instead of ia.getregion and the subsequent arithmetic (because it should be faster)

useThresholdWithMask: if False, then just take mean rather than meanAboveThreshold

when a mask has been specified

Returns 8 items:

• avgspectrum (vector)

• avgspectrumAboveThresholdNansRemoved (vector)

• avgspectrumAboveThresholdNansReplaced (vector)

• threshold (scalar)

• edgesUsed: 0=lower, 1=upper, 2=both

• nchan (scalar)

• nanmin (the value used to replace NaNs)

• percentagePixelsNotMasked

pipeline.extern.findContinuum.meanSpectrumFromMom0Mom8JointMask(cube, imageInfo, nchan, pbcube=None, psfcube=None, minbeamfrac=0.3, projectCode='', overwriteMoments=False, overwriteMasks=True, phase2=True, normalizeByMAD=True, minPixelsInJointMask=3, initialQuadraticImprovementThreshold=1.6, userJointMask='', snrThreshold=23, mom0minsnr=5, mom8minsnr=4, rmStatContQuadratic=True, bidirectionalMaskPhase2=False, outdir='', avoidExtremaInNoiseCalcForJointMask=False, momentdir='', statistic='mean')

This function is called by runFindContinuum when meanSpectrumMethod=’mom0mom8jointMask’. This is the new heuristic for Cycle 6 which creates the moment 0 and moment 8 images for a cube, takes their union and determines the mean spectrum by calling computeStatisticalSpectrumFromMask(), which uses ia.getprofile. pbcube: if not specified, then assume ‘.residual’ should be replaced in the name by ‘.pb’ overwriteMoments: rebuild the mom0 and mom8 images even if they already exist overwriteMasks: rebuild the mom0 and mom8 mask images even if they already exist phase2: if True, then run a second phase if SNR is high minPixelsInJointMask: if fewer than these pixels are found, then use all pixels above pb=0.3 userJointMask: if specified, use this joint mask instead of computing one; if it is a cube mask,

as from tclean, then this function will form a flattened version first, and then use that

snrThreshold: if SNR is higher than this, and phase2==True, then run a phase 2 mask calculation mom0minsnr: sets the threshold for the mom0 image (i.e. median + mom0minsnr*scaledMAD of the whole image) mom8minsnr: sets the threshold for the mom8 image (i.e. median + mom8minsnr*scaledMAD of thw whole image) rmStatContQuadratic: if True, then do not remove the old-style quadratic in this function in

favor of the new-style removal elsewhere

bidirectionalMaskPhase2: True=extend mask to negative values beyond threshold; False=Cycle6 behavior outdir: directory to write the mom0, mom8, masks, .dat and mean spectrum text files avoidExtremaInNoiseCalcForJointMask: experimental Boolean to avoid pixels <5%ile and >95%ile

in the chauvenet imstat of mom0 and mom8 images

momentdir: alternate directory to look for existing mom0 and mom8 images Returns: 13 things:

• 1. the meanSpectrum

• 2. a Boolean which states whether normalization was applied

• 3. the number of pixels in the mask

• 4. a Boolean for whether the mask reverted to pb-based

• 5. a Boolean for whether a quadratic was removed

• 6. the initialQuadraticImprovementRatio

• 7. the list of 3 mom0snrs

• 8. the list of 3 mom8snrs

• 9. the number of regions pruned

• 10. number of pixels in moment 8 mask

• 11. mom0peak (Jy*km/s)

• 12. mom8peak (Jy)

• 13. name of jointMask file

pipeline.extern.findContinuum.medianCorrected(baselineMode, percentile, median, mad, signalRatio, useLowBaseline)

This function is called by findContinuumChannels. Computes the true median of a datastream from the observed median and MAD of the lowest Nth percentile points. signalRatio: when there is a lot of signal, we need to reduce the

correction factor because it is less like Gaussian noise It is 1.0 if no lines are present, 0.25 if half the channels have signal, etc.

pipeline.extern.findContinuum.mjdSecondsToMJDandUT(mjdsec, debug=False, prec=6, delimiter='-')

This function is called by mjdToUT. Converts a value of MJD seconds into MJD, and into a UT date/time string. prec: 6 means HH:MM:SS, 7 means HH:MM:SS.S example: (56000.0, ‘2012-03-14 00:00:00 UT’) Caveat: only works for a scalar input value

pipeline.extern.findContinuum.mjdToUT(mjd, use_metool=True, prec=6)

This function is called by getDateObs. Converts an MJD value to a UT date and time string such as ‘2012-03-14 00:00:00 UT’ use_metool: whether or not to use the CASA measures tool if running from CASA.

This parameter is simply for testing the non-casa calculation.

-Todd Hunter

pipeline.extern.findContinuum.nanmean(a, axis=0)

This function is called by findContinuumChannels, runFindContinuum, and avgOverCube. Takes the mean of an array, ignoring the nan entries

pipeline.extern.findContinuum.nanmedian(x, axis=0, preop=None)

This function is called by MAD, avgOverCube, and meanSpectrum. Compute the median along the given axis ignoring nan values.

Parameters

• x (array_like) – Input array.

• axis (int or None, optional) – Axis along which the median is computed. Default is 0. If None, compute over the whole array x.

• preop (function) – function to apply on 1d slice after removing the NaNs and before computing median

Returns

m – The median of x along axis.

Return type

float

See also

nanstd, nanmean, numpy.nanmedian

Examples

>>> from scipy import stats
>>> a = np.array([0, 3, 1, 5, 5, np.nan])
>>> stats.nanmedian(a)
array(3.0)

>>> b = np.array([0, 3, 1, 5, 5, np.nan, 5])
>>> stats.nanmedian(b)
array(4.0)

Example with axis:

>>> c = np.arange(30.).reshape(5,6)
>>> idx = np.array([False, False, False, True, False] * 6).reshape(5,6)
>>> c[idx] = np.nan
>>> c
array([[  0.,   1.,   2.,  nan,   4.,   5.],
       [  6.,   7.,  nan,   9.,  10.,  11.],
       [ 12.,  nan,  14.,  15.,  16.,  17.],
       [ nan,  19.,  20.,  21.,  22.,  nan],
       [ 24.,  25.,  26.,  27.,  nan,  29.]])
>>> stats.nanmedian(c, axis=1)
array([  2. ,   9. ,  15. ,  20.5,  26. ])

pipeline.extern.findContinuum.numberOfChannelsInCube(img, returnFreqs=False, returnChannelWidth=False, verbose=False)

This function is called by findContinuum, cubeLSRKToTopo, computeStatisticalSpectrumFromMask, and meanSpectrum. Finds the number of channels in a CASA image cube. returnFreqs: if True, then also return the frequency of the

first and last channel (in Hz)

returnChannelWidth: if True, then also return the channel width (in Hz) verbose: if True, then print the frequencies of first and last channel -Todd Hunter

pipeline.extern.findContinuum.oneEvent(npts, events=1.0, positive=True, verbose=False)

This function is called by meanSpectrumFromMom0Mom8JointMask. For a specified size of a Gaussian population of data, compute the sigma that gives just less than one event, by using scipy.special.erfinv. Inputs: positive: if True, then only considers positive events. events: how many events to allow verbose: passed to sigmaEvent Return: sigma: floating point value

pipeline.extern.findContinuum.onlyExtraMaskYesOrNo(badAtmosphere, median, momDiffSNR, momDiffSNRCube, NpixMomDiff, NpixMomDiffBadAtm, NpixCubeAnywhere, TenEventSigma, momDiffMAD, MADCubeOutside, cubePeak, cubeMedian, momDiffLevel, momDiffLevelBadAtm, cubeLevel, sigmaThreshold=7.5, npixThreshold=7, verbose=True)

Used in Pipeline2020 going forward. badAtmosphere: either a boolean or a string (‘goodAtm’ or ‘badAtm’) momDiffSNR: SNR where peak is measured over whole image (instead of outside mask),

(mom scaledMAD is in denominator)

momDiffSNRCube: SNR where peak is measured over whole image (instead of outside mask),

(cube scaledMAD is in denominator)

NpixCubeAnywhere: unused now

pipeline.extern.findContinuum.parseFrequencyArgument(bandwidth)

This function is called by parseFrequencyArgumentToGHz, topoFreqToChannel and cubeLSRKToTopo. Converts a string frequency into floating point in Hz, based on the units. If the units are not present, then the value is simply converted to float.

pipeline.extern.findContinuum.parseFrequencyArgumentToGHz(bandwidth)

This function is called by frames and lsrkToRest. Converts a frequency string into floating point in GHz, based on the units. If the units are not present, then the value is assumed to be GHz if less than 1000.

pipeline.extern.findContinuum.pickAutoTrimChannels(totalChannels, length, maxTrim, medianWidthOfRanges)

This function is called by splitListIntoContiguousListsAndTrim and pickTrimString. Automatic choice of number of trimChannels as a function of the number of potential continuum channels in an spw. length: number of channels in the list of potential continuum channels Returns: an integer or floating point value

pipeline.extern.findContinuum.pickNarrow(length)

This function is called by pickNarrowString and findContinuumChannels. Automatically picks a setting for the narrow parameter of findContinuumChannels() based on the number of channels in the spectrum. Returns: an integer Examples: This formula results in the following values: length: 64,128,240,480,960,1920,3840,7680: return: 2, 3, 3, 3, 3, 4, 4, 4 (ceil(log10)) **** current function ****

2, 2, 2, 3, 3, 3, 4, 4 (round_half_up(log10)) 1, 2, 2, 2, 2, 3, 3, 3 (floor(log10)) 5, 5, 6, 7, 7, 8, 9, 9 (ceil(log)) 4, 5, 5, 6, 7, 8, 8, 9 (round_half_up(log)) 4, 4, 5, 6, 6, 7, 8, 8 (floor(log))

pipeline.extern.findContinuum.pickNarrowString(narrow, length, narrowValueModified=None)

This function is called by runFindContinuum. Generate a string describing the setting of the narrow parameter. Returns: a string

pipeline.extern.findContinuum.pickRandomError(seed=None)

Picks a random value from a Gaussian distribution with mean 0 and standard deviation = 1. seed: if specified, then first reseed with random.seed(seed) -Todd Hunter

pipeline.extern.findContinuum.pickRandomErrors(nvalues=1)

Picks a series of random values from a Gaussian distribution with mean 0 and standard deviation = 1 and returns it as an array. -Todd Hunter

pipeline.extern.findContinuum.pickTrimString(totalChannels, trimChannels, length, maxTrim, selection)

This function is called by runFindContinuum. Generate a string describing the setting of the trimChannels parameter. totalChannels: total number of points in spectrum trimChannels: string (e.g. ‘auto’) or integer (e.g. 20) or float (e.g. 0.1) length: number of channels in the spectrum (now unused) selection: continuum channel selection string Returns: a string

pipeline.extern.findContinuum.pick_sFC_TDM(meanSpectrumMethod, singleContinuum)

This function is called by runFindContinuum. Chooses the value of sigmaFindContinuum for TDM datasets based on the meanSpectrumMethod and whether the user requested single continuum in the Observing Tool.

pipeline.extern.findContinuum.plotChannelSelections(ax1, selection, separator, avgspectrumAboveThreshold, skipchan, medianTrue, threshold, secondSelectionAdded='', lineColor='c')

avgspectrumAboveThreshold: computed in runFindContinuum medianTrue, threshold: computed by findContinuumChannels Returns labelDescs: list of plot artists that may be removed and replaced later

pipeline.extern.findContinuum.plotMeanSpectrum(meanSpectrumFile, plotfile='', selection='')

Reads a *.findcont.residual.meanSpectrum.<method> file and plots it as intensity vs. channel. selection: channel selection string to draw in cyan horizontal lines

pipeline.extern.findContinuum.plotStatisticalSpectrumFromMask(cube, jointMask='', pbcube=None, statistic='mean', normalizeByMAD=False, png='', jointMaskForNormalize='')

Simple wrapper to call computeStatisticalSpectrumFromMask and plot it. jointMask: either a mask image, or an expression with < or > on an mask

image

pipeline.extern.findContinuum.polyfit(x, y, yerror, pinit=[0, 0, 0, 0])

This function is called by removeInitialQuadraticIfNeeded in Cycle 6, and by runFindContinuum in Cycle 4+5. Basic second-order polynomial function fitter with error bars in y-axis data points. Uses scipy.optimize.leastsq(). Accepts either lists or arrays. Input:

x, y: x and y axis values yerror: uncertainty in the y-axis values (vector or scalar) pinit contains the initial guess of [slope, intercept] y = order2coeff*(x-xoffset)**2 + slope*(x-xoffset) + y-intercept

Output:

The fit result as: [xoffset, order2coeff, slope, y-intercept]

pipeline.extern.findContinuum.propagateMaskToAllChannels(mask, axis=3)

This function is called by meanSpectrum. Takes a spectral mask array, and propagates every masked spatial pixel to all spectral channels of the array. Returns: a 2D mask (of the same spatial dimension as the input,

but with only 1 channel)

pipeline.extern.findContinuum.pruneMask(mymask, psfcube=None, minbeamfrac=0.3, prunesize=6.0, nchan=1, overwrite=True)

available in CASA >= 5.4.0-X (see CAS-11335) Removes any regions smaller than prunesize contiguous pixels. psfcube: if specified, then use minbeamfrac * pixelsPerBeam, otherwise use prunesize prunesize: value used if psfcube is None; use 6 because pipeline size mitigation could

produce images that have only 3 pixels across the beam, meaning only ~7-8 pts/beam

nchan: number of channels to process, will always be 1 in findContinuum’s use case overwrite: if True, replaces mymask with pruned mask, putting original in mymask.prepruned

if False, is simply creates: mymask.pruned, but only if it pruned anything

If all regions are pruned, it checks if it is ACA7m (maxBaseline < 60m) and if so, tries again with minbeamfrac*0.5.

pipeline.extern.findContinuum.rad2radec(ra=0, dec=0, prec=5, verbose=True, component=0, replaceDecDotsWithColons=True, hmsdms=False, delimiter=', ', prependEquinox=False, hmdm=False)

This function is called by cubeLSRKToTopo. Convert a position in RA/Dec from radians to sexagesimal string which is comma-delimited, e.g. ‘20:10:49.01, +057:17:44.806’. The position can either be entered as scalars via the ‘ra’ and ‘dec’ parameters, as a tuple via the ‘ra’ parameter, or as an array of shape (2,1) via the ‘ra’ parameter. replaceDecDotsWithColons: replace dots with colons as the Declination d/m/s delimiter hmsdms: produce output of format: ‘20h10m49.01s, +057d17m44.806s’ hmdm: produce output of format: ‘20h10m49.01, +057d17m44.806’ (for simobserve) delimiter: the character to use to delimit the RA and Dec strings output prependEquinox: if True, insert “J2000” before coordinates (i.e. for clean or simobserve)

pipeline.extern.findContinuum.readContDat(filename)

Reads the channel selection for a _findContinuum.dat file

pipeline.extern.findContinuum.readContDatAggregateContinuum(filename)

Reads the aggregate continuum from a _findContinuum.dat file. Returns: value in GHz

pipeline.extern.findContinuum.readMeanSpectrumFITSFile(meanSpectrumFile, unit=0, nanBufferChannels=1)

++++++ This function is not used by the pipeline. Reads a spectrum from a FITS table, such as one output by spectralcube. Returns: 8 items * average spectrum * average spectrum with the NaNs replaced with the minimum value * threshold used (currently hardcoded to 0.0) * edges used (currently hardcoded to 2) * number of channels * the minimum value * first frequency * last frequency

pipeline.extern.findContinuum.readPreviousMeanSpectrum(meanSpectrumFile, verbose=False, invert=False)

++++++ This function is not used by the pipeline. Read a previous calculated mean spectrum and its parameters, or an ASCII file created by the CASA viewer (or tt.ispec). Note: only the intensity column(s) are returned, not the

channel/frequency columns.

This function will not typically be used by the pipeline, only manual users. Returns: 11 things:

avgspectrum, avgSpectrumNansReplaced, threshold, edgesUsed, nchan, nanmin, firstFreq, lastFreq, centralArcsec, mask, percentagePixelsNotMasked

pipeline.extern.findContinuum.readViewerOutputFile(lines, debug=False)

++++++ This function is not used by the pipeline. Reads an ASCII spectrum file produced by the CASA viewer or by tt.ispec. Returns: 4 items: 2 arrays and 2 strings: * array for x-axis, array for y-axis * x-axis units, intensity units

pipeline.extern.findContinuum.rejectNarrowInnerWindowsChannels(channels, fCCiteration=0)

This function is called by findContinuumChannels. If there are 3-15 groups of channels, then remove any inner window that is narrower than both edge windows. Returns: a list of channels

pipeline.extern.findContinuum.removeIfNecessary(imgname)

pipeline.extern.findContinuum.removeInitialQuadraticIfNeeded(avgSpectrum, initialQuadraticImprovementThreshold=1.6)

This function is called by runFindContinuum when meanSpectrumMethod=’mom0mom8jointMask’. Fits a quadratic to the specified spectrum, and removes it if the MAD will improve by more than a factor of threshold

pipeline.extern.findContinuum.removeNaNs(a, replaceWithMin=False, verbose=False, nanBufferChannels=0, replaceWithZero=False)

+++++++ This function is not used for meanSpectrumMethod == ‘mom0mom8jointMask’ by pipeline,

But it is called by readMeanSpectrumFITSFile available to manual users.

Remove or replace the nan values from an array. replaceWithMin: if True, then replace NaNs with np.nanmin of array

if False, then simply remove the NaNs

replaceWithZero: if True, then replace NaNs with np.nanmin of array

if False, then simply remove the NaNs

nanBufferChannels: only active if replaceWithMin=True Returns: * new array * if replaceWithMin=True, then also return the value used to replace NaNs

pipeline.extern.findContinuum.removeStatContQuadratic(y, nsigma=2.0, returnExtra=False, minPercentage=50, makeMovie=False, img='', keepContinuum=True)

Iteratively removes outlier points down to nsigma, then fits a quadratic to the remaining points, removes this quadratic from the full initial spectrum, and returns it, along with the list of channels used for the fit and the fit evaluated at all channels. minPercentage: stop removing outliers if number of remaining goes below this threshold makeMovie: if True, then make a png for each quadratic fit as channels are removed one-by-one img: only used to name the movie frame pngs

pipeline.extern.findContinuum.removeZeros(a)

+++++ This function is not used for meanSpectrumMethod=’mom0mom8jointMask’. Takes an array, and replaces all appearances of 0.0 with the minimum value of all channels not equal to 0.0. This is used to reduce the bias caused by the edge channels being 0.0 in the profiles returned by ia.getprofile. If the absolute value of the minimum is greater than the absolute value of the maximum, then set the channels to the maximum value.

pipeline.extern.findContinuum.replaceLineFullRangesWithNoise(intensity, selection, median, scaledMAD)

Used in Pipeline2020 going forward. intensity: an array of intensity values of all channels in the spectrum selection: a channel selection string in CASA format, whose intensities should be replaced median: of the channels not in the selection string (for the noise generation) scaledMAD: of the channels not in the selection string (for the noise generation) Returns: a new intensity array, of the same dimension as the input array

pipeline.extern.findContinuum.restToTopo(restFrequency, velocityLSRK, datestring, ra, dec, equinox='J2000', observatory='ALMA', veltype='radio', verbose=False)

This function is called by lsrkToTopo. Converts a rest frequency, LSRK velocity, and observing date/direction to the corresponding frequency in the TOPO frame. Inputs: restFrequency: floating point value in Hz or GHz, or a string with units velocityLSRK: floating point value in km/s datestring: “YYYY/MM/DD/HH:MM:SS” ra: string “HH:MM:SS.SSSS” dec: string “DD.MM.SS.SSSS” or “DD:MM:SS.SSSS” (colons will be replaced with .) prec: only used to display the value when verbose=True Returns: the TOPO frequency in Hz

pipeline.extern.findContinuum.robustMADofContinuumRanges(myChannelList, intensity)

Used in Pipeline2020 going forward. myChannelList: subset of channels that are currently the continuum selection intensity: list of intensities in all channels of the spectrum

pipeline.extern.findContinuum.roundFigures(value, digits)

This function is called by runFindContinuum and drawYlabel. This function rounds a floating point value to a number of significant figures. value: value to be rounded (between 1e-20 and 1e+20) digits: number of significant digits, both before or after decimal point Returns: a floating point value

pipeline.extern.findContinuum.round_half_up(x)

pipeline.extern.findContinuum.runFindContinuum(img='', pbcube=None, psfcube=None, minbeamfrac=0.3, spw='', transition='', baselineModeA='min', baselineModeB='min', sigmaCube=3, nBaselineChannels=0.19, sigmaFindContinuum='auto', sigmaFindContinuumMode='auto', verbose=False, png='', pngBasename=False, nanBufferChannels=2, source='', useAbsoluteValue=True, trimChannels='auto', percentile=20, continuumThreshold=None, narrow='auto', separator=';', overwrite=False, titleText='', maxTrim=20, maxTrimFraction=1.0, meanSpectrumFile='', centralArcsec=- 1, channelWidth=0, alternateDirectory='.', imageInfo=[], chanInfo=[], plotAtmosphere='transmission', airmass=1.5, pwv=1.0, channelFractionForSlopeRemoval=0.75, mask='', invert=False, meanSpectrumMethod='peakOverMad', peakFilterFWHM=15, fullLegend=False, iteration=0, meanSpectrumMethodMessage='', minSlopeToRemove=1e-08, minGroupsForSFCAdjustment=10, regressionTest=False, quadraticFit=False, megapixels=0, triangularPatternSeen=False, maxMemory=- 1, negativeThresholdFactor=1.15, byteLimit=- 1, singleContinuum=False, applyMaskToMask=False, plotBaselinePoints=False, dropBaselineChannels=2.0, madRatioUpperLimit=1.5, madRatioLowerLimit=1.15, projectCode='', useIAGetProfile=True, useThresholdWithMask=False, dpi=150, normalizeByMAD=False, overwriteMoments=False, initialQuadraticImprovementThreshold=1.6, minPeakOverMadForSFCAdjustment=19, maxMadRatioForSFCAdjustment=1.2, minPixelsInJointMask=3, userJointMask='', signalRatioTier1=0.965, signalRatioTier2=0.94, snrThreshold=23, mom0minsnr=5, mom8minsnr=4, overwriteMasks=True, rmStatContQuadratic=True, quadraticNsigma=2, bidirectionalMaskPhase2=False, plotQuadraticPoints=True, makeMovie=True, outdir='', allowBluePruning=True, avoidance='', enableRejectNarrowInnerWindows=True, avoidExtremaInNoiseCalcForJointMask=False, amendMask=False, momentdir='', skipchan=1, amendMaskIterationName='', fontsize=10)

This function is called by findContinuum. It calls functions that: 1) compute the mean spectrum of a dirty cube 2) find the continuum channels 3) plot the results Inputs: channelWidth: in Hz Returns: 23 items *1 A channel selection string suitable for the spw parameter of clean. *2 The name of the png produced. *3 The slope of the linear fit (or None if no fit attempted). *4 The channel width in Hz (only necessary for the fitsTable option). *5 nchan in the cube (only necessary to return this for the fitsTable option, will be computed otherwise). *6 Boolean: True if it used low values as the baseline (as opposed to high values) *7 SNRs in the moment0 image (raw, outside mask, outside phase2 mask) *8 SNRs in the moment8 image (raw, outside mask, outside phase2 mask) *9 Boolean: True if the middle-valued channels were used as the baseline (as opposed to low or high) *10 list: selectionPreBluePruning *11 float: finalSigmaFindContinuum *12 string: name of jointMask *13 float array: avgspectrumAboveThreshold *14 float: the true median of the spectrum *15 list: of plot artist descriptors (things to potentially remove and replot) *16 axis instance: ax1 (lower x-axis) *17 axis instance: ax2 (upper x-axis) *18 float: threshold (the positive threshold for line detection) *19 string: final line of upper text legend *20 int: number of narrow central groups that were dropped (or would have been dropped

if enableRejectNarrowWindows had been True, but wasn’t), summed over all runs of findContinuumChannels()

*21 float: effectiveSigma (product of finalSigmaFindContinuum*correctionFactor) *22 float: baselineMAD *23 string: upper x-axis label Inputs: img: the image cube to operate upon spw: the spw name or number to put in the x-axis label transition: the name of the spectral transition (for the plot title) baselineModeA: ‘min’ or ‘edge’, method to define the baseline in meanSpectrum() sigmaCube: multiply this value by the MAD to get the threshold above which a pixel

is included in the mean spectrum

nBaselineChannels: if integer, then the number of channels to use in:

1. computing the mean spectrum with the ‘meanAboveThreshold’ methods.

2. computing the MAD of the lowest/highest channels in findContinuumChannels

   if float, then the fraction of channels to use (i.e. the percentile) default = 0.19, which is 24 channels (i.e. 12 on each side) of a TDM window

sigmaFindContinuum: passed to findContinuumChannels, ‘auto’ starts with 3.5 sigmaFindContinuumMode: ‘auto’, ‘autolower’, or ‘fixed’ verbose: if True, then print additional information during processing png: the name of the png to produce (‘’ yields default name) pngBasename: if True, then remove the directory from img name before generating png name nanBufferChannels: when removing or replacing NaNs, do this many extra channels

beyond their extent

source: the name of the source, to be shown in the title of the spectrum.

if None, then use the filename, up to the first underscore.

findContinuum: if True, then find the continuum channels before plotting overwrite: if True, or ASCII file does not exist, then recalculate the mean spectrum

writing it to <img>.meanSpectrum

if False, then read the mean spectrum from the existing ASCII file

trimChannels: after doing best job of finding continuum, remove this many

channels from each edge of each block of channels found (for margin of safety) If it is a float between 0..1, then trim this fraction of channels in each group (rounding up). If it is ‘auto’, use 0.1 but not more than maxTrim channels and not more than maxTrimFraction

percentile: control parameter used with baselineMode=’min’ continuumThreshold: if specified, only use pixels above this intensity level narrow: the minimum number of channels that a group of channels must have to survive

if 0<narrow<1, then it is interpreted as the fraction of all

channels within identified blocks

if ‘auto’, then use int(ceil(log10(nchan)))

titleText: default is img name and transition and the control parameter values meanSpectrumFile: an alternative ASCII text file to use for the mean spectrum.

Must also set img=’’.

meanSpectrumMethod: ‘peakOverMad’, ‘peakOverRms’, ‘meanAboveThreshold’,

‘meanAboveThresholdOverRms’, ‘meanAboveThresholdOverMad’,

where ‘peak’ refers to the peak in a spatially-smoothed version of cube

peakFilterFWHM: value used by ‘peakOverRms’ and ‘peakOverMad’ to presmooth

each plane with a Gaussian kernel of this width (in pixels) Set to 1 to not do any smoothing.

centralArcsec: radius of central box within which to compute the mean spectrum

default=’auto’ means start with whole field, then reduce to 1/10 if only one window is found (unless mask is specified); or ‘fwhm’ for the central square with the same radius as the PB FWHM; or a floating point value in arcseconds

mask: a mask image equal in shape to the parent image that is used to determine the

region to compute the noise (outside the mask) and the mean spectrum (inside the mask) option ‘auto’ will look for the <filename>.mask that matches the <filename>.image and if it finds it, it will use it; otherwise it will use no mask

plotAtmosphere: ‘’, ‘tsky’, or ‘transmission’ airmass: for plot of atmospheric transmission pwv: in mm (for plot of atmospheric transmission) channelFractionForSlopeRemoval: if at least this many channels are initially selected, or

if there are only 1-2 ranges found and the widest range has > nchan*0.3 channels, then fit and remove a linear slope and re-identify continuum channels. Set to 1 to turn off.

quadraticFit: if True, fit quadratic polynomial to the noise regions when appropriate;

otherwise fit only a linear slope

megapixels: simply used to label the plot triangularPatternSeen: if True, then slightly boost sigmaFindContinuum if it is in ‘auto’ mode maxMemory: only used to name the png if it is set applyMaskToMask: if True, apply the mask inside the user mask image to set its masked pixels to 0 plotBaselinePoints: if True, then plot the baseline-defining points as black dots dropBaselineChannels: percentage of extreme values to drop in baseline mode ‘min’ useIAGetProfile: if True, then for meanAboveThreshold and baselineMode=’min’, then

use ia.getprofile instead of ia.getregion and subsequent arithmetic (faster)

initialQuadraticImprovementThreshold: if removal of a quadratic from the raw meanSpectrum reduces

the MAD by this factor or more, then proceed with removing this quadratic (new Cycle 6 heuristic)

maxMadRatioForSFCAdjustment: madRatio must be below this value (among other things) in order

for automatic lowering of sigmaFindContinuum value to occur

avoidance: a CASA channel selection string to avoid selecting (for manual use)

Parameters only relevant to mom0mom8jointMask: mom0minsnr: sets the threshold for the mom0 image (i.e. median + mom0minsnr*scaledMAD) mom8minsnr: sets the threshold for the mom8 image (i.e. median + mom8minsnr*scaledMAD) snrThreshold: if SNR is higher than this, and phase2==True, then run a phase 2 mask calculation minPixelsInJointMask: if fewer than these pixels are found, then use all pixels above pb=0.3

(when meanSpectrumMethod = ‘mom0mom8jointMask’)

overwriteMoments: if True, then overwrite any existing moment0 or moment8 image overwriteMasks: if True, then overwrite any existing moment0 or moment8 mask image rmStatContQuadratic: if True, then fit&remove a quadratic to subset of channels that lack signal quadraticNsigma: the stopping threshold to use when ignoring outliers prior to fitting quadratic bidirectionalMaskPhase2: True=extend mask to negative values beyond threshold; False=Cycle6 behavior makeMovie: if True, then make a png for each quadratic fit as channels are removed one-by-one outdir: where to write the .mom0 and .mom8 images, .dat, meanSpectrum, and the .png file

Parameters passed to findContinuumChannels (and only used in there): baselineModeB: ‘min’ or ‘edge’, method to define the baseline in findContinuumChannels() separator: the character to use to separate groups of channels in the string returned maxTrim: in trimChannels=’auto’, this is the max channels to trim per group for TDM spws; it is automatically scaled upward for FDM spws. maxTrimFraction: in trimChannels=’auto’, the max fraction of channels to trim per group negativeThresholdFactor: scale the nominal negative threshold by this factor (to adjust

sensitivity to absorption features: smaller values=more sensitive)

signalRatioTier1: threshold for signalRatio, above which we desensitize the level to

consider line emission in order to avoid small differences in noise levels from affecting the result (e.g. that occur between serial and parallel tclean cubes) signalRatio=1 means: no lines seen, while closer to zero: more lines seen

signalRatioTier2: second threshold for signalRatio, used for FDM spws (nchan>192) and

for cases of peakOverMad < 5. Should be < signalRatioTier1.

dropBaselineChannels: percentage of extreme values to drop in baseline mode ‘min’ madRatioUpperLimit, madRatioLowerLimit: determines when dropBaselineChannels is used

pipeline.extern.findContinuum.setYLimitsAvoidingEdgeChannels(avgspectrumAboveThreshold, mad, chan=1)

Called by runFindContinuum to set the y-axis limits.. 1) Avoid spikes in edge channels from skewing the plot limits, by

setting plot limits on the basis of channels chan..-chan

2. Enforce a maximum dynamic range (peak-median)/mad so that weak features can be seen, allowing strongest ones to overflow the top.

pipeline.extern.findContinuum.sigmaCorrectionFactor(baselineMode, npts, percentile)

This function is called by findContinuumChannels. Computes the correction factor for the fact that the measured rms (or MAD) on the N%ile lowest points of a datastream will be less than the true rms (or MAD) of all points. Returns: a value between 0 and 1, which will be used to reduce the

estimate of the population sigma based on the sample sigma

pipeline.extern.findContinuum.splitListIntoContiguousLists(mylist)

This function is called by findContinuumChannels. Called by copyweights. See also splitListIntoHomogenousLists. Converts [1,2,3,5,6,7] into [[1,2,3],[5,6,7]], etc. -Todd Hunter

pipeline.extern.findContinuum.splitListIntoContiguousListsAndRejectNarrow(channels, narrow=3)

This function is called by findContinuumChannels. Split a list of channels into contiguous lists, and reject those that have a small number of channels. narrow: if >=1, then interpret it as the minimum number of channels that

a group must have in order to survive

if <1, then interpret it as the minimum fraction of channels that

a group must have relative to the total number of channels

Returns: a new single list (as an array) Example: [1,2,3,4,6,7,9,10,11] –> [ 1, 2, 3, 4, 9, 10, 11]

pipeline.extern.findContinuum.splitListIntoContiguousListsAndRejectZeroStd(channels, values, nanmin=None, verbose=False)

This function is called by findContinuumChannels. Takes a list of numerical values, splits into contiguous lists and removes those that have zero standard deviation in their associated values. Note that values must hold the values of the whole array, while channels can be a subset. If nanmin is specified, then reject lists that contain more than 3 appearances of this value.

pipeline.extern.findContinuum.splitListIntoContiguousListsAndTrim(totalChannels, channels, trimChannels=0.1, maxTrim=20, maxTrimFraction=1.0, verbose=False)

This function is called by findContinuumChannels. Split a list of channels into contiguous lists, and trim some number of channels from each edge. totalChannels: number of channels in the spectrum channels: a list of channels selected for potential continuum trimChannels: if integer, use that number of channels. If float between

0 and 1, then use that fraction of channels in each contiguous list (rounding up). If ‘auto’, then use 0.1 but not more than maxTrim channels and not more than maxTrimFraction of channels.

maxTrim: used in ‘auto’ mode Returns: a new single list

pipeline.extern.findContinuum.splitListIntoHomogeneousLists(mylist)

This function is called only by removeNaNs, and hence is not used in Cycle 6-onward pipeline. Converts [1,1,1,2,2,3] into [[1,1,1],[2,2],[3]], etc. -Todd Hunter

pipeline.extern.findContinuum.submask(mask, region)

This function is called by meanSpectrum, but only in Cycle 4+5 heuristic. Takes a spectral mask array, and picks out a subcube defined by a blc,trc-defined region region: dictionary containing keys ‘blc’ and ‘trc’

pipeline.extern.findContinuum.tdmSpectrum(channelWidth, nchan)

This function is called by runFindContinuum and findContinuum. Use 15e6 instead of 15.625e6 because LSRK channel width can be slightly narrower than TOPO. Works for single, dual, or full-polarization. Returns: True or False

pipeline.extern.findContinuum.tooLittleBandwidth(selection, chanInfo, fraction=0.05)

Used in Pipeline2020 going forward. Returns: a string if there is a warning, or None if not

pipeline.extern.findContinuum.tooLittleSpread(selection, chanInfo, fraction=0.33)

Used in Pipeline2020 going forward. Returns: a string if there is a warning, or None if not

pipeline.extern.findContinuum.topoFreqRangeListToChannel(contdatline='', vispath='./', spw=- 1, freqlist='', vis='', mymsmd='', returnFlatlist=False, writeChannelsInIncreasingOrder=True)

++++ This function is used by pipeline in writeContDat() Converts a semicolon-delimited string list of topocentric frequency ranges to channel ranges in the specified ms. contdatline: line cut-and-pasted from .dat file (e.g. ‘my.ms 150.45~151.67GHz;151.45~152.67GHz’) vispath: set the path to the measurement set whose name was read from contdatline freqlist: ‘150.45~151.67GHz;151.45~152.67GHz’ spw: integer ID or string ID writeChannelsInIncreasingOrder: if True, then ensure that the list is in increasing order Returns: a string like: ‘29:134~136;200~204’ if flatlist==True, then return [134,136,200,204]

pipeline.extern.findContinuum.topoFreqToChannel(freqlist, vis, spw, mymsmd='')

++++ This function is used by pipeline in writeContDat() Converts a python floating point list of topocentric frequency ranges to channel ranges in the specified ms. freqlist: [150.45e9,151.67e9] or [150.45, 151.67] spw: integer ID vis: reads the channel frequencies from this measurement set for the specified spw Returns: a python list of channels

pipeline.extern.findContinuum.transition(vis, spw, source='', intent='OBSERVE_TARGET', verbose=True, mymsmd=None)

+++++++ This function is not used by pipeline, because it is only used by isSingleContinuum. Returns the list of transitions for specified spw (and source). vis: measurement set spw: can be integer ID or string integer ID Optional parameters: source: can be integer ID or string name; if not specified, use the first one intent: if source is blank then use first one with matching intent and spw

pipeline.extern.findContinuum.updateChannelRangesOnPlot(labelDescs, selection, ax1, separator, avgspectrumAboveThreshold, skipchan, medianTrue, positiveThreshold, upperXlabel, ax2, channelWidth, fontsize=10, remove=True)

Used in Pipeline2020 going forward. channelWidth: in Hz Returns: new aggregate bandwidth in GHz

pipeline.extern.findContinuum.version(showfile=True)

Returns the CVS revision number.

pipeline.extern.findContinuum.versionStringToArray(versionString)

This function is called by casaVersionCompare. Converts ‘5.3.0-22’ to np.array([5,3,0,22], dtype=np.int32)

pipeline.extern.findContinuum.widthOfMaskArcsec(mask, maskInfo)

++++ This function is not used by pipeline when meanSpectrumMethod=’mom0mom8jointMask’ or if mask=’’ Finds width of the smallest central square that circumscribes all masked pixels. Returns the value in arcsec.

pipeline.extern.findContinuum.writeContDat(meanSpectrumFile, selection, png, aggregateBandwidth, firstFreq, lastFreq, channelWidth, img, imageInfo, vis='', numchan=None, chanInfo=None, lsrkwidth=None, spw='')

This function is called by findContinuum. Writes out an ASCII file called <meanSpectrumFile>_findContinuum.dat that contains the selected channels, the png name and the aggregate bandwidth in GHz. Returns: None meanSpectrumFile: only used to generate the name of the .dat file, unless

firstFreq <= 0, in which case the readPreviousMeanSpectrum is called on it (It splits off name before ‘.meanSpectrum’ and appends _findContinuum.dat)

vis: if specified, then also write a line with the topocentric velocity ranges spw: integer or string ID number; if specified (along with vis), then also write

a final line with the topocentric channel ranges for this spw

pipeline.extern.findContinuum.writeMeanSpectrum(meanSpectrumFile, frequency, avgspectrum, avgSpectrumNansReplaced, threshold, nchan, edgesUsed=0, nanmin=0, centralArcsec='auto', mask='', iteration=0)

This function is called by meanSpectrum. Writes out the mean spectrum (and the key parameters used to create it), so that it can quickly be restored. This allows the manual user to quickly experiment with different parameters of findContinuumChannels applied to the same mean spectrum. Units are Hz and Jy/beam. meanSpectrumFile: name of file to create threshold: this is interpreted as mom0threshold for meanSpectrumMethod=’mom0mom8jointMask’ edgesUsed: this is interpreted as numberPixelsInMask for meanSpectrumMethod=’mom0mom8jointMask’ nanmin: this is interpreted as mom8threshold for meanSpectrumMethod=’mom0mom8jointMask’ frequency: list or array of frequencies Returns: None

pipeline.extern.ps_mem module

class pipeline.extern.ps_mem.Proc

Bases: object

open(*args)

path(*args)

pipeline.extern.ps_mem.cmd_with_count(cmd, count)

pipeline.extern.ps_mem.getCmdName(pid, split_args)

pipeline.extern.ps_mem.getMemStats(pid)

pipeline.extern.ps_mem.get_memory_usage(pids_to_show, split_args, include_self=False, only_self=False)

pipeline.extern.ps_mem.help()

pipeline.extern.ps_mem.human(num, power='Ki')

pipeline.extern.ps_mem.kernel_ver()

pipeline.extern.ps_mem.parse_options()

pipeline.extern.ps_mem.print_header()

pipeline.extern.ps_mem.print_memory_usage(sorted_cmds, shareds, count, total)

pipeline.extern.ps_mem.shared_val_accuracy()

http://wiki.apache.org/spamassassin/TopSharedMemoryBug

pipeline.extern.ps_mem.show_shared_val_accuracy(possible_inacc)

pipeline.extern.ps_mem.std_exceptions(etype, value, tb)

pipeline.extern.ps_mem.verify_environment()

pipeline.extern.sensitivity_improvement module

pipeline.extern.sensitivity_improvement.onlineChannelAveraging(vis, spw, mymsmd='')

For Cycle 3-onward data, determines the channel averaging factor from the ratio of the effective channel bandwidth to the channel width. -Todd Hunter

pipeline.extern.sensitivity_improvement.sensitivityImprovement(vis, spw, newchanwidth, useCAS8534=True, window='hanning', cubechanwidth=None)

Computes the expected factor of improvement in sensitivity expected when making images of ALMA data with channel widths wider than the observed width. vis: name of measurement set spw: single spw index (integer) newchanwidth: width of image channel in units of the uvdata channel width,

or a string with any common frequency units (‘MHz’ etc.) or ‘km/s’

useCAS8534: if True, use the approximate formula developed for the imaging

pipeline;

if False, use a spline fit to the integral table of values (1,2,4,8,16)

cubechanwidth: if specified, then compute the improvement with respect to

this chanwidth (which might be larger than the observed chanwidth). Units: Width of image channel in units of the uvdata channel width, or a string with any common frequency units (‘MHz’ etc.) or ‘km/s’

-Todd Hunter

pipeline.extern.sensitivity_improvement.windowFunction(window='', channelAveraging=1, returnValue='FWHM', splineDegree=3, splineSmoothing=None, ratio=False, useCAS8534=False, nchan=1, spwchan=128)

Print the FWHM and Effective sensitivity bandwidth of each of the ALMA correlator window functions, or return the value for a specific choice. The values are taken from the tables in Richard Hills’ note of April 8, 2012. window: one of [‘’, ‘uniform’,’hanning’,’welch’,’cosine’,’hamming’,’bartlett’,

‘blackmann’,’blackmann-harris’] ‘’=prints the whole table

channelAveraging: 1, 2, 4, 8, or 16; >16 will return channelAveraging, other values

will be spline interpolated

returnValue: ‘FWHM’ or ‘EffectiveBW’ or ‘dictionary’ splineDegree: passed as the k parameter to scipy.interpolate.UnivariateSpline splineSmoothing: passed as the s parameter to scipy.interpolate.UnivariateSpline ratio: if True, then divide by the channelAveraging factor useCAS8534: uses the approximate formula developed for the pipeline rather than a spline spwchan: number of channels in spw (only used if useCAS8534=True) nchan: number of channels being combined (only used if useCAS8534=True) Returns: The effective number of native channels supplied by a single channel. -Todd Hunter

Module contents