pipeline.hifv.heuristics package¶

Submodules¶

pipeline.hifv.heuristics.bandpass module¶

pipeline.hifv.heuristics.bandpass.computeChanFlag(vis, caltable, context)[source]¶

pipeline.hifv.heuristics.bandpass.do_bandpass(vis, caltable, context=None, RefAntOutput=None, spw=None, ktypecaltable=None, bpdgain_touse=None, solint=None, append=None, executor=None)[source]¶: Run CASA task bandpass

pipeline.hifv.heuristics.bandpass.do_bandpassweakbp(vis, caltable, context=None, RefAntOutput=None, spw=None, ktypecaltable=None, bpdgain_touse=None, solint=None, append=None)[source]¶: Run CASA task bandpass

pipeline.hifv.heuristics.bandpass.removeRows(caltable, spwids)[source]¶

pipeline.hifv.heuristics.bandpass.weakbp(vis, caltable, context=None, RefAntOutput=None, ktypecaltable=None, bpdgain_touse=None, solint=None, append=None, executor=None, spw='')[source]¶

pipeline.hifv.heuristics.lib_EVLApipeutils module¶

pipeline.hifv.heuristics.lib_EVLApipeutils.cont_file_to_CASA(vis, context, contfile='cont.dat')[source]¶

Take the dictionary created by _read_cont_file and put it into the format: spw = ‘0:1.380~1.385GHz;1.390~1.400GHz’

If the frequencies specified in the contfile are in LSRK, they will be converted to TOPO.

pipeline.hifv.heuristics.lib_EVLApipeutils.find_EVLA_band(frequency, bandlimits=[0.0, 150000000.0, 700000000.0, 2000000000.0, 4000000000.0, 8000000000.0, 12000000000.0, 18000000000.0, 26500000000.0, 40000000000.0, 56000000000.0], BBAND='?4PLSCXUKAQ?')[source]¶

pipeline.hifv.heuristics.lib_EVLApipeutils.getBCalStatistics(calTable, innerbuff=0.1)[source]¶

Version 2012-11-20 v1.0 STM casa 4.0 version Version 2012-12-17 v1.0 STM casa 4.1 version, phase, real, imag stats

This method will look at the specified B calibration table and return the statistics of unflagged solutions for each Antenna, SPW, Poln. This assumes that the specified cal table will not have any channel dependent flagging.

return structure is a dictionary with AntennaID and Spectral Window ID as the keys and returns a list of statistics per polarization in the order specified in the Cal Table.

Input:

calTable - name of MS innerbuff - fraction of spw bandwidth to ignore at each edge

(<0.5, default=0.1 use inner 80% of channels each spw)

Output: OutDict (return value)

STM 2012-11-19 dictionary structure: key: ‘antspw’ indexed by antenna and spectral window id per poln

[‘antspw’][<antid>][<spwid>][<polid>]

‘antband’ indexed by antenna and rx/baseband summed over subbands and poln
[‘ant’][<antid>][<rxband>][<baseband>] Note: this requires that the spectral window NAME follow JVLA convention of

rxband#baseband#spw (e.g. ‘EVLA_K#A0C0#48’)

For each of these there is a data structure for [‘all’] and [‘inner’] that contains the fields: [‘total’], [‘number’], [‘mean’], [‘min’], [‘max’]

Some useful indexing dictionaries are also output:: [‘antDict’][ant] the name of antenna with index ant [‘spwDict’][spw] {‘RX’:rx, ‘Baseband’:bb, ‘Subband’:sb} correspoding to a given spw [‘rxBasebandDict’][rx][bb] list of spws corresponding to a given rx and bb

Example:

!cp /home/sandrock2/smyers/casa/pipeline/pipeline4.1/lib_EVLApipeutils.py . from lib_EVLApipeutils import getBCalStatistics result = getBCalStatistics(‘testBPcal.b’)

This is a B Jones table, proceeding Found 1 Rx bands Rx band EVLA_K has basebands: [‘A0C0’, ‘B0D0’] Within all channels: Found 864 total solutions with 47824 good (unflagged) Within inner 0.8 of channels: Found 0 total solutions with 42619 good (unflagged)

AntID AntName Rx-band Baseband min/max(all) min/max(inner) ALERT?

0 ea01 EVLA_K A0C0 0.3541 0.3575 0 ea01 EVLA_K B0D0 0.4585 0.4591 1 ea02 EVLA_K A0C0 0.3239 0.3298 1 ea02 EVLA_K B0D0 0.2130 0.2179 2 ea03 EVLA_K A0C0 0.3097 0.3124 2 ea03 EVLA_K B0D0 0.3247 0.3282 3 ea04 EVLA_K A0C0 0.3029 0.3055 3 ea04 EVLA_K B0D0 0.2543 0.2543 4 ea05 EVLA_K A0C0 0.2657 0.2715 4 ea05 EVLA_K B0D0 0.2336 0.2357 5 ea06 EVLA_K A0C0 0.3474 0.3501 5 ea06 EVLA_K B0D0 0.2371 0.2398 6 ea07 EVLA_K A0C0 0.2294 0.2322 6 ea07 EVLA_K B0D0 0.3684 0.3690 7 ea08 EVLA_K A0C0 0.2756 0.2809 7 ea08 EVLA_K B0D0 0.5269 0.5269 8 ea10 EVLA_K A0C0 0.2617 0.2686 8 ea10 EVLA_K B0D0 0.2362 0.2407 9 ea11 EVLA_K A0C0 0.2594 0.2607 9 ea11 EVLA_K B0D0 0.3015 0.3111

10 ea12 EVLA_K A0C0 0.2418 0.2491 10 ea12 EVLA_K B0D0 0.3839 0.3849 11 ea13 EVLA_K A0C0 0.3017 0.3156 11 ea13 EVLA_K B0D0 0.3299 0.3299 12 ea14 EVLA_K A0C0 0.2385 0.2415 12 ea14 EVLA_K B0D0 0.2081 0.2101 13 ea15 EVLA_K A0C0 0.3118 0.3151 13 ea15 EVLA_K B0D0 0.1749 0.1844 * 14 ea16 EVLA_K A0C0 0.2050 0.2051 14 ea16 EVLA_K B0D0 0.4386 0.4537 15 ea17 EVLA_K A0C0 0.4084 0.4107 15 ea17 EVLA_K B0D0 0.2151 0.2189 16 ea18 EVLA_K A0C0 0.3124 0.3147 16 ea18 EVLA_K B0D0 0.3712 0.3730 17 ea19 EVLA_K A0C0 0.3084 0.3132 17 ea19 EVLA_K B0D0 0.2453 0.2522 18 ea20 EVLA_K A0C0 0.4618 0.4672 18 ea20 EVLA_K B0D0 0.2772 0.2790 19 ea21 EVLA_K A0C0 0.3971 0.4032 19 ea21 EVLA_K B0D0 0.2812 0.2818 20 ea22 EVLA_K A0C0 0.2495 0.2549 20 ea22 EVLA_K B0D0 0.3777 0.3833 21 ea23 EVLA_K A0C0 0.3213 0.3371 21 ea23 EVLA_K B0D0 0.2702 0.2787 22 ea24 EVLA_K A0C0 0.2470 0.2525 22 ea24 EVLA_K B0D0 0.0127 0.0127 * 23 ea25 EVLA_K A0C0 0.3173 0.3205 23 ea25 EVLA_K B0D0 0.0056 0.0066 * 24 ea26 EVLA_K A0C0 0.3505 0.3550 24 ea26 EVLA_K B0D0 0.4028 0.4071 25 ea27 EVLA_K A0C0 0.3165 0.3244 25 ea27 EVLA_K B0D0 0.2644 0.2683 26 ea28 EVLA_K A0C0 0.1695 0.1712 * 26 ea28 EVLA_K B0D0 0.3213 0.3318

result[‘antband’][22][‘EVLA_K’][‘B0D0’]

Out:

{‘all’: {‘amp’: {‘max’: 3.3507643208628997,

‘mean’: 1.2839660621605888, ‘min’: 0.042473547230042749, ‘var’: 0.42764307104292559},

‘imag’: {‘max’: array(1.712250828742981),
‘mean’: -0.0042696134968516824, ‘min’: array(-2.2086853981018066), ‘var’: 0.43859873853192027},

‘number’: 900, ‘phase’: {‘max’: 112.32456525225054,

‘mean’: -0.70066399447640126, ‘min’: -108.23612296337477, ‘var’: 1445.9499062237583},

‘real’: {‘max’: array(3.3105719089508057),
‘mean’: 1.0975328904785548, ‘min’: array(-0.13342639803886414), ‘var’: 0.433613996950728},

‘total’: 1024},

‘inner’: {‘amp’: {‘max’: 3.3507643208628997,

‘mean’: 1.2913493102726084, ‘min’: 0.042473547230042749, ‘var’: 0.45731194085061877},

‘imag’: {‘max’: array(1.712250828742981),: ‘mean’: -0.0019235184823728287, ‘min’: array(-2.2086853981018066), ‘var’: 0.43650578363669668},

‘number’: 802, ‘phase’: {‘max’: 112.32456525225054,

‘mean’: -0.74464737957566107, ‘min’: -108.23612296337477, ‘var’: 1496.4536912732597},

‘real’: {‘max’: array(3.3105719089508057),: ‘mean’: 1.1054519462910002, ‘min’: array(-0.13342639803886414), ‘var’: 0.4665390728630604},

‘total’: 816}}

result[‘rxBasebandDict’][‘EVLA_K’][‘B0D0’]

Out: [8, 9, 10, 11, 12, 13, 14, 15]

result[‘antspw’][22][10]

Out:

{0: {‘all’: {‘amp’: {‘max’: 3.3507643208628997,

‘mean’: 1.6887832659451747, ‘min’: 0.054888048286296932, ‘var’: 0.84305637195708472},

‘imag’: {‘max’: array(1.4643619060516357),
‘mean’: -0.13419617990315968, ‘min’: array(-2.2086853981018066), ‘var’: 0.97892254522390565},

‘number’: 59, ‘phase’: {‘max’: 112.32456525225054,

‘mean’: -1.0111711887628112, ‘min’: -102.4143890204484, ‘var’: 2885.5110793438112},

‘real’: {‘max’: array(3.3105719089508057),
‘mean’: 1.2845739619253926, ‘min’: array(-0.13342639803886414), ‘var’: 1.0099576839663802},

‘total’: 64},

‘inner’: {‘amp’: {‘max’: 3.3507643208628997,

‘mean’: 1.1064667538670538, ‘min’: 0.054888048286296932, ‘var’: 0.94882587512210448},

‘imag’: {‘max’: array(1.4643619060516357),: ‘mean’: 0.056129313275038409, ‘min’: array(-2.2086853981018066), ‘var’: 0.55401604820332395},

‘number’: 201, ‘phase’: {‘max’: 112.32456525225054,

‘mean’: 17.99353823044413, ‘min’: -102.4143890204484, ‘var’: 5065.0038289850836},

‘real’: {‘max’: array(3.3105719089508057),: ‘mean’: 0.75592079894228781, ‘min’: array(-0.13342639803886414), ‘var’: 0.89091406078797231},

‘total’: 51}},

1: {‘all’: {‘amp’: {‘max’: 1.0603890232940267,

‘mean’: 0.9966154059201201, ‘min’: 0.92736792814263314, ‘var’: 0.032951023002670755},

‘imag’: {‘max’: array(0.28490364551544189),
‘mean’: 0.10552169008859259, ‘min’: array(-0.08266795426607132), ‘var’: 0.017548692059680165},

‘number’: 59, ‘phase’: {‘max’: 17.237656150196859,

‘mean’: 6.0660176622725466, ‘min’: -4.9577763029789832, ‘var’: 40.224751029787001},

‘real’: {‘max’: array(1.047441840171814),
‘mean’: 0.98505325640662234, ‘min’: array(0.90479201078414917), ‘var’: 0.032541082320460629},

‘total’: 64},

‘inner’: {‘amp’: {‘max’: 1.0603890232940267,

‘mean’: 1.0327309832106102, ‘min’: 0.92837663017787742, ‘var’: 0.19543922221719845},

‘imag’: {‘max’: array(0.26414120197296143),: ‘mean’: 0.13256895535348001, ‘min’: array(-0.08266795426607132), ‘var’: 0.036957882304178208},

‘number’: 201, ‘phase’: {‘max’: 15.395755389451365,

‘mean’: 7.3133434743802201, ‘min’: -4.9577763029789832, ‘var’: 18.607793373980815},

‘real’: {‘max’: array(1.047441840171814),: ‘mean’: 1.0216352563400155, ‘min’: array(0.92740714550018311), ‘var’: 0.19275019403126262},

‘total’: 51}}}

NOTE: You can see the problem is in poln 0 of this spw from the amp range and phase.

pipeline.hifv.heuristics.lib_EVLApipeutils.getCalFlaggedSoln(calTable)[source]¶

Version 2012-05-03 v1.0 STM to 3.4 from original 3.3 version, new dictionary Version 2012-05-03 v1.1 STM indexed by ant, spw also Version 2012-09-11 v1.1 STM correct doc of <polid> indexing, bug fix, median over ant Version 2012-09-12 v1.2 STM median over ant revised Version 2012-11-13 v2.0 STM casa 4.0 version with new call mechanism Version 2013-01-11 v2.1 STM use getvarcol

This method will look at the specified calibration table and return the fraction of flagged solutions for each Antenna, SPW, Poln. This assumes that the specified cal table will not have any channel dependent flagging.

return structure is a dictionary with AntennaID and Spectral Window ID as the keys and returns a list of fractional flagging per polarization in the order specified in the Cal Table.

STM 2012-05-03 revised dictionary structure: key: ‘all’ all solutions

[‘all’][‘total’] = <number> [‘all’][‘flagged’] = <number> [‘all’][‘fraction’] = <fraction>

‘antspw’ indexed by antenna and spectral window id per poln
[‘antspw’][<antid>][<spwid>][<polid>][‘total’] = <total number sols per poln> [‘antspw’][<antid>][<spwid>][<polid>][‘flagged’] = <flagged number sols per poln> [‘antspw’][<antid>][<spwid>][<polid>][‘fraction’] = <flagged fraction per poln>

‘ant’ indexed by antenna summed over spectral window per poln
[‘ant’][<antid>][<polid>][‘total’] = <total number sols per poln> [‘ant’][<antid>][<polid>][‘flagged’] = <flagged number sols per poln> [‘ant’][<antid>][<polid>][‘fraction’] = <flagged fraction per poln>

‘spw’ indexed by spectral window summed over antenna per poln
[‘spw’][<spwid>][<polid>][‘total’] = <total number sols per poln> [‘spw’][<spwid>][<polid>][‘flagged’] = <flagged number sols per poln> [‘spw’][<spwid>][<polid>][‘fraction’] = <flagged fraction per poln>

‘antmedian’ median fractions over antenna summed over spw and polarization
[‘total’] = median <total number sols per ant> [‘flagged’] = median <flagged number sols per ant> [‘fraction’] = median <flagged fraction of sols per ant> [‘number’] = number of antennas that went into the median

Note that fractional numbers flagged per poln are computed as a fraction of channels (thus a full set of channels for a given ant/spw/poln count as 1.0)

Example:

!cp /home/sandrock2/smyers/casa/pipeline/lib_EVLApipeutils.py . from lib_EVLApipeutils import getCalFlaggedSoln result = getCalFlaggedSoln(‘calSN2010FZ.G0’) result[‘all’]

Out: {‘flagged’: 1212, ‘fraction’: 0.16031746031746033, ‘total’: 7560}

result[‘antspw’][21][7]

Out: {0: {‘flagged’: 3.0, ‘fraction’: 0.29999999999999999, ‘total’: 10},: 1: {‘flagged’: 3.0, ‘fraction’: 0.29999999999999999, ‘total’: 10}}

result[‘ant’][15]

Out: {0: {‘flagged’: 60.0, ‘fraction’: 0.42857142857142855, ‘total’: 140},: 1: {‘flagged’: 60.0, ‘fraction’: 0.42857142857142855, ‘total’: 140}}

result[‘spw’][3]

Out: {0: {‘flagged’: 39.0, ‘fraction’: 0.14444444444444443, ‘total’: 270},: 1: {‘flagged’: 39.0, ‘fraction’: 0.14444444444444443, ‘total’: 270}}

Bresult = getCalFlaggedSoln(‘calSN2010FZ.B0’) Bresult[‘all’]

Out: {‘flagged’: 69.171875, ‘fraction’: 0.091497189153439157, ‘total’: 756}

Bresult[‘ant’][15]

Out: {0: {‘flagged’: 6.03125, ‘fraction’: 0.43080357142857145, ‘total’: 14},: 1: {‘flagged’: 6.03125, ‘fraction’: 0.43080357142857145, ‘total’: 14}}

Bresult[‘antmedian’]

Out: {‘flagged’: 0.0625, ‘fraction’: 0.002232142857142857, ‘number’: 27, ‘total’: 28.0}

Another example, to make a list of spws in the caltable that have any unflagged solutions in them:

G2result = getCalFlaggedSoln(‘calSN2010FZ.G2.2’) goodspw = [] for ispw in G2result[‘spw’].keys():

tot = 0 flagd = 0 for ipol in G2result[‘spw’][ispw].keys():

tot += G2result[‘spw’][ispw][ipol][‘total’] flagd += G2result[‘spw’][ispw][ipol][‘flagged’]

if tot>0:
fract = flagd/tot if fract<1.0:

goodspw.append(ispw)

goodspw: Out: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]

pipeline.hifv.heuristics.lib_EVLApipeutils.set_add_model_column_parameters(context)[source]¶

pipeline.hifv.heuristics.standard module¶

class pipeline.hifv.heuristics.standard.Standard[source]¶

Bases: pipeline.infrastructure.api.Heuristic

calculate(field: str) → str [source]¶

Make a calculation based on the given parameters.

This is an abstract method and must be implemented by all Heuristic subclasses.

Note

The signature and return types of calculate() are intended to be implementation specific. Refer to the documentation of the implementing class for the appropriate signature.

ephemeris_fields = {'Callisto', 'Ceres', 'Davida', 'Europa', 'Ganymede', 'Io', 'Juno', 'Jupiter', 'Mars', 'Neptune', 'Pallas', 'Pluto', 'Titan', 'Triton', 'Uranus', 'Vesta', 'Victoria'}¶

pipeline.hifv.heuristics.standard_test module¶

Tests for the hifv/heuristics/standard.py module.

pipeline.hifv.heuristics.standard_test.test_standard(source, expectedmodel)[source]¶

Test standard() heuristics class

This heuristics function takes a source field name and returns the appropriate model.

pipeline.hifv.heuristics.uvrange module¶

pipeline.hifv.heuristics.uvrange.uvrange(setjy_results, field_id, spw_id=2)[source]¶

Parameters

setjy_results – Flux domain object read in from the import stage
field_id – integer field id
spw_id – integer spw id, default is spw_id=2 for VLASS
** However, currently it just picks the first index of zero

uvmin and uvmax are of type Decimal

Units are always assumed to be lambda

Returns: uvrange string

pipeline.hifv.heuristics.uvrange_test module¶

Tests for the hifv/heuristics/uvrange.py module.

pipeline.hifv.heuristics.uvrange_test.test_uvrange(result, expecteduvrange)[source]¶

Test uvrange() heuristics function

This utility function takes a flux result object and determines what the uvrange string should be. A blank string, greater than value (>), or value~value can be returned in each of the examples. This function is primarily used in VLASS processing.

pipeline.hifv.heuristics.vlascanheuristics module¶

class pipeline.hifv.heuristics.vlascanheuristics.VLAScanHeuristics(vis)[source]¶

Bases: object

calibratorIntents()[source]¶

# Identify scans and fields associated with different calibrator intents

# NB: the scan intent definitions changed in the OPT on Feb 21, # 2012. So test on date: # Test Implementation: JSK April 11, 2016

calibratorIntentsOld()[source]¶

# Identify scans and fields associated with different calibrator intents

# NB: the scan intent definitions changed in the OPT on Feb 21, # 2012. So test on date:

determine3C84()[source]¶: #Determine if 3C84 was used as a bandpass or delay calibrator

find_3C84(positions)[source]¶

makescandict()[source]¶: Run Steve’s buildscans

pipeline.hifv.heuristics.vlascanheuristics.buildscans(msfile, scd)[source]¶

buildscans: compile scan information for msfile

Created S.T. Myers 2012-05-07 v1.0 Updated S.T. Myers 2012-05-14 v1.1 add corrtype Updated S.T. Myers 2012-06-27 v1.2 add corrdesc lookup Updated S.T. Myers 2012-11-13 v2.0 STM casa 4.0 new calls

Usage:: from lib_EVLApipeutils import buildscans

Input:

msfile - name of MS

Output: scandict (return value)

Examples:

CASA <2>: from lib_EVLApipeutils import buildscans

CASA <3>: msfile = ‘TRSR0045_sb600507.55900.ms’

CASA <4>: myscans = buildscans(msfile) Getting scansummary from MS Found 16 DataDescription IDs Found 4 StateIds Found 3422 times in DD=0 Found 3422 times in DD=1 Found 3422 times in DD=2 Found 3422 times in DD=3 Found 3422 times in DD=4 Found 3422 times in DD=5 Found 3422 times in DD=6 Found 3422 times in DD=7 Found 3422 times in DD=8 Found 3422 times in DD=9 Found 3422 times in DD=10 Found 3422 times in DD=11 Found 3422 times in DD=12 Found 3422 times in DD=13 Found 3422 times in DD=14 Found 3422 times in DD=15 Found total 54752 times Found 175 scans min=1 max=180 Size of scandict in memory is 248 bytes

CASA <5>: myscans[‘Scans’][1][‘intents’]: Out[5]: ‘CALIBRATE_AMPLI#UNSPECIFIED,UNSPECIFIED#UNSPECIFIED’
CASA <6>: myscans[‘Scans’][1][‘dd’]: Out[6]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
CASA <7>: myscans[‘DataDescription’][0]: Out[7]:

{‘corrdesc’: [‘RR’, ‘RL’, ‘LR’’, ‘LL’], ‘corrtype’: [5, 6, 7, 8], ‘ipol’: 0, ‘nchan’: 64, ‘npol’: 4, ‘reffreq’: 994000000.0, ‘spw’: 0, ‘spwname’: ‘Subband:0’}

CASA <8>: myscans[‘Scans’][1][‘times’][0]: Out[8]:

[4829843281.500001, 4829843282.5, 4829843283.5, 4829843284.5, … 4829843336.5]

The last of these returns the integration midpoints for scan 1 DD 0.

Note that to get spw and pol info you use the DD indexes from [‘dd’] in the ‘Scans’ part to index into the ‘DataDescription’ info.

You can also list the keys available in the Scans sub-dictionary:

CASA <9>: myscans[‘Scans’][1].keys(): Out[9]:

[‘scan_mid’, ‘intents’, ‘field’, ‘dd’, ‘npol’, ‘rra’, ‘spw’, ‘scan_int’, ‘scan_start’, ‘times’, ‘scan_end’, ‘rdec’]

pipeline.hifv.heuristics.vlascanheuristics.testCalibratorIntents(vis)[source]¶