From the point when data are recorded on the ten data disks on computers paf1 through paf10, data processing to produce cross-correlation spectra happens in four steps
The data_copy.py, collate.py, and correlate.py scripts each allow you to process large numbers of data scans with each command, but, if you want to correlate just one scan without manually executing these three commands, you can use the process_scan.py script as follows.
/opt/local/bin/python process_scan.py TINT_072313 2012_07_15_14:59:20 where the first argument is the program name, the second is the scan name, and the third is the optional number of data points input to the FX correlator, which must be a power of 2 and is twice the number of frequency bins that appear in the cross-product spectra.
Input data for the software correlator is a collated data file whose creation is described HERE. The collated data file names have the suffix '.agg' and the same date/time root name, e.g., 2012_07_15_14:59:20, as all other files associated with a telescope data scan. The directory containing the aggregated raw data files is defined in the paf_directories.py code file with the string variable, DATA_DIR, plus the observing program name plus "AGG", for example,
The software correlator is an "FX" correlator, which means that it executes Fourier Transforms on simultaneously recorded blocks of data from the analog-to-digital converters that sample signals from the 19 array elements. The output complex spectra are multiplied together in all possible combinations of the 19 elements, two polarizations, and each cross-product spectrum is accumulated for the duration of the scan. The accumulated cross-product spectra are stored in a FITS format correlation file that is then available for further analysis and beamforming.
The data files are accessible from the Green Bank Linux computer 'paf0' so log onto 'paf0' and change directories with
cd /home/paf/PAF0where you will find the Python files
paf_directories.py, correlate.py, FXcorFits.py, RcvrFits.py, and paf_correlator.py.These source files, along with the Python modules pyfits and numpy, are required to run the correlator. The correlator is run with the script correlate.py, for example:
/opt/local/bin/python correlate.py TINT_072313 2012_07_15_14:59:20 where the first argument is the observing session name and the last argument is the name of the scan for which you are generating a correlation FITS file. The last argument is the number of input data samples to the FFT of the FX correlator. It must be a power of 2 and is defaulted to 256 if the last argument is omitted. The number of spectral channels in the FFT output will be half this number.
The main directory location of the various data files used in the collation and correlation process is set by the DATA_DIR parameter in the paf_directories.py code file. This may need to be modified if the location of any of the directories is changed. The location of the aggregated data sample file is a concatination of the DATA_DIE parameter, the observing session name, and the directory name, 'AGG', for example,
/export/raid1/TINT_072313/AGGLikewise, the location of the raw cross-correlation data files is a concatination of the DATA_DIR parameter, the observing session name, and the directory name, 'CrossSpec', for example,
/export/raid1/TINT_072313/CrossSpecThe directory where the FITS format cross-correlation data file are stored is set by the COR_FITS_DIR parameter in paf_directories.py.
The correlate.py script creates both the raw cross-correlation data file and the FITS format cross-correlation data file. The correlation FITS files are the ones that should be used for all further data processing.
The directory where the PAF receiver FITS files were stored when the data were acquired is given by the FITS_DIR parameter in paf_directories.py. When the PAF is run stand-alone (without GBT control) this directory is set to
/export/raid1/gbtdata/Both FITS_DIR and COR_FITS_DIR will need to be changed to match the GBT archive directories when the PAF is run on the GBT, probably to
The output of the software correlator is a FITS file, which is defined by the FXcorFits python class in
on the gb.nrao.edu network.
After the data have been correlated the resulting cross-correlation spectra in the FITS file may be displayed with python scripts, examples of which are described here. The code in these examples show how to access the matrix of correlation spectra using the FXcorFits class.
To show the relative power levels in all cross product spectra use
/opt/local/bin/python show_matrix_pwr.py TINT_072313 2013_07_25_18:41:31 X 0where the arguments are the observing program name, scan name, X or Y array polarization, and an optional power offset value in dB. The display will be a 20 x 20 text table of cross-spectrum powers averaged over the spectrum channels expressed in dB. The power levels are relative and, hence, have an arbitrary offset. The top left to bottom right diagonal of this table contains the self-product value for each channel.
To plot the power spectrum of a cross-product of two PAF channels use
/opt/local/bin/python plot_pwr_spectrum.py TINT_072313 2012_07_15_14:59:20 X 2 2where the arguments are the observing program name, scan name, X or Y array polarization, and the two receiver channels for which you want the cross product, numbered from 1 to 20. If the two channels are the same number, the total power spectrum in that channel will be shown. If the channel numbers are different, the vector amplitude of the complex spectrum will be shown as a fraction of the mean total power in the two channels.
To plot the phase of a cross-product spectrum of two PAF channels use
/opt/local/bin/python plot_phase.py TINT_072313 2012_07_15_14:59:20 X 2 2where the arguments are the observing program name, scan name, X or Y array polarization, and the two receiver channels for which you want the cross product, numbered from 1 to 20. The two receiver channels must be different since the phase of a self-product will always be zero.