John's collection of words on metrics: ------------------------------------------------------------- From: "David Woody" To: "Min Yun" Subject: Re: dynamic range and fidelity Date: Wed, 23 Aug 2000 14:15:28 -0700 MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-MimeOLE: Produced By Microsoft MimeOLE V5.50.4133.2400 Min I haven't been keeping up on the details of the image simulations that are being done, but I assume that the simulations are ignoring additive noise, i.e. no receiver noise. Then the noise in the images as given by the difference maps will scale with the total or peak flux. So the diff maps should be turned into dimensionless fractions by dividing by either the total or peak flux (probably the peak flux). We don't know the flux of the sources we will be looking at, so the diff maps in Jy are not very useful. The rms of the diff map is composed of roughly two parts, the on-source part and the off-source part. The various definitions of dynamic range and fidelity try to get at these two components of the errors. Using masks and thresholds can be misleading, because the threshold that an astronomer might want to use to decide which statistic to apply for determining whether a feature in an image is real will depend on the amount of additive noise, which is not known and is not included in the simulations. There should be a simple method to separate out the two error components that does not require masks or thresholds. What about doing a simple linear fit of the (diff-map)^2 to A + B*(original simulation image)^2. 1/sqrt(B) would be interpreted as the fidelity, i.e. the errors in the map that are proportional to the image. 1/sqrt(A) would be the "off-source" dynamic range. This fit is not computationally time consuming or difficult. This gives two simple and mathematically well defined imaging quality measures that should contain most of the information we need to evaluate the images. Anything more complicated risks the danger of applying a particular slant to the results that depends upon "what one expects to see". Note: pointing and other calibration type errors are multiplicative and can be included in the simulation studies and the above type of image quality analysis can be applied. Cheers David ------------------------------------------------------------------ From mholdawa@cv3.cv.nrao.edu Fri Jan 5 15:40:15 2001 Date: Wed, 23 Aug 2000 11:25:29 -0700 (MST) From: Mark Holdaway To: alma-config@zia.aoc.NRAO.EDU Subject: [alma-config] image fidelity, etc 1) Dynamic range: off source errors Select an off-source region (you could do this by taking your model image, convolving with Gaussian; then setting everything below 1e-5 to 1.0, everything above 1e-5 to 0.0), calculate the rms offsource, divide that into the reconstructed peak. 2) on-source dynamic range: Invert the region defined above to that it is "on-source". may want to change the clip level to emphasize "on-source" (ie, take out the barely off-source region), calculate the on-source rms in the difference image, divide that into the rconstrcuted peak 3) median fidelity: Make error (difference) image, make a mask based on the convolved model clipping below some level (ie, you now have a mask that defines the on-source region of interest). divide the absolute error into the pixel values of the convolved image. As some pixel errors will accidentally be very small, take the median of this fractional error image. Often the median fidelity will reflect the fractional error level in weak pixels. To overcome this, you could set the clip value higher, or you could weight the fidelity image by the pixel values: 4) moment fidelity: Define mask as above; calculate the fidelity image within the masked region: convolved model / abs( convolved model - reconstruction ). Weight each pixel by the pixel value (in the convolved model); Take the average, normalize. -mark -------------------------------------------------------------------------- From guillote@iram.fr Fri Jan 5 15:40:50 2001 Date: Thu, 9 Nov 2000 16:32:03 +0100 From: Stephane Guilloteau To: John Conway , Steven Cc: alma-config@zia.aoc.NRAO.EDU Subject: Re: [alma-config] Difference images for the simulations using theSIL 6) We compute the Fidelity image as the ratio of Smooth image to the absolute value of the Difference image = abs(Smooth-Clean), truncated at 0.7 rms(Smooth-Clean). 7) We compute the "Fidelity-Range" as defined by Leonia Kogan (max(Clean)/rms(Smooth-Clean)), and the histogram of the distribution of the Fidelity image. An example of the graphic output is attached to that. The whole procedure has been tested to give accurate results for simple images (i.e. there is no registration error, ------------------------------------------------------------------------ From: John Conway To: Steven Cc: alma-config@zia.aoc.NRAO.EDU Date: Thursday, November 09, 2000 4:02 PM >fairly automatically. > >On the issue of metrics rather than waiting for us all to agree >on a final metric I suggest you should go ahead and do the >simlpest things first(!) Pure difference >images side by side of the two arrays will >give us a first order idea of if there is any difference in >imaging quality at all and what the nature of any difference >is. Fractional errors on the bright regions are also easy to >compute - you could start with simply dividing the difference >between the difference and the true image, setting the >'magic-blanking' cutoff at say 1% of the peak brightness. >This would be just to get a quantitative idea of what the sizes of >the 'on-source' fractional errors are, these >can then be compared to other sources of errors (pointing, phase >etc etc). For a final comparison we may need more metrics >(e.g 'off-source' dynamic range) or fractional errors at >different model cutoff levels but I think if we can look at the >first order results we will be much better placed to discuss this >(at the next telecon perhaps). > > John. > ------------------------------------------------------------------- > From steven@heddle97.freeserve.co.uk Fri Jan 5 15:41:16 2001 Date: Mon, 18 Dec 2000 23:02:10 -0000 From: Steven Reply-To: Steven To: alma-config@zia.aoc.NRAO.EDU Subject: [alma-config] Histograms, max, min, mean, rms, and fidelity plots for the C arrays The results expand on the earlier CLEAN and difference results and now show statistics of the difference images, and histograms of pixel values. Further, as a first attempt at a fractional difference measure I have (I believe) implemented the fidelity plots and histograms shown by Stephane. I have taken the fidelity image as the ratio of (convolved model image) and (absolute value of the difference image truncated at 0.7 x rms of difference image). Importantly, in the figures shown this latter rms value is different for each of the K and C array cases (i.e. the difference images for the K and C arrays have different rms values), resulting in the C array Fidelity image showing more structure despite (or as a consequence of) somteimes having half the rms value for its difference image. For comparison I feel that the same value should be used to truncate the denominator in both the K and C cases. Not being experienced in the interpretation of the fidelity images, I am not sure whether I am raising more questions than are answered by their provision, and realise now that it might have been useful to calculate Leonia's fidelity range value in addition. Perhaps the simple histograms of pixel value for the difference images are most immediately informative. Nonetheless, the scripts to obtain these results are wriiten, and can provide results for the A and B cases in about a day if required, or can easily be changed. -------------------------------------------------------------------------- Date: Tue, 19 Dec 2000 09:54:48 -0700 (MST) From: Mark Holdaway To: Steven Cc: alma-config@zia.aoc.NRAO.EDU Subject: Re: [alma-config] Histograms, max, min, mean, rms, and fidelity plots for the C arrays A few notes on the fidelity calculations and Dec 18 simulations: General: All of the pages (ie, all model sources) say the mean fidelity was calculated over 66049 pixels. I would have to say that the mean fidelities cited are incorrect (ie, they represent the fidelity calculated with the wrong mask or something). For example, look at the Spiral configuration results for a long track observations on Mars, -23 d case: the distribution is fairly flat, out to 50; you'd think the fidelity would be 25, but its quoted as 11. ------------------------------------------------------------------------ From steven@heddle97.freeserve.co.uk Fri Jan 5 15:41:44 2001 Date: Wed, 20 Dec 2000 12:52:53 -0000 From: Steven Reply-To: Steven To: alma-config@zia.aoc.NRAO.EDU Subject: [alma-config] Re. Mark's comments on the Fidelity images Thanks for the comments Mark. Regarding the technical points raised regarding the creation of the results: 1) Negative fidelities- these do indeed arise from negative values in the model brightness distribution (or in this case the brightness distribution of the model dowsampled to the appropriate pixel scale and convolved with the specified CLEAN beam). Doesthe consensus say redo the results using the absolute value of the smoothed model for the division, or use the absolute value of the model from the word go i.e. redo the imaging and CLEANing as well, prior to differencing etc. 2) I believe the quoted mean fidelities are correct, with the log scale of the histogram perhaps being deceptive. Considering the CCMAR_4-23B7 case (spiral, 4 hour track, imaging Mars, -23 d) with a quoted mean of 11, the attached text file shows the full output from running IMEAN on the fidelity image, with the same task used to generate the histogram. The histogram is indeed approximately flat out to a fidelity of 50, with the important exception of the first bar of fidelities between plus and minus 0.19 which contains 44,000 of the 66,000 pixels, implying a ball park mean of approximately (25*22/66)=8.3, with the actual figure plausibly raised to 11 by the slight peak in the histogram between 37 and 45. Keep the comments coming folks. In particular, what are your feelings about having different truncation values for the denominators of corresponding K and C (ring and spiral) cases when calculating the fidelity image? Also do you wish the fidelity range to be calculated? Cheers, Steven ------------------------------------------------------------------------- From guillote@iram.fr Fri Jan 5 15:41:59 2001 Date: Wed, 20 Dec 2000 14:56:10 +0100 From: Stephane Guilloteau To: Mark Holdaway Cc: alma-config@zia.aoc.NRAO.EDU Subject: Re: [alma-config] Histograms, max, min, mean, rms, and fidelityplots for the C arrays > > Original message from Mark > >The mean fidelity, etc: >These numbers are dominated by the very faint emission that isn't showing >up on your contour plots or grey scale images. You might want to also >put in a fidelity weighted by the pixel brightness, or have a higher >flux cutoff to examine how well the bright stuff is imaged. > One thing we found out usefull in understanding the fidelity is a correlation (scatter) plot of residuals vs intensity. The slope of that gives you the mean fidelity... A similar plot of fidelity vs intensity is quite interesting too > >I don't understand how you are getting negative fidelities, unless the >model brightness distribution has negatives in it, in which case we should >probably take the absoulte value of the model image while doing the >division. > Yeah, take the absolute value. It doesn't matter anyhow. -----------------------------------------------------------------------------> From mholdawa@nrao.edu Fri Jan 5 15:42:14 2001 Date: Wed, 20 Dec 2000 10:40:23 -0700 (MST) From: Mark Holdaway To: Steven Cc: alma-config@nrao.edu Subject: Re: [alma-config] Re. Mark's comments on the Fidelity images > > > > I am still lacking some understanding here. Why do all of the images have > > 66,000 on-source pixels? It seems to me that you must be counting some > > off-source pixels or something to get so many pixels with very low > > fidelity. > > > > -Mark > > > > For generality all of the cases have been treated the same way when > CLEANing, i.e. no CLEAN boxes, no default images. Thus when I am calculating > the statistics for the images shown, they are derived from the whole image > as posted, i.e. a 257x257 area (I intended 256x256, but it doesn't matter). > So yes, there will be a lot of off-source pixels counted. However, as I have > mentioned before, I can do CLEAN boxes and any other bespoke CLEAN parameter > for that matter on a per model basis, if necessary. > > Cheers, > Steven I've probably missed some decision the alma-config group made while I wasn't paying attention, but: This is why there are so many low fidelity pixels in your results, because you are basically dividing zero by some error number. My understanding of the fidelity is that it measures the on-source SNR. When I calculate the fidelity, I screen out off source pixels based on a lower limit on the pixel values in the convolved model ... ie, when you convolve the model, you will have Gaussian wings going off source, and the point of the cutoff was to ignore these regions way off source. I would recommend another procedure for studying the off-source errors, rather than lump the on and off-source errors into a single measure; they tend to be about an order of magnitude different. -Mark ------------------------------------------------------------------------ From guillote@iram.fr Fri Jan 5 15:42:26 2001 Date: Wed, 20 Dec 2000 20:17:50 +0100 From: Stephane Guilloteau To: alma-config@nrao.edu Subject: Re: [alma-config] Re. Mark's comments on the Fidelity images -----Original Message----- From: Mark Holdaway To: Steven Cc: alma-config@nrao.edu Date: Wednesday, December 20, 2000 6:40 PM Subject: Re: [alma-config] Re. Mark's comments on the Fidelity images > >> > >> > I am still lacking some understanding here. Why do all of the images have >> > 66,000 on-source pixels? It seems to me that you must be counting some >> > off-source pixels or something to get so many pixels with very low >> > fidelity. >> > >> > -Mark >> > >> >> For generality all of the cases have been treated the same way when >> CLEANing, i.e. no CLEAN boxes, no default images. Thus when I am calculating >> the statistics for the images shown, they are derived from the whole image >> as posted, i.e. a 257x257 area (I intended 256x256, but it doesn't matter). >> So yes, there will be a lot of off-source pixels counted. However, as I have >> mentioned before, I can do CLEAN boxes and any other bespoke CLEAN parameter >> for that matter on a per model basis, if necessary. >> >> Cheers, >> Steven >pixel values in the convolved model ... ie, when you convolve the model, >you will have Gaussian wings going off source, and the point of the cutoff >was to ignore these regions way off source. Yes, but there is an arbitrary decision here. What criterium do you use ? > >I would recommend another procedure for studying the off-source errors, >rather than lump the on and off-source errors into a single measure; >they tend to be about an order of magnitude different. > We have two proposals there: - John Conway suggestions of using 3 different thresholds to define the "on-source" region to compute the mean fidelity (e.g. 10 %, 1%, 0.1 % of the peak intensity) - some variations of what I mentionned in my previous E-Mail, i.e. getting the slope of the Error/Intensity scatter plot. That plot in itself contains the whole information on fidelity, and you could just get the previous result by fitting the slope for all points above 10 %, 1% or 0.1 % of the peak intensity... I personally don't like defining the thesholds in terms of Peak Source fraction, but rather in terms of Mean Residual value. In other words, I would prefer to have the number of pixels with fidelity higher than some value, rather than the mean fidelity value of some arbitrary number of pixels. This was the idea of the Fidelity plot. Stephane -----------------------------------------------------------------------------