Alan H. Bridle (Chair)
Donald C. Backer
Edward B. Churchwell
Martha P. Haynes
Jacqueline N. Hewitt
David E. Hogg
K. Y. (Fred) Lo
28 January 1997
Document as compressed Postscript (22k)
Document as PDF (31k)
Amendment to policy adopted by NRAO in April 2000
The first link for each topic is to the committee's recommendation, the second link is to background discussion.
The NRAO should have a written, disseminated, policy for the treatment of large proposals. It is important, however, that this written policy be flexible enough to cover a wide range of circumstances.
All proposals that ask for more observing time than a (telescope-specific) threshold, and, at the NRAO Director's discretion, some proposals requesting less time than this, should initially be evaluated by an expanded "skeptical review" panel of five or more referees.
The panel should be drawn from the normal pool of proposal referees for the telescope, augmented if necessary by others who have recently been proposal referees. The panel should be roughly balanced between "experts" in the astronomical sub-discipline ad dressed by the large proposal, and cross-disciplinary "skeptics".
The panel should assess:
The panel will provide the Director with a recommended course of action and a summary of its deliberations.
(Also see Recommendation 5.)
For the VLA and VLBA, the threshold for skeptical review should be around 300 hours of observing time. For the 12-meter telescope, it should be around 1000 hours. For the GBT, the threshold should change as new instruments and higher-frequency capabilities are commissioned, and will need continual review. In all cases, these thresholds should be explicitly "fuzzy", i.e. the policy should make it clear that the NRAO Director has the option to send some proposals below these thresholds for expanded "skeptical review".
Proposers of "moderate-sized" (below-threshold) projects may also volunteer for expanded "skeptical review" of their proposals. This option provides a way to obtain a stronger guarantee of observing time for moderate-sized projects whose science could clearly be advanced by receiving such guarantees, in return for submitting them to a more demanding initial review. We emphasize that we see this as an option to be used rarely, and only in exceptional cases where the science would suffer if the project was done piecemeal through the regular proposal process.
The skeptical review panel for a large proposal should also advise the NRAO Director whether any further "expert" review of the proposal is needed in four main areas:
Not all large proposals will require further review in all of these areas, and many may not require further review at all. If a highly-rated large proposal is of sufficient scope or technical complexity to warrant ongoing review, the NRAO should make every effort to achieve this without over-burdening either the proposers or the expert referees. The arrangements for any ongoing "expert review" would be made at the discretion of the NRAO Director on a case-by-case basis.
If several large proposals for a given telescope are highly rated by the skeptical review panels, the NRAO Director should seek advice from a cross-disciplinary subset of the regular proposal referees about upper limits to the fraction of all observing time that should be devoted to them. Any policy statement about such upper limits must emphasize they will not be interpreted as "quotas" to be filled with large projects, however.
The NRAO should not make Announcements of Opportunity for the submission of large proposals. Large proposals should be submitted at the normal proposal deadlines, without special solicitation by the observatory.
We believe that it is.
Our reasons for concluding this are twofold:
(a) Large projects will (by any definition) be ones that impact other NRAO users' work to an unusual extent. The NRAO should therefore have a process that can reassure its users that the few large projects which do get scheduled have met unusual standards of scientific importance and of uniqueness, and also that they are of finite length. To the extent that the constraints imposed on 'standard' proposals by the VLA surveys have been widely accepted, there is consensus not only that these surveys are scientifically important but also that they could only have been done with the VLA. It is also important that the proposed disruption to other work ends eventually. We believe that the NRAO must be able to show that it is carefully balancing the scientific worth of large projects against their impact on smaller ones when making future decisions about scheduling large projects. We suggest that a key ingredient in this will be a more extensive "skeptical review" process for proposals that are above a certain threshold.
(b) Most large projects will also generate databases that are of interest to a large community of astronomers. It is therefore appropriate to seek that community's advice about the scope of a large project, about its data selection parameters, about data reduction methods, and about archiving and dissemination plans. A further, and possibly ongoing, "expert review" of large projects may therefore also be needed once they have passed initial "skeptical review". We also note that some large projects are merely long projects (e.g. large sample studies in which the individual observations are not especially challenging) but others may push the limits of the instrumentation in sensitivity, data rate or data volume. The latter may benefit from expert technical advice from an expanded community at an early stage of planning. We do not see how the NRAO could address either of the above areas satisfactorily just by extending the normal proposal review process to projects of arbitrarily large scope.
We do not see how to measure the breadth of support for large proposals, or to satisfy the user community that their observing parameters have been optimized, without having a threshold above which proposals get extra initial scrutiny.
Thus, a new policy is needed.
It also seems clear that no single-forum review could address all of the above issues well. Our proposal for a new policy has several optional stages (after the initial review) to deal with this.
The first question in the charge to the Committee also asked us whether, if a new policy is needed, it should be written down and disseminated. It will be important to strike an appropriate balance between (a) clarifying the observatory's future intentions about large projects and (b) specifying a policy in detail now that proves to be ill-suited to particular cases in future, or which is unnecessarily burdensome either to proposers or reviewers. We therefore seek an approach that has built-in flexibility, but which can and should be written down and disseminated to the user community.
The NRAO should have a written, disseminated, policy for the treatment of large proposals. It is important, however, that this written policy be flexible enough to cover a wide range of circumstances.
"Normal" proposals are reviewed by small specialized panels of discipline experts from outside the NRAO. A favorable review from within the discipline is a necessary, but we believe an insufficient, condition for scheduling a "large" proposal. A project large enough to constrain work in other areas of astronomy significantly should be asked to impress a review panel that also includes astronomers whose work will not directly benefit from the project's final database.
Such an initial review should ask :
An important ingredient in community acceptance of large proposals that displace other research significantly is that the duration of the large proposal is well understood, finite, and commensurate with the scientific priority of the project. It is therefore important to establish before a project begins that a specific (finite) grant of observing time is involved, and that any extension beyond this must be re-applied for either through the normal proposal process (if small) or by further skeptical review (if large).
The review panels should also be asked to advise about the appropriateness, and length, of any proprietary "holding time" for the data from large proposals. It is essential that the proposers and the user community clearly understand what has been agreed about the time scale of public release of data before a project is scheduled. Large proposals must therefore address this issue as part of their submission.
If the telescope is one on which dynamic scheduling is used, the "skeptical review" panel might also be asked to comment on whether a proposal is appropriate for use as part of that scheduling strategy.
The expanded "skeptical review" panels for large proposals should be drawn from people who are already refereeing other discipline areas for that telescope. For the single dishes, it may be necessary to augment the current referee group because there may not be enough current referees for the job. Recent referees, and other cross-disciplinary experts, should then be co-opted.
The heart of our suggestion is therefore that any proposal exceeding some threshold (in hours, discussed quantitatively below) be reviewed first by a "skeptical review" panel drawn from the pool of proposal referees for that telescope, but representing all major astronomical sub-disciplines served by the telescope. This would allow some of the same referees who judge smaller projects to weigh their priority against those of any large projects that might use up all their time. It ensures that large projects will be judged in the specific context of their impact on the other work currently proposed for the telescope, by a group of people well positioned to do so.
The questions of :
are also best handled by panels whose members referee both large and small proposals. In other words, with the proposed composition of the skeptical review panels, questions of priority among large and small proposals with similar science goals can be handled as they are now within the normal proposal process. We strongly prefer this approach to that of having a separate standing committee of "large proposal reviewers" who do not participate in the normal proposal-review process. Such a separate committee would be less aware of the overall scientific context with which large proposal(s) would compete. Also, the act of setting up a separate process for reviewing large proposals could itself generate pressure to have some such proposals scheduled. We do not think this is desirable.
Before discussing how to set the threshold for the "skeptical review" process at each telescope, we note that the focus of this report is the "large" proposal that is relatively short in duration but wide in its scheduling impact. A time-based threshold for skeptical review could also be exceeded by long-term monitoring projects (variability, pulsar timing, astrometry) that are long in duration but narrow in scheduling impact per observing period. Should such proposals also be subject to "skeptical review"? We believe so, if and only if it is crucial to their scientific goals that the full duration of the program be guaranteed "up front" (to the extent that the NRAO's contract makes sense of guarantees beyond 5 years). Absent a clear scientific reason for such a guarantee, we believe that long-term monitoring programs are better handled through the normal proposal process, via progress reports and follow-up proposals every few years.
All proposals that ask for more observing time than a (telescope-specific) threshold, and, at the NRAO Director's discretion, some proposals requesting less time than this, should initially be evaluated by an expanded "skeptical review" panel of five or more referees.
The panel should be drawn from the normal pool of proposal referees for the telescope, augmented if necessary by others who have recently been proposal referees. The panel should be roughly balanced between "experts" in the astronomical sub-discipline addressed by the large proposal, and cross-disciplinary "skeptics".
The panel should assess:
The panel will provide the Director with a recommended course of action and a summary of its deliberations.
(Also see Recommendation 5.)
At the VLA and VLBA:
We suggest that the threshold for an expanded initial review should be set in an explicitly "fuzzy" range of 200-300 hours of observing time. (300 hours corresponds to about 2 weeks of schedule time if done in one session.) Since 1990, use of this criterion to trigger additional review would have affected only about one project previously treated as "standard" at the VLA, plus the two VLA surveys. It would have affected five previously treated as "standard" at the VLBA. (These statements are based on statistics for the VLA and VLBA furnished to us by Barry Clark.) The number of past proposals that would have been exposed to "skeptical review" remains modest wherever the threshold could be set in the few-hundred-hour range (for these telescopes).
At the 12-meter telescope:
We believe that the main criterion for setting the threshold is "significant displacement of other proposals", so a reasonable criterion is that the threshold should be around 10 times the mean length of scheduled proposals. A threshold around 1000 hours might therefore be more appropriate for the 12-meter telescope.
At the GBT:
In the case of the GBT, we can expect proposal pressure to be a strong function of time as new instrumental capabilities are commissioned. There may however be times early on when instruments are unexpectedly unavailable, and dynamic scheduling is needed. There may be good reasons to seek to combine some classes of large proposal with a dynamic scheduling strategy: e.g., some survey observations at low frequencies might be appropriate as "backup" projects at times when higher frequencies are unavailable due to weather or equipment problems. We suggest that a working group be established to examine such issues for large proposals at the GBT, both with regard to setting appropriate upper limits to the time allocation for large proposals, and with regard to their role in any dynamic scheduling strategy for the telescope.
Our main reason for suggesting a "fuzzy" threshold for the initial "skeptical review", i.e. an explicit statement that discretion will be exercised by the NRAO in applying the criterion, is to discourage attempts to avoid the process by tailoring proposals to be just under a strict threshold.
For the VLA and VLBA, the threshold for skeptical review should be around 300 hours of observing time. For the 12-meter telescope, it should be around 1000 hours. For the GBT, the threshold should change as new instruments and higher-frequency capabilities are commissioned, and will need continual review. In all cases, these thresholds should be explicitly "fuzzy", i.e. the policy should make it clear that the NRAO Director has the option to send some proposals below these thresholds for expanded "skeptical review".
Some 200-300 hour projects have already been done at the VLA and VLBA via series of consecutive proposals for 100 or so hours. This approach provides a way to do moderate-sized projects through the normal channels. We see no reason to discourage it. It amounts to an ongoing, but not guaranteed, grant of observing time on the basis of demonstrable progress, with the review time scale being set by the proposers' success with, and capacity for, the project.
This approach may not be well-suited to all moderate-sized projects, however. Data subsets or pilot projects do not always produce good science. Doing a moderate-sized project piecemeal so as to maximize short-term "excitement" at proposal deadlines may distort its overall strategy. Some VLA observations of source samples that interest a wide community have been fragmented into small proposals carried out by different groups. The resulting loss of homogeneity limits the long-term benefit to the community, which would be better served by the data produced by a small number of moderate-sized and well-coordinated proposals rather than a large number of small, independent ones. (VLA observations of the 3CR continuum sources, and of galactic water vapor masers are particular examples of this known to us.)
The "volunteer" mechanism may also be appropriate for proposals that require coordinated observing at several telescopes. A "skeptical review" committee might be better able to evaluate the whole plan, rather than leaving each part for independent (uncoordinated) review through different channels in the normal proposal process.
We therefore see some merit in inviting proposers to volunteer projects of moderate size (100-300 hours) for the enhanced "skeptical review". This might be a way for a proposer to ensure that moderate-sized proposals obtain all the time that they need (regardless of graduate student involvement or the status of intermediate results). It might also encourage attempts to produce more homogeneous, moderate-sized databases that would benefit a wider community. Success in such proposals would also allow proposers to marshal resources (staff, computer resources, funding, etc.) better for moderate-sized projects, simply by clarifying that all of the requested observing time would be granted (the current "will be considered further" status at the VLA leaves some uncertainties hanging over proposals in the present queue.) The fact that a proposal had successfully passed a more demanding skeptical review process at the NRAO could make it more attractive to funding agencies.
PI's will therefore have some incentives to "volunteer" for extra review, and it seems advantageous to offer this possibility as an option. We should however aim for a situation wherein only a small minority of all proposals goes for "skeptical review". This goal could be reached by holding proposals that undergo skeptical review to a significantly higher standard, in recognition of the greater long-term commitment that would be made to successful ones. (This has happened with the "key projects" category at the KPNO, where the success rate is small.)
Proposers of "moderate-sized" (below-threshold) projects may also volunteer for expanded "skeptical review" of their proposals. This option provides a way to obtain a stronger guarantee of observing time for moderate-sized projects whose science could clearly be advanced by receiving such guarantees, in return for submitting them to a more demanding initial review. We emphasize that we see this as an option to be used rarely, and only in exceptional cases where the science would suffer if the project was done piecemeal through the regular proposal process.
Some, but not necessarily all, large projects, may need further review by more a narrowly-focused expert panel before they are scheduled.
The impact of large proposals on other NRAO users requires us to ensure that their observing techniques and time allocations are optimized both to the science and to the telescope involved, and that the final databases are made available promptly and in scientifically robust forms.
The main areas in which further expert review may be appropriate before a project is scheduled are:
In some cases, it may be important to require a pilot or demonstration project to prove an observational technique before going ahead with the project as initially proposed. This area may require review by a group that involves scientists, engineers familiar with the instruments, and telescope operations staff.
Ongoing review of a project after it has been started may also be appropriate, to monitor:
a) Data-processing progress: ongoing review may be particularly appropriate for projects whose data volume presents a major computing challenge. If such review is required, the supervision should be "strong". By this we mean that the review panel must be able to recommend withholding later installments of observing time if the project does not meet data-processing targets (quality and speed of the data analysis) in a timely way. Such a panel will in effect re-referee the project while it is in progress, and could recommend no further time allocation if agreed data-processing milestones were not met.
b) Construction of an accessible public repository for data products. If a big community's observing time is "taxed" to make room for large projects, then that community should expect to share the benefits of the final database quickly. This implies a review process aimed at ensuring prompt access to calibrated data whose quality are uniform and well-understood. It also requires that large proposals clearly state their plans for public access to the data (and the nature of the proposed data products) in order to be sent for skeptical review. If ongoing "expert" review of the data products is required, it would typically be done by a panel with a mix of scientific and computer expertise.
To the extent that any of these issues apply to a particular large proposal, they imply review by groups different in composition from the initial "skeptical review" panel. Unlike this panel, which should be cross-disciplinary and is probably best drawn from the existing (external to NRAO) referee pool, "expert review" panels would benefit by including people who are not currently acting as NRAO referees. They could include NRAO scientific and technical staff with special knowledge about the telescope, the science, or data processing relevant to the proposal. Some would need to be ongoing. These might use a range of formats, including telephone conferences, face-to-face meetings or workshops, that are not traditionally used for proposal refereeing at the NRAO.
We emphasize that not all "large" projects should need exposure to all of the above forms of ongoing monitoring and supervision. It is likely that all projects above some very large (1000-hour?) threshold should have some ongoing supervision by an ad hoc "expert panel". But length of observing time alone is not the only criterion for whether ongoing expert review is necessary. The technical "degree of difficulty" of the project is clearly significant. For example, proposals that are straightforward in terms of observing technique and data analysis, but which require 'simply' large amounts of time, might be selected on the basis of a favorable evaluation of the skeptical review committee. But a proposal which challenges the current technical frontier (Zeeman work on the GBT, a dramatic new pulsar search strategy) and which requires an extensive block of time should surely be reviewed by a group with a strong technical background before being scheduled.
It is important that contentious areas, e.g. "research" issues about data processing, etc. should not be allowed to stymie progress on a proposal. Issues such as timely completion and accessible archiving of the data will be important for many large proposals, however. The heart of the issue here is that the style and extent of any ongoing supervision of large projects should be determined on a case-by-case basis. Any policy that is written down now should simply define a process that is flexible enough to make this case-by-case determination. It should not try to anticipate all of the possible supervisory issues in advance (though we have pointed to a few above).
We therefore suggest that when a "skeptical review" panel for a proposal assigns it high scientific priority, they should also recommend whether the proposal should be subject to further expert review, and if so in what areas. The scope and style of any further review process should however be decided by the NRAO Director, with advice from any other appropriate sources. It is important that the process begin with input from representatives of the whole astronomical community served by the telescope involved, but the "skeptical review" panel should neither be expected to, or expect to, specify the entire subsequent review process.
(We note parenthetically that in discussing this area, we were guided by the recent experience with the two VLA surveys. We understand that there were significant technical issues that had to be settled for each of the surveys, primarily in the area of data analysis. We also believe that the community relied on the survey oversight committee(s) to ensure that the data were made readily available to the public in a timely manner. Perhaps this would have occurred anyway, but we believe that it was helpful to have a mechanism in place to strengthen the resolve of the PI's!)
The skeptical review panel for a large proposal should also advise the NRAO Director whether any further "expert" review of the proposal is needed in four main areas:
Not all large proposals will require further review in all of these areas, and many may not require further review at all. If a highly-rated large proposal is of sufficient scope or technical complexity to warrant ongoing review, the NRAO should make every effort to achieve this without over-burdening either the proposers or the expert referees. The arrangements for any ongoing "expert review" would be made at the discretion of the NRAO Director on a case-by-case basis.
There must be some upper limit, or we could have a situation where all the time goes to a few large projects -- an inappropriate asymptote for a national facility with a large, diverse user base.
The limits should be expected to vary from telescope to telescope, and with time at any given telescope, just as the overall proposal pressures vary in response to major changes in instrumentation, to discipline-wide shifts in astronomical emphasis, or to astronomical transients such as supernovae and comets.
In general, we feel that while the over-subscription rate on a telescope remains under 2:1, the question of exactly how upper limits are set for large proposals may not be too pressing. But if a large proposal or proposals raise the over-subscription rate much over 2:1, their effects would likely be noticeable across a broad community, and the upper-limit question would be more pressing.
For the more heavily over-subscribed facilities such as the VLA, VLBA, and GBT (presumably) the appropriate upper limits would be below those appropriate for instruments such as the former 300-ft, the 140-ft and the Green Bank interferometer in the years before their shutdown. In the later years of a telescope's operation, doing large-scale surveys becomes attractive for operational, as well as scientific, reasons. (Simplifying telescope schedules and minimizing equipment changes are often good operational strategies as a facility ages).
Within this committee, our thresholds for discomfort about large proposals displacing smaller ones on an instrument in the prime of its scientific life ranged from 1/6 to 1/3 of the total observing time. (Large projects that require time in the most "popular" LST ranges for galactic and extragalactic work would obviously constrain other work more severely than those with intrinsic LST flexibility.)
We concluded however that it is probably inappropriate for us to go beyond this to assess general large-proposal upper limits for any particular telescope as part of this report. Instead, we wish to recommend how such an assessment should be obtained for any telescope when it is needed.
In our opinion, the best group to assess this issue would be a cross-disciplinary panel of scientists with access to the statistics of observing time requests from, and an appraisal of the scientific vigor in, the different sub-disciplines that dominate t he proposal demand at the telescope. This description matches that of the "cross-disciplinary" parts of our proposed "skeptical review" panels.
We also believe that advice on upper limits to the observing time for large proposals will be needed only on the (presumably rare) occasions when more than one large proposal at a time is highly rated by the skeptical review panels for a given telescope. We therefore suggest that, on these occasions, the NRAO Director seek such advice from the cross-disciplinary cohort of those skeptical review panels.
It is important that any upper limits that are established at such times not be re-interpreted later as quotas of time that "should" be filled by large proposals. High scientific priority based on reviewing proposals that were initiated on the "open market" by users should be the only driver for assigning time to a large proposal in competition with smaller projects.
If several large proposals for a given telescope are highly rated by the skeptical review panels, the NRAO Director should seek advice from a cross-disciplinary subset of the regular proposal referees about upper limits to the fraction of all observing time that should be devoted to them. Any policy statement about such upper limits must emphasize they will not be interpreted as "quotas" to be filled with large projects, however.
The committee considered whether the NRAO should explicitly solicit proposals for large projects via Announcements of Opportunity, targeted either to specific disciplines or to special deadlines (other than those of the regular proposal process.)
It was our unanimous opinion that this would be undesirable.
It would separate "opportunities" for proposing large projects from the regular proposal process, whereas we see merit in keeping the processes for large and small proposals well-coupled. It is also hard to see what benefit would come by encouraging the whole user community to think about large proposals simultaneously.
The NRAO-operated telescopes are ground-based and flexible in their capabilities, so operational and planning considerations differ greatly from those needed to establish the scientific program of space-borne instruments, for example. The AO approach would however place some obligation on the NRAO to schedule some large projects after a period in which it had encouraged the whole user community to make proposals for them.
It is particularly undesirable to create an artificial imbalance between the pressures for large and regular proposals when our ultimate goal is to find an appropriate balance. We believe that balance is more likely to be achieved through a proposal process that is driven mainly by the scientific interests of individual investigators, rather than through one driven by ad hoc deadlines.
The NRAO should not make Announcements of Opportunity for the submission of large proposals. Large proposals should be submitted at the normal proposal deadlines, without special solicitation by the observatory.