COV Response

Response to Issues Raised in the April 2004 Committee of Visitor's Report to the Office of High Energy Physics

Introduction

This document responds to the Report of the 2004 Committee of Visitors (COV) for the Office of High Energy Physics (OHEP) which met on March 8-9, 2004, and submitted its report to the High Energy Physics Advisory Committee on April 18, 2004. The report was subsequently transmitted by HEPAP to the Director, Office of Science, on April 22, 2004.

We are pleased that, based on its review of the major activities of OHEP?national laboratories, universities, technology R&D, and projects?the Committee found that overall the Office?s functioning was very professional and that they were impressed with the responsible and excellent job the Office was doing in soliciting and evaluating proposals, making grants, and monitoring the funded programs. However, the Committee did find some areas of concern and made several observations, recommendations, and suggestions for improving the functioning of the office. This response focuses on those issues, provides feedback, and indicates what actions are planned to address each issue.

Staffing Issues

Issue 1. The COV found that the most serious problem throughout its review is that the OHEP is very seriously understaffed, due to a combination of unfilled positions and need for new positions to carry out functions where the office is currently deficient. This includes the areas of program planning, large facility and laboratory oversight, and university oversight. Thus, the COV recommended that OHEP strive to fill its unfilled positions as soon as possible and to request authorization to create the new positions outlined.

Action: OHEP has been aware for some time of the staffing issues outlined in the report and has been striving to fill its vacant positions. The vacant physicist position was offered to an individual and, after a protracted period of time, the offer was declined in March 2004. The position has been reposted for three months, closing on July 23, 2004. The position has been advertised in the CERN Courier and Physics Today and a write-up seeking applicants has been sent to the APS Division of Particles and Fields (DPF) members, OHEP Principal Investigators, spokespersons of major high energy physics experiments, and to the management of five laboratories (FNAL, SLAC, LBNL, BNL, ANL) which have a sizable HEP program. Our request for applicants for the vacant IPA physicist position has been sent to the same sources.

With regard to additional staff, the OHEP is in the process of submitting a staffing plan that takes into consideration the COV recommendations. Additional staff is requested as appropriate. These additional needs will be considered in the context of overall SC staffing needs, available slots, and program direction funds. Decisions will be made on a timely basis.

Travel Funds

Issue 2. The COV found that a lack of travel funds is limiting the ability of OHEP to carry out its program evaluations and review processes in an effective manner. In addition, they felt that site visits are of critical importance in evaluating physics research and technology R&D groups. For this reason, they recommended that OHEP make every effort to increase the travel funds available for site visits to review and monitor the program.

Response and Action: The Office of Science agrees with the Committee as to the importance of site visits to reviewing and monitoring its programs, including those of the Office of High Energy Physics. While every effort will be made to provide additional resources for this purpose; it is important to note that OHEP needs must be balanced against those of the other offices within SC when competing for available resources within the separate Congressional allocation for Program Direction which includes funds for salaries, travel, etc.

Program Planning

Issue 3. The COV stated that the functions of the office would be greatly improved by adding a dedicated program planning function. The COV believed that, with dedicated personnel, development of a database, and the use of modern computer tools, the Office would be able to analyze budget action implications and improve the ability to do long range studies or analysis.

Action: We agree that OHEP would benefit from a program planning function that could, among other things, focus on program level strategic planning and how to integrate this planning into the budget. The resources for this function were addressed above under "Staffing Issues."

Program Funding Decisions

Issue 4. The COV felt that a concerted effort should be made to ensure that, as much as possible, funding decisions are based primarily on factors that will lead to the strongest possible program. Therefore, the Committee recommended that OHEP should make funding decisions based primarily on excellence, priorities within HEP and the overall success of the program. Furthermore, they recommended that, where possible, budget reductions or increases should be implemented strategically, rather than across-the-board.

Response and Action: We agree that funding decisions should be based primarily on excellence and that funding reductions and increases should be implemented strategically rather than across-the-board or, like "peanut butter" as it is sometimes called. We believe that the portfolio review, discussed in more detail in Issue 17, will be a good mechanism to reevaluate all research from a common point of view and a visible means to ensure that the research is of the highest quality overall. OHEP is making greater efforts to plan strategically for the HEP program, and to make implementation of that plan more transparent in funding decisions. OHEP has engaged HEPAP in assessing priorities and developing a long-term strategy for the HEP program, and we anticipate more interactions on these issues in the future. Also with the addition of a dedicated planning function in the office (proposed above), OHEP should be able to respond to changes in priorities in a more strategic manner.

Integrity and Efficacy of the Process of Treating Proposals

Issue 5. The COV concluded that OHEP is "continuing its traditionally excellent oversight of the university program." They went on to suggest that, in order to assure that resource allocations best reflect the quality of the research programs at the individual universities, some form of regular comparative review of the entire university program should be instituted.

Response and Action: "Comparative review" has been employed in the university program peer review process in the following ways. It is a standard feature of the OHEP review request letter to ask for comparison with other research programs with which the reviewer is familiar. In addition, OHEP often sends more than one proposal to a given reviewer so that the proposals can be compared and contrasted. This is the kind of comparative review recommended in the "Gilman" HEPAP subpanel report in 1998, and OHEP continues to utilize as needed this kind of comparative review. As to a comparative review of the entire university program, this will be handled as part of the overall OHEP "Research Portfolio" review recommended in Issue 17.

Issue 6. The COV found that the Advanced Technology R&D program continues to have significant success given the modest funds allocated to it, that HEP continues to lead the way for all fields in the development of accelerator concepts, and that this is a large contribution to the overall science and technology program in the U.S. While acknowledging that processes appear satisfactory, they suggested several possible refinements. These include establishing deadlines for proposal submission to allow comparisons among the proposals for the purpose of establishing priorities; adding additional technologically knowledgeable particle physicists to the roster of reviewers to more evenly distribute the review load; adding more expert accelerator consultants on program review committees to enhance the quality and continuity of the review and monitoring process; and conducting periodic external reviews of the program.

Response: Improvements in processes, including better documentation and make-up of reviewer lists, are being proposed and discussed and will be implemented by January 2005. We concur with the recommendation to add more expert accelerator consultants to the program review committees, and every effort will be made to do so commensurate with scale and balance of the review. As to an external review of the entire program, this will be implemented within the overall OHEP "Research Portfolio" review recommended in Issue 17.

Issue 7. Overall, the COV was satisfied that the process in place to review, recommend, authorize, and document funding actions at the laboratories is adequate. However, the COV noted that Field Work Proposals (FWPs) are not used due to the fact that the schedule on which they are received is poorly matched with DOE?s budget-making schedule, and because some labs do not adhere to realistic funding scenarios and do not represent the labs actual priorities. The Committee found that in some cases the content of the FWPs was out of date and/or inaccurate. Therefore, the COV recommended that FWPs either be made useful, or eliminated; and that making them useful would minimally require advancing their due date by about three months and developing a format that makes the laboratory priorities apparent within realistic funding scenarios.

Response: FWPs are a Department-wide requirement, and changing them is not solely within the purview of SC. As an alternative to the FWP, OHEP has been gathering necessary information for its laboratory budget formulations and priorities from the laboratories during the annual budget briefings and program reviews.

Issue 8. The Committee found the university grants documentation to be difficult to navigate primarily because the information is only available in paper format and because of the sheer volume of documentation. They recommended improvements in documentation organization so that ready comparison between requests and awards would be clear in the documentation and the distinction between continuing proposals and awards and supplementary proposals and awards would be clearer.

Response and Action: We concur in this recommendation. Improvements in documentation organization and format are being proposed and discussed and will be implemented by January 2005. Limits on proposal lengths are being implemented (see below).

Issue 9: In conjunction with the above recommendation, the COV suggested imposing length limits on new proposals to not only contribute to record keeping, but also as a way to make proposals more accessible to reviewers. In addition, they suggested that a more uniform proposal format and list of requirements would also help

Response and Action: We concur in this recommendation, and OHEP is preparing a letter for grantees that discusses proposal length limits and other requirements not contained in the standard solicitation announcements. This is an evolving process, but it should converge over the next year.

Issue 10: The COV was concerned with the lack of electronic documentation available on the universities, especially numerical, fiscal, and demographic data. They believed that steering the evolution of university programs would be significantly improved with the availability of a database with critical information and some analysis tools. Therefore, the COV recommended requiring data from grantees containing useful demographic information such as the number of faculty, senior scientific staff, postdocs, graduate students, undergraduate students, engineers, and technicians. They recommended that the data be kept in a modern database so that it is easily accessible for studying funding trends, responding to changes in priorities, and more generally planning for the future of the field.

Response and Action: Data from grantees containing demographic information is being collected on physicist faculty, senior research scientists, postdocs, and graduate students. However, information on undergraduate students is not collected because our mission is graduate research training, and we do not normally support undergraduates on grants, except as hourly technicians. Similarly, information on engineers and technicians is not collected because many of these are supported on project funds, and this support often does not flow through the grants, or is hard to track. However, we are considering trying to track engineers and technicians that are considered part of the base grants (i.e., not considered supported on project funds). While we agree with the COV that developing a database with this information and maintaining it would be very useful, this would require considerable effort that additional staff could enable. This requirement is factored into the OHEP staffing plan.

Issue 11: The question of making more information regarding funding decisions publicly available was raised by the COV. While they acknowledged that this question needed more thought, the COV believed that comparative review and consequently more openness would help strengthen the program?s credibility. Therefore, they recommended that more statistical, average information should be made available on the World Wide Web.

Action: New statistical, average information was presented to HEPAP at its April 18-19, 2004 meeting. This presentation has been made available on the OHEP website (View presentation) and it will be updated periodically.

Integrity and Efficacy of the Program Management of the National Laboratories and for Large Facilities

Issue 12: The COV found that decisions on laboratory funding are based primarily on information received during budget presentations made by laboratory managers to OHEP staff in a format consistent with formats of internal budget documents. While this format is useful to OHEP, the Committee felt that, as a result, a large fraction of the laboratories? budgets (about 90%) are not given much scrutiny during the budget making process. Therefore, the COV recommended that laboratory HEP budgets be subjected to a bottom-up analysis periodically, although the period should not be every year due to the complexity of the task.

Related to getting a better understanding of laboratory budgets, the Committee felt that the budget information received across the laboratory program would benefit from being in a standardized format and felt that this would also facilitate meaningful comparisons between the costs of similar activities in different labs. Therefore, the COV recommended that information on laboratory budgets should be collected in a uniform format and tracked annually.

Response and Action: We concur with both of these recommendations. In fall 2002, OHEP initiated annual budget briefings by ANL, BNL, FNAL, LBNL and SLAC. Starting in 2004, OHEP also initiated Operations Reviews of its major facilities, where the detailed breakdown of operating budgets is presented. Every three years, OHEP will expand these annual budget briefings and Operations Reviews to include a bottoms-up analysis of each labs? budget. With regard to a standardized format for the information presented from the laboratories, OHEP currently provides a template to the labs and asks that budget material be presented according to this format. In the past, however, the labs have modified this format somewhat to better fit individual laboratory internal tracking. In the future, OHEP will work with the laboratories to improve the template and require all the laboratories to submit their budget information in the same format. Laboratory budgets (both requests and actual costs) have been tracked annually by the DOE budget office to develop a detailed understanding of changes from year-to-year as well as long-term trends. OHEP is planning to store the data in a standardized format and at a more detailed level for tracking and analysis.

Issue 13. With regard to monitoring advanced technology R&D, the COV noted that the documentation is generally adequate, but could be improved with the addition of a summary sheet for each proposal. Also, as suggested for the university program, they recommended that basic information be recorded in a database. Finally, the procedures for carrying out funding actions should be formalized and recorded.

Response and Action: As stated in Issue 8 for the university program, such improvements in documentation organization and format for the advanced technology R&D program are being proposed and discussed and will be implemented by January 2005. With regard to the database, such a system does exist for technology R&D; however, in recent years it has not been kept up-to-date due to a lack of trained personnel. With the recent addition of a program support specialist to the staff, this database has been updated and will be maintained. The formal procedures for carrying out funding actions as described to the Committee will be documented.

Issue 14. The COV felt that, even though the Facilities Operations team is developing new strength and improved practices, OHEP project oversight and management remains principally adaptive and reactive. The COV recommended that program-level strategic planning should provide the basis for OHEP project development and budget planning. The stages of early support, R&D, baselining, project execution, integration, operations, and decommissioning or upgrade should be fully included in OHEP strategic planning. Cradle-to-grave lifecycle costs should be included in planning. The COV went on to suggest that a computer model should be developed to encompass the full lifecycle plan for projects, and this model should include budget and program impacts on laboratory and research programs. The computer model should be maintained and used to develop program options, and it should be fully consistent with models for the University and Laboratory program planning.

Response: Starting in 2003, OHEP has been performing analysis of the "Cradle-to-grave lifecycle costs" in the form of an internal white paper for newly proposed projects and major facilities. This practice is being used as input to program-level strategic planning but is still in its infancy and only conducted for specific cases. Pending approval of additional staff, OHEP will expand this practice program-wide. Also a strawman plan will be formulated to implement the results of the analysis into an integrated computer model.

Issue 15. The COV acknowledged, as a positive development, the OHEP?s initiation of laboratory operations reviews, and pointed out that the use of Dan Lehman?s capabilities is a good way to start this new process at a time when OHEP?s project oversight staff is too small. However, the Committee felt that operations oversight falls within the purview of OHEP and that these reviews should be conducted by OHEP staff. At the same time, the COV acknowledged that additional staff would be required before OHEP would be capable of operating this new review process within the organization.

Action: The Office of Science understands the importance of OHEP ownership of the laboratory operations reviews and also appreciates Dan Lehman?s assistance and guidance to OHEP as they start this new process. OHEP intends to take over these reviews once the process has been established and the staffing shortage has been addressed. As stated earlier, OHEP is in the process of submitting a staffing plan where this need and others are being addressed.

Issue 16. The COV noted that oversight of OHEP major projects involves a line management role provided by a project director at a DOE field office who must be a experienced and skilled project manager. The COV pointed to a proposed DOE certification process to mentor and identify qualified project managers with defined levels of training and experience corresponding to graded levels of project oversight authority. The COV recommended that this certification process be implemented.

Response: The certification process was put into place in late 2003, and it will be implemented by summer 2005.

Outcome of the Program?s Proposal Processes and Program Management Functions

Issue 17. The Office of Science appreciates the COV?s statement that "the current program has consistently produced, and continues to produce, much of the leading research in high energy physics worldwide. The university program of the OHEP is one of the great successes of publicly funded research, and it is impossible to imagine high energy physics without it." The COV, however, raised a general question about the balance of support to large in-house laboratory-based research programs, as compared with university-based research programs and could not assess whether the balance, or the process by which the balance is decided, is appropriate. Therefore, the Committee believed that a review of the "Research Portfolio" is warranted.

Response and Action: OHEP has initiated discussions with laboratory management about a uniform approach for peer reviewing of physics research efforts at the laboratories. We believe that such uniform peer review should lead to improved physics research efforts at the laboratories and universities. Furthermore, the peer review information collected should allow for better management and balancing of the lab and university efforts. Once a uniform peer review mechanism for research proposals from both laboratories and universities is in place, we will conduct a broad review of the overall Research Portfolio, spanning laboratory and university research.

Issue 18. The COV identified as a problem the difficulty investigators have in seeking funding from the agency that is not their traditional source. The COV recommended that DOE and NSF continue to work toward building on the strengths of having two funding agencies, while finding ways to minimize the problems associated with separately funded individuals and research groups.

Response: The university programs of DOE and NSF are extensively coordinated on projects such as Auger, CDMS, and VERITAS, and on any grant funding actions that are of mutual interest. While the two agencies? programs have different missions, approaches and priorities, there is considerable scientific common ground that allows many joint efforts. However, it is important to note that neither agency seeks to support or supplement the support of scientists who are already receiving support for peer-reviewed research from the other agency. Both agencies? university programs are an essential element of the overall HEP program, provide an essential balance for the program, and offer alternative sources of funding for researchers. DOE and NSF will continue to work together to ensure that the very best research is supported.

Opportunities for Proposal Process and Program Management Improvement

Issue 19. The COV was impressed with the laboratory annual program review process and its ability to identify problems at an early stage. No major changes in the review process were recommended; however, the COV did suggest that more timely feedback to the laboratories was needed as well as tracking of the labs? responses. Therefore, the COV recommended that OHEP set an internal deadline of providing the letter report to laboratory management no later than two months after the review is completed. Also, they recommended that OHEP implement a mechanism to follow up on laboratory responses when specific problems have been identified.

Action: In 2003, OHEP established procedures to ensure that the letter reports are prepared and sent to the laboratories within two months of the review, and they will strive to ensure that these deadlines are met. Also, procedures have been put in place to follow-up with the laboratories to receive timely responses from laboratory management.

Issue 20. The COV pointed out that while the laboratory annual program reviews focus on the physics program at the laboratories in an appropriate way, there does not appear to be a mechanism to review the physics research groups of those laboratories. The Committee strongly recommended that OHEP develop such a process, most likely outside the annual program reviews, to provide more uniformity of review between physics research groups in the national labs and those in the university community.

Response: OHEP has initiated discussions with laboratory management about a uniform approach for peer reviewing of physics research efforts at the laboratories. Such uniform peer review, which we hope to have implemented within the next year, should lead to improved physics research efforts at both the laboratories and universities.

Issue 21. For most Project Execution Plans for large projects, significant change control actions are approved by the project director in the local field offices. While this is appropriate, the COV was concerned about a potential communication gap with the program oversight function in OHEP. Therefore, the COV recommended that change control approval by field office staff be carried out in close and prompt communication with the cognizant staff in the OHEP office.

Action: OHEP program managers are in regular and frequent contact with project directors in the field office and very much aware of change control approvals. However, to formalize the process, OHEP, in May 2004 asked that project directors send OHEP program managers a formal notification for the change controls when they approve them.

Issue 22. Major acquisitions/contracts are reviewed by acquisition officials in the DOE Forrestal organization to ensure that the Federal and DOE acquisition process is properly followed. The COV was concerned about the OHEP being in a position to provide programmatic oversight. They recommended that OHEP staff be included in parallel in the DOE acquisition review.

Action: The OHEP is informed of the reviews of the major acquisitions and contracts; however, by the end of summer 2004, they will formalize the request for the notification process with a simple template and a few short instructions.

Issue 23. The COV felt that, since similar project management capabilities are being developed in other offices (e.g., DOE/BES and NSF), lessons learned and best practices should be shared in a timely manner. Therefore, they recommended that OHEP project oversight staff should interact with counterparts in other government organizations to learn techniques, practices and insights.

Response and Action: OHEP agrees with this recommendation. In fact, this activity already takes place in an informal manner across SC with Dan Lehman?s office acting as a conduit for this knowledge. In addition, OHEP will explore ways to coordinate with NSF, NASA, and other pertinent agencies.

Issue 24. The COV pointed out that, since many OHEP projects are hosted and managed by OHEP laboratories, it is essential that laboratory human resources must include a critical mass of project management and project engineering skills in order to sustain the queue of projects over time. The COV suggested that an assessment be made of the existing skills at the HEP laboratories and the results used when planning for future hiring needs.

Response and Action: This is an excellent point and one which OHEP was planning to address. This has been partially addressed as part of the "staffing" portion of the annual budget briefings and program reviews. In addition, for operating facilities and large scale construction projects, OHEP will address this issue as a specific component of future operations or project reviews.

Further Observations and Recommendations

Issue 25. The Committee noted that large experiments receive their funding from many sources and that good communication between OHEP, NSF, and non-U.S. funding agencies is important. Equally important, the COV felt that OHEP should internally optimize the distribution of funding through the different channels it provides. The COV questioned whether support for large experiments is optimized with respect to the division of funding between the national labs and university groups, and whether distribution of funds among different university groups on a given experiment is optimal.

Response: Communication between funding agencies for many projects and R&D efforts is handled on a case-by-case basis between their cognizant program managers, but some of the larger international projects (e.g., GLAST) are now moving toward the model of a "Joint Oversight Group" which DOE and NSF have used for the LHC in both the fabrication and research phases. OHEP recognizes and agrees with the importance of resource planning and balance of distribution. Optimization of OHEP funding to research groups on particular experiments is done routinely in the peer-reviewed process of funding laboratories and universities. Overall optimization of resources designated for particular experiments requires coordination between the management of the experiments and the agencies, and this information feeds into the funding process. This is a question we expect to be addressed in the context of the broad HEP Research Portfolio review.

Issue 26. The Committee commented that both Fermilab and SLAC are active in a number of non-accelerator experiments and noted that their process for obtaining funding for these experiments is not necessarily comparable to that for universities. The Committee believed the process for laboratories is unlikely to address some important questions such as whether laboratory participation is cost-effective in comparison to the same effort being located in a university. The COV stated that OHEP needs to define a process for appropriate consideration of such laboratory initiatives.

Response and Action: Currently, non-accelerator groups at Fermilab and SLAC present their proposals to the Laboratory Program Advisory Committees for laboratory approval. If appropriate, they then present the proposals to SAGENAP (currently a HEPAP subpanel) for community input. If the proposal is sufficiently large (estimated cost of $50M-600M), DOE asks for a recommendation from HEPAP/P5 before making a decision and seeking DOE levels of project approval. OHEP has initiated discussions with laboratory management about implementing a peer review process for the new initiatives proposed by the non-accelerator groups at the laboratories. Such a review process can evaluate and validate the mission justification, cost effectiveness, and the uniqueness of the roles by the laboratory groups.

In addition, all proposals for new research initiatives at HEP laboratories (whether accelerator-based or not) will be considered in the uniform peer review context discussed above, which will allow for comparison between laboratory and university contributions on a more equal footing.

Item 27. The Committee noted that increasingly, major project opportunities involve multi-agency support, and they recommended that the process for identifying, developing, executing, and monitoring interagency projects be better defined. It further recommend that orderly and consistent means of consultation and coordinated review and funding decisions be developed.

Response: As the Committee pointed out, OHEP currently uses many means for accomplishing this coordination. HEPAP is a joint advisory committee, subpanels to HEPAP provide advice on joint planning for the field, SAGENAP is a joint panel for reviewing non-accelerator proposals, and P5 is a joint committee that looks at proposed large projects. In addition, there is a Joint Oversight Group (JOG) for coordinating LHC accelerator and detector projects, and another JOG for coordinating the VERITAS project. Also, there are "Technical, Cost, Schedule, and Management "(TCSM) reviews that are jointly attended. In the future, the combination of JOG and TCSM reviews will probably become the norm for coordination of projects. For smaller projects, there are joint progress and TCSM reviews. In addition, the Associate Director for High Energy Physics regularly communicates with his counterparts at NSF to coordinate the program, and the senior program officer for the University Program interacts frequently with his counterparts at NSF to coordinate the University Program. OHEP is exploring more formal, comprehensive survey/review mechanisms with other funding agencies and will report progress on this effort by the end of CY 2004.

Issue 28. The COV pointed out that the system did not work as well as it should have with regard to recent difficulties with the Tevatron. The Committee stated that the issue needed further attention to ascertain "lessons learned" so that the problems could be avoided in the future.

Response and Action: In summer 2003, OHEP performed its study of the Analysis and Recommendations for the Tevatron Run 2 Program and generated an internal "white paper." A "lessons learned" section was included in this white paper. OHEP will ask Fermilab management to prepare its own "lessons learned" white paper and submit to DOE within the year.