SOW 2024-v05.pdf

Attachment A – Statement of Work May 2024, Rev 5.

Statement of Work Open-Source Compiler Capability and Performance Enhancements, May 2024

1. Scope

1.1 Introduction

The Oak Ridge Leadership Computing Facility (OLCF) at Oak Ridge National Laboratory (ORNL), UT- Battelle (UT-B) or Company) fields some of the most powerful supercomputers in the world in support of the U.S. Department of Energy's (DOE) scientific mission. Recent and current-generation supercomputers (Titan, Summit, and Frontier) all utilize GPU accelerators to deliver this computational capability within an affordable power envelope. Additionally, other U.S. Department of Energy laboratories are also deploying supercomputers based on GPU accelerators (e.g., Perlmutter at Lawrence Berkeley National Laboratory, Aurora at Argonne National Laboratory, and El Capitan at Lawrence Livermore National Laboratory).

The OLCF has strongly supported and promoted directive-based approaches for GPU programming in order to facilitate the portability of user code between systems with and without such accelerators, specifically, the OpenACC and OpenMP languages, which are layered on C, C++, and Fortran. The OLCF also finds it useful to offer multiple implementations (compilers) for each of the languages it makes available.

The GNU Compiler Collection (GCC) and the LLVM Compiler Infrastructure are both open-source compiler suites which are widely used by OLCF users, and DOE users more broadly, on DOE and non-DOE platforms. These compilers, and in the case of LLVM, derivatives enhanced by the vendors, are frequently included in the programming environment provided by supercomputer vendors and are also widely deployed on other platforms. Both compiler suites support the C, C++, and Fortran programming languages. Both have some support for OpenACC1 and OpenMP, but there are capability and performance gaps, particularly on the GPU accelerated platforms of interest to DOE (currently AMD, Intel, and NVIDIA). In order for these compilers to meet the needs of DOE users, they need to provide both the capabilities and performance that ensures they constitute viable tools for users needing those languages.

The primary purpose of this Statement of Work (SOW) is to improve and enhance these compiler implementations to support the latest versions of the various base language (C, C++, Fortran) and accelerator programming languages (OpenACC, OpenMP) to the latest versions of the standards, so that the compilers meet the needs of DOE users, including suitable performance, particularly on the GPU platforms of interest to DOE users. As new versions of the various language standards are released, we may specify additional tasks to be defined under the Basic Ordering Agreement to update the compilers.

ORNL expects that the GCC compiler will be the primary focus of work under this Basic Ordering Agreement (BOA) but prefers a Seller capable of working in LLVM as well, should the need arise.

Similarly, we expect that GPU accelerators from AMD, Intel, and NVIDIA will be the primary focus of work under this BOA, it is possible that new accelerator hardware will become important during the lifetime of the BOA. ORNL may elect to carry out some or all of this work in collaboration with other

1 For LLVM, OpenACC support has been largely developed by the DOE Exascale Computing Project and has not yet been upstreamed to the main LLVM repository, but it is available.

organization, within or outside of DOE in order to share the costs of the developments. ORNL desires that to the maximum extent possible, enhancements under this BOA should be upstreamed to the main repositories of the compilers rather than having to be maintained as separate patches.

Performance of the code generated by the compilers is important to DOE users and to ORNL. The Seller is expected to consider the performance potential for their approach to implementing all tasks under this BOA, even if they are primarily focused on capability. Task statements of work should include a written design document and implementation plan, which addresses performance, maintainability, potential concerns about the work being accepted by the GCC maintainers, and other considerations, as an early deliverable, to be reviewed and approved by the ORNL technical project officer (TPO).

Exceptions to this requirement may be agreed between the Seller and the TPO on a case-by-case basis.

The expectation is that work under this BOA will be completed as quickly as reasonably possible. Where specific deadlines are required, they will be included in the specification of the tasks.

1.2 Work Description

The following tasks are examples of the types of tasks expected to be issued under the BOA, and Seller’s responses will be used in the evaluation of proposals. Additional tasks may be issued throughout the lifetime of the BOA. When there are multiple tasks active, the TPO will provide guidance on their prioritization.

1. The OpenMP implementation in GCC will be extended to support version 5.2 of the standard for both NVIDIA and AMD targets.

2. The OpenMP implementation in GCC will be extended to support version 6.0 of the standard once it is published (expected in November 2024) for both NVIDIA and AMD targets. For purposes evaluation, Seller should base their bid on OpenMP Technical Report 11, as a preview version of 6.0. The Seller will have an opportunity to update their bid after version 6.0 is finalized and published.

3. The OpenMP implementation in GCC will be extended to support the OMPT tracing interface for both NVIDIA and AMD targets. For purposes evaluation, Seller should base their bid on OpenMP Technical Report 12, as a preview version of 6.0. The Seller will have an opportunity to update their bid after version 6.0 is finalized and published.

4. The following Fortran 2023, as defined in the ISO/IEC 1539:2023 standard, features will be implemented:

a. Interoperability with C: c_f_pointer with lower bound (per section 18.2.3.3 of the standard; see also UK01 in ISO/IECJTC1/SC22/WG5 N2212) and Fortran/C string conversion (per section 18.2 of the standard, particularly 18.2.3.5 and 18.2.3.9; see also US09 in ISO/IECJTC1/SC22/WG5 N2212)

b. Additional named constants to specify kinds in the ISO_FORTRAN_ENV module:

LOGICAL8, LOGICAL16, LOGICAL32, LOGICAL64, and REAL16 (per section 16.10.2 of the standard, particularly 16.10.2.21 and 16.10.2.27; see also US07 & US08 in

ISO/IECJTC1/SC22/WG5 N2212)

5. The performance of the GCC compiler using OpenMP and OpenACC offloading will be improved to achieve at least 50% of the performance achieved by freely available vendor compilers and the LLVM compiler suite for the SPEChpc benchmark suite for NVIDIA and AMD targets.

https://wg5-fortran.org/N2201-N2250/N2212.pdf https://wg5-fortran.org/N2201-N2250/N2212.pdf https://wg5-fortran.org/N2201-N2250/N2212.pdf

a. The Seller will propose a general methodology for performance improvement tasks that will be pursued through the BOA. The methodology should describe how performance improvement tasks will be identified and statements of work defined, progress monitored, and how results will be assessed, and should involve both the Seller and the TPO.

b. The Seller will propose a specific activity to improve GPU kernel launch times in the gcc compiler suite. The proposal should outline the technical strategy and discuss the degree of performance improvement expected. The proposal should be consistent with the general performance improvement methodology proposed above.

6. General support and maintenance will be provided for GCC (bug fixes, performance improvements, etc.) at the direction of the TPO. At a minimum, support and maintenance will be provided 8 hours per day, 5 days per week excluding holidays.

These activities are expected to be primarily in the newly implemented capabilities but may be in any portion of the GCC suite. For example, a bug in a Fortran feature not part of this SOW may adversely impact OLCF users, making it a requirement that it be addressed promptly under this Task.

2. Applicable Code, Procedures, and References Information about the Oak Ridge Leadership Computing Facility (OLCF) and the Titan, Summit, and Frontier systems can be found at https://www.olcf.ornl.gov/

Information about the National Energy Research Scientific Computing Center (NERSC) Perlmutter system can be found at https://www.nersc.gov/systems/perlmutter/

Information about the Argonne Leadership Computing Facility (ALCF) Aurora system can be found at https://www.alcf.anl.gov/aurora

Information about the El Capitan system at Lawrence Livermore National Laboratory can be found at https://www.llnl.gov/news/llnl-scientists-eagerly-anticipate-el-capitans-potential-impact

Information about the GNU Compiler Collection (GCC) can be found at https://gcc.gnu.org/

Information about the LLVM Compiler Infrastructure can be found at https://llvm.org/

Information and the current specification for OpenACC can be found at https://www.openacc.org/

Information and specifications for OpenMP can be found at https://www.openmp.org/

References to “Fortran 2023” mean the ISO/IEC 1539:2023 standard, or equivalently, the J3/24-007 Fortran 2023 Interpretation Document. Document ISO/IECJTC1/SC22/WG5 N2212 concisely summarizes the new features of Fortran 2023, and labels from this document are also noted for the features requested to be bid.

The SPEChpc 2021benchmarks (v 1.1 and later, https://www.spec.org/hpc2021/) and the SPEC ACCEL benchmarks (v 1.3 and later, https://www.spec.org/accel/) will be used to help evaluate the performance and correctness of the OpenACC and OpenMP implementations.

https://www.olcf.ornl.gov/ https://www.nersc.gov/systems/perlmutter/ https://www.alcf.anl.gov/aurora https://gcc.gnu.org/ https://llvm.org/ https://j3-fortran.org/doc/year/24/24-007.pdf https://wg5-fortran.org/N2201-N2250/N2212.pdf https://www.spec.org/hpc2021/ https://www.spec.org/accel/

The SOLLVE V&V Suite (under development) will be used to help evaluate the correctness of the OpenMP implementation: https://bitbucket.org/crpl_cisc/sollve_vv.

The OpenACC V&V Suite (under development) will be used to help evaluate the correctness of the OpenACC implementation: https://github.com/OpenACCUserGroup/OpenACCV-V

3. Technical and Functional Requirements The SOLLVE V&V toolkit and benchmarks from the SPEC High-Performance Group will be used to help assess the capabilities developed under this subcontract. However, these and other codes of interest to OLCF (including user codes) are generally under continual development, and there is no single code base that represents a definitive test of the capabilities in this SOW.

Performance of the generated executables will be comparable to that obtained with other freely available compilers on the same hardware.

Decisions as to acceptance of the deliverables are ultimately the sole responsibility of the Company's Technical Project Officer (TPO).

4. Responsibilities Company will work with Seller to facilitate the establishment of any required nondisclosure agreements required to support forthcoming hardware devices (GPUs, CPUs, etc.).

Company will work with Seller to obtain any additional computer hardware that may be required by Seller to support development and testing under this SOW. Seller shall obtain hardware from appropriate vendors by way of loan or purchase. Company's preference is that Seller pursue loaned hardware. Where a loan is not feasible, Company's TPO will approve the Seller's purchase in writing before the purchase occurs. Company may also pursue such a loan or agree to purchase the hardware directly from the vendor. Computer hardware purchased under this subcontract will be property of the Company and shall be returned to the Company upon completion of the subcontract.

Seller and Company staff will hold regular teleconferences to discuss the status of the project. The Company holds at least two (2) programmatic meetings yearly. At least one member of the Seller's senior staff that can discuss the work shall plan on supporting Company's yearly programmatic meetings. Travel may be required.

Company will work with Seller to identify and obtain codes suitable for evaluating the performance and correctness of work under this subcontract. This will include test suites, benchmarks, and user codes.

Access to some codes (particularly SPEC HPG benchmarks) may require the purchase of a license or organizational membership.

5. Deliverables/Submittals Deliverables will be in the form of changes and additions to the appropriate compiler suite, including appropriate documentation, made available under the corresponding open-source license. Deliverables are to be timely upstreamed to the maximum extent possible. Until deliverables are accepted upstream, and for any changes that are refused upstream, Seller will maintain a public repository and/or download site which makes available the complete version of the compiler suite including all contributions under this SOW integrated with the most recent release version of the compiler to make the results of the work available to the OLCF and other interested parties.