High-performance computing (HPC) uses parallel data processing to provide the fastest possible computing performance. Whether it’s a supercomputer like the Exabyte Fast Frontier HPE Cray Supercomputer or a cluster of hundreds or thousands of servers, it’s all about taking complex tasks and running them on machines as fast as possible.
now, The Linux FoundationA leading open-source innovation company, announced two new foundations for advancing HPC and storage solutions: High-Performance Software Foundation (HPSF) and DAOS Foundation.
Also: How to Choose the Right Linux Desktop Distribution for You
HPSF’s mission is to develop, promote, and advance a portable core software stack for high-performance computing. The stack consists of 10 different programs. These are:
-
spac: HPC Package Manager
-
Cocos: A Performance-Portable Programming Model for Writing Modern C++ Applications in a Hardware-Agnostic Way.
-
AMREX: A performance-portable software framework designed to rapidly solve partial differential equations on block-structured, adaptively refined meshes.
-
WarpX: A performance-portable particle-in-cell code with improved algorithms that won the 2022 Gordon Bell Prize
-
Trillinos: A collection of reusable scientific software libraries, particularly known for linear, non-linear, and transient solvers, as well as optimization and uncertainty quantification.
-
Apptainer: A container system and image format specifically designed for secure, high-performance computing.
-
VTK-Mr: A Toolkit of Scientific Visualization Algorithms for Accelerator Architectures.
-
HPCToolkit: Computer performance measurement and analysis tools from laptops to the world’s largest GPU-accelerated supercomputers.
-
E4S: Extreme scientific software stack
-
Charliecloud: An HPC-friendly, lightweight, fully featureless container implementation.
What do these programs have in common? They’re all open source, which only makes sense. Over the years, Linux and open source computing have powered all TOP500 supercomputers
All of these are designed to make life easier for high-performance software developers With them, programmers can achieve continuous integration, turnkey software stacks and performance testing. It will make their life much easier.
This new foundation addresses the increasing ubiquity of HPC in scientific computing, digital engineering and AI applications. Plan to leverage investment from foundations US Department of Energy’s Exascale Computing ProjectThe EuroHPC Joint Undertakingand other international projects to improve performance across different computing architectures.
Also: Linux over 3% of desktop market? It’s more complicated than that
Key supporters of HPSF include Amazon Web Services, Argonne National Laboratory, CEA, CIQ, Hewlett Packard Enterprise, Intel, and NVIDIA, among others. The Foundation will form a Technical Advisory Committee to conduct working groups on various HPC topics, following a governance model inspired by Cloud Native Computing Foundation (CNCF).
In short, Rick Stevens, Argonne National LaboratoryIts associate laboratory director for computing, environment and life sciences, explained, “HPSF will provide a collaborative environment and access to resources that will enable the community to collectively develop the software needed to run current and future generations of HPC systems, and Argonne is joining as a founding member. Honored to do so.”
The DAOS Foundation focuses on governance and development Distributed Asynchronous Object Storage (DAOS) Coined with Project “Boss”, DAOS is a high-performance open-source storage system that has revolutionized performance for HPC and AI workloads. It offers a fully distributed key-standard storage architecture that overcomes the limitations of traditional POSIX-based HPC storage systems.
Also: AI pioneer Cerebras is having ‘a monster year’ in hybrid AI computing
DAOS supports high bandwidth, low latency, and high I/O operations per second (IOPS). Again the name of the game is speed, speed, speed. It is a mature project that Intel started working on in 2012. Intel will donate the DAOS source code to the Foundation and will continue as a major contributor.
Like HPFS, DAOS brings support for a laundry list of HPC movers and shakers, such as Argonne National Laboratory, Enakta Labs, Google Cloud, Hewlett Packard Enterprise, and Intel.
Under the Linux Foundation umbrella, these two foundations represent a major leap in collaborative efforts toward the advancement of open-source solutions in high-performance computing and storage. They will encourage innovation, lower barriers to entry and support the development of critical technologies in HPC and AI/ML.