Integrating Linux Clusters into the GridIan Lumb and Chris Smith Linux clustering is pervasive. Next to the attractive price/performance of COTS components, smart system software plays a key role in this pervasiveness. In the context of clustering, it is smart system software that allows a number of distinct systems to appear as one -- even though each runs its own instance of the Linux operating system. Figure 1 illustrates the possibilities. At one extreme, the single-system environment (SSE) is smart system software that runs in user space as a layered service. Often referred to as middleware, there exist a number of open source and commercial implementations of SSE solutions. At the other end, the single-system image (SSI) is smart system software that spreads operating-system functions across systems and involves modification of the Linux kernel. Such tightly coupled integrations permit global process spaces (i.e., PIDs that span separate instances of the Linux operating system, such as Beowulf BPROC), use of algorithms for preemptive process migration (e.g., the MOSIX management package), etc. Though presented as extremes, examples of SSI-SSE integration do exist. To varying degrees, these solutions enable computing for capacity (i.e., throughput of serial, parametric, and embarrassingly parallel applications) and/or capability (i.e., multithreaded and distributed-memory parallel applications). However, our purpose in this article is not to discuss Linux clustering in detail. Rather, it is make a simple observation: Use of smart system software allows distinct instances of the Linux operating system to be virtualized as a cluster; a natural extension allows clusters to be virtualized into grids. In the next section, we define "grid computing" and follow this definition with examples of enterprise and partner grids. After providing an overview of the exciting convergence between the grid and Web services in the Open Grid Services Architecture (OGSA), we close with a summary and some recommendations plus resources for further investigation. Grid Computing Much like the Web, grid computing originated in the research community to facilitate collaboration for "Big Science", such as sharing terascale volumes of data from high-energy physics (HEP) experiments between hundreds of globally distributed scientists, aggregating hundreds of CPUs to perform "grand challenge" computations. Because the private sector shares a common interest in high performance computing (HPC), grid computing is seeing early adoption in the commercial sector as well. (See Resources section for additional background and examples of grid computing.) With adoption in its earliest phases, awareness of grid computing is evident, but an understanding is often unclear. To fix ideas, we adopt a three-point grid checklist (see Resources): "... a Grid is a system that (1) coordinates resources that are not subject to centralized control using (2) standard, open, general-purpose protocols and interfaces to (3) deliver nontrivial qualities of service". Each of these technical points requires elaboration:
1. Coordinates resources that are not subject to centralized control. Because geographic distribution of people and resources is common, coordination between multiple departments within a single organization or multiple organizations may be necessary. In some cases, cooperation among organizations exists only for a finite period of time, so the term "virtual organizations" is often used. Trust relationships and connectivity are examples of concerns between cooperating parties. 2. Uses standard, open, general-purpose protocols and interfaces. An increasingly real vision at the present time, the emerging Open Grid Services Architecture (OGSA) holds the promise for refactoring existing and developing new technology around open standards. 3. Delivers nontrivial qualities of service (QoS). These qualities of service are often combined into Service Level Agreements (SLAs) or policies. In the grid computing context, QoS translates business objectives into objectives for the IT infrastructure, thus enabling effective utilization, resource aggregation, and remote access to specialized resources.
There are two noteworthy consequences of this three-point grid checklist. First, Linux clusters are not grids. Even though they may have grid-like attributes, Linux clusters fail to satisfy the checklist's first point (i.e., they are centrally controlled but not distributed geographically). Second, grid computing represents the next phase in the evolution of distributed computing. In the next two sections, we illustrate this evolution in terms of enterprise and partner grids. Enterprise Grids In this section and the next, we assume that SSE smart system software (e.g., Platform LSF) has virtualized a number of distinct systems into an HPC cluster -- as before, each system is running its own instance of the Linux operating system. Two or more clusters can be transformed into an enterprise grid with Platform MultiCluster. An overview of the transformation process follows:
Platform MultiCluster facilitates collaboration at the enterprise level without loss of local autonomy; to implement such a solution, inter-departmental discussions are required to complement the technical efforts. Customers use this enabling technology in their production deployments across a variety of industries (e.g., semiconductor design, industrial manufacturing, government and education, bioinformatics, computational chemistry, petroleum exploration, financial services, etc.). In practice, this combination supports a variety of submission-execution topologies (Figure 2). With the exception of the cluster case (Single Submission, Single Execution), real-world implementations of these topologies exist. From this topological consideration, it is clear that enterprise grids are a cluster of clusters. Platform MultiCluster fits the three-point grid checklist with the following qualifications on the first two points:
1. A single organization is involved -- or multiple organizations operating as one. This simplification means that the organizational firewall can serve as the primary means of security. 2. Because this technology predates the still emerging open standards, proprietary protocols and interfaces remain in use today.
Non-trivial QoS, through enterprise-wide scheduling, is enabled by Platform MultiCluster. Partner Grids Again, we assume that SSE smart system software has virtualized a number of distinct systems into an HPC cluster -- as before, each system is running its own instance of the Linux operating system. Additionally, multiple (virtual) organizations cause the transition from enterprise to partner grids. In the partner grid case, the enterprise firewall is meaningless, and the need for resource discovery arises. The Globus Toolkit addresses these extra-enterprise tensions in security and discovery. Figure 3 provides a functional overview of the toolkit:
Figure 4 illustrates how the components of the Globus Toolkit might be deployed both with and without a cluster-level workload manager. Again, the natural affinity for grid use is evident in environments that have already been virtualized through SSE smart system software. The Globus Project (a research consortium lead by the Argonne National Laboratory (Chicago, IL) and the Information Sciences Institute of the University of Southern California (Marina del Rey, CA)) makes the toolkit available via a liberal open source license called the Globus Toolkit Public License (GTPL). The GTPL permits the following distributions of the toolkit:
Although the specifics do vary from distribution to distribution, a generic overview of the installation and configuration process will include:
Once installed, correct operation of each component can be determined. The Globus Project provides certificates through a publicly accessible Certificate Authority (CA). This means that those who are keen to experiment with the toolkit do not need to set up their own CA at the outset. To this point, we have referenced version 2.x of the Globus Toolkit; as of this writing, version 2.4 is the current production release. Version 2.x is used in a number of grid projects (see Resources). Overall, the toolkit complies well with the three-point grid checklist. However, against the final point, regarding non-trivial QoS, there is much opportunity for improvement. The Open Grid Services Architecture Each of the components in version 2.x of the Globus Toolkit is based directly on a protocol (Figure 5). Although this was a pragmatic and understandable decision at the outset, these underpinnings started to increasingly limit the ability to develop on top of the toolkit. Starting in late 2001, and based on these concerns of modularity and extensibility, the Globus Project sought to refactor the toolkit. Around the same time, IBM Research was investigating autonomic computing -- computer systems that regulate themselves much in the same way our autonomic nervous system regulates and protects our bodies. The cross-fertilization between the Globus Project and IBM Research lead to the OGSA. OGSA is the consequence of the convergence between grid computing and Web services. Based on experience with the Globus Toolkit, a number of functional components (e.g., resource and allocation management, data management, directory services) and common services (e.g., security) are identifiable. From Web services, the ability to leverage SOAP (Simple Object Access Protocol), WSDL (Web Services Description Language), and other capabilities is clearly appealing. This fusion has already resulted in a significant outcome -- an emerging standard and implementation of the Open Grid Services Infrastructure (OGSI, Figure 6). Built on top of Web services (particularly WSDL), the Grid Service Specification is an enhancement that takes grid computing into account through additions like persistence, lifetime management, etc. The current version of this specification is passing through the standards-approval process of the Global Grid Forum (GGF). The GGF is a community-initiated forum that serves as the authoritative body in the standards process. The Grid Service Specification is a deliverable of the OGSI Working Group within the GGF. Originally released in mid-January of 2003, version 3 of the Globus Toolkit (GT3) includes an implementation of the OGSI. As of this writing, version 3 of the toolkit is in a beta release, and is expected to enter production status later this year. The OGSI implementation in GT3 is typically used in a Java 2 Enterprise Edition (J2EE) hosting environment, though other hosting environments (e.g., Microsoft .NET) are emerging. The hosting environment allows all resources (the lowest layer in Figure 6), including Linux clusters, to be virtualized for grid use. From the bottom up, these first three layers (Figure 6) are tangible. Current efforts in GGF Working Groups seek to co-evolve standards and implementations for core OGSA services and policies. The final three layers (at the top of Figure 6) place user interfaces, applications and an application-enabling API in this OGSA context; these layers are also evolving. Although the transition to OGSA will be evolutionary, this is a revolutionary change to a service-oriented architecture. Summary Linux clusters are predisposed towards the grid. The established practice is to virtualize these clusters through SSE smart system software. Enterprise and partner grids provide examples of early adoption based around established products (Platform MultiCluster) and toolkits (Globus Toolkit, version 2), respectively. We have also introduced the exciting convergence of grid computing and Web services. This approach provides a modular and extensible foundation upon which grid standards and implementations are starting to co-evolve. Understandably, this is a time of significant change. Existing technologies are being refactored under OGSA, and new technologies are starting to emerge. Getting involved will depend on a number of factors, such as exploratory investigation versus production implementation, testbed versus enterprise versus cross-organizational deployment, readiness for a service-oriented approach, etc. Again, the experiences of those familiar with Linux clustering are transferable to the grid. However, out-of-the-cluster thinking is needed to identify and address the requirements on a broader scale of collaboration. Acknowledgements The authors acknowledge Carla Lotito of Platform for providing Figure 4. Resources Autonomic Computing -- http://www.research.ibm.com/autonomic Commodity Grid Kits -- http://www-unix.globus.org/cog Foster, I., "What is The Grid? A Three Point Checklist", GRIDtoday, 1(6), July 22, 2002. Available online at http://www.gridtoday.com/02/0722/100136.html. Foster, I., "The Grid: Computing without Bounds", Scientific American, April 2003. The Globus Project -- http://www.globus.org The Global Grid Forum (GGF) -- http://www.ggf.org Grid Portal Toolkit -- https://gridport.npaci.edu/ MOSIX -- http://www.mosix.org Platform Computing -- http://www.platform.com The Open Grid Services Architecture (OGSA) -- http://www.globus.org/ogsa Ian Lumb has been with Platform Computing Inc. for 5 years, starting in training, and then business development, before taking his current role as a Systems Engineer focused on Grid Computing solutions for Government, Education and the Life Sciences. He has an M.Sc. in Earth and Atmospheric Science from York University, and his interests include High Performance Computing (HPC) for scientific insight. Chris Smith has been with Platform Computing Inc. for 6 years, starting in the development organization, moving to his current role of Integration Architect focused on Grid Computing solutions in Life Science and Government. He has a B.Sc. in Computer Science from the University of British Columbia, and his interests include distributed computing, parallel programming, operating systems and communication protocols. |