HHS Data Center Optimization Multi-Year Plan 2023-2026

Purpose
Background
Data Center Definition
HHS Goals
The HHS Environment - Changes since 2016 Strategic Plan
Clarification on the Prioritization of Concerns
Quarterly IDC Reporting
IDC Reporting Is Checked for Accuracy
Cost Savings and Cost Avoidance
Data Center Consolidation and Closures
Data Center Optimization
Risk and Risk Mitigation
Cloud Smart Approach
DCOI Moving Forward

1.0 Purpose

The purpose of this Data Center Optimization Initiative (DCOI) Strategic Plan is to provide information on HHS’ Office of the Chief Information Officer’s (OCIO) response to M-19-19 and to provide the current status of HHS’ compliance with Federal-wide DCOI guidance for 2023-2026.

2.0 Background

The Data Center Optimization Initiative (DCOI) supersedes the Federal Data Center Consolidation Initiative (FDCCI) passed in 2010 and is an initiative implemented in accordance with the Federal Information Technology Acquisition Reform Act (FITARA). DCOI was first introduced by Office of Management and Budget (OMB) Memorandum M-16-19, then revised by M-19-19. It establishes requirements for the consolidation and optimization of Federal data centers to promote the use of green Information Technology (IT), reduce the energy and real estate footprint of government data centers, reduce the cost of associated data center hardware, software, and operations, and increase the IT security posture of the Federal Government and shift IT investments to more efficient technologies.

M-19-19, introduced and effective June 25, 2019, reduced the focus from the closure of agency-owned data centers to targeting specific areas of optimization and closures to improve cost savings and modernize IT across the federal government. This requirement included the submission of a quarterly Integrated Data Collection (IDC) to OMB. The requirements of M-19-19 were extended by OMB through Memorandum M-21-05, released November 13, 2020, and effective to September 30, 2022.

In accordance with requirements defined by the Federal Information Technology Acquisition Reform Act (FITARA), beginning in FY2016, agencies shall publish an Annual Strategic Plan to include comprehensive data center inventories, multi-year strategies to consolidate and optimize data centers, performance metrics and a timeline for agency activities, and yearly calculations of investment cost savings. In accordance with FITARA requirements, HHS consistently publishes this information on the HHS Digital Strategy page.

3.0 Data Center Definition

Per M-16-19, any room with at least one server, providing services (whether in production, test, staging, development, or any other environment) is considered a data center. The complete definitions of tiered and non-tiered data centers compared to the FDCCI definition are defined in Table 1, FDCCI Definitions vs. DCOI Definitions found in the HHS Strategic Plan for Data Center Optimization v1.0.

Table 1: FDCCI Definitions vs DCOI Definitions found in HHS Strategic Plan for Data Center Optimization v. 1.0¹

FDCCI Definitions	DCOI Definitions
Tier 1 Basic Capacity: A Tier 1 data center provides dedicated site infrastructure to support information technology beyond an office setting. Tier 1 infrastructure includes a dedicated space for IT systems; an uninterruptible power supply (UPS) to filter power spikes, sags, and momentary outages; dedicated cooling equipment that won’t get shut down at the end of normal office hours; and an engine generator to protect IT functions from extended power outages.	Tiered Tiered data centers are defined as those that utilize each of the following: 1) a separate physical space for IT infrastructure; 2) an uninterruptible power supply; 3) a dedicated cooling system or zone; and 4) a backup power generator for prolonged power outages. Data centers previously classified as tiered in past inventories will automatically be classified as tiered under the DCOI.
Tier 2 (Includes Tier 1 capabilities) Capacity Components: Tier 2 facilities include redundant critical power and cooling components to provide select maintenance opportunities and an increased margin of safety against IT process disruptions that would result from site infrastructure equipment failures. The redundant components include power and cooling equipment such as UPS modules, chillers or pumps, and engine generators.	Tiered Tiered data centers are defined as those that utilize each of the following: 1) a separate physical space for IT infrastructure; 2) an uninterruptible power supply; 3) a dedicated cooling system or zone; and 4) a backup power generator for prolonged power outages. Data centers previously classified as tiered in past inventories will automatically be classified as tiered under the DCOI.
Tier 3 (Includes Tier 1-2 capabilities) Concurrently Maintainable: A Tier 3 data center requires no shutdowns for equipment replacement and maintenance. A redundant delivery path for power and cooling is added to the redundant critical components of Tier 2 so that each and every component needed to support the IT processing environment can be shut down and maintained without impact on the IT operation.	Tiered Tiered data centers are defined as those that utilize each of the following: 1) a separate physical space for IT infrastructure; 2) an uninterruptible power supply; 3) a dedicated cooling system or zone; and 4) a backup power generator for prolonged power outages. Data centers previously classified as tiered in past inventories will automatically be classified as tiered under the DCOI.
Tier 4 (Includes Tier 1-3 capabilities) Fault Tolerance: Tier 4 site infrastructure builds on Tier 3, adding the concept of Fault Tolerance to the site infrastructure topology. Fault Tolerance means that when individual equipment failures or distribution path interruptions occur, the effects of the events are stopped short of the IT operations.	Tiered Tiered data centers are defined as those that utilize each of the following: 1) a separate physical space for IT infrastructure; 2) an uninterruptible power supply; 3) a dedicated cooling system or zone; and 4) a backup power generator for prolonged power outages. Data centers previously classified as tiered in past inventories will automatically be classified as tiered under the DCOI.
Non-Tiered All other data centers shall be considered non-tiered data centers.	Non-Tiered All other data centers shall be considered non-tiered data centers.

The term “tiered” and the definitions that follow are derived from the Uptime Institute’s Tier Classification System; however, this shall not be construed as requiring any certification in order for a data center to be considered tiered by OMB.

Data centers shall be categorized into two groups: tiered data centers and non-tiered data centers. Tiered data centers are defined as those that utilize each of the following: 1) a separate physical space for IT infrastructure; 2) an uninterruptible power supply; 3) a dedicated cooling system or zone; and 4) a backup power generator for prolonged power outages. All other data centers shall be considered non-tiered data centers. Private sector-provided cloud services are not considered data centers for the purposes of this memorandum but must continue to be included in agencies’ quarterly inventory data submissions to OMB. Agencies shall self-classify data centers as either tiered or non-tiered data centers based on the above criteria; however, any data center previously reported to OMB as a Tier 1-4 data center shall be automatically categorized as a tiered data center.

4.0 HHS Goals

HHS’s goals for data center consolidation and optimization are in alignment with DCOI mandates:

Leverage Cloud-based technologies where feasible under budgetary and security constraints
Reduce expenditures on data center operations by optimizing power usage and real estate, automating monitoring and power metering, and implementing best practices
Employ and optimize Green IT to reduce power consumption and cooling needs
Leverage virtualization strategies and technology where applicable and feasible
Reduce costs on infrastructure through efficient, Green IT purchases and internal shared services, where optimal
Employ the most efficient architecture of datacenters that consolidates where appropriate, allows for geographically dispersed datacenters where appropriate, and leverages cloud and distributing computing where appropriate.
Eliminate or minimize security risks for all data centers

5.0 The HHS Environment - Changes since 2016 Strategic Plan

M-19-19 includes changes intended to lessen the burden on agencies. It considers M-16-19 insufficient in demonstrating savings from the consolidation of non-tiered facilities and therefore has adjusted the requirements. It is now optional to include, in inventories, data centers that were deemed invalid by the M-16-19 memorandum. Most HHS Operating Divisions voluntarily removed these data centers from their inventories.

Although, OMB has removed the energy efficiency requirement from reporting, HHS continues to seek ways to leverage Green IT solutions for DCOI and instead report improvements over time in the strategic plan.

Internal Shared Services is one major element of the data center consolidation initiative. For instance, the HHS Office of the Secretary (OS) leads the way by serving as a Key Mission Facility (KMF) for several of the smaller HHS Operating Divisions. These Operating Divisions are consolidated with the OS data center inventory and retain this KMF presence on their own inventory submissions by marking the data centers as part of an aggregate.

HHS continued to comply with M-21-05’s requirements by submitting updates accordingly until the FITARA data center metrics sunset September 30, 2022.

6.0 Clarification on the Prioritization of Concerns

With numerous metrics and possible methods to optimize a data center, agencies could select a variety of solutions and ways to invest in this process. However, not all methods yield the same level of impact. Moreover, after many years of optimizing data centers, many agencies have already created long-term plans for their investments in these areas. OMB prioritizes the effort to consolidate and optimize data centers in the following hierarchy, to maximize the return on investment and based on the level of impact for the effort. When considering the allocation of limited resources or conflicting priorities, agencies should consider their mission goals first, and this prioritization second, in the following descending order of importance:

1) Consolidation and Closure
2) Optimization
i) Virtualization
ii) Availability
iii) Energy Metering
iv) Server Utilization (Underutilized Servers)

7.0 Quarterly IDC Reporting

Since the issuance of M-16-19, the requirements for DCOI reporting changed to de-emphasize focus from closure of data centers. In streamlining the data center inventory reporting for the DCOI IDC, each Operating Division in HHS has been enabled the opportunity to shift focus to additional priority targets and performance metrics. Through improved agency DCOI inventory initiatives, improvements were made to reporting to avoid under-reporting facilities that met the data center criteria.

Specifically, defined as a” purpose-built physically separate and dedicated space; contains one or more racks of servers, mainframes and/or HPCs; has a dedicated uninterruptible power supply and/or backup generator for prolonged power outages; and/or has a dedicated cooling system or zone.” Privately owned data centers do not count toward DCOI metrics. HHS’s Operating Divisions maintain their own repositories for DCOI inventories, and the collective HHS repository is uploaded to and housed in the in the Department’s Microsoft Teams site.

8.0 IDC Reporting Is Checked for Accuracy

The HHS OCIO collects updates about the HHS data center inventory through the integrated quarterly data collection (IDC). The data call is delivered to the HHS Operating Division via email which includes an attached spreadsheet. Each HHS Operating Division individually reviews and approves its data before submission to the HHS OCIO. The consolidated spreadsheet is checked for errors. In the event errors are detected, the HHS Subject Matter Expert (SME) from OCIO will contact the appropriate Operating Division representative to make the appropriate changes. The HHS OCIO SME relies on the Operating Divisions to submit accurate data. The HHS agency DCOI SME hosts integrated project team meetings (IPTs) with the Operating Division SMEs who report DCOI inventory and metrics, as necessary. These meetings are held to identify any changes made to the Quarterly DCOI data call regarding DCOI reporting, discussion of upcoming deadlines, expectations of Operating Division action items, and provides a platform for open discussion of DCOI progress and relevant questions.

9.0 Cost Savings and Cost Avoidance

HHS uses the Capital Planning and Investment Control (CPIC) process to track cost and those savings are collected and reported through the HHS Quarterly IDC. For FY22, HHS had a goal of $12.612M and exceeded the goal by $5.9M, achieving $18.514M. HHS exceeded planned cost savings every year since the start of M-16-19 and has achieved $340.3M in cost savings and avoidance as of August 2023.

While HHS exceeded the cost savings and avoidance targets, HHS Operating Divisions expressed that they do not expect to see continued savings or any further large-scale closures from ongoing data center consolidation and optimization efforts.

HHS publishes quarterly FITARA updates to the HHS Digital Strategy Page which can be found here: Digital Strategy | HHS.gov

10.0 Data Center Consolidation and Closures

Most HHS data centers closed per the requirements of M-16-19, leaving few invalid data centers, and confirming the removal of so-called “low-hanging fruit” described in M-19-19. HHS continues to monitor data centers’ eligibility for closures in each Operating Division. HHS has effectively consolidated data centers, migrated to cloud, and demonstrated continuous cost savings on an annual basis.

During FY2022, HHS’s target value for data center consolidation and closure required that HHS continue to follow planned timelines of closure and migration executions. The closure effort for FY2022 was successfully achieved. From FY2016 – FY2022 HHS closed forty-nine (49) data centers by consolidating data centers, migrating centers to the cloud and outsourcing data centers. For FY22, HHS met 100% of the planned closure goal.

Based on the most current information, HHS operates forty-nine (49) data centers and is inclusive of open invalid and valid facilities, data centers that are not closing or are in migration execution, not using cloud providers, and are agency owned. The total number of valid and invalid facilities decreased by three (3) data centers since FY22 Q4.

As part of the Department’s continuous optimization efforts, HHS plans to close five (5) data centers between FY2024 and FY2026, demonstrating continued adherence to the guidance and recommendations provided by OMB, Congress, and GAO.

HHS is advocating virtualization and cloud migration of data centers. However, there are remaining federal data centers that are vital to the Agency’s operations and will not be closing. These data centers will remain open for the following reasons:

At least one HHS Operating Division has consolidated their data center footprint down to two (2) key facilities that provide 24/7 mission critical support and focus on supporting mission-critical needs that aren’t currently suitable for cloud migration.
Two (2) Department data centers are mission critical locations that hosts the infrastructure and/or serve as the redundancy facility for a few of the smaller Operating Divisions.
The remaining federal data centers are mission critical for several reasons. These data centers support the extensive research and massive amounts of data in genomics, medical imaging, and clinical trials. The speed and low latency needed require on-premises data centers. This research also includes data processing, which requires computational power to process and analyze complex datasets, enabling researchers to perform advanced scientific investigations and gain insights into various health-related phenomena. Additionally, one Operating Division has over 75 buildings in one main campus and several smaller locations across the United States. These data centers provide the backbone for the network, disaster recovery, and connections to external researchers worldwide and are vital to the continuity of the Operating Division’s daily operations.

Several smaller HHS Operating Divisions are already 100% in the cloud with no physical footprint, and other HHS Operating Divisions plan to virtualize or migrate data centers to the cloud when feasible and when resources are available to complete the effort from on-premises to the cloud.

11.0 Data Center Optimization

HHS understands that it may not budget any funds or resources toward initiating a new data center or significantly expanding an existing data center without approval from OMB, unless it’s a KMF. HHS plans to open one (1) data center to serve as an Operating Division KMF. The data to open this KMF is to be determined.

11.1 Virtualization

The virtualization metric is based on the number of virtual hosts. OMB requires agencies to report the number of servers and mainframes that are currently serving as hosts for virtualized or containerized systems in their agency-managed data centers. Several HHS Operating Divisions implemented a cloud strategy and migrated many data centers to virtual hosts rather than physical data center facilities.

In FY2022, HHS met three (3) of the four (4) optimization targets. The virtualization metric is the only target HHS did not achieve for FY2022. After achieving 100% of the virtualization target in 2021, HHS set its goal higher in 2022. HHS’ intention to meet the virtualization target of 1906 was not achieved in FY2022 because some of the HHS Operating Divisions mistakenly over-estimated the number of OMB “valid” virtual host data centers they maintained in their DCOI inventory.

This error occurred due to internal procedural and personnel changes within certain Operating Divisions. HHS took appropriate steps to ensure the errors have since been identified and corrections were implemented by those Operating Divisions. Additionally, HHS will continue to collect updates to the DCOI inventory through an Internal Quarterly Integrated Data Call to ensure data center optimization efforts continue. Through this process, the Department will be in position to review and monitor the progress of data center inventories which allows HHS to be able to better support the Office of Management and Budget’s objectives for the data center virtualization objective.

HHS realizes the benefits of data center optimization and will continue to use strategies to prioritize virtualization and cloud migration as resources become available.

11.2 Advanced Energy Metering

Facilities with energy metering will be tracked by measuring the number of facilities having advanced energy metering. OMB prioritizes the increased virtualization of Federal systems as critical for IT modernization efforts, to drive efficiency and application portability. In line with OMB requirements, HHS expects all new hosted applications to use virtualization whenever possible and appropriate.

The energy metering metric is based on the number of data centers with metering. In FY2022, HHS achieved the metering goal. In FY2022, twenty-five (25) of the planned twenty-six (26) data centers were metered; one (1) of the data centers closed.

HHS will continue to monitor the advanced energy metering to improve energy efficiency of its facilities when cost-effective to do so.

11.3 Server Utilization (Underutilized Servers)

The metric defined for Server Utilization in M-16-19 set a baseline for utilization as a calculated average, which did not help in finding and removing underutilized servers. Moreover, this was measured as a combined metric with “automated monitoring,” making the target more difficult for agencies to achieve.

HHS faces challenges with the server utilization metric because “server efficiency” is a relative term, depending on the nature of the hardware and applications running on the server. HHS Operating Divisions have established their own interpretation of server utilization due to agency-specific factors. Some Operating Divisions have shifted to cloud entirely, while others still rely on physical data center presence in addition to ongoing virtualization efforts. Data center closures are an important component of M-19-19, but some Operating Divisions face challenges with continuing to close data centers.

Since FY2019, the HHS DCOI strategy has demonstrated a steady decrease in the number of underutilized servers across the department. Per M-19-19, underutilized server definitions are now the responsibility of each federal department, as they are no longer prescribed entirely by OMB. The operating divisions have unique requirements; therefore, the practice of the agency is to rely on the operating divisions’ mission goals as described by the operating divisions themselves as well as industry best practices.

This revealed there is not a consensus on interpretation: underutilization is often indicated based on processors, memory, and disk utilization, including defining an underutilized server as one with an average utilization less than 75% of which it was designed. HHS has maintained an underutilized metric of no more than 15%. OS measures server utilization and availability with a variety of tools known across the industry, including Orion Solar Winds, VRealize, APC StruxureWare, XDMod, Ganglia, and CheckMK. In FY2022, HHS had 3.8% agency-owned underutilized servers meeting its internal target.

11.4 Availability

This metric tracks the data center’s availability, not individual server, or application uptime or availability. OMB requires agencies to report the planned hours of availability for each data center, as well as any unplanned outages for that data center over the reporting period, also measured in hours.

Unavailability includes unplanned outages of most of the facility due to disaster, systems failure, cybersecurity events, or other negative events as well as any other hours that would otherwise be planned as available hours. In FY2022, HHS met the availability target for agency-owned tiered data centers.

12.0 Risk and Risk Mitigation

Risk is assessed in different areas including mission impact, criticality, and visibility by using an Inherent Risk Assessment Score and their IT Investment Control Reviews.

HHS takes into consideration a couple major risks to cloud-based solutions and on-premises virtualization. Some of the risks associated with cloud-based solutions include interruptions or failures in network access can cause loss in service, problems with cloud vendor security practices can result in security vulnerabilities. Mitigation strategies include spreading data across multiple data centers to minimize impact of network interruptions, disaster recovery, and FISMA certification to confirm appropriate implementation of security practices.

One of the risks associated with on-premises virtualization is that many servers are on a central system resulting in the possibility that a failure single virtualized hosting system. This failure could negatively impact a high-volume of servers. The mitigation plan for this is to setup backups, disaster recovery, and virtual hardware/software resiliency so that single failures will not result in an outage.

13.0 Cloud Smart Approach

HHS is committed to adopting cloud solutions that streamline transformation and embrace modern capabilities in line with the Federal Cloud Smart strategy.

HHS has successfully closed many data centers by migrating to the cloud. This strategy continues to evolve with industry trends, and the progress made so far serves as a basis for continuing technology collaboration across the agency.

In a further attempt to streamline transformation and embrace modern capabilities, HHS will leverage the Nonrecurring Expenses Fund (NEF) to help the agency advance DCOI efforts, when available. Further benefits include moving HHS data centers to cloud services, improving energy efficiency and continuous virtualization of data centers’ network infrastructure.

14.0 DCOI Moving Forward

HHS repeatedly demonstrates a dedication to virtualization, keeping up with industry standards, and improving HHS’s cybersecurity posture. However, costs of cloud computing operation are a concern. Several Operating Divisions relayed that the current funding models do not support utilization-based services and the flexibility needed to expand cloud and subscription-based services. Aside from the cost of operating and maintenance of legacy systems, a significant investment into staffing Cloud engineers and training current federal employees is also required. There are also competing priorities including Zero Trust mandates.

A continuous combination of efforts from all Operating Divisions in a highly federated Department such as HHS is critical to achieve both the agency’s mission and DCOI objectives. Ongoing collaboration is a challenge for future goals given competing objectives.

HHS is dedicated to its focus on targeted improvements in key areas where the Department can make meaningful improvements and or achieve further cost savings. The HHS CIO will continue to communicate the importance of data center optimization efforts to Operating Divisions and Staff Divisions. For 2023 and beyond, HHS will continue advancing the Agency’s implementation of DCOI through strategies and DCOI IDC reporting. Through this process, the Department will be in position to review and monitor the progress of data center inventories which allows HHS to be able to meet the Office of Management and Budget’s objectives for the Data Center Optimization Initiative.

Endnotes

¹ From OMB Memorandum 16-19, The term “tiered” and the definitions that follow are derived from the Uptime Institute’s Tier Classification System; however, this shall not be construed as requiring any certification in order for a data center to be considered tiered by OMB.