Best Practice Designs for High Availability and Disaster Recovery

by | January, 2026 | architecture, availability, data, data recovery, high availability, recovery, recovery point objective, recovery time objective, rpo, rto, systems, unix

HA and DR with 2-Site Architectures

Designing Your IT Infrastructure for High Availability and Disaster Recovery and Best Practices

Introduction

The purpose of this document is to discuss some possible multi-site (or two-site) IT infrastructure designs for high availability and summarise benefits and drawbacks to address business uptime requirements. We will focus on some best practice considerations in order to achieve better application or services high-availability and disaster recovery to meet your business needs.

Remember that high availability applies to all layers on the information stack. That means that you start at the bottom physical tier by addressing things such as multipathing network and storage, mirroring disks, redundant operating systems (more on that in another post when we discuss Clustering in more detail) and eventually the applications and the core data or database.

For the purposes of keeping this document short, we will focus on the infrastructure layers.

You cannot have a document about high availability and disaster recovery, without addressing business continuity (BC), recovery time objectives (RTO) and recovery point objectives (RPO). These are business objectives that should drive the design of the architecture.

General High Availability and Disaster Recovery Concepts

Business Continuity (BC) is the business objective that deals with a company’s ability to continue critical business functions, in the face of unforeseen events.

Disaster Recovery (DR) refers to specific tasks undertaken in the event of a loss by an organisation.

Business Impact Analysis (BIA) identifies and prioritises the minimum enterprise business continuity requirements to stay in business at certain levels of disruption.

A disaster is a sudden and unplanned event that causes damage or loss.

Disasters can be internal and external to a business. Examples of disasters:

  • Technical: hardware failures, facility failures (H/W and S/W single points of failure)
  • Natural: earth quakes, floods, fires, etc
  • Man-made: accidents, mistakes, malicious

The Recovery Time Objective (RTO) for an application is how quickly you need to have that application running again after an outage has occurred.

The Recovery Point Objective (RPO) for an application is the point in time that data must be successfully saved and can continue processing from.

When discussing RTO and RPO requirements with any organisation, the requirement always starts with “we cannot afford any down time at all” and “we must be up all the time”.

Possible? Maybe.

Only a business owner can determine the areas (applications) of risk, the RTO/RPO and cost to protect against them. The biggest surprise then becomes when you explain that as RTO and RPO approach zero (0), the costs to implement such as architecture increase (nearly) exponentially. So trade-offs are accepted.

Architecture Methodologies

Solution architectures needs to take into account the RTO, RPO objectives and the financial impact to provide the cost of recovery.

Here is a list of rules for best High Availability and Disaster Recovery Practices:

  • no single points of failure (hardware or software)
  • all hardware must have good reliability, availability and serviceability (RAS) features
  • requirement of 2 or more geographically-separated sites
  • all server hardware must be clustered
  • all storage must be protected by RAID
  • all data duplicated by log, mirroring, duplication
  • data mirroring or replication must be done synchronously
  • all hardware and software configured with strict security practices
  • all hardware and software under strict change control practices
  • all equipment located in a secure facility

Other 2-site considerations:

  • active-active vs active-passive (per application)
  • backup and restore design
  • system provisioning and implementation
  • system management, monitoring, logging, etc
  • production and development and testing environments

Business Requirements

When designing any system architecture you need to start at the top. The business requirements drive the applications to use. The applications drive the operating system that supports it. The operating system drives the supported server platform.

And on top of that you need to address all the “*ilities” ie scalability, serviceability, etc and not just the high availability needs.

For our hypothetical customer, we will use the following RTO and RPO requirements:

  • Recovery Time Objective (RTO):
    • Maximum=4hrs.
    • Preferred=90min
  • Recovery Point Objective (RPO):
    • Worst Case=60sec.
    • Preferred=Nil or very-near nil data loss

There will be many possible constraints when designing any system, especially when the architecture needs to address all layers of the information stack.

  • Physical: adequate datacentre, power and cooling, rack space, IP and storage networks, etc.
  • Environmental: location of offices, distances, availability of services, etc.
  • External Providers: telecommunications, facility management, etc.
  • Personnel: adequate number, skills and maturity necessary to administer and support critical environments, etc.
  • And more

For our hypothetical client, we will assume the following:

  • Two datacentres to allow a 2-site configuration.
  • Commercial grade data centre (redundant power and colling).
  • IP network can support layer 2 and layer 3 spanning across the sites.
  • Storage network (SAN fabrics) does span across the sites.
  • Commercial grade servers. The size and type will depend on application needs.

High Availability Architecture Options

Metro-Cluster with Host-Based Mirroring

Design

This is known as a campus or metropolitan area cluster (or potentially geographic if the infrastructure supports is). Essentially this is a normal cluster except it spans over some distance by virtue of some dark fibre technology. Also commonly known as a “stretched cluster”. The benefit of such a design is that it offers all the protection from any single point of failure but also protects from site failures and some geographic (local to that area) disasters.

Advantages

This is a standard and simple cluster configuration using host-based mirroring between the storage devices (LUNs) across some distance.

  • no single points of failure.
  • fast automatic failover of data services between sites as if they were local clustered nodes. This exceeds the RTO requirement.
  • no difference in administrative overhead compared to local clusters. ie existing clusters.
  • relatively cheap solution in terms of clustering technology – no additional licenses such as Geo-Cluster or storage-level replication.
  • host-based mirroring tools such as ZFS and SVM come bundled (free) with the operating system. This will meet the RPO requirement.
  • fast/synchronous mirroring ensures data is copied at both sites.
  • host-based mirroring is storage-vendor independent.

Disadvantages

In 2-site cluster configurations the main overhead is the communications infrastructure between the sites to facilitate replication and failover. Therefore the LAN and SAN must span between the datacentres.

  • cost of inter-site SAN infrastructure in terms of hardware, software and licenses.
  • cost of inter-site LAN infrastructure in terms of hardware, software and licenses.

RTO/RPO Implications

In terms of RPO the data will be the same at both sites. Host-based disk mirroring will synchronously mirror data between the LUNs. Therefore when experiencing a hardware failure data loss will be nil (excluding for any traffic on the wire at the time of the event).

In terms of RTO only a short time will elapse when a failover occurs to the alternate master in case of any hardware failure. Therefore when experiencing a hardware failure return of service times will be within minutes depending on the application restart times.

General Notes

To achieve highly available infrastructure you must start with server and storage hardware with high reliability, availability and serviceability (RAS) features. Generally, this architecture will require:

  • Multiple commercial grade servers – Sun, HP, IBM, Dell, etc.
  • Multiple commercial grade storage arrays – HDS, EMC, IBM, NetApp, etc.
  • Clustering software – Solaris Cluster, and some Linux variants.
  • Host-based mirroring software – ZFS, SVM (Solaris), LVM (linux), etc.
  • Storage-level replication software is optional for servers with no host-based mirroring capability.
  • Extended (SAN) Fabrics.
  • Spanning layer-2 IP LANs.
  • A cluster quorum server (or device) will be located at site-3.

Even though the diagram shows Oracle/Sun servers and HDS storage arrays, this infrastructure can be applied using other vendors in a similar fashion.

Metro-Cluster with Storage-Based Replication

Design

This is known as a campus or metropolitan area cluster (or potentially geographic if the infrastructure supports is). This also is a standard cluster that spans sites as the cluster above except that is uses storage-level replication to copy the data between the sites. The benefit of such a design is that it offers all the protection from any single point of failure but also protects from site failures and some geographic (local to that area) disasters. In order to guarantee data consistency at either end synchronous storage-level replication is mandatory. Asynchronous replication is not supported with some clustering software as the delivery mechanism cannot guarantee the consistency of data at a point in time at both sites.

Advantages

This is a cluster configuration using storage-level replication to copy the data between the storage array LUNs across some distance.

  • no single points of failure.
  • fast automatic failover of data services between sites as if they were local clustered nodes. This exceeds the RTO requirement.
  • fast/synchronous storage replication ensures data is copied at both sites. This will meet the RPO requirement.
  • server-vendor and operating system independent as replication occurs at LUN level. May however need other clustering product or manual failover systems to switch between sites.

Disadvantages

In 2-site cluster configurations the main overhead is the communications infrastructure between the sites to facilitate replication and failover. Therefore the LAN and SAN must span between the datacentres.

  • cost of inter-site SAN infrastructure in terms of hardware, software and licenses.
  • cost of inter-site LAN infrastructure in terms of hardware, software and licenses – Layer-2/3.
  • some administrative overhead is required in terms of integrating the Cluster software and the storage array replication software – particularly for automation and switching sites.
  • replication software licenses may be required.

RTO/RPO Implications

In terms of RPO the data will be the same at both sites. Storage-level replication will synchronously mirror data between the LUNs across arrays. Therefore when experiencing a hardware failure data loss should be close to nil.

In terms of RTO only a short time will elapse when a failover occurs to the alternate master in case of any hardware failure. Therefore when experiencing a hardware failure return of service times will be within minutes depending on the application restart times.

General Notes

To achieve highly available infrastructure you must start with server and storage hardware with high reliability, availability and serviceability (RAS) features. Generally this architecture will require:

  • Multiple commercial grade servers – Sun, HP, IBM, Dell, etc.
  • Multiple commercial grade storage arrays – HDS, EMC, IBM, NetApp, etc.
  • Clustering software – Solaris Cluster, and some Linux variants.
  • Extended Fabric licenses may be required.
  • Spanning IP layer-2 licenses may be required.
  • Synchronous replication software and licenses.
  • A cluster quorum server (or device) will be located at site-3.

Even though the diagram shows Oracle/Sun servers and HDS storage arrays, this infrastructure can be applied using other vendors in a similar fashion.

If no clustering software is used, then switch or failovers can be manually performed to take over those replicated LUNs in case of a disaster.

Geographic-Cluster with Storage-Based Replication

Design

This is known as a Geographic area cluster. This is essentially a cluster of clusters. A local cluster exists in each site where highly available data services automatically failover/switchover between the nodes in the local cluster. In the event of a site or geographic disaster, manual failover can be initiated to start the data services on the cluster on the alter site.

This design also use operating system or storage-level replication to copy the data between the sites.

The benefit of such a design is that it offers all the protection from any single point of failure but also protects from site failures and geographic (local to that area) disasters.

This architecture applies specifically to Solaris Cluster and Solaris Cluster Geographic Edition software.

Advantages

This is a cluster of clusters and uses storage-level replication to copy the data between the storage array LUNs across some distance.

  • no single points of failure.
  • fast automatic failover of data services between site-local clustered nodes only.
  • fast/synchronous replication ensures data is copied at both sites.
  • IP layer-3 transport is already suitable and available.

Disadvantages

In 2-site cluster configurations the main overhead is the communications infrastructure between the sites to facilitate replication and failover. IP LAN can span at Layer-3 but the SAN must span between the datacentres.

  • cost of inter-site (SAN) infrastructure in terms of hardware software and licenses.
  • failover between the datacentres requires user intervention. However, you can still provide a “big red button” (command or script) recovery to switch the services between the datacentres.
  • failover of the clients between the datacentres requires the DNS to be updated and client caches to also be updated.
  • some administrative overhead is required in terms of integrating eg the Cluster (local and geographic), Solaris Geographic Cluster software and the HDS True Copy replication software.
  • HDS True Copy synchronous replication software (CLI) and licenses.
  • HDS True Copy asynchronous (TCA) and Universal Replicator (HUR) can be configured outside of Solaris Cluster framework. There is great potential of data loss when using any asynchronous replication such as TCA or HUR.

RTO/RPO Implications

In terms of RPO the data will be the same at both sites. Storage-level replication will synchronously mirror data between the array LUNs. Therefore, when experiencing a hardware failure, data loss should be close to nil.

When configuring an asynchronous form of replication RPOs will be affected as there will be an undetermined amount of data loss. In these cases, one must plan for worse case scenarios.

In terms of RTO only a short time will elapse when a failover occurs to the alternate master in case of hardware failure as it will be handled within the local-site cluster. In the event of a site failure then the RTO will be extended due to human intervention decisions and procedures.

General Notes

To achieve highly available infrastructure you must start with server and storage hardware with high reliability, availability and serviceability (RAS) features. Generally this architecture will require:

  • Multiple commercial grade servers – Sun, HP, IBM, Dell, etc.
  • Multiple commercial grade storage arrays – HDS, EMC, IBM, NetApp, etc.
  • Clustering software – Solaris Cluster, and some Linux variants.
  • Clustering Geographic Edition software – Solaris Cluster, and some Linux variants.
  • Extended Fabric licenses may be required.
  • Spanning IP layer-2 licenses may be required.
  • Synchronous replication software and licenses.
  • A cluster quorum server (or device) will be located locally at each site for the local cluster.

This infrastructure is specific for the Clustering software that can manage multiple Clusters ie Solaris Cluster Geographic Edition software. The underlying server and storage hardware can also be configured in any combination as the previously described architectures and make use of the storage-level LUN replication where a standby server at the alternate site can be manually configured to take over those replicated LUNs in case of a disaster.

Manual “Cluster” with Storage-Based Replication

Design

This can be a either a stretched cluster or a Geographic area cluster. This is covering the same architecture as previously mentioned but without any automation ie no commercial Cluster framework.

This may sound as if providing some cost savings but effectively you have removed all the high-availability features that a clustering product such as Solaris Cluster offers. Such features include logical resource group management, hardware failure detection and recovery, application-level fault probes (agents), validation, failure fencing, split-brain protection, etc.

For the purposes of the scope, this option is also documented.

For high-availability requirements this is not a suitable option.

Advantages

This is a cluster of clusters and uses storage-level replication to copy the data between the storage array LUNs across some distance.

  • no single points of failure.
  • relatively cheap in terms od savings from Sun Cluster licenses.
  • can make use of existing SAN/LAN infrastructure if already implemented.
  • can be used for platforms that do not have a clustering option.

Disadvantages

In 2-site cluster configurations the main overhead is the communications infrastructure between the sites to facilitate replication and failover. Therefore, the LAN and SAN must span between the datacentres.

  • cost of inter-site infrastructure if new implementation.
  • no automation of fault and failure detection.
  • no automatic failover in case of hardware failure.
  • no validation of single points of failure.
  • perhaps the use of customised scripts – testing, support, etc.
  • faults and failures require a human to be available to detect them.
  • decision-making and escalation process upon failure detection and reaction could take time.
  • failover between the datacentres requires user intervention. Which may include the coordination between multiple groups and escalation.
  • storage synchronous replication software licenses are required.
  • great deal of administrative overhead is required in terms of integrating and ongoing administering all the software products.
  • great deal of configuration and administration documentation is required and to be maintained.
  • There is great potential of catastrophic data loss if procedures are not thoroughly documented and thoroughly tested or in the case of human errors.

RTO/RPO Implications

These are very hard to judge due to the amount of human intervention and the number of people that could be involved in the detection, escalation and correction of the problem. The more people and groups involved the greater the risk that RPO and RTO could be affected.

Assuming that all the hardware and software products are integrated properly and thoroughly documented then in terms of RPO the data will be the same at both sites. There is the option of host-based mirroring or storage-level replication. Therefore, when experiencing a hardware failure data loss should be close to nil.

In terms of RTO this would be a guess. Could be as fast as a couple of hours to many hours and  days if an accidental mistake was made.

General Notes

To achieve highly available infrastructure you must start with server and storage hardware with high reliability, availability and serviceability (RAS) features. Generally this architecture will require:

  • Multiple commercial grade servers – Sun, HP, IBM, Dell, etc.
  • Multiple commercial grade storage arrays – HDS, EMC, IBM, NetApp, etc.
  • Clustering software – Solaris Cluster, and some Linux variants.
  • Extended Fabric licenses may be required.
  • Spanning IP layer-2 licenses may be required.
  • Synchronous replication software and licenses.
  • many man hours of management.

In this case it’s a matter of creating the local clusters at each site, configure the replication across the sites.

Note: if this is manually managed, great care needs to be taken to ensure only one cluster is running at any one time and the replication is going in the right direction.

So in the event of a manual switch over from site 1 to site 2, the high level steps include:

  • stop the applications
  • the cluster at site 1 needs to be stopped
  • the storage replication needs to be suspended
  • volumes at site 2 need to be made read/writable
  • the cluster at site 2 can be brought up and start the applications
  • replication restarted back to site 1

This infrastructure presents a relatively cheap solution but at great risk.

Ultimately, how you implement a high availability and disaster recovery solution will mainly be driven the organisations business needs, then technology capabilities and costs (monetary and personnel skills) to implement that solution. Remember that an architecture always starts at the top of the information stack.

KAOS Data can assist with designing and implementing such infrastructure architectures. Just ask our team.

Check out our other Cheat Sheets and Blogs and if you would like us to write a cheat sheet for you, for FREE, (and we find it suitable) Contact Us.