Disaster Recovery Plans for Your LIMS or ELN
Imagine…historic rainfalls flood your production facility and its lab, while knocking out power for miles around. Your analytical equipment is ruined. Your server is under water. Or, wildfires ignite suddenly after a lightning strike or car accident. The fires spread rapidly. Lab personnel have just enough time to escape with what they can carry. Your facility is burned to the ground. These are just two scenarios that have occurred in the last couple of years. And all of us have experienced a network failure or the dreaded blue screen of death at some point. System downtime is going to happen. You may even have a total loss on your hands in the future. How you plan for it is the key.
Do you know how to protect your LIMS or ELN in case of a catastrophic failure? It’s important to have a detailed plan in place in case a disaster happens.
Informatics Disasters and Recovery Plans
There are many kinds of disasters (natural and manmade) that can befall an informatics system. Hardware failures remain the most common cause of downtime, including failures of universal power sources, generators, or server room environmental systems, in addition to failures of the actual equipment. Human error is the next biggest source of data loss. Cyber attacks have increased dramatically in the last decade to become the third biggest cause. Weather is the fourth most common cause of downtime, and weather-related disruptions are also increasing. In developing or updating your LIMS or ELN disaster recovery plan, each of these potential causes and their remediation paths should be considered.
Preparedness training and remediation paths vary for the different causes of informatics system downtime. Hardware failure, human error, and cyber attack remediation are well documented in most recovery plans. There is guidance from the U.S government about information technology disaster recovery plans, if you find that your company needs one. If you have a LIMS or ELN and haven’t yet got informatics-specific language in your disaster recovery plan, you can find some help here.
For the purposes of this blog we are going to look at weather-related failures, because climate change is happening. Companies that fail to recognize the inherent danger this poses to their infrastructure may come to regret it.
Climate Change and Its Potential for LIMS and ELN Disruptions
Even if your data is stored in the cloud at data centers with redundant generators that are far from flood plains, tornado alleys, or drought and fire prone areas, those centers may be served by aging infrastructure. Inadequate infrastructure could be overwhelmed by extreme temperatures or precipitation. One of the hallmarks of climate change is the increase in precipitation that falls as an intense rain event. Aging rain sewer systems from the latter half of the 20th century were not built to carry the amount of water that can fall in one of these events. The increase in extreme rainfall events also results in increased street-level flooding, even in areas that are not near streams or rivers. As we all know, even a little bit of water can ruin your informatics day.
According to the United Nations Office for Disaster Risk Reduction, climate-related disasters caused the US $945 billion in direct economic losses in the United States from 1998 to 2017. The vast majority of those losses were the result of floods and storms. These extreme weather events are resulting in increased downtime. Research has indicated that power failures caused by weather events have doubled since 2003.
Some questions related to disaster recovery that you may want to bring up with your IT team can be found here. The figure shows the increase in land area affected by extreme precipitation events in the continental United States since approximately 1980. This is a trend that is not likely to reverse itself any time soon.
LIMS and ELN Disaster Recovery Plans for Weather-related Disasters
So what should you add to your LIMS or ELN disaster recovery plan to address weather events that could result from climate change? If your company isn’t taking advantage of cloud computing, you should definitely consider keeping copies of your frequent backups in multiple, physically separate locations.
You may want to consider a contract laboratory agreement or analytical equipment redundancy in separate locations so your lab personnel can work at an alternate location temporarily, if the need arises. Taking this step would result in minimal disruptions to essential work and prevent losses related to missed deadlines or underutilized staff.
Moving your LIMS or ELN to the cloud is one way to alleviate risk, but it isn’t as sufficient as a disaster recovery plan in itself. As mentioned above, the physical locations of data centers are vulnerable to climate change, as well as the other systems of the data center. Some of those other systems include roof drains and cooling systems. In a 2018 survey by the Uptime Institute sent to data center operators and IT professionals, 46% of respondents indicated that their data centers were not prepared for climate change-related disruptions. Because your data center may not have sufficient disaster recovery plans themselves, there are additional considerations you may want to add to your own disaster recovery plan. One way to address this is to ensure that server monitoring for early detection of any kind of failure is part of any hardware or hosting agreement.
In an era of increasing climate uncertainty, saying that you should be prepared for anything doesn’t seem like hyperbole anymore. Make sure that your disaster recovery plan is, too.
Are you confident that your data infrastructure is sufficient to handle an extreme weather event? If not, does your disaster recovery plan include the possibility of a system downtime as a result of such an event?
Comments