Edge data centre resilience is critical to safeguarding precious genome sequencing data at the Wellcome Sanger Institute. The 4MW data centre contains the potential answers to tackling the world’s deadliest diseases
When the Ebola outbreak hit Sierra Leone, leading to more than 11,000 deaths and 28,600 infections, genetic sequencing at the Wellcome Sanger Institute played a vital part in understanding and curbing the further spread of the deadly disease. The charity is one of the premier centres of genomic discovery in the world and the data held within its 4MW data centre is of the highest global importance – playing a leading role in national and international projects spanning: cancer, infectious disease, human epidemiology and developmental disorders.
In recent years, researchers at the Wellcome Sanger Institute have discovered thousands of genes essential for cancer’s survival and ranked which ones show the most promise as drug targets for developing new treatments. The institute is also embarking on other ambitious projects, including The Darwin Tree of Life project, which aims to sequence all complex life within the British Isles. This is part of a much wider plan, known as the Earth BioGenome Project, which aims to map the genomics for all life on earth.
Resilience at the edge
The heavy demands of data generation and analytics requires compute at the point of data generation and the proximity requirement makes it one of the largest edge data centres in the world. To support the huge growth in demand for genome sequencing, the Institute’s data centre campus – based at Hinxton, Cambridgeshire – has moved from being operational five days per week, 9-5; to being ‘always on’, 24/7, 365 days per year.
Sequencing requires a huge amount of computing power and the output at the site produces a staggering 2TB data per day, per genome. The Wellcome Trust site currently holds over 150PB of data, which cannot be deleted, due to the global importance of this scientific knowledge. This vast amount of data is not tolerant of outages, however, so ensuring the resilience of the physical infrastructure is vital. The campus has previously experienced utility power issues (with 12 brownout periods during 2018), so monitoring the health of the site’s UPS systems was an important aspect of delivering reliable uptime.
Data centre manager Simon Binley required a software solution that could offer complete visibility for three ageing, legacy data centres and a new 1.2 MW high density rack facility, built by ISG. EfficiencyIT, a UK-based reseller and Elite Partner to Schneider Electric, installed EcoStruxure IT Expert software to enable a small team (comprising just three personnel) to manage the facility efficiently and reliably.
“For the Wellcome Sanger Institute, ‘the edge’ is where the scientists operate the sequencing machines. There are around 25 of these machines, each costing around £2m – all sequencing the human genome for the betterment of mankind. In the past, it took 10 years to map just one human genome. Today, these machines are mapping one to two human genomes per day. If they lose a UPS and there is an outage – a day of genome sequencing research could potentially be lost.
“There is a lot of discussion around the cost of downtime for data centres, but in this instance, downtime means that £36,000 worth of chemical components have to be replaced for each sequencing machine. As there are 25 machines, this gives a potential cost of £900,000 per day. Added to this is the intangible cost of not being able to perform vital research. This makes data centre uptime absolutely mission critical,” comments Nick Ewing, managing director at EfficiencyIT.
“Ensuring availability for our researchers is vital,” adds Binley. “It is really important that we understand the state of the data centres and the environment. The software gives us a granular view of the metrics that really drive the data centres – including the temperature and power. The ability to see what is going on, through the EcoStruxure IT software, also allows us to build efficiencies into the data centres. When we drive efficiencies, we make cost savings – money that goes straight back into the charity’s live-saving research.”
Data centre insights
The system provides a physical representation of the data centre – including the inlet and outlet temperature for every single rack, as well as providing intelligence on capacity and load. Using this tool, Simon Binley can manage capacity planning, optimise power and cooling, as well as the physical space and, ultimately, achieve the goal of detecting any issues before they become problem.
“The system allows me to remotely monitor what is happening, at any time of day, via the EcoStruxure smart-phone app, and converse with our engineers when there are potential issues. It provides an early warning and avoids any delays. It has been a real game-changer for us,” Binley comments.
The institute needed a single interface that would bring together the disparate metrics into one place. The engineers no longer have to access many different platforms to obtain all the measurements they need to understand exactly how the data centre is performing – from the UPSs to individual PDUs.
“The visibility and information obtained, via EcoStruxure IT, allows the institute to unlock the engineers’ time to support the physical IT hardware throughout the data centre,” comments Binley. “This has led to an efficiency gain in their ability to respond to faulty hard drives.
He explains that the disks have a limited shelf life and need replacing on a regular basis: “The software enables the engineers to focus on the tasks they really need to be doing rather than spending time looking through half a dozen different systems to find an answer on how the data centre is performing. This is a huge advantage.”
“We have basically given Simon the eyes into his data centre,” adds Ewing. “The institute’s edge comprises 25-30 APC Smart-UPSs and we are able to provide intelligent, predictive information, based on Schneider Electric’s huge amount of acquired analytic data.”
Regular reports are provided, detailing which UPSs are nearing the end of their lives, so that the Institute can schedule their replacement.
“By taking this proactive step, we can save the institute many thousands of pounds worth of lost chemicals and several days of lost days of genome sequencing. The ability to give this forward planning and visibility, is a powerful tool,” Ewing continues.
“It enables us to understand how individual assets are performing and replace equipment before it fails. You can diagnose whether a PDU has a fault, for example. This means maintenance can be carried out in a more proactive way. The plan is also to look at mechanical elements to make the data centre more efficient. We are at the beginning of that journey,” says Binley.
The Wellcome Trust is also concerned about its carbon footprint and the sustainability of its compute operations. Simon Binley wanted a software solution that offered enhanced visibility of energy use, with the ability to identify areas for improvement. The EcoStruxure IT Energy Efficiency Module provides PUE and granular electrical data for all four data centres. The most recent data centre build on the campus has also incorporated diesel rotary uninterruptible power supply units (DRUPS), which has eliminated the use of batteries in the fourth quadrant.
“This means we no longer have to swap out batteries every five years, which has an environment impact around disposal,” says Binley.
In the future, the data centre team plans to use the software to raise the temperature of the data centre to an optimum level, to create additional efficiencies, while alternative approaches to cooling may also be considered.
“We are in the process of investigating whether we may need to increase densities and move to immersion cooling technology. The plan is to reach a point where, for individual racks and rows, the cooling can be matched to the load and, therefore, drive efficiencies even further. In the long term, the ambition is to ensure that the costs of data centre operation are decreased even further, ensuring the Institute can maximise its budget for important scientific research,” Binley concludes.
For more information click here
Standardisation and integration
Standardisation was an important requirement for the Wellcome Sanger Institute and the campus includes a variety of infrastructure from Schneider Electric. Today, these components enable visibility of the physical layer and include:
• APC Racks NetShelter in 95% of the data centre space
• APC Metered Gen 2 PDUs in 95% of the data centre
• APC NetBotz devices throughout for environmental monitoring
• APC Temp and Humidity devices
• APC Symmetra Megawatt UPS
The Schneider Electric management tools include:
• EcoStruxure IT Data Center Expert to monitor the core data centre. This is being expanded to the edge comms rooms and sequencing environment.
• EcoStruxure IT Data Center Operations
• EcoStruxure IT – Energy Efficiency Module – to provide PUE and granular electrical data to all four data centres individually and jointly
• EcoStruxure IT Expert – this brings all the core devices into the Schneider Electric data lake, providing access via a smart phone app, including analytical data