post

The Switch SuperNAP data centre – one of the most impressive I’ve been in

by

Switch SuperNAP data centre

If you were going to build one of the world’s largest data centre’s you wouldn’t intuitively site it in the middle of the Nevada desert but that’s where Switch sited their SuperNAPs campus. I went on a tour of the data centre recently when in Las Vegas for IBM’s Pulse 2012 event.

The data centre is impressive. And I’ve been in a lot of data centre’s (I’ve even co-founded and been part of the design team of one in Ireland).

The first thing which strikes you when visiting the SuperNAP is just how seriously they take their security. They have outlined their various security layers in some detail on their website but nothing prepares you for the reality of it. As a simple example, throughout our entire guided tour of the data centre floor space we were followed by one of Switch’s armed security officers!

The data centre itself is just over 400,000 sq ft in size with plenty of room within the campus to build out two or three more similarly sized data centres should the need arise. And although the data centre is one of the world’s largest, at 1,500 Watts per square foot it is also quite dense as well. This facilitates racks of 25kW and during the tour we were shown cages containing 40 x 25kW racks which were being handled with apparent ease by Switch’s custom cooling infrastructure.

Switch custom cooling infrastructure

Because SuperNAP wanted to build out a large scale dense data centre, they had to custom design their own cooling infrastructure. They use a hot aisle containment system with the cold air coming in from overhead and the hot air drawn out through the top of the contained aisles.

The first immediate implication of this is that there are no raised floors required in this facility. It also means that walking around the data centre, you are walking in the data centre’s cold aisle. And as part of the design of the facility, the t-scif’s (thermal seperate compartment in facility – heat containment structures) are where the contained hot aisle’s air is extracted and the external TSC600 quad process chillers systems generate the cold air externally for delivery to the data floor. This form of design means that there is no need for any water piping within the data room which is a nice feature.

Through an accident of history (involving Enron!) the SuperNAP is arguably the best connected data centre in the world, a fact they can use to the advantage of their clients when negotiating connectivity pricing. And consequently, connectivity in the SuperNAP is some of the cheapest available.

As a result of all this, the vast majority of enterprise cloud computing providers have a base in the SuperNAP. As is the 56 petabyte ebay hadoop cluster – yes, 56 petabyte!

US electricity generation

Given that I have regularly bemoaned cloud computing’s increasing energy and carbon footprint on this blog, you won’t be surprised to know that one of my first questions to Switch was about their energy provider, NV Energy.

According to NV Energy’s 2010 Sustainability Report [PDF] coal makes up 21% of the generation mix and gas accounts for another 63.3%. While 84% electricity generation from fossil fuels sounds high, the 21% figure for coal is low by US standards, as the graph on the right details.

Still, it is a long way off the 100% of electricity from renewables that Verne Global’s new data centre has.

Apart from the power generation profile, which in fairness to Switch, is outside their control (and could be considerably worse) the SuperNAP is, by far, the most impressive data centre I have ever been in.

Photo Credit Switch

post

Data Center War Stories – Maxim Samo from UBS

by

Back at the start of the summer I mentioned that Sentilla had asked us to run a series of chats with data center practitioners to talk about their day-to-day challenges.

This was very much a hands-off project from Sentilla – I sourced all the interviewees, conducted all the interviews and am publishing the results without any Sentilla input or oversight. This had advantages and disadvantages – obviously, from an independence perspective, this was perfect but from a delivery point of view it was challenging. It turns out that data center practitioners are remarkably camera shy, so it far longer than anticipated to produce this series, however, finally I’m ready with the first of the series, with more to come every week in the coming weeks.

This first episode in the series is with Maxim Samo, who is responsible for managing the European data center operations of global financial services firm UBS.

Here’s a transcript of our conversation:

Tom Raftery: Hi everyone and welcome to GreenMonk?s Data Centers War Stories series, sponsored by Sentilla. This week?s guest is Maxim Samo who works for the Swiss financial services company UBS. Maxim, do you want to start off by telling us what you do for UBS and what the kind of size of your datacenter fleet is?

Maxim Samo: Yeah, I run the Swiss and European operation for UBS, at the moment we have five datacenters in Switzerland and three outside of Switzerland spread around Europe, the total size or capacity probably being around six megawatts.

Tom Raftery: Okay and what kind of age is the fleet, is it like you know the last five years or 10, or is that — it?s obviously a variety that you didn?t build all eight in the one go.

Maxim Samo: Right, it?s anywhere between — they were built anywhere between 1980 and 2004, there is a couple of colo?s that are probably newer than that, but yeah.

Tom Raftery: So if they were built starting in 1980, I mean I assume that this is one of the reasons why you think more in terms of power as supposed to space because your — they weren?t optimized around power at that time I?m sure.

Maxim Samo: Oh not at all, exactly. They were built with a density of around 300 watts per square meter or even less right, I mean they were mainframe datacenters and we kind of ? well, we did some refurbishments in there and as a matter of fact one of those datacenters is undergoing a major renovation right now to increase the amount of power that we can put in there.

Tom Raftery: Power is obviously one of the more pressing issues you guys are running up against, but what are the other kind of issues you have in the datacenters in the day-to-day operations?

Maxim Samo: So the way our datacenters are built in Europe at least within UBS is that, we don?t have like big data halls, but we have a number of smaller rooms within the datacenter building and in order to be cost effective you know we don?t have every single network available in all the rooms, we don?t have every single storage device and storage network in terms of production storage or tester development storage available in all the rooms.

So some of our constraints or else or around that we have to — not only do we have to manage the capacity, but we have to figure out which rooms the servers come in and then try to get adequate forecasts of how much the business and the developers want to put into what datacenter rooms and try to juggle the capacity around that.

Tom Raftery: We are calling the show the Datacenter War Story. So, have you any interesting problems that you came across in the last number of years and resolved any interesting issues that you hit up against?

Maxim Samo: So, in terms of war stories I guess we are — one thing is we are going to have the interesting issue about switching the electrical network of the datacenter that is undergoing renovation and we are currently looking at the options of how we can do that.

One option would be that we would switch — well, that we would put both ups into utility bypass, runoff the utility, and then switch over the network, where of course you have the risk of a power blip coming through which takes down your datacenter. So, in order to mitigate that we are also talking about a full scale shutdown of the datacenter, which right now is being received very well by the people involved, so that?s going to be an interesting one.

So we had, actually very recently we had a very funny case where we — what we do is, we conduct black star tests, black star test is when you almost, you like pull the plug and see what happens, right. So you literally cut off the utility network, your ups will carry the power, the diesel generators will start and you make sure everything works smoothly.

The last time we did this test that was a week ago on the weekend when the diesel generator started it created so much smoke that a pedestrian out on the street actually called the fire department and we had the fire department come in and lot of people were panicking and asking what is going on, we have a fire in the datacenters, like no, we just tested our diesel generators, that was a very funny instance.

I can really remember a war story in terms of the datacenter going down luckily that has not happened for a very long time, we absolutely, we probably — well, we did have a partial failure at one point where pretty big power switch within the switch gear has failed and brought down one side of the power.

However, since most of our servers and IT equipment is dual power attachsed it did not have any impact on their production.

Tom Raftery: Great, that?s been fantastic. Max, thanks for coming on the show.

Maxim Samo: All right, thanks Tom.

Disclaimer – Maxim asked me to mention that any views he expressed in this video are his own, and not those of his employer, UBS AG.

post

Data center war stories sponsored blog series – help wanted!

by

Data center work

Sentilla are a client company of ours. They have asked me to start a discussion here around what are the day-to-day issues data center practitioners are coming up against.

This is a very hands-off project from their point of view.

The way I see it happening is that I’ll interview some DC practitioners either via Skype video, or over the phone, we’ll have a chat about DC stuff (war stories, day-to-day issues, that kind of thing), I’ll record the conversations and publish them here along with transcriptions. They’ll be short discussions – simply because people rarely listen to/watch rich media longer than 10 minutes.

There will be no ads for Sentilla during the discussions, and no mention of them by me – apart from an intro and outro simply saying the recording was sponsored by Sentilla. Interviewees are free to mention any solution providers and there are no restrictions whatsoever on what we talk about.

If you are a data center practitioner and you’d like to be part of this blog series, or simply want to know more, feel free to leave a comment here, or drop me an email to [email protected]

You should follow me on Twitter here
Photo credit clayirving

post

Top 10 Data Center blogs

by

Data center air and water flows

Out of curiosity, I decided to see if I could make a list of the top 10 data center focussed blogs. I did a bit of searching around, found around thirty blogs related to data centers (who knew they were so popular!). I went through the thirty blogs and eliminated them based on arbitrary things I made up on the spot like post frequency, off-topic posts, etc. until I came up with a list I felt was the best. Then I counted them and lo! I had exactly 10 – phew, no need to eliminate any of the good ones!

So without further ado – and in no particular order, I present you with my Top 10 Data Center blogs:

What great data center blogs have I missed?

The chances are there are superb data center blogs out there which my extensive 15 seconds of research on the topic failed to uncover. If you know of any, feel free to leave them in the comments below.

Image credit Tom Raftery

post

FaceBook open sources building an energy efficient data center

by

FaceBook's new custom-built Prineville Data Centre

Back in 2006 I was the co-founder of a Data Centre in Cork called Cork Internet eXchange. We decided, when building it out, that we would design and build it as a hyper energy-efficient data centre. At the time, I was also heavily involved in social media, so I had the crazy idea, well, if we are building out this data centre to be extremely energy-efficient, why not open source it? So we did.

We used blogs, flickr and video to show everything from the arrival of the builders on-site to dig out the foundations, right through to the installation of customer kit and beyond. This was a first. As far as I know, no-one had done this before and to be honest, as far as I know, no-one since has replicated it. Until today.

Today, Facebook is lifting the lid on its new custom-built data centre in Prineville, Oregon.

Not only are they announcing the bringing online of their new data centre, but they are open sourcing its design, specifications and even telling people who their suppliers were, so anyone (with enough capital) can approach the same suppliers and replicate the data centre.

Facebook are calling this the OpenCompute project and they have released a fact sheet [PDF] with details on their new data center and server design.

I received a pre-briefing from Facebook yesterday where they explained the innovations which went into making their data centre so efficient and boy, have they gone to town on it.

Data centre infrastructure
On the data centre infrastructure side of things, building the facility in Prineville, Oregon (a high desert area of Oregon, 3,200 ft above sea level with mild temperatures) will mean they will be able to take advantage of a lot of free cooling. Where they can’t use free cooling, they will utilise evaporative cooling, to cool the air circulating in the data centre room. This means they won’t have any chillers on-site, which will be a significant saving in capital costs, in maintenance and in energy consumption. And in the winter, they plan to take the return warm air from the servers and use it to heat their offices!

By moving from centralised UPS plants to 48V localised UPS’s serving 6 racks (around 180 Facebook servers), Facebook were able to re-design the electricity supply system, doing away with some of the conversion processes and creating a unique 480V distribution system which provides 277V directly to each server, resulting in more efficient power usage. This system reduces power losses going in the utility to server chain, from an industry average 11-17% down to Prineville’s 2%.

Finally, Facebook have significantly increased the operating temperature of the data center to 80.6F (27C) – which is the upper limit of the ASHRAE standards. They also confided that in their next data centre, currently being constructed in North Carolina, they expect to run it at 85F – this will save enormously on the costs of cooling. And they claim that the reduction in the number of parts in the data center means they go from 99.999% uptime, to 99.9999% uptime.

New Server design
Facebook also designed custom servers for their data centres. The servers contain no paint, logos, stickers bezels or front panel. They are designed to be bare bones (using 22% fewer materials than a typical 1U server) and for ease of serviceability (snap-together parts instead of screws).

The servers are 1.5U tall to allow for larger heat sinks and larger (slower turning and consequently more efficient) 60mm fans. These fans only take 2-4% of the energy of the server, compared to 10-20% for typical servers. The heat sinks are all spread at the back of the mother board so none of them will be receiving pre-heated air from another heat sink, reducing the work required of the fans.

The server power supply accepts both 277V AC power from the electrical distribution system and 44V DC from the UPS in the event of a utility power failure. These power supplies have a peak efficiency of 94.5% (compared to a more typical 90% for standard PSU’s) and they connect directly to the motherboard, simplifying the design and reducing airflow impedance.

Open Compute
Facebook relied heavily on open source in creating their site. Now, they say, they want to make sure the next generation of innovators don’t have to go through the same pain as Facebook in building out efficient data centre infrastructure. Consequently, Facebook is releasing all of the specification documentation which it gave to its suppliers for this project.

Some of the schematics and board layouts for the servers belong to the suppliers so they are not currently being published, though Facebook did say they are working with their suppliers to see if they will release them (or portions of them) but they haven’t reached agreement with the suppliers on this just yet.

Asked directly about their motivations for launching Open Compute Facebook’s Jay Park came up with this classic reply

… it would almost seem silly to do all this work and just keep it closed

Asking Facebook to unfriend coal
Greenpeace started a campaign to pressure Facebook into using more renewable energy resources due to the fact that Pacific Power, the energy supplier Facebook will be using for its Prineville data center produces almost 60% of its electricity from burning coal.

Greenpeace being Greenpeace, created a very viral campaign, using the Facebook site itself, and the usual cadre of humurous videos etc., to apply pressure on Facebook to thinking of sourcing its electricity from more renewable sources.

When we asked Facebook about this in our briefing, they did say that their data centre efforts are built around many more considerations than just the source of energy that comes into the data centre. They then went on to maintain that they are impressed by Pacific Power’s commitment to moving towards renewable sources of energy (they are targeting having 2,000MW of power from renewables by 2013). And they concluded by contending that the efficiencies they have achieved in Prineville more than offsets the use of coal which powers the site.

Conclusion
Facebook tell us this new custom data centre at Prineville has a PUE of 1.07, which is very impressive.

They have gone all out on innovating their data centre and the servers powering their hugely popular site. More than that though, they are launching the Open Compute Project giving away all the specs and vendor lists required to reproduce an equally efficient site. That is massively laudable.

It is unfortunate that their local utility has such a high gen-mix of coal in its supply to besmirch an otherwise great energy and sustainability win for Facebook. The good thing though is that as the utility adds to its portfolio of renewables, Facebook’s site will only get greener.

For more on this check out the discussions on Techmeme

You should follow me on Twitter here

Photo credit FaceBook’s Chuck Goolsbee

post

Sentilla thinks of data centers, as data factories!

by

Data center

If you have been following this blog, you’ll know I have been profiling Data Center efficiency companies over the last few weeks. This week I take a look at Sentilla.

I talked to Sentilla’s CTO and co-founder, Joe Polastre, the other day and Joe told me that Sentilla came out of Berkeley where they had been looking at data analytics problems around large, potentially incomplete or inaccurate, streaming datasets. The challenge was how to turn that into a complete picture of what’s going on so people could make business decisions.

Sentilla takes an industrial manufacturing approach to Data Centers – in manufacturing you have power going in one side, and products and (often) waste heat coming out the other. In the same way in data centers you have power going in one side and coming out the other side you have the product (compute cycles) and waste heat. To optimise your data center you need to get the maximum data/compute (product) output with the minimum power in and the least waste heat generated. Sentilla thinks of data centers, as data factories!

Unlike most of the data center people I have talked to, Sentilla don’t talk so much about energy savings. Instead they emphasise maximising performance – getting the most out of your existing data centers, your existing storage, your existing servers, your existing power supply. By far the greatest saving from deploying Sentilla, Joe claimed, is not from the energy savings. That pales in comparison to the capital deferment savings gained from being able to delay the building of extra data center facilities by however many years, he said.

So how does Sentilla help?

Well Sentilla analyses the energy profile of every asset in the data center, whether metered or not, and makes recommendations to improve the planning and management of data center operations. I highlighted the “whether metered or not” bit because this is an important differentiator for Sentilla – they have developed and patented what they call “virtual meters”. These are algorithms which look at the work that a device is doing, and based on models which Sentilla have built up, and measurements they have done, as well as some benchmarks which are out there, Sentilla computes how much power is being used by that equipment.

The reason this is so important is because the most inefficient equipment in the data center is not the new stuff (which is likely to already be metered) but the legacy devices. These are the ones which need to be most carefully managed, and the ones where the greatest performance gains for the data center can be made. And because Sentilla can pull usage information from management databases like Tivoli, it means the Sentilla doesn’t need to poll every piece of equipment in the data center (with the increased network traffic and data that would generate).

Also, because Sentilla has its virtual meters, it is a software-only product and can therefore be rolled out very quickly.

The other nice feature Sentilla has is that it can identify the energy utilisation of virtualised servers. This is important because with the increasing ease of deployment of virtual servers, under-utilised VM’s and VM clutter are starting to become issues for data centers. VM clutter isn’t just an issue for energy reasons – there are also implications for software licensing, maintenance and SLA requirements.

I asked Joe about whether Sentilla is a SaaS product and he said that while they have a SaaS version of the product, so far most of Sentilla’s clients prefer to keep their data in-house and they haven’t gone for the SaaS option.

Finally I asked about pricing and Joe said that Sentilla is priced on a subscription basis and, apparently, it is priced such that for any modest sized data center, for every $1 you put into Sentilla, you get $2 back. Or put another way, Joe said, deploying Sentilla will generally mean that you reclaim around 18-20% of your power capacity.

Disclosure: Sentilla are a client (but this post is not part of their client engagement)

You should follow me on Twitter here

Photo credit The Planet

post

Power Assure automates the reduction of data center power consumption

by

Data centre

If you’ve been following this blog in the last couple of weeks you’ll have noticed that I have profiled a couple of data centre energy management companies – well, today it is the turn of Power Assure.

The last time I talked to Power Assure was two years ago and they were still very early stage. At that time I talked to co-founder and CTO, Clemens Pfeiffer, this time I spoke with Power Assure’s President and CEO, Brad Wurtz.

The spin that Power Assure put on their energy management software is that, not only do they offer their Dynamic Power Management solution which provides realtime monitoring and analytics of power consumption across multiple sites, but their Dynamic Power Optimization application automatically reduces power consumption.

How does it do that?

Well, according to Brad, clients put an appliance in each of the data centres they are interested in optimising (Power Assure’s target customer base are large organisations with multiple data centres – government, financial services, healthcare, insurance, telco’s, etc.). The appliance uses the management network to gather data – data may come from devices (servers, PDU’s, UPS’s, chillers, etc.) directly, or more frequently, it gathers data directly from multiple existing databases (i.e. a Tivoli db, a BMS, an existing power monitoring system, and/or inventory system) and performs Data Centre analytics on those data.

Data centre

The optimisation module links into existing system management software to measures and track energy demand on a per applications basis in realtime. It then calculates the amount of compute capacity required to meet the service level agreements of that application and adds a little bit of headroom. From the compute it knows the number of servers needed, so it communicates with the load balancer (or hypervisor, depending on the data centre’s infrastructure) and adjusts the size of the server pool to meet the required demand.

Servers removed from the pool can be either power capped or put in sleep mode. As demand increases the servers can be brought fully online and the load balancer re-balanced so the enlarged pool can meet the new level of demand. This is the opposite of the smart grid demand response concept – this is supply-side management – matching your energy consumption (supply to the demand for compute resources).

A partnership with Intel means that future versions will be able to turn off and on individual components or cores to more precisely control power usage.

The software is agentless and interestingly, given the customer profile Brad outlined (pharmas, financial institutions, governments, etc.), customers log in to view and manage their power consumption data because it is SaaS delivered.

The two case studies on their site make for interesting reading and show reductions in power consumption from 56% – 68% which are not to be sneezed at.

The one client referred to in the call is NASA and Power Assure are involved in a data centre consolidation program with them. Based on the work they have done with Power Assure, Brad informed me that NASA now expects to be able to consolidate their current 75 Data Centres significantly. That’ll make a fascinating case study!

You should follow me on Twitter here

Photo credit cbowns

post

Viridity’s new President and CEO Arun Oberoi speaks to GreenMonk

by

Viridity EnergyCheck Screen Shot

We all know data centre’s are massive consumers of energy but just how much? The European data centre consumption was 50 terawatt hours (TWh) in 2008, according to a recent article in the Guardian. This will rise to 100TWh by 2020, roughly the same as the electricity consumption of Portugal.

I mentioned on here just before Christmas that data center energy management company Viridity had named Arun Oberoi as their new President and CEO. Arun has an impressive CV which is outlined in Viridity’s press release about the appointment.

I had the opportunity to chat with Arun recently and he talked to me about Viridity’s solutions.

Data centre with cold aisle containment

As Arun put it, the world has done a great job of mapping dependencies to IT Services in the Enterprise Management world but very little has been done so far on bridging the physical world (think power, space and cooling) to the logical world. These are resources which are becoming very expensive but whose ability to be measured and managed has been hampered by the traditional separation of roles between facilities and IT, for example.

Three areas Viridity can help company’s with, according to Arun are

  1. Power and cost savings
  2. Sustainability – emissions reduction and
  3. Mapping physical to logical to ensure optimisation of resources and managing data centre physical constraints (which, unlike IT, can’t be virtualised!)

Viridity’s software takes the data from many, often disparate sources and provides analysis and trending information to allow managers decide how best to reduce their electricity and space costs. The next version will have automation built-in to enable even greater savings!

In an ideal world this would mean that European data centre consumption might only rise to 60 terawatt hours (TWh) by 2020, instead of the projected 100TWh. However, Parkinson’s Law teaches us that data centre’s expand to fill the power available to run them!

Photo credit Tom Raftery

post

How to build a hyper Energy-efficient Data Center

by

Building a Hyper Energy Efficient Data Center

View SlideShare presentation. (tags: cold aisle containment cix)

I am speaking next week at a virtual conference called “bMighty – A Deep Dive on IT Infrastructure for SMBs” – apologies in advance for the state of the website(!)

My talk is titled “How to build a hyper Energy-efficient Data Center” and is based on the CIX data center which I helped develop (and am still a director of).

This is the slide deck I will be presenting there.