You have moved some of your IT infrastructure to the Cloud. You have probably moved some applications to SaaS (Software as a Service), development or testing to Platform as a Service (PaaS), or some of your custom applications to an IaaS (Infrastructure as a Service) Cloud. It is possible the whole process went smoothly; but, more likely, you have had a few learning opportunities during the journey so far.
If it has not happened yet, someday you will wake up and realize that you cannot see your Cloud. The two most likely causes are that your Cloud Service Provider (CSP) has a problem, or that your connection to the Internet has failed.
If your CSP fails, then not only are you not able to see your Cloud, your customers and partners can not see your Cloud either. I first wrote about this problem about two years ago, and the basic facts are unchanged. CSPs are in general very reliable and provide high availability. They almost all will “guarantee” 99.9% availability or better, although you must be careful to understand exactly how they are measuring their availability. When they miss their availability commitment, most credit you for some amount of your monthly fee, up to some percentage of that monthly fee usually in the 20% to 100% range. My rule of thumb is that if you can stand being down for a day on rare occasions, with 99% availability to your customers then the CSP’s basic availability should be sufficient. Almost every CSP will offer, for an additional price, to provide automatic disaster recovery to another of their data centers. You can usually specify where the alternate data center is located for your Cloud. Ideally, it should be at least 500 miles away from the primary center, but located in the same country to preserve compliance with governmental privacy laws. Whether you should have a disaster recovery option for some or all of your workloads in the Cloud is a business decision, not a technical decision. Measure the benefits of staying available to your customers against the CSP price for the disaster recovery option, and your own additional costs in terms of the effort to maintain privacy and compliance certifications in that expanded IT environment.
The second likely cause for not being able to see your Cloud is called “the last mile” problem: you have lost your connection from your physical facility to the Internet and thus to your Cloud. The good news is that your Internet presence is likely still visible, and your customers may be able to place orders and get some information. The bad news is that you may have trouble finding out about those orders, and you have lost your communication connection to your customers. It is not just access to your Cloud that has disappeared, but also access to your email and, for many companies, access to your phones.
Where I live, the two biggest ISPs (Internet Service Providers) are Verizon, with FIOS, and Comcast, with Xfinity. Both companies push their bundle of Internet, TV and phone with both residential and business options. Both send me weekly “you should use our bundle” notices via snail mail and email, plus the occasional phone call. I use Comcast for Internet and TV, Verizon for cell and landline (“copper”) phones. Why do I pay the extra each month for both? Neither is very reliable, although in my experience Comcast is a whole lot more reliable than Verizon.
For a simple local outage, I can access the Internet through my Verizon cell phone, and access email and monitor my web site in that manner. I could even take my laptop or tablet to Starbucks, Staples, a local library, or any of a number of other places that provide free Wi-Fi; I just need to be careful from a security standpoint.
I can easily call Comcast from my landline phone, and usually get a real person in a matter of a few minutes. That would be hard to do in a full Xfinity bundle, since your phone would also be out. Verizon is a real problem. It is almost impossible to get to a live person via any means; they really only want you to report problems via their web site. Try doing that if your complete FIOS bundle goes down, taking your phone and Internet with it.
Perhaps the biggest flaw with plans like what Comcast or Verizon offer is that your facility must have power. Verizon, for example, provides battery backup. “A battery unit will supply backup power for standard voice service (but not other voice over IP services), including E-911, for up to eight hours. The customer is responsible for periodically replacing the battery. Certain telephones, answering machines and other telephone equipment not meeting industry standards may not work with service provided on the Verizon network.” Note that this battery backup does not include your Internet or TV connection. Personally, I think 8 hours is woefully inadequate. Recent weather events have left thousands without power for days. That is a long time to go without have access to emergency responders.
Do not expect to be able to make or receive a cell phone call during an area-wide emergency.
Copper phones work even if you do not have power, as long as your phone instrument itself does not need to be plugged in. We always keep a couple of the old-style phones with a phone cord to the wall jack and another between the base unit and the handset. Ugly, clunky, and tied to the wall; but it works fine during a power failure. Your younger employees may not even recognize the thing as a phone.
There is an obvious solution to the “last mile” problem: pay for two different ISPs and make sure there is no single physical location where both ISPs can be impacted by a single failure, along with some means of providing power to the critical components to keep you connected and communicating. Just like for your own IT equipment, the decisions need to be made on a business basis; what is the value to not going down for different applications compared with the cost of the second ISP and backup power. Whether you use an expensive solution or “let’s all go to Starbucks for the afternoon” disaster recovery plan, you need to document the plan, and then test it at least once a year. Part of that plan must be how you will communicate with your critical employees necessary to enable the plan.
Remember that, depending on why you have a “last mile” problem, some critical component of your disaster recovery plan might also be impacted. Does the Starbucks have power? Can you get there through the hurricane, flood, fire, or earthquake damage?
The last word:
About 20 years ago I attended a full-day session on disaster recovery by Pacific Telesis Group (“PacTel”), one of the “Baby Bells” formed during the breakup of AT&T in 1983. In 1997, PacTel was acquired by SBC Communications, but not until its cellular and paging unit was spun off to become AirTouch Communications, which is now part of Verizon Wireless. Yep, them again.
This disaster recovery session was focused on small business and dealt as much with how to keep employees and families safe as to keep the business going in a disaster. At the time I was in San Diego, with boringly wonderful weather punctuated by earthquakes, mudslides, and wild fires that could move faster than you can drive a car. One key point they made: Always make sure you and each employee at home has a landline and a phone that does not require electrical power.
Cell phones do not work in an area-wide emergency. Cell phone connections are severely limited by the number of simultaneous calls each tower antenna can handle. As I have experienced in San Diego in an earthquake and within the past two years in a snowstorm here in Pennsylvania, this problem is real and still exists.
We use Verizon for our landline and cell phones. Verizon is constantly telling us to “move up to FIOS,” their fiber optics phone, Internet and TV solution. Part of that reason is that Verizon has “stopped investing in copper” (their exact words from an engineer and sales manager). In fact, for many copper subscribers, the copper connection only goes from your end point (house or business facility) to the nearest FIOS connection box. In our case, that is a file cabinet sized metal box at the edge of a field about a quarter mile from the nearest road and about a mile from us. From there, the connection is all fiber-optic cable back to the local central office and the rest of the Verizon network. I know this because we recently had an eight-day period without a dial tone. In tracking down exactly what failed, it all started with a failure of the FIOS electronics in that connection box. That took down hundreds of FIOS customers and a dozen or two of us remaining copper customers. When the FIOS tech went out to investigate, he simply rebooted the FIOS equipment. He could not check it out completely since there was no technician at the central office on a Sunday. So much for their 7×24 service. The tech had to go back out Monday, and finally got FIOS back up after a 30-hour outage. However, he did not check out the copper connections. Somehow my trouble report just disappeared in the system, and it wasn’t until I contacted them again, via their web page since they absolutely will not answer their phone, that they actually sent a tech to our house. The tech did not come until three days later; as he said, they want to make sure there is a real problem. They wait two or three days and then robo-call you to see if you still have a problem before they really send a tech. He replaced the board in the connection box that translated copper to fiber, and we had our dial tone back.
I suggest you check with your phone company to see if they have a completely copper network, or have converted everything but the last mile to some different technology. I would ask more than one person; I recently received yet another phone call from a FIOS sales lady who was absolutely convinced that a copper connection was copper all the way.
This is not unique to the Philadelphia area. Verizon is doing this big time in New York state, starting with little Fire Island but also including the Catskills and even into New York City. Check out the Huffington Post article and a portion of the July 7 Le Show by Harry Shearer (28:42 into the show). This move by Verizon, and probably other legacy phone providers, is for more than just cost savings. Verizon believes it can completely abandon most of the current federal and state communications regulations by shifting away from copper. “The phone companies and a group called the American Legislative Exchange Council, ALEC, created ‘model’ legislation that has been burning through the states, removing basic obligations, from quality of service requirements, state commission oversight to even removing ‘carrier of last resort,’ where the company no longer has to provide basic services.”
Remember this also applies to your home connection. In an emergency, with the cell phones clogged, could you still contact help?
Keep your sense of humor.