1. Physical connectivity to the Internet
   • telco (copper) wired T1-OC local loop: your basic connection. While a second loop pointing at an independent feed is a good thing, if your telco broadband connections die, you'll be trying to get to the Net on two dead connects instead of one. Look for a backup connect to your feed that does NOT go through telco.
   • telco, etc. fiber optic: I'm not sure if the fiber optic will stay up if the copper wired part of the network goes down, I don't know how many of the connections are in common. Fiber optic is independent of telco if you go with another provider of fiber optic dial tone with its own network. Make sure they aren't using telco facilities to get to the Net if you're using this for improving redundancy.
   • cable modems, e.g. @Work While I know of one ISP that uses or used a standard home account to support a few hundred users on a regular basis, I think this can only be justified in the event of emergency. Running several hundred users through a single user account costing $30/month is difficult to justify doing even to your local cable company, as much as you may hate them. More to the point, that kind of scam is likely to be unplugged without any notice or warning, and might be considered illegal theft of cable service. If available, check into @Work. If you do, check into what @Work uses as a backbone provider. The worst news here is that your cable broadband access provider may depend on having local utility power continously available.
   • wireless point-to-point - for 10-20 miles, figure $1500 or so for each end. This price may drop if the Tuscon Amateur Radio Society ever manages to get their spread-spectrum transceiver kit complete and ready to sell. (the kit looks like about $350 from what I can see, if it is ever available) While the bandwidth is only 500Kbps, it should be fairly inexpensive. I really like the wireless solution best in a lot of ways. Note that an Ethernet 802.11 wireless connection will work over long distances using high-gain antennas at either or both ends. High gain microwave antennas aren't that difficult to build (Check the ARRL Antenna Handbook for designs.) and if your people have RF background, this is worth thinking about. See wireless in the Customer Connectivity section below.
   • satellite: Haven't had a chance to check into the costs yet. If you've got numbers from 128Kbps - OC3 service and bi-directional terminals, please let me know. While DishNetwork is available everywhere in the US, you would have to make special arrangements with them to plug your ISP into them for an emergency connection. Also, latency is much longer in a satellite connection, running one means adding at least 50,000 miles (1/3 second each way) to response times, meaning that real-time video and/or VOIP gets painful to use.
   • Backbone redundancy - Keeping your computers running when the lights go out doesn't help all that much if your feed to the Net went out with the lights. If you've got alternate path(s) to the Internet, they also don't do you much good if they connect to the same backbone and it fails, or your separate feed sites connect to a common upstream link between your ISP and a backbone and that link breaks. Check to make sure that this is not true. Specify this in your service agreement if need be, and make sure at least one of your paths has emergency power available for all the nodes between you and the backbone and the backbone site as well.
   • Peering Points - In the US, MAE East, MAE West, and the smaller regional providers. Europe, Asia, and other continents and countries have peering points of their own. You probably want to know whether the one you're ultimately connected to has generator backup, and how many days of fuel their generators have.
2. Connectivity to your customers
   • dialup POTS: Make sure your modem pool is connected to the backup power. Not much you can do about telco / local dial tone availability.
   • ISDN: It is conceivable though unlikely that ISDN might be working even with the regular dial network down.
   • cellular / PCS: It is possible that cell phones might be up with regular dial network down. This actually has happened during earthquakes and other natural disasters. However, this simply means you probably should have one around the office in case you need to call for help. You won't want to plug cell phones into your modem pool.
   • wireless point-to-point: see above, you may be working with reduced bandwidths and less expensive transceivers. You might want to look into providing this and setup to your customers for an appropriate fee if you can find some good turnkey packages. Given the problems local telcos have given ISPs over providing DSL connectivity and given the low probabilty that AOL/Time Warner will comply with any merger conditions requiring that they must provide cablemodem connectivity to ISPs, this might be a good thing in any case as an option to provide broadband customer connectivity. If you use a point-to-point to connect to the backbone, telco won't be taking your network down by "accident".
   • voice phones: If you have a PBX or the usual business key stations, battery backup is a good idea. Keep a POTS phone, preferably the old fashioned Western Electric type desk/wall phones, lying around just in case your PBX / business phones lose power or take a dive for other reasons. If it has a transformer brick attached, it is not suitable as an emergency phobe.
3. Power: It is quite possible that your power will be down and some telco will be up. If so, you might as well stay in operation. Communications in a power outage are extremely valuable for things like organizing relief efforts, getting medical advice in emergencies, and even calling for help if 911 is down, as well as keeping critical business operations going. For extended discussion of generators, alternative power, low power/no power lighting, emergency heating etc., go to the Personal Preparation section of my site and to the complete chapters of my unpublished Y2K preparation book or my recent article at 8wire on power protection.
4. In the long run, the best power backup solution will probably be fuel cells burning natural gas / propane. These units will supply several kilowatts of power indefinitely or as long as the fuel holds out, they are both quiet and safe. While you will probably still need UPS units on critical systems (in case something goes wrong with the fuel cell or in case the transfer relay is a bit slow), these units should not require massive battery banks. However, it will probably be a year or two before these units become generally available, these systems are mostly still in the R&D stage. You might consider applying to become a beta test site for one of the manufacturers. (I'll look this up when I have time, if you've got an immediate problem, catch me in e-mail.)
5. backups: Are you keeping at least weekly offsite tape (or whatever) backups of your server and business files? If your place burns down and you can get an insurance company setup, you'll be very happy to have these. The farther away your offsite backups are stored, the better, remember that you can get them back from anywhere in the US via Fed Ex. If your pipe to the Net is big enough, offsite backup to Web backup service providers might be a viable strategy, my general advice to avoid these services only applies to people with dialup links or enough data storage requirement that an image backup will take longer than overnight. If you're running T-3 or above to the backbone, a Web service provider might make a lot of sense. However, test this before you need it, between you and your Web service provider might be a node that might slow your 4.5 million character datastream to dialup speeds. Anybody who's been alone on a T-1 or larger link and watch a Web page download at 2 cps knows what I mean.
6. UPS - batteries: These are valuable mainly for dealing with power fluctuations and very short term black/brownouts. If you plan to stay in business during blackouts, see above. I've had good experiences with the Belkin PRO GOLD SERIAL UPS 425VA unit. (model F6C425-SER for RS232, also available for USB connection) Good voltage regulation (+/- under 10%), software for monitoring including remote Web monitoring / automated shutdown / e-mail and pager notification. Also, software versions available for Wintel / Mac / various -ix flavors.
7. generators: see above.
8. alternative power - For solar, figure about $10/watt. For windmill / hydro / steam (i.e. run your ISP on firewood. No, I'm not kidding.) Economizing on power is the best way to save money on the size of your generating system, but compromising reliability in the process is fairly easy and needs to be watched for. For information on power generation, see above. Good starting points for information on this are the vendor sites Jade Mountain and Home Power Magazine.
9. How to save power: Remember that every watt you don't burn is one you don't have to provide via batteries. Your most important power savings will be in the area of lighting (recommended: compact fluorescents or LED based fixtures, though oil lamps/ Coleman type lanterns will work in emergencies. The efficiency range of lighting is from 5% (incandescent light bulbs) to 95%+ (LED). LCD flat panel monitors are also good power-savers but this is rather expensive. You already know what your power company has to say about setting thermostats, etc. During cold weather, you might want to get space heaters (propane, heating oil, etc.) unless you're running a large generator. A laptop-based server could save substantial power, but there are certain problems with the idea. If you do decide to try a laptop server, you're best advised to turn all power management features off, and due to the probability that laptop hard drives are less reliable than regular hard drives, consider running a SCSI/100 Base T PCMCIA card to connect to your network and a SCSI chain running enough standard hard drives in a separate AC-powered box with a small power supply to support whatever RAID level you can afford and run RAID SCSI compatible software. (I suggest RAID 5.) Put your tape backup in the same box. You should get substantial power savings.
10. Servers and software.
    • This, of course, is critical. You'll want at least a few hours of power backup here, since if these don't work, neither do your customers. CHOOSE POWER-SAVING MODES on the server with extreme care. In particular, do not save power by letting the hard drive(s) shut down when idle. However, the monitor is always a good candidate for automatic power shutdown. Remember that power-saver modes for Wintel boxes can usually be found both in the OS and often, in the BIOS, even if your Linux distribution doesn't support power saving, your BIOS usually will.
    • spare parts: Supply chain disruptions may interfere with getting spares. Try to have at least one extra of everything, more than one extra of things you have quite a few identical iterations of, e.g. network cards, video cards and other spare cards, hard drives, etc. Keep lots of extra tapes or whatever it is you use for system backups. Keep maintenance supplies like tape head cleaners for tape drives, floppy disk drive cleaners, solder, whatever it is you would really miss if you had to go to a computer store to get more and found the store closed for a few days or weeks. Maintenance / diagnostic software / hardware for your PC is a good thing. Don't depend on being able to get a hardware technician when you need one if you can avoid it, if anyone on your staff can use things like oscilloscopes and other electronics test hardware, set up an electronics test bench. If you're looking for electronics part / tool information, the electronics section of my home page is a good place to start.
    • documentation : Make sure you've got documentation on all your equipment, both hardware and software, run an inventory on everything (create one if you don't already have one) and ensure this. If you don't have documentation, get it while it's still easy to do so. Documentation includes docs on motherboards, video cards, and anything else inside your computers.
    • Make sure your virus scanners are up to date and keep them that way. Use those scanners on every downloadable executable or macro file or system , including the ones you buy on CDROM at your software store in shrinkwrap. One good place to get virus scanners which will run on not only Wintel, but Mac and Linus and I think, other -ix variants is McAfee.
11. Business
    • workstations: Plug the ones you expect to be using during a power failure into the power backup. Use the "green" power saving modes for the monitor and for the CPU box, they'll buy you lots of battery backup time. As for the ones that won't be used, the main consideration is orderly shutdown in the event of power failure.
    • billing systems (might be midrange / mainframe in larger ISPs) If you're a smaller ISP, this is probably running on your workstations. See above, and pay attention to spreadsheets and database data files. The main consideration here is orderly shutdown. Consider a separate UPS that doesn't draw power from the main battery backup if you are using a battery backup system intended to run "everything".
12. Building considerations
    • Climate control: If you're running alternative power, you probably can't afford to run either air conditioning or electric heating. See above for preparation with respect to heating.
    • Water / Sewage: If power goes down for any length of time, your water and sewage go with them. Or an earthquake might simply take out the sewage and water pipes. If you're still trying to do business, i.e. if telco is still up, you'll need stored water and some method of disposing of sewage. Check the Personal Preparation section of my Y2K page.
    • Physical security
      • burglar alarm: Make sure it's got a hefty battery backup. If you're using a system that reports to a central monitoring station, make sure that vendor has backup power. If you can't get the right straight answers out of them, find another vendor. Of course, if telco goes down, chances are, so does your access to your alarm company and their access to the police, though they may be able to contact private security vehicles via radio, though without backup power on their part, this is irrelvant.
      • physical measures: Steel doors, with high-quality locks, are a good idea. Being able to board up windows or glass doors is a good thing. Concrete buildings are also a good thing.
      • You do have one or more fire extinguishers (if you don't, what are you waiting for?), and if you've got a server room, you might think of a Halon or CO2 based automated fire extinguisher system.
      • Of course, the ultimate backup to your physical security is your insurance company, but not only will having preparations in place reduce your risk and your rates, but your offsite backups should get you running as soon as your vendors get you the replacement systems your insurance claims paid for.
13. Other Emergency / Disaster Considerations
    • Once you have your preparations in place, work out a written disaster / emergency procedure and put it in a binder whose location is known to everybody. (What has to be done, who has to be called, contingency plans, etc.)
    • Civil Disorder: If rioting mobs burn down the city your ISP is in, it isn't likely you'll have many customers left or that the ones that can connect to you have a reasonable expectation of continued service. I recommend either locking up the place and getting out or if you enjoy a bit of personal risk, bring in a few guns, lots of ammo, a few non-lethal items like stun guns and pepper spray, tear gas grenades if you can get it legally, and a videocam or two to provide evidence that your people were justified in opening fire when necessary. If you've got Fortune 500 backing, look into a Class I Federal Firearms License, that covers possession and use of automatic weapons. Very few mobs or individual looters want to deal with armed machine-gun toting crazies for some reason. If you decide to acquire weapons, get training in their use, whatever those weapons are.
    • If you're a regional provider, though, your out of town customers will not care if your city was burned to ashes, they are still going to expect you to be up short of a nuclear detonation. Also remember that you may have emergency traffic running through your system, from disaster management agencies to people trying to find out about the safety of their loved ones. If there are transportation problems, the Internet may be more important than ever for business communication.
    • Finally, in a disaster, what can you do to help as an ISP? Check this article out about lessons learned at the Los Alamos fire where the local ISP community came together and put together online clearing houses for disaster-related information almost instantaneously. This is not only good for the community, but good public relations as well. The business or government agency you gave special help to is also a potential future customer or likely to put in a good word for you with potential customers. Having friends is a good thing.
    • It might be smartest with respect to helping as an ISP if you are proactive, as in contact your local emergency management agencies and/or city government and let them know that you have backup power and redundant Internet backbone connections, and plan to stay connected to the Internet during emergencies and disaster, and ask how you can help out. Note that if local governments have a personal interest in keeping your company running during disasters, you might be able to borrow generators or get priorities on fuel or get access to government-based emergency Internet backbone connections or even extra police protection if need be.

You can contact me at alizard@ecis.com. If you have suggestions for improvements for this page, if you see problems, etc., do please contact me.