Y2K and ISPs

This page is being targeted to disaster preparation for ISPs and will probably be expanded. To see the old page, go to Archived ISP/Y2K.

This discussion is oriented towards the smaller ISP, though much of this is relevant to keeping a business LAN connected to the Internet. This page has been revised to delete the specific Y2K coverage that's no longer relevant, since I regard that as of historical interest, the old page is here. Of course, if you, like some other organizations find that your routers, etc. are suddenly having date-related problems, check the old page.

If you're a customer, you might get from this the questions you ought to ask your ISP to see if it's ready for potential trouble. The bad news is that the expensive stuff required to do business if there is serious trouble will probably be out of reach of your Internet Service Provider. Remember that ISP service is now commodity that everyone pretty much pays $20 a month for. Profit margins on $20 a month and in some cases, even lower tend to be razor thin. If you are a business that must have Internet service if there's any Internet to connect to, you might consider making a deal with your provider to provide the hardware and facilities to get it to the compliance level your business requires in exchange for reduced service costs or something else your organization wants. If you think your ISP needs to be in an underground concrete bunker, yuu might be able to sell your ISP on it, but you get to pay for it.

There's a limit to what an ISP can normally do about disasters. To function, an ISP or office LAN needs connectivity to the Internet (broadband links), connectivity to your customers (usually dialup POTS and/or ISDN), power, working servers and software, working network components and software, (bridges, routers,etc.) and if you want to do business, your workstations and larger systems that process billing and other customer interaction had better be compliant. Building services required include climate control (the temperature level computers are most comfortable with is the same one you are; if the temperature hits 100 F (39 C) in the computer room, the service life of the computer is drastically reduced. Then, there's physical security. Reduced police protection is probable in the event of serious Y2K-related trouble. if there's civil disorder outside the place, a mob could do a good job on either the computers or both. However, what an ISP can do about that is pretty severely limited.

Even if a generator backup isn't possible, an inverter / battery bank setup capable of providing a few hours of power (more if monitors not in use are turned off) isn't that difficult or expensive to do. This is perfectly adequate for the great majority of power outages.

ISP Emergency Readiness Outline

Physical connectivity to the Internet
- telco (copper) wired T1-OC5 local loop: your basic connection. While a second loop pointing at an independent feed is a good thing, if your telco broadband connections die, you'll be trying to get to the Net on two dead connects for the price of one.
- telco, etc. fiber optic: I'm not sure if the fiber optic will stay up if the copper wired part of the network goes down, I don't know how many of the connections are in common. Fiber optic is independent of telco if you go with another provider of fiber optic dial tone with its own network. Make sure they aren't using telco local loops to get to the Net if you're using this for improving redundancy.
- cable modems, e.g. @Work While I know of one ISP that uses or used a standard home account to support a few hundred users on a regular basis, I think this can only be justified in the event of emergency. Running several hundred users through a single user account costing $30/month is difficult to justify doing even to your local cable company, as much as you may hate them. More to the point, that kind of scam is likely to be unplugged without any notice or warning, and might be considered illegal theft of cable service. If available, check into @Work. If you do, check into what @Work uses as a backbone provider. The worst news here is that your cable provider or home.com may not be Y2K ready.
- wireless point-to-point - for 10-20 miles, figure $1500 or so for each end. This price may drop if the Tuscon Amateur Radio Society ever manages to get their spread-spectrum transceiver kit complete and ready to sell. While the bandwidth is only 500Kbps, it should be fairly inexpensive. I really like the wireless solution best in a lot of ways.
- satellite: Haven't had a chance to check into the costs yet. If you've got numbers from 128Kbps - OC3 service and bi-directional terminals, please let me know.
- Backbone redundancy - If you've got alternate path(s) to the Internet, they don't do you much good if they connect to the same backbone and it fails, or your separate feed sites connect to a common upstream link between your ISP and a backbone and that link breaks. Check to make sure that this is not true. Specify this in your service agreement if need be.
- Peering Points - In the US, MAE East, MAE West, and the smaller regional providers. Europe, Asia, and other continents and countries have peering points of their own. You probably want to know whether the one you're ultimately connected to has generator backup, and how many days of fuel that generator has.
Connectivity to your customers
- dialup POTS: Make sure your modem pool is connected to the backup power. Not much you can do about telco / local dial tone availability.
- ISDN: It is conceivable though unlikely that ISDN might be working even with the regular dial network down.
- cellular / PCS: It is possible that cell phones might be up with regular dial network down. This actually has happened during earthquakes and other natural disasters. However, this simply means you probably should have one around the office in case you need to call for help, you won't want to plug cell phones into your modem pool.
- wireless point-to-point: see above, you may be working with reduced bandwidths and less expensive transceivers. You might want to look into providing this and setup to your customers for an appropriate fee if you can find some good turnkey packages. Given the problems local telcos have given ISPs over providing DSL connectivity and given the low probabilty that AOL/Time Warner will comply with any merger conditions requiring that they must provide cablemodem connectivity to ISPs, this might be a good thing in any case as an option to provide broadband customer connectivity. If you use a point-to-point to connect to the backbone, telco won't be taking your network down by "accident".
- voice phones: If you have a PBX or the usual business key stations, battery backup is a good idea. Keep a POTS phone, preferably the old fashioned Western Electric type desk/wall phones, lying around just in case your PBX / business phones lose power or take a dive for other reasons. If it has a transformer brick attached, it is not suitable as an emergency phobe.
Power: It is quite possible that your power will be down and some telco will be up. If so, you might as well stay in operation. Communications in a power outage are extremely valuable for things like organizing relief efforts, getting medical advice in emergencies, and even calling for help if 911 is down, as well as keeping critical business operations going. For extended discussion of generators, alternative power, low power/no power lighting, emergency heating etc., go to the Personal Preparation section of my site and to the complete chapters of my unpublished Y2K preparation book
- backups: Are you keeping at least weekly off-site tape (or whatever) backups of your server and business files? If your place burns down and you can get an insurance company setup, you'll be very happy to have these.
- UPS - batteries: These are valuable mainly for dealing with power fluctuations and very short term black/brownouts. If you plan to stay in business during blackouts, see above.
- generators: see above.
- alternative power - For solar, figure about $10/watt. For windmill / hydro / steam (i.e. run your ISP on firewood. No, I'm not kidding.) Economizing on power is the best way to save money on the size of your generating system, but compromising reliability in the process is fairly easy and needs to be watched for. For information on power generation, see above.. If you do decide to try a laptop server server, you're best advised to turn all power management features off, and due to the probability that laptop hard drives are less reliable than regular hard drives, consider running a SCSI/100 Base T PCMCIA card to connect to your network and a SCSI chain running enough standard hard drives in a separate box with a small power supply to support whatever RAID level you can afford and run RAID SCSI compatible software. (I suggest RAID 5.) Put your tape backup in the same box. You should get substantial power savings. Another option might be LCD flat panel monitors and your regular system, but this is rather expensive. However, your most important power savings will be in the area of lighting (recommended: compact fluorescents or LED based fixtures, though oil lamps/ Coleman type lanterns will work in emergencies. The efficiency range of lighting is from 5% (incandescent light bulbs) to 95%+ (LED).
Servers and software. You really ought to read through the entire computer section of my Y2K page after you finish this part of the outline. Make sure that you replace the CMOS or equivalent on your computer (clock/low level configuration) batteries no earlier than 11/1999. And keep spare batteries. And put the configuration info on a floppy.
- Windows NT. The first question is "Have you considered upgrading to Linux?" If you don't have the latest Y2K fixes (i.e. all the patches available) installed, your system isn't Y2K ready. If you've got a Wintel box, even a new one, it needs to be tested at the hardware (BIOS and RTC level) for compliance. Click here for more information. If you're running NT on DEC Alpha, check with DEC. (Compaq) Check your applications and data files, if relevant.
- Unix (all variations including linux) There should be no OS compliance issues unless you're using an older version, in which case you probably want to upgrade anyway. Check with your vendors on the OS and your applications, grep through your data files for two digit date fields. If you're running a unix variant in a Wintel box, check Wintel Hardware. There is other Y2K relevant information in the Other section of my Y2K page.
- everything else: see this section of my Y2K page.
- Network components and software: (routers, switches, internal software) Check with your vendors for Y2K compliance. If you need to download new software to make it compliant, get and install it, but if you make a policy to check your vendor site on a regular basis for the latest upgrades, you're probably already OK.
  The Cisco Y2K Compliance Assessment Tool provides a quick and accurate way to identify the state of compliance of Cisco IOS, Catalyst and WAN switching device software running in your network.
- spare parts: Supply chain disruptions may interfere with getting spares. Try to have at least one extra of everything, more than one extra of things you have quite a few identical iterations of, e.g. network cards, video cards and other spare cards, hard drives, etc. Keep lots of extra tapes or whatever it is you use for system backups. Keep maintenance supplies like tape head cleaners for tape drives, floppy disk drive cleaners, solder, whatever it is you would really miss if you had to go to a computer store to get more and found the store closed for a few months. Maintenance / diagnostic software / hardware for your PC is a good thing. Don't depend on being able to get a hardware technician when you need one if you can avoid it, if anyone on your staff can use things like oscilloscopes and other electronics test hardware, set up an electronics test bench. If you're looking for electronics part / tool information, the electronics section of my home page is a good place to start.
- documentation : Make sure you've got documentation on all your equipment, both hardware and software, run an inventory on everything (create one if you don't already have one) and ensure this, you can do your Y2K assessment as part of the inventory process if you haven't already. If you don't have documentation, get it while it's still easy to do so. Documentation includes docs on motherboards, video cards, and anything else inside your computers.
- Make sure your virus scanners are up to date and keep them that way until the last possible minute before 2000. Hopefully, your virus vendor will still be doing business and offering updates after 12/31/1999, but don't figure on it. It is rumored that there are virus variants set to trigger on or shortly after the rollover, so use those scanners on every downloadable executable or macro file or system , including the ones you buy on CDROM at your software store in shrinkwrap. One good place to get virus scanners which will run on not only Wintel, but Mac and Linus and I think, other -ix variants is McAfee.
Business
- workstations: If you're running a Windows machine that hasn't been Y2K-fixed, it's Y2K broken, you can't depend on the computer or the operating system or the applications or your data files to handle Y2K correctly. Go here for help. If you're running Mac, check your applications and data files. Your applications should be checked at your vendor Y2K pages, I don't know which scan tools for data files run on Mac.
- billing systems (might be midrange / mainframe in larger ISPs) If you're a smaller ISP, this is probably running on your workstations. See above, and pay attention to spreadsheets and database data files. If you're running midrange or mainframe systems and custom applications, there isn't much I can do for you, other than advise checking with your vendors, find a good automated scan tool that'll check your custom code, and start RIGHT NOW if you haven't already. You probably should have started in 1998 or 1997. In general, you'll want to eyeball every single use of date commands in your software. There appear to be ways of writing non-compliant software in all popular computer languages from COBOL, C (all variants), Java, Javascript. Good luck, if you're starting now (this was written in 4/1999) you're really going to need it.
Building considerations
- Climate control: If you're running alternative power, you probably can't afford to run either air conditioning or electric heating. See above for preparation with respect to heating.
- Water / Sewage: If power goes down for any length of time, your water and sewage go with them. If you're still trying to do business, i.e. if telco is still up, you'll need stored water and some method of disposing of sewage. Check the Personal Preparation section of my Y2K page.
- Physical security
  - burglar alarm: Make sure it's got a hefty battery backup. If you're using a system that reports to a central monitoring station, make sure that vendor is Y2K compliant. If you can't get the right straight answers out of them, find another vendor. Of course, if telco goes down, chances are, so does your access to your alarm company.
  - physical measures: Steel doors, with high-quality locks, are a good idea. Being able to board up windows or glass doors is a good thing. Concrete buildings are also a good thing.
  - Of course, the ultimate backup to your physical security is your insurance company. Check their Y2K compliance, and have fun reading their Y2K compliance statements.
Personnel considerations
- On duty for the rollover? Having some senior level people available at your ISP for the rollover is a good idea. I hate to say this, but you probably should limit the alcohol consumption at this New Year's company party. If you're in an urban area, having an armed person or two might be a good idea, and that person should avoid the booze. I agree that this sucks.
- After the rollover depends on what happens. If rioting mobs burn down the city your ISP is in, it isn't likely you'll have many customers left or that the ones that can connect to you have a reasonable expectation of continued service. If you're a regional provider, though, your out of town customers will not care if your city was burned to ashes, they are still going to expect you to be up short of a nuclear detonation. Also remember that you may have emergency traffic running through your system, from disaster management agencies to people trying to find out about the safety of their loved ones. If there are transportation problems, the Internet may be more important than ever for business communication.
- One possible thing you might consider is to do a company sponsored Y2K preparation program, group purchases of Y2K supplies, etc. See above for more information about that.
Note that SBA loans are available for Y2K remediation and upgrading. Click for information. There is also some special tax treatment for Y2K preparation, click here for information on both and other information available for small business Y2K in general.
Finally, in a disaster, what can you do to help? Check this article out about lessons learned at the Los Alamos fire where the local ISP community came together and put together online clearing houses for disaster-related information almost instantaneously.