Business Continuity For Administrators
A Presentation given by Dermot S. L. Butler at the
"Hedge Fund Services Summit"
held at the Crowne Plaza Hotel, London
Good morning ladies and gentlemen.
I am going to discuss Business Continuity for Administrators
In fact, I am going to discuss Business Continuity for any of us here but from my perspective and experience.
This is an important topic, which, like many such topics has come to the top of the pile, in priority terms, because of some extraordinary or disastrous event. At least one such event seems to happen every year.
Let me give you some examples of the sort of topics that have become the Flavour of the Month, over the past few years, just in the context of Hedge Funds:
1. Transparency, which is now a much-discussed topic, the importance of which, I suggest, was exacerbated by the Long-Term Capital Management debacle;
2. Another example is the attention now being paid to the Administration of Hedge Funds, such as this very conference. This is, I suggest, a direct result of the various hedge fund frauds in the United States over the past couple of years, where many, if not most, hedge funds are self-administered; - by that I mean administered by and the financial books kept by the Funds manager.
3. Thirdly, the demand by investors for assurance that administrators use Independent Price Sources and Independent Data Feeds, stressing Independence. I suggest that this is a direct result of the Manhattan fraud. For those of you who dont know what I am talking about the Manhattan Case was one in which the Funds manager sent forged statements, purporting to come from a prime broker, to the administrator and falsified the funds returns;
4. Fourthly, the proliferation of Computer Viruses I always want to say Virae which shows not only that I had a classical education, but also how long ago I went to school. Be that as it may, headline viruses, such as Melissa, the Love Bug and NIMDA have, if nothing else, ensured the future of the leading firewall and anti-virus programme companies.
5. Finally, of course, the current interest in Business Continuity and Disaster Recovery Plans is a direct result of the World Trade Center tragedy.
When I knew I was going to give this presentation today, I had a conversation with a friend who had attended a Business Continuity Seminar, at which the first speaker was a senior officer of the Metropolitan Police Force. This fellow opened his speech, somewhat dramatically, by showing the audience a lengthy tome, explaining that it was a Disaster Recover Plan.
He followed up that explanation by throwing it into the wastepaper basket, saying that no disaster recovery plan is worth the paper its written on, unless the organisation has a leader, who can and will take charge in the first hour following the disaster.
He went on to tell a story about the awful Piper Alpha oil rig disaster in the North Sea several years ago. Apparently, according to our fellow in blue, the leader, who it appeared on this occasion was, surprisingly, a chef, who grabbed a group of workmates, made them all hold hands and then got them to jump into the sea, which was a very gutsy thing to do, when you remember the height of the Piper Alpha rig. Anyway, they were all rescued.
The irony was that the majority of those who didn't jump, but who stuck to the Disaster Recovery Plan and procedures, perished.
Now this, of course, is a very dramatic example of an extraordinary and unique disaster and not one that any administrators are likely to experience - or so we would have thought before September 11th. On that awful occasion, the "leaders" would have had less than an hour to carry out an effective decision and the only decision that worked was to evacuate immediately.
I would not go so far as to say that you should dump your Business Continuity Plan, but I would look at it again to ensure that your Business Continuity Plan - which incorporates the Disaster Recovery Plan - covers every conceivable, and indeed, perhaps inconceivable, type of disaster. It must be a detailed plan with detailed procedures for carrying out the plan. You must select personnel for specific responsibilities and provide adequate back-up; you must provide effective training; and you must test the plan.
Inevitably, there are some things that you cannot plan for, such as a massive loss of life as a result of an act of terrorism. I regret to say that, although it is unlikely, this is, nevertheless, a realistic possibility in Dublin and will remain so until there is a genuine and long-lasting peace in Northern Ireland. Of course, many of those of you who have worked in or near the City and Docklands in London, may have firsthand experience of these types of disaster.
As I have already said, Business Continuity is not just another name for Disaster Recovery the Disaster Recovery Plan is one component, albeit a very important component, of the Business Continuity Plan. Therefore, I will address the Disaster Recovery Plan first, and explain what we, at Custom House, have done and why.
We have set up a Disaster Recovery Site in a purpose built facility about 12 or 15 miles due north of Dublin. We selected somewhere that distance from the office, obviously, because the facility was where it was, but also because it seemed a sensible compromise between being too far away for quick access and far enough away from our present office to be immune from contagion by the disaster that had struck us. We took the morbid view that, if the disaster was a nuclear one, then the Disaster Recovery Site would probably be redundant anyway.
We could have moved about the same distance South of the city, but decided to go North because:
* Firstly, our existing office is on the North Bank of the Liffey and we thought that, if a disaster struck, such as a gas explosion, a massive fire or a terrorist attack, the bridges across the river would be grid locked, as would Central Dublin traffic, which is focused predominantly south of the river. We would have to get through that jammed traffic to get to the site and, on a normal day, in Dublins new era of prosperity, crossing the city can take over an hour. So relative ease of access was an initial influence in the choice of sites;
Secondly, a high proportion of our staff live on the North Side;
* Thirdly, it was a better site; and
* Fourthly, it was relatively economical compared with other sites. I say relatively economical because obviously, this sort of facility costs a lot of money but, as with some insurance premiums, even if the protection provided is expensive, the cost is negligible when compared to the cost and potential loss of not being able to handle a disaster. That cost could be terminal.
* This leads to the fifth consideration, which is the reputation and financial standing of the operator of the facility. This could be critical. Recently, one of the higher profile operators in the UK filed for administration the UK equivalent of Chapter 11 due diligence always pays off.
We have now contracted to take up, at a moment's notice,forty fully equipped desks that are configured to operate exactly as our existing network. The operators claim that this facility can be up and running in four hours, so we, more cautiously, anticipate that we can be up and running in six hours.
We recently held our first systems functionality test, with one of our teams and everything went exceedingly, perhaps surprisingly, smoothly. We plan a company wide test shortly. Thereafter, we will hold tests with staff at least once a year and probably more frequently. Regular tests are essential, to avoid complacency, as well as cobwebs and rust in the procedures.
We have installed a lease line for permanent data communications between our existing office and the Disaster Recovery Site. We have also installed a new main server on the site, so we are now able to back up our entire existing network data onto the offsite server, every seven minutes so that, if a disaster strikes, we can only lose, at a maximum, seven minutes work.
One bonus, if you like, of this arrangement is that we now use the offsite server as a backup for our onsite server, so that if, for any reason, our main server in our office crashes, the offsite server can take over, without a break.
Other Problems and Risks
Of course, Disaster Recovery Sites only come into their own when a business cannot use its existing premises, for whatever reason. However the Business Continuity Plan must take into account problems that can disrupt, even cripple, your business, but do not necessarily require moving offsite. Such problems could include, inter alia:
1. 'Natural' Disasters
-Fire
-Flooding
2. Technical Faults
-Power failure
-Circuit and terminal failure
-Computer data and electronic communication failure
-Voicemail and telephonic failure
-Computer hardware problems
3. External Technical Problems
-Viruses
-Computer Hackers
4. Political Problems
-Demonstrations and civil strife
-Bomb alerts and other threatening calls
-Transport Strikes
5. Staff Problems
-Staff turnover
-Illness especially epidemic type illnesses, like flu
-Disgruntled employees
6. Legal and Regulatory Issues
-Changes in domestic regulations
-Changes in international regulations
-Deep pocket attacks
-Class Actions
7. Information Storage
- Archiving
8. Financial and Operational Controls
As I've said, this is not an exhaustive list. What you have to do is to go through and identify every conceivable risk that you can think of, that could disrupt your business, and, once you have prioritised that list, which is not necessarily that easy to do, you must then work out a plan as to how to handle each particular problem in the different circumstances under which they might occur.
Financial and Operational Controls
When we did our risk assessment, we identified over 60 different risks, many of which were risks related to operational and financial controls - the last item on this list - which most people don't associate with Business Continuity. However, weak financial and operational controls are a very real risk and could affect Business Continuity. I believe that this particular aspect demonstrates that Business Continuity, like health, can be maintained in many ways by following the old adage of "prevention is better than cure" - or to put it another way, make sure that the stable door is bolted before the horse makes any attempt to get out.
Financial controls may not stop the master crook - that is often very nearly impossible to do, but effective controls and oversight, which make the environment too difficult for the opportunistic thief to operate, can prevent fraud or theft by eliminating the temptation.
We all assume, on the basis of what we read in the papers, that the Allfirst debacle was possible because of inadequate controls at company level and parent company level. I would also suggest that Allfirst was an example of not learning by experience, even other people's experience, such as Barings. AIB was relatively lucky, because they only lost half of one year's profits and the business continues Barings were not so lucky, because the business failed. This is a perfect example of the importance of financial controls in the context of Business Continuity planning.
Moving onto the more commonly perceived Business Continuity risks, I propose to go through most of those on the list that I have just highlighted and comment on how we have handled them.
Tailored Plan to Suit Your Business
I would also like to say that I think of Business Continuity planning in a way, as the artistic application of science, rather like tailoring, because each business is different - and each administrator is different to another and will have its own problems and its own priorities. This could be a function of size and money, or both.
For example, within the Citibank group's Business Continuity processes, they monitor political change around the world. This seems like a very sensible precaution for an organisation with the largest proprietary branch network in the world, but for Custom House, and most of us here, I suggest that would be "over-egging the pudding". Nevertheless, in the context of regulatory and political change, we do have to monitor what is going on in the financial centers in which we operate including, the BVIs, the Bahamas and Bermuda, as well as the Cayman Islands.
Natural Disasters
Moving onto the more mundane topics on the list.Firstly, "Natural" Disasters - in this I have included fire and flooding, which seem to occur more often out of business hours than during business hours.
Fire
With regard to fire, obviously, if you have a fire which incapacitates the office, then you immediately move into the Disaster Recovery mode and proceed from there. So the point is, what can you do to prevent fire? I suspect, although I have not heard any statistics on this, that the laws relating to the prohibition of smoking in offices has reduced the number of office fires substantially. I would also expect that most fires, that aren't caused by arson, are caused by some form of electrical or equipment failure. Obviously you should do all you can to prevent that by maintaining the equipment on a regular and efficient basis.
Similarly it is essential that you ensure that your office complies with all building regulations in the context of fire prevention, because if you should have a fire during office hours and if anyone gets injured, or worse, and you have not covered that aspect, then you are liable to be sued and the damages alone could put you out of business. The only thing is to follow the regulations and comply with them, perhaps 110% - i.e. do a bit more than they ask in the interests of protecting your staff, as well as yourselves.
Flooding
From our point of view, flooding is not a major concern, even though we occupy three four-story over-basement buildings right beside Dublin's river, the Liffey. This is because we "tanked" the basements, which means we lined all of the walls and floors with steel sheeting and concrete and installed a drainage pump system under the new floors. We did this precisely because of the vulnerability of these basements to flooding during high tides. We recently witnessed record high tides in Dublin and the Liffey broke its banks in a couple of places, but our basements remained dry, whilst our neighbours were awash.
I assume many of you will not have this particular problem, however, you may be vulnerable to water damage, if not by flooding, then perhaps as a result of some plumbing problem. Therefore, you should always take the obvious precautions - nothing stored directly on the floors, etc.
Technical Faults
Power Failure
We have huge generator, about the size of a Transit van, which can provide enough power for all three of our buildings. In the event of a power outage, this generator kicks in automatically and immediately. We test this at least once a week, either by substituting the generator for the mains power in an individual building for a few hours, or in all three buildings.
Circuit and Terminal Failure
We are supported by back up, but I am told that 99% of those failures are resolved by changing the fuse.
Communications Failure
These include both lines that permit data feeds and Internet access, as well as voicemail. As far as data feeds goes, we have two lease lines, which is overkill, but it gives us the second line in the event of a problem with the first line.
Telephones
Our telephone systems are provided by an independent company, but they use Eircom (the Irish national telephone service) telephone lines. If the independent company fails, we are automatically switched into the Eircom network.
Our communications hardware is provided by a group who give us a guaranteed four-hour service to repair or replace any broken or faulty parts, under our system service agreement. To date, we have not had many causes to call on them, and they have yet to let us down.
Our main risk in this area is the complete collapse of Eircom, which would be like a complete collapse of BT in the UK or perhaps AT&T in the USA. But this is possible, so we are investigating and pricing the installation of, either two satellite lines, or an independent lease line, fixed through a wireless point-to-point arrangement on the roof. At the moment we are maybe slightly vulnerable, in the event of the national telephone system collapsing - but only very slightly.
Computer Hardware Problems
In the context of computer network hardware crashing, as I have already mentioned, we have the back-up Disaster Recover Site, which covers us.
External Technical Problems
Next, external technical problems, by which I mean the effect of viruses and computer hackers.
Viruses and Hackers
This is a much-publicised subject and not one that will go away. There seems to be a breed of computer super-literate oddballs, whose joy in life is to cause as much disruption of other people's lives as they can, often with no opportunity for profit - it's their idea of fun.I think of them rather like intellectual football hooligans. Of course, there are the super crooks, who try to hack into the mother-load, such as the fellow who diverted just a few cents on every transaction of the daily turnover of a major bank and built up an account with millions of Dollars in a very short time. In this case, he proved that intellect is not necessarily a reflection of intelligence, because, if I recall correctly, he opened account at the same bank and so, when they discovered the scam, they froze the account.
Terrorist Attacks
But this pales into insignificance when one thinks of the chaos that could be caused by a well organised terrorist or anarchistic group, whose intent is massive worldwide disruption. Every time I hear of a new super virus, it crosses my mind that it's the 14-year-old son of Ossama Bin Laden at work.
We have all heard of the viruses, called I believe "worms", which sneak into your address book and replicate messages which are then sent out to everybody on that address list. Once they have got into the recipient's system, they then sneak into their address book and do the same thing over again. Even in a relatively small company, with, say, 50 employees, this could amount to several hundred thousand messages in a very short time, totally jamming up the company's network system and effectively blocking both normal outgoing and incoming Internet traffic.
Finally, in my pessimistic and gloomy mode, I ask you what would happen to international business, which today relies on the Internet - I give it ten years before many, if not most, people have forgotten how to use a pen - anyway, can you envisage the chaos if the Internet system was crippled. Of course, it wouldn't be chaos, which implies some movement, it would be like a seizure - as if the world of finance experienced a massive heart attack. I also dread to think what would happen if the banking system was crippled, perhaps by a systematic failure in the international bank settlement process - another of my perennial nightmares - the whole world of commerce would grind to a halt in hours.
I'm probably just getting paranoid and, no wonder that I might be paranoid, when you remember that our company is a specialist hedge fund administrator and the most effective virus last year - by effective, I mean destructive - was "NIMDA", which is "ADMIN" spelt backwards.
In order to avoid these problems, both with regard to hacking and viruses, we have very extensive and expensive firewall protection and we subscribe to anti-viral programmes, which are reviewed and updated almost constantly. For example, I, personally, cannot receive an e-mail from a third party, with an attachment, without that attachment being reviewed and checked for a virus, on an individual and specific basis, by our IT department.
At the time of Y2K scare, at the turn of the Millennium in December 1999, we switched off our external computers on Christmas Eve and switched them back on again on January 5th, 2000, not because we thought that we had any Y2K problems - that had been eliminated because we had completely re-equipped, replacing all our hardware in November 1999 - but because I was nervous that the Millennium was the perfect time for some "nutter" to release, what, at that stage, could have been, the ultimate virus and we wanted to avoid being vulnerable at all.
In the short time since then, firewall and anti-viral protection has advanced by leaps and bounds and we havenot been adversely affected by any virus since the Love Bug, which did get in to our system for a short while - less than a day before it was cleaned out. The irony was that the virus was transported into our system from an e-mail from our computer suppliers at the time, Gateway.
We believe that we are now fully protected but, again, its something that you cannot be complacent about, because anti-viral protection is only as good as the last virus it identified. Suffice it to say that, over a recent two-week period, our anti-viral programmes identified over 100,000 viruses attempting to get into our system. That is not a reflection of a specific attack on Custom House, but is just an example of the amount of activity and the number of viruses that are out there and, no doubt, many of the 100,000 attacks were by the same viruses.
Political Problems
Demonstrations, Civil Strife, etc. These are really matters for following procedures. - How do you handle a bomb alert? - How do you handle a threatening telephone call?
What procedures do you have for getting staff in and out of the office in the event of a transport strike? The latter, for example, is just a matter of organising car pools, or taxis and providing or paying for parking.
As with all sections of your Business Continuity Plan, in the end, it all comes down to well tested procedures and training.
I cannot stress that enough.
Staff Problems
Staff Turnover
Staff problems can be difficult, particularly in the context of staff turnover. In Ireland, from 1999 to almost this year, the scarcity of qualified and intelligent staff was a daily talking point. In fact, many people are under the impression that this was predominantly an Irish problem, whereas this malaise infected the whole international financial services industry, partially as a result of the "dot.com" hysteria. Certainly in Ireland that has changed, but staff turnover was something that should have been considered within a Business Continuity Plan, as a reasonably high priority. That has now fallen to a lower priority, but is still important. If a member of your staff leaves, there is undoubtedly a stressful period for other members while that person's duties are reallocated, pending the arrival and training of a new member.
Vacations and Illness
Similarly, and this is all going to sound obvious, but, you need to have plans in place to cover vacations and in our business in particular, study leave and exams, and illness, especially epidemic type illnesses like flu - we offer the flu jab, by the way. This all seems like standard operating and business matters, but, nevertheless, they should form part of the Business Continuity Plan.
Disgruntled Employees
Sometimes, you are bound to have unhappy employees who leave under a cloud and who believe they have been hard done by. If you don't have very strong internal security, particularly with regard to the computer network, such a disgruntled employee could cause havoc. This means that you must have a very tight policy with regard to e-mails and the Internet.
Legal and Regulatory
As I have already mentioned, we keep a close watch on changes in regulations relating to our business, both in Ireland and in the jurisdictions in which our client funds are domiciled. I am not going to elaborate on this, but I do want to say a word about legal actions.
In this litigious age all of us are vulnerable to litigious clients, or litigious investors and, indeed, anyone with a grudge who believes that the service provider must have "deep pockets". We have also seen numerous class actions, including our client funds - or rather companies in which our clients have invested - which are often settled on a nuisance value basis. Stupid as it may seem, in this day and age, as an administrator, we could be sued for not encouraging the funds to take part in such an action. The main point here is to make sure that your insurance cover is adequate.
Information Storage
I have already discussed backing up the data on the network, on a regular basis, which is, in itself, information storage. Nevertheless, as information becomes dated - the NAV calculations for past years, for example - it should be stored offsite on either disc or in hard copy format. We are legally obliged to keep documents for 7 years, so the procedure that should be followed in the "prevention rather than cure" attitude, is regular and comprehensive archiving of hard copy documentation and files.
Because the work that we do often involves documentation with original signatures, including, for example, Board Minutes, subscription application forms and redemption notices, we have a process where such documents are scanned and filed on disc. This is partially because it makes it easier to forward copies of documents to interested parties, but also because it means that we have a copy of the original on file, as well as on memory, in our computer network. The original can then be sent offsite to a secure document archiving center.
Of course, there is a risk that that archiving center could suffer a disaster, but in the end, you have to live in the hope that all of the gods are not against you and your Disaster Recovery and Business Continuity planning should be prudent, well thought out, but not paranoiac.
Summary
In summary, I think that the most important factors that should be considered when preparing your Business Continuity Plan are:
1. Identify and prioritise the risks;
2. Work out how best to prevent, rather than cure, the problems;
3. Work out a procedure for the most efficient and effective handling of each problem, should it occur;
4. Train everybody carefully and completely in all of those procedures;
5. Train them again;
6. Test them;
7. Test them again;
8. and do not be frightened of getting help from professionals, after all, you will happily pay for an outside contractor to install your fire alarm system;
9. Finally, make sure your insurance premiums are up to date;
10. Then tell everyone that you are open for business.
11. For example, it may take a day or two to get your telephone numbers forwarded to the Disaster Recovery Site so you need to communicate with every client.
Thank you.
Dermot S.L. Butler is Chairman of Dublin-based Custom House Administration & Corporate Services Limited ("Custom House"), a company that specialises in assisting clients in the organisation, establishment and administration of alternative investment and hedge funds. Custom House is authorised by the Financial Regulator, (formerly the Central Bank of Ireland), under the Investment Intermediaries Act, 1995.
Fri 22.Mar