Teaching Case

Bank SolutionsDisasterRecoveryand Business

Continuity:ACaseStudyfor CSIA 485

SteveCamara

Senior Manager, KPMG LLP

1021 E CaryStreet, Suite2000

Richmond, VA23219 scamara@kpmg.com

Robert Crossler Vishal Midha Assistant Professor

ComputerInformation Systems

TheUniversityof Texas– Pan American recrossler@utpa.edu, vmidha@utpa.edu

Linda Wallace

AssociateProfessor

AccountingandInformation Systems VirginiaTech wallacel@vt.edu

ABSTRACT

DisasterRecovery andBusinessContinuity(DR/BC) planningis anissue thatstudents willlikely come incontactwithas they enterindustry. Many differentfieldsrequirethisknowledge,whetheremployeesareadvisingacompanyimplementinganew DR/BCprogram,auditingacompany’sexistingprogram,orimplementingand/orservingasakeyparticipantinacompany program. Oftentimesintheclassroom itisdifficulttofindrealworldpracticeforstudentstoapply thetheoriestaught. The informationinthiscase providesstudentswithrealworlddatatopracticewhattheywoulddoif theywereonanengagement teamevaluatingaDR/BCplan. Providingstudentswiththisopportunitybetterpreparesthemforoneofthejobstheycould perform aftergraduation.

Keywords: Casestudy,Computer security,Criticalthinking,Experientiallearning& education,Informationassurance and security,Role-play, Security,Teamprojects

2.CASE TEXT

2.1CompanyBackground

BankSolutions,Inc.(apseudonym),foundedin1973bythe

First Presidential Bank, a major bank of its time, is a providerofitem processingservicesitocommunitybanks, savingsandloanassociations,Internetbanks,andsmall-to mid-sizecreditunions. It offersafullrangeof services, includingin-clearingand Proof ofDeposit(POD) processing, itemcapture,returnandexceptionitem processing,image archive storageandretrieval,andcustomerstatement rendering.

Bank Solutions wasformedin1973whenthe Chief OperatingOfficerof First PresidentialBank,amajor commercial bank, recognizedanopportunity. Since item processingfunctionsarestandardized(they havetobein orderfor originating andreceiving financialinstitutionsto clearcustomertransactions) andscalablewithincreases in item processingvolumes,theywereabletoofferthese servicestootherfinancialinstitutionswishing to reduce operating expenseandfocus on growthstrategiesandother core business functions. FirstPresidentialmarketedthese services underthe BankSolutionsbrandname.

Overthe next15years,Bank Solutionsenjoyedmodest growth. By1988,itserved41small-tomid-sizefinancial institutions.    It had not, however, developed a market

presenceoutsideoftheNorthwesternRegionoftheUnited States,asmanagement hadhoped.  Thiswas primarily because Bank Solutionswasunabletocompetewithother item-processing  service  providers  that  had  developed

proprietarysoftwaresystemsconsidered“topoftheline.” Tomakemattersworse,atthe timealmost one quarter of BankSolutions‟clientbasewassavingandloanassociations (savingandloans).  AsaresultoftheSavingsandLoan crisis,60%ofBankSolutions‟savingsandloancustomer base failedoverthe sixyears spanning 1985–1991,thus stuntingtheoutsourcer‟sgrowth. Therelatedslowdownof the financialservicesandrealestateindustries andthe recessionof1990–1991presentedfurtherheadwindstothe growth objectives of  First Presidential management.     In

1994,FirstPresidentialsoldoffBankSolutions.

Undernewmanagement,BankSolutionsthrived. Keys

tothe company‟s renewedsuccess includedthe following:

·    The development of key strategic partnerships with other industry participants,   including   data  clearing housesandfinancialinstitutioncore processing system outsourcers.ii

·    Theintroductionofanewcompanyculturethatfocused onopen doormanagement,mentoring,andenhanced employee benefits.

·    Thedevelopmentofaproprietary,stateoftheartitem processingsystem thatusesstate-of-the-artOptical CharacterRecognition(OCR)technology toachieve characterrecognitionaccuraciesthat were previously unheardof.

·Theimplementationof“remotecapture”technologiesiii

to  meetelectronicbankinginitiativesand  regulations suchas“Check21.”

·    The upgrade or replacement of other administrative informationsystems,includingthecompany‟sfinancial reporting system. Thishelpedtoincrease operational effectivenessandefficiencies.

From  1995–2008, Bank Solutions enjoyed unprecedentedgrowth. During thattimeframe,the company expanded operations to 18 item  processing  facilities, two

datacentersinwhichtheitem processing systemwashosted, and345financial institutions.

2.2Current Scenario(2011)

DouglasSmith,theChief InformationOfficerforBank Solutions,wasoneof theoriginalmembersof“new management”andresponsibleformanyofBankSolutions‟ pastsuccesses.   A solid,middle-sizedcompanywith continuedgrowthpotential,BankSolutionshasbecomea

targetfora leveragedcorporatebuyout.  Thisisanattractive situationforDouglasandothermembersof executive management.  Severalof theseindividualsarecloseto retirement;andinitialindicationsarethatthepriceofthe

buyoutwillbeveryfavorableformembersof executive management.

TheCEOand other influentialmembersof executive managementwantBankSolutionsto  remain  an  attractive

purchase optionand,asaresult,havecontractedtheservices ofyourteamasanoutsideconsultanttoidentifyoperating andregulatory risksandadvisethem oncontrolmeasuresto mitigate the risks.

2.3RiskAssessmentTask

Asmembersoftheengagementteamperformingtherisk

assessment,yourteamhasbeengiventhetaskofassessing

BankSolutions‟incidenthandling,businesscontinuity,and disasterrecoverystrategy.

Inordertoperform theassessment,preliminary interviewswithDouglasSmith,theDataCenterManagers,

Systems Engineers  and Network Architect in each of BankingSolutions‟datacenters,andtheITManagersand Day  and Night Operations Managers from seven of the largest   item   processing   facilities   were   conducted.

Additionally,the following documentationrelatedtoBank Solutions‟securityincidentmanagement,DR/BCplanning activitieswas reviewed:

·Flowchartsthatdiagram theitemprocessingoperations anddataflow betweenBankSolutionsitem processing facilities and data centers and outside entities (see

AppendixA)

·AdiagramofBankSolutions‟network architecture

·Bank Solutions‟Data Center Disaster Recovery and

BusinessContinuityPlan(DRBCP)

·Policies,procedures,guidelines,andstandardsrelated tosecurityincidentresponse

·ItemProcessingFacilityDRBCPs

·Results from the most recently completed DRBCP

test/exercise

·Distributionlist forthe DRBCP

·BankSolutions‟BackupandRecoveryPolicy.

·Screen  prints  of  the  configurations  from  Bank

Solutions‟backup utility (these configurations show

what serversharesaresubject toautomated backupand the frequencyofthosebackups)

·Contracts withtheoff-site storageprovider

·A system-generatedlisting of accesstoeventlogging servers

·Alistofindividualswhohavebeenprovidedaccessto recall backuptapes fromthe off-site storage vendor.

·ScreenshotsoftheIntrusionDetectionSystem (IDS), firewall,and othereventlogging capability configurations

·Excerptsfrom theIDSandfirewalleventlogsand management‟s manuallymaintainedincidenttracking log.

2.4 Facts: RiskAssessmentFindings

Based onthe discussionsheldwiththe managementanda

reviewofthe documentationprovided,younote the followingfacts:

1.     With the assistance of an external consultant, Bank Solutions wrote its current data center DRBCPin2007. Itwas last updatedinJanuary2009.

2.     AccordingtoDouglas,thedatacenterDRBCPwaslast

testedin 2007.  Testingactivitiesconsistedof a conceptual,table-topwalkthroughof theDRBCP conductedbyDouglaswiththeDataCenterManagers andNetworkandSystemsEngineers. Itemprocessing facilityDRBCPs have notyet beentested.

3.     Site-specificDRBCPshavebeenwrittenforthefive largestitemprocessingfacilities.    Theremainingitem processing facilities have a generic “small center”

DRBCPtemplate thatwas distributedtoandcustomized by facility managementinJune 2010.  Fouritem processing facilities have notyetcompletedthe customizationexercise.

4.     DRBCPs  contain  several  sections,  including  the following:

·Emergency/crisis responseprocedures

·Businessrecoveryprocedures

·“Returnto normal” procedures

·Various appendices

RecoveryTimeObjectivesandRecovery Point Objectivesiv  for each critical business process and system  were  not  identified  in  the  DRBCP.  The

following details,mostofwhichareincludedinthe DRBCPappendices,are also documentedinthe text of the DRBCP:

·Criticalsystems,includingdetailedhardwareand software inventories

·Critical businessprocesses andprocessowners

·Alternative  processing  facility  addresses  and

directions

·“CallingTrees” (notificationlistings)

·Critical plan participant roles, responsibilities,

andrequirements

·Criticalvendorcontactlistings

·Keybusinessforms

·Specific recoveryprocedures forkeysystems

·Procedures for managing public relations and

communications

5.     Based on a review of DRBCP distribution lists, it appearsthatnotallkeyplanparticipantshaveacopyof

theplan.Whenthiswasdiscussed  withDouglas,he

respondedthatcopiesof allDRBCPsarestoredonthe network(whichisreplicatedacrossbothdatacenters

andvia backuptape).

6.     Criticalplanparticipantshavenotbeentrainedtouse

DRBCPs.

7.     BankSolutionshasimplemented  arobusthost-based

IDS,including detailedeventlogging andreporting capabilities.   However,  neither  the  DRBCP  nor any otherpolicy,standard,guideline,or procedure addresses security incident handlingsteps,including escalation pointsof contactand proceduresforpreservingthe forensic qualities oflogicalevidence.

8.     Event logging is also performed when power users perform specific privileged activities on production

serversandselectedadministrative back office systems. Interestingly, it was noted that several  of  the  same poweruserswhose actionsarerecorded ontoeventlogs also have write accesstothe logsthemselves.

9.     A review  of the network diagram  and conversations withthe Network Architectrevealthatredundancies have beenimplementedatthe network perimeter (e.g., routers,firewalls,IDS,loadbalancers,etc.).

10. BankingSolutionshasorganizedtheirDR/BCprogram

according toa“sistercenter”format;thatis,eachdata center serves as the other‟s “hot site” processing locationandeachitem processingfacility hasbeen assignedacorrespondingitem processingfacility to serve asa backupprocessing location.  Neitherthe DRBCPsnoranyotherdocumentationoutlinespecific processingresponsibilities for backupfacilities.

11. Onadailybasis,transactiondetailanditemimagefiles

fromthecurrentday‟s processingoperationsare uploadedfromeachitem processingfacility totheir regional data center (see AppendixA).

12. At   the data centers,   electronic vaulting has been

establishedwhereby alle-mail,file,andapplication serversand databases at the datacenter arecontinuously backedupto the other data centervia dual dedicated fiber optic lines.

13.A  data backup and recovery  utility  has been implemented in each data center and  the item processingfacilities. Fullbackupsofcriticaldatafiles, softwareprograms, and configurations are  performed

onceaweekandincrementalbackupsareperformedon a dailybasis MondaythroughFriday.

14. At one item processing facility, backup jobs have

routinely failed due tounknown causes. Whenthe topic was discussed with the IT Manager on duty, he shruggedthefailuresoffnotingthatthecorefinancial institutiontransactiondataandimagesaretransmitted toandarchivedatthe BankSolutionsDataCenterEast onadailybasis.

15.Attheitemprocessingfacilities,themanagementhas beentaskedwithcontracting the off-sitestorage of backuptapes. Atoneoftheitemprocessingfacilities, management has contractedthe bank across the streetto store its backup tapes in a safety  deposit box.   At anotheritem processingfacility,thenightOperations Managerstoresthebackuptapesinasafeathishome. Atathirditem processingcenter,tapesarestoredina shedatthe backofthe building.

ii

Thisisindividualproject. Asa memberofanengagementteamincharge of performingthe incident handling, DR/BC risk  assessment for Bank Solutions.youshouldreadthecase backgroundand the facts identifiedinthe interviews.

IndividualWork:Forallofthe facts/findings,preparea writtenreportthatliststhecondition(s)that presentrisksto Bank Solutionsaswellas proposedrecommendationsfor addressingthoseconditions.

JournalofInformationSystems Education,Vol.22(2)

Appendix A

Thiscasewasdevelopedsolelyforclassdiscussion.Whilethesituationdescribedinthiscaseisbasedonrealisticevents,theBankSolutionsisafictionalorganization. Further,thenames,product/serviceofferings,andthenamesofallindividualsinthecasearefictional.Anyresemblancetoactualcompanies,offerings,orindividualsis accidental.

122