About Me

My photo
Scott Arnett is an Information Technology & Security Professional Executive with over 30 years experience in IT. Scott has worked in various industries such as health care, insurance, manufacturing, broadcast, printing, and consulting and in enterprises ranging in size from $50M to $20B in revenue. Scott’s experience encompasses the following areas of specialization: Leadership, Strategy, Architecture, Business Partnership & Acumen, Process Management, Infrastructure and Security. With his broad understanding of technology and his ability to communicate successfully with both Executives and Technical Specialists, Scott has been consistently recognized as someone who not only can "Connect the Dots", but who can also create a workable solution. Scott is equally comfortable playing technical, project management/leadership and organizational leadership roles through experience gained throughout his career. Scott has previously acted in the role of CIO, CTO, and VP of IT, successfully built 9 data centers across the country, and is expert in understanding ITIL, PCI Compliance, SOX, HIPAA, FERPA, FRCP and COBIT.

Wednesday, May 30, 2012

Identity - Who are you?

Identity, Identity Management, Username - what about all this identity topics these days.  You have an identity login for work, personal, and some public.  Hard to remember all these logins and passwords I know, and we need to keep focus on security these days. 

Have you read some of the discussion around owning your own identity.  Your personal identity is federated to your work systems, your personal banking, how you vote and the list goes on.  Would that be a bad thing?  Think of it as a digital identity you now own, this is me and I have authorization to be in your system.  Sounds cool to me, as I can own, and manage my digital identity and what systems I interact with, either on desktop or mobile. 

So who would broker these digital identities? Should it be a private company?  Government?  Should it be a cloud provider?  This is the challenge, as you will need specifications, guidelines and rules on how  these digital identities are used, read, and secure.  Would it make sense to tag this to a driver license process?  If I am going to vote with this identity, perhaps use it for online government sites, like DNR, IRS, etc - then we need a government agency interaction to this effort. 

I am all in favor of a personal digital identity.  My concern would be around the issuing body, governance body and how we deal with security events around these.  I think this can be accomplished and the benefits would out weigh the concerns, but we need to have all the process(s) in place up front. 

The training effort around this would also be enormous.  We would need to make is simple and self service for as much as possible.  Perhaps a government agency using ServiceNow to manage the personal digital identities.  How about that?  Now that would be a good investment.

This topic will continue to grow in the months / years ahead.  Better to be engaged up front and help steer the direction, then to change something on the back end. 

Keep it positive!

Scott Arnett
scott.arnett@charter.net

Friday, May 18, 2012

DR Test - What Test?

I got your emails, and some new jokes from the readers of this blog.  Thank you.  The emails asked if I would give some high level recommendations on the DR test.  How would you go about setting up a real test of your environment and yet keep the business running.  So, at a high level, let me see if I can answer your questions.

First of all you are not going to be able to do your test during the week, this is a Saturday activity for most of you.  So schedule your DR test out far enough to give your team advance notice and you time to plan.  Next, get the big conference room scheduled (war room) for the day, have plenty of coffee, soda and some food, it can be a long day.  Other tools to have on hand is a working phone in the data center to be up and on speaker phone with the war room, and some application testers lined up.  Have a few work from home to test remote access, web applications, etc. 

I always recommend if this is your first test, start small - fail one application.  Don't make this so big your first time that it becomes unmanageable and confusion.  You will learn a great deal about your process, environment and plan with just 1 to start with.  If this is not your first time and you are ready to call a Disaster on your data center, then here are some suggestions:

  • Communicate, communicate and communicate.  Send out emails to all your staff, both IT and non-IT.  Put up posters, put a notice on the intranet page, have some staff working the service desk.  No matter how much you communicate, someone will still call the service desk their email is down or they can't get to their presentation to work on it.  Make sure everyone is aware that you are doing a DR test on Saturday and systems will be unavailable.  I would also put it upon each department manager to communicate in their staff meetings this same message. 
  • Ensure Friday night all your backups are complete, and verified.  You may need to start your backups early to ensure they complete on time.  When you start moving things around, things happen. 
  • Make sure your Saturday team has a updated DR plan prior to Saturday.  I like to send them out a week in advance telling them all to ready and prepare. 
  • Have a plan for the test - document it, how will this be done, who is doing what, when, and how will they document their portion of the test.  What worked, what did not work, and lessons learned.  What can we do different next time.
  • Fail the primary data center.  Now let me give a few words of caution because there have been organizations that turned everything off, etc.  What you are doing is testing users can get to the systems, data, applications in your backup environment.  Some of this legacy hardware when you turn it down, that has been running for years, may not come back up.  Be careful.  There was a question, can we incorporate a generator load test during this - sure.  If your DR test is a loss of utility power - yes.  To stop user access to data center A - take away the network.
  • Now that you failed over to your backup site or systems, make sure from a hardware perspective you can see everything. 
  • Bring up the applications and have the test users ready to start using the applications.
  • Don't forget to test print, EDI, and those other important transactions. 
  • Document your test as you go along.  Detail out what needs improvement, clarity, or re-write.
  • If your test failed - don't take it personal.  If you gained insight, lessons learned, you hit a home run.  Take all those notes and start fixing the issues one at a time.  Better to find that it does not work now than in a real crisis. 
Have a post test gathering at the end of the day.  Keep it positive, what we learned, what are next steps, and thank everyone.  I usually have thank you cards made up ahead of time with a little gift card in there from Applebee's or something as a way of thanks.  Your next job is to write up an executive report to management on the results of the test and what are the next steps to improve.  They will want to know when the next test will be, so be prepared to address that. 

I hope that helps.  One of your biggest challenges will be data if you have to recover from tape or drives.  If you don't have an archive policy in place, some of these large databases will be a challenge to accomplish in your agreed upon RTO.  Your test will help the organization understand that.

Keep it positive!

Scott Arnett
scott.arnett@charter.net

Tuesday, May 15, 2012

My Disaster Recovery Plan is GOLD

Disaster Recovery is always a great topic of discussion.  I had a colleague contact me recently and asked if I would look at their DR plan and poke holes in it.  So, naturally I said sure.

I spent the time looking through the plan and it looks good, very well thought out, and has some areas that need some attention.  It also has some major flaws - and it is not what is in the plan, it is what is not in the plan.  That is the plan itself.  So I called him up and said let me ask you some questions.

  1. If your data center goes down, where is your plan?  On SharePoint?  So you can't get to your plan then?  Right?  Where is your off site copy?
  2. Network is down, can't get to Outlook - where is your notification list - in Outlook?  Where is the off site copy?
  3. Where is your runbook copies?  Runbooks - those documents you need to ensure anyone can help you recover a system or application.  Don't forget the people aspects of your DR plan.  If you have a disaster that hit your data center, chances are some of your staff could be impacted. 
Find a cloud based solution to help you manage your disaster recovery documents.  You can get access to the Internet from a fast food place, hotel or a staff member home.  Don't have the only copies in your own data center that you just wrote your DR plan for. 

The other thing I recommended was to have a process for updating the plan as the infrastructure changes, applications put in production or retired, and testing the plan.  Do an actual test, not just go through a whiteboard session in a meeting room.  Make sure you can actually recover to your RPO and RTO agreements. 

One more important step I saw missing was a clear process and role responsibility for declaring a Disaster.  Don't have just 1 person with authority - have a few folks with the authority to declare a Disaster or a committee.  During your test, flush all these process(s) out.  Make sure you adjust and update your plans with lessons learned. 

So not Gold yet, but getting there.  Continuous improvement will get you to Gold my friend.

Keep it positive!

Scott Arnett
scott.arnett@charter.net