Change Improvement Plan

Many changes happen on the servers, from nightly yum updates, to our security scanner updates, to our server configuration updates. In order to improve our change management process we promise to do the following:

Classify change types into categories with specific notification requirements

  • Identify every change that is made and classify the change
  • Write a change policy for each change
  • All changes must happen outside of business hours Pacific Time
  • Green changes are routine maintenance and can be scheduled 12 hours in advance
  • Yellow changes are changes that might affect customers and must be scheduled 7 days in advance
  • Red changes are changes that will affect customers and must be scheduled 2 weeks in advanced

Standardize release days and times

  • Server releases will be tested on internal servers on Mondays during a standard time window 4am to 8am
  • Server releases will be released to customer servers on Tuesdays during a standard time window 4am to 8am
  • Network changes will happen after 9pm

Notify customers ahead of time

  • A status page was created for both scheduled and unplanned events
  • All customers primary email, alert emails, mojo developer emails, and additional emails are subscribed
  • Customers can also subscribe by phone number for text alerts
  • Customers email/phone are synched to all components on status page they are using including physical hosts and racks
  • Every time we touch the infrastructure, even if routine, we will schedule in advance based on classification type

Validate releases thoroughly

  • Server releases will be released and tested on incremental groups of internal servers
  • Server releases will then be released on incremental groups of customers servers

Open communication during events

  • We will proactively notify affected customers
  • We will provide status updates every 30 minutes until resolution
  • For customers with special configurations (ex clusters) we will have a team lead available to communicate with directly

Provide a full transparency of all changes

  • Any change that applies to more than 1 customer will be listed on the status page
  • Nightly yum updates will be listed in mhm log
  • Saltstack changes to servers will be listed in mhm log
  • Internal and external servers users commands will be listed in mhm log
  • Server config change diffs will be listed in mhm log
Leave a Reply
  • Cisco
  • Intel
  • Redis
  • Magento
  • Nginx
  • Dell
  • Percona
  • MemCached
  • PCI Compliant
  • BBB