KNS > Operations > Change Control

Change Control

Change control is designed to limit outages by ensuring that engineers promoting code to production coply with all relevant processes and procedures and notify all relevant parties.

The goal of change control is *not* to limit code promotion. Kynetx expects to release new features often, sometimes daily.

Release Impact

Impact is defined according to the following levels:

  • *low* -- service will remain up during change with only minor disruptions to some service calls.
  • *medium* -- service will intermittently unavailable over the service period and could affect most service calls during those periods.
  • *high* -- service will be unavailable for the entire service period and all service calls during that period will be rejected.

Notification

Prior to code release or system reconfiguration an email will be sent to the Kynetx Engineering mailing list and the customer account representatives for any affected customers. Account representatives are responsible for notifying customers of relevant details of the change.

The email must be sent according to the following table:

Impact Timeframe
Low 30 minutes
Medium 2 hours
High 24 hours

The email should contain the following information:

  • Description
  • Expected outcome
  • System components affected by change
  • Duration, including start and end times
  • Risk assessment & mitigation
  • Post-deployment test plan & rollback plan
  • Overall impact assessment

Risk Assessment

The risk assessment should consider the following:

  • Number of affected systems
  • Importance of affected systems (production, logging, corporate)
  • Number of customers affected
  • Impact on service - negligible, brief, intermittent, sustained
  • Impact of not making the change
  • QA certification and status of Operational Acceptance Testing
  • Probability of failure
  • Impact of failure

Using this information, the deploying engineer and product engineer should determine if this is a low, medium, or high impact release.

Specifics

  • Releases that mutate existing structures in the abstract syntax tree should be classified as "medium" risk because of the need to flush the memcache simultaneous with the update for the release to work reliably.

Release Procedures

Low Impact Releases

Low impact releases can be made at any time provided that the following criteria are met:

  • A Rolling Release can be used
  • Only one service type is changed or the changes to multiple service types are orthoganal and can be deployed independently.

If these criteria can't be met then the release must be made during a maintenance window.

Other Releases

The following restrictions apply to medium and high impact releases:

  • Medium and high impact releases should be done in scheduled maintenance windows.
  • medium and high impact releases happen so infrequently that maintenance windows are scheduled irregularly as of the date at the bottom of this document.
  • Notification as defined above is required.
  • Specific procedures for the release must be written as part of the change control process.