Is it really necessary to have an Automation Change Management System in your plant?
Many plants use a manual approach to managing automation program changes. The change is made, and the Technician or Engineer saves the new program onto the server. This works, right? Yep, until it doesn’t. What if that person forgets? They are human after all. What if inadvertent changes were made and the new program is bad? Do you have to completely rewrite it? What if your program is hacked and the program is modified without anyone’s knowledge? These are just some scenarios where a manual approach or even an open-source version control product is not going to help you.
First, let’s define an Automation Change Management System, or CMS. A CMS is a centralized system that manages changes to program logic for controls programs and devices such as PLCs, CNCs, HMIs, PC control systems, robots, drives and general automation programs. At a minimum, a CMS should have the following features:
A good CMS will also:
Every plant’s automation layer is at risk from human error, equipment failure, sabotage, power surges/interruptions and fire. So, how will a CMS protect against these risks?
Human error: If someone makes changes to a program that results in undesired performance, or corrupts the program due to inadvertent changes, the prior version of the program is readily available.
Equipment failure: Equipment can and does fail. If the hardware fails and the only good copy of the program logic was in that hardware, the plant has a problem. With a CMS, the hardware is replaced, and maintenance personnel download the latest version of the program to the processor resulting in only a few minutes of downtime.
Sabotage: As unfortunate as this threat is, someone can connect directly to many devices (especially those in remote, unsecured locations) and modify the program with harmful results. A CMS is designed to store processor passwords (in the case of some processors) so these are not available without going through the CMS. Also, the CMS will periodically upload the logic from the processor for comparison with a copy on file. Changes can be identified in graphical detail, and immediate notification can be sent to responsible individuals.
Power surges / interruptions: Power issues can cause equipment to lock up or go off-line. If these situations result in a loss of the program, it can be downloaded from the CMS after the hardware is reset.
Fire: Any fire will be a major disruption. Whether a single device or an entire facility is lost, having all program logic stored in a central, organized CMS repository accelerates the time and decreases the cost associated with resuming production. Insurance underwriters are beginning to factor in the use of a Change Management System in assessing the risk profile of facilities.
Without proper system safeguards these events can lead to increased downtime, product errors and safety issues. While a manual backup approach may appear adequate at first glance, experience has shown that plant personnel have too many tasks that compete for the time to manually back up programs on a consistent basis. Also, increased visibility of changes, through better reporting and the potential for process improvement brought about by the effective use of a CMS application, can quickly pay for the CMS.
For instance, the following are real-world examples of plants that never had any negative consequence to using a manual or open-source version control/backup process…until they did.
Automotive encounters a PLC failure: A U.S. plant for one of the world’s largest automotive manufacturers experienced a failure with one of their PLC devices and the program configuration was lost, so an older version of the device program was used. Extensive editing was required to match the old program data to the currently required processes. Locating the old program and making the necessary updates resulted in 6 hours of lost production. The plant stated that if they had a CMS at the time of the PLC failure, the reduced downtime would have paid for the system. Read entire Application Story here.
Metals manufacturer experiences product loss after an undetected PLC code change: A specialty metals producer was given new material specifications by the customer that required a change to the PLC programming code. The change was made to the master copy, but it was not loaded into the device itself. Therefore, the program code on file did not match the code running on the device. The code running on the device was still following the ‘old recipe’ resulting in the production of three months’ worth of incorrect material. The product had to be recalled, reproduced and resent. Read entire Application Story here.
These, and countless other plants, implemented a CMS after experiencing downtime and product loss. A CMS is necessary to safeguard your automation systems from these risk events but can also benefit the plant with process improvements, compliance with regulatory requirements and more.
Download the white paper: “Safeguarding Your Plant Automation Systems with Change Management” for more information on features of a CMS, device type considerations, implementing a CMS, regulatory considerations and more.