We are pleased to bring you a guest blog post today from Martin Klubeck, a Books24x7 BusinessPro Collection author. Klubeck is a strategy and planning consultant at the University of Notre Dame and a recognized expert in the field of practical metrics. His passion for simplifying the complex has led to the development of a simple system for developing meaningful metrics. Klubeck is also the founder of the Consortium for the Establishment of Information Technology Performance Standards, a nonprofit organization focused on providing much-needed standards for measures. His most recent book published by Apress - Metrics: How to Improve Key Business Results - is available in the Books24x7 BusinessPro collection.
By Martin Klubeck, Strategy and Planning Consultant for the University of Notre Dame
Metrics are at their best when they are designed to answer a root question – not when they are driven by a need to compare ourselves to our peers or in finding “interesting” viewpoints. In the case of learning solutions, the root question often times is, how effective is the training?
Or another way of looking at that question is, how does the learning solution affect the capability of the students? In business, these students are normally workers and managers attending a diverse set of learning opportunities for the purpose of improving their performance of specific tasks. I highly recommend using tasks (and subsequent task-breakdowns) for focusing training metrics on what really matters – the attainment of skills.
By applying the rules for developing metrics found in Metrics: How to Improve Key Business Results, my recently published book from Apress, you can identify the right measures. Let’s look at one sample task to demonstrate – writing HTML5 code:
Capability can be measured by three outputs – specifically speed, accuracy, and efficiency.
Speed. How fast does the worker write code before and after training? You’ll want to know how long it takes to perform the task, not how long it takes in total time since this would include non-value added time (breaks, time to power up the computer, answering the phone, etc.) So this requires the worker to track the actual time-to-task. It is important to use measures which if the training was successful, we would expect to see (positive) change and speed fits that description.
Accuracy. How accurate is the coding effort? Does it have to be reworked or rewritten? If you use rework as a measure of accuracy, it can be in the form of time spent allowing you to use the same data used for Speed. You can also use defects or errors. This is when the classic “defects per lines-of-code (LOC)” comes in handy. If one worker has 10 errors and the other has 5, which is better? Well it depends, if the worker with 10 errors wrote 1000 lines of code and the one with “only” five errors wrote only 10 lines of code, which would you choose?
Efficiency of the Code. Efficiency helps us to see why Triangulation is important. In the scenario for Accuracy, it isn’t enough to know that the five-error-programmer only wrote 10 LOC. It is important to know how efficient both programmers’ code is. If you only look at errors-per-LOC, you might choose the one with 10 errors since it was spread over many more lines of code (1 error for every 100 lines vs. 1 error for every 2 lines of code). But if the one writing 1000 lines of code was not very efficient, and wrote more code than needed, your choice of the better programmer could be wrong (for example if the code they wrote did exactly the same things).
By looking at all three measures together, Speed, Accuracy, and Efficiency of the code, we can build a picture of how capable or skilled the worker is. By measuring this performance before and after the learning solution, we can determine the effectiveness of the training for improving the skill of the worker to perform given tasks.
The steps for measuring the effectiveness of training can be summarized as:
- Identify the performance (in the form of specific tasks) you are trying to improve. If necessary perform task breakdowns, as was done in the example of writing HTML5 code.
- Determine which attributes you want to see positively affected. Speed, Accuracy, and Efficiency of the code were used in our example.
- Measure those attributes before the training.
- Measure those attributes after the training.
The final step is to again measure the attributes after an extended period of time (say six months) to determine the retention of the improvements. Many times performance improves for reasons other than effective training so measuring performance over time helps ensure you are getting a true picture.