Statistics Abuse In QA/QC: 3 Lessons Learned

The tools we use to develop new or improved products are essential in ensuring we deliver the product as soon as possible and as efficacious as technology, testing, and knowledge/experience permits. But the caution one must take in using tools is to be mindful that we use the tools correctly and effectively. The worst that can occur is to use a tool incorrectly and, as a result, make errors in conclusion that could lead to injury, costly litigation, or a tarnished reputation.

One of the best tools at our disposal in a quality assurance/quality control context (manufacturing, supplier quality, etc.) is statistics. Statistics helps to quantify uncertainty. This allows us to make conclusions with a degree of certainty regarding this uncertainty. Statistics also helps us extrapolate and interpolate information in the form of data to make informed decisions. The key word here is informed. Statistics is not a substitute for thinking. Remember, statistics is a tool. As a tool, it provides more clarity around uncertainty so you can make a more appropriate conclusion on the data that relates to your product’s performance. This is extremely important regardless of the project’s phase, whether it is feasibility, development, validation, or an aspect of process improvement. Statistics can be abused through purposely misapplying statistics or it can be abused by improper application or interpretation.

Lesson #1: Statistics Is Not A Substitute For Thinking

A form of statistics abuse can be demonstrated in the following example. A company where a manufacturing site was updating a testing machine to a more automated version applied statistics to test data that showed the P-Value of a 2-gram difference in test comparisons between the two testing machines is significant. The engineer determined that difference meant the machines were not the same. Significant debate ensued. The main concern was the 2-gram difference was not a measurable difference by the customer, yet statistics implied the 2-gram difference was significant. The test was a subjective destructive test measuring a constantly changing resistance force. Things were at an impasse until informal testing of prepared samples was conducted with customers. The customer feedback showed that differences of less than 10 grams were undetectable. In fact, for larger products, differences of less than 35 grams were undetectable. The conclusion was, despite statistics indicating this small difference to be significant, it did not impact the performance or the perception of performance by the customer. In other words, there was no practical difference between the two machines, even though statistics concluded there was a difference; the difference was not practical.

This scenario illustrates where a practical litmus test coupled with experience and additional testing can provide valuable information or clarity around statistical uncertainty. Always remember that statistics is a tool to help you make a decision — it is not a substitute for thinking.

Lesson #2: Set A Realistic Confidence Interval

I’ll share another example from my personal experiences: A manufacturing site was switching to another supplier and was under pressure to complete the conversion quickly. The quality engineers from the site presented their data statistically comparing the two manufacturers’ products, with the conclusion that all five materials being switched were the same. This, of course, was great news and would eliminate a significant amount of work by the site. However, this euphoric conclusion was dashed when the statistical analysis was reviewed. The site engineers used a confidence interval of 99% rather than 95%, which is typically used for analysis. This seemingly slight difference was far from slight, because the higher the confidence interval percentage, the greater the margin of error. For example, let’s say there are 10 horses in a race and you want to be sure to pick the winner with 99% confidence. The only way to do this is by stating that the winner is one of the 10 horses. If we used a 95% confidence interval, there is less certainty. We would have to make the statement that the winning horse will come from one of the nine horses you selected. This makes for more uncertainty as the 10th horse could be the winning horse. What the site engineer did in stating the two suppliers were the same at a confidence interval of 99% was in fact using the statistics to conclude they are the same. When they redid the statistics at a confidence interval of 95%, it showed that four out of the five products were actually quite different.

In this case, a knowledge of statistics was being used to avoid doing the proper testing and, conversely, knowledge of statistics helped avoid a potential error that could have been very troubling. Statistics is like a double-edged sword — it can work for you or against you depending upon how it is employed and the intentions behind its use. This is a good justification for having a statistician on a company’s payroll to ensure statistics are properly applied.

Lesson #3: Look Into The Causes Of Testing Anomalies

Statistics is a powerful and insightful tool. When used properly it is extremely helpful and timesaving, and it can help drive you to the right solution. Unfortunately, statistics is only as good as the data and how the data is collected. Further, it has to be applied correctly.

During another personal experience, a supplier was testing a material for release, but when the test was performed by the receiving company, the results indicated the material failed. A review of the testing suggested the problem lay with the testing technicians, with one in particular having very different results from the other technicians; obviously, this technician was one of the reasons for the difference in test findings. Using statistics over many weeks and many different test protocols, we discovered that the testing variations were due to the fact that each technician had to dissolve samples of the supplied material to make a solution. Variations in the amount of the solution dispensed by the different technicians was what was causing the testing anomalies. The fix was to purchase an automatic dispensing machine that removed the variation from the mixing step, and we bought one for the supplier and one for the receiving company. Over 44% of the variation was eliminated.  And that one rogue technician whose testing was significantly different from the other technicians? Turned out the technicians was actually the most accurate and consistent of all of the technicians.

Points To Remember In Using Statistics

These cases and the points made show why it is important for your company to ensure that your project teams have solid training and solid ongoing training in statistics. It is important that teams know how to collect data and how to preserve data integrity. Employ or consult with a statistician who can review project analyses and guide those using statistics to ensure they are using the right statistics and drawing the right conclusions. A statistician is also a valuable resource to verify that statistics are applied and interpreted correctly, to be available as a consultant to product teams, and to educate and improve project teams’ knowledge and use of statistics. After all, statistics are only as good as the inputted data and how they are collected, and the decisions made based upon statistics are only as good as the statistics. In the end, product safety, customer safety, product efficacy, and your company’s reputation and vitality are at stake.

Continuity Is the Key to Success for Your Medical Device Company

Continuity, by definition, is the state or quality of being continuous. To a company, continuity could be the ability to provide a product that consistently performs as expected with consistent quality. It could also refer to a company’s ability to consistently make a product available without supply chain disruption. Financially, continuity could refer to a company’s ability to continuously return an increasing profit.

Critical to continuity in these aforementioned objectives are a company’s short- and long -term plans that provide the approaches on how to fulfill the overall company strategy. It is easy to see that continuity in various areas is key to a company’s well-being. The challenge for medical device manufacturing and medtech companies is maintaining continuity in all of these areas simultaneously. This is not an easy task because it takes coordinated efforts throughout a company’s infrastructure that are tied back to the comprehensive company strategy. Continuity can be impacted by cost-containment initiatives or by cultural factors.

How Is Continuity Impacted By Cost-Containment Initiatives?

When a company focuses on “improving” its bottom line, this is generally done by reducing overhead costs. One common method is to reduce the size of the workforce through layoffs and retirements. Sometimes, a company replaces those employees with contracted services. Contracted services costs can be expensed, which effectively moves a bottom-line cost into the expense column. This move could also result in tax benefits.

Another cost-containment initiative that impacts a company’s continuity is when a company divests itself of direct manufacturing responsibility. Contracting out specific manufacturing processes not only eliminates the associated workforce as a cost burden but also removes direct production responsibilities. In essence, this strips down the company to the perceived minimum needed to maintain or improve current financial metrics.

Workforce reduction or conversion to contract or third-party services does improve the company’s bottom line in the short term, but this approach can be disruptive over the long term if the necessary planning is not put into place. What is concerning about these short-term solutions is their potential to dilute a company’s depth of experience. Outsourcing manufacturing, the increased reliance on contract services, retirements, buyouts, and layoffs all reduce a company’s experience levels. In some cases, a company will gradually return to the pre-cut conditions to compensate for issues and emerging directives.

How Is Continuity Impacted By Cultural Factors?

The dynamics of the work environment are evolving. Gone are the days when a person landed a job and stayed with the company until they retired. A company’s allegiance to its workforce is diminishing while a worker’s allegiance to their employer is likewise diminishing. As soon as an employee lands their current position, they are typically searching for their next job or promotion to improve their earnings or to land a higher-level position. This is quickly becoming the inherent cultural persona of the typical worker. It is not about “what you have done for me?” but more of “what are you doing for me now?”. Resting or relying upon past accomplishments is no longer de rigueur for companies or employees. This lack of mutual commitment translates into a revolving door of personnel, and it is not solely a workforce issue — it is seen in upper management as well. Upper management have similar career desires and shift into different roles within a company or move to different companies as opportunity presents itself. This creates a lot more disruption than at the workforce level because changes in upper management usually result in changes in the company strategy.

What Can A Company Do To Address Knowledge Drain?

The subtle knowledge drain caused by cost-cutting initiatives and cultural changes can affect a company in the long term through lost time, wasted expenses, delivery issues, and manufacturing and quality issues. Employees or management struggle to resolve problems that experienced or well-trained associates could have solved immediately. Without the proper training or experience, new employees or contractors may not have the intimate knowledge of the company’s product(s) or manufacturing processes necessary to avoid needless delays or unexpected costs. If new employees or contractors do not have access to comprehensive documentation, then the typical reaction to resolve a problem is to conduct trial-and-error solutions; these typically do not lead to a prompt solution and often generate additional problems to be resolved.

In one situation I observed, an overseas manufacturing site was struggling for months with a process step. The step was well-defined in our domestic operations but not at this overseas locale. When I visited the site with an associate about an unrelated issue, the problem was brought to our attention. We both had extensive experience in the process they were struggling with. In less than an hour, we had identified the root cause of their problem (there were several problems confounding the situation) and we were able to get them back on track and eliminate the manufacturing issues. In this example, the site had no visibility to the knowledge base of other manufacturing sites performing the same manufacturing steps, nor did they have adequate internal documentation to guide associates on how to resolve the problems they were facing. Management was fairly new as the site had significant turnover, so they were not able to help. If the team had access to a company database or a directory of company resources or had better overall documentation, they could have resolved their issues quickly rather than having months of frustration. We held impromptu manufacturing training and awareness sessions and showed where they could improve their manufacturing documentation. The problems are no longer an issue.

Manufacturing problems can potentially lead to quality issues that could negatively impact the consumer’s perception of a product’s efficacy or performance. Therefore, the two key denominators contributing to a company’s sustained continuity that are often overlooked or marginalized is knowledge retention and comprehensive training. Elements that should be integrated into the company’s short- and long-term plans accordingly. This is especially vital when a company operates globally with manufacturing sites having similar operations. In this instance, it is imperative that test methods, manufacturing processes, and accept/reject criteria be the same. I often saw different manufacturing sites that did not communicate with each other, reject product that another site would accept and vice versa. I like to refer to these scenarios as gems. They represent situations that can be easily resolved that immediately result in two forms of cost savings: reducing product waste and freeing up manufacturing capacity.

While it is true that product evolves by the application of emerging technology and techniques, there are basic elements that tend to be present with any project, and experience can help avoid delays when implementing these new technologies and techniques. Companies should utilize their experienced workers to develop or add to their product and process knowledge prior to leaving the company in the form of thoroughly documenting processes, development work, and troubleshoot techniques/solutions. This creates a solid knowledge base for a company that could also be tapped into during the development of new products and processes to avoiding an unknowing repeat of past negative efforts.

In another example from my experiences, my team was working on developing a new material for a current product line. The material was so unique that current methods did not work. We had to develop almost all new manufacturing techniques. However, we had a lot of experienced people at the time. By culling together all of the product and process experiences, we were able to identify the most promising approaches. These proved successful and saved the project team from very prohibitive delays.

Before enacting significant forms of cost-cutting techniques, a company should have a well-developed product and training/education program for new employees, contractors, and consultants. If a company is global with manufacturing sites around the world, it is imperative that a universal database be implemented for knowledge sharing. Sites performing the same processes should use the same documents. This way, as changes are made and improvements are identified, all sites benefit from the efforts of one site since they are using the same documents. This is especially true for test methods, acceptance criteria, and operating parameters.

In summary, cost-cutting initiatives and high levels of workforce and/or management turnover can be disruptive to a company’s continuity, competitiveness, and long-term vitality. Companies need to develop a flexibility to accommodate and adapt to management and workforce fluidness. It is okay for a company to be aggressive in cost cutting initiatives. However, in order to avoid paying the price down the road, a company must have comprehensive documentation and a thorough workforce training program in place, as they are ideal vehicles for a seamless transfer and retention of knowledge and experience from one “generation” of the workforce to the next. This basic approach could have far-reaching positive impact on companies and bridge the gap between short-term focus and long-term objectives.