Measurement Uncertainty Vs Total Error
Measurement Uncertainty Vs Total Error
In a recent article, Error Methods Are More Practical, But Uncertainty Methods May Still Be Preferred, James Westgard comments on the latest developments in the debate on the use of analytical total error (TE) and measurement uncertainty (MU), a debate which has been regularly revisited for the last twenty years. This blog aims to briefly explore the benefits of MU and TE and attempt to draw a conclusion on which is most beneficial in the clinical laboratory.
Many things can undermine a measurement. Measurements are never made under perfect conditions and in a laboratory, errors and uncertainties can come from (Good Practice Guide No. 11, 2012):
- The measuring instrument – instruments can suffer from errors including bias, changes due to ageing, wear, poor readability, and noise.
- The item being measured – the sample may be unstable.
- The measurement process – the analyte may be difficult to measure
- ‘Imported’ uncertainties – calibration of the instrument.
- User error – skill and judgement of the operator can affect the accuracy of a measurement.
- Sampling issues – the measurements you make must be properly representative of the process you are trying to assess. I.e. not using fully commutable controls will mean your quality control process is not reflective of a true patient sample.
Random and systematic errors
The effects that give rise to uncertainty in a measurement can be either random or systematic, below are some examples of these in a laboratory.
- Random – bubbles in reagent, temperature fluctuation, poor operator technique.
- Systematic – sample handling, reagent change, instrument calibration (bias), inappropriate method.
Total Error (TE) or Total Analytical Error (TAE) represents the overall error in a test result that is attributed to imprecision (%CV) and inaccuracy (%Bias), it is the combination of both random and systematic errors. The concept of error assumes that the difference between the measured result and the ‘true value’, or reference quantity value, can be calculated (Oosterhuis et al., 2017).
TE is calculated using the below formula:
TE = %BIAS + (1.96 * %CV)
Measurement Uncertainty is the margin of uncertainty, or doubt, that exists about the result of any measurement.
There is always margin of doubt associated with any measurement as well as the confidence in that doubt, which states how sure we are that the ‘true value’ is within that margin. Both the significance, or interval, and the confidence level are needed to quantify an uncertainty.
For example, a piece of string may measure 20 cm plus or minus 1 cm with a 95% confidence level, so we are 95% sure that the piece of string is between 19 cm and 21 cm in length (Good Practice Guide No. 11, 2012).
Standards such as ISO 15189 require that laboratories must determine uncertainty for each test. Measurement Uncertainty is specifically mentioned in section 5.5.8.3:
“The laboratory shall determine measurement uncertainty for each measurement procedure in the examination phases used to report measured quantity values on patients’ samples. The laboratory shall define the performance requirements for the measurement uncertainty of each measurement procedure and regularly review estimates of measurement uncertainty.”
Uncertainty is calculated using the below formula:
u = √A2+B2
U = 2 x u
Where:
A = SD of the Intra-assay precision
B = SD of the Inter-assay precision
u = Standard Uncertainty
U = Uncertainty of Measurement
Error methods, compared with uncertainty methods, offer simpler, more intuitive and practical procedures for calculating measurement uncertainty and conducting quality assurance in laboratory medicine (Oosterhuis et al., 2018).
It is important not to confuse the terms ‘error’ and ‘uncertainty’.
- Error is the difference between the measured value and the ‘true value’.
- Uncertainty is a quantification of the doubt about the measurement result.
Whenever possible we try to correct for any known errors: for example, by applying corrections from calibration certificates. But any error whose value we do not know is a source of uncertainty (Good Practice Guide No. 11, 2012).
While Total Error methods are firmly rooted in laboratory medicine, a transition to the Measurement Uncertainty methods has taken place in other fields of metrology. TE methods are commonly intertwined with quality assurance, analytical performance specifications and Six Sigma methods. However, Total Error and Measurement Uncertainty are different but very closely related and can be complementary when evaluating measurement data.
Whether you prefer Measurement Uncertainty, Total Error, or believe that they should be used together, Randox can help. Our interlaboratory QC data management software, Acusera 24•7, automatically calculates both Total Error and Measurement Uncertainty. This makes it easier for you to meet the requirements of ISO:15189 and other regulatory bodies.
This is an example of the type of report generated by the 247 software. MU is displayed for each test and each lot of control in use therefore eliminating the need for manual calculation and multiple spreadsheets.
Fig. A
Fig. B
Fig. A and Fig. B above are examples of report generated by the 24•7 software. Fig.A shows how MU is displayed for each test and each lot of control in use therefore eliminating the need for manual calculation and multiple spreadsheets. Fig. B shows TE displayed for each test.
Acusera Third Party Controls
The Importance of ISO 15189
Good Practice Guide No. 11. (2012). Retrieved from http://publications.npl.co.uk/npl_web/pdf/mgpg11.pdf
Hill, E. (2017). Improving Laboratory Performance Through Quality Control.
Oosterhuis, W., Bayat, H., Armbruster, D., Coskun, A., Freeman, K., & Kallner, A. et al. (2017). The use of error and uncertainty methods in the medical laboratory. Clinical Chemistry and Laboratory Medicine (CCLM), 56(2). http://dx.doi.org/10.1515/cclm-2017-0341
Westgard, J. (2018). Error Methods Are More Practical, But Uncertainty Methods May Still Be Preferred. Clinical Chemistry, 64(4), 636-638. http://dx.doi.org/10.1373/clinchem.2017.284406
The Importance of Meeting ISO 15189 Requirements
Laboratory accreditation provides formal recognition to competent laboratories, providing a means for customers to identify and select reliable services (CALA, n.d.). Use of accreditation standards by clinical laboratories enables them to drive gains in quality, customer satisfaction, and financial performance. This is essential at a time when laboratory budgets are shrinking.
Some key benefits include:
- Recognition of testing competence – as mentioned above, customers can recognise the competence of a lab with an internationally recognised standard.
- Marketing advantage – accreditation can be an effective marketing tool as labs can demonstrate their quality and overall competence.
- Benchmark for performance – laboratories can determine whether they are performing to the appropriate standards and provides them with a benchmark to maintain that standard.
To maintain the global recognition gained from accreditation, labs are evaluated regularly by an accreditation body to ensure their continued compliance with requirements, and to check that standards are being maintained. (CALA, n.d.).
In a comprehensive study conducted by Rohr et al. (2016) it was found that, while accounting for as little as 2% of total healthcare expenditure, in vitro diagnostics (IVD) account for 66% (two thirds) of clinical decisions. Despite such a small percentage of budget dedicated to it, IVD plays a huge role in patient care so it is vital that there is guidance in place to ensure quality standards are met. Poor performance of tests at any stage of care and treatment can reduce the effectiveness of treatment and deny appropriate care to patients in need (Peter et al., 2010).
ISO 15189 is an international accreditation standard that specifies the quality management system requirements particular to medical laboratories and exists to encourage interlaboratory standardisation, it is recognised globally.
Meeting ISO Requirements
Scroll through below to learn how ISO 15189 regulates aspects of a clinical laboratory and how Randox can help you meet these suggestions.
Review of QC data
“The laboratory shall have a procedure to prevent the release of patient results in the event of quality control failure. When the QC rules are violated and indicate that examination results are likely to contain clinically significant errors, the results shall be rejected…QC data shall be reviewed at regular intervals to detect trends in examination performance”
– ISO 15189:2012
Acusera 24∙7 will automatically apply QC multi-rules, alert you to or reject any results that violate the QC multi-rules or performance limits, generate a variety of charts allowing visual identification of trends and provide access to real-time peer group data to assist with the troubleshooting process.
Calculation of MU
“The laboratory shall determine measurement uncertainty for each measurement procedure in the examination phases used to report measured quantity values on patients’ samples. The laboratory shall define the performance requirements for the measurement uncertainty of each measurement procedure and regularly review estimates of measurement uncertainty.”
– ISO 15189:2012
Acusera 24∙7 is the only QC data management platform that incorporates the automatic calculation of Measurement Uncertainty (MU) as well as other performance metrics, including Total Error.
More about Measurement Uncertainty and how Acusera 24∙7 can help
Commutability
“The laboratory shall use quality control materials that react to the examining system in a manner as close as possible to patient samples”
– ISO 15189:2012
Acusera True Third Party Controls are fully commutable, behaving like a real patient sample, reducing the need to re-assign QC target values when the reagent batch is changed, reducing labour and costs.
Medical decision levels
“The laboratory should choose concentrations of control materials, wherever possible, especially at or near clinical decision values, which ensure the validity of decisions made”
– ISO 15189:2012
Acusera True Third Party Controls are designed to challenge instruments across the entire clinical reporting range.
Comparison of results across instruments
“Laboratories with two or more analysers for examinations, should have a defined mechanism for comparison of results across analysers”
– ISO 15189:2012
Acusera 24∙7 is capable of combining multiple data sets on a single Levey-Jennings, Histogram of Performance Summary chart, enabling at-a-glance performance review and comparative performance assessment. A unique multi-instrument report is also available via our RIQAS EQA programme allowing performance of each instrument to be compared.
Third Party Control
“Use of independent third party control materials should be considered, either instead of, or in addition to, any control materials supplied by the reagent or instrument manufacturer”
– ISO 15189:2012
Acusera True Third Party Controls are manufactured completely independently of and calibrators and assigned values through a pool of instruments across the world, making them true third party controls.
At a conference in Belgium in 2016, data, which highlighted the most common areas of non-conformance in laboratories, showed that nonconformities were most prevalent in sections 5.5 and 5.6 of ISO 15189. This data is visualised in fig. A below. Furthermore, a study by Munene et al. (2017) has had similar findings, as visualised in fig. B. The greatest number of nonconformities occur in the sections that are concerned with insufficient assay validation and quality of examination procedures. These studies specifically identified the lack of independent controls, QC not at clinically relevant levels, commutability issues, and a lack of interlaboratory comparison as major issues.
Randox Quality Control products are designed to target these areas, making it easier to conform to ISO 15189 standards.
Fig. A
Fig. B
Acusera Third Party Controls
Interlaboratory Data Management
CALA. The Advantages of Being an Accredited Laboratory. Canadian Association for Laboratory Accreditation. Retrieved from http://www.cala.ca/ilac_the_advantages_of_being.pdf
Munene, S., Songok, J., Munene, D., & Carter, J. (2017). Implementing a regional integrated laboratory proficiency testing scheme for peripheral health facilities in East Africa. Biochemia Medica, 110-113. http://dx.doi.org/10.11613/bm.2017.014
Peter, T., Rotz, P., Blair, D., Khine, A., Freeman, R., & Murtagh, M. (2010). Impact of Laboratory Accreditation on Patient Care and the Health System. American Journal Of Clinical Pathology, 134(4), 550-555. http://dx.doi.org/10.1309/ajcph1skq1hnwghf
Rohr, U., Binder, C., Dieterle, T., Giusti, F., Messina, C., & Toerien, E. et al. (2016). The Value of In Vitro Diagnostic Testing in Medical Practice: A Status Report. PLOS ONE, 11(3), e0149856. http://dx.doi.org/10.1371/journal.pone.0149856
Benefits of High-Sensitivity Troponin I (hs-TnI)
Benefits of High-Sensitivity Troponin I (hs-TnI)
Chest pain is a common symptom; 20% to 40% of the population will experience chest pain during their lifetime. There are many causes of chest pain, some of which are benign, while others are potentially life threatening. Importantly, in patients with chest pain caused by an acute coronary syndrome (ACS) or angina, there are effective treatments to improve symptoms and prolong life, emphasising the importance of early diagnosis in patients where chest pain may be of cardiac origin (Skinner et al, 2010). Chest pain is one of the most common reasons for emergency admission to hospital and is a heavy burden on health-care resources. A strategy to identify low-risk patients suitable for immediate discharge would have major benefits (Shah et al., 2015).
RIQAS Liquid Cardiac Programme
Interlaboratory data Management
Ford, C. (2017). Benefits of High Sensitivity Cardiac Troponin I at Admission. Clinical Laboratory Management Association, (July/August 2017), 22-24.
Shah, A., Anand, A., Sandoval, Y., Lee, K., Smith, S., & Adamson, P. et al. (2015). High-sensitivity cardiac troponin I at presentation in patients with suspected acute coronary syndrome: a cohort study. The Lancet, 386(10012), 2481-2488. http://dx.doi.org/10.1016/s0140-6736(15)00391-8
Skinner, J., Smeeth, L., Kendall, J., Adams, P., & Timmis, A. (2010). NICE guidance. Chest pain of recent onset: assessment and diagnosis of recent onset chest pain or discomfort of suspected cardiac origin. Heart, 96(12), 974-978. http://dx.doi.org/10.1136/hrt.2009.190066
The Benefits of True Third Party Controls
In the clinical laboratory, Quality Control (QC) refers to the process of detecting analytical errors to ensure both the reliability and accuracy of patient test results. Poor performance can result in misdiagnosis, delayed/inappropriate treatment, increased costs and may even be potentially life threatening for the patient.
1. Dependent / First Party Controls
Dependent controls refer to any control material that has been produced by the instrument or reagent manufacturer for use on a specific test system. Such controls are often manufactured from the same raw materials as the calibrator, making them less sensitive to subtle changes in performance.
As dependent controls are generally optimised for use with the manufacturer’s test system, these controls can mask weaknesses, and therefore, are increasingly considered less effective than independent controls.
2. Semi-Dependent Controls
Semi-dependent control material, although produced independently of the instrument or reagent, is often supplied or recommended by the instrument/reagent manufacturer. It is this manufacturing relationship between the two that requires close scrutiny when considering if these controls are fit-for-purpose.
Although the control material is not produced by the instrument manufacturer, it is produced according to their exact specifications and therefore, optimised to work with a specific platform.
3. Independent / Third Party Controls
Independent or third party quality control material has not been designed or optimised for use with any instrument, kit or method. This complete independence enables the quality control material to closely mirror the performance of patient samples, and in doing so, provide an unbiased, independent assessment of analytical performance across multiple platforms.
Third party controls have been designed to deliver an independent, unbiased assessment of performance with any instrument or method helping you gain accreditation. Below are some of the key benefits of third party controls:
- Values assigned using a large number of independent laboratories ensuring statistically valid targets.
- Highly consolidated controls allow for space, time, and ultimately, cost savings.
- Boosted shelf life ensures continuity of supply and reduced costs
- Reduced preparation times by removing the need for multiple instrument controls
Regulatory Requirements
Third party controls are growing in popularity across the globe. More and more laboratories are beginning to use third party controls as part of their daily QC strategy. The benefits of such controls are widely accepted and recommended by both key opinion leaders and regulatory bodies in the field of Quality Control.
“Use of independent third party control materials should be
considered, either instead of, or in addition to, any control
materials supplied by the reagent or instrument manufacturer”
ISO 15189:2012 Section 5.6.2.2
ISO 15189:2012 Medical Laboratories – Requirements for Quality and Competence
The benefit of running third party controls in your laboratory cannot be underestimated. The following case studies highlight their many benefits and how they have helped laboratories across the world to provide more accurate and reliable test results.
Case Study One: Identifying Lot-to-Lot Variability with Third Party Controls
A laboratory in the UK contacted Randox Technical Services, reporting higher than expected QC results for
Thyroglobulin. When using a third party control (Acusera Immunoassay Premium Plus) the results were four
times higher on their main analyser compared to other systems. However, when they ran the instrument
manufacturer’s control alongside the third party control it did not show the same problem.
After reviewing EQA data, the Technical Services team confirmed there was a significant difference in results
compared to other instruments, and set about helping the laboratory troubleshoot. After an exhaustive
review of procedures and processes, the customer contacted the instrument manufacturer, who advised of a
positive bias with several batches of reagent, including the batch the laboratory was using.
Conclusion: By using a third party control the laboratory was able to detect a shift in results after changing
reagent batch that the instrument manufacturer’s control did not.
Case Study Two: Overcoming Instrument Errors with Third Party Controls
A laboratory using the Acusera Assayed Chemistry Premium Plus control contacted Randox Technical Services
after observing a consistent negative bias for ALT which was not replicated by the instrument control. They
had previously contacted their instrument manufacturer who advised that the problem was with the control
and not the reagent or instrument.
Randox investigated the problem and demonstrated that patient results were also wrongly reported low. This
later led the instrument manufacturer to recommend a wash stage to eliminate any interference.
Conclusion: The use of a third party control in this instance enabled the identification of a procedural error
with the instrument that the recommended control did not.
Explore the benefits of Randox Acusera third party controls below.
Our extensive range of assayed quality controls are supplied with highly accurate target values for a wide range of instruments and methods. Our unique value assignment process utilises thousands of independent laboratories globally ensuring target values won’t change throughout the shelf life of the control and eliminates the need to spend time and money performing value assignment in-house.
Accuracy coupled with unrivalled traceability to International Reference Laboratories, provides a product of unsurpassed accuracy and reliability.
We take quality seriously, that’s why all QC products are manufactured to the highest possible standard delivering control products of unrivalled quality. Our superior manufacturing processes ensure stability claims and analyte levels won’t differ significantly from lot to lot. You can therefore be sure of receiving the same standard of product time and time again.
Regular shifts in QC results when a reagent batch is changed can be both costly and frustrating for many labs, resulting in a frequent need to reassign target values. Designed to be commutable, the Acusera range of Internal Quality Controls will react to the test system in a manner as close as possible to the patient sample helping you to meet ISO 15189:2012 requirements while ultimately ensuring accurate & reliable instrument performance. Furthermore our lyophilized controls contain no added preservatives or stabilisers ensuring a sample matrix that closely matches the patient sample.
ISO 15189:2012 states, “The laboratory shall use quality control materials that react to the examining system in a manner as close as possible to patient samples”.
All controls for use with immunoassay/immunology based methods are manufactured using only 100% human components demonstrating our commitment to quality and eliminating costly QC shifts when reagent batch is changed.
Working stability and product shelf life are important considerations for any lab when choosing which internal quality control material is best suited to their needs. Labs often spend up to one month validating new material, a process which can be minimised by opting for a control with an extended shelf life. At Randox our lyophilised controls have a shelf life of up to four years and our liquid controls a shelf life of up to two years from the date of manufacture ensuring continuity of lot supply and ultimately reducing the need for expensive new lot validation studies.
Each of our third party controls will have its own reconstituted or open vial stability, some of the analytes will have limitations, however we pride ourselves on not misleading customers with false claims. The extended open vial and reconstituted stabilities will help laboratories to minimise waste and reduce costs.
Randox is a leading provider of multi-analyte, third party controls designed to allow any lab to carry out highly accurate QC using fewer controls. In an industry where budgets and resources are increasingly under pressure, highly consolidated controls will ensure high levels of throughput without compromising on accuracy. Uniquely comprising up to 100 analytes in a single control product, costs, preparation time and storage space are dramatically reduced without sacrificing on quality.
The presence of analytes at key decision levels in all Acusera controls will not only ensure accurate test system performance across the clinical range, but will further aid consolidation and maximise laboratory efficiency by eliminating the need to purchase additional high or low level controls, which are often expensive.
Available in multiple levels, the Acusera range of third party controls are designed to challenge laboratory instruments throughout the patient reportable range. The presence of analytes at clinically relevant decision levels not only helps to ensure accurate instrument performance but maximizes laboratory efficiency by eliminating the need to purchase additional low/high concentration controls at extra expense.
Randox also employs easy to use colour-coded packaging to help distinguish between different levels and reduce costly mix-ups.
ISO 15189:2012 states, “The laboratory should choose concentrations of control materials wherever possible, especially at or near clinical decision values, which ensure the validity of decisions made”.
Randox Acusera is a world leading manufacturer of true third party controls providing a cost effective, high quality solution for any lab – regardless of their size or budget. Designed to provide an unbiased, independent assessment of performance, our internal quality controls have not been manufactured in line with, or optimised for use with any particular reagent, method or instrument helping you to easily meet ISO 15189:2012 recommendations.
ISO 15189:2012 states that the “use of independent third party control material should be considered, either instead of, or in addition to, any control materials supplied by the reagent or instrument manufacturer.”
Traceability refers to the property of a measured result or calibrator value to be related or traced back to a reference measurement procedure or reference material through a series of measurements with known uncertainties.
The European parliament and council directive 98/79 EC require values assigned to both calibrators and control materials on in vitro diagnostic medical devices to be traceable to a recognised reference material or reference measurement procedure of higher order, e.g. SI units.
Guidelines have been set for diagnostic manufacturers to follow when assigning calibrator values and establishing traceability. These guidelines were set by the European standards EN/ISO 17511 and also EN/ISO 18153. The Randox traceability pathway has been established with reference to standards ISO 17511 and ISO 18153. The pathway has been followed to establish traceability for Randox calibrators allowing all patient results to be traced right back to the source.
With an extensive range of assayed/unassayed, liquid/lyophilised and single/multi-analyte controls, the Acusera portfolio has a solution to suit all laboratory preferences.
Custom Controls
Randox is a market leader in the manufacture of customised quality controls designed to meet the individual and unique requirements of even the most specialised laboratories.