Understanding Reproducibility in Flow Cytometry – It’s the Antibodies!

Reproducibility is key to the scientific method. After the results of a study are published, the community validates the findings and extends them. If the findings are not reproducible, the second step is impossible. With performable experiments increasing in complexity, and the concurrent increase in the cost of equipment and reagents to perform these experiments, it is important to find the best way to maximize the money spent on advancing research.The focus of this blog entry will be on the reagents we use to identify the cells of interest: Antibodies and fluorochromes.

In flow cytometry, there are many places where improvements can be made to increase the consistency and reproducibility of an experiment. The most obvious place is in the instrument, which was the focus of a previous blog post. The focus of this blog entry will be on the reagents we use to identify the cells of interest: Antibodies and fluorochromes.

Why the focus on antibodies? The commentary on this topic (with 110 co-signers) by Bradbury and Plückthun (2015) is informative. The authors cite a review from 2008, in which the authors stated that “fewer than half of around 6,000 routinely-used commercial antibodies recognized only their specified target.” For a field that relies on these reagents on a daily basis, this is cause for some alarm.

In Begley and Ellis (2012), Begley states that in one set of experiments, only 6 of 53 “landmark preclinical studies” were able to be replicated. Bradbury and Plückthun postulate that this was “due in large part to poorly characterized antibodies.” They go on to estimate that these poorly defined reagents are costing an estimated $350 million US dollars in lost productivity in the US alone!

Baker M (2015) provided some excellent commentary on the impact it has on research when poor and uncharacterized antibodies are sold to the scientific community. The stories of David Rimm and Loannis Prassas are cautionary tales for all researchers using antibodies, as are the results published in Egelhofer et al. (2011), where 25% of 200+ tested antibodies failed specificity tests.

The most common issues to watch out for include:

  • Batch-to-batch variability of the antibodies – both the production of the antibody and its labeling with fluorochromes.
  • Incorrect use of the antibody – most vendors will tell you about the applications in which their antibodies have been tested.
  • Poor characterization – the antibody may have binding to secondary (off-target) proteins. This is very difficult to catch in flow cytometry, so extensive controls are necessary.

While everyone has a favorite vendor from whom they purchase common reagents, not all antibodies are created equal. It may be difficult to know whom to trust. The website Benchsci.com is a site every flow cytometrist should bookmark when searching for reagents. Benchsci’s searchable database, which contains over 3.9 million antibodies, allows researchers to narrow down species, tissue type, and even application. These can be used to identify not only antibody vendors but publications that have used the reagent in question. Recently, publishers “Springer Nature” and “Wiley” allowed Benchsci users access to figures that are normally “closed-access,” increasing the power of this tool.

Now that the problem is clear–that the reagents used in flow cytometry can negatively impact research and discovery, or worse, clinical diagnostic testing–it will be important to understand:

  • How antibodies are sold and classified
  • Which tests MUST be performed for each new lot of reagents
  • How best to use these reagents
Reagents can have a negative impact on flow cytometry measurements

Before looking at the different classifications of reagents, it is important to look at a set of guidelines known as Good Manufacturing Practices, or GMP. These guidelines are overseen by governmental regulatory agencies. They are designed to ensure that products made under the GMP are consistently produced, and that the manufacturing process is rigorously controlled so that the final product meets a predefined set of quality standards.

Researchers rely on various manufacturers to produce reagents used in the laboratory. Antibodies can be purchased with several different designations, and these designations have more to do with the reagent than its cost alone. The ones most commonly used in the research lab are labeled RUO, which stands for “Research Use Only.” The RUO designation doesn’t offer a guarantee of quality, and generally, there is little effort to validate the reagent itself. In fact, when reading the technical specifications of an RUO reagent, one often finds a statement like this:

“… this product is for research use only and not intended for human or animal diagnostic, therapeutic or commercial use.”

While the vendor may do some testing and qualify the assays it will work in, there can be dramatic lot-to-lot variation using the same reagent from different batches. It is up to the vendors to decide how much to test or validate the reagent, and what they will report to the end-user; so caveat emptor applies.

The Analyte Specific Reagent designation, or ASR, is based on the US guideline 21 CFR 864.4020. These reagents must have some level of validation, and are prepared more carefully than an RUO reagent. They are intended for use in diagnostic application but must be validated in the laboratory. The FDA, the regulatory body governing these products, does not have a requirement on the performance of ASR reagents. It requires that these products carry the following on their label: “Analyte Specific Reagent. Analytical and performance characteristics are not established.”

Finally, there is IVD, or “In Vitro Diagnostic” reagents. These are usually sold as kits, and as the name implies, they are used for diagnostics. These reagents pass much stricter regulations and have been validated on actual clinical samples. Because of these regulations and validations, they are also the most expensive. Specifically, IVD reagents must meet a specific level of performance, and this must be demonstrated and documented the manufacturer.

Confused? Here is a table that summarizes the important characteristics of the differences in these three levels.

    
 RUOASRCE-IVD
Research UseYESYESYES
Diagnostic UseNOYESYES
FDA General ControlsYESYESNO
Manufactured under GMPNOYESYES
Labeling requirementsYESYESYES
Performance claimsYESN/AYES
Generate data to support performance claimsYESN/AYES
Facilitated laboratory accreditationNONOYES
Manufacturer audit by certification bodyYESYESYES

Unfortunately, there are many reagents that don’t exist at the IVD level, and the ASR designation is US-centric. This means that for most work, only RUO reagents are available. In this case, it becomes important to understand the concerns over the RUO and how to use these reagents in flow cytometry experiments.

Why? Because there can be significant lot-to-lot variation between RUO reagents, and if the lab using them does not address this lot-to-lot variation, it can lead to an extensive loss of work, a lack of reproducibility, and even incorrect conclusions.

Here is the take-home message as far as the consistency and reproducibility of flow cytometry assays: Look at the manufacturing standard that the vendor is following. Vendors who follow GMP will have the best consistency across reagent lots. Clinical reagents must be manufactured under GMP; this is mandatory. While RUO reagents do not share this requirement, some vendors have made it a policy to manufacture all reagents under GMP and its strict guidelines.

Good manufacturing practices ensures high-quality reagents for accurate flow cytometry measurements

FIGURE 1: “Good Manufacturing Practices” is the gold standard for ensuring high-quality reagents with lot-to-lot consistency.

In developing a polychromatic flow cytometry panel for research or clinical use, a great deal of thought needs to go into the development and pairing of fluorochrome and antibody. It is recommended that researchers and clinicians pair antigens expressed at a low level with brighter fluorochromes. They should also take into account any sensitivity-reducing issues brought about by the other fluorochromes in the panel. When determining brightness, the concept of the staining index is useful. As described in Maecker et al. (2004), the same antibody, conjugated to different fluorochromes, is used to label cells. After acquisition, the Staining Index (SI) is calculated. This is shown in Figure 2.

Calculating the staining index in flow cytometry

Figure 2: Calculating the Staining Index, as described by Maecker et al. (2004). The distance is the difference between the mean of the positive minus the mean of the negative. This is then divided by two times the spread of the negative, as measured by the standard deviation.

Of course, titration is a critical parameter for validating that the reagent is working, and that the correct staining concentration is being used. Too much antibody and there is a loss of signal due to non-specific binding of the reagent. Too little antibody and the signal decreases. An excellent review of this, and other issues that reduce sensitivity and impact the correct interpretation of data, can be found in Hulspar et al. (2006).

One other critical process for using antibodies to ensure high-quality data is the concept of the “master mix,” or antibody cocktails. This process is a very powerful tool for ensuring that each tube gets the correct about of antibody, and it saves on errors caused by pipetting small volumes.

What happens when an antibody cocktail is not used to stain flow cytometry samples

Figure 3: The impact of not using an antibody cocktail when staining samples. Sample 1 was stained with an antibody mix made on the fly, while Sample 2 and 3 were run from an antibody cocktail.

There are those who worry about antibody cocktails for staining, especially when staining using tandem dyes, which are known to have their own issues. In Johansson and Maecy (2014), the authors addressed this issue and demonstrated the stability of different tandem dyes within cocktails up through 8 weeks. The authors also evaluated the lot-to-lot variation between different tandem dyes, and recommend that each lot of tandem dyes be characterized to determine how similar or different they are from previously used reagents.

As Bernard of Chartres wrote in the 12th century, “nanos gigantum humeris insidentes,” which was later popularized by Sir Isaac Newton as, “If I have seen further, it is by standing on the shoulders of Giants.” Scientific progress is made as each new set of experiments builds on the previous one to see farther, peer deeper, and understand more fully the biology that is being studied. This can only happen when the earlier results are robust and reproducible. The paths to cures are out there, but when only 11% of the findings from 53 preclinical papers hold up to scrutiny, that road looks longer and harder to travel than ever. Reproducibility is a mindset that every scientist must apply to their research at every step of the way, and even the littlest detail needs to be explored to ensure that published results hold up to review and scrutiny.

References

Hulspar et al. (2006)

Maecker et al. (2004)

Johansson and Maecy (2014)

Baker M (2015)

Bradbury and Plückthun (2015)

Begley and Ellis (2012)

Begley et al., (2015)

Begley and Ioannidis (2015)

To learn more about Understanding Reproducibility in Flow Cytometry – It’s the Antibodies, and to get access to all of our advanced materials including 20 training videos, presentations, workbooks, and private group membership, get on the Flow Cytometry Mastery Class wait list.

Join Expert Cytometry's Mastery Class
Tim Bushnell, PhD
Tim Bushnell, PhD

Tim Bushnell holds a PhD in Biology from the Rensselaer Polytechnic Institute. He is a co-founder of—and didactic mind behind—ExCyte, the world’s leading flow cytometry training company, which organization boasts a veritable library of in-the-lab resources on sequencing, microscopy, and related topics in the life sciences.

Similar Articles

The Power Of Spectral Viewers And Their Use In Full Spectrum Flow Cytometry

The Power Of Spectral Viewers And Their Use In Full Spectrum Flow Cytometry

By: Tim Bushnell, PhD

What photon from yonder fluorochrome breaks?  It is … umm… hmmm. Let me see. Excitation off a 561 nm laser, with an emission maximum of 692 nm. I’m sure if Shakespeare was a flow cytometrist, he might have written that very scene. But the play is lost in time. However, since the protagonist had difficulty determining what fluorochrome was emitting photons, let’s consider how this could be figured out. In my opinion, one of the handiest flow cytometry tools is the spectral viewer. This tool helps visualize the excitation and emission profile of different fluorochromes, as well as allowing you…

Fickle Markers: Solutions For Antibody Binding Specificity Challenges

Fickle Markers: Solutions For Antibody Binding Specificity Challenges

By: Tim Bushnell, PhD

Reproducibility has been an ongoing, and important, concept in the sciences for years.  In the area of biomedical research, the alarm was sounded by several papers published in the early 2010’s.  Authors like Begley and Ellis, Prinz and coworkers, and Vasilevsky and colleagues, among others reported an alarming trend in the reproducibility of pre-clinical data.  These reports indicated between 50% to almost 90% of published pre-clinical data were not reproducible.  This was further highlighted in the article by Freedman and coworkers, who tried to identify and quantify the different sources of error that could be causing this crisis.  Figure 1,…

5 Common Flow Cytometry Questions, Answered

5 Common Flow Cytometry Questions, Answered

By: Tim Bushnell, PhD

I want to thank all of you who send us your questions about flow cytometry, so I thought I would dip into the old email bag and answer a few of the common ones here.  If your question isn’t answered this time, look for it to be answered in a future blog post.  Of course, if you want us to cover a specific topic, drop us a line.  1. How Fast Can I Go? This is  a common question. The allure of the ‘hi’ button is hard to resist.  The faster you go, the sooner you are finished with data…

Combining Flow Cytometry With Plant Science, Microorganisms, And The Environment

Combining Flow Cytometry With Plant Science, Microorganisms, And The Environment

By: Tim Bushnell, PhD

My first introduction to flow cytometry was talking to a professor who’d brought one on a research cruise to study phytoplankton. It was only later that I was introduced to the marvelous world that’s been my career for over 20 years.   In that time, I’ve had the opportunity to work with researchers in many different areas, exposing me to a wide variety of cell types and more important assays. What continues to amaze me is the number of different parameters we can measure, not just the number of fluorochromes, but the information we can extract from samples – animal, vegetable…

Common Numbers-Based Questions I Get As A Flow Cytometry Core Manager And How To Answer Them

Common Numbers-Based Questions I Get As A Flow Cytometry Core Manager And How To Answer Them

By: Tim Bushnell, PhD

Numbers are all around us.  My personal favorite is ≅1.618 aka ɸ aka ‘the golden ratio’.  It’s found throughout history, where it has influenced architects and artists. We see it in nature, in plants, and it is used in movies to frame shots. It can be approximated by the Fibonacci sequence (another math favorite of mine). However, I have not worked out how to apply this to flow cytometry.  That doesn’t mean numbers aren’t important in flow cytometry. They are central to everything we do, and in this blog, I’m going to flit around numbers-based questions that I have received…

3 Must-Have High-Dimensional Flow Cytometry Controls

3 Must-Have High-Dimensional Flow Cytometry Controls

By: Tim Bushnell, PhD

Developments such as the recent upgrade to the Cytobank analysis platform and the creation of new packages such as Immunocluster are reducing the computational expertise needed to work with high-dimensional flow cytometry datasets. Whether you are a researcher in academia, industry, or government, you may want to take advantage of the reduced barrier to entry to apply high-dimensional flow cytometry in your work. However, you’ll need the right experimental design to access the new transformative insights available through these approaches and avoid wasting the considerable time and money required for performing them. As with all experiments, a good design begins…

The Fluorochrome Less Excited: How To Build A Flow Cytometry Antibody Panel

The Fluorochrome Less Excited: How To Build A Flow Cytometry Antibody Panel

By: Tim Bushnell, PhD

Fluorochrome, antibodies and detectors are important. The journey of a thousand cells starts with a good fluorescent panel. The polychromatic panel is the combination of antibodies and fluorochromes. These will be used during the experiment to answer the biological question of interest. When you only need a few targets, the creation of the panel is relatively straightforward. It’s only when you start to get into more complex panels with multiple fluorochromes that overlap in excitation and emission gets more interesting.  FLUOROCHROMES Both full spectrum and traditional fluorescent flow cytometry rely on measuring the emission of the fluorochromes that are attached…

Flow Cytometry Year in Review: Key Changes To Know

Flow Cytometry Year in Review: Key Changes To Know

By: Meerambika Mishra

Here we are, at the end of an eventful year 2021. But with the promise of a new year 2022 to come. It has been a long year, filled with ups and downs. It is always good to reflect on the past year as we move to the future.  In Memoriam Sir Isaac Newton wrote “If I have seen further, it is by standing upon the shoulders of giants.” In the past year, we have lost some giants of our field including Zbigniew Darzynkiwicz, who contributed much in the areas of cell cycle analysis and apoptosis. Howard Shapiro, known for…

What Star Trek Taught Me About Flow Cytometry

What Star Trek Taught Me About Flow Cytometry

By: Tim Bushnell, PhD

It is no secret that I am a very big fan of the Star Trek franchise. There are many good episodes and lessons explored in the 813+ episodes, 12 movies (and counting). Don’t worry, this blog is not going to review all 813, or even 5 of them. Instead, some of the lessons I have taken away from the show that have applicability to science and flow cytometry.  “Darmok and Jalad at Tanagra.”  (ST:TNG season 5, episode 2) This is probably one of my favorite episodes, which involves Picard and an alien trying to establish a common ground and learn…

Top Technical Training eBooks

Get the Advanced Microscopy eBook

Get the Advanced Microscopy eBook

Heather Brown-Harding, PhD

Learn the best practices and advanced techniques across the diverse fields of microscopy, including instrumentation, experimental setup, image analysis, figure preparation, and more.

Get The Free Modern Flow Cytometry eBook

Get The Free Modern Flow Cytometry eBook

Tim Bushnell, PhD

Learn the best practices of flow cytometry experimentation, data analysis, figure preparation, antibody panel design, instrumentation and more.

Get The Free 4-10 Compensation eBook

Get The Free 4-10 Compensation eBook

Tim Bushnell, PhD

Advanced 4-10 Color Compensation, Learn strategies for designing advanced antibody compensation panels and how to use your compensation matrix to analyze your experimental data.