4 Ways To Achieve Reproducible Flow Cytometry Results

Reproducibility is one of the hallmarks of the Scientific Method. One reason scientists publish their results is so that other labs can attempt to reproduce the results and extend the investigation into new areas.

From the Begley and Ellis commentary in Nature, to the development of the Rigor and Reproducibility initiative at NIH, to articles in the mainstream media, reproducibility is on everyone’s mind. This is not a bad thing, and following the best practices in flow cytometry provides investigators, peer reviewers, and colleagues with more confidence in your data.

There are several areas within the process of developing, implementing, and reporting a flow cytometry experiment where a little additional work, attention to proper controls, and careful planning will ensure reproducible data generation.

Instrumentation: Characterization, Optimization, and Quality Control.

The first line of defense for good, reproducible data is the instrument.

A properly maintained machine ensures that it is not introducing error in the data. A researcher needs to know that when they find a difference, it’s due to the experiment, not the instrument. Thus, quality control is the key.

In the recent “Best Practices” article from the ISAC Shared Resource Laboratory Taskforce (Barksy et al. 2016), the authors discuss the importance of QA and QC, and highlight the different possible levels — including the idea of an external audit.

It is critical to remember that QC is only good if it is being monitored. Running beads without looking back at the trends over a week, a month, or a quarter doesn’t help anyone.

The end-user should not be shy about asking those in charge of the instrument to see the QC reports or Levey-Jennings plots (Figure 1).

The end-user can also take ownership of QC for their experiments by incorporating a bead standard that is run before each experiment in the series, and monitored. Such a practice was discussed in a recent paper by Misra and co-workers (2016).

Figure 1. Levey-Jennings graph of QC data monitoring PMT voltage changes over a 100-day period. The blue line represents the mean, and the black lines +/- 1 and 2 standard deviations around the mean. Depending on the experimental needs, this data can be used to determine when an intervention is necessary before running a critical sample.

Another area that investigators should look for, regarding the instrument, is optimization of the system. At a minimum, the optimal PMT voltages and linear dynamic range should be available to all users. Additional characterization, such as the protocols discussed in this paper by Perfetto and co-workers (2012), can help with panel design.

It may seem like a lot of work, but what is the cost of a ruined experiment? Lost data? It is critical that researchers begin to ask for this level of characterization from their instruments, and that those operating them are given the time and resources necessary to perform such characterizations.

Reagents: Cells, Antibodies, and Buffers.

In many cases, our flow cytometry experiments will be measuring a biological process in cells and cell lines using fluorescently tagged antibodies.

The issue of cross-contamination of cell lines is well-documented, and it is incumbent on the investigator using cell lines to provide validation of the cell line in question. With advances in next generation sequencing and proliferation of commercial services to perform this validation, it is critical that you know the cell line you are using.

A more deeply-rooted problem may be naming conventions. In a recent Nature article, Yu and colleagues suggested a new framework for naming cell lines.

Antibody naming conventions have benefited from the Human Leucocyte Differentiation Antigens (HLDA) workshops, which share information on their website about different antigens and antibody clones available for investigators.

Regarding antibodies, optimization of these reagents is an essential component to good flow cytometry. Titration is the first important step towards characterization of a given antibody. Performing this assay allows for determining the best concentration for your experimental assays, as well as validating that the antibody is working.

Figure 2. Titration of an antibody. On the left is the concatenated file with increasing antibody concentration, and on the right is the calculated Staining Index calculated per Maecker et al. (2004).

Additional validation steps of the antibody is a critical discussion that is ongoing in the literature.

While we currently rely on vendors to ensure their processes provide us with what the label says, a recent letter in Nature by Bradbury and Plückthun (and 110 co-signers) highlighted our dependence on protein binding reagents and their current limitations, as well as the magnitude of research dollars lost to poor quality reagents. It will bear watching to see how the industry moves in the directions that this letter advocates.

A quick comment on buffers and reagents — many investigators rely on purchasing pre-mixed reagents as a way to improve consistency and lot-to-lot variation over house-made reagents. It is important to ensure that you have a proper training and validation protocol for reagents, regardless of their origin, to make sure that they are not negatively affecting your data.

A cautionary tale on using pre-made reagents is found in this paper by Zarkowsky and co-workers (2011). Here, the researchers noticed a loss of quantum dot fluorescence with some lots of commercial formalin solutions.

After characterization of different lots, it was discovered that copper ions, a micro-contaminant, were the culprits (Figure 3). The solution to the problem was simply to add EDTA to the fixative, which chelated the Cu++ ions.

Figure 3. Figure 1 from Zarkowsky’s paper, showing the effect on Qdot fluorescence of three different lots of fixation buffer (+/- EDTA).

One can only imagine how many investigators lost data between the adoption of quantum dots in flow cytometry and the publication of this paper. It serves as a cautionary tale for researchers adopting newly available fluorochromes to delve a little deeper into their buffers in the case of anomalous results.

Process: Monitoring Human Error.

Variability in data can be a result of sample processing. There are many steps between isolating the cells and acquiring data. Development and use of standard operating procedures (SOPs) can help mitigate variations. A mainstay of clinical labs and GMP production facilities, more and more basic research labs are beginning to adopt SOPs. The aforementioned Barksy et al. 2016 paper spends a good deal of time talking about their role in flow cytometry.

Another way to monitor the sample preparation process is to develop and validate a “reference” or “process” control. This is a standard sample that is run each time the assay is performed.

Knowing how this standard sample performs in the assay will allow the researcher to set acceptable limits and know if there was an issue with part of the process (Figure 4). The severity of the deviation determines whether it is possible to troubleshoot and move the assay forward, or if the run has to be performed again. Likewise, it can point to instrument issues that developed between the QC and the time the investigator ran the assay. Of course, as with any QC system, it must be monitored.

Figure 4. Process control sample showing the range of expression for different populations in an assay.

Analysis: Extracting the Correct Data.

The fourth area for consideration to improve reproducibility of flow cytometry results is data analysis. This is the area where the ‘science’ of cytometry meets the ‘art’ of analysis, and the one that causes the most headaches.

At present, most flow cytometry analysis relies on manual bi-variant hierarchical gating strategies. As more robust automated workflows are developed, they may become the dominant analysis methods, reducing or removing human bias from the process. Until that time, however, a few simple rules can help improve analysis and reduce errors.

Before analysis begins, it is important to understand what the biological question is, what data needs to be extracted from the experiment, and what downstream analysis will be performed on the data. This will drive the development of the analysis strategy.

To develop a robust analysis strategy, one must understand best practices for gate setting. This is the basis of a separate blog post, as there are many factors involved in that process.

Good analysis requires using appropriate controls, understanding display options, and developing consistent rules, once again leading to the SOP.

In the recent update to FCS Express, they included an SOP builder in the software. This allows for the creation of an analysis SOPs for lab members to follow. Checkpoints can be built in throughout the analysis, so that the researcher moves through step-by-step, and will not proceed to the next step unless the requirements are met. It’s a great feature for a lab performing a standard analysis over time, so that everyone performs the analysis the same way.

Consistent analysis is critical, as was shown in a study published by Maecker and coworkers (2010), which compared the analysis of prestained cells in 15 experienced labs versus analysis in one central lab. The results showed a mean CV of 20.5% across the four samples for analysis by the remote labs, versus 4% when the analysis was performed in a central lab. That level of data spread can make it extremely difficult to find rare differences, thus reducing the likelihood of advancing the field.

For consistency in gating, relying on more normal shapes for gates (rather than free-form drawings) is an important consideration (Figure 5). Couple that with the use of FMO controls, establishing cut-off percentages for gates, and acquiring appropriate biological controls, one can improve the consistency of results.

Figure 5. Sample analysis showing a gating strategy for analysis of a multi-year study.

Finally, because there is a great deal of information that cannot makes its way into a standard materials and methods section, the ISAC Data Standards Task Force developed and published the “Minimal Information about a Flow Cytometry Experiment” or MIFLowCyt standard. This standard establishes a detailed checklist covering four major areas:

  • Experimental overview
  • Sample and specimen description
  • Instrument details
  • Data analysis details

Papers published that are compliant with this standard ensure an additional level of information is available to researchers seeking to reproduce the data.

At present, this standard has not been widely adopted, but is strongly encouraged for submissions to Cytometry A. A checklist is available from the journal to assist in ensuring a submitted paper has met the reporting requirements of this standard.

Additionally, ISAC has developed and supports the FlowRepository, a database where investigators can submit their data. This database, first described in a paper by Spidlen and coworkers, allows peer reviewers access to the raw data used in a paper so they can examine the data to improve the review process.

After publication, the data is available to investigators as a way for them to compare their results with published data.

An excellent example of the FlowRepository in action are the Optimized Multicolor Immunofluorescence Panels (OMIPS). These papers describe the development of new polychromatic flow cytometry panels for the research community, and are required to show the gating strategy used by the authors. Having the primary data available to the research community again offers a way to combat issues in reproducibility.

In conclusion, there are several areas that researchers can focus on to improve the reproducibility of their flow cytometry experiments. From instrument quality control, through validation of reagents, to reporting out the findings, a little effort will go a long way to ensure that flow cytometry data is robust, reproducible, and accurately reported to the greater scientific community. Initiatives by ISAC have further offered additional levels of standards to support these initiatives, which were developed even before the Reproducibility Crisis came to a head in both scientific and popular literature.

As scientists, we owe it to ourselves, our colleagues, and the public to ensure the data we generate is of the highest quality. It is Isaac Newton who is attributed to saying, “If I have seen further, it is by standing on the shoulders of giants.” Together, following the steps outlined above, we can each stand on the shoulders of our colleagues to move scientific discovery forward, with the ultimate goal of improving the health and well-being of our fellow man.

To learn more about 4 Ways To Achieve Reproducible Flow Cytometry Results, and to get access to all of our advanced materials including 20 training videos, presentations, workbooks, and private group membership, get on the Flow Cytometry Mastery Class wait list.

Join Expert Cytometry's Mastery Class
Tim Bushnell, PhD
Tim Bushnell, PhD

Tim Bushnell holds a PhD in Biology from the Rensselaer Polytechnic Institute. He is a co-founder of—and didactic mind behind—ExCyte, the world’s leading flow cytometry training company, which organization boasts a veritable library of in-the-lab resources on sequencing, microscopy, and related topics in the life sciences.

Similar Articles

How To Profile DNA And RNA Expression Using Next Generation Sequencing

How To Profile DNA And RNA Expression Using Next Generation Sequencing

By: Deepak Kumar, PhD

Why is Next Generation Sequencing so powerful to explore and answer both clinical and research questions. With the ability to sequence whole genomes, identifying novel changes between individuals, to exploring what RNA sequences are being expressed, or to examine DNA modifications and protein-DNA interactions occurring that can help researchers better understand the complex regulation of transcription. This, in turn, allows them to characterize changes during different disease states, which can suggest a way to treat said disease.  Over the next two blogs, I will highlight these different methods along with illustrating how these can help clinical diagnostics as well as…

Optimizing Flow Cytometry Experiments - Part 2         How To Block Samples (Sample Blocking)

Optimizing Flow Cytometry Experiments - Part 2 How To Block Samples (Sample Blocking)

By: Tim Bushnell, PhD

In my previous blog on  experimental optimization, we discussed the idea of identifying the best antibody concentration for staining the cells. We did this through a process called titration, which  focuses on finding the best signal-to-noise ratio at the lowest antibody concentration. In this blog we will deal with sample blocking As a reminder, there are two other major binding concerns with antibodies. The first is the specific binding of the Fc fragment of the antibody to the Fc Receptor expressed on some cells. This protein is critical for the process of destroying microbes or other cells that have been…

What Is Next Generation Sequencing (NGS) And How Is It Used In Drug Development

What Is Next Generation Sequencing (NGS) And How Is It Used In Drug Development

By: Deepak Kumar, PhD

NGS methodologies have been used to produce high-throughput sequence data. These data with appropriate computational analyses facilitate variant identification and prove to be extremely valuable in pharmaceutical industries and clinical practice for developing drug molecules inhibiting disease progression. Thus, by providing a comprehensive profile of an individual’s variome — particularly that of clinical relevance consisting of pathogenic variants — NGS helps in determining new disease genes. The information thus obtained on genetic variations and the target disease genes can be used by the Pharma companies to develop drugs impeding these variants and their disease-causing effect. However simple this may allude…

How To Determine The Optimal Antibody Concentration For Your Flow Cytometry Experiment (Part 1 of 6)

How To Determine The Optimal Antibody Concentration For Your Flow Cytometry Experiment (Part 1 of 6)

By: Tim Bushnell, PhD

Over the next series of blog posts, we will explore the different aspects of optimizing a polychromatic flow cytometry panel. These steps range from figuring out the best voltage to use, which controls are critical for data interpretation, what quality control tools can be integrated into the assay; how to block cells, and more. This blog will focus on determining the optimal antibody concentration.  As a reminder about the antibody structure, a schematic of an antibody is shown below.  Figure 1: Schematic of an antibody. Figure from Wikipedia. The antibody is composed of two heavy chains and two light chains that…

Brightness Is In The Eye Of The Detector - What To Consider When Designing Your Panel

Brightness Is In The Eye Of The Detector - What To Consider When Designing Your Panel

By: Tim Bushnell, PhD

The heart and soul of the flow cytometry experiment is the ‘panel.’ The unique combinations of antibodies, antigens, fluorochromes, and other reagents are central to identifying the cells of interest and extracting the data necessary to answer the question at hand. Designing the right panel for flow cytometry is essential for detecting different modalities. The more parameters that can be interrogated will yield more information about the target cells. Current instruments can measure as many as 40 different parameters simultaneously. This is exciting, as it allows for more complex questions to be studied. Panel design is also valuable for precious samples,…

The 5 Fundamental Methods For Imaging Nucleic Acids

The 5 Fundamental Methods For Imaging Nucleic Acids

By: Heather Brown-Harding, PhD

There are 4 major ways to sort cells. The first way can use magnetic beads coupled to antibodies and pass the cells through a magnetic field. The labeled cells will stick, and the unlabeled cells will remain in the supernatant. The second way is to use some sort of mechanical force like a flapper or air stream that separates the target cells from the bulk population. The third way is the recently introduced microfluidics sorter, which uses microfluidics channels to isolate the target cells. The last method, which is the most common––based on Fuwyler’s work––is the electrostatic cell sorter. This…

Avoid Flow Cytometry Faux Pas: How To Set Voltage The Right Way

Avoid Flow Cytometry Faux Pas: How To Set Voltage The Right Way

By: Tim Bushnell, PhD

There are 4 major ways to sort cells. The first way can use magnetic beads coupled to antibodies and pass the cells through a magnetic field. The labeled cells will stick, and the unlabeled cells will remain in the supernatant. The second way is to use some sort of mechanical force like a flapper or air stream that separates the target cells from the bulk population. The third way is the recently introduced microfluidics sorter, which uses microfluidics channels to isolate the target cells. The last method, which is the most common––based on Fuwyler’s work––is the electrostatic cell sorter. This…

Designing Microscopy Experiments Related To Infectious Diseases And Antivirals

Designing Microscopy Experiments Related To Infectious Diseases And Antivirals

By: Heather Brown-Harding, PhD

There are 4 major ways to sort cells. The first way can use magnetic beads coupled to antibodies and pass the cells through a magnetic field. The labeled cells will stick, and the unlabeled cells will remain in the supernatant. The second way is to use some sort of mechanical force like a flapper or air stream that separates the target cells from the bulk population. The third way is the recently introduced microfluidics sorter, which uses microfluidics channels to isolate the target cells. The last method, which is the most common––based on Fuwyler’s work––is the electrostatic cell sorter. This…

My 3-Step Panel Validation Pocket Guide

My 3-Step Panel Validation Pocket Guide

By: Tim Bushnell, PhD

There are 4 major ways to sort cells. The first way can use magnetic beads coupled to antibodies and pass the cells through a magnetic field. The labeled cells will stick, and the unlabeled cells will remain in the supernatant. The second way is to use some sort of mechanical force like a flapper or air stream that separates the target cells from the bulk population. The third way is the recently introduced microfluidics sorter, which uses microfluidics channels to isolate the target cells. The last method, which is the most common––based on Fuwyler’s work––is the electrostatic cell sorter. This…

Top Technical Training eBooks

Get the Advanced Microscopy eBook

Get the Advanced Microscopy eBook

Heather Brown-Harding, PhD

Learn the best practices and advanced techniques across the diverse fields of microscopy, including instrumentation, experimental setup, image analysis, figure preparation, and more.

Get The Free Modern Flow Cytometry eBook

Get The Free Modern Flow Cytometry eBook

Tim Bushnell, PhD

Learn the best practices of flow cytometry experimentation, data analysis, figure preparation, antibody panel design, instrumentation and more.

Get The Free 4-10 Compensation eBook

Get The Free 4-10 Compensation eBook

Tim Bushnell, PhD

Advanced 4-10 Color Compensation, Learn strategies for designing advanced antibody compensation panels and how to use your compensation matrix to analyze your experimental data.