4 Ways To Achieve Reproducible Flow Cytometry Results

Reproducibility is one of the hallmarks of the Scientific Method. One reason scientists publish their results is so that other labs can attempt to reproduce the results and extend the investigation into new areas.

From the Begley and Ellis commentary in Nature, to the development of the Rigor and Reproducibility initiative at NIH, to articles in the mainstream media, reproducibility is on everyone’s mind. This is not a bad thing, and following the best practices in flow cytometry provides investigators, peer reviewers, and colleagues with more confidence in your data.

There are several areas within the process of developing, implementing, and reporting a flow cytometry experiment where a little additional work, attention to proper controls, and careful planning will ensure reproducible data generation.

Instrumentation: Characterization, Optimization, and Quality Control.

The first line of defense for good, reproducible data is the instrument.

A properly maintained machine ensures that it is not introducing error in the data. A researcher needs to know that when they find a difference, it’s due to the experiment, not the instrument. Thus, quality control is the key.

In the recent “Best Practices” article from the ISAC Shared Resource Laboratory Taskforce (Barksy et al. 2016), the authors discuss the importance of QA and QC, and highlight the different possible levels — including the idea of an external audit.

It is critical to remember that QC is only good if it is being monitored. Running beads without looking back at the trends over a week, a month, or a quarter doesn’t help anyone.

The end-user should not be shy about asking those in charge of the instrument to see the QC reports or Levey-Jennings plots (Figure 1).

The end-user can also take ownership of QC for their experiments by incorporating a bead standard that is run before each experiment in the series, and monitored. Such a practice was discussed in a recent paper by Misra and co-workers (2016).

Figure 1. Levey-Jennings graph of QC data monitoring PMT voltage changes over a 100-day period. The blue line represents the mean, and the black lines +/- 1 and 2 standard deviations around the mean. Depending on the experimental needs, this data can be used to determine when an intervention is necessary before running a critical sample.

Another area that investigators should look for, regarding the instrument, is optimization of the system. At a minimum, the optimal PMT voltages and linear dynamic range should be available to all users. Additional characterization, such as the protocols discussed in this paper by Perfetto and co-workers (2012), can help with panel design.

It may seem like a lot of work, but what is the cost of a ruined experiment? Lost data? It is critical that researchers begin to ask for this level of characterization from their instruments, and that those operating them are given the time and resources necessary to perform such characterizations.

Reagents: Cells, Antibodies, and Buffers.

In many cases, our flow cytometry experiments will be measuring a biological process in cells and cell lines using fluorescently tagged antibodies.

The issue of cross-contamination of cell lines is well-documented, and it is incumbent on the investigator using cell lines to provide validation of the cell line in question. With advances in next generation sequencing and proliferation of commercial services to perform this validation, it is critical that you know the cell line you are using.

A more deeply-rooted problem may be naming conventions. In a recent Nature article, Yu and colleagues suggested a new framework for naming cell lines.

Antibody naming conventions have benefited from the Human Leucocyte Differentiation Antigens (HLDA) workshops, which share information on their website about different antigens and antibody clones available for investigators.

Regarding antibodies, optimization of these reagents is an essential component to good flow cytometry. Titration is the first important step towards characterization of a given antibody. Performing this assay allows for determining the best concentration for your experimental assays, as well as validating that the antibody is working.

Figure 2. Titration of an antibody. On the left is the concatenated file with increasing antibody concentration, and on the right is the calculated Staining Index calculated per Maecker et al. (2004).

Additional validation steps of the antibody is a critical discussion that is ongoing in the literature.

While we currently rely on vendors to ensure their processes provide us with what the label says, a recent letter in Nature by Bradbury and Plückthun (and 110 co-signers) highlighted our dependence on protein binding reagents and their current limitations, as well as the magnitude of research dollars lost to poor quality reagents. It will bear watching to see how the industry moves in the directions that this letter advocates.

A quick comment on buffers and reagents — many investigators rely on purchasing pre-mixed reagents as a way to improve consistency and lot-to-lot variation over house-made reagents. It is important to ensure that you have a proper training and validation protocol for reagents, regardless of their origin, to make sure that they are not negatively affecting your data.

A cautionary tale on using pre-made reagents is found in this paper by Zarkowsky and co-workers (2011). Here, the researchers noticed a loss of quantum dot fluorescence with some lots of commercial formalin solutions.

After characterization of different lots, it was discovered that copper ions, a micro-contaminant, were the culprits (Figure 3). The solution to the problem was simply to add EDTA to the fixative, which chelated the Cu++ ions.

Figure 3. Figure 1 from Zarkowsky’s paper, showing the effect on Qdot fluorescence of three different lots of fixation buffer (+/- EDTA).

One can only imagine how many investigators lost data between the adoption of quantum dots in flow cytometry and the publication of this paper. It serves as a cautionary tale for researchers adopting newly available fluorochromes to delve a little deeper into their buffers in the case of anomalous results.

Process: Monitoring Human Error.

Variability in data can be a result of sample processing. There are many steps between isolating the cells and acquiring data. Development and use of standard operating procedures (SOPs) can help mitigate variations. A mainstay of clinical labs and GMP production facilities, more and more basic research labs are beginning to adopt SOPs. The aforementioned Barksy et al. 2016 paper spends a good deal of time talking about their role in flow cytometry.

Another way to monitor the sample preparation process is to develop and validate a “reference” or “process” control. This is a standard sample that is run each time the assay is performed.

Knowing how this standard sample performs in the assay will allow the researcher to set acceptable limits and know if there was an issue with part of the process (Figure 4). The severity of the deviation determines whether it is possible to troubleshoot and move the assay forward, or if the run has to be performed again. Likewise, it can point to instrument issues that developed between the QC and the time the investigator ran the assay. Of course, as with any QC system, it must be monitored.

Figure 4. Process control sample showing the range of expression for different populations in an assay.

Analysis: Extracting the Correct Data.

The fourth area for consideration to improve reproducibility of flow cytometry results is data analysis. This is the area where the ‘science’ of cytometry meets the ‘art’ of analysis, and the one that causes the most headaches.

At present, most flow cytometry analysis relies on manual bi-variant hierarchical gating strategies. As more robust automated workflows are developed, they may become the dominant analysis methods, reducing or removing human bias from the process. Until that time, however, a few simple rules can help improve analysis and reduce errors.

Before analysis begins, it is important to understand what the biological question is, what data needs to be extracted from the experiment, and what downstream analysis will be performed on the data. This will drive the development of the analysis strategy.

To develop a robust analysis strategy, one must understand best practices for gate setting. This is the basis of a separate blog post, as there are many factors involved in that process.

Good analysis requires using appropriate controls, understanding display options, and developing consistent rules, once again leading to the SOP.

In the recent update to FCS Express, they included an SOP builder in the software. This allows for the creation of an analysis SOPs for lab members to follow. Checkpoints can be built in throughout the analysis, so that the researcher moves through step-by-step, and will not proceed to the next step unless the requirements are met. It’s a great feature for a lab performing a standard analysis over time, so that everyone performs the analysis the same way.

Consistent analysis is critical, as was shown in a study published by Maecker and coworkers (2010), which compared the analysis of prestained cells in 15 experienced labs versus analysis in one central lab. The results showed a mean CV of 20.5% across the four samples for analysis by the remote labs, versus 4% when the analysis was performed in a central lab. That level of data spread can make it extremely difficult to find rare differences, thus reducing the likelihood of advancing the field.

For consistency in gating, relying on more normal shapes for gates (rather than free-form drawings) is an important consideration (Figure 5). Couple that with the use of FMO controls, establishing cut-off percentages for gates, and acquiring appropriate biological controls, one can improve the consistency of results.

Figure 5. Sample analysis showing a gating strategy for analysis of a multi-year study.

Finally, because there is a great deal of information that cannot makes its way into a standard materials and methods section, the ISAC Data Standards Task Force developed and published the “Minimal Information about a Flow Cytometry Experiment” or MIFLowCyt standard. This standard establishes a detailed checklist covering four major areas:

  • Experimental overview
  • Sample and specimen description
  • Instrument details
  • Data analysis details

Papers published that are compliant with this standard ensure an additional level of information is available to researchers seeking to reproduce the data.

At present, this standard has not been widely adopted, but is strongly encouraged for submissions to Cytometry A. A checklist is available from the journal to assist in ensuring a submitted paper has met the reporting requirements of this standard.

Additionally, ISAC has developed and supports the FlowRepository, a database where investigators can submit their data. This database, first described in a paper by Spidlen and coworkers, allows peer reviewers access to the raw data used in a paper so they can examine the data to improve the review process.

After publication, the data is available to investigators as a way for them to compare their results with published data.

An excellent example of the FlowRepository in action are the Optimized Multicolor Immunofluorescence Panels (OMIPS). These papers describe the development of new polychromatic flow cytometry panels for the research community, and are required to show the gating strategy used by the authors. Having the primary data available to the research community again offers a way to combat issues in reproducibility.

In conclusion, there are several areas that researchers can focus on to improve the reproducibility of their flow cytometry experiments. From instrument quality control, through validation of reagents, to reporting out the findings, a little effort will go a long way to ensure that flow cytometry data is robust, reproducible, and accurately reported to the greater scientific community. Initiatives by ISAC have further offered additional levels of standards to support these initiatives, which were developed even before the Reproducibility Crisis came to a head in both scientific and popular literature.

As scientists, we owe it to ourselves, our colleagues, and the public to ensure the data we generate is of the highest quality. It is Isaac Newton who is attributed to saying, “If I have seen further, it is by standing on the shoulders of giants.” Together, following the steps outlined above, we can each stand on the shoulders of our colleagues to move scientific discovery forward, with the ultimate goal of improving the health and well-being of our fellow man.

To learn more about 4 Ways To Achieve Reproducible Flow Cytometry Results, and to get access to all of our advanced materials including 20 training videos, presentations, workbooks, and private group membership, get on the Flow Cytometry Mastery Class wait list.

Join Expert Cytometry's Mastery Class
Tim Bushnell, PhD
Tim Bushnell, PhD

Tim Bushnell holds a PhD in Biology from the Rensselaer Polytechnic Institute. He is a co-founder of—and didactic mind behind—ExCyte, the world’s leading flow cytometry training company, which organization boasts a veritable library of in-the-lab resources on sequencing, microscopy, and related topics in the life sciences.

Similar Articles

The Power Of Spectral Viewers And Their Use In Full Spectrum Flow Cytometry

The Power Of Spectral Viewers And Their Use In Full Spectrum Flow Cytometry

By: Tim Bushnell, PhD

What photon from yonder fluorochrome breaks?  It is … umm… hmmm. Let me see. Excitation off a 561 nm laser, with an emission maximum of 692 nm. I’m sure if Shakespeare was a flow cytometrist, he might have written that very scene. But the play is lost in time. However, since the protagonist had difficulty determining what fluorochrome was emitting photons, let’s consider how this could be figured out. In my opinion, one of the handiest flow cytometry tools is the spectral viewer. This tool helps visualize the excitation and emission profile of different fluorochromes, as well as allowing you…

Fickle Markers: Solutions For Antibody Binding Specificity Challenges

Fickle Markers: Solutions For Antibody Binding Specificity Challenges

By: Tim Bushnell, PhD

Reproducibility has been an ongoing, and important, concept in the sciences for years.  In the area of biomedical research, the alarm was sounded by several papers published in the early 2010’s.  Authors like Begley and Ellis, Prinz and coworkers, and Vasilevsky and colleagues, among others reported an alarming trend in the reproducibility of pre-clinical data.  These reports indicated between 50% to almost 90% of published pre-clinical data were not reproducible.  This was further highlighted in the article by Freedman and coworkers, who tried to identify and quantify the different sources of error that could be causing this crisis.  Figure 1,…

5 Common Flow Cytometry Questions, Answered

5 Common Flow Cytometry Questions, Answered

By: Tim Bushnell, PhD

I want to thank all of you who send us your questions about flow cytometry, so I thought I would dip into the old email bag and answer a few of the common ones here.  If your question isn’t answered this time, look for it to be answered in a future blog post.  Of course, if you want us to cover a specific topic, drop us a line.  1. How Fast Can I Go? This is  a common question. The allure of the ‘hi’ button is hard to resist.  The faster you go, the sooner you are finished with data…

Combining Flow Cytometry With Plant Science, Microorganisms, And The Environment

Combining Flow Cytometry With Plant Science, Microorganisms, And The Environment

By: Tim Bushnell, PhD

My first introduction to flow cytometry was talking to a professor who’d brought one on a research cruise to study phytoplankton. It was only later that I was introduced to the marvelous world that’s been my career for over 20 years.   In that time, I’ve had the opportunity to work with researchers in many different areas, exposing me to a wide variety of cell types and more important assays. What continues to amaze me is the number of different parameters we can measure, not just the number of fluorochromes, but the information we can extract from samples – animal, vegetable…

Common Numbers-Based Questions I Get As A Flow Cytometry Core Manager And How To Answer Them

Common Numbers-Based Questions I Get As A Flow Cytometry Core Manager And How To Answer Them

By: Tim Bushnell, PhD

Numbers are all around us.  My personal favorite is ≅1.618 aka ɸ aka ‘the golden ratio’.  It’s found throughout history, where it has influenced architects and artists. We see it in nature, in plants, and it is used in movies to frame shots. It can be approximated by the Fibonacci sequence (another math favorite of mine). However, I have not worked out how to apply this to flow cytometry.  That doesn’t mean numbers aren’t important in flow cytometry. They are central to everything we do, and in this blog, I’m going to flit around numbers-based questions that I have received…

3 Must-Have High-Dimensional Flow Cytometry Controls

3 Must-Have High-Dimensional Flow Cytometry Controls

By: Tim Bushnell, PhD

Developments such as the recent upgrade to the Cytobank analysis platform and the creation of new packages such as Immunocluster are reducing the computational expertise needed to work with high-dimensional flow cytometry datasets. Whether you are a researcher in academia, industry, or government, you may want to take advantage of the reduced barrier to entry to apply high-dimensional flow cytometry in your work. However, you’ll need the right experimental design to access the new transformative insights available through these approaches and avoid wasting the considerable time and money required for performing them. As with all experiments, a good design begins…

The Fluorochrome Less Excited: How To Build A Flow Cytometry Antibody Panel

The Fluorochrome Less Excited: How To Build A Flow Cytometry Antibody Panel

By: Tim Bushnell, PhD

Fluorochrome, antibodies and detectors are important. The journey of a thousand cells starts with a good fluorescent panel. The polychromatic panel is the combination of antibodies and fluorochromes. These will be used during the experiment to answer the biological question of interest. When you only need a few targets, the creation of the panel is relatively straightforward. It’s only when you start to get into more complex panels with multiple fluorochromes that overlap in excitation and emission gets more interesting.  FLUOROCHROMES Both full spectrum and traditional fluorescent flow cytometry rely on measuring the emission of the fluorochromes that are attached…

Flow Cytometry Year in Review: Key Changes To Know

Flow Cytometry Year in Review: Key Changes To Know

By: Meerambika Mishra

Here we are, at the end of an eventful year 2021. But with the promise of a new year 2022 to come. It has been a long year, filled with ups and downs. It is always good to reflect on the past year as we move to the future.  In Memoriam Sir Isaac Newton wrote “If I have seen further, it is by standing upon the shoulders of giants.” In the past year, we have lost some giants of our field including Zbigniew Darzynkiwicz, who contributed much in the areas of cell cycle analysis and apoptosis. Howard Shapiro, known for…

What Star Trek Taught Me About Flow Cytometry

What Star Trek Taught Me About Flow Cytometry

By: Tim Bushnell, PhD

It is no secret that I am a very big fan of the Star Trek franchise. There are many good episodes and lessons explored in the 813+ episodes, 12 movies (and counting). Don’t worry, this blog is not going to review all 813, or even 5 of them. Instead, some of the lessons I have taken away from the show that have applicability to science and flow cytometry.  “Darmok and Jalad at Tanagra.”  (ST:TNG season 5, episode 2) This is probably one of my favorite episodes, which involves Picard and an alien trying to establish a common ground and learn…

Top Technical Training eBooks

Get the Advanced Microscopy eBook

Get the Advanced Microscopy eBook

Heather Brown-Harding, PhD

Learn the best practices and advanced techniques across the diverse fields of microscopy, including instrumentation, experimental setup, image analysis, figure preparation, and more.

Get The Free Modern Flow Cytometry eBook

Get The Free Modern Flow Cytometry eBook

Tim Bushnell, PhD

Learn the best practices of flow cytometry experimentation, data analysis, figure preparation, antibody panel design, instrumentation and more.

Get The Free 4-10 Compensation eBook

Get The Free 4-10 Compensation eBook

Tim Bushnell, PhD

Advanced 4-10 Color Compensation, Learn strategies for designing advanced antibody compensation panels and how to use your compensation matrix to analyze your experimental data.