5 Experimental Errors That Prevent Your Flow Cytometry Data From Passing The Peer Review Process

Reproducibility is a critical component of the scientific process. One cannot publish data if the experiments cannot be replicated are are full of errors.

Unfortunately, as Begley and Ellis pointed out in a commentary in Nature that when Amgen attempted to reproduce 53 “landmark” papers in the area of cancer research, only 6 papers were “confirmed.”

What does that mean to you, the flow cytometry researcher?

To avoid publishing errors that reviews despise, it’s important to follow and promote the best practices in the field, thus ensuring that your data is reproducible to investigators attempting to validate your research.

In particular, you must follow these 5 experimental tips before embarking into the peer review process…

Error 1: Know what your hypothesis is and design the experiments to test your hypothesis.

It may seem obvious, but the goal of the experiments is to prove some hypothesis that you are testing. In examining the hypothesis, there are several questions that should be on your mind…

A. Is flow cytometry the right tool to answer the question?—Sometimes flow isn’t the best way to answer the question so it’s better to make the decision before starting experiments.

B. Is flow cytometry a path to another technique?—For example, will sorted cells be used to perform experiments such as RNAseq, Proteomic analysis, cell culture etc? If so, what controls will you need to build into the experiment to help evaluate the handoff between technologies?

C. Is there another technology that will confirm the findings?—It is always useful to have data from two different technologies that can confirm each other. For instance, flow cytometry brings statistical power to examining cell populations from other experiments.

Error 2: Know which statistical tests are appropriate for your data.

Proving the hypothesis will require statistical analysis of the data. The experiments must be designed to withstand the statistical tests that you will be using. It is a best practice to decide on which of the statistical tests you will be performing BEFORE you start your experiments. This means you must do the following…

A. Define your thresholdThe alpha value is typically set to 0.05 based on work by RA Fisher. However, this is only a guideline and should be evaluated for each experiment.

B. Define your statistical test—Depending on what is being compared and what assumptions are being made about the data (e.g. Gaussian distribution or not), the appropriate test should be identified before starting.

C. Determine the appropriate sample size—The number of replicates, number of samples is critical to determine before performing the experiments. As this little vignette shows, it is more than just “always using 3 patients.” As the power of an experiment increases, the chance of a Type II (false negative) error decreases.

Error 3: Know your instrument.

The operation and characterization of the instrument is essential for good reproducible data. The individuals who manage the instruments (core directors, shared-resource facility managers) tend to establish and run quality control on the instruments. This is so they can identify and repair issues before they impact the end-user.

However, there is a lot of other information that the end-user needs to be able to design good multi-color experiments, run at optimal voltages, maintain consistency between runs and the like, including…

A. Understand the configuration—It may seem obvious, but the first step is knowing the instrumentation configuration, such as the excitation lines, the power of the lasers, the emission filters and PMTs. For example, if the instrument has a high-power 488 nm laser PerCP is not recommended. Also, when using fluorochromes on multi-lasers instruments it helps to model the excitation and emission profile, to see where there may be issues. Spectral viewers from various vendors are quite useful in this. Here are some useful links to these viewers…

B. Understand the PMT (Part I)—With analog systems, investigators were taught to put the negative populations in the first decade. With newer digital systems, it is a best practice to determine the optimal voltage settings of your instrument. This was described in a paper by Maecker and Trotter. There are other methods out there, but the “peak 2 (RCP-30-5A-2)” method is instrument and vendor agnostic. Using this method helps establish where the PMTs are most sensitive.

C. Understand the PMT (Part II)—When the first 17 color flow cytometry flow cytometry panel was published, the authors discussed how to determine which PMTs were most sensitive and which PMTs received the most error from the fluorochromes used in the panel. Determining PMT sensitivity helps determine which fluorochromes to use and which to avoid. This work has been expanded recently in two articles that carefully evaluate spectral spillover—Nguyen et al., (2013) and Perfetto et al., (2014).

Error 4: Know your antibody panel.

Your antibody panel is the lifeblood of your flow cytometry analysis. Properly designed, optimized and executed, and the answers will be revealed. Poorly designed panels or a lack of optimization will limit the utility of the panel for interpretation. To avoid this, make sure you follow these four tips…

A. Automate your antibody panel—Starting with the hypothesis and knowing the intimate details of the machine you will be using is the start of the process. Next becomes the selection of the antibodies and fluorochromes. This can be a tedious process of searching through multiple catalogues, knowing fluorochrome names (e.g. TriColor = CyChrome + PE-Cy5), and keeping up-to-date with the latest releases on the market. Instead, it’s best to consider using software automation for flow cytometry panel design. These automated programs will help you search the various databases and ensure that you find fluorochromes that (1) are available (2) work on your instrument. The three most common tools for antibody panel automation are (in alphabetical order)…

B. Know how bright your fluorochromes are—Unless you keep updating your staining index panel as new fluorochromes come out, you might want to look to these lists of brightness for different fluorochrome—BD Biosciences and Biolegend.

C. Match fluorochromes to the right antibodies—The goal of panel design is to maximize the signal on the markers where critical measurements are to be made, while minimizing the effects of spectral spillover in those channels. Remember, the more fluorochromes used in a panel, the greater the spread of the data and impact on sensitivity. Design the panel based on the following rules…

  • Highly expressed antigens paired with dimmer fluorochromes
  • Antigens for subset determination paired with mid-bright fluorochromes
  • Unknown and/or low expression antigens paired with brightest fluorochromes
  • Always strive to minimize spectral overlap that will reduce sensitivity in your panel.
  • Don’t forget a dump channel and viability dye – slot those in where you have room, but don’t forget them.

D. Research and use the proper OMIP—The Optimized Multicolor Immunofluorescent Panel is a journal type describing polychromatic panels that have been vetted and optimized. OMIP panels are a great place to start if you’re in a quandary on how to build a panel to answer the hypothesis you’re working on. There are over 25 published OMIPs, and each are required to provide the sample types the OMIP has been validated on, list the antibodies and fluorochromes used, and provide an instrument configuration along with a typical analysis.

Error 5: Know your data.

After all your cells are stained and run on the instrument, it’s time for another important step in the process, gating and data analysis. Extracting the critical information that will be used in statistical analysis to prove (or disprove) the hypothesis follows a path from beginning to end. As you approach the end, keep these four things in mind…

A. Begin with the right controls—Proper data interpretation requires the correct controls to be run as part of the experiment. One of the best articles on flow cytometry controls is this one by Hulspas et al. Without the right controls, the data will be difficult (even impossible) to interpret. The right controls include…

  • Compensation controls
  • Gating controls (aka – FMO controls)
  • Stimulation controls
  • Reference controls
  • Instrument controls

B. Gate generously—Gating is the process of excluding data from the final analysis. Thus, it is critical to be careful with your gating. Don’t rely on forward and side scatter alone to define your populations. Instead, let the antigens in the panel do the required cutting and fine-tuning of the data. Here are four gates you should become familiar with…

  • Viability gate—This gate is used to define the live cells in your populations. This can be combined with a dump gate to give the viability + dump-population. Here, the reagents are doing the heavy lifting helping to get rid of the things that you don’t want to analyze.
  • Singlet gate—This gate is used to define the non-clumping cells based on pulse geometry. Don’t forget to collect the height and width measures for the physical parameters, as well as area, when you set up the instrument.
  • FSCxSSC gate—This gate is used to remove debris, including cell fragments and other “smutz” in lower left-hand portion of the plot, as well the very large (off-scale) debris usually found in the upper right-hand portion of the plot.
  • Phenotypic antigen gate—This gate is used to define your final populations of interest. Use these gates carefully as if you were a sculptor trying to uncover the beauty beneath the rough.

C. Backgate and analyze the ancestry—Backgating is a process that allows you to review the gates and the shape of your final populations of interest. These gates allow you to determine if you’ve missed part of the population. A typical example would be in a simulation experiment, where the cells became larger and are therefore missed because of a too restrictive forward and side scatter gate. Likewise, if the cells are fixed, they can get smaller and again be missed with restrictive gating.

D. Grab the right statistics—With all that work done, make sure you collect the right statistics to answer your hypothesis.

E. Share your flow cytometry data—consider using the MiFlowCyt standard to communicate the essential pieces of your flow experiments to the world. This will add a level of detail to your process that makes peer reviewers happy. You can also share your data with the greater scientific community using the Flow Repository. This repository allows curious researchers to review your data and perform independent analyses on it, increasing their confidence in your conclusions. Data repositories are not a new idea, and have been used for many years in the protein and nucleic acid research communities, but have only recently (2012) come to the field of flow cytometry.

In the peer review process when reviewers evaluate article submissions, they look at the above criteria and that helps guide their review of the material. If there are flaws, cut corners, or missing information—then the material is destined to be rejected. Those papers that answer the questions above, and demonstrate a complete understanding of the process will be reviewed positively. By answering the above questions and following flow cytometry best practices, your experiments will be next in line to be published.

To learn more about publishing your flow cytometry data, and to get access to all of our advanced materials including 20 training videos, presentations, workbooks, and private group membership, get on the Flow Cytometry Mastery Class wait list.

Join Expert Cytometry's Mastery Class
Tim Bushnell, PhD
Tim Bushnell, PhD

Tim Bushnell holds a PhD in Biology from the Rensselaer Polytechnic Institute. He is a co-founder of—and didactic mind behind—ExCyte, the world’s leading flow cytometry training company, which organization boasts a veritable library of in-the-lab resources on sequencing, microscopy, and related topics in the life sciences.

Similar Articles

We Tested 5 Major Flow Cytometry SPADE Programs for Speed - Here Are The Results

We Tested 5 Major Flow Cytometry SPADE Programs for Speed - Here Are The Results

By: Tim Bushnell, PhD

In the flow cytometry community, SPADE (Spanning-tree Progression Analysis of Density-normalized Events) is a favored algorithm for dealing with highly multidimensional or otherwise complex datasets. Like tSNE, SPADE extracts information across events in your data unsupervised and presents the result in a unique visual format. Given the growing popularity of this kind of algorithm for dealing with complex datasets, we decided to test the SPADE algorithm in 5 software packages, including Cytobank, FCS Express, FlowJo, R, and the original, free software made available by the author of SPADE. Which was the fastest?

5 FlowJo Hacks To Boost The Quality Of Your Flow Cytometry Analysis

5 FlowJo Hacks To Boost The Quality Of Your Flow Cytometry Analysis

By: Tim Bushnell, PhD

FlowJo is a powerful tool for performing and analyzing flow cytometry experiments, if you know how to use it to the fullest. This includes understanding embedding and using keywords, the FlowJo compensation wizard, spillover spreading matrix, FlowJo and R, and creating tables in FlowJo. Extending your use of FJ using these hacks will help organize your data, improve analysis and make your exported data easier to understand and explain to others. Take a few moments and explore all you can do with FJ beyond just gating populations.

Statistical Challenges Of Rare Event Measurements In Flow Cytometry

Statistical Challenges Of Rare Event Measurements In Flow Cytometry

By: Tim Bushnell, PhD

It is necessary to sort through hundreds of thousands or millions of cells to find the few events of interest. With such low event numbers, we move away from the comfortable domain of the Gaussian distribution and move into the realm of Poisson statistics. There are 3 points to consider to build confidence in the data that the events being counted are truly events of interest and not random events that just happen to fall into the gates of interest.

How to Optimize Flow Cytometry Hardware For Rare Event Analysis

How to Optimize Flow Cytometry Hardware For Rare Event Analysis

By: Tim Bushnell, PhD

Preparing for rare event analysis requires an understanding of the power and limitation of the instrument to be used. From how fast to run the fluidics, to how the signal is processed to the number of gates that can be used in the sorting experiment, each factor impacts the outcome of the experiment.

How To Choose The Correct Antibody For Accurate Flow Cytometry Results

How To Choose The Correct Antibody For Accurate Flow Cytometry Results

By: Tim Bushnell, PhD

With the added emphasis on reproducibility, it is critical to look at every step where experiments can be improved. No single step makes an experiment more reproducible, rather it is a process, making changes at each stage that leads to reproducibility. Antibodies comprise a critical component that needs to be reviewed. As Bradbury et al. in a commentary in Nature pointed out, the global spending on antibodies is about $1.6 billion a year, and it is estimated about half of that money is spent on “bad” antibodies. This does not include the additional costs of wasted time and effort by…

How To Achieve Accurate Flow Cytometry Calcium Flux Measurements

How To Achieve Accurate Flow Cytometry Calcium Flux Measurements

By: Tim Bushnell, PhD

Dyes exist for the detection of everything from large nucleic acids to reactive oxygen species, and from lipid aggregates to small ions. Concentrations of physiologically important ions such as sodium, potassium, and calcium can be important indicators of health and disease. Calcium ions play an especially critical role in cellular signaling. As a signaling messenger, calcium is involved in everything from muscle contractions, to cell motility, to enzyme activity. Calcium experiments can be very informative, and with the advent of cheaper UV lasers, more and more researchers can use ratiometric measurements to evaluate the signaling processes in phenotypically defined populations.

How to Perform Doublet Discrimination In Flow Cytometry

How to Perform Doublet Discrimination In Flow Cytometry

By: Tim Bushnell, PhD

You are probably familiar with the term, “doublet discrimination” or “doublet exclusion”, and have likely included this flow cytometry measurement into at least some (if not all) of your gating strategies. Even though you may utilize this important gating strategy, you may not have had the chance to delve deeper to explore exactly what doublets are and why it’s critical to exclude them. This article aims to give you insight on the what, why, and how of doublet discrimination.

4 Considerations For Assessing Protein Phosphorylation Using Flow Cytometry

4 Considerations For Assessing Protein Phosphorylation Using Flow Cytometry

By: Tim Bushnell, PhD

For those working in the signaling field, having the ability to take a sample and phenotypically identify it, while knowing what is happening inside the cell to the target molecules of choice opens up a host of new opportunities. These assays are amenable to high throughput setup, meaning that biologically relevant outcomes in pre-clinical drug discovery can be measured directly. All told, with a little forethought, some careful planning and validation, and our helpful tips, phosphoflow assays are within your reach.

5 Essential Calculations For Accurate Flow Cytometry Results

5 Essential Calculations For Accurate Flow Cytometry Results

By: Tim Bushnell, PhD

Flow cytometry is a numbers game. There are percentages of a population, fluorescence intensity measurements, sample averages, data normalization, and more. Many of these common calculations are useful, but surrounded by misconceptions. This primer will help you decide which calculation to use, when to use it, and how to interpret the results.

Top Technical Training eBooks

Get the Advanced Microscopy eBook

Get the Advanced Microscopy eBook

Heather Brown-Harding, PhD

Learn the best practices and advanced techniques across the diverse fields of microscopy, including instrumentation, experimental setup, image analysis, figure preparation, and more.

Get The Free Modern Flow Cytometry eBook

Get The Free Modern Flow Cytometry eBook

Tim Bushnell, PhD

Learn the best practices of flow cytometry experimentation, data analysis, figure preparation, antibody panel design, instrumentation and more.

Get The Free 4-10 Compensation eBook

Get The Free 4-10 Compensation eBook

Tim Bushnell, PhD

Advanced 4-10 Color Compensation, Learn strategies for designing advanced antibody compensation panels and how to use your compensation matrix to analyze your experimental data.