5 Experimental Errors That Prevent Your Flow Cytometry Data From Passing The Peer Review Process

Reproducibility is a critical component of the scientific process. One cannot publish data if the experiments cannot be replicated are are full of errors.

Unfortunately, as Begley and Ellis pointed out in a commentary in Nature that when Amgen attempted to reproduce 53 “landmark” papers in the area of cancer research, only 6 papers were “confirmed.”

What does that mean to you, the flow cytometry researcher?

To avoid publishing errors that reviews despise, it’s important to follow and promote the best practices in the field, thus ensuring that your data is reproducible to investigators attempting to validate your research.

In particular, you must follow these 5 experimental tips before embarking into the peer review process…

Error 1: Know what your hypothesis is and design the experiments to test your hypothesis.

It may seem obvious, but the goal of the experiments is to prove some hypothesis that you are testing. In examining the hypothesis, there are several questions that should be on your mind…

A. Is flow cytometry the right tool to answer the question?—Sometimes flow isn’t the best way to answer the question so it’s better to make the decision before starting experiments.

B. Is flow cytometry a path to another technique?—For example, will sorted cells be used to perform experiments such as RNAseq, Proteomic analysis, cell culture etc? If so, what controls will you need to build into the experiment to help evaluate the handoff between technologies?

C. Is there another technology that will confirm the findings?—It is always useful to have data from two different technologies that can confirm each other. For instance, flow cytometry brings statistical power to examining cell populations from other experiments.

Error 2: Know which statistical tests are appropriate for your data.

Proving the hypothesis will require statistical analysis of the data. The experiments must be designed to withstand the statistical tests that you will be using. It is a best practice to decide on which of the statistical tests you will be performing BEFORE you start your experiments. This means you must do the following…

A. Define your thresholdThe alpha value is typically set to 0.05 based on work by RA Fisher. However, this is only a guideline and should be evaluated for each experiment.

B. Define your statistical test—Depending on what is being compared and what assumptions are being made about the data (e.g. Gaussian distribution or not), the appropriate test should be identified before starting.

C. Determine the appropriate sample size—The number of replicates, number of samples is critical to determine before performing the experiments. As this little vignette shows, it is more than just “always using 3 patients.” As the power of an experiment increases, the chance of a Type II (false negative) error decreases.

Error 3: Know your instrument.

The operation and characterization of the instrument is essential for good reproducible data. The individuals who manage the instruments (core directors, shared-resource facility managers) tend to establish and run quality control on the instruments. This is so they can identify and repair issues before they impact the end-user.

However, there is a lot of other information that the end-user needs to be able to design good multi-color experiments, run at optimal voltages, maintain consistency between runs and the like, including…

A. Understand the configuration—It may seem obvious, but the first step is knowing the instrumentation configuration, such as the excitation lines, the power of the lasers, the emission filters and PMTs. For example, if the instrument has a high-power 488 nm laser PerCP is not recommended. Also, when using fluorochromes on multi-lasers instruments it helps to model the excitation and emission profile, to see where there may be issues. Spectral viewers from various vendors are quite useful in this. Here are some useful links to these viewers…

B. Understand the PMT (Part I)—With analog systems, investigators were taught to put the negative populations in the first decade. With newer digital systems, it is a best practice to determine the optimal voltage settings of your instrument. This was described in a paper by Maecker and Trotter. There are other methods out there, but the “peak 2 (RCP-30-5A-2)” method is instrument and vendor agnostic. Using this method helps establish where the PMTs are most sensitive.

C. Understand the PMT (Part II)—When the first 17 color flow cytometry flow cytometry panel was published, the authors discussed how to determine which PMTs were most sensitive and which PMTs received the most error from the fluorochromes used in the panel. Determining PMT sensitivity helps determine which fluorochromes to use and which to avoid. This work has been expanded recently in two articles that carefully evaluate spectral spillover—Nguyen et al., (2013) and Perfetto et al., (2014).

Error 4: Know your antibody panel.

Your antibody panel is the lifeblood of your flow cytometry analysis. Properly designed, optimized and executed, and the answers will be revealed. Poorly designed panels or a lack of optimization will limit the utility of the panel for interpretation. To avoid this, make sure you follow these four tips…

A. Automate your antibody panel—Starting with the hypothesis and knowing the intimate details of the machine you will be using is the start of the process. Next becomes the selection of the antibodies and fluorochromes. This can be a tedious process of searching through multiple catalogues, knowing fluorochrome names (e.g. TriColor = CyChrome + PE-Cy5), and keeping up-to-date with the latest releases on the market. Instead, it’s best to consider using software automation for flow cytometry panel design. These automated programs will help you search the various databases and ensure that you find fluorochromes that (1) are available (2) work on your instrument. The three most common tools for antibody panel automation are (in alphabetical order)…

B. Know how bright your fluorochromes are—Unless you keep updating your staining index panel as new fluorochromes come out, you might want to look to these lists of brightness for different fluorochrome—BD Biosciences and Biolegend.

C. Match fluorochromes to the right antibodies—The goal of panel design is to maximize the signal on the markers where critical measurements are to be made, while minimizing the effects of spectral spillover in those channels. Remember, the more fluorochromes used in a panel, the greater the spread of the data and impact on sensitivity. Design the panel based on the following rules…

  • Highly expressed antigens paired with dimmer fluorochromes
  • Antigens for subset determination paired with mid-bright fluorochromes
  • Unknown and/or low expression antigens paired with brightest fluorochromes
  • Always strive to minimize spectral overlap that will reduce sensitivity in your panel.
  • Don’t forget a dump channel and viability dye – slot those in where you have room, but don’t forget them.

D. Research and use the proper OMIP—The Optimized Multicolor Immunofluorescent Panel is a journal type describing polychromatic panels that have been vetted and optimized. OMIP panels are a great place to start if you’re in a quandary on how to build a panel to answer the hypothesis you’re working on. There are over 25 published OMIPs, and each are required to provide the sample types the OMIP has been validated on, list the antibodies and fluorochromes used, and provide an instrument configuration along with a typical analysis.

Error 5: Know your data.

After all your cells are stained and run on the instrument, it’s time for another important step in the process, gating and data analysis. Extracting the critical information that will be used in statistical analysis to prove (or disprove) the hypothesis follows a path from beginning to end. As you approach the end, keep these four things in mind…

A. Begin with the right controls—Proper data interpretation requires the correct controls to be run as part of the experiment. One of the best articles on flow cytometry controls is this one by Hulspas et al. Without the right controls, the data will be difficult (even impossible) to interpret. The right controls include…

  • Compensation controls
  • Gating controls (aka – FMO controls)
  • Stimulation controls
  • Reference controls
  • Instrument controls

B. Gate generously—Gating is the process of excluding data from the final analysis. Thus, it is critical to be careful with your gating. Don’t rely on forward and side scatter alone to define your populations. Instead, let the antigens in the panel do the required cutting and fine-tuning of the data. Here are four gates you should become familiar with…

  • Viability gate—This gate is used to define the live cells in your populations. This can be combined with a dump gate to give the viability + dump-population. Here, the reagents are doing the heavy lifting helping to get rid of the things that you don’t want to analyze.
  • Singlet gate—This gate is used to define the non-clumping cells based on pulse geometry. Don’t forget to collect the height and width measures for the physical parameters, as well as area, when you set up the instrument.
  • FSCxSSC gate—This gate is used to remove debris, including cell fragments and other “smutz” in lower left-hand portion of the plot, as well the very large (off-scale) debris usually found in the upper right-hand portion of the plot.
  • Phenotypic antigen gate—This gate is used to define your final populations of interest. Use these gates carefully as if you were a sculptor trying to uncover the beauty beneath the rough.

C. Backgate and analyze the ancestry—Backgating is a process that allows you to review the gates and the shape of your final populations of interest. These gates allow you to determine if you’ve missed part of the population. A typical example would be in a simulation experiment, where the cells became larger and are therefore missed because of a too restrictive forward and side scatter gate. Likewise, if the cells are fixed, they can get smaller and again be missed with restrictive gating.

D. Grab the right statistics—With all that work done, make sure you collect the right statistics to answer your hypothesis.

E. Share your flow cytometry data—consider using the MiFlowCyt standard to communicate the essential pieces of your flow experiments to the world. This will add a level of detail to your process that makes peer reviewers happy. You can also share your data with the greater scientific community using the Flow Repository. This repository allows curious researchers to review your data and perform independent analyses on it, increasing their confidence in your conclusions. Data repositories are not a new idea, and have been used for many years in the protein and nucleic acid research communities, but have only recently (2012) come to the field of flow cytometry.

In the peer review process when reviewers evaluate article submissions, they look at the above criteria and that helps guide their review of the material. If there are flaws, cut corners, or missing information—then the material is destined to be rejected. Those papers that answer the questions above, and demonstrate a complete understanding of the process will be reviewed positively. By answering the above questions and following flow cytometry best practices, your experiments will be next in line to be published.

To learn more about publishing your flow cytometry data, and to get access to all of our advanced materials including 20 training videos, presentations, workbooks, and private group membership, get on the Flow Cytometry Mastery Class wait list.

Join Expert Cytometry's Mastery Class

ABOUT TIM BUSHNELL, PHD

Tim Bushnell holds a PhD in Biology from the Rensselaer Polytechnic Institute. He is a co-founder of—and didactic mind behind—ExCyte, the world’s leading flow cytometry training company, which organization boasts a veritable library of in-the-lab resources on sequencing, microscopy, and related topics in the life sciences.

Tim Bushnell, PhD

Similar Articles

Common Numbers-Based Questions I Get As A Flow Cytometry Core Manager And How To Answer Them

Common Numbers-Based Questions I Get As A Flow Cytometry Core Manager And How To Answer Them

By: Tim Bushnell, PhD

Numbers are all around us.  My personal favorite is ≅1.618 aka ɸ aka ‘the golden ratio’.  It’s found throughout history, where it has influenced architects and artists. We see it in nature, in plants, and it is used in movies to frame shots. It can be approximated by the Fibonacci sequence (another math favorite of mine). However, I have not worked out how to apply this to flow cytometry.  That doesn’t mean numbers aren’t important in flow cytometry. They are central to everything we do, and in this blog, I’m going to flit around numbers-based questions that I have received…

How To Do Variant Calling From RNASeq NGS Data

How To Do Variant Calling From RNASeq NGS Data

By: Deepak Kumar, PhD

Developing variant calling and analysis pipelines for NGS sequenced data have become a norm in clinical labs. These pipelines include a strategic integration of several tools and techniques to identify molecular and structural variants. That eventually helps in the apt variant annotation and interpretation. This blog will delve into the concepts and intricacies of developing a “variant calling” pipeline using GATK. “Variant calling” can also be performed using tools other than GATK, such as FREEBAYES and SAMTOOLS.  In this blog, I will walk you through variant calling methods on Illumina germline RNASeq data. In the steps, wherever required, I will…

Understanding Clinical Trials And Drug Development As A Research Scientist

Understanding Clinical Trials And Drug Development As A Research Scientist

By: Deepak Kumar, PhD

Clinical trials are studies designed to test the novel methods of diagnosing and treating health conditions – by observing the outcomes of human subjects under experimental conditions.  These are interventional studies that are performed under stringent clinical laboratory settings. Contrariwise, non-interventional studies are performed outside the clinical trial settings that provide researchers an opportunity to monitor the effect of drugs in real-life situations. Non-interventional trials are also termed observational studies as they include post-marketing surveillance studies (PMS) and post-authorization safety studies (PASS). Clinical trials are preferred for testing newly developed drugs since interventional studies are conducted in a highly monitored…

How To Profile DNA And RNA Expression Using Next Generation Sequencing (Part-2)

How To Profile DNA And RNA Expression Using Next Generation Sequencing (Part-2)

By: Deepak Kumar, PhD

In the first blog of this series, we explored the power of sequencing the genome at various levels. We also dealt with how the characterization of the RNA expression levels helps us to understand the changes at the genome level. These changes impact the downstream expression of the target genes. In this blog, we will explore how NGS sequencing can help us comprehend DNA modification that affect the expression pattern of the given genes (epigenetic profiling) as well as characterizing the DNA-protein interactions that allow for the identification of genes that may be regulated by a given protein.  DNA Methylation Profiling…

How To Profile DNA And RNA Expression Using Next Generation Sequencing

How To Profile DNA And RNA Expression Using Next Generation Sequencing

By: Deepak Kumar, PhD

Why is Next Generation Sequencing so powerful to explore and answer both clinical and research questions. With the ability to sequence whole genomes, identifying novel changes between individuals, to exploring what RNA sequences are being expressed, or to examine DNA modifications and protein-DNA interactions occurring that can help researchers better understand the complex regulation of transcription. This, in turn, allows them to characterize changes during different disease states, which can suggest a way to treat said disease.  Over the next two blogs, I will highlight these different methods along with illustrating how these can help clinical diagnostics as well as…

What Is Next Generation Sequencing (NGS) And How Is It Used In Drug Development

What Is Next Generation Sequencing (NGS) And How Is It Used In Drug Development

By: Deepak Kumar, PhD

NGS methodologies have been used to produce high-throughput sequence data. These data with appropriate computational analyses facilitate variant identification and prove to be extremely valuable in pharmaceutical industries and clinical practice for developing drug molecules inhibiting disease progression. Thus, by providing a comprehensive profile of an individual’s variome — particularly that of clinical relevance consisting of pathogenic variants — NGS helps in determining new disease genes. The information thus obtained on genetic variations and the target disease genes can be used by the Pharma companies to develop drugs impeding these variants and their disease-causing effect. However simple this may allude…

7 Key Image Analysis Terms For New Microscopist

7 Key Image Analysis Terms For New Microscopist

By: Heather Brown-Harding, PhD

As scientists, we need to perform image analysis after we’ve acquired images in the microscope, otherwise, we have just a pretty picture and not data. The vocabulary for image processing and analysis can be a little intimidating to those new to the field. Therefore, in this blog, I’m going to break down 7 terms that are key when post-processing of images. 1. RGB Image Images acquired during microscopy can be grouped into two main categories. Either monochrome (that can be multichannel) or “RGB.” RGB stands for red, green, blue – the primary colors of light. The cameras in our phones…

We Tested 5 Major Flow Cytometry SPADE Programs for Speed - Here Are The Results

We Tested 5 Major Flow Cytometry SPADE Programs for Speed - Here Are The Results

By: Tim Bushnell, PhD

In the flow cytometry community, SPADE (Spanning-tree Progression Analysis of Density-normalized Events) is a favored algorithm for dealing with highly multidimensional or otherwise complex datasets. Like tSNE, SPADE extracts information across events in your data unsupervised and presents the result in a unique visual format. Given the growing popularity of this kind of algorithm for dealing with complex datasets, we decided to test the SPADE algorithm in 5 software packages, including Cytobank, FCS Express, FlowJo, R, and the original, free software made available by the author of SPADE. Which was the fastest?

5 FlowJo Hacks To Boost The Quality Of Your Flow Cytometry Analysis

5 FlowJo Hacks To Boost The Quality Of Your Flow Cytometry Analysis

By: Tim Bushnell, PhD

FlowJo is a powerful tool for performing and analyzing flow cytometry experiments, if you know how to use it to the fullest. This includes understanding embedding and using keywords, the FlowJo compensation wizard, spillover spreading matrix, FlowJo and R, and creating tables in FlowJo. Extending your use of FJ using these hacks will help organize your data, improve analysis and make your exported data easier to understand and explain to others. Take a few moments and explore all you can do with FJ beyond just gating populations.

Top Industry Career eBooks

Get the Advanced Microscopy eBook

Get the Advanced Microscopy eBook

Heather Brown-Harding, PhD

Learn the best practices and advanced techniques across the diverse fields of microscopy, including instrumentation, experimental setup, image analysis, figure preparation, and more.

Get The Free Modern Flow Cytometry eBook

Get The Free Modern Flow Cytometry eBook

Tim Bushnell, PhD

Learn the best practices of flow cytometry experimentation, data analysis, figure preparation, antibody panel design, instrumentation and more.

Get The Free 4-10 Compensation eBook

Get The Free 4-10 Compensation eBook

Tim Bushnell, PhD

Advanced 4-10 Color Compensation, Learn strategies for designing advanced antibody compensation panels and how to use your compensation matrix to analyze your experimental data.