The Right Way To Read A Flow Cytometry Scientific Paper

Scientific reproducibility and the public’s confidence in scientific results is critically important. One has to look no further than the website Retraction Watch to learn of the published papers that are being retracted for a host of reasons. Reasons for retractions can range from plagiarism to scientific misconduct. The cost of these retractions is more than losing a paper, it can lead to financial penalties and even prison. In March, 2019 Duke University settled a lawsuit brought by the US government for over 100 million dollars arising from fraudulent data being used by a researcher in grants and papers from that institution over the years.

These types of stories erode public confidence in scientific data and lead to issues like the discredited link between Autism and vaccination. If you couple this the issues of reproducibility that have been discussed in several articles like this one.

But right now science is facing a reproducibility crisis and the public’s trust in scientific results is faltering. As scientists, it is our responsibility to do everything we can to address these issues.

Recently, a colleague of mine forwarded me a tweet that had been sent by a journal sharing a paper this publication recently published. The tweet implied that this type of research was cutting-edge and so was the journal.

And he asked me, “What do you think of the flow data?” And I was aghast that this paper had been published based upon the quality and the lack of information, and the lack of controls, and the lack of consistency in the data.

It was disappointing to see that a paper like this made it through peer-review. Of course, it doesn’t help when stories like this one highlight another issue in science. It can be easy to set up a website and start a journal in an effort to make easy money. These predatory journals are characterized by charging fees to the scientists seeking to get published. There is a list of these journals here.

When you are reading a paper, look at three areas that should get a critical read in helping to assess the quality of the data and commitment to reproducibility that the authors have.

1. Methods section.

The methods section can reveal a lot about the paper. Here is where the details of how the experiments were performed and the information needed to assess how much confidence one should have in the data. When reading papers, there are a couple of critical areas to check out.

  1. Did the authors describe the instrument? Understanding the instrument characteristics is important to help assess the quality of the fluorescent data. Sub-optimal excitation lines or poorly chosen emission filters can call into question conclusions based upon poor sensitivity the instrument in question would have to the target fluorochrome.
  2. Did the authors mention antigen and clone name? Mentioning CD3 from Company X in the methods section doesn’t provide enough information to reproduce the experiment. Take the common target anti-Human CD3k there are at least 7 different commercially available antigens. Using two different sources – Benchsci.com and the OMIPs, there is no clear clone that is used more than another so if the paper says CD3, which one are the authors referring to?

Figure 1: The distribution of anti-CD3 clones used in publications and the OMIPs

  1. What controls were used? Gating data to identify the populations of interest is a critical step in the data analysis process. Proper gating requires the use of the best controls. These include FMO controls, reference controls, unstimulated controls and more. These should be spelled out and how they were used. In the figure below, shows the comparison of the unstimulated, FMO and isotype control in setting gates on a simulated sample.

Figure 2: Comparison of three different controls to determine the appropriate gate. The data is from Maecker and Trotter (2006). Red and blue lines are added for illustration.

The FMO control is designed to identify the spreading of signal in the channel of interest and is illustrated by the red dashed line above. The isotype control is supposed to address the background staining and is illustrated by the blue dashed line. The isotype control is not recommended.

On the fully stained sample, the FMO control would overestimate the number of positive cells. However, the unstimulated sample (quadrants) addresses both the spectral issues and any NSB issues for the target antigen. This is the type of detail that needs to be in the methods section to improve the reproducibility of the data by other researchers.

Other important details in the methods section include was a viability dye used, what software was used for analysis and more. The methods section should provide sufficient detail that the data can be reproduced.

2. Results section.

Moving to the figures and results section, the information from the methods section should prepare the reader to know what information should be presented in results. One of the most common mistakes made in papers is how the figures are labeled.

Don’t rely on the labels that come with the FCS header. Each machine has its unique labeling system which may not be correct when it comes for publication. Take, for example, an instrument that has a channel labeled PerCp, and the published figure has PerCp on the axis of the bivariate plot – but there is no PerCp mentioned in the methods section. What does this mean? The axes should be labeled with the antigen name and ideally the excitation line and emission filters to make sure the reader knows the correct details.

A second point to look at on the axes of bivariate plots. Looking at the axes can indicate if the data has been transformed in some manner. This is a big issue in papers, that they do not reveal the transformation or indicating that all data had the same transformation applied to the data.

Another area to review in the results section is the gating strategy. The use of pulse geometry is a powerful way to help remove doublets. However, if the ASF is set wrong, cells of interest can be lost. Especially if you are using larger cells, the ASF needs to be adjusted. The consequences of poorly set ASF are discussed in this paper.

Statistical analysis is another area to examine closely. The hypothesis should clearly be stated, along with the threshold and the type of test that was used. The number of replicates should also be listed. This information will help the reader assess the strength of any results.

3. MIFlowCyt standard and the Flow Repository.

As cytometrists, we have a tool that can be used to help improve the communication of experimental information. This is the Minimum Information for a Flow Cytometry Experiment or the MIFlowCyt standard. The development of this standard was under the auspices of the ISAC Data Standards Taskforce and represents a trend in biomedical sciences for these comprehensive checklists to ensure the authors include critical information.

Using the MIFlowCyt standard requires the researcher to include descriptions about the sample, about the instrument, and the experimental overview in the data analysis.

At the present time, the MIFlowCyt is only used by a handful of journals from publishers Wiley-Blackwell and Nature Publishing group. For articles submitted to Cytometry A, if the paper is submitted and is MIFloCyt compliant, it gains a special distinction, and a badge to indicate the paper is compliant.

In the case of publishing in another journal, investigators can encourage other journals to adopt this standard.

Another way for researchers to improve the quality of publications and confidence in data is to share the information in a way that others can access it. The flow repository that was started as a partnership between ISAC with the support of the Wallace H. Coulter Foundation. Investigators can upload their data to this database. The data can remain embargoed until publication, but a special access link can be provided to the reviewers so they can have access to the full data as they make a decision on the paper.

Once a paper is published, the data can be released and anybody can access it. This allows researchers to review the data a paper is based on. Further, it can be used to train new researchers in analytical techniques using published data.

Looking to see if a paper is MIFlowCyt compliant, and the data is archived in the Flow Repository is an excellent way to gain confidence in the conclusions found in the publication, or identify how the analysis was skewed in one way or the other.

When you are reviewing papers, Glen Begley’s rules, which were discussed by Bruce Booth are an excellent source of the way to review a paper. These 6 rules are a powerful tool to aid the researcher and help demonstrate the reproducibility of the data.

In the end, no one wants to be the headline in Retraction Watch. No one wants to see their institution to be fined 100 million dollars because of data fraud. Read every paper with a critical eye, and doubly so for your own manuscripts. Even if not submitting a paper to a journal that doesn’t use the MiFlowCyt standard, it is a good practice to annotate each paper in this manner. In so doing, the authors will go through and make sure none of the critical information is missing. With estimates of the wasted money due to irreproducible results in the 70-90% range, it is essential that researchers do better.

To learn more about The Right Way To Read A Flow Cytometry Scientific Paper, and to get access to all of our advanced materials including 20 training videos, presentations, workbooks, and private group membership, get on the Flow Cytometry Mastery Class wait list.

Join Expert Cytometry's Mastery Class
Tim Bushnell, PhD
Tim Bushnell, PhD

Tim Bushnell holds a PhD in Biology from the Rensselaer Polytechnic Institute. He is a co-founder of—and didactic mind behind—ExCyte, the world’s leading flow cytometry training company, which organization boasts a veritable library of in-the-lab resources on sequencing, microscopy, and related topics in the life sciences.

Similar Articles

The 5 Fundamental Methods For Imaging Nucleic Acids

The 5 Fundamental Methods For Imaging Nucleic Acids

By: Heather Brown-Harding, PhD

There are 4 major ways to sort cells. The first way can use magnetic beads coupled to antibodies and pass the cells through a magnetic field. The labeled cells will stick, and the unlabeled cells will remain in the supernatant. The second way is to use some sort of mechanical force like a flapper or air stream that separates the target cells from the bulk population. The third way is the recently introduced microfluidics sorter, which uses microfluidics channels to isolate the target cells. The last method, which is the most common––based on Fuwyler’s work––is the electrostatic cell sorter. This…

Avoid Flow Cytometry Faux Pas: How To Set Voltage The Right Way

Avoid Flow Cytometry Faux Pas: How To Set Voltage The Right Way

By: Tim Bushnell, PhD

There are 4 major ways to sort cells. The first way can use magnetic beads coupled to antibodies and pass the cells through a magnetic field. The labeled cells will stick, and the unlabeled cells will remain in the supernatant. The second way is to use some sort of mechanical force like a flapper or air stream that separates the target cells from the bulk population. The third way is the recently introduced microfluidics sorter, which uses microfluidics channels to isolate the target cells. The last method, which is the most common––based on Fuwyler’s work––is the electrostatic cell sorter. This…

Designing Microscopy Experiments Related To Infectious Diseases And Antivirals

Designing Microscopy Experiments Related To Infectious Diseases And Antivirals

By: Heather Brown-Harding, PhD

There are 4 major ways to sort cells. The first way can use magnetic beads coupled to antibodies and pass the cells through a magnetic field. The labeled cells will stick, and the unlabeled cells will remain in the supernatant. The second way is to use some sort of mechanical force like a flapper or air stream that separates the target cells from the bulk population. The third way is the recently introduced microfluidics sorter, which uses microfluidics channels to isolate the target cells. The last method, which is the most common––based on Fuwyler’s work––is the electrostatic cell sorter. This…

My 3-Step Panel Validation Pocket Guide

My 3-Step Panel Validation Pocket Guide

By: Tim Bushnell, PhD

There are 4 major ways to sort cells. The first way can use magnetic beads coupled to antibodies and pass the cells through a magnetic field. The labeled cells will stick, and the unlabeled cells will remain in the supernatant. The second way is to use some sort of mechanical force like a flapper or air stream that separates the target cells from the bulk population. The third way is the recently introduced microfluidics sorter, which uses microfluidics channels to isolate the target cells. The last method, which is the most common––based on Fuwyler’s work––is the electrostatic cell sorter. This…

Easy-To-Forget Flow Fundamentals That Thwart Bad Science

Easy-To-Forget Flow Fundamentals That Thwart Bad Science

By: Tim Bushnell, PhD

There are 4 major ways to sort cells. The first way can use magnetic beads coupled to antibodies and pass the cells through a magnetic field. The labeled cells will stick, and the unlabeled cells will remain in the supernatant. The second way is to use some sort of mechanical force like a flapper or air stream that separates the target cells from the bulk population. The third way is the recently introduced microfluidics sorter, which uses microfluidics channels to isolate the target cells. The last method, which is the most common––based on Fuwyler’s work––is the electrostatic cell sorter. This…

Important Controls For Your Flow Cytometry Lab

Important Controls For Your Flow Cytometry Lab

By: Tim Bushnell, PhD

There are 4 major ways to sort cells. The first way can use magnetic beads coupled to antibodies and pass the cells through a magnetic field. The labeled cells will stick, and the unlabeled cells will remain in the supernatant. The second way is to use some sort of mechanical force like a flapper or air stream that separates the target cells from the bulk population. The third way is the recently introduced microfluidics sorter, which uses microfluidics channels to isolate the target cells. The last method, which is the most common––based on Fuwyler’s work––is the electrostatic cell sorter. This…

4 Factors To Improve Flow Cytometry Cell Sorting Speed

4 Factors To Improve Flow Cytometry Cell Sorting Speed

By: Tim Bushnell, PhD

There are 4 major ways to sort cells. The first way can use magnetic beads coupled to antibodies and pass the cells through a magnetic field. The labeled cells will stick, and the unlabeled cells will remain in the supernatant. The second way is to use some sort of mechanical force like a flapper or air stream that separates the target cells from the bulk population. The third way is the recently introduced microfluidics sorter, which uses microfluidics channels to isolate the target cells. The last method, which is the most common––based on Fuwyler’s work––is the electrostatic cell sorter. This…

5 Techniques For Dramatic Improvements In Reproducibility

5 Techniques For Dramatic Improvements In Reproducibility

By: Heather Brown-Harding, PhD

It’s not easy to improve reproducibility in your experiments. Image manipulation has become a major problem in science, whether intentional or accidental. This has exploded with the advent of digital imaging and software like Photoshop. There are even mobile applications like Instagram filters that can be used for imaging trickery. It should go without saying that image reuse/manipulation represents profound dishonesty in science – a field intended to uphold the most stringent possible standards of truthful inquiry! But what about studies with a sloppy or stunted capacity for reproduction? These, too, plague science and hinder our ability to seamlessly move…

3 Compensation Mistakes That Will Ruin Your Flow Cytometry Experiments

3 Compensation Mistakes That Will Ruin Your Flow Cytometry Experiments

By: Tim Bushnell, PhD

Compensation is necessary due to the physics of fluorescence. Basically, compensation is the mathematical process of correcting spectral spillover from a fluorochrome into a secondary detector so that it is possible to identify single positive events in the context of a multidimensional panel. Good compensation requires that your controls tightly adhere to three rules. If the controls don’t meet this criteria, it will lead to faulty compensation resulting in false conclusions and poorly reproducible data. Even among flow cytometry veterans, a strong foundation is occasionally in need of a tune-up. And in a topic as complex as flow cytometry, it’s…

Top Technical Training eBooks

Get the Advanced Microscopy eBook

Get the Advanced Microscopy eBook

Heather Brown-Harding, PhD

Learn the best practices and advanced techniques across the diverse fields of microscopy, including instrumentation, experimental setup, image analysis, figure preparation, and more.

Get The Free Modern Flow Cytometry eBook

Get The Free Modern Flow Cytometry eBook

Tim Bushnell, PhD

Learn the best practices of flow cytometry experimentation, data analysis, figure preparation, antibody panel design, instrumentation and more.

Get The Free 4-10 Compensation eBook

Get The Free 4-10 Compensation eBook

Tim Bushnell, PhD

Advanced 4-10 Color Compensation, Learn strategies for designing advanced antibody compensation panels and how to use your compensation matrix to analyze your experimental data.