Using Begley’s Rules To Improve Reproducibility In Flow Cytometry

Isaac Newton was famous for saying “If I have seen further than others, it is by standing upon the shoulders of giants.” Implicit in that statement is that the information that the giants provided was reproducible. In fact, reproducibility is central to the scientific method and as far back as the 10th century, the concept of reproducibility of data was being discussed by Ibn al-Haytham.

In 2011, Prinz et al. published an article that indicated a case study looking at reproducibility by Bayer Healthcare found only 25% of academic studies were reproducible. This was followed up in 2012 by a report from Begley and Ellis that indicated on 11% of 53 landmark oncology studies were able to be replicated. So it seems that while we are trying to see farther, our lens may be out of focus.

Bruce Booth, writing for Forbes, published an article called “Scientific Reproducibility: Begley’s Six Rules” and in this article, he proposed the following 6 rules that should serve as a roadmap in evaluating scientific work, both published and your own work. These rules are:

  1. Were the studies blinded?
  2. Were all the results shown?
  3. Were the experiments repeated?
  4. Were the positive and negative controls shown?
  5. Were the reagents validated?
  6. Were the statistical tests appropriate?

While these rules are focused more on clinical trials, they are readily adopted for basic scientific inquiry. By starting to think about these questions in the early stages of discovery and into pre-clinical studies, which should increase the confidence and reproducibility of later stages of the process.

Using Begley’s Rules

Reproducibility is a mindset, it’s not one simple tweak and the data is reproducible. It is a matter of critically evaluating each process in the experiment and identifying areas that can be improved. It involves complete communication of the process. It involves relying on well-developed and documented standard operating procedures that everyone involved in the project are trained on.

Turning our attention back to Begley’s rules, how can these rules help you improve your research? They help provide a roadmap on how to design, validate, execute and report experimental data in a way that is more robust and reproducible.

1. Take the first rule, “Were the studies blinded?

This is a critical component of clinical trials. In blinded studies, the subject does not know if they are part of the control group or the experimental group. In a double-blinded studied the experimenter also does not know what group the subjects are part of.

This helps prevent experimenter bias impacting the data. In the research setting, this technique is not often used, but with a little coordination within the laboratory, this could be implemented in the research setting.

2. Thinking about the second rule: “Were all the results shown?”

Flow cytometry is a data-rich technology and numbers are the name of the game. Experiments looking at the change in percentage a population or the change in the expression pattern of a given protein.

For this reason, the results of any experiments can often be summarized and presented as a table or graph that provides statistical information about the experiment, which is used to support (or refute) the thesis of the argument. A histogram or bivariant gating strategy is useful, but the meat of the argument will be in these summary figures, such as the one shown below.

Figure 1: Summary figure showing all the results of an experiment measuring the the changes in CD4+ cells after drug treatment. All the data is shown with the mean and standard deviation indicated. The number of data points and the p-value between the two datasets is indicated.

In addition to showing the data, thanks to the support of the Wallace H. Coulter Foundation and ISAC, there is a public database where flow cytometry data can be deposited. The Flow Repository allows researchers to upload their data for their published experiments. This allows for all researchers to review the data that the paper is based on, thus improving the ability of researchers to repeat and extend findings of interest.

3. In lines with showing all the data is the thir rule: “Were the experiments repeated?”

For any experiment, it is critical that there the experiments are replicated. This becomes the ‘n’ in any graph and helps evaluate how robust the experiment has been tested. Based on discovery-based work, estimates of the magnitude of the different and the expected variance in the data can be estimated. This, in turn, allows for a Power calculation, which can help guide the researcher in determining the ‘n’.

The smaller the difference that the researchers wish to test, the more samples that they will need to run. The program Statmate is one useful tool for performing these calculations. Figure 2 shows how to determine the number of samples to run based on the Statmate output.

Figure 2: Statmate output used to determine the number of replicates needed.

4. This leads to the fourth rule: “Were the positive and negative controls shown?”

In flow cytometry the controls that are used to determine the population of interest are very important to show. Since gating is a data reduction technique, incorrect gating can impact the data and conclusions.

Without showing and explaining the use of the controls, gating is more a subjective art than an objective evaluation. For those starting out in flow cytometry, using the data available in the Flow Repository along with the paper it came from is a good way to practice. The OMIPs are especially useful for this purpose.

5. Next is to examine the fifth rule :“Were the reagents validated?”

When thinking about flow cytometry, reagent validation is a critical step in the validation and optimization of any polychromatic panel. This is especially true of the antibodies used in experiments.

In Bradbury and Plückthun’s commentary in Nature, the authors estimate about 50% of the money spent on antibodies is wasted due to the quality of the antibodies. Issues with antibodies can include cross-reactivity, lot-to-lot variability and even the wrong antibody for the application.

With the advent of recombinant antibodies, this should begin to become less and less of an issue, but it will take time to for these reagents to penetrate the market. At a minimum, every antibody that comes into the lab should be tested and titrated to ensure the reagent is working properly.

Beyond the antibodies, any other reagent that is being used should be tested and validated. This includes the flow cytometer. While not a reagent, per se, it is essential to gathering the data and the results of the quality control being performed on the system should be accessible to the investigator. In fact, the investigator can build into their procedures their own QC steps that show the instrument and assay are working.

6. The last of Begley’s rules is “Were the statistical tests appropriate?”

All researchers want their data to be shown to be statistically significant because there is an inherent bias in published articles. It was shown by Dickersin et al., in their 1987 paper that papers containing data shown to be statistically significant were 3 times more likely to be published. Issues with HARKing and p-Hacking are troubling but can be reduced or avoided with a simple change to the mindset in experimental design.

Before any experiments are performed, it is critical to consider the outcome and how one would validate the hypothesis being tested. Doing this at the beginning of the process, rather than towards the end, allows the researcher to define the statistical testing that will be used and the threshold for significance.

Any deviations from this plan need to be reported so that readers can understand and evaluate the statistical analysis. Second, by defining the power of the experiment, this reduces the potential to stop collecting data early when the results support your hypothesis. While outliers can be very interesting in their own right, define a rule for excluding them from the analysis and report it.

In summary, Begley’s rules are a useful tool to use to evaluate the quality and reproducibility of data. It helps you look for important issues in a report and your own experiments. Couple these with the best practices in flow cytometry, and you are well on your way to improving the rigor and reproducibility of your work.

To learn more about Using Begley’s Rules To Improve Reproducibility In Flow Cytometry, and to get access to all of our advanced materials including 20 training videos, presentations, workbooks, and private group membership, get on the Flow Cytometry Mastery Class wait list.

Join Expert Cytometry's Mastery Class
Tim Bushnell, PhD
Tim Bushnell, PhD

Tim Bushnell holds a PhD in Biology from the Rensselaer Polytechnic Institute. He is a co-founder of—and didactic mind behind—ExCyte, the world’s leading flow cytometry training company, which organization boasts a veritable library of in-the-lab resources on sequencing, microscopy, and related topics in the life sciences.

Similar Articles

The Power Of Spectral Viewers And Their Use In Full Spectrum Flow Cytometry

The Power Of Spectral Viewers And Their Use In Full Spectrum Flow Cytometry

By: Tim Bushnell, PhD

What photon from yonder fluorochrome breaks?  It is … umm… hmmm. Let me see. Excitation off a 561 nm laser, with an emission maximum of 692 nm. I’m sure if Shakespeare was a flow cytometrist, he might have written that very scene. But the play is lost in time. However, since the protagonist had difficulty determining what fluorochrome was emitting photons, let’s consider how this could be figured out. In my opinion, one of the handiest flow cytometry tools is the spectral viewer. This tool helps visualize the excitation and emission profile of different fluorochromes, as well as allowing you…

Fickle Markers: Solutions For Antibody Binding Specificity Challenges

Fickle Markers: Solutions For Antibody Binding Specificity Challenges

By: Tim Bushnell, PhD

Reproducibility has been an ongoing, and important, concept in the sciences for years.  In the area of biomedical research, the alarm was sounded by several papers published in the early 2010’s.  Authors like Begley and Ellis, Prinz and coworkers, and Vasilevsky and colleagues, among others reported an alarming trend in the reproducibility of pre-clinical data.  These reports indicated between 50% to almost 90% of published pre-clinical data were not reproducible.  This was further highlighted in the article by Freedman and coworkers, who tried to identify and quantify the different sources of error that could be causing this crisis.  Figure 1,…

5 Common Flow Cytometry Questions, Answered

5 Common Flow Cytometry Questions, Answered

By: Tim Bushnell, PhD

I want to thank all of you who send us your questions about flow cytometry, so I thought I would dip into the old email bag and answer a few of the common ones here.  If your question isn’t answered this time, look for it to be answered in a future blog post.  Of course, if you want us to cover a specific topic, drop us a line.  1. How Fast Can I Go? This is  a common question. The allure of the ‘hi’ button is hard to resist.  The faster you go, the sooner you are finished with data…

Combining Flow Cytometry With Plant Science, Microorganisms, And The Environment

Combining Flow Cytometry With Plant Science, Microorganisms, And The Environment

By: Tim Bushnell, PhD

My first introduction to flow cytometry was talking to a professor who’d brought one on a research cruise to study phytoplankton. It was only later that I was introduced to the marvelous world that’s been my career for over 20 years.   In that time, I’ve had the opportunity to work with researchers in many different areas, exposing me to a wide variety of cell types and more important assays. What continues to amaze me is the number of different parameters we can measure, not just the number of fluorochromes, but the information we can extract from samples – animal, vegetable…

Common Numbers-Based Questions I Get As A Flow Cytometry Core Manager And How To Answer Them

Common Numbers-Based Questions I Get As A Flow Cytometry Core Manager And How To Answer Them

By: Tim Bushnell, PhD

Numbers are all around us.  My personal favorite is ≅1.618 aka ɸ aka ‘the golden ratio’.  It’s found throughout history, where it has influenced architects and artists. We see it in nature, in plants, and it is used in movies to frame shots. It can be approximated by the Fibonacci sequence (another math favorite of mine). However, I have not worked out how to apply this to flow cytometry.  That doesn’t mean numbers aren’t important in flow cytometry. They are central to everything we do, and in this blog, I’m going to flit around numbers-based questions that I have received…

3 Must-Have High-Dimensional Flow Cytometry Controls

3 Must-Have High-Dimensional Flow Cytometry Controls

By: Tim Bushnell, PhD

Developments such as the recent upgrade to the Cytobank analysis platform and the creation of new packages such as Immunocluster are reducing the computational expertise needed to work with high-dimensional flow cytometry datasets. Whether you are a researcher in academia, industry, or government, you may want to take advantage of the reduced barrier to entry to apply high-dimensional flow cytometry in your work. However, you’ll need the right experimental design to access the new transformative insights available through these approaches and avoid wasting the considerable time and money required for performing them. As with all experiments, a good design begins…

The Fluorochrome Less Excited: How To Build A Flow Cytometry Antibody Panel

The Fluorochrome Less Excited: How To Build A Flow Cytometry Antibody Panel

By: Tim Bushnell, PhD

Fluorochrome, antibodies and detectors are important. The journey of a thousand cells starts with a good fluorescent panel. The polychromatic panel is the combination of antibodies and fluorochromes. These will be used during the experiment to answer the biological question of interest. When you only need a few targets, the creation of the panel is relatively straightforward. It’s only when you start to get into more complex panels with multiple fluorochromes that overlap in excitation and emission gets more interesting.  FLUOROCHROMES Both full spectrum and traditional fluorescent flow cytometry rely on measuring the emission of the fluorochromes that are attached…

Flow Cytometry Year in Review: Key Changes To Know

Flow Cytometry Year in Review: Key Changes To Know

By: Meerambika Mishra

Here we are, at the end of an eventful year 2021. But with the promise of a new year 2022 to come. It has been a long year, filled with ups and downs. It is always good to reflect on the past year as we move to the future.  In Memoriam Sir Isaac Newton wrote “If I have seen further, it is by standing upon the shoulders of giants.” In the past year, we have lost some giants of our field including Zbigniew Darzynkiwicz, who contributed much in the areas of cell cycle analysis and apoptosis. Howard Shapiro, known for…

What Star Trek Taught Me About Flow Cytometry

What Star Trek Taught Me About Flow Cytometry

By: Tim Bushnell, PhD

It is no secret that I am a very big fan of the Star Trek franchise. There are many good episodes and lessons explored in the 813+ episodes, 12 movies (and counting). Don’t worry, this blog is not going to review all 813, or even 5 of them. Instead, some of the lessons I have taken away from the show that have applicability to science and flow cytometry.  “Darmok and Jalad at Tanagra.”  (ST:TNG season 5, episode 2) This is probably one of my favorite episodes, which involves Picard and an alien trying to establish a common ground and learn…

Top Technical Training eBooks

Get the Advanced Microscopy eBook

Get the Advanced Microscopy eBook

Heather Brown-Harding, PhD

Learn the best practices and advanced techniques across the diverse fields of microscopy, including instrumentation, experimental setup, image analysis, figure preparation, and more.

Get The Free Modern Flow Cytometry eBook

Get The Free Modern Flow Cytometry eBook

Tim Bushnell, PhD

Learn the best practices of flow cytometry experimentation, data analysis, figure preparation, antibody panel design, instrumentation and more.

Get The Free 4-10 Compensation eBook

Get The Free 4-10 Compensation eBook

Tim Bushnell, PhD

Advanced 4-10 Color Compensation, Learn strategies for designing advanced antibody compensation panels and how to use your compensation matrix to analyze your experimental data.