PREMIER Model Planning / Study Design

Introduction

The functional and performance requirements for models and methods depend on the planned research project in which they are to be used, the research question and the disease or physiopathology to be investigated. The scientists involved determine the requirements and objectives of the model planning.

The experimental design is defined as:

A plan for the collection and use of data so that the desired information can be obtained with sufficient accuracy or so that a hypothesis can be tested properly (ref. 1) (ref. 2).

A good experimental design is important to answer correctly the research question of interest in an unbiased way that can be generalized to the desired or intended audience. This includes defining the measurements to be performed, the timing, etc. It also includes identifying the relevant statistical analyses, determining the appropriate sample size and establishing a random system (ref. 3).

Requirements for good experimental design and analysis:

  • An a priori explanation of the objectives/hypotheses.
  • Appropriate experimental methods.
  • Sufficient but not overly large sample size to ensure statistical significance of biologically relevant effects.

A priori determination of appropriate statistical methods.

First and most important is the choice of method (e.g. in vivo, in vitro).  

A more detailed description of these two options is included below.

In vivo models

In vivo studies should be designed so that all meaningful biological effects are statistically significant. In an exploratory study, this "significant effect" could correspond to any pharmacologically relevant effect. The power and sample size analyses are particularly relevant for studies designed to address important endpoints (REF?). Biologically significant effects are not always known in advance, so that in this case a number of plausible effects should be considered. The design of experiments is largely about identifying and establishing a strategy for dealing with different types of variables. The types of variables that occur in research include:

  • Manipulated variable (independent/explanatory variable)
  • Response variable (dependent/outcome variable)
  • Extraneous variables (uncontrolled/random)

The manipulated variable is a targeted attempt to introduce variability into the experiment, for example by administering different doses of a drug. Extraneous variables can interfere with an experiment and alter the results in an undesirable way, or in a way we do not know about. Examples of extraneous variables could be inherent, such as animal variations, time of day, body weight, laboratory noise, etc. Design of experiments is largely about designing a strategy for dealing with extraneous variables. Ignoring them can lead to biased results and the demand for larger sample sizes. Fixing (keeping constant) or eliminating, e.g. by considering only a subgroup of animals, can reduce bias and sample sizes, but also reduce the general validity of the results to only those conditions considered in the experiment. (REF?) Another approach is to control them by integrating them into the experimental design, ideally at the design stage or, if this is not possible, in statistical analysis.

Other additional factors that should be considered are:

  • Appropriate random allocation of animals to treatment groups
  • Blinding of observers in the allocation of medication whenever possible, especially when subjective assessments are to be made by observers.
  • Correct selection of dosages.
  • Optimal selection of control groups.
  • Optimal timing of sampling.
  • Appropriate statistical methodology.

Different design strategies should be carefully considered to minimize variability and maximize information from the experiment. The above design issues should be addressed in the context of the main endpoints (or summary measures) of the study. Examples of such endpoints may include survival rate, glucose normalization, etc. If there are multiple endpoints of interest for a study, certain design questions such as the significance of the study should be evaluated in relation to the endpoints that the researcher and the project team consider to be the most important. (ref. 3)

Endpoints

The key endpoints of the study must first be identified, as all other design decisions should be based on these results. Typical results include:

Statistical significance from control using analysis of variance (ANOVA). ED50 (either absolute or relative) from a dose-response model, such as the 4-parameter logistic model (4PL).

Control Groups

Control groups serve three purposes:

  • a comparison with the test groups,
  • as a quality control marker and
  • to normalize the response for comparison between studies.

A "positive" control is a combination that has a different response than the negative control and is usually the maximum response of a standard treatment. Positive controls are accompanying experiments in which a phenomenon (or effect) achieved with the main experiment is certain to occur. Positive controls are used to demonstrate that a method works with the known values of the variables (method validation) and can therefore exclude false negative results of an experiment.

Negative controls are accompanying experiments in which a phenomenon (or effect) achieved with the main experiment does not occur (zero value) or should not occur. This ensures that a positive result in the main experiment can only have been due to the change in the variables. In contrast, a positive result of the negative control indicates a lack of specificity of the respective main experiment, i.e. the effect achieved in the main experiment also occurs due to other influences. Negative controls help exclude other reasons, or error sources different from the hypothesis for a phenomenon and serve to avoid incorrect interpretations of false positive results. This reduces the possibility of misrepresentations of the hypothesis after publication. If the results of the experiment and the negative controls are negative, it can be concluded that the experiment output is independent of the variables, i.e. the change in the variables had no influence on the experiment output. In animal experiments, placebo administration is a typical negative control. Blind and double-blind experiments avoid the use of additional negative controls to investigate the influence of the level of knowledge of the subject and, in the latter case, of the experimenter on an experimental result.

In-vitro Cell Culture Models

If a question is to be processed in an in-vitro model, the following decision flow can be used.

Selection of cells

What kind of cells are of interest (neurons, endothelium, etc.)?

Should cell lines or primary cell cultures be used?

Cell lines
  • Are the cell lines available in the institution? Is there a SOP about handling of Liquid Nitrogen and retrieval of samples from storage?
  • Are the cell lines characterized (data sheets ATCC, publications, SOP) and tested for mycoplasma?
  • Are genetically modified lines available from the selected cell line?
Primary cell cultures
  • Are there protocols for the preparation of primary cells?
  • Who can provide training?
  • Where do you get the material from? Is it possible to use tissue from other projects?
  • What are the quality criteria for the primary cultures?
Selection of the model
  • Chemical hazard (toxicity of substances)
  • Physical hazard (temperature, pH, flow)

Once the target parameters have been defined, the model can be selected:

Influencing  Impact factors

The previous steps must be followed by considering the possible influencing factors (both known and unknown) and the resulting outcome.

There is a very informative paper (https://www.pmi.org/learning/library/characterizing-unknown-unknowns-6077 ) on this topic.

Below some factors that (may) influence the results of cell culture experiments are listed:

  • Cell density
  • Age of the cells - especially in primary cultures
  • Cell-culture passage protocol (Interval from the beginning of the experiment)
  • Protocol of medium change (interval, complete or partial medium change))
  • Volume of the medium used during the experiment
  • CO2, bicarbonate, pH value, medium supplements
  • Temperature

A reference group is a control group within a scientific experiment in which, experimental the independent variable or intervention is not changed or applied. Ideally, the group distribution is randomized, with identical conditions in the control and experimental groups. To achieve valid and robust results, control groups are essential in every experiment.

New methods must be exhaustively validated, i.e. it must be checked whether they meet the pre-set requirements and specifications and thus, fulfills the intended purpose.

It is highly recommended to get professional advice well in advance of the starting of a project on the specific biometrics, recommended statistical software and analysis plan.

  1. MESH Definition: Research Design. https://www.ncbi.nlm.nih.gov/mesh/?term=experimental%20design
  2. Fry DJ. Teaching experimental design. ILAR J. 2014;55(3):457-71. doi: 10.1093/ilar/ilu031. Review. PubMed PMID: 25541547. https://academic.oup.com/ilarjournal/article/55/3/457/643598
  3. Sittampalam GS, Coussens NP, Brimacombe K, et al., editors. Assay Guidance Manual [Internet]. Bethesda (MD): Eli Lilly & Company and the National Center for Advancing Translational Sciences; 2004-. Available from: https://www.ncbi.nlm.nih.gov/books/NBK53196/
  4. Charité Institut für Biometrie und Klinische Epidemiologie (iBikE)
CC BY 4.0