The flexibility of text provides researchers
with a lot of freedom to select different options. For example, a
researcher can select among many different layers (in BERT base 12 and
in BERT large 24); and these layers can be aggregated in different ways
including using mean, minimum or maximum. It is also possible to use
different number of PCA components (or not use PCA at all) in training;
as well as selecting different regression algorithms including (multiple
linear regression or ridge). All these options are great for learning
more about these methods. However, when hypotheses testing is important
to not fall pray for researcher degrees of freedom and avoid
the risk of (unconsciously) p-hacking (e.g., see Simmons, Nelson, &
Simonsohn, 2011).
Researcher degrees of freedom refers to the inherent flexibility
involved in conducting research including carrying out experiments as
well as analyzing the data. Researchers can choose among many ways of
analyzing their data, and these ways can, for example, be selected
arbitrarily or on the basis that certain ways result in more desirable
outcomes such as a statistically significant result (Simmons, Nelson,
& Simonsohn, 2011). Or put another way, the flexibility in
text is a double edged sward where abusing the options
leads to p-hacking: the analytic process of consciously or unconsciously
trying several types of analyses until achieving the desired
results.
Specify language model , specify which layers that will be used and how they will be aggregated.
Example of aspects to consider in a pre-registration of
hypotheses testing
This is not an exhaustive list; rather think through your analyses as
carefully as possible and consider which decisions that can be
appropriately be made in advance. For example,
Type of model (e.g., BERT-base, BERT-large, multilingual BERT, RoBERTa, XLnet, etc.)
Which layers (e.g., all, 11 and 12 etc.)
Layer aggregation method (e.g., mean, minimum, and maximum)
Exclusion of some token (e.g., [CLS] and [SEP])
Type of ML algorithm (e.g., ridge, Random Forest etc.)
Number of cross validation folds in textTrain
Criteria for plotting (e.g., number of words to significance test, plots etc.)
Number of permutations (e.g., in textSimilarityTest, textProjection)
Not(ing) change of random seed. In computer science literature it has recently been discussed that different random seeds can give very different results (e.g., see Mosbach et al., 2020). So perhaps even consider pointing out that seeds will not be changed or commit to a specific seed
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological science. Mosbach, M., Andriushchenko, M., & Klakow, D. (2020). On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines.