This outline of steps to insure reproducibility of collection and analysis of complex datasets is very timely. As pointed out by the authors, software already exists that can automate many of the steps (in some cases accomplishing two or more of the steps in one go, such as automatically logging exact records of the commands used in an analysis in a format that can be simply and quickly re-run).

Nonetheless, those of us working in Psychology departments face a real challenge with such research, particularly when it involves PhD students, who have only a limited amount of time to complete their studies. It is rare to find psychology students who have the requisite computational and/or programming skills to carry out all of the steps outlined in this article. In fact, it is a challenge bringing students with no computational background up to speed in understanding just the basics of computational techniques such as scripting. Getting such students to the stage where they can, for example, use versioning systems is perhaps a step too far. This is particularly the case in smaller departments or labs, in which there is limited support by postdocs or other staff with a more computational background.

The limited time afforded PhD students is a major constraint. In the UK in particular, we are faced with a 3-year time limit on PhD studentships, or in some cases 4 years if one secures funding for an initial 1-year MSc programme (increasingly rare). Is it at all feasible to train psychology students in the requisite computational techniques to the level outlined in this article, while allowing them to complete a programme of research that will enable them to finish a PhD within 3 or 4 years? I have my doubts.

We (especially those of us in the UK) need to have a re-think about the way we train our future cognitive neuroscience researchers. One possibility would be to lengthen the (funded) time to get a PhD. European degrees already tend to be somewhat longer than UK degrees (e.g. a 2-year Masters preceding a PhD, rather than no Masters or a 1 year MSc). Given current limitations on public funding of universities, however, lengthening degrees seems exceedingly unlikely. Another option is to develop undergraduate programmes that are more interdisciplinary from the outset, instead of being constrained by traditional academic disciplines such as Psychology, though this would require universities to loosen their accounting structures that tend to be organised around single-discipline departments, or establish completely new academic departments that reflect the the shape of current scientific research. For example, currently most undergraduate psychologists finish their degrees with a decent training in statistics, but absolutely no training in more general computational techniques such as scripting, modeling or simulation.

Good science is good science. If current educational and funding structures are not sufficient to insure that PhD research is thorough, reliable and can be reproduced, and that PhD students go on to propagate such good research practice, then they need to be changed.