Tuesday, December 20, 2011

Incentives for replication: A collective action problem of journals

Science of December 2 has a great special issue on Data Replication and Reproducibility. This might sounds like a dull topic, but it is critical for science. In the publish or perish era there is a focus on novel results and not on replication of existing studies. Given the high profile fraud cases in previous years where scholars fabricated their own data, we should put more emphasis on documenting and archiving the data. There is a cost of doing this, and that's why it is less common scholars volunteer their time in those activities, although most of them are paid from tax revenue.
Fortunately NSF start requiring a data management plan, although there is not a consensus what the minimum conditions for a valid plan are. Journals may require archiving of data, but many don't.
I am involved in different fields and experience different standards. Journals who publish experimental economics papers require typically a full documentation of the experimental protocol as a condition of acceptable of the paper. Although it is not always stated in the guidelines, reviewers will deny acceptance without proper documentation.
In journals publishing results of agent-based modeling, the culture of model archiving and replication is diverse and problematic. Journals often don't require model documentation and expect reviewers to evaluate papers on pretty figures. From the journal's perspective the increasing requirement will reduce the submission of interesting papers that will boost their impact factor. Since other journals don't require documentation, they don't too. As such many papers have been published, including in journals like Science, Nature and PNAS, which results could not be reproduced, or only after finding out the shaky assumptions behind the models. This shows that better documentation and model archiving for the development of the field. Not doing so will led agent-based modeling becoming a temporary fashion without being taken seriously by the broader scientific community.
I am involved with openabm.org which contains a model archive and more than 100 models are now archived. This is only a small fraction of the published models. Being involved in a small conference that required, but not enforced, model archiving, only 10% of the models were archived.
There is also some positive news. We worked with the journal Ecology & Society to make it a requirement for authors to archive their work on openabm.org if they use an agent-based model. We will have to learn what kind of problems will pop up, but we plan to approach more journals to start requiring proper documentation and archiving.
So, if you are involved with agent-based modeling tools, archive your work. If you review papers, request proper documentation to replicate the results. Contact journals on which you serve on the editorial board to increase their standards of replicability of the research.

Thursday, December 8, 2011

Green Revolution

Yesterday attended a very interesting talk of Carlo Jaeger who gave a seminar in our School of Sustainability. He presented a long term view and systems analysis of the financial crises and this led to different insights than we hear in the popular debate. Carlo wrote an article on his main message in the Financial Times 2 months ago.
There has been a long disbalance of imports and exports. USA is importing more than it exports for a long time, while countries like China and Germany export more. The earned money was used to lend it to USA, Greece, etc. This mechanism worked fine until the irresponsible banking activities led to a financial crises in 2007/2008 and governments bailed out banks and had to borrow more money. The actors with money surplus (including companies (like Apple)) start to hold their money due to uncertainty in what are good investment options. The increasing interests rates are largely caused by distrust and uncertainty, not because of financial realities.
Carlo Jaeger argues that institutional changes are needed to provide incentives for those actors with money to invest it to increase productivity growth. This could be done by a focus on investments in a green low carbon economy: retro fitting of buildings, R&D of renewable energy, etc. Not all these investments will be green, but stimulating an upgrade of the infrastructure will create jobs, investment opportunities, innovation, and a lower energy bill.
In his talk, Jaeger showed a lot of data from the 1930s depression which was resolved by World War 2 when there was a large investment in the military complex. Such a solution to the current economic crisis is not desirable. An alternative major focus would be a serious investment in a low carbon economy which will repay itself by the savings it will costs in the long term.
Unfortunately, the fact free politics of today is likely pursuing other solutions and may dig deeper holes.