Thursday, December 4, 2014

Archiving practice for model code of agent-based models

There is increasing concern over the repeatability and reproducibility of computational science (see also here, here, here, here and here). If computational scientific enterprises want to be accumulative more transparency is required including the archiving of computer code in public repositories. This also holds for agent-based modeling, an increasingly popular methodology in the social and life sciences.
            I show here some initial results of an analysis of the practice of archiving agent-based models. Five journals were selected that regularly publish research that use agent-based models:  Advances in Complex Systems, Computational and Mathematical Organization Theory, Ecological Modelling, Environmental Modeling and Software, Journal of Artificial Societies and Social Simulation. Using the ISI web of science we searched for all articles in those 5 journals in the years 2010 to 2014 using the search term "agent-based model*". This resulted in 255 articles on September 5, 2014 of which 56 articles were disregarded since they did not discuss an agent-based model itself.
            Out of the 199 remaining articles 135 were found not to provide the computational model’s source code. 21 articles referred to an institutional or individual homepage. In 5 cases, the link resulted in a 404 not found error and we recorded that the code was not available. In 17 cases the code was included as an electronic appendix of the journal.  Only 31 articles provided the model code in a public archive, out of which 26 were stored at the CoMSES Computational Model Library. The other 5 models were archived in repositories like Bitbucket, Git Hub, Google code, Sourceforge and the Netlogo community models site.
Percentage of archived model per year.
            Over the years there has been improvement in model archiving. In 2010 75% of models were not archived. The increasing availability of public archives has enabled authors to archive their models more frequently and in 2014 50% of the models are archived. The majority of those models are archived in OpenABM.
Percentage of archived model per journal.
            As we can see, most models are still not archived. One journal has championed model archiving with more than 50% of its publications associated with a publicly archived model, whereas the other journals have an archiving percentage between 10% and 20%.
          Since most research is sponsored by tax money, sponsors sometimes explicitly require that the data, including software code, is made publicly available. We find that papers from the 2 main sponsors (16 by European Commission and 21 by the National Science Foundation) experience a low compliance rate to best practices. In both cases we find that only 15% of the models are available in public archives, significantly lower than the articles that do not list a sponsor (29%), or list other sponsors (24%).
        Currently the scope of the analysis is extended to about 3000 articles (using search term agent-based model* unrestricted to years and journals). Besides getting a better picture of current archiving practices we also hope this activity lead to more awareness of the problem and the need for journals to increase requirements for archiving code and documentation in public repositories. 



Thursday, September 18, 2014

Open Science; Science for a 21th century experiences roadblocks from 19th century incentive structures


The book Reinventing Discovery by Michael Nielsen is a joy to read. Technological development enables the production of knowledge by a large group of people making small contributions, instead of the isolated inventions in 19th century science. Nielsen discussed examples like the Polymath project where mathematicians - from field medalist to high school students - collaborate to solve problems, the GalaxyZoo (classifying galaxies) and Foldit (folding proteins). These success stories brought large number of people together - often non-specialists - to solve problems faster and better than individuals may do. Key components for success are the ability to split the problem in smaller modules, and clear measures of performance (getting a score when folding proteins).
So citizens start solving science problems, but a harder problem is to have scientists doing research in the open, Open science, which enable others to build on it. The incentives structures in Science are perverse to stimulate discovery. In fact we use incentives structures from the 19th century ignoring the potential increase in knowledge production if we use a 21th century approach. This networked open science approach aspired by Nielsen experience major challenges due to the incentives of scientists to publish results in high profile journals, but not getting recognition for sharing data and/or computer code, which are often the main outcomes of a project.
As we experience in the development of openabm scientists like to download the models of others, but are reluctant to archive their own work. Furthermore, journals are reluctance to increase the standards of transparency and sponsors like the National Science Foundation require data management plans but do not invest sufficiently in cyberinfrastructure to make this possible.
Although there is self-organization at the small scale in the science community, the sponsors need to step up and invest seriously to make a science for the 21th century possible.

Sunday, July 27, 2014

The re-emergence of the roving bandit

Mancur Olson introduced the concept of roving and stationary bandits to explain why dictators - stationary bandits - have a self-interest to the society they rule productive to maximize the rent they can collect from the population. This in contrast to roving bandits who have no obligation to provide protection to the population, or keep the land productive. Roving bandits just plunder and steal. A stationary bandit uses taxation.
With the development of nation states during the last few centuries we seem to have entered a period where we don't have roving bandits. Some countries experience democratic systems, others autocratic. But in recent years we seem to observe the re-emergence of roving bandits, as a nasty side-effect in panacea thinking of the benefits of market solutions and democratic systems. Note that democracy is typically portrait as voting, not the involvement of people in decision making, which is the key to democracy according to Vincent Ostrom.
We have multinational companies who move around to avoid taxation and exploit natural resources. When resources are depleted they can move on to other countries. Whether this is shrimp farming in rice field in south east Asia, gold mines in Africa, or soy bean production in Latin America, the multi-national companies lack the incentives to care about the long term for the people, and often local governments are too weak to implement and enforce regulations to reduce the negative impact.
Former stationary bandits
At a different scale, we see roving bandits emerging in Northern Africa and the Middle East. The so-called Arab Spring led to the removal of some dictators who suppressed a large part of their population. Unfortunately we now learn that those dictators were able to provide some security compared to the anarchy currently ruling in 'countries' like Libya, Syria and Iraq. Those dictator were able to suppress the violence between different ethnic and religious groups within their countries. These suppressive regimes favored their own ethnic and religious groups, but compared to the current anarchy, we may wonder whether everyone did benefit.
Douglas North and his colleagues public a book in 2009 on the difficulty of societies to transition of controlling social order via violence, to the control social order via votes and democratic institutions. Those transitions have been rare, had a long duration and are embedded in a long history of social norm development that supports democratic institutions. As we have seen in recent years, removing dictators is not a solution to establish a less violent and prosperous society. I don't have a solution to this problem, but we should at least learn from history and avoid creating more anarchy as has happened in recent years.


Wednesday, January 1, 2014

The biology of trust

The book "The moral molecule" by Paul Zak is a great book discussing the research on the relation between oxytocin and decision making. Using the trust game and measurements of oxytocin in the blood before and after decisions are make, dr. Zak shows that a higher increase with oxytocin correlates with higher levels of trust. By artificially increasing oxytocin levels participants increase their level of cooperation in experimental games. Dr. Zak also did experiments with non-traditional groups such as people who are abused, religious groups, and tribes in Papua New Guinea.
The result is a fascinating discussion how hormone levels like oxytocin and testosterone affect decision making. These hormone levels are defined by genetics, social context and change during the life cycle of our lives. A better understanding how hormone levels, due to frequent hugging as practiced in the lab of dr. Zak, affect decision making has consequences how we organize our social interactions. Although new technologies lead to "social snacking" they do not replace the biological responses to physical and face-to-face contacts with other people. If we want to cope with increasingly complex collective action problems at higher levels of scale we need to take into account the biological context of our decision making next to the efficiency of information exchange by the new technologies.