Service provider: Datopian
Client: Organisation for Economic Co-operation and Development (OECD)
- professional consulting on open-source software
- training sessions
- stakeholder interviews
- technical analysis
The Organisation for Economic Co-operation and Development (OECD) is an international organisation that provides a data hub for official statistics, called the .Stat Suite. The software is co-developed by community members, of which there are currently 15. These include UNICEF, the International Labour Organization and the National Bank of Belgium, in addition to numerous national statistics offices, such as Australia, New Zealand, Italy and Tunisia. The SIS-CC also works with partners such as EuroStat and PARIS21.
In 2019, the community made the strategic decision to transition to open-source software. The internal directive to go open source also had the advantage that it catered to those users that increasingly wanted to be able to integrate their existing systems with the .Stat Suite. The .Stat Suite technology had been around for about 10 years in its modern form (it was an evolution of the 2003 .Stat Software) and the transition to open-source was its most technically complex iteration yet.
The .Stat Suite was built with SDMX at its core, which is an ISO standard. It is made up of three components: 1) the data management part (.Stat Data Lifecycle Manager); 2) the service layer (.Stat Core); and 3) the dissemination end (.Stat Data Explorer).
To meet the challenges presented by the transition to open source, the SIS-CC was interested in tapping into the experiences of similar, open-source projects. This meant finding an expert in open-source software with a long history of working with open-source environments. Their ultimate goal was to move towards a fully open-source model, such as CKAN.
“We looked at CKAN as a good benchmark to work towards” – Jonathan Challener, SIS-CC Community Manager at OECD
The SIS-CC took part in some initial workshops with Datopian to plan their open-source journey. This introduced the SIS-CC team to some key points to think about, such as documentation and the continuous integration/delivery aspects. After exposing the team to these new concepts, Datopian then worked with internal stakeholders at the OEDC, as well as some external stakeholders from the wider community, to produce a detailed report of recommendations for how to best go about open-sourcing their codebase. This involved Datopian advising the SIS-CC on everything from community engagement through to more technical issues on how codebases work.
Since Datopian’s review, the .Stat Suite has been developed heavily and the SIS-CC has put in place a full devops operation. They now have three different delivery mechanisms: one is .Stat Suite as Containers; .Stat Suite as Codebase; and .Stat Suite as a Service.
They’ve also adopted technologies such as docker and kubernetes, which facilitate easy deployment and scalability. In some cases, the SIS-CC has been able to build docker-compose files that can be deployed by developers in a matter of minutes, rather than days. This is because .Stat Suite in its most recent form operates as a set of microservices, which allows developers to work on one part of the system without necessarily knowing how other parts work. With the microservices approach, the codebase is broken up into modular parts that are not dependent on the other parts in order to function. This has the knock-on effect that the barrier to entry for new developers to work on the open-source codebase is much lower.
“The Stat Suite is really opening up new projects and being leveraged by others in different contexts” – Jonathan Challener, SIS-CC Community Manager at OECD
The SIS-CC was able to use their newfound open-source knowledge to engage others. At a training workshop in Bangkok with 6 National Statistics Offices run by the United Nations Statistics division – supported by UNICEF and the OECD – The .Stat Suite was deployed in a multi-tenant environment for training purposes. It was available to each participant in their own environment.
The SIS-CC is on the path to building its open-source community. Datopian hopes to engage further with them in the future and assess the progress of the community.