Organization for Economic Co-operation and Development

Key facts

Brief summary of the project

The project offers a compelling case for how Datopian created a custom-built discovery portal and search engine that streamlined processes, enabling a Fortune 500 pharmaceutical company to significantly improve dataset access and collaboration for their global research and development teams.

Problem

The OECD faced the challenge of transitioning their data hub for official statistics called .Stat Suite to open-source software to accommodate users' increasing demand for seamless integration with existing systems, while ensuring the software's technical complexity was effectively addressed. This change marked the most technically complex iteration of the software, and OECD needed expert guidance to ensure a smooth and successful transition.

Need

OECD sought a partner with a strong background in open-source software to help them move towards a fully open-source model. They aimed to learn from similar projects and benefit from expertise in areas such as documentation, continuous integration/delivery, and community engagement.

Solution

We collaborated with the OECD, providing expert guidance on their open-source transition journey. By conducting workshops and generating a detailed report of recommendations, Datopian advised the SIS-CC on various aspects from community engagement to technical codebase management. Consequently, the .Stat Suite was transformed with the implementation of full DevOps operations, leveraging microservices architecture, and adopting technologies such as Docker and Kubernetes for enhanced scalability and deployment efficiency. This strategic approach enabled the OECD to empower a growing open-source community and deliver a more accessible, user-friendly data management solution. As a result, OECD successfully transitioned to a microservices-based, open-source model, lowering the barrier for new developers and enabling rapid deployment of the .Stat Suite in various environments.

Main technologies & tools used

Context

The Organisation for Economic Co-operation and Development (OECD) is an international organisation that provides a data hub for official statistics, called the .Stat Suite. The software is co-developed by community members, of which there are currently 15. These include UNICEF, the International Labour Organization and the National Bank of Belgium, in addition to numerous national statistics offices, such as Australia, New Zealand, Italy and Tunisia. The SIS-CC also works with partners such as EuroStat and PARIS21.

The situation

In 2019, the community made the strategic decision to transition to open-source software. The internal directive to go open source also had the advantage that it catered to those users that increasingly wanted to be able to integrate their existing systems with the .Stat Suite. The .Stat Suite technology had been around for about 10 years in its modern form (it was an evolution of the 2003 .Stat Software) and the transition to open-source was its most technically complex iteration yet.

The .Stat Suite was built with SDMX at its core, which is an ISO standard. It is made up of three components: 1) the data management part (.Stat Data Lifecycle Manager); 2) the service layer (.Stat Core); and 3) the dissemination end (.Stat Data Explorer).

The criteria

To meet the challenges presented by the transition to open source, the SIS-CC was interested in tapping into the experiences of similar, open-source projects. This meant finding an expert in open-source software with a long history of working with open-source environments. Their ultimate goal was to move towards a fully open-source model, such as CKAN.

The solution

The SIS-CC took part in some initial workshops with Datopian to plan their open-source journey. This introduced the SIS-CC team to some key points to think about, such as documentation and the continuous integration/delivery aspects. After exposing the team to these new concepts, Datopian then worked with internal stakeholders at the OEDC, as well as some external stakeholders from the wider community, to produce a detailed report of recommendations for how to best go about open-sourcing their codebase. This involved Datopian advising the SIS-CC on everything from community engagement through to more technical issues on how codebases work. 

Outcomes

Since Datopian’s review, the .Stat Suite has been developed heavily and the SIS-CC has put in place a full devops operation. They now have three different delivery mechanisms: one is .Stat Suite as Containers; .Stat Suite as Codebase; and .Stat Suite as a Service.

They’ve also adopted technologies such as docker and kubernetes, which facilitate easy deployment and scalability. In some cases, the SIS-CC has been able to build docker-compose files that can be deployed by developers in a matter of minutes, rather than days. This is because .Stat Suite in its most recent form operates as a set of microservices, which allows developers to work on one part of the system without necessarily knowing how other parts work. With the microservices approach, the codebase is broken up into modular parts that are not dependent on the other parts in order to function. This has the knock-on effect that the barrier to entry for new developers to work on the open-source codebase is much lower.

The SIS-CC was able to use their newfound open-source knowledge to engage others. At a training workshop in Bangkok with 6 National Statistics Offices run by the United Nations Statistics division - supported by UNICEF and the OECD - The .Stat Suite was deployed in a multi-tenant environment for training purposes. It was available to each participant in their own environment.

What’s next?

The SIS-CC is on the path to building its open-source community. Datopian hopes to engage further with them in the future and assess the progress of the community.