Revolutionizing Open Data in the United Arab Emirates: The Creation of a New Open Data Portal with PortalJS and CKAN

Key facts

Brief summary of the project.

The Federal Competitiveness and Statistics Centre (FCSC) of the UAE partnered with Datopian to upgrade and enhance their existing Open Data portal. Rather than merely upgrading it, we opted for a ground-up redevelopment, retaining only the datasets from the original portal. The new platform, now accessible at https://opendata.fcsc.gov.ae, is designed to set new benchmarks in data quality, user engagement, and multifunctionality, positioning the UAE as a leader in the open data field. Experience the transformation by comparing the old portal and the new one at https://opendata.fcsc.gov.ae.

Problem

The existing open data portal, built on CKAN 2.6 and SiteCore, had limitations in usability, functionality, and user engagement. The platform lacked modern features like a compelling UI/UX design, dynamic charts, and content management integration. This made it less appealing and usable for the public, hindering the FCSC's goal to be among the top 10 countries in the open data field. Here, you can see the old portal.

Need

There was a need for a comprehensive upgrade that would overcome the existing platform's limitations. The new system needed to be bilingual, have a more user-friendly interface and support dynamic features like charts, easy sharing, and user feedback. Moreover, the content management system needed integration to improve SEO capabilities and user engagement.

Solution

We rebuild the portal with cutting-edge UX/UI design, seamlessly integrating storytelling to immerse users in a journey through various UAE-focused topics. Utilizing our PortalJS framework for the front end enabled high customizability and simplified administrative tasks. We implemented bilingual support for English and Arabic, including right-to-left content flow for Arabic. On the backend, we modernized the platform by upgrading to CKAN 2.9.5 and migrated the database to a secure Azure Kubernetes Cluster. We also seamlessly integrated a custom GhostCMS instance, empowering administrators with the ability to create and update content such as pages, news articles, navigation links, and homepage visuals. We added resource citation, sharing, rating, and dynamic visualization features to elevate user experience. Further, we designed a CKAN dataset approval workflow aligned with FCSC's data quality standards, ensuring only vetted and approved datasets are publicly accessible.

Main technologies & tools used

Context

The Federal Competitiveness and Statistics Centre (FCSC) is a government center affiliated with the Ministry of Cabinet Affairs in the United Arab Emirates (UAE). FCSC is actively involved in raising awareness of the culture of competitiveness and the importance of providing quality, accurate data and the powerful impact of statistics and competitiveness on policy-making and strategic planning. FCSC aims to become a global role model that contributes to the UAE’s advancement and prosperity. As part of the UAE government's intention to enhance sustainable communities based on participation and transparency, and to promote the concept of the digital knowledge economy, there is an official UAE Open Data portal based on CKAN that provides open data systems for public use, managed by FCSC. Datasets include economy, education, society, technology, transportation, environment, government, health, and infrastructure.

As part of their goal to be in the top 10 countries in the open data field, FCSC were seeking partners with enough expertise and experience to help them move forward to a new generation of an open data platform. Before reaching out to Datopian, they contacted many other vendors but Datopian came out as the best candidate. As the creators of CKAN, our company has been building data portals for longer than anyone else and has extensive experience and know-how on how to do open data projects. Our team has clear, established methodologies for implementing open data projects which takes the burden off the shoulders of our clients. We make things easy.

The situation

This project was born as a means of helping FCSC achieve their goals of providing good quality UAE-centered data to the public in a more sophisticated, modern, and good-looking way, taking advantage of cutting-edge front-end technologies, and therefore thrusting their open data portal discoverability, usability, and engagement.

The UAE government already had an official open data portal (https://bayanat.ae) which was built using CKAN (version 2.6), а SiteCore as a content management system, and more than 2600 datasets. It was obvious that we had to overcome the lack of support and the limitations of the old platform that was implemented in 2017 by upgrading everything.

The criteria

Our task was to enhance and expand the platform functionality, usability, and integration flexibility by leveraging best practices and the latest technologies.

The most important requirements for this project were:

• Meaningful, elegant, and clean UI/UX
• Dynamic charts to display insightful data
• Enhanced data catalog experience with easy navigation
• Content management system integration, allowing the administrators to publish content, therefore serving as a communication medium and also improving the portal’s SEO capabilities
• Bilingual support for English and Arabic, also adjusting the content flow to right-to-left for the latter one
• Resource citation and sharing capability, allowing users to easily share resources with others and researchers, students, and journalists to easily cite resources as sources
• Resource rating capability, empowering the users to provide feedback to the team

The solution

UX/UI Design

The project first got started with the development of the portal mock-ups in Figma. A really important aspect of that was storytelling. Our intention was to make the users feel like they embarked on a journey to the UAE, allowing the users to discover and travel to a variety of topics of interest.

Database migration

We then proceeded to work on the database migration to a brand new CKAN 2.9.5 instance hosted on Azure Kubernetes Cluster, altogether with the decoupled front end and a GhostCMS instance. Two environments were set up for this project: one for production and one for staging. This setup ensures that only complete features are released to the final users and everything can be properly quality tested before going to production.

Dataset approval workflow

Once the project environments were set up, the team commenced the development of the dataset approval workflow. It was imperative to the client that the workflow mirrored the activities and business processes of their Data Quality and Curation team. To achieve this, the Datopian team worked closely with the client to better understand these processes and then designed and developed a dataset approval workflow that not only mirrored the client’s current processes but included features that resolved the pain points identified in these processes. Since the dataset approval workflow is integrated into the CKAN platform, it harnesses all the benefits of the CKAN platform.

Frontend

After that, we were ready to start the front-end development of the brand-new data portal, using PortalJS as the framework. PortalJS is an open-source React framework built by Datopian that allows developers to create high-quality data portals using CKAN back end with ease, providing the most usual components that can easily be customized and used as the building blocks of the portal, therefore allowing the development team to focus its efforts on the client-specific features that have to be developed. You can learn more about PortalJS here.

Easy customization

Taking advantage of GhostCMS, we also enabled the client to keep on evolving the data portal in many ways without the need for tech assistance. Administrators can easily change the homepage hero image, and the links presented on the navigation bar and the footer, add pages and news articles, and everything in both supported languages.

Bilingual support for English and Arabic

As a bilingual website, one of the things we had to keep in mind during the development process was the support of the two content flows: right-to-left and left-to-right. Take a look at our homepage hero in both languages:

Developing special features - rate, preview, download, share, cite, build charts

As a means of increasing the open data portal usability, and having the users in mind, many special features were added. Users are now encouraged to leave their feedback on the resources and can easily share, cite, preview and download directly from the resources page:

Going even beyond that, we implemented a chart builder. Dynamic charts can be constructed based on the selected resource data:

The outcome

The UAE official data portal (UAE Open Data portal), now provides an enriched feature set, making it even more secure and optimized, and users now have access to a more appealing data portal, with a rich UI/UX and storytelling alluding to a trip across the UAE.

What’s next?

The FCSC team is planning to expand and develop their open data strategy with high expectations for substantial benefits to society. Open data doesn’t end up with opening the data. You need to stimulate and support an ecosystem of data producers, innovators, and users. You also need to make sure an ecosystem approach to planning and design is used to assess existing conditions and to consider policies, strategies, and relationships that address realistic barriers and stimulate desired benefits.

Datopian is renowned for its integrity, agility, discretion, and dedication to clients and for providing value-added services. We maintain long-term relationships with our clients around the globe and are happy that we’ll continue to collaborate with our clients at FCSC and provide them with our array of services.