The Department for Levelling Up, Housing and Communities has begun piloting an open source data pipeline architecture to underpin its Planning Data platform, replacing earlier batch-processing scripts. The new stack uses Apache Kafka for event streaming and dbt for data transformation, with all code published under an MIT licence on GitHub. The pilot is running against live data feeds from six local planning authorities including Southwark and Sheffield, ahead of a wider rollout anticipated in early 2025.

Source: PublicTechnology.net