Software as a heritage for humanity: How Ceph will help store and preserve code for future generations

Software Heritage

Software Heritage is a non-profit organization, headquartered in Paris, with the mission of collecting, preserving and sharing all software source code for present and future generations as a heritage for humanity.

In 2021, Software Heritage partnered with the Red Hat Social Innovation Program, committed to the use of open source technologies for the greater good, and started a collaboration around the development of a massively scalable software-defined storage architecture built on Ceph storage.

By building an object data store, Software Heritage is able to take advantage of a cost-effective, highly scalable, distributed system, and with enriched metadata, they can ensure that software repositories will be easily searched and retrieved for use by future generations.

Listen to this informative interview on Software Engineering Radio.

Storing all the code of the world in a reliable, scalable, and affordable way requires building one of the largest archives ever created.

Software Heritage aims to not only collect and preserve the world’s code, but also to share it for practical use, and offer it as a reference guide for future generations to inherit and learn from. The open sharing of all code will also help humanity reuse valuable code, providing solid, common foundations to serve the different needs of heritage preservation, science, and industry.

But how can we ensure that all this data is stored with resiliency and organized in a reliable, scalable and affordable way? That’s why, to build its next-generation object storage system, Software Heritage chose Ceph, an open source solution that is scalable, has no vendor lock-in and is highly reliable.

An endeavor of this magnitude needs a reliable, affordable, and scalable solution. For Software Heritage, storing and sharing all of the world’s code is their core mission. Ceph will support this goal by providing industry-leading performance, reliability and flexibility with rapid scalability–minimizing storage costs, and building durable, resilient clusters. 

Ceph makes working with data at scale a great option for any organization because it runs on any commodity hardware platform, allowing full control over the capabilities and capacity of the cluster. Basic offerings include enhanced search capabilities, as well as native support for file and block storage protocols. Additionally, Ceph is a distributed software-defined storage system, meaning that there are no dedicated controllers that could fail. So, if Software Heritage ever loses a node, the cluster will rebuild itself with no downtime. Plus, storage capacity can easily be increased by adding new nodes to the cluster while the data is live, and Ceph redistributes the load accordingly. 

Best of all, Ceph is cost-effective. With native support for file, block, and object storage, Software Heritage can create and tune storage pools, delivering the performance required for high-latency workloads in addition to an archive object data store. This flexibility gives data engineers the ability to manage costs while concentrating on delivering the service levels required by the organization — all in one solution, with one management plane, providing incremental operational cost savings.

Red Hat’s Social Innovation Program provides consulting services and product support to nonprofit-based open source projects that are solving social and environmental issues. Through our joint commitment of using open source technologies for the greater good, Red Hat is partnering with Software Heritage and utilizing Red Hat Consulting services to help design a successful foundation for their shared vision of creating an archival database to track the origin and development of software.

When it comes to implementing such a large-scale project, having a solid architecture is key to building a strong, scalable base, not only to preserve, but to share all the code of the world. This is why Software Heritage engineers worked with Red Hat Ceph experts, including the team that created and supports Ceph, to ensure that the design can support the massive scalability of Software Heritage’s deployment. 

As this project unfolds, Red Hat will continue to contribute resources, helping to validate performance, implement desired features, provide deployment guidance, and consult on the development and operation of Ceph storage. Our continued partnership helps to fulfill Red Hat’s goal of fostering collaboration towards building better open source solutions towards the benefit of all.