We began the process of automating our software dependency updates and, as of today, we extensively utilize automation to keep our dependencies current. In this blog, I share an in-depth exploration of our journey, detailing not only the ‘what’ and ‘how’ but also the unforeseen benefits.
The Dependency Landscape at SAP
Before we delve into the details, let’s set the stage by providing a glimpse into the scale of operations we’re dealing with. At SAP, we have roughly 35,000 active source code repositories, housing both official SAP products and internal services. A vast majority of these repositories rely on both open source and SAP-internal dependencies. We need to ensure that we’re running the correct versions with the latest bug-fixes and features for our services. With such a massive volume of repositories and their dependencies, this becomes a tedious task.
Historically, developers manually checked for new versions and updated them as needed. Some teams even went a step further, building automation for the dependencies they release to notify internal consumers about new versions via automatic pull requests. There were also teams that began using existing open-source automation tools like Renovate or Dependabot to manage their team’s dependencies.
Introduction of Renovate
In 2021, we started utilizing the open-source project Renovate within our own team. Huge shout out here to the people behind Renovate for being really awesome! After demonstrating our experience and setup with neighboring teams, they wanted to leverage Renovate for their own repositories as well.
To avoid having each team duplicate our technical Renovate setup, we decided to allow other teams the reuse of our setup. This way, not every team needs to set up and configure Renovate on their own. Similar to the officially hosted Renovate App on github.com we offer a GitHub App, which teams can activate independently and doesn’t require any central intervention from us. We just provide the infrastructure centrally, along with guidance and support to address any arising questions or issues. If you’re interested in running Renovate yourself, the public documentation is an excellent resource.
Fast-forward two years to today, and we now have 12,000 repositories with Renovate enabled, which accounts for a third of all active SAP repositories. During this period, Renovate has created more than 230k dependency update pull requests.
The Benefits of Automating Dependency Updates
Leveraging Renovate has allowed us to maintain our dependencies at their most current state, and it has provided us with automated notifications whenever new versions are released. This alone has greatly improved our dependency management process. Beyond the evident benefits of using up-to-date dependencies, we’ve seen additional, and somewhat unexpected, advantages from automating our dependency management.
Adoption of Standard Patterns for Dependency Declaration
Although Renovate supports a wide variety of dependency definitions, it doesn’t support all variations which are technically possible. It is possible to get Renovate to recognize custom patterns through the “Regex-Manager” or custom configuration, but this can get complicated fast.
We started to adjust our approach of defining project dependencies to be compatible with Renovate’s capabilities. This adjustment led to more stable parsing compared to Regex-Managers and facilitated the consumption of these dependencies by other automation without additional changes. Moreover, it allowed developers to predict these dependencies in a consistent manner across various projects.
Strengthening of Update Processes
The introduction of automated updates initially resulted in a surge of update pull requests, requiring teams to meticulously review, analyze, and merge them. Some teams, who were used to infrequent dependency updates, perceived the constant review of each pull request as tedious and burdensome.
Over the medium term we observed that many teams improved their processes, strengthening their CI-builds and automated reviews. This refinement minimized the need for manual intervention in merging and deploying modifications, resulting in a more efficient deployment process for all types of changes, not just those related to dependencies. Having high confidence in the automated update process has proven to be highly beneficial, especially when urgent dependency updates, like security updates, are required.
Cultural Shift Towards Automation
The two preceding points have also precipitated a subtle but significant cultural shift towards embracing more automation in all aspects. The adoption of automation, which triggers frequent changes, nudges even smaller development projects to gradually automate more. This, in turn, enables them to deploy smaller increments more confidently, bolstering their faith in automation.
Outlook
Overall, our journey towards automating software dependency updates has not only enhanced the process of keeping our dependencies up-to-date but has also indirectly improved related processes and fostered a culture that embraces automation.
I hope that by sharing our journey, I have provided some valuable insights into our initiatives. I would be interested in learning how you have tackled dependency management and what lessons you’ve learned from it. Please share your experiences and insights about dependency management in the comments section below.
- SAP Managed Tags: