1 - Contributing Bigger Changes
Contributing Bigger Changes
Here are the guidelines you should follow when contributing larger changes to Gardener:
We strongly recommend to write a Gardener Enhancement Proposal (GEP) to get a common understanding what you want to achieve. This makes it easier for reviewers to understand the big picture.
Avoid proposing a big change in one single PR. Instead, split your work into multiple stages which are independently mergeable and create one PR for each stage. For example, if introducing a new API resource and its controller, these stages could be:
- API resource types, including defaults and generated code.
- API resource validation.
- API server storage.
- Admission plugin(s), if any.
- Controller(s), including changes to existing controllers. Split this phase further into different functional subsets if appropriate.
If you realize later that changes to artifacts introduced in a previous stage are required, by all means make them and explain in the PR why they were needed.
Consider splitting a big PR further into multiple commits to allow for more focused reviews. For example, you could add unit tests / documentation in separate commits from the rest of the code. If you have to adapt your PR to review feedback, prefer doing that also in a separate commit to make it easier for reviewers to check how their feedback has been addressed.
To make the review process more efficient and avoid too many long discussions in the PR itself, ask for a “main reviewer” to be assigned to your change, then work with this person to make sure he or she understands it in detail, and agree together on any improvements that may be needed. If you can’t reach an agreement on certain topics, comment on the PR and invite other people to join the discussion.
Even if you have a “main reviewer” assigned, you may still get feedback from other reviewers. In general, these “non-main reviewers” are advised to focus more on the design and overall approach rather than the implementation details. Make sure that you address any concerns on this level appropriately.
2 - CI/CD
CI/CD
As an execution environment for CI/CD workloads, we use Concourse.
We however abstract from the underlying “build executor” and instead offer a
Pipeline Definition Contract
, through which components declare their build pipelines as
required.
In order to run continuous delivery workloads for all components contributing to the
Gardener project, we operate a central service.
Typical workloads encompass the execution of tests and builds of a variety of technologies,
as well as building and publishing container images, typically containing build results.
We are building our CI/CD offering around some principles:
- container-native - each workload is executed within a container environment. Components may customise used container images
- automation - pipelines are generated without manual interaction
- self-service - components customise their pipelines by changing their sources
- standardisation
Learn more on our: Build Pipeline Reference Manual
3 - Dependencies
Testing
We follow the BDD-style testing principles and are leveraging the Ginkgo framework along with Gomega as matcher library. In order to execute the existing tests, you can use
make test # runs tests
make verify # runs static code checks and test
There is an additional command for analyzing the code coverage of the tests. Ginkgo will generate standard Golang cover profiles which will be translated into a HTML file by the Go Cover Tool. Another command helps you to clean up the filesystem from the temporary cover profile files and the HTML report:
make test-cov
open gardener.coverage.html
make test-cov-clean
sigs.k8s.io/controller-runtime env test
Some of the integration tests in Gardener are using the sigs.k8s.io/controller-runtime/pkg/envtest
package.
It sets up a temporary control plane (etcd + kube-apiserver) against the integration tests can run.
The test
and test-cov
rules in the Makefile
prepare this env test automatically by downloading the respective binaries (if not yet present) and set the necessary environment variables.
You can also run go test
or ginkgo
without the test
/test-cov
rules.
In this case you have to set the KUBEBUILDER_ASSETS
environment variable to the path that contains the etcd + kube-apiserver binaries or you need to have the binaries pre-installed under /usr/local/kubebuilder/bin
.
Dependency Management
We are using go modules for depedency management.
In order to add a new package dependency to the project, you can perform go get <PACKAGE>@<VERSION>
or edit the go.mod
file and append the package along with the version you want to use.
Updating Dependencies
The Makefile
contains a rule called revendor
which performs go mod vendor
and go mod tidy
.
go mod vendor
resets the main module’s vendor directory to include all packages needed to build and test all the main module’s packages. It does not include test code for vendored packages.
go mod tidy
makes sure go.mod
matches the source code in the module. It adds any missing modules necessary to build the current module’s packages and dependencies, and it removes unused modules that don’t provide any relevant packages.
The dependencies are installed into the vendor
folder which should be added to the VCS.
Warning
Make sure that you test the code after you have updated the dependencies!4 - Security Release Process
Gardener Security Release Process
Gardener is a growing community of volunteers and users. The Gardener community has adopted this security disclosure and response policy to ensure we responsibly handle critical issues.
Gardener Security Team
Security vulnerabilities should be handled quickly and sometimes privately. The primary goal of this process is to reduce the total time users are vulnerable to publicly known exploits. The Gardener Security Team is responsible for organizing the entire response, including internal communication and external disclosure, but will need help from relevant developers and release managers to successfully run this process. The Gardener Security Team consists of the following volunteers:
Disclosures
Private Disclosure Process
The Gardener community asks that all suspected vulnerabilities be privately and responsibly disclosed. If you’ve found a vulnerability or a potential vulnerability in Gardener, please let us know by writing an e-mail to secure@sap.com. We’ll send a confirmation e-mail to acknowledge your report, and we’ll send an additional e-mail when we’ve identified the issue positively or negatively.
Public Disclosure Process
If you know of a publicly disclosed vulnerability please IMMEDIATELY write an e-mail to secure@sap.com to inform the Gardener Security Team about the vulnerability so they may start the patch, release, and communication process.
If possible, the Gardener Security Team will ask the person making the public report if the issue can be handled via a private disclosure process (for example, if the full exploit details have not yet been published). If the reporter denies the request for private disclosure, the Gardener Security Team will move swiftly with the fix and release process. In extreme cases GitHub can be asked to delete the issue but this generally isn’t necessary and is unlikely to make a public disclosure less damaging.
Patch, Release, and Public Communication
For each vulnerability, a member of the Gardener Security Team will volunteer to lead coordination with the “Fix Team” and is responsible for sending disclosure e-mails to the rest of the community. This lead will be referred to as the “Fix Lead.” The role of the Fix Lead should rotate round-robin across the Gardener Security Team. Note that given the current size of the Gardener community it is likely that the Gardener Security Team is the same as the “Fix Team” (i.e., all maintainers).
The Gardener Security Team may decide to bring in additional contributors for added expertise depending on the area of the code that contains the vulnerability. All of the timelines below are suggestions and assume a private disclosure. The Fix Lead drives the schedule using his best judgment based on severity and development time.
If the Fix Lead is dealing with a public disclosure, all timelines become ASAP (assuming the vulnerability has a CVSS score >= 7; see below). If the fix relies on another upstream project’s disclosure timeline, that will adjust the process as well. We will work with the upstream project to fit their timeline and best protect our users.
Fix Team Organization
The Fix Lead will work quickly to identify relevant engineers from the affected projects and packages and CC those engineers into the disclosure thread. These selected developers are the Fix Team. The Fix Lead will give the Fix Team access to a private security repository to develop the fix.
Fix Development Process
The Fix Lead and the Fix Team will create a CVSS using the CVSS Calculator. The Fix Lead makes the final call on the calculated CVSS; it is better to move quickly than make the CVSS perfect.
The Fix Team will notify the Fix Lead that work on the fix branch is complete once there are LGTMs on all commits in the private repository from one or more maintainers.
If the CVSS score is under 7.0 (a medium severity score) the Fix Team can decide to slow the release process down in the face of holidays, developer bandwidth, etc. These decisions must be discussed on the private Gardener Security mailing list.
Fix Disclosure Process
With the fix development underway, the Fix Lead needs to come up with an overall communication plan for the wider community. This Disclosure process should begin after the Fix Team has developed a Fix or mitigation so that a realistic timeline can be communicated to users. The Fix Lead will inform the Gardener mailing list that a security vulnerability has been disclosed and that a fix will be made available in the future on a certain release date. The Fix Lead will include any mitigating steps users can take until a fix is available. The communication to Gardener users should be actionable. They should know when to block time to apply patches, understand exact mitigation steps, etc.
Fix Release Day
The Release Managers will ensure all the binaries are built, publicly available, and functional before the Release Date. The Release Managers will create a new patch release branch from the latest patch release tag + the fix from the security branch. As a practical example, if v0.12.0
is the latest patch release in gardener.git, a new branch will be created called v0.12.1
which includes only patches required to fix the issue. The Fix Lead will cherry-pick the patches onto the master branch and all relevant release branches. The Fix Team will LGTM and merge. The Release Managers will merge these PRs as quickly as possible.
Changes shouldn’t be made to the commits, even for a typo in the CHANGELOG, as this will change the git sha of the already built commits, leading to confusion and potentially conflicts as the fix is cherry-picked around branches. The Fix Lead will request a CVE from the SAP Product Security Response Team via email to cna@sap.com with all the relevant information (description, potential impact, affected version, fixed version, CVSS v3 base score, and supporting documentation for the CVSS score) for every vulnerability. The Fix Lead will inform the Gardener mailing list and announce the new releases, the CVE number (if available), the location of the binaries, and the relevant merged PRs to get wide distribution and user action.
As much as possible, this e-mail should be actionable and include links how to apply the fix to users environments; this can include links to external distributor documentation. The recommended target time is 4pm UTC on a non-Friday weekday. This means the announcement will be seen morning Pacific, early evening Europe, and late evening Asia. The Fix Lead will remove the Fix Team from the private security repository.
Retrospective
These steps should be completed after the Release Date. The retrospective process should be blameless.
The Fix Lead will send a retrospective of the process to the Gardener mailing list including details on everyone involved, the timeline of the process, links to relevant PRs that introduced the issue, if relevant, and any critiques of the response and release process. The Release Managers and Fix Team are also encouraged to send their own feedback on the process to the Gardener mailing list. Honest critique is the only way we are going to get good at this as a community.
Communication Channel
The private or public disclosure process should be triggered exclusively by writing an e-mail to secure@sap.com.
Gardener security announcements will be communicated by the Fix Lead sending an e-mail to the Gardener mailing list (reachable via gardener@googlegroups.com), as well as posting a link in the Gardener Slack channel.
Public discussions about Gardener security announcements and retrospectives will primarily happen in the Gardener mailing list. Thus Gardener community members who are interested in participating in discussions related to the Gardener Security Release Process are encouraged to join the Gardener mailing list (how to find and join a group).
The members of the Gardener Security Team are subscribed to the private Gardener Security mailing list (reachable via gardener-security@googlegroups.com).