Add validating admission webhook to Helm chart #75
Conversation
This change adds a validating admission webhook that validates the driver-specific opaque parameters that can be specified in ResourceClaims and ResourceClaimTemplates.
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: nojnhuh The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
cncf-cla: yes
k8s-ci-robot
added
the
size/XL
| - name: Test | ||
| run: go test -v -race ./... |
There was a problem hiding this comment.
I also made this particular change in #70, but added that again here to run the new tests added by this PR in CI.
| @@ -71,6 +71,22 @@ kube-system kube-scheduler-dra-example-driver-cluster-control-plane | |||
| local-path-storage local-path-provisioner-7dbf974f64-9jmc7 1/1 Running 0 1m | |||
| ``` | |||
|
|
|||
| When the validating admission webhook is enabled (as it is by default), cert-manager and its CRDs must be installed. | |||
There was a problem hiding this comment.
Instead of taking on a cert-manager dependency, I experimented with Helm's builtin in genSignedCert and adjacent template functions. The main functional caveat with those is that every helm upgrade will generate a new cert, but if the Deployment doesn't change, then a new Pod will not be created to pick up to that new cert, leading to connection issues until a new Pod is created. That might be acceptable for what this example driver will be used for, but probably isn't a pattern we want to establish for "real" drivers.
| limitations under the License. | ||
| */ | ||
|
|
||
| package main |
There was a problem hiding this comment.
For context, this webhook implementation is largely based on the agnhost one.
| gpuConfig, ok := decodedConfig.(*configapi.GpuConfig) | ||
| if !ok { | ||
| errs = append(errs, fmt.Errorf("expected v1alpha1.GpuConfig at %s but got: %T", fieldPath, decodedConfig)) | ||
| continue | ||
| } | ||
| err = gpuConfig.Validate() | ||
| if err != nil { | ||
| errs = append(errs, fmt.Errorf("object at %s is invalid: %w", fieldPath, err)) | ||
| } |
There was a problem hiding this comment.
Since this is about the extent of the driver-specific logic, I wonder if we could eventually make most of this a library in https://github.com/kubernetes/dynamic-resource-allocation where driver authors would only have to provide this custom type, a validation function, and an invocation from a main package. Getting some mileage on this implementation though seems wise before making it too easy :)
| @@ -35,3 +35,4 @@ LABEL summary="Example DRA resource driver for Kubernetes" | |||
| LABEL description="See summary" | |||
|
|
|||
| COPY --from=build /artifacts/dra-example-kubeletplugin /usr/bin/dra-example-kubeletplugin | |||
| COPY --from=build /artifacts/dra-example-webhook /usr/bin/dra-example-webhook | |||
There was a problem hiding this comment.
I know it would be generally more desirable to have separate container images for the kubelet plugin and the webhook, but I wasn't 100% sure that's desired here based on the above make cmds invocation building all of the packages under ./cmd. It certainly does simplify some things in the Helm chart if there's only one image in play.
| {{- define "dra-example-driver.webhookServiceAccountName" -}} | ||
| {{- $name := printf "%s-webhook-service-account" (include "dra-example-driver.fullname" .) }} | ||
| {{- if .Values.webhook.serviceAccount.create }} | ||
| {{- default $name .Values.webhook.serviceAccount.name }} | ||
| {{- else }} | ||
| {{- default "default-webhook" .Values.webhook.serviceAccount.name }} | ||
| {{- end }} | ||
| {{- end }} |
There was a problem hiding this comment.
I opted to create a separate ServiceAccount for the webhook since the webhook doesn't require any RBAC permissions. The Helm chart already seemed to be assuming that there will be only one ServiceAccount though, so the values structure is a little clunky where the kubelet plugin's ServiceAccount is rooted at serviceAccount where the webhook's is rooted at webhook.serviceAccount. Happy to make that more consistent how if that's desired.
| @@ -6,10 +6,12 @@ metadata: | |||
| namespace: {{ include "dra-example-driver.namespace" . }} | |||
| labels: | |||
| {{- include "dra-example-driver.labels" . | nindent 4 }} | |||
| app.kubernetes.io/component: kubeletplugin | |||
There was a problem hiding this comment.
This label is used to distinguish the kubeletplugin from the webhook so selectors don't get confused.
| cdiapi "tags.cncf.io/container-device-interface/pkg/cdi" | ||
| cdiparser "tags.cncf.io/container-device-interface/pkg/parser" | ||
| cdispec "tags.cncf.io/container-device-interface/specs-go" | ||
| ) | ||
|
|
||
| const ( | ||
| cdiVendor = "k8s." + DriverName | ||
| cdiVendor = "k8s." + consts.DriverName |
There was a problem hiding this comment.
I'm not generally a fan of consts packages but that seemed at least better than duplicating the variable. Open to suggestions for a better name or a place for this variable where it can be used by two different main packages. This change is in a separate commit from the rest of the webhook changes in case that makes this easier to review at all, but please let me know if I should squash.
This change adds a validating admission webhook that validates the
driver-specific opaque parameters that can be specified in
ResourceClaims and ResourceClaimTemplates.
Fixes #49