Вопрос/проблема:

Под создается успешно несмотря на то,что для всех service account для namespace feature-1 и namespace default действует
PodSecurityPolicy с именем default, в которой запрещено создавать привиллегированные поды

# kubectl --as=system:serviceaccount:default:myuser -n feature-1 create -f- <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: privileged
spec:
  containers:
    - name: pause
      image: k8s.gcr.io/pause
      securityContext:
        privileged: true
EOF


Аналогично, запрещено создавать поды в namespace feature-1 с
 - hostNetwork: true
 - hostPID: true
 - hostPath отличным от "/opt/k8svolumes"

Но под успешно создается
# kubectl --as=system:serviceaccount:default:myuser -n feature-1 create -f- <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: hackers-pod
spec:
  containers:
  - command: ["/bin/bash", "-c", "sleep 100000"]
    image: debian
    name: hackers-pod
    volumeMounts:
    - mountPath: /host
      name: host
  hostNetwork: true
  hostPID: true
  tolerations:
  - effect: NoSchedule
    operator: Exists
  nodeSelector:
    node-role.kubernetes.io/master: ""
  volumes:
  - hostPath:
      path: /
      type: Directory
    name: host
EOF



# kubectl get sa -n default
NAME            SECRETS   AGE
default         1         101d
myuser     1         60d

# kubectl get sa -n feature-1
NAME      SECRETS   AGE
default   1         62d

Видно, что несмотря на то, что под запускается с service account default и как для default-аккаунта, таки и для аккаунта myuser
должна использоваться psp с именем default, которая запрещает запуск подов
Однако почему-то используется psp с именем system, которая ожидаемо разрешает запуск привиллегироваонного пода

# kubectl get pod privileged -o yaml | grep -Ew 'serviceAccount|serviceAccountName|psp'
    kubernetes.io/psp: system
  serviceAccount: default
  serviceAccountName: default

# kubectl get pod hackers-pod -o yaml | grep -Ew 'serviceAccount|serviceAccountName|psp'
    kubernetes.io/psp: system
  serviceAccount: default
  serviceAccountName: default

Тут ожидаемо service account myuser, созданный в namespace default имеет доступ к psp default, для которой создана rolebinding  в namespace-ах default и feature-1
# kubectl --as=system:serviceaccount:default:myuser auth can-i use podsecuritypolicy/default  2>/dev/null
yes
# kubectl --kubeconfig ~/.kube/myuser  auth can-i use podsecuritypolicy/default  2>/dev/null
yes


Тут непонятно почему service account который сушествует в namespace default имеет доступ к psp, для которых созданы rolebinding  в namespace-ах, в которых пользователь не имеет доступ (kube-system,lens-metrics,ingress-nginx)

# kubectl --as=system:serviceaccount:default:myuser auth can-i use podsecuritypolicy/system  2>/dev/null
yes
# kubectl --kubeconfig ~/.kube/myuser  auth can-i use podsecuritypolicy/system  2>/dev/null
no

# kubectl --as=system:serviceaccount:default:myuser auth can-i use podsecuritypolicy/default-hostnetwork  2>/dev/null
yes
# kubectl --kubeconfig ~/.kube/myuser auth can-i use podsecuritypolicy/default-hostnetwork  2>/dev/null
no


# kubectl --as=system:serviceaccount:default:myuser auth can-i use podsecuritypolicy/default-monitoring  2>/dev/null
yes
# kubectl --kubeconfig ~/.kube/myuser  auth can-i use podsecuritypolicy/default-monitoring  2>/dev/null
no



Подготовительные действия:

Addmission controller в настройках kube-api-севрера включен на всех master-серверах
# grep PodSecurityPolicy /etc/kubernetes/manifests/kube-apiserver.yaml
    - --enable-admission-plugins=NodeRestriction,PodSecurityPolicy

Поды с kube-api перезапущены автоматически после добавления PodSecurityPolicy в конфиг kube-api(/etc/kubernetes/manifests/kube-apiserver.yaml)
После перезапуска в логах видно, что включен PodSecurityPolicy
# kubectl logs kube-apiserver-XXX -n kube-system | head -n 7  | grep PodSecurityPolicy
I1011 22:08:34.851037       1 plugins.go:158] Loaded 13 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,LimitRanger,ServiceAccount,NodeRestriction,TaintNodesByCondition,PodSecurityPolicy,Priority,DefaultTolerationSeconds,DefaultStorageClass,StorageObjectInUseProtection,RuntimeClass,DefaultIngressClass,MutatingAdmissionWebhook.
I1011 22:08:34.851068       1 plugins.go:161] Loaded 11 validating admission controller(s) successfully in the following order: LimitRanger,ServiceAccount,PodSecurityPolicy,Priority,PersistentVolumeClaimResize,RuntimeClass,CertificateApproval,CertificateSigning,CertificateSubjectRestriction,ValidatingAdmissionWebhook,ResourceQuota.



Для всех system:authenticated во всех namespace-ах использовать PSP default (реализовано через ClusterRoleBinding psp-default)

Для всех service account-ов через указание группы system:serviceaccounts для namespace feature-1 использовать PSP default (реализовано через RoleBinding psp-default-feature-1)

Для всех service account-ов через указание группы system:serviceaccounts для namespace default использовать PSP default (реализовано через RoleBinding psp-default-default)

Для всех system:serviceaccounts|system:authenticated для namespace kube-system использовать PSP system (реализовано через RoleBinding psp-system)

Для всех system:serviceaccounts|system:authenticated для namespace ingress-nginx использовать PSP psp-default-hostnetwork (реализовано через RoleBinding psp-default-hostnetwork-ingress-nginx)

Для всех system:serviceaccounts для namespace lens-metrics использовать PSP psp-default-monitoring (реализовано через RoleBinding psp-default-monitoring-lens-metrics)


Проверка созданных необходимых объектов:

# kubectl get psp
NAME                  PRIV    CAPS   SELINUX    RUNASUSER   FSGROUP    SUPGROUP   READONLYROOTFS   VOLUMES
default               false          RunAsAny   RunAsAny    RunAsAny   RunAsAny   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath
default-hostnetwork   false          RunAsAny   RunAsAny    RunAsAny   RunAsAny   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim
default-monitoring    false          RunAsAny   RunAsAny    RunAsAny   RunAsAny   false            configMap,emptyDir,projected,secret,downwardAPI,persistentVolumeClaim,hostPath
system                true    *      RunAsAny   RunAsAny    RunAsAny   RunAsAny   false            *

# kubectl get clusterrole | grep psp
psp-default                                                            2021-10-11T16:47:06Z
psp-default-hostnetwork                                                2021-10-11T16:33:48Z
psp-default-monitoring                                                 2021-10-11T19:58:20Z
psp-system                                                             2021-10-11T16:41:13Z

# kubectl get clusterrolebindings.rbac.authorization.k8s.io | grep psp
psp-default   ClusterRole/psp-default   17h

# kubectl get rolebindings.rbac.authorization.k8s.io -A | grep psp
feature-1              psp-default-feature-1                               ClusterRole/psp-default                               17h
ingress-nginx          psp-default-hostnetwork-ingress-nginx               ClusterRole/psp-default-hostnetwork                   14h
kube-system            psp-system                                          ClusterRole/psp-system                                17h
lens-metrics           psp-default-monitoring-lens-metrics                 ClusterRole/psp-default-monitoring                    17h
default                psp-default-default                                 ClusterRole/psp-default                               17h




Использовались следующие конфигурации для создания объектов psp,clusterrole,clusterrolebinding,rolebinding

### Default

---
kind: PodSecurityPolicy
apiVersion: policy/v1beta1
metadata:
  name: default
spec:
  privileged: false
  hostNetwork: false
  hostIPC: false
  hostPID: false
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'downwardAPI'
    - 'persistentVolumeClaim'
    - 'hostPath'
  allowedHostPaths:
    - pathPrefix: "/opt/k8svolumes"
      readOnly: false
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp-default
rules:
  - apiGroups:
      - policy
    resources:
      - podsecuritypolicies
    verbs:
      - use
    resourceNames:
      - default
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp-default-feature-1
  namespace: feature-1
roleRef:
  kind: ClusterRole
  name: psp-default
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: Group
    name: system:serviceaccounts
    apiGroup: rbac.authorization.k8s.io
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp-default-default
  namespace: default
roleRef:
  kind: ClusterRole
  name: psp-default
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: Group
    name: system:serviceaccounts
    apiGroup: rbac.authorization.k8s.io


---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp-default
roleRef:
  kind: ClusterRole
  name: psp-default
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:authenticated



### System

---
kind: PodSecurityPolicy
apiVersion: policy/v1beta1
metadata:
  name: system
spec:
  privileged: true
  allowPrivilegeEscalation: true
  hostNetwork: true
  hostPID: true
  hostIPC: true
  hostPorts:
    - min: 0
      max: 65535
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
    - '*'
  allowedCapabilities:
    - '*'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp-system
rules:
  - apiGroups:
      - policy
    resources:
      - podsecuritypolicies
    verbs:
      - use
    resourceNames:
      - system
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp-system
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: psp-system
  apiGroup: rbac.authorization.k8s.io
subjects:
# Authorize all authenticated users in a namespace:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:authenticated
# Authorize all service accounts in a namespace:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccounts


### Monitroring

---
kind: PodSecurityPolicy
apiVersion: policy/v1beta1
metadata:
  name: default-monitoring
spec:
  privileged: false
  hostNetwork: false
  hostPID: true
  hostIPC: false
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'downwardAPI'
    - 'persistentVolumeClaim'
    - 'hostPath'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp-default-monitoring
rules:
  - apiGroups:
      - policy
    resources:
      - podsecuritypolicies
    verbs:
      - use
    resourceNames:
      - default-monitoring
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp-default-monitoring-lens-metrics
  namespace: lens-metrics
roleRef:
  kind: ClusterRole
  name: psp-default-monitoring
  apiGroup: rbac.authorization.k8s.io
subjects:
  # Authorize all service accounts in a namespace:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccounts


### Hostnetwork

---
kind: PodSecurityPolicy
apiVersion: policy/v1beta1
metadata:
  name: default-hostnetwork
spec:
  privileged: false
  hostNetwork: true
  hostIPC: false
  hostPID: false
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  fsGroup:
    rule: RunAsAny
  volumes:
    - 'configMap'
    - 'emptyDir'
    - 'projected'
    - 'secret'
    - 'downwardAPI'
    - 'persistentVolumeClaim'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp-default-hostnetwork
rules:
  - apiGroups:
      - policy
    resources:
      - podsecuritypolicies
    verbs:
      - use
    resourceNames:
      - default-hostnetwork
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: psp-default-hostnetwork-ingress-nginx
  namespace: ingress-nginx
roleRef:
  kind: ClusterRole
  name: psp-default-hostnetwork
  apiGroup: rbac.authorization.k8s.io
subjects:
  # Authorize all service accounts in a namespace:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccounts
  # Or equivalently, all authenticated users in a namespace:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:authenticated