Network Policies📜
To increase the overall security posture of Big Bang, network policies are put in place to only allow ingress and egress from package namespaces to other needed services. A deny by default policy is put in place to deny all traffic that is not explicitly allowed. The following is how to implement the network policies per Big Bang standards.
Table of Contents📜
Prerequisites📜
- Understanding of ports and communications of applications and other components within BigBang
chart/templates/bigbang
andchart/templates/bigbang/networkpolicies
folders within package for committing bigbang specific templates.
Integration📜
All examples in this documentation will center on podinfo.
Default Deny📜
In order to keep Big Bang secure, a default deny policy must be put into place for each package. Create default-deny-all.yaml
inside chart/templates/bigbang/networkpolicies
with the following details:
{{ if .Values.networkPolicies.enabled }}
# Default deny everything to/from this namespace
kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
name: default-deny-all
namespace: {{ .Release.Namespace }}
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
egress: []
ingress: []
{{- end }}
Default Allow📜
For packages with more than one pod/deployment and those pods/deployments need to talk to each other, add a policy that allows all ingress/egress between pods in the namespace. Create default-allow-ns.yaml
inside chart/templates/bigbang/networkpolicies
with the following details:
{{- if .Values.networkPolicies.enabled }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-allow-ns
namespace: {{ .Release.Namespace }}
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector: {}
egress:
- to:
- podSelector: {}
{{- end }}
Was Something Important Blocked?📜
There are a few ways to determine if a network policy is blocking egress or ingress to or from a pod.
* Test things from the pod’s perspective using ssh/exec. See this portion of the keycloak quickstart for an example of how do to that.
* Curl a pod’s IP from another pod to see if network polices are blocking that traffic. Use kubectl pod -o wide -n <podNamespace>
to see pod IP addresses.
* Check the pod logs (or curl from one container to the service) for a context deadline exceeded
or connection refused
message.
Allowing Exceptions📜
- Egress exceptions to consider:
- pod to pod
- SSO
- When available, use a value from the helm values for the port
- Otherwise, use the SSO default and allow egress to all IPs, except the cloud metadata IP. The default port should be 443.
- storage database
- When available, use a value from the helm values for the port
- Otherwise, use the database default and allow egress to all IPs, except the cloud metadata IP.
- Istiod for istio-proxy sidecars
- Ingress exceptions to consider:
- Kube-api
- Prometheus
- Istio for virtual service
- web endpoints
- Once you have determined an exception needs to be made, create a template in
chart/templates/bigbang/networkpolicies
. - NetworkPolicy templates follow the naming convention of
direction-destination.yaml
(eg: egress-dns.yaml). - Each networkPolicy template in the package will have an if statement checking for
networkPolicies.enabled
and will only be present whenenabled: true
For example, if the podinfo package needs to send information to istiod, add the following content to a file named egress-istio-d.yaml
:
{{- if and .Values.networkPolicies.enabled .Values.istio.enabled }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: egress-istiod
namespace: {{ .Release.Namespace }}
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
app.kubernetes.io/name: istio-controlplane
podSelector:
matchLabels:
app: istiod
ports:
- port: 15012
{{- end }}
Similarly, if prometheus needs access to podinfo to scrape metrics, create an ingress-monitoring-prometheus.yaml
file with the following contents:
{{- if and .Values.networkPolicies.enabled .Values.monitoring.enabled }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: ingress-monitoring-prometheus
namespace: {{ .Release.Namespace }}
spec:
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
app.kubernetes.io/name: monitoring
podSelector:
matchLabels:
app: prometheus
ports:
# Port numbers will vary, dependent on the pod
- port: 9797
podSelector:
matchLabels:
app.kubernetes.io/name: podinfo
{{- end }}
Additional Configuration📜
Sample chart/values.yaml
code at the package level:
# BigBang specific Network Policy Configuration
networkPolicies:
enabled: false
# See `kubectl cluster-info` and then resolve to IP
controlPlaneCidr: 0.0.0.0/0
ingressLabels:
app: istio-ingressgateway
istio: ingressgateway
- Use the
enabled: false
code above in order to disable networkPolicy templates for the package. The networkPolicy templates will be enabled by default when deployed from BigBang because it will inherit thenetworkPolicies.enabled
value. - The ingressLabels portion supports packages that have an externally accessible UIs. Values from BigBang will also be inherited in this portion to ensure traffic from the correct istio ingressgateway is whitelisted.
Example of a BigBang value configuration, bigbang/templates/podinfo/values.yaml
, when adding a package into BigBang with networkPolicies:
networkPolicies:
enabled: {{ .Values.networkPolicies.enabled }}
ingressLabels:
{{- $gateway := default "public" .Values.addons.podinfo.ingress.gateway }}
{{- $default := dict "app" (dig "gateways" $gateway "ingressGateway" nil .Values.istio) "istio" nil }}
{{- toYaml (dig "values" "gateways" $gateway "selector" $default .Values.istio) | nindent 4 }}
controlPlaneCidr: {{ .Values.networkPolicies.controlPlaneCidr }}
- If the package needs to talk to the kube-api service (eg: operators) then the
controlPlaneCidr
value will be required. - The
controlPlaneCidr
will control egress to the kube-api and be wide open by default, but will inherit thenetworkPolicies.controlPlaneCidr
value from BigBang so the range can be locked down.
Sample chart/templates/bigbang/networkpolicies/egress-kube-api.yaml
:
{{- if .Values.networkPolicies.enabled }}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: egress-kube-api
namespace: {{ .Release.Namespace }}
spec:
podSelector: {}
egress:
- to:
- ipBlock:
cidr: {{ .Values.networkPolicies.controlPlaneCidr }}
{{- if eq .Values.networkPolicies.controlPlaneCidr "0.0.0.0/0" }}
# ONLY Block requests to cloud metadata IP
except:
- 169.254.169.254/32
{{- end }}
policyTypes:
- Egress
{{- end }}
- The networkPolicy template for kube-api egress will look like the above, so that communication to the AWS Instance Metadata and Azure Instance Metadata can be limited unless required by the package.
Supporting additional network policies through values.yaml📜
All bigbang core and supported addon packages are expected to provide support for the deployment of additional network policies through the values yaml as per the user guide. There is a standard mechanism for the implementation of this pattern, with two use cases:
- where a package will only be deployed into its own namespace (the majority of bigbang packages)
- where a package may be used in inside another package’s namespace or deployed into its own namespace (such as the gitlab-runner)
Single namespace📜
For this use case, a simple iteration over the values is sufficient to create the needed functionality. The standard pattern is to place this into <package>/chart/templates/bigbang/networkpolicies/additional-networkpolicies.yaml
:
{{- if .Values.networkPolicies.enabled }}
{{- range $policy := .Values.networkPolicies.additionalPolicies -}}
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: {{ $policy.name }}
spec:
{{ tpl ($policy.spec | toYaml) $ | nindent 2 }}
---
{{- end }}
{{- end }}
Multiple namespaces📜
For this use case, refer to the gitlab runner implementation. In this pattern, a given chart may be deployed into one or more namespaces. However, you may only want to enable to control of additional network policies in a certain subset of those namespaces. In these cases, it is sufficient to extend the conditional at the top that checks for the flag in the values:
{{- if and .Values.networkPolicies.enabled (ne .Release.Namespace "NAMESPACE-YOU-DONT-WANT-TO-DO-THIS-IN") }}
Validation📜
- Package functions as expected and is able to communicate with all BigBang touchpoints.