# KubeArmor Release Notes v0.7

### ARM support for KubeArmor

<img src="https://user-images.githubusercontent.com/9133227/201508738-31ba6324-9d01-4425-8f02-c6f9c16c30dd.png" width="256" class="center" alt="ARM KubeArmor support">


KubeArmor is increasingly deployed in [near-edge and far-edge scenarios](https://tech.ginkos.in/2019/06/far-edge-vs-near-edge-in-edge-computing.html) for enhanching security for IoT/Edge aspects. KubeArmor has added support to handle the characteristics of edge scenarios such as:
1. Support for heterogeneous platforms (most of the edge is deployed on ARM, Intel). KubeArmor ensures that the same policy enforcement techniques are delivered across different linux kernel versions. It does this by abstracting the policy layer from the enforcement layer and the enforcement layer uses the techniques that are available in the context of the deployed worker node.
2. Edge devices are constrained in terms of CPU and memory availability. The overall resource usage on KubeArmor has been drastically reduced. This has not only helped edge scenarios but cloud based deployments as well.
3. KubeArmor supports the philosophy of "Do not assume always-on connectivity with the cloud platform".

Recently KubeArmor finished its [integration with LF Edge Open Horizon](https://open-horizon.github.io/docs/kubearmor-integration/docs/). KubeArmor is also now listed on [Intel SmartEdge Marketplace](https://networkbuilders.intel.com/commercial-applications/accuknox). LF Edge Open Horizon and Intel® Smart Edge are edge computing platforms for deploying edge networks and delivering multi-access edge computing (MEC) for applications, containers, k8s, and virtual machines.

### Which ARM platforms were tested?
1. Raspberry Pi (RPi) - Both as a systemd service and on k8s
2. Azure ARM VM
3. Apple Macbook M1 laptops

## Network Policy & Microsegmentation support using KubeArmor

In Kubernetes, the [network policy resource](https://kubernetes.io/docs/concepts/services-networking/network-policies/) is a set of network traffic rules that are applied to a group of pods in a Kubernetes cluster. The network policy specifies how a pod is allowed to communicate with others. Network policy controllers (running as pods in the Kubernetes cluster) convert the requirements and restrictions of the network policies that are retrieved from the Kubernetes API into the network infrastructure.

KubeArmor has visibility into the network connections made into or from the pods. It has visibility across `socket()`, `bind()`, `connect()`, `accept()` calls. KubeArmor along with the [discovery-engine](https://github.com/kubearmor/discovery-engine/) could now auto-generate [kubernetes network policy](https://kubernetes.io/docs/concepts/services-networking/network-policies/) which are enforceable using most of the k8s CNIs (Container Network Interface). Note that the discovered policies use k8s abstractions for identifying services, deployments, and pods such as k8s labels, namespaces.

This release allows kubearmor and discovery-engine to do automatic network microsegmentation by detecting the network connections and creating appropriate network policies based on it. These auto-generated network policies are enforceable using any CNI that supports k8s network policy namely, Cilium, Calico, AWS VPC CNI, Weave.

<img src="https://user-images.githubusercontent.com/9133227/201518064-f52b897e-3c60-4440-88ba-bc3cd1c0f147.png" width="512" class="center" alt="K8s network policy rule support using kubearmor">

## Container based policy support

KubeArmor currently applies policy at the pod level i.e. in the `KubeArmorPolicy` you specify the selector labels which selects a set of pods across the cluster. However, a single pod may contain multiple containers and each container usually has a very different purpose and the set of binaries or file paths may vary across multiple containers within the pod.

This release adds support to apply the policies at specific container level by extending the selector label policy construct. Following is an instantiation of this construct:

```yaml
apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: block-ls-at-container-1
  namespace: wordpress-mysql
spec:
  severity: 2
  selector:
    matchLabels:
      app: wordpress
      kubearmor.io/container.name: "[container-1]"
  process:
    matchPaths:
    - path: /bin/ls
  action:
    Block
```

Note the use of `kubearmor.io/container.name: "[container-1]"` construct as part of the selector labels that allows KubeArmor to select specific container(s) within the pod.

<img src="https://user-images.githubusercontent.com/9133227/201514997-c73c59c7-af47-45b5-9e62-a00f7b2ea704.png" width="512" class="center" alt="Container specific policies">

### Need for this support

The file system paths and binaries present in different containers might be completely different. Without this feature, the block of a process operation in one container would also be blocked in other containers. The [discovery-engine](https://github.com/kubearmor/discovery-engine) automatically discovers the security posture of the k8s application discovers what operations should be allowed. These operations are container specific and should be restricted to specific containers. Thus without the ability to specify container specific rules, there are chances that either the security posture is too generic (in the case of allow based policies) or too restrictive (in the case of block based policies).

### Backward compatibility

Specifying container specific labels are optional i.e., if the container information is not specified then the previous way of applying the policies at pod level is retained.

# Using un-privileged container for KubeArmor daemonset (as a part of LFX Mentorship)
KubeArmor is a security policy engine and it needs to be ensured that the engine itself follows all the right security practices.
Privileged containers are usually frowned upon. Almost every static scanning engine will flag this as an issue. In lot of cases, organizations deploy admission controllers that would not allow containers to be installed in privileged mode.

This release achieves following in the context:
1. Removes the use of `privilege: true` flag in the context of kubearmor and associated accessory pods.
2. Enables individual capabilities in place of enabling all capabilities for the KubeArmor containers.
3. Allows KubeArmor to be installed in non-kube-system namespace.

Kudos to [Anurag Kumar](https://github.com/kranurag7) who achieved this as part [LFX Mentorship Project](https://mentorship.lfx.linuxfoundation.org/project/3cc962b4-cd8b-46ea-9c77-83304145fd51).

## Workload behavior summary with KubeArmor and Discovery Engine

KubeArmor can observe system events at different dimensions including, Process, File, and Network events. Discovery Engine connects to KubeArmor and does aggregation of this information over a time period to provide app behavior summary. The use-cases for this summary view are:
1. User can view what are the different process getting spawned in the given workload. Workload could be filtered based on namespace, labels etc.
2. User can view what file system paths are accessed. For e.g., user can check which processes within the workload are accessing service account token.
3. User can check what network connections are handled by the workload. Note that the ingress/egress connections are pinned down to the final process which handles it.

![karmor-summary](https://user-images.githubusercontent.com/9133227/201572596-0de140c6-7ea2-468c-b182-9387c037bae3.gif)


```
❯ karmor summary -n wordpress-mysql -l app=wordpress --agg
local port to be used for port forwarding knoxautopolicy-8587dfd464-28dpq: 9089

  Pod Name        wordpress-bf95888cb-rkcwm
  Namespace Name  wordpress-mysql
  Cluster Name    default
  Container Name  wordpress
  Labels          app=wordpress

Process Data
+------------------------------------------------------------------------------------------------------------------------+-----------------------------------+-------+------------------------------+--------+
|                                                      SRC PROCESS                                                       |     DESTINATION PROCESS PATH      | COUNT |      LAST UPDATED TIME       | STATUS |
+------------------------------------------------------------------------------------------------------------------------+-----------------------------------+-------+------------------------------+--------+
| /bin/bash                                                                                                              | /bin/mkdir                        | 1     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /bin/bash                                                                                                              | /bin/sed                          | 19    | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /bin/bash                                                                                                              | /usr/bin/cut                      | 3     | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /bin/bash                                                                                                              | /usr/bin/dirname                  | 4     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /bin/bash                                                                                                              | /usr/bin/head                     | 3     | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /bin/bash                                                                                                              | /usr/bin/sha1sum                  | 3     | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /bin/bash                                                                                                              | /usr/local/bin/php                | 5     | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /var/lib/rancher/k3s/data/8307e9b398a0ee686ec38e18339d1464f75158a8b948b059b564246f4af3a0a6/bin/containerd-shim-runc-v2 | /usr/local/bin/apache2-foreground | 1     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /var/lib/rancher/k3s/data/8307e9b398a0ee686ec38e18339d1464f75158a8b948b059b564246f4af3a0a6/bin/containerd-shim-runc-v2 | /usr/sbin/apache2                 | 1     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
+------------------------------------------------------------------------------------------------------------------------+-----------------------------------+-------+------------------------------+--------+


File Data
+-----------------------------------+--------------------------------------------------------------------+-------+------------------------------+--------+
|            SRC PROCESS            |                       DESTINATION FILE PATH                        | COUNT |      LAST UPDATED TIME       | STATUS |
+-----------------------------------+--------------------------------------------------------------------+-------+------------------------------+--------+
| /bin/mkdir                        | /etc/                                                              | 1     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /bin/mkdir                        | /lib/x86_64-linux-gnu/                                             | 4     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /bin/rm                           | /lib/x86_64-linux-gnu/                                             | 1     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /bin/sed                          | /etc/                                                              | 10    | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /bin/sed                          | /lib/x86_64-linux-gnu/                                             | 25    | Mon Nov 14 03:49:35 UTC 2022 | Allow  |
| /usr/bin/dirname                  | /etc/                                                              | 2     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /usr/bin/dirname                  | /lib/x86_64-linux-gnu/                                             | 2     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /usr/bin/head                     | /dev/urandom                                                       | 1     | Mon Nov 14 03:49:34 UTC 2022 | Allow  |
| /usr/bin/head                     | /etc/                                                              | 1     | Mon Nov 14 03:49:35 UTC 2022 | Allow  |
| /usr/bin/sha1sum                  | /etc/                                                              | 2     | Mon Nov 14 03:49:34 UTC 2022 | Allow  |
| /usr/bin/sha1sum                  | /lib/x86_64-linux-gnu/                                             | 1     | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /usr/local/bin/apache2-foreground | /dev/tty                                                           | 1     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /usr/local/bin/apache2-foreground | /lib/x86_64-linux-gnu/                                             | 4     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /usr/local/bin/php                | /dev/urandom                                                       | 3     | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /usr/local/bin/php                | /etc/                                                              | 11    | Mon Nov 14 03:49:35 UTC 2022 | Allow  |
| /usr/local/bin/php                | /lib/x86_64-linux-gnu/                                             | 31    | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /usr/local/bin/php                | /tmp/                                                              | 4     | Mon Nov 14 03:49:35 UTC 2022 | Allow  |
| /usr/local/bin/php                | /usr/lib/ssl/openssl.cnf                                           | 3     | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /usr/local/bin/php                | /usr/lib/x86_64-linux-gnu/                                         | 29    | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /usr/local/bin/php                | /usr/local/etc/php/conf.d                                          | 2     | Mon Nov 14 03:49:34 UTC 2022 | Allow  |
| /usr/local/bin/php                | /usr/local/etc/php/conf.d/                                         | 10    | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /usr/local/bin/php                | /usr/local/lib/php/extensions/no-debug-non-zts-20131226/gd.so      | 2     | Mon Nov 14 03:49:35 UTC 2022 | Allow  |
| /usr/local/bin/php                | /usr/local/lib/php/extensions/no-debug-non-zts-20131226/mysqli.so  | 3     | Mon Nov 14 03:49:35 UTC 2022 | Allow  |
| /usr/local/bin/php                | /usr/local/lib/php/extensions/no-debug-non-zts-20131226/opcache.so | 3     | Mon Nov 14 03:49:36 UTC 2022 | Allow  |
| /usr/sbin/apache2                 | /dev/urandom                                                       | 1     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /usr/sbin/apache2                 | /etc/                                                              | 8     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /usr/sbin/apache2                 | /lib/x86_64-linux-gnu/                                             | 4     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
| /usr/sbin/apache2                 | /usr/lib/ssl/openssl.cnf                                           | 1     | Mon Nov 14 03:49:44 UTC 2022 | Allow  |
+-----------------------------------+--------------------------------------------------------------------+-------+------------------------------+--------+


Egress connections
+----------+--------------------+------------+------+-----------------+-----------+-------+------------------------------+
| PROTOCOL |      COMMAND       | POD/SVC/IP | PORT |    NAMESPACE    |  LABELS   | COUNT |      LAST UPDATED TIME       |
+----------+--------------------+------------+------+-----------------+-----------+-------+------------------------------+
| TCP      | /usr/local/bin/php | svc/mysql  | 3306 | wordpress-mysql | app=mysql | 1     | Mon Nov 14 03:49:44 UTC 2022 |
+----------+--------------------+------------+------+-----------------+-----------+-------+------------------------------+

```

## Policy Recommendations and Reports
KubeArmor already had a community driven curated list of System and Network policy templates at [policy-template](https://github.com/kubearmor/policy-templates) repository.
With the templates, it was upto the user to change values like `namespace` `labels` etc to make sure that the policies are actually enforcing on their cluster.

With the new `karmor recommend` it is made sure that the user doesn't have to change anything on the policy but rather simply apply them to get a secure environment for the Kubernetes deployments.

![karmor-recommend](https://user-images.githubusercontent.com/9133227/201516809-6c664041-f5f9-4628-be1b-12252a1155c7.gif)

`karmor recommend` recommends policies based on container image, k8s manifest or the actual runtime environment itself.

```sh
$ karmor recommend --help
Recommend policies based on container image, k8s manifest or the actual runtime env

Usage:
  karmor recommend [flags]
  karmor recommend [command]

Available Commands:
  update      Updates policy-template cache

Flags:
  -c, --config string      absolute path to image registry configuration file
  -h, --help               help for recommend
  -i, --image strings      Container image list (comma separated)
  -l, --labels strings     User defined labels for policy (comma separated)
  -n, --namespace string   User defined namespace value for policies
  -o, --outdir string      output folder to write policies (default "out")
  -r, --report string      report file (default "report.txt")
  -t, --tag strings        tags (comma-separated) to apply. Eg. PCI-DSS, MITRE

Global Flags:
      --context string      Name of the kubeconfig context to use
      --kubeconfig string   Path to the kubeconfig file to use

Use "karmor recommend [command] --help" for more information about a command.
```

## Bug fixes and improvements
- Ability to specific LSM order ([#788](https://github.com/kubearmor/KubeArmor/issues/788)). On certain platforms it is possible that multiple LSMs are available for enforcing a policy. For e.g., in case of Google COS (Container Optimized OS), both AppArmor and BPF-LSM are available. By default, KubeArmor chooses BPF-LSM if available, but with this change it is possible to specify the LSM order that could be used for policy enforcement.
- Support for readOnly flag with BPF-LSM ([#962](https://github.com/kubearmor/KubeArmor/pull/962))
- Fix AppArmor policies behavior ([#954](https://github.com/kubearmor/KubeArmor/issues/954))
- Discovery engine fixes
  - Relative path are not shown anymore
  - Improved `karmor summary` output
  - Improved discovered policies rules to include all necessary paths
