K8s开源安全平台kubescape实现Pod的安全合规检查镜像漏洞扫描

　　写在前面生产环境中的 k8s 集群安全不可忽略，即使是内网环境 容器化的应用部署虽然本质上没有变化，始终是机器上的一个进程 但是提高了安全问题的处理的复杂性 分享一个开源的 k8s 集群安全合规检查/漏洞扫描 工具  kubescape 博文内容涉及：  kubescape  简介介绍kubescape  命令行工具安装，扫描运行的集群kubescape  在集群下安装，通过 kubescape Clound 可视化查看扫描信息理解不足小伙伴帮忙指正 需要有科学上网环境
　　对每个人而言，真正的职责只有一个：找到自我。然后在心中坚守其一生，全心全意，永不停息。所有其它的路都是不完整的，是人的逃避方式，是对大众理想的懦弱回归，是随波逐流，是对内心的恐惧 ——赫尔曼·黑塞《德米安》 简单介绍k8s 安全问题不可忽略
　　镜像的不可变性使我们能够方便地部署、测试和发布镜像到其他环境，这是一个很大的优势，但也带来了潜在的风险：镜像及其依赖在过时或者被弃用时，无法自动更新或打新的安全补丁，尤其一些基础镜像，重新做镜像，需要依赖一些厂商，或者开源项目 team 来维护。
　　RedHat 在 2022 年 Kubernetes 安全报告中对 300 多名 DevOps、工程和安全专业人士进行了调查，发现对  容器安全威胁的担忧以及对容器安全投资的缺乏  是 31% 的受访者对容器策略最普遍的担忧。
　　支持这些担忧的是 93% 的受访者在过去 12 个月内在其 Kubernetes 环境中经历过至少一次安全事件，这些事件有时会导致收入或客户流失。超过一半的受访者 (55%) 在过去一年中还因为安全问题而不得不推迟应用程序的推出。
　　尽管媒体广泛关注网络攻击，但该报告强调， 实际上是错误配置让 IT 专业人员彻夜难眠 。Kubernetes 是高度可定制的，具有可以影响应用程序安全状况的各种配置选项。因此，受访者最担心的是容器和 Kubernetes 环境中的错误配置导致的风险暴露 (46%)  —— 几乎是对攻击的担忧程度 (16%) 的三倍。尽可能自动化配置管理有助于缓解这些问题，因此安全工具  —— 而不是人类 —— 提供了帮助开发人员和 DevOps 团队更安全地配置容器和 Kubernetes 的护栏。
　　以上内容来自：Redhat Blog(The State of Kubernetes Security in 2022) Kubescape 简单介绍
　　Kubescape 是一个开源的 Kubernetes 安全平台。它的功能包括  风险分析 、安全合规性  和 错误配置扫描 。针对 DevSecOps 从业者或平台工程师，提供易于使用的 CLI 界面、灵活的输出格式和自动扫描功能。同时对于小集群提供了免费的 在线 面板工具，它为 Kubernetes 用户和管理员节省了宝贵的时间、精力和资源。
　　Kubescape 可以扫描运行的集群、静态 YAML 文件和 本地 Helm Charts。它根据多个框架（包括  NSA-CISA、MITRE ATT&CK®和CIS Benchmark ）检测错误配置。
　　Kubescape  由 ARMO  创建，是一个 Cloud Native Computing Foundation (CNCF) 沙盒项目。
　　Kubescape 如果小伙伴觉得重的话， kubectl 有一个类似的插件，个人十分推荐，叫做  kube-score  ,很轻量，也可以做一些简单的合规性扫描。但是只有扫描合规性的提示，没有规范出处。Kubescape 安装
　　当前的集群信息 ┌──[root@vms100.liruilongs.github.io]-[~] └─$kubectl get  nodes NAME                          STATUS   ROLES           AGE   VERSION vms100.liruilongs.github.io   Ready    control-plane   8d    v1.25.1 vms101.liruilongs.github.io   Ready    control-plane   8d    v1.25.1 vms102.liruilongs.github.io   Ready    control-plane   8d    v1.25.1 vms103.liruilongs.github.io   Ready              8d    v1.25.1 vms105.liruilongs.github.io   Ready              8d    v1.25.1 vms106.liruilongs.github.io   Ready              8d    v1.25.1 vms107.liruilongs.github.io   Ready              8d    v1.25.1 vms108.liruilongs.github.io   Ready              8d    v1.25.1 ┌──[root@vms100.liruilongs.github.io]-[~] └─$
　　集群节点信息 ┌──[root@vms100.liruilongs.github.io]-[~] └─$hostnamectl    Static hostname: vms100.liruilongs.github.io          Icon name: computer-vm            Chassis: vm         Machine ID: e93ae3f6cb354f3ba509eeb73568087e            Boot ID: a1150b6d97dc4afbb81dae58f131a487     Virtualization: vmware   Operating System: CentOS Linux 7 (Core)        CPE OS Name: cpe:/o:centos:centos:7             Kernel: Linux 5.4.230-1.el7.elrepo.x86_64       Architecture: x86-64 ┌──[root@vms100.liruilongs.github.io]-[~] └─$
　　Kubescape 命令行工具安装 Kubescape CLI 安装
　　通过下面的方式自动安装 curl -s https://raw.githubusercontent.com/kubescape/kubescape/master/install.sh | /bin/bash
　　如果没有科学上网，找一台可以访问的集群，下载 install.sh 文件，按照下面方式修改，获取 curl 命令，自行下载。 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$vim install.sh ....... # curl --progress-bar -L $DOWNLOAD_URL -o $OUTPUT  echo ＂curl --progress-bar -L $DOWNLOAD_URL -o $OUTPUT＂ exit 1 ...
　　运行 shell，获取下载命令，找有网的机器下载 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$sh install.sh Installing Kubescape...  curl --progress-bar -L https://github.com/kubescape/kubescape/releases/latest/download/kubescape-ubuntu-latest -o /root/.kubescape/kubescape
　　上传到指定位置 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$ls install.sh  kubescape-ubuntu-latest ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$mv kubescape-ubuntu-latest  /root/.kubescape/kubescape mv：是否覆盖＂/root/.kubescape/kubescape＂？ y ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$
　　修改脚本，再次运行。 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$vim install.sh # exit 1 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$sh install.sh Installing Kubescape...  curl --progress-bar -L https://github.com/kubescape/kubescape/releases/latest/download/kubescape-ubuntu-latest -o /root/.kubescape/kubescape  Finished Installation.  Your current version is: v2.0.183 [git enabled in build: true]  Usage: $ kubescape scan --enable-host-scan
　　到这里 ，kubescape 命令行工具即安装成功，扫描当前运行的集群可以运行如下命令。 ┌──[root@vms100.liruilongs.github.io]-[~/ansible] └─$kubescape scan --enable-host-scan   --format html --output results.html  --verbose [info] Kubescape scanner starting [info] Installing host scanner [info] Downloading/Loading policy definitions [success] Downloaded/Loaded policy [info] Accessing Kubernetes objects [success] Accessed to Kubernetes objects [info] Requesting images vulnerabilities results [success] Requested images vulnerabilities results [info] Requesting Host scanner data [info] Host scanner version : v1.0.39 ◑[error] failed to get data. path: /controlPlaneInfo; podName: host-scanner-xshr6; error: the server could not find the requested resource (get pods http:host-scanner-xshr6:7888) ◒[error] failed to get data. path: /controlPlaneInfo; podName: host-scanner-4tgnp; error: the server could not find the requested resource (get pods http:host-scanner-4tgnp:7888) ....... [success] Done scanning. Cluster: kubernetes-admin-kubernetes  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  ################################################################################ ApiVersion: hostdata.kubescape.cloud/v1beta0 Kind: KubeletInfo Name: vms105.liruilongs.github.io  Controls: 21 (Failed: 6, Excluded: 0)  +----------+--------------------------------+------------------------------------+------------------------+ | SEVERITY |          CONTROL NAME          |                DOCS                | ASSISTANT REMEDIATION  | +----------+--------------------------------+------------------------------------+------------------------+ | High     | CIS-4.1.7 Ensure that the      | https://hub.armosec.io/docs/c-0168 |                        | |          | certificate authorities file   |                                    |                        | |          | permissions are set to 600 or  |                                    |                        | |          | more restrictive               |                                    |                        | +          +--------------------------------+------------------------------------+------------------------+ |          | CIS-4.1.9 If the kubelet       | https://hub.armosec.io/docs/c-0170 |                        | |          | config.yaml configuration      |                                    |                        | |          | file is being used validate    |                                    |                        | |          | permissions set to 600 or more |                                    |                        | |          | restrictive                    |                                    |                        | +----------+--------------------------------+------------------------------------+------------------------+ ........ +----------+--------------------------------+------------------------------------+------------------------+ | Low      | CIS-4.2.6 Ensure that the      | https://hub.armosec.io/docs/c-0177 | protectKernelDefaults  | |          | --protect-kernel-defaults      |                                    |                        | |          | argument is set to true        |                                    |                        | +          +--------------------------------+------------------------------------+------------------------+ |          | CIS-4.2.7 Ensure that the      | https://hub.armosec.io/docs/c-0178 | makeIPTablesUtilChains | |          | --make-iptables-util-chains    |                                    |                        | |          | argument is set to true        |                                    |                        | +----------+--------------------------------+------------------------------------+------------------------+  ################################################################################ ApiVersion: v1 Kind: Namespace Name: kubescape  Controls: 18 (Failed: 17, Excluded: 0)  +----------+--------------------------------+------------------------------------+----------------------------------------------------------------+ | SEVERITY |          CONTROL NAME          |                DOCS                |                     ASSISTANT REMEDIATION                      | +----------+--------------------------------+------------------------------------+----------------------------------------------------------------+ | High     | CIS-5.2.11 Minimize the        | https://hub.armosec.io/docs/c-0202 | metadata.labels[pod-security.kubernetes.io/enforce]=baseline   | |          | admission of Windows           |                                    |                                                                | |          | HostProcess Containers         |                                    |                                                                | +          +--------------------------------+------------------------------------+                                                                + |          | CIS-5.2.2 Minimize the         | https://hub.armosec.io/docs/c-0193 |                                                                | |          | admission of privileged        |                                    |                                                                | |          | containers                     |                                    |                                                                | +----------+--------------------------------+------------------------------------+----------------------------------------------------------------+ | Medium   | CIS-5.2.1 Ensure that          | https://hub.armosec.io/docs/c-0192 | metadata.labels[pod-security.kubernetes.io/enforce]=YOUR_VALUE | |          | the cluster has at least       |                                    |                                                                | |          | one active policy control      |                                    |                                                                | |          | mechanism in place             |                                    |                                                                | +          +--------------------------------+------------------------------------+----------------------------------------------------------------+ ....... .......... ################################################################################ ApiVersion: hostdata.kubescape.cloud/v1beta0 Kind: ControlPlaneInfo Name: vms102.liruilongs.github.io  Controls: 25 (Failed: 1, Excluded: 0)  +----------+--------------------------------+------------------------------------+-----------------------+ | SEVERITY |          CONTROL NAME          |                DOCS                | ASSISTANT REMEDIATION | +----------+--------------------------------+------------------------------------+-----------------------+ | High     | CIS-1.1.20 Ensure that the     | https://hub.armosec.io/docs/c-0111 |                       | |          | Kubernetes PKI certificate     |                                    |                       | |          | file permissions are set to    |                                    |                       | |          | 600 or more restrictive        |                                    |                       | +----------+--------------------------------+------------------------------------+-----------------------+  ################################################################################ ApiVersion: apps/v1 Kind: Deployment Name: local-path-provisioner Namespace: local-path-storage  Controls: 35 (Failed: 18, Excluded: 0) ... ........ ......
　　输出的合规信息和漏洞信息。
　　异常问题
　　我在 kubescape 多次次扫描中，集群因部分节点因为端口问题，无法发生调度，节点上的 kubescape 对于的 pod 无法自行删除，如果上一次扫描的 pod 或则 ns 没有删除，那么下一次的扫描无法进行，之前创建的 pod 和 ns 状态一直为  Terminating , 解决办法需要 对命名空间进行彻底删除。
　　这里执行完命令会进入阻塞状态。 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubevirt] └─$kubectl delete ns kubescape-host-scanner namespace ＂kubescape-host-scanner＂ deleted
　　运行脚本删除 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s_shell_secript] └─$cat delete_namespace.sh #!/bin/bash  coproc kubectl proxy --port=30990 &  if [ $# -eq 0 ] ; then         echo ＂后面加上你所要删除的ns.＂         exit 1 fi  kubectl get namespace $1 -o json > logging.json sed  -i ＂/＂finalizers＂/{n;d}＂ logging.json curl -k -H ＂Content-Type: application/json＂ -X PUT --data-binary @logging.json http://127.0.0.1:30990/api/v1/namespaces/${1}/finalize  kill %1 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/k8s_shell_secript] └─$sh delete_namespace.sh kubescape-host-scanner
　　也可以离线运行 Kubescape ，时间网络原因，这里不做分享，有需要的小伙伴可以到 github 上的项目地址查看详细信息。一些其他的用法
　　扫描正在运行的 Kubernetes 集群： kubescape scan --enable-host-scan  --verbose
　　使用替代的 kubeconfig 文件： kubescape scan --kubeconfig cluster.conf
　　扫描特定的命名空间： kubescape scan --include-namespaces development,staging,production
　　排除某些命名空间： kubescape scan --exclude-namespaces kube-system,kube-public
　　部署前扫描本地 YAML/JSON 文件： kubescape scan *.yaml ┌──[root@vms100.liruilongs.github.io]-[~/ansible/helm] └─$kubescape scan --enable-host-scan  kube-prometheus-stack.yaml  --format html  --output resout.html [info] Kubescape scanner starting [warning] in setContextMetadata. case: git-url; error: repository host ＂gitee.com＂ not supported [info] Downloading/Loading policy definitions [success] Downloaded/Loaded policy [info] Accessing local objects [success] Done accessing local objects [info] Scanning GitLocal [success] Done scanning GitLocal  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  Controls: 55 (Failed: 31, Excluded: 0, Skipped: 0) Failed Resources by Severity: Critical — 0, High — 34, Medium — 151, Low — 24  +----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+ | SEVERITY |                                        CONTROL NAME                                        | FAILED RESOURCES | EXCLUDED RESOURCES | ALL RESOURCES | % RISK-SCORE | +----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+ | High     | Resources memory limit and request                                                         |        7         |         0          |       7       |     100%     | | High     | Resource limits                                                                            |        7         |         0          |       7       |     100%     | | High     | List Kubernetes secrets                                                                    |        4         |         0          |       7       |     57%      | … | Medium   | Allow privilege escalation                                                                 |        6         |         0          |       7       |     86%      | | Medium   | Ingress and Egress blocked                                                                 |        7         |         0          |       7       |     100%     | … | Medium   | CIS-5.4.1 Prefer using secrets as files over secrets as environment variables              |        1         |         0          |       7       |     14%      | | Medium   | CIS-5.7.2 Ensure that the seccomp profile is set to docker/default in your pod definitions |        7         |         0          |       7       |     100%     | | Medium   | CIS-5.7.4 The default namespace should not be used                                         |        56        |         0          |      61       |     92%      | | Low      | Immutable container filesystem                                                             |        6         |         0          |       7       |     86%      | | Low      | Configured readiness probe                                                                 |        5         |         0          |       7       |     71%      | | Low      | Malicious admission controller (validating)                                                |        1         |         0          |       1       |     100%     | | Low      | Pods in default namespace                                                                  |        7         |         0          |       7       |     100%     | | Low      | Naked PODs                                                                                 |        1         |         0          |       1       |     100%     | | Low      | Label usage for resources                                                                  |        3         |         0          |       7       |     43%      | | Low      | K8s common labels usage                                                                    |        1         |         0          |       7       |     14%      | +----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+ |          |                                      RESOURCE SUMMARY                                      |        66        |         0          |      72       |    43.26%    | +----------+--------------------------------------------------------------------------------------------+------------------+--------------------+---------------+--------------+ FRAMEWORKS: ArmoBest (risk: 33.51), cis-v1.23-t1.0.1 (risk: 62.75), DevOpsBest (risk: 61.72), AllControls (risk: 36.46), MITRE (risk: 11.79), NSA (risk: 33.74)  [success] Scan results saved. filename: resout.html  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Scan results have not been submitted: run kubescape with the ＂--account＂ flag For more details: https://hub.armosec.io/docs/installing-kubescape?utm_campaign=Submit&utm_medium=CLI&utm_source=GitHub ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  Run with ＂--verbose＂/＂-v＂ flag for detailed resources view  ┌──[root@vms100.liruilongs.github.io]-[~/ansible/helm] └─$l
　　从 Git 存储库扫描 Kubernetes 清单文件： kubescape scan https://github.com/kubescape/kubescape
　　扫描 Helm 图表,kubescape 将加载默认的 VALUES 文件。 kubescape scan 
　　扫描 Kustomize 目录,Kubescape 将使用 kustomize 文件生成 Kubernetes YAML 对象并扫描它们以确保安全。 kubescape scan 
　　使用 NSA 框架扫描正在运行的 Kubernetes 集群： kubescape scan framework nsa
　　使用 MITRE ATT&CK® 框架扫描正在运行的 Kubernetes 集群： kubescape scan framework mitre
　　使用控件名称或控件 ID 扫描特定控件。请参阅控件列表。 kubescape scan control ＂Privileged container＂ 指定报告输出格式
　　JSON： kubescape scan --format json --format-version v2 --output results.json
　　XML： kubescape scan --format junit --output results.xml
　　PDF: kubescape scan --format pdf --output results.pdf
　　普罗米修斯指标： kubescape scan --format prometheus
　　HTML kubescape scan --format html --output results.html
　　显示所有扫描到的资源（包括通过的资源）： kubescape scan --verbose Kubescape 在集群下安装
　　Kubescape 也可以在集群下安装，通过集群安装，可以在 cloud 里的 UI 界面查看具体扫描信息，集群中安装 Kubescape，先决条件 确保您拥有 Kubescape Cloud 帐户——如果没有，请在此处注册 您需要拥有对集群的安装权限（您应该能够创建 Deployments、CronJobs、ConfigMaps 和 Secrets） 你必须有 Kubectl 和 Helm
　　集群要求 Kubescape 运算符组件至少需要 400Mib RAM 和 400m CPU
　　具体的安装可以参考下的教程： 需要注册 Kubescape Cloud，并且  只有的工作节点小于 10 个的时候才免费 。
　　https://hub.armosec.io/docs/installation-of-armo-in-cluster#install-a-pre-registered-cluster
　　注册登录
　　这里登录完会弹出一个安装部署，安装安装部署安装即可
　　添加 Helm 源，并更新 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$helm repo add kubescape https://kubescape.github.io/helm-charts/ ＂kubescape＂ has been added to your repositories ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the ＂rancher-stable＂ chart repository ...Successfully got an update from the ＂botkube＂ chart repository ...Successfully got an update from the ＂awx-operator＂ chart repository ...Successfully got an update from the ＂kubescape＂ chart repository Update Complete. ⎈Happy Helming!⎈
　　运行 charts ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$helm upgrade --install kubescape kubescape/kubescape-cloud-operator -n kubescape --create-namespace --set account=97f09924-0c06-42e4-bdad-5b333321af77 --set clusterName=`kubectl config current-context` Release ＂kubescape＂ does not exist. Installing it now. ......... ...................... NAME: kubescape LAST DEPLOYED: Sat Feb  4 10:03:36 2023 NAMESPACE: kubescape STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Thank you for installing kubescape-cloud-operator version 1.9.5.  In a few minutes your scan results will be available in the following link: https://cloud.armosec.io/config-scanning/kubernetes-admin-kubernetes  You can see and change the values of your＂s recurring configurations daily scan in the following link: https://cloud.armosec.io/settings/assets/clusters/scheduled-scans?cluster=kubernetes-admin-kubernetes > kubectl -n kubescape get cj kubescape-scheduler -o=jsonpath=＂{.metadata.name}{＂	＂}{.spec.schedule}{＂ ＂}＂  You can see and change the values of your＂s recurring images daily scan in the following link: https://cloud.armosec.io/settings/assets/images > kubectl -n kubescape get cj kubevuln-scheduler -o=jsonpath=＂{.metadata.name}{＂	＂}{.spec.schedule}{＂ ＂}＂  See you!!! ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$
　　运行完成，安装提示操作，点击刚才注册生成的页面。
　　验证 Kubescape 在集群中运行状态 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$kubectl -n kubescape get deployments.apps NAME        READY   UP-TO-DATE   AVAILABLE   AGE gateway     1/1     1            1           19m kubescape   1/1     1            1           19m kubevuln    1/1     1            1           19m operator    1/1     1            1           19m
　　kubescape 会定期扫描 ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$kubectl get cronjobs.batch -n kubescape NAME                  SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGE kubescape-scheduler   21 3 * * *    False     0                  20m kubevuln-scheduler    13 22 * * *   False     0                  20m ┌──[root@vms100.liruilongs.github.io]-[~/ansible/kubescape] └─$
　　在登录的主页中可以看到集群和第一次扫描结果
　　安全合规性扫描信息
　　可以通过不同的维度来看
　　合规性的维度查看
　　集群 Pod/Deploy 的维度查看
　　具体的 合规配置信息查看
　　漏洞扫描
　　关于  kubescape  和小伙伴分享到这里，时间关系，没有深入太多。偶尔听大佬谈到，所以研究一下，有需要的小伙伴快去尝试吧。博文部分内容参考
　　文中涉及参考链接内容版权归原作者所有，如有侵权请告知
　　https://github.com/kubescape/kubescape
　　https://www.redhat.com/en/blog/state-kubernetes-security-2022-1
　　https://betterprogramming.pub/image-vulnerability-scanning-for-optimal-kubernetes-security-c3ba933190ef
　　https://hub.armosec.io/docs/installation-of-armo-in-cluster#install-a-pre-registered-cluster
　　© 2018-2023 liruilonger@gmail.com，All rights reserved. 保持署名-非商用-自由转载-相同方式共享(创意共享 3.0 许可证)