Ubuntu18。04下部署k8s

　　一、更新Ubuntu源mvetcaptsources。listetcaptsources。list。bakcatetcaptsources。list。bakgrepvgrepv34；sources。listsedisarchive。ubuntu。commirrors。ustc。edu。cngetcaptsources。listsedissecurity。ubuntu。commirrors。ustc。edu。cngetcaptsources。listaptyupdateaptyupgrade2、timedatectlsedisenUSCgetcdefaultlocaletimedatectlsettimezoneAsiaShanghai3、bashcompletionsedi97，99sgroot。bashrc4、sshechoPermitRootLoginyesetcsshsshdconfigpasswdrootEOFpasswordpasswordEOFsystemctlreloadssh5、hostsvimetchosts10。0。0。20k8smaster0010。0。0。21k8smaster0110。0。0。22k8smaster0210。0。0。23k8snode0110。0。0。24k8snode0210。0。0。25k8sblmaster6、sshkeygensshkeygentrsaforiincatroot。txt；doechoi；sshcopyidi。sshidrsa。pubi；done7、swapswapoffasediswaps（。）1getcfstab8、networknetcatetcnetplan00installerconfig。yamlawkNR4｛print1｝sedis｛net｝eth0：getcnetplan00installerconfig。yamlsedi11snet。ifnames0biosdevname0getcdefaultgrubupdategrubreboot二、安装ipvsaptyinstallipvsadmipsetsysstatconntracklibseccomp2libseccompdevcatetcmodulesload。dipvs。confEOFipvsipvslcipvswlcipvsrripvswrripvslblcipvslblcripvsdhipvsshipvsfoipvsnqipvssedipvsftpnfconntrackiptablesipsetxtsetiptsetiptrpfilteriptREJECTipipEOFsystemctlrestartsystemdmodulesload。servicelsmodgrepeipvsenfconntrackipv4三、下载安装containerdwgethttps：github。comcontainerdcontainerdreleasesdownloadv1。6。1cricontainerdcni1。6。1linuxamd64。tar。gztarnooverwritedirCxzfcricontainerdcni1。6。1linuxamd64。tar。gzsystemctldaemonreloadsystemctlenablenowcontainerd修改config。tomlcontainerdconfigdefaultetccontainerdconfig。tomlsedisk8s。gcr。ioregistry。aliyuncs。comgooglecontainersgetccontainerdconfig。tomlsedisSystemdCgroupfalseSystemdCgrouptruegetccontainerdconfig。tomlsedi153a〔plugins。io。containerd。grpc。v1。cri。registry。mirrors。docker。io〕etccontainerdconfig。toml8个空格endpoint10个空格sedi154aendpoint〔https：registry。aliyuncs。com〕etccontainerdconfig。toml修改crictl。yamlmvetccrictl。yamletccrictl。yaml。bakcatetccrictl。yamlEOFruntimeendpoint：unix：runcontainerdcontainerd。sockimageendpoint：unix：runcontainerdcontainerd。socktimeout：0debug：falsepullimageoncreate：falsedisablepullonrun：falseEOF四、安装nginx做四层代理aptyinstallnginxcpetcnginxnginx。confetcnginxnginx。conf。bakvimetcnginxnginx。conf。。。。。。stream｛logformatmainremoteaddrupstreamaddr〔timelocal〕statusupstreambytessent；accesslogvarlognginxk8saccess。logmain；upstreamk8sapiserver｛server10。0。0。20：6443；server10。0。0。21：6443；server10。0。0。22：6443；｝server｛listen6444；proxypassk8sapiserver；｝｝http｛logformatmainremoteaddrremoteuser〔timelocal〕requeststatusbodybytessenthttprefererhttpuseragenthttpxforwardedfor；。。。。。。｝systemctlenablenownginx。servicesystemctlstatusnginx。service五、安装keepalive做高可用aptyinstallkeepalivedkeepalivedconfigcatetckeepalivedkeepalived。confEOFglobaldefs｛notificationemail｛acassenfirewall。locfailoverfirewall。locsysadminfirewall。loc｝notificationemailfromAlexandre。Cassenfirewall。locsmtpserver127。0。0。1smtpconnecttimeout30routeridNGINXMASTER｝vrrpscriptchecknginx｛scriptetckeepalivedchecknginx。shinterval5weight1fall2rise1｝vrrpinstanceVI1｛stateMASTERinterfaceeth0修改为实际网卡名virtualrouterid51VRRP路由ID实例，每个实例是唯一的priority100优先级，备服务器设置90advertint1指定VRRP心跳包通告间隔时间，默认1秒authentication｛authtypePASSauthpassK8SHAKAAUTH｝虚拟IPvirtualipaddress｛10。0。0。2524｝trackscript｛checknginx｝｝EOFhealthconfigcatetckeepalivedchecknginx。shEOF！binbashcount（psefgrepnginxgrepsbinegrepcvgrep）if〔counteq0〕；thensystemctlstopkeepalivedfiEOFsystemctlenablenowkeepalived。servicesystemctlstatuskeepalived。service六、master端部署cfssl、etcd、cacertificate、etcdcertificate
　　6。1、下载cfsslwgethttps：github。comcloudflarecfsslreleasesdownloadv1。6。1cfssl1。6。1linuxamd64Ousrlocalbincfsslwgethttps：github。comcloudflarecfsslreleasesdownloadv1。6。1cfssljson1。6。1linuxamd64Ousrlocalbincfssljsonwgethttps：github。comcloudflarecfsslreleasesdownloadv1。6。1cfsslcertinfo1。6。1linuxamd64OusrlocalbincfsslcertinfochmodxusrlocalbincfsslchownRfroot：rootusrlocalbincfssl
　　6。2、etcd目录规划allMaster1、etcdsslmkdirpetcetcdssl2、etcdWorkingDirectorymkdirpvarlibetcddefault。etcd3、kubernetessslmkdirpetckubernetesssl4、kuberneteslogmkdirpvarlogkubernetes
　　6。3、ca证书生成mkdirpworkcdworkcatcacsr。jsonEOF｛CN：kubernetes，key：｛algo：rsa，size：2048｝，names：〔｛C：CN，ST：Shanghai，L：Shanghai，O：k8s，OU：system｝〕｝EOFcatcaconfig。jsonEOF｛signing：｛default：｛expiry：87600h｝，profiles：｛kubernetes：｛usages：〔signing，keyencipherment，serverauth，clientauth〕，expiry：87600h｝｝｝｝EOFcfsslgencertinitcacacsr。jsoncfssljsonbarecacpca。pemetcetcdsslsendtoothermasterforiincatMasterNodes。txt；doechoi；scpetcetcdsslca。pemi：etcetcdssl；done
　　6。4配置etcd证书catetcdcsr。jsonEOF｛CN：etcd，hosts：〔127。0。0。1，10。0。0。20，10。0。0。21，10。0。0。22，10。0。0。25〕，key：｛algo：rsa，size：2048｝，names：〔｛C：CN，ST：Shanghai，L：Shanghai，O：k8s，OU：system｝〕｝EOFcfsslgencertcaca。pemcakeycakey。pemconfigcaconfig。jsonprofilekubernetesetcdcsr。jsoncfssljsonbareetcdcpetcd。pemetcetcdsslsendtootherforiincatMasterNodes。txt；doechoi；scpetcetcdssletcd。pemi：etcetcdssl；done
　　6。5、下载及配置etcddownloadetcdwgethttps：github。cometcdioetcdreleasesdownloadv3。5。0etcdv3。5。0linuxamd64。tar。gztaretcd。tar。gztarxfetcdv3。5。0linuxamd64。tar。gzstripcomponents1Cworketcdv3。5。0linuxamd64etcd｛，ctl｝chownRfroot：rootetcdcparpetcdusrlocalbinsendtootherforiincatMasterNodes。txt；doechoi；scpusrlocalbinetcd｛，ctl｝i：usrlocalbin；donecatetcetcdetcd。confEOFETCDNAMEetcd1ETCDDATADIRvarlibetcddefault。etcdETCDLISTENPEERURLShttps：10。0。0。20：2380changeipETCDLISTENCLIENTURLShttps：10。0。0。20：2379，http：127。0。0。1：2379changeipETCDINITIALADVERTISEPEERURLShttps：10。0。0。20：2380changeipETCDADVERTISECLIENTURLShttps：10。0。0。20：2379changeipETCDINITIALCLUSTERetcd1https：10。0。0。20：2380，etcd2https：10。0。0。21：2380，etcd3https：10。0。0。22：2380ETCDINITIALCLUSTERTOKENetcdclusterETCDINITIALCLUSTERSTATEnewEOF
　　6。6、添加etcdsystemd启动catusrlibsystemdsystemetcd。serviceEOF〔Unit〕DescriptionEtcdServiceAfternetwork。targetAfternetworkonline。targetWantsnetworkonline。target〔Service〕TypenotifyEnvironmentFileetcetcdetcd。confWorkingDirectoryvarlibetcdExecStartusrlocalbinetcdcertfileetcetcdssletcd。pemkeyfileetcetcdssletcdkey。pemtrustedcafileetcetcdsslca。pempeercertfileetcetcdssletcd。pempeerkeyfileetcetcdssletcdkey。pempeertrustedcafileetcetcdsslca。pempeerclientcertauthclientcertauthRestartonfailureRestartSec10LimitNOFILE65536〔Install〕WantedBymultiuser。targetEOFsendtootherforiincatMasterNodes。txt；doechoi；scpusrlibsystemdsystemetcd。servicei：usrlibsystemdsystem；done
　　启动etcd1、startetcdsystemctldaemonreloadsystemctlenablenowetcd。servicesystemctlstatusetcd。service2、checketcdETCDCTLAPI3etcdctlendpointshttps：10。0。0。20：2379，https：10。0。0。21：2379，https：10。0。0。22：2379writeouttablecacertetcetcdsslca。pemcertetcetcdssletcd。pemkeyetcetcdssletcdkey。pemendpointhealthENDPOINTHEALTHTOOKERRORhttps：10。0。0。20：2379true16。188005mshttps：10。0。0。21：2379true16。693314mshttps：10。0。0。22：2379true16。089367ms七、安装k8smaster1、downloadwgethttps：dl。k8s。iov1。23。5kubernetesserverlinuxamd64。tar。gz2、tartarxfkubernetesserverlinuxamd64。tar。gzstripcomponents3Cworkkubernetesserverbinkube｛let，ctl，apiserver，controllermanager，scheduler，proxy｝scpkube｛ctl，apiserver，controllermanager，scheduler｝usrlocalbin3、kube｛let，ctl，apiserver，controllermanager，scheduler，proxy｝foriincatMasterNodes。txt；doechoi；scpworkkube｛ctl，apiserver，controllermanager，scheduler｝i：usrlocalbin；done4、kube｛let，proxy｝foriincatWorkNodes。txt；doechoi；scpworkkube｛let，proxy｝i：usrlocalbin；done5、sendpemcpetcetcdsslca。pemetckubernetessslforiincatWorkNodes。txt；doechoi；scpetcetcdsslca。pemi：etckubernetesssl；done添加kubeapiservertokencatetckubernetestoken。csvEOF（headc16devurandomodAntxtrd），kubeletbootstrap，10001，system：kubeletbootstrapEOFforiincatMasterNodes。txt；doechoi；scpetckubernetestoken。csvi：etckubernetes；done
　　7。2、添加kubeapiserver证书catkubeapiservercsr。jsonEOF｛CN：kubernetes，hosts：〔127。0。0。1，10。0。0。20，10。0。0。21，10。0。0。22，10。0。0。23，10。0。0。24，10。0。0。25，10。96。0。1，kubernetes，kubernetes。default，kubernetes。default。svc，kubernetes。default。svc。cluster，kubernetes。default。svc。cluster。local〕，key：｛algo：rsa，size：2048｝，names：〔｛C：CN，ST：Shanghai，L：Shanghai，O：k8s，OU：system｝〕｝EOFcfsslgencertcaca。pemcakeycakey。pemconfigcaconfig。jsonprofilekuberneteskubeapiservercsr。jsoncfssljsonbarekubeapiservercpkubeapiserver。pemetckubernetessslforiincatMasterNodes。txt；doechoi；scpworkkubeapiserver。pemi：etckubernetesssl；done
　　7。3、天kubeapiserver配置文件changebindaddressandadvertiseaddresscatetckuberneteskubeapiserver。confEOFKUBEAPISERVEROPTSenableadmissionpluginsNamespaceLifecycle，NodeRestriction，LimitRanger，ServiceAccount，DefaultStorageClass，ResourceQuotaanonymousauthfalsebindaddress10。0。0。20secureport6443advertiseaddress10。0。0。20insecureport0authorizationmodeNode，RBACruntimeconfigapialltrueenablebootstraptokenauthserviceclusteriprange10。96。0。016tokenauthfileetckubernetestoken。csvservicenodeportrange3000050000tlscertfileetckubernetessslkubeapiserver。pemtlsprivatekeyfileetckubernetessslkubeapiserverkey。pemclientcafileetckubernetessslca。pemkubeletclientcertificateetckubernetessslkubeapiserver。pemkubeletclientkeyetckubernetessslkubeapiserverkey。pemserviceaccountkeyfileetckubernetessslcakey。pemserviceaccountsigningkeyfileetckubernetessslcakey。pemserviceaccountissuerhttps：kubernetes。default。svc。cluster。localetcdcafileetcetcdsslca。pemetcdcertfileetcetcdssletcd。pemetcdkeyfileetcetcdssletcdkey。pemetcdservershttps：10。0。0。20：2379，https：10。0。0。21：2379，https：10。0。0。22：2379enableswaggeruitrueallowprivilegedtrueapiservercount3auditlogmaxage30auditlogmaxbackup3auditlogmaxsize100auditlogpathvarlogkubeapiserveraudit。logeventttl1halsologtostderrtruelogtostderrfalselogdirvarlogkubernetesv4EOF
　　7。4、添加kubeapiserversystemd启动catusrlibsystemdsystemkubeapiserver。serviceEOF〔Unit〕DescriptionKubernetesAPIServerDocumentationhttps：github。comkuberneteskubernetesAfteretcd。serviceWantsetcd。service〔Service〕EnvironmentFileetckuberneteskubeapiserver。confExecStartusrlocalbinkubeapiserverKUBEAPISERVEROPTSRestartonfailureRestartSec5TypenotifyLimitNOFILE65536〔Install〕WantedBymultiuser。targetEOFforiincatMasterNodes。txt；doechoi；scpusrlibsystemdsystemkubeapiserver。servicei：usrlibsystemdsystem；done
　　启动kubeapiserversystemctldaemonreloadsystemctlenablenowkubeapiserver。servicesystemctlstatuskubeapiserver。servicecheckcurlinsecurehttps：10。0。0。20：6443｛kind：Status，apiVersion：v1，metadata：｛｝，status：Failure，message：Unauthorized，reason：Unauthorized，code：401
　　7。5、kubectl安装添加admincertificatecatadmincsr。jsonEOF｛CN：admin，hosts：〔〕，key：｛algo：rsa，size：2048｝，names：〔｛C：CN，ST：Shanghai，L：Shanghai，O：system：masters，OU：system｝〕｝EOFcfsslgencertcaca。pemcakeycakey。pemconfigcaconfig。jsonprofilekubernetesadmincsr。jsoncfssljsonbareadmincpadmin。pemetckubernetessslforiincatMasterNodes。txt；doechoi；scpetckubernetesssladmin。pemi：etckubernetesssl；done添加admin。config1、设置集群参数kubectlconfigsetclusterkubernetescertificateauthorityca。pemembedcertstrueserverhttps：10。0。0。25：6444kubeconfigadmin。config2、设置客户端认证参数kubectlconfigsetcredentialskubernetesadminclientcertificateadmin。pemclientkeyadminkey。pemembedcertstruekubeconfigadmin。config3、设置上下文参数kubectlconfigsetcontextkubernetesclusterkubernetesuserkubernetesadminkubeconfigadmin。config4、设置当前上下文kubectlconfigusecontextkuberneteskubeconfigadmin。configkuberneteskubeletapikubectlcreateclusterrolebindingkubeapiserver：kubeletapiserverclusterrolesystem。kubeletapiadminuserkuberneteskubectlcreateclusterrolebindingkubernetesclusterroleclusteradminuserkubernetes其它节点cpworkadmin。configetckubernetesmkdirpHOME。kubecpietckubernetesadmin。configHOME。kubeconfigchown（idu）：（idg）HOME。kubeconfigforiincatMasterNodes。txt；doechoi；scpetckubernetesadmin。configi：etckubernetes；doneechoexportKUBECONFIGetckubernetesadmin。configetcprofilesourceetcprofilekubectl（bashcompletion）kubectl（bashcompletion）source（kubectlcompletionbash）echosource（kubectlcompletionbash）etcprofilesourceetcprofilekubectlclusterinfo｛Kubernetescontrolplaneisrunningathttps：10。0。0。20：6443｝kubectlgetcomponentstatusesNAMESTATUSMESSAGEERRORschedulerUnhealthyGethttps：127。0。0。1：10259healthz：dialtcp127。0。0。1：10259：connect：connectionrefusedcontrollermanagerUnhealthyGethttps：127。0。0。1：10257healthz：dialtcp127。0。0。1：10257：connect：connectionrefusedetcd0Healthy｛health：true，reason：｝etcd1Healthy｛health：true，reason：｝etcd2Healthy｛health：true，reason：｝kubectlgetallallnamespacesNAMESPACENAMETYPECLUSTERIPEXTERNALIPPORT（S）AGEdefaultservicekubernetesClusterIP10。96。0。1none443TCP56mverifykubectlkubectlclusterinfo｛Kubernetescontrolplaneisrunningathttps：10。0。0。20：6443｝kubectlgetcomponentstatusesNAMESTATUSMESSAGEERRORschedulerUnhealthyGethttps：127。0。0。1：10259healthz：dialtcp127。0。0。1：10259：connect：connectionrefusedcontrollermanagerUnhealthyGethttps：127。0。0。1：10257healthz：dialtcp127。0。0。1：10257：connect：connectionrefusedetcd0Healthy｛health：true，reason：｝etcd1Healthy｛health：true，reason：｝etcd2Healthy｛health：true，reason：｝kubectlgetallallnamespacesNAMESPACENAMETYPECLUSTERIPEXTERNALIPPORT（S）AGEdefaultservicekubernetesClusterIP10。96。0。1none443TCP56m八、kubecontrollermanagerkubecontrollermanagercertificatecatkubecontrollermanagercsr。jsonEOF｛CN：system：kubecontrollermanager，hosts：〔127。0。0。1，10。0。0。20，10。0。0。21，10。0。0。22，10。0。0。25〕，key：｛algo：rsa，size：2048｝，names：〔｛C：CN，ST：Shanghai，L：Shanghai，O：system：kubecontrollermanager，OU：Kubernetes｝〕｝EOFcfsslgencertcaca。pemcakeycakey。pemconfigcaconfig。jsonprofilekuberneteskubecontrollermanagercsr。jsoncfssljsonbarekubecontrollermanagercpkubecontrollermanager。pemetckubernetessslforiincatMasterNodes。txt；doechoi；scpetckubernetessslkubecontrollermanager。pemi：etckubernetesssl；donekubecontrollermanager。kubeconfig1、设置集群参数kubectlconfigsetclusterkubernetescertificateauthorityca。pemembedcertstrueserverhttps：10。0。0。25：6444kubeconfigetckuberneteskubecontrollermanager。kubeconfig2、设置客户端认证参数kubectlconfigsetcredentialssystem：kubecontrollermanagerclientcertificatekubecontrollermanager。pemclientkeykubecontrollermanagerkey。pemembedcertstruekubeconfigetckuberneteskubecontrollermanager。kubeconfig3、设置上下文参数kubectlconfigsetcontextsystem：kubecontrollermanagerclusterkubernetesusersystem：kubecontrollermanagerkubeconfigetckuberneteskubecontrollermanager。kubeconfig4、设置当前上下文kubectlconfigusecontextsystem：kubecontrollermanagerkubeconfigetckuberneteskubecontrollermanager。kubeconfigkubecontrollermanager。confcatetckuberneteskubecontrollermanager。confEOFKUBECONTROLLERMANAGEROPTSv2secureport10257bindaddress127。0。0。1kubeconfigetckuberneteskubecontrollermanager。kubeconfigserviceclusteriprange10。96。0。016clusternamekubernetesclustersigningcertfileetckubernetessslca。pemclustersigningkeyfileetckubernetessslcakey。pemallocatenodecidrstrueclustercidr10。244。0。016experimentalclustersigningduration87600hrootcafileetckubernetessslca。pemserviceaccountprivatekeyfileetckubernetessslcakey。pemleaderelecttruefeaturegatesRotateKubeletServerCertificatetruecontrollers，bootstrapsigner，tokencleanerhorizontalpodautoscalersyncperiod10stlscertfileetckubernetessslkubecontrollermanager。pemtlsprivatekeyfileetckubernetessslkubecontrollermanagerkey。pemuseserviceaccountcredentialstrueEOFforiincatMasterNodes。txt；doechoi；scpetckuberneteskubecontrollermanageri：etckubernetes；donekubecontrollermanager。servicesystemd启动catusrlibsystemdsystemkubecontrollermanager。serviceEOF〔Unit〕DescriptionKubernetesControllerManagerDocumentationhttps：github。comkuberneteskubernetes〔Service〕EnvironmentFileetckuberneteskubecontrollermanager。confExecStartusrlocalbinkubecontrollermanagerKUBECONTROLLERMANAGEROPTSRestartonfailureRestartSec5〔Install〕WantedBymultiuser。targetEOFforiincatMasterNodes。txt；doechoi；scpusrlibsystemdsystemkubecontrollermanager。servicei：usrlibsystemdsystem；donestartkubecontrollermanager。servicesystemctldaemonreloadsystemctlenablenowkubecontrollermanager。servicesystemctlstatuskubecontrollermanager。service九、调度器kubeschedulerkubeschedulercertificatecatkubeschedulercsr。jsonEOF｛CN：system：kubescheduler，hosts：〔127。0。0。1，10。0。0。20，10。0。0。21，10。0。0。22，10。0。0。25〕，key：｛algo：rsa，size：2048｝，names：〔｛C：CN，ST：Shanghai，L：Shanghai，O：system：kubescheduler，OU：system｝〕｝EOFcfsslgencertcaca。pemcakeycakey。pemconfigcaconfig。jsonprofilekuberneteskubeschedulercsr。jsoncfssljsonbarekubeschedulercpkubescheduler。pemetckubernetessslforiincatMasterNodes。txt；doechoi；scpetckubernetessslkubescheduler。pemi：etckubernetesssl；donekubescheduler。kubeconfig1、设置集群参数kubectlconfigsetclusterkubernetescertificateauthorityca。pemembedcertstrueserverhttps：10。0。0。25：6444kubeconfigetckuberneteskubescheduler。kubeconfig2、设置客户端认证参数kubectlconfigsetcredentialssystem：kubeschedulerclientcertificatekubescheduler。pemclientkeykubeschedulerkey。pemembedcertstruekubeconfigetckuberneteskubescheduler。kubeconfig3、设置上下文参数kubectlconfigsetcontextsystem：kubeschedulerclusterkubernetesusersystem：kubeschedulerkubeconfigetckuberneteskubescheduler。kubeconfig4、设置当前上下文kubectlconfigusecontextsystem：kubeschedulerkubeconfigetckuberneteskubescheduler。kubeconfigkubescheduler。confcatetckuberneteskubescheduler。confEOFKUBESCHEDULEROPTSaddress127。0。0。1kubeconfigetckuberneteskubescheduler。kubeconfigleaderelecttruealsologtostderrtruelogtostderrfalselogdirvarlogkubernetesv2EOFforiincatMasterNodes。txt；doechoi；scpetckuberneteskubescheduleri：etckubernetes；donekubescheduler。servicecatusrlibsystemdsystemkubescheduler。serviceEOF〔Unit〕DescriptionKubernetesSchedulerDocumentationhttps：github。comkuberneteskubernetes〔Service〕EnvironmentFileetckuberneteskubescheduler。confExecStartusrlocalbinkubeschedulerKUBESCHEDULEROPTSRestartonfailureRestartSec5〔Install〕WantedBymultiuser。targetEOFforiincatMasterNodes。txt；doechoi；scpusrlibsystemdsystemkubescheduler。servicei：usrlibsystemdsystem；donestartkubescheduler。servicesystemctldaemonreloadsystemctlenablenowkubescheduler。servicesystemctlstatuskubescheduler。service十、k8snode节点安装
　　1、kubeletBOOTSTRAPTOKENBOOTSTRAPTOKEN（awkF，｛print1｝etckubernetestoken。csv）
　　1。2kubeletbootstrap。kubeconfig1、设置集群参数kubectlconfigsetclusterkubernetescertificateauthorityca。pemembedcertstrueserverhttps：10。0。0。25：6444kubeconfigrootworkkubeletbootstrap。kubeconfig2、设置客户端认证参数kubectlconfigsetcredentialskubeletbootstraptoken｛BOOTSTRAPTOKEN｝kubeconfigrootworkkubeletbootstrap。kubeconfig3、设置上下文参数kubectlconfigsetcontextdefaultclusterkubernetesuserkubeletbootstrapkubeconfigrootworkkubeletbootstrap。kubeconfig4、设置当前上下文kubectlconfigusecontextdefaultkubeconfigrootworkkubeletbootstrap。kubeconfig5、创建clusterrolebindingkubectldeleteclusterrolebindingkubeletbootstrapkubectlcreateclusterrolebindingkubeletbootstrapclusterrolesystem：nodebootstrapperuserkubeletbootstrapkubectlcreateclusterrolebindingclustersystemanonymousclusterroleclusteradminuserkubeletbootstrap
　　1。3kubelet。jsoncatworkkubelet。jsonEOF｛kind：KubeletConfiguration，apiVersion：kubelet。config。k8s。iov1beta1，authentication：｛x509：｛clientCAFile：etckubernetessslca。pem｝，webhook：｛enabled：true，cacheTTL：2m0s｝，anonymous：｛enabled：false｝｝，authorization：｛mode：Webhook，webhook：｛cacheAuthorizedTTL：5m0s，cacheUnauthorizedTTL：30s｝｝，address：10。0。0。23，port：10250，readOnlyPort：10255，cgroupDriver：systemd，hairpinMode：promiscuousbridge，serializeImagePulls：false，clusterDomain：cluster。local。，clusterDNS：〔10。96。0。2〕｝EOF
　　1。4kubelet。servicecatworkkubelet。serviceEOF〔Unit〕DescriptionKubernetesKubeletDocumentationhttps：github。comkuberneteskubernetesAftercontainerd。serviceRequirescontainerd。service〔Service〕WorkingDirectoryvarlibkubeletExecStartusrlocalbinkubeletcontainerruntimeremotecontainerruntimeendpointunix：runcontainerdcontainerd。sockbootstrapkubeconfigetckuberneteskubeletbootstrap。kubeconfigcertdiretckubernetessslkubeconfigetckuberneteskubelet。kubeconfigconfigetckuberneteskubelet。jsonpodinfracontainerimageregistry。aliyuncs。comgooglecontainerspause：3。2v2RestartonfailureRestartSec5〔Install〕WantedBymultiuser。targetEOFforiincatWorkNodes。txt；doechoi；scpworkkubelet。jsonworkkubeletbootstrap。kubeconfigi：etckubernetes；doneforiincatWorkNodes。txt；doechoi；scpworkkubelet。servicei：usrlibsystemdsystem；doneforiincatWorkNodes。txt；doechoi；scpetckubernetessslca。pemi：etckubernetesssl；done
　　1。5startkubelet。servicesmkdirpvarlibkubeletsystemctldaemonreloadsystemctlenablenowkubelet。servicesystemctlstatuskubelet。service
　　1。6ApproveNodeskubectlgetcsrgrepnodeawk｛print1，6｝nodecsrBV7RZ1Mc1RFkWhH9jzJH8h8ondRMB3an7FgBUwWhkPendingnodecsrwZOIACKylv7DlEPRK8iMg3sYyBErjbGjxkMkRyPoPendingkubectlcertificateapprovenodecsrcsrBV7RZ1Mc1RFkWhH9jzJH8h8ondRMB3an7FgBUwWhknodecsrwZOIACKylv7DlEPRK8iMg3sYyBErjbGjxkMkRyPokubectlgetnodesNAMESTATUSROLESAGEVERSIONk8snode01Readynone118mv1。23。5k8snode02Readynone118mv1。23。5
　　2、kubepproxykubeproxycertificatecatkubeproxycsr。jsonEOF｛CN：system：kubeproxy，key：｛algo：rsa，size：2048｝，names：〔｛C：CN，ST：Shanghai，L：Shanghai，O：k8s，OU：system｝〕｝EOFcfsslgencertcaca。pemcakeycakey。pemconfigcaconfig。jsonprofilekuberneteskubeproxycsr。jsoncfssljsonbarekubeproxycpkubeproxy。pemetckubernetessslforiincatWorkNodes。txt；doechoi；scpworkkubeproxy。pemi：etckubernetesssl；done
　　2。3kubeproxy。kubeconfig1、设置集群参数kubectlconfigsetclusterkubernetescertificateauthorityca。pemembedcertstrueserverhttps：10。0。0。25：6444kubeconfigrootworkkubeproxy。kubeconfig2、设置客户端认证参数kubectlconfigsetcredentialskubeproxyclientcertificatekubeproxy。pemclientkeykubeproxykey。pemembedcertstruekubeconfigrootworkkubeproxy。kubeconfig3、设置上下文参数kubectlconfigsetcontextdefaultclusterkubernetesuserkubeproxykubeconfigrootworkkubeproxy。kubeconfig4、设置当前上下文kubectlconfigusecontextdefaultkubeconfigrootworkkubeproxy。kubeconfikubeproxy。yaml以下bindAddress均为宿主机ip，clusterCIDR为宿主机网段catworkkubeproxy。yamlEOFapiVersion：kubeproxy。config。k8s。iov1alpha1bindAddress：10。0。0。23clientConnection：kubeconfig：etckuberneteskubeproxy。kubeconfigclusterCIDR：10。244。0。024healthzBindAddress：10。0。0。23：10256kind：KubeProxyConfigurationmetricsBindAddress：10。0。0。23：10249mode：ipvsEOFforiincatWorkNodes。txt；doechoi；scpworkkubeproxy。yamlworkkubeproxy。kubeconfigi：etckubernetes；done
　　2。4kubeproxy。servicecatworkkubeproxy。serviceEOF〔Unit〕DescriptionKubernetesKubeProxyServerDocumentationhttps：github。comkuberneteskubernetesAfternetwork。target〔Service〕WorkingDirectoryvarlibkubeproxyExecStartusrlocalbinkubeproxyconfigetckuberneteskubeproxy。yamlalsologtostderrtruelogtostderrfalselogdirvarlogkubernetesv2RestartonfailureRestartSec5LimitNOFILE65536〔Install〕WantedBymultiuser。targetEOFforiincatWorkNodes。txt；doechoi；scpworkkubeproxy。servicei：usrlibsystemdsystem；done
　　2。5startkubeproxy。services
　　mkdirpvarlibkubeproxysystemctldaemonreloadsystemctlenablenowkubeproxy。servicesystemctlstatuskubeproxy。service
　　3、网络calicowgethttps：docs。projectcalico。orgmanifestscalico。yamlkubectlapplyfcalico。yaml
　　3。1corednsmvetcresolv。confetcresolv。conf。baklnsrunsystemdresolveresolv。confetcsystemctlrestartsystemdresolved。servicesystemctlenablesystemdresolved。service
　　coredns。yamlapiVersion：v1kind：ServiceAccountmetadata：name：corednsnamespace：kubesystemapiVersion：rbac。authorization。k8s。iov1kind：ClusterRolemetadata：labels：kubernetes。iobootstrapping：rbacdefaultsname：system：corednsrules：apiGroups：resources：endpointsservicespodsnamespacesverbs：listwatchapiGroups：discovery。k8s。ioresources：endpointslicesverbs：listwatchapiVersion：rbac。authorization。k8s。iov1kind：ClusterRoleBindingmetadata：annotations：rbac。authorization。kubernetes。ioautoupdate：truelabels：kubernetes。iobootstrapping：rbacdefaultsname：system：corednsroleRef：apiGroup：rbac。authorization。k8s。iokind：ClusterRolename：system：corednssubjects：kind：ServiceAccountname：corednsnamespace：kubesystemapiVersion：v1kind：ConfigMapmetadata：name：corednsnamespace：kubesystemdata：Corefile：。：53｛errorshealth｛lameduck5s｝readykubernetescluster。localinaddr。arpaip6。arpa｛fallthroughinaddr。arpaip6。arpa｝prometheus：9153forward。etcresolv。conf｛maxconcurrent1000｝cache30loopreloadloadbalance｝apiVersion：appsv1kind：Deploymentmetadata：name：corednsnamespace：kubesystemlabels：k8sapp：kubednskubernetes。ioname：CoreDNSspec：replicas：notspecifiedhere：1。Defaultis1。2。WillbetunedinrealtimeifDNShorizontalautoscalingisturnedon。strategy：type：RollingUpdaterollingUpdate：maxUnavailable：1selector：matchLabels：k8sapp：kubednstemplate：metadata：labels：k8sapp：kubednsspec：priorityClassName：systemclustercriticalserviceAccountName：corednstolerations：key：CriticalAddonsOnlyoperator：ExistsnodeSelector：kubernetes。ioos：linuxaffinity：podAntiAffinity：preferredDuringSchedulingIgnoredDuringExecution：weight：100podAffinityTerm：labelSelector：matchExpressions：key：k8sappoperator：Invalues：〔kubedns〕topologyKey：kubernetes。iohostnamecontainers：name：corednsimage：corednscoredns：1。8。4imagePullPolicy：IfNotPresentresources：limits：memory：170Mirequests：cpu：100mmemory：70Miargs：〔conf，etccorednsCorefile〕volumeMounts：name：configvolumemountPath：etccorednsreadOnly：trueports：containerPort：53name：dnsprotocol：UDPcontainerPort：53name：dnstcpprotocol：TCPcontainerPort：9153name：metricsprotocol：TCPsecurityContext：allowPrivilegeEscalation：falsecapabilities：add：NETBINDSERVICEdrop：allreadOnlyRootFilesystem：truelivenessProbe：httpGet：path：healthport：8080scheme：HTTPinitialDelaySeconds：60timeoutSeconds：5successThreshold：1failureThreshold：5readinessProbe：httpGet：path：readyport：8181scheme：HTTPdnsPolicy：Defaultvolumes：name：configvolumeconfigMap：name：corednsitems：key：Corefilepath：CorefileapiVersion：v1kind：Servicemetadata：name：kubednsnamespace：kubesystemannotations：prometheus。ioport：9153prometheus。ioscrape：truelabels：k8sapp：kubednskubernetes。ioclusterservice：truekubernetes。ioname：CoreDNSspec：selector：k8sapp：kubednsclusterIP：10。96。0。2ports：name：dnsport：53protocol：UDPname：dnstcpport：53protocol：TCPname：metricsport：9153protocol：TCP
　　安装corednskubectlapplyfcoredns。yaml二进制安装很繁琐，请耐心看完