ITPub博客
首页 博客 专栏 学院 社区 IT新媒体评选
发博

首页 > IT基础架构 > 网络通信/物联网 > TF实战丨使用Vagrant安装Tungsten Fabric

TF实战丨使用Vagrant安装Tungsten Fabric

原创 网络通信/物联网 作者：TF中文社区 时间：2020-06-19 09:27:49 0 删除 编辑

本文为苏宁网络架构师陈刚的原创文章。


01
准备测试机


在16G的笔记本没跑起来，就干脆拼凑了一台游戏工作室级别的机器：双路E5-2860v3 CPU，24核48线程，128G DDR4 ECC内存，NVME盘 512G。在上面开5个VM，假装是物理服务器。


· 192.16.35.110  deployer

· 192.16.35.111  tf控制器

· 192.16.35.112  openstack服务器，同时也是计算节点

· 192.16.35.113  k8s master

· 192.16.35.114  k8s的Node k01，同时也是ops的计算节点

 

直接使用vagrant拉镜像会很慢，就先下载下来：


https://cloud.centos.org/centos/7/vagrant/x86_64/images/

 

下载对应的VirtualBox.box文件。

 

然后使用命令, 命名为vagrant的box：

 

vagrant box add centos/7 CentOS-7-x86_64-Vagrant-2004_01.VirtualBox.box




    

    
cat << EEOOFF > vagrantfile


### start 


# -*- mode: ruby -*-


# vi: set ft=ruby :

Vagrant.require_version 
">=2.0.3"



    


# All Vagrant configuration is done below. The "2" in Vagrant.configure


# configures the configuration version (we support older styles for


# backwards compatibility). Please don't change it unless you know what


# you're doing.



    

ENV[
"LC_ALL"] = 
"en_US.UTF-8"



    

VAGRANTFILE_API_VERSION = 
"2"



    

Vagrant.configure(
"2") 
do |config|

  
# The most common configuration options are documented and commented below.

  
# For a complete reference, please see the online documentation at

  
# https://docs.vagrantup.com.



    

  
# Every Vagrant development environment requires a box. You can search for

  
# boxes at https://atlas.hashicorp.com/search.



    

  config.vm.box = 
"geerlingguy/centos7"

  
# config.vbguest.auto_update = false

  
# config.vbguest.no_remote = true  



    

  config.vm.define 
"deployer" 
do | dp |

    dp.vm.provider 
"virtualbox" 
do | v |

      v.memory = 
"8000"

      v.cpus = 2

    end

    dp.vm.network 
"private_network", ip: 
"192.16.35.110", auto_config: 
true

    dp.vm.hostname = 
"deployer"

  end



    

  config.vm.define 
"tf" 
do | tf |

    tf.vm.provider 
"virtualbox" 
do | v |

      v.memory = 
"64000"

      v.cpus = 16

    end

    tf.vm.network 
"private_network", ip: 
"192.16.35.111", auto_config: 
true

    tf.vm.hostname = 
"tf"

  end



    

  config.vm.define 
"ops" 
do | os |

    os.vm.provider 
"virtualbox" 
do | v |

      v.memory = 
"16000"

      v.cpus = 4

    end

    os.vm.network 
"private_network",ip: 
"192.16.35.112",  auto_config: 
true

    os.vm.hostname = 
"ops"

  end



    

  config.vm.define 
"k8s" 
do | k8 |

    k8.vm.provider 
"virtualbox" 
do | v |

      v.memory = 
"8000"

      v.cpus = 2

    end

    k8.vm.network 
"private_network", ip: 
"192.16.35.113", auto_config: 
true

    k8.vm.hostname = 
"k8s"

  end



    

  config.vm.define 
"k01" 
do | k1 |

    k1.vm.provider 
"virtualbox" 
do | v |

      v.memory = 
"4000"

      v.cpus = 2

    end

    k1.vm.network 
"private_network", ip: 
"192.16.35.114", auto_config: 
true

    k1.vm.hostname = 
"k01"

  end



    

  config.vm.provision 
"shell", privileged: 
true, path: 
"./setup.sh"



    

end



    



    

EEOOFF



    

cat << EEOOFF > setup.sh


#!/bin/bash


#


# Setup vagrant vms.


#



    


set -eu



    


# Copy hosts info

cat <<EOF > /etc/hosts

127.0.0.1 localhost

127.0.1.1 vagrant.vm vagrant



    

192.16.35.110 deployer

192.16.35.111 tf

192.16.35.112 ops

192.16.35.113 k8s

192.16.35.114 k01



    



    


# The following lines are desirable for IPv6 capable hosts

::1     localhost ip6-localhost ip6-loopback

ff02::1 ip6-allnodes

ff02::2 ip6-allrouters

EOF



    

systemctl stop firewalld

systemctl 
disable firewalld

iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat

iptables -P FORWARD ACCEPT



    

swapoff -a 

sed -i 
's/.*swap.*/#&/' /etc/fstab


# swapoff -a && sysctl -w vm.swappiness=0



    


# setenforce  0 

sed -i 
"s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/sysconfig/selinux 

sed -i 
"s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config 

sed -i 
"s/^SELINUX=permissive/SELINUX=disabled/g" /etc/sysconfig/selinux 

sed -i 
"s/^SELINUX=permissive/SELINUX=disabled/g" /etc/selinux/config  



    


# modprobe ip_vs_rr

modprobe br_netfilter



    

yum -y update



    


# sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-ip6tables: No such file or directory


# sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory


# yum install -y bridge-utils.x86_64


# modprobe bridge


# modprobe br_netfilter


# Setup system vars



    

yum install -y epel-release

yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools vim chrony python python-setuptools python-pip iproute lrzsz tree git



    

yum install -y libguestfs-tools libvirt-python virt-install libvirt ansible



    

pip install wheel --upgrade -i https://mirrors.aliyun.com/pypi/simple/

pip install pip --upgrade -i https://mirrors.aliyun.com/pypi/simple/

pip install ansible  netaddr --upgrade -i https://mirrors.aliyun.com/pypi/simple/



    


# python-urllib3 should be installed before "pip install requests"


# if install failed, pip uninstall urllib3, then reinstall python-urllib3


# pip uninstall -y urllib3 | true


# yum install -y python-urllib3 

pip install requests -i https://mirrors.aliyun.com/pypi/simple/



    

systemctl 
disable libvirtd.service

systemctl 
disable dnsmasq

systemctl stop libvirtd.service

systemctl stop dnsmasq



    


if [  -d 
"/root/.ssh" ]; 
then

      rm -rf /root/.ssh


fi



    

ssh-keygen -q -t rsa -N 
"" -f ~/.ssh/id_rsa



    

cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys

chmod go-rwx ~/.ssh/authorized_keys



    


# 


# timedatectl set-timezone Asia/Shanghai



    


if [ -f 
"/etc/chrony.conf" ]; 
then

   mv /etc/chrony.conf /etc/chrony.conf.bak


fi



    

cat <<EOF > /etc/chrony.conf

      allow 192.16.35.0/24

      server ntp1.aliyun.com iburst

      
local stratum 10

      logdir /var/
log/chrony

      rtcsync

      makestep 1.0 3

      driftfile /var/lib/chrony/drift

EOF



    

systemctl restart chronyd.service

systemctl 
enable chronyd.service



    


echo 
"* soft nofile 65536" >> /etc/security/limits.conf


echo 
"* hard nofile 65536" >> /etc/security/limits.conf


echo 
"* soft nproc 65536"  >> /etc/security/limits.conf


echo 
"* hard nproc 65536"  >> /etc/security/limits.conf


echo 
"* soft  memlock  unlimited"  >> /etc/security/limits.conf


echo 
"* hard memlock  unlimited"  >> /etc/security/limits.conf



    


if [ ! -d 
"/var/log/journal" ]; 
then

  mkdir /var/
log/journal


fi



    


if [ ! -d 
"/etc/systemd/journald.conf.d" ]; 
then

  mkdir /etc/systemd/journald.conf.d


fi



    

cat <<EOF > /etc/systemd/journald.conf.d/99-prophet.conf 

[Journal]

Storage=persistent



    

Compress=yes



    

SyncIntervalSec=5m

RateLimitInterval=30s

RateLimitBurst=1000



    

SystemMaxUse=10G



    

SystemMaxFileSize=200M



    

ForwardToSyslog=no



    

EOF



    

systemctl restart systemd-journald



    



    

EEOOFF


    02
    在所有的节点上安装docker


    CentOS


    例如： 如果pip安装软件的速度很慢，可以考虑使用基于aliyun的pip加速


    · 各个节点设置pip加速

      mkdir .pip && tee ~/.pip/pip.conf <<-'EOF'[global]trusted-host =  mirrors.aliyun.comindex-url = https://mirrors.aliyun.com/pypi/simpleEOF


      注意requests包不能在urllib3之后安装，否则会出错：

        pip uninstall urllib3pip uninstall chardetpip install requests


        (这些命令应该都已经在 setup.sh 中执行过了)

          yum install -y yum-utils device-mapper-persistent-data lvm2 net-tools iproute lrzsz tree gityum-config-manager   --add-repo   https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repoyum makecache fastyum install -y docker-ceyum -y install epel-releasesystemctl daemon-reloadsystemctl enable dockersystemctl restart dockeryum install -y chronysystemctl start chronydsystemctl enable chronyd


          03
          拉取并启动Contrail-Kolla-Ansible-Deployer容器


          容器的Nightly builds 可以从这里访问：  Docker Hub


          https://hub.docker.com/r/opencontrailnightly/contrail-kolla-ansible-deployer/tags


          例如：

            
vim /etc/docker/daemon.json

{ "registry-mirrors" : [ "https://hub-mirror.c.163.com",

    "https://registry.docker-cn.com" ] }

systemctl restart docker



            

export CAD_IMAGE=opencontrailnightly/contrail-kolla-ansible-deployer:master-latest

docker run -td --net host --name contrail_kolla_ansible_deployer $CAD_IMAGE


            04
              将配置文件复制到容器


            instance.yaml： 用于配置Tungsten Fabric集群的模板文件。


            https://github.com/Juniper/contrail-ansible-deployer/wiki/Contrail-with-Openstack-Kolla#13-configure-necessary-parameters-configinstancesyaml-under-appropriate-parameters


            要获得有关如何配置该文件中所有可用参数的信息，可阅读这里：


            https://github.com/Juniper/contrail-ansible-deployer/blob/master/README.md#configuration

             

              

cat 
<< EOF > instances.yaml


provider_config:  
bms:    ssh_pwd: vagrant    ssh_user: root    ntpserver: ntp1.aliyun.com    domainsuffix: localinstances:  tf:    provider: bms    ip: 192.16.35.111    roles:      config_database:      config:      control:      analytics_database:      analytics:      webui:  ops:    provider: bms    ip: 192.16.35.112    roles:

      
openstack:

      
openstack_compute:  

      
vrouter:

        
PHYSICAL_INTERFACE: 
enp0s8

  
k8s:    
provider: bms    ip: 192.16.35.113    roles:

      
k8s_master:

      
k8s_node:

      
kubemanager:

      
vrouter:

        
PHYSICAL_INTERFACE: 
enp0s8

  
k01:    
provider: bms    ip: 192.16.35.114    roles:

      
openstack_compute:

      
k8s_node:

      
vrouter:

        
PHYSICAL_INTERFACE: 
enp0s8


contrail_configuration:  
AUTH_MODE: keystone  KEYSTONE_AUTH_URL_VERSION: /v3

  
KEYSTONE_AUTH_ADMIN_PASSWORD: 
vagrant

  
CLOUD_ORCHESTRATOR: 
openstack

  
CONTRAIL_VERSION: 
latest

  
UPGRADE_KERNEL: 
true

  
ENCAP_PRIORITY: 
"VXLAN,MPLSoUDP,MPLSoGRE"

  
PHYSICAL_INTERFACE: 
enp0s8


global_configuration:

  
CONTAINER_REGISTRY: 
opencontrailnightly


kolla_config:

  
kolla_globals:    
enable_haproxy: no    enable_ironic: "no"    enable_swift: "no"

    
network_interface: 
"enp0s8"

  
kolla_passwords:    
keystone_admin_password: vagrant



              


EOF



              


export 
INSTANCES_FILE=instances.yaml


docker 
cp $INSTANCES_FILE contrail_kolla_ansible_deployer:/root/contrail-ansible-deployer/config/instances.yaml

              05
              准备好所有节点的环境


              除了deployer，我在所有节点上都做了一遍。


              正常的做法是建个自己的repository放各种image，实验环境节点少，直接国内下载也很快的。

               

              注意python和python-py这两个包是冲突的，只能安装其中之一，最好先全卸载，再安装其中一个：

                
pip uninstall docker-py docker

 pip install python



                

yum -y install python-devel python-subprocess32 python-setuptools python-pip



                

 pip install --upgrade pip



                

 find / -name *subpro*.egg-info

 find / -name *subpro*.egg-info |xargs rm -rf



                

pip install -I sixpip install -I docker-compose


                将k8s  repository改成阿里的，缺省的Google源太慢或不通：vi

                playbooks/roles/k8s/tasks/RedHat.yml

                  yum_repository:name: Kubernetesdescription: k8s repobaseurl: https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64gpgkey: https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpgrepo_gpgcheck: yesgpgcheck: yeswhen: k8s_package_version is defined


                  playbook中安装这些需要访问海外网站，可以从国内下载，然后改个tag：

                    k8s.gcr.io/kube-apiserver:v1.14.8k8s.gcr.io/kube-controller-manager:v1.14.8k8s.gcr.io/kube-scheduler:v1.14.8k8s.gcr.io/kube-proxy:v1.14.8k8s.gcr.io/pause:3.1k8s.gcr.io/etcd:3.3.10k8s.gcr.io/coredns:1.3.1


                    换个方法变通处理一下

                      docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.8docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.8docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.8docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.8docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10docker pull coredns/coredns:1.3.1docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.8.3


                      再重新给下载的打个tag

                        docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.14.8 k8s.gcr.io/kube-apiserver:v1.14.8docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.14.8 k8s.gcr.io/kube-controller-manager:v1.14.8docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.14.8 k8s.gcr.io/kube-scheduler:v1.14.8docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.14.8 k8s.gcr.io/kube-proxy:v1.14.8docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.3.10 k8s.gcr.io/etcd:3.3.10docker tag docker.io/coredns/coredns:1.3.1 k8s.gcr.io/coredns:1.3.1docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kubernetes-dashboard-amd64:v1.8.3  k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3


                        06
                        启动deployer容器，进入其中进行部署


                          docker start contrail_kolla_ansible_deployer


                          进入deployer容器：

                            
docker exec -it contrail_kolla_ansible_deployer bashcd /root/contrail-ansible-deployer

ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/provision_instances.yml

ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/configure_instances.yml

ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/install_openstack.yml

ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/install_k8s.yml

ansible-playbook -i inventory/ -e orchestrator=openstack playbooks/install_contrail.yml



                            



                            

kubectl taint nodes k8s node-role.kubernetes.io/master-


                            最后一次kubelet升级到最新，遇到CSI的bug，修改一下配置文件后重启kubelet即可：

                              After experiencing the same issue, editing /var/lib/kubelet/config.yaml to add:featureGates:  CSIMigration: false

                              07
                              安装完成后，建2个VM和容器测试一下


                                
yum install -y gcc python-devel

pip install python-openstackclient

pip install python-ironicclient



                                

source /etc/kolla/kolla-toolbox/admin-openrc.sh


                                如果openstack命令有如下“queue”的报错，是需要python3：

                                  File "/usr/lib/python2.7/site-packages/openstack/utils.py", line 13, in <module>    import queueImportError: No module named queue


                                    
rm -f /usr/bin/python

ln -s /usr/bin/python3 /usr/bin/python

pip install python-openstackclient

pip install python-ironicclient

yum install -y python3-pip



                                    

yum install -y gcc python-devel wgetpip install --upgrade setuptoolspip install --ignore-installed python-openstackclient



                                    

我每次都需要python3，所以干脆也安装了这个：

pip3 install python-openstackclient -i https://mirrors.aliyun.com/pypi/simple/

pip3 install python-ironicclient -i https://mirrors.aliyun.com/pypi/simple/


                                    进入Tungsten Fabric，用浏览器： https://192.16.35.111:8143


                                    进入openstack，用浏览器： https://192.16.35.112

                                     

                                    在k8s master上（ 192.16.35.113 ）：

                                      scp root@192.16.35.114:/opt/cni/bin/contrail-k8s-cni /opt/cni/bin/mkdir /etc/cni/net.dscp root@192.16.35.114:/etc/cni/net.d/10-contrail.conf /etc/cni/net.d/10-contrail.conf


                                      wget 

                                      https://github.com/cirros-dev/cirros/releases/download/0.4.0/cirros-0.4.0-x86_64-disk.img


                                      官方下载地址 

                                      https://download.cirros-cloud.net/ 

                                       

                                      curl -O 

                                      https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img

                                       

                                      wget 

                                      http://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img


                                      wget 

                                      http://download.cirros-cloud.net/daily/20161201/cirros-d161201-x86_64-disk.img 

                                      （都没有找到带tcpdump的版本）

                                       

                                      reboot

                                      source /etc/kolla/kolla-toolbox/admin-openrc.sh

                                        openstack image create cirros --disk-format qcow2 --public --container-format bare --file cirros-0.4.0-x86_64-disk.imgnova flavor-create m1.tiny auto 512 1 1openstack network create net1openstack subnet create --subnet-range 10.1.1.0/24 --network net1 mysubnet1NET_ID=`openstack network list | grep net1 | awk -F '|' '{print $2}' | tr -d ' '` nova boot --image cirros --flavor m1.tiny --nic net-id=${NET_ID} VM1nova boot --image cirros --flavor m1.tiny --nic net-id=${NET_ID} VM2


                                        进入k8s_master, 192.16.35.113:

                                          yum install -y gitgit clone https://github.com/virtualhops/k8s-demokubectl create -f k8s-demo/po-ubuntuapp.ymlkubectl create -f k8s-demo/rc-frontend.ymlkubectl expose rc/frontendkubectl exec -it ubuntuapp curl frontend # many times


                                          参考方案：

                                          https://github.com/Juniper/contrail-ansible-deployer/wiki/%5B-Container-Workflow%5D-Deploying-Contrail-with-OpenStack


                                          推荐阅读

                                          Tungsten Fabric实战：对接vMX虚拟路由平台填坑

                                          Tungsten Fabric实战：基于K8s的部署踩坑

                                          TF实战Q&A丨你不理解透，出了问题都不知道怎么弄

                                          TF 实战Q&A丨只在此网中，云深不知处





                                          来自 “ ITPUB博客 ” ，链接：https://blog.itpub.net/69957171/viewspace-2699332/ ，如需转载，请注明出处，否则将追究法律责任。

                                          0
                                          0
                                          上一篇： Tungsten Fabric入门宝典丨多编排器用法及配置
                                          下一篇： Tungsten Fabric解决方案指南-Kubernetes集成
                                          请登录后发表评论 登录
                                          全部评论
                                          TF中文社区
                                          Tungsten Fabric项目是一个开源项目协议，它基于标准协议开发，并且提供网络虚拟化和网络安全所必需的所有组件。项目的组件包括：SDN控制器，虚拟路由器，分析引擎，北向API的发布，硬件集成功能，云编排软件和广泛的REST API。

                                          注册时间：2019-12-11

                                          • 博文量
                                            102
                                          • 访问量
                                            74503

                                          最新文章

                                          支持我们 作者招募 用户协议 FAQ Contact Us

                                          北京盛拓优讯信息技术有限公司. 版权所有  京ICP备16024965号-8 北京市公安局海淀分局网监中心备案编号：11010802021510 niuxiaotong@pcpop.com 17352615567 未成年人举报专区

                                          广播电视节目制作经营许可证(京) 字第1234号 中国互联网协会会员

                                          违法和不良信息举报电话:18101376593