Pain, So...

CloudStack 4.2.1+KVM部署实战。

拓扑图:

Topology

相关说明:

系统都为CentOS6.4 x64

Management端不需要任何虚拟化的支持,只是做WEB控制台和管理用,任意机器即可

Agent端的CPU需要虚拟化支持,我这里用的是R710,已开启VT并且cpu支持flag:vmx

主存储和二级存储都放在Agent端

配置网络:

[root@management ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=static
ONBOOT=yes
IPADDR=10.10.16.250
NETMASK=255.255.255.0
GATEWAY=10.10.16.254
DNS1=8.8.4.4

[root@management ~]# ifconfig
eth1 Link encap:Ethernet HWaddr 00:1D:60:E4:01:F2
inet addr:10.10.16.250 Bcast:10.10.16.255 Mask:255.255.255.0
inet6 addr: fe80::21d:60ff:fee4:1f2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:31460 errors:0 dropped:0 overruns:0 frame:0
TX packets:9113 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:25547046 (24.3 MiB) TX bytes:704232 (687.7 KiB)
Interrupt:18

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)

[root@agent ~]# cat /etc/sysconfig/network-scripts/ifcfg-em2
DEVICE=em2
BOOTPROTO=static
ONBOOT=yes
IPADDR=10.10.16.253
NETMASK=255.255.255.0
GATEWAY=10.10.16.254
DNS1=8.8.4.4

[root@agent ~]# ifconfig
em2 Link encap:Ethernet HWaddr 78:2B:CB:64:B7:A2
inet addr:10.10.16.253 Bcast:10.10.16.255 Mask:255.255.255.0
inet6 addr: fe80::7a2b:cbff:fe64:b7a2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:189 errors:0 dropped:0 overruns:0 frame:0
TX packets:58 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19800 (19.3 KiB) TX bytes:6460 (6.3 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
清空防火墙(两台都需要):

[root@management ~]# iptables  -F
[root@management ~]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables: [ OK ]

配置NTP:

management:

[root@management ~]# yum -y install ntp

编辑/etc/ntp.conf,找到server部分,修改如下(把management做NTP服务器,不从互联网更新,也可参考其他配置或者事先从网络更新一下):

#server 0.rhel.pool.ntp.org iburst

#server 1.rhel.pool.ntp.org iburst

#server 2.rhel.pool.ntp.org iburst

#server 3.rhel.pool.ntp.org iburst
server 127.127.1.0
fudge 127.127.1.0 stratum 10

[root@management ~]# service ntpd restart
Shutting down ntpd: [ OK ]
Starting ntpd: [ OK ]
[root@management ~]# chkconfig ntpd on

agent:

[root@agent ~]# yum -y install ntp

编辑/etc/ntp.conf,找到server部分,修改如下:

#server 0.rhel.pool.ntp.org iburst

#server 1.rhel.pool.ntp.org iburst

#server 2.rhel.pool.ntp.org iburst

#server 3.rhel.pool.ntp.org iburst
server 10.10.16.250

[root@management ~]# service ntpd restart
Shutting down ntpd: [ OK ]
Starting ntpd: [ OK ]
[root@management ~]# chkconfig ntpd on

重启网络和关闭SELINUX、hosts文件添加住被控IP和主机名(2台都需要)

[root@management ~]# service network restart
Shutting down interface eth1: [ OK ]
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface eth1: [ OK ]

[root@agent ~]# service network restart
Shutting down interface em2: [ OK ]
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface em2: [ OK ]

[root@agent ~]# cat /etc/selinux/config

This file controls the state of SELinux on the system.

SELINUX= can take one of these three values:

enforcing - SELinux security policy is enforced.

permissive - SELinux prints warnings instead of enforcing.

disabled - SELinux is fully disabled.

SELINUX=disabled

SELINUXTYPE= type of policy in use. Possible values are:

targeted - Only targeted network daemons are protected.

strict - Full SELinux protection.

SELINUXTYPE=targeted

[root@management ~]# hostname –fqdn
management
[root@management ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.10.16.250 management
10.10.16.253 agent
添加源:

只需要默认源和cloudstack源即可,由于网速太慢,我事先准备了另外一台服务器做本地源

repo

[root@management ~]# cat /etc/yum.repos.d/1.repo
[rhel]
name=rhel
baseurl=http://10.10.16.100/repo/rhel
enable=1
gpgcheck=0
[cloudstack]
name=cloudstack
baseurl=http://10.10.16.100/repo/cloudstack
enable=1
gpgcheck=0

更新重启:

[root@management ~]# yum -y update;reboot
[root@agent ~]# yum -y update;reboot

配置management端:

安装MYSQL:

[root@management ~]# yum -y install mysql-server
[root@management ~]# service mysqld start
[root@management ~]# chkconfig mysqld on

设置MYSQL密码:

[root@management ~]# mysql_secure_installation

这里我的mysql密码为:cloudstack

安装management:

[root@management ~]# yum -y install cloudstack-management

等待ing:

install-management

导入数据库:

[root@management ~]# cloudstack-setup-databases 新建的数据库帐号:密码@主机名 –deploy-as=root:mysql管理员密码

setup-database

设置cloudstack用户对/var/log/cloudstack*日志目录的访问权限(不然会无法启动):

[root@management ~]# chown -R cloud:cloud /var/log/cloudstack*

cloud账户为安装management时自动创建的,用于运行相关服务 。

初始化management服务:

[root@management ~]# cloudstack-setup-management
Starting to configure CloudStack Management Server:
Configure sudoers … [OK]
Configure Firewall … [OK]
Configure CloudStack Management Server …[OK]
CloudStack Management Server setup is Done!
[root@management ~]# netstat -antp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0: LISTEN 941/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:
LISTEN 1018/master
tcp 0 0 0.0.0.0:3306 0.0.0.0: LISTEN 1272/mysqld
tcp 0 52 10.10.16.250:22 10.10.16.220:57436 ESTABLISHED 1052/sshd
tcp 0 0 :::51022 :::
LISTEN 2193/java
tcp 0 0 :::48686 ::: LISTEN 2193/java
tcp 0 0 :::8080 :::
LISTEN 2193/java
tcp 0 0 :::22 ::: LISTEN 941/sshd
tcp 0 0 ::1:25 :::
LISTEN 1018/master
tcp 0 0 :::45219 ::: LISTEN 2193/java
tcp 0 0 :::7080 :::
LISTEN 2193/java
tcp 0 0 ::1:55059 ::1:34910 TIME_WAIT -

如上,8080 WEB控制台已经正常启动,

访问http://10.10.16.250:8080/client 默认帐号:admin 密码:password

dashboard

配置NFS服务:

之前说了,我准备把主存储和二级存储都放agent,所以在agent上安装NFS

[root@agent ~]# yum -y install nfs-utils

挂载磁盘:

[root@agent ~]# mkdir -p /storage/{primary,secondary}
[root@agent ~]# mount /dev/sdb /storage/secondary/
[root@agent ~]# mount /dev/sdc /storage/primary/

[root@agent ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 20G 1003M 18G 6% /
tmpfs 56G 0 56G 0% /dev/shm
/dev/sdb 135G 188M 128G 1% /storage/secondary
/dev/sdc 3.6T 197M 3.4T 1% /storage/primary

我对cloudstack存储的理解:

主存储存放VM文件和磁盘文件

辅助存储存放ISO镜像和系统模板

记得添加开机自动挂载

fstab

编辑/etc/exports 添加NFS共享信息:

[root@agent ~]# vim /etc/exports
/storage/primary (rw,async,no_root_squash)
/storage/secondary
(rw,async,no_root_squash)

启动NFS服务:

[root@agent ~]# service rpcbind start
Starting rpcbind: [ OK ]
[root@agent ~]# chkconfig rpcbind on
[root@agent ~]# service nfs start
Starting NFS services: [ OK ]
Starting NFS mountd: [ OK ]
Starting NFS daemon: [ OK ]
Starting RPC idmapd: [ OK ]
[root@agent ~]# chkconfig nfs on

在management测试nfs是否工作正常:

[root@management ~]# showmount -e 10.10.16.253
Export list for 10.10.16.253:
/storage/secondary
/storage/primary

能正常显示则OK;

management端导入KVM系统模板(用于cloudstack系统VM):

我之前用官方的一直不正常,推荐使用下方地址的模板

可以在http://jenkins.buildacloud.org/job/build-systemvm64-master/ 这里下载

或者用这个:http://soft.painso.com/Linux/CloudStack/templates/systemvm64template-2013-12-09-master-kvm.qcow2.bz2 (本次安装用的)

systemvm-template

挂载NFS二级存储(之前讲过是放系统模板和ISO用):

[root@management ~]# mount 10.10.16.253:/storage/secondary /opt

导入系统VM模板:

[root@management ~]# /usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m NFS挂载二级存储的路径 -f 系统模板文件路径 -h 虚拟化平台名称 -F

import-systemvm-template

等待。。。

导入完毕后卸载二级存储:

[root@management ~]# umount /opt

进入WEB管理界面点击”我以前使用过 CloudStack,跳过此指南“,进入全局设置选项

设置存储所在网段:

secstorage

设置延时时间(删除VM等等操作的延时):

expunge

重启management:

[root@management ~]# service cloudstack-management restart
Stopping cloudstack-management: [ OK ]
Starting cloudstack-management: [ OK ]

配置Agent:

安装qemu等组件支持,不然待会添加主机会加不上;

[root@agent ~]# yum -y install qemu virt libvirt*

install-qemu

[root@agent ~]# yum -y install cloudstack-agent

编辑/etc/libvirt/qemu.conf找到vnc_listen = “0.0.0.0”取消注释

qemu

编辑/etc/cgconfig.conf添加:

group virt {
cpu {
cpu.shares = 9216;
}
}

cgconfig

启动服务:

[root@agent ~]# service cgconfig start
Starting cgconfig service: [ OK ]
[root@agent ~]# service libvirtd restart
Stopping libvirtd daemon: [ OK ]
Starting libvirtd daemon: [ OK ]
[root@agent ~]# chkconfig cgconfig on
[root@agent ~]# chkconfig libvirtd on

注意:此时cloudstack-agent服务不需要启动,management添加该主机时会自动启动

访问http://10.10.16.250:8080/client management端开始配置cloudstack:

cloudstack-1

修改新密码继续:

cloudstack-pass

区域类似一个数据中心;

add-area

添加提供点:我的理解是给CloudStack内部系统用的IP,如系统VM

add-offer

来宾网络:我的理解是给我们创建的VM用的IP

add-guest-network

add-cluster

添加主机:就是添加agent端,输入IP、用户名和密码即可

add-host

add-primary

add-secondary

最后点击启动,等待数分钟

cs-successfully

点击启动后可进入主面板,可查看agent配置信息和相关资源使用信息

cs-dashboard

点击“基本架构”,可查看相关信息

cs-base

点击系统VM下的“查看全部”可看到系统VM的状态,都为running+up则表示正常

systemvm-satate

至此CloudStack Management Agent分离安装完毕,如果有任何批评和疑问请在此提出。