Setting up multi nodes live migration in Openstack Juno with devstack

Summary

• Live migration overview

• Setup Openstack with devstack: one controller node, two compute nodes

• Configure migrations

• Live migration test

• Issue summary

Live migration Overview

Migration enables an administrator to move a virtual-machine instance from one compute host to another. This feature is useful when a compute host requires maintenance. Migration can also be useful to redistribute the load when many VM instances are running on a specific physical machine.

The migration types are:

• Non-live migration (sometimes referred to simply as 'migration'). The instance is shut down for a period of time to be moved to another hypervisor. In this case, the instance recognizes that it was rebooted.

• Live migration (or 'true live migration'). Almost no instance downtime. Useful when the instances must be kept running during the migration. The different types of live migration are:

  • Shared storage-based live migration. Both hypervisors have access to shared storage.

  • Block live migration. No shared storage is required. Incompatible with read-only devices such as CD-ROMs and Configuration Drive (config_drive).

  • Volume-backed live migration. Instances are backed by volumes rather than ephemeral disk, no shared storage is required, and migration is supported (currently only available for libvirt-based hypervisors

This page's live migration uses block live migration which does not need shared storage.

Setup Openstack with devstack: one controller node, two compute nodes

Prepare three servers with ubuntu installed. One server is worked as control node, the other two are compute nodes(Note: please named three servers with different hostnames.) and get updates and install git

sudo apt-get update

sudo apt-get install git

Configure controller node

1. Git devstack Juno release:  git clone https://github.com/openstack-dev/devstack.git -b stable/juno

2. Add localrc under devstack and add content to it as following

   HOST_IP=your configured IP address
   MULTI_HOST=1
   FIXED_RANGE=10.0.0.0/24
   FIXED_NETWORK_SIZE=256
   ADMIN_PASSWORD=123456
   MYSQL_PASSWORD=123456
   RABBIT_PASSWORD=123456
   SERVICE_PASSWORD=123456
   SERVICE_TOKEN=123456
   SCREEN_LOGDIR=/home/stack/log/screen
   LOGFILE=/home/stack/log/stack.log
   LOGDAYS=1
   PUBLIC_INTERFACE=eth0 #actual name of your network interface
   FLAT_INTERFACE=eth0

    

3. Run devstack to install openstack: ./stack.sh

Configure compute node

1. Git devstack Juno release:  git clone https://github.com/openstack-dev/devstack.git -b stable/juno

2. Add localrc under devstack and add content as done in controller node. and also add following to it.

   SERVICE_HOST=your controller ip
   
   HOST_IP=host ip
   
   MULTI_HOST=1
   
    FIXED_RANGE=10.0.0.0/24
   
   FIXED_NETWORK_SIZE=256
   
    ADMIN_PASSWORD=123456
   
   MYSQL_PASSWORD=123456
   
   RABBIT_PASSWORD=123456
   
   SERVICE_PASSWORD=123456
   
   SERVICE_TOKEN=123456
   
   SCREEN_LOGDIR=/home/stack/log/screen
   
   LOGFILE=/home/stack/log/stack.log
   
   LOGDAYS=1
   
   #PUBLIC_INTERFACE=eth0
   
   FLAT_INTERFACE=eth0
   
   DATABASE_TYPE=mysql
   
   Q_HOST=$SERVICE_HOST
   
   MYSQL_HOST=$SERVICE_HOST
   
   RABBIT_HOST=$SERVICE_HOST
   
   GLANCE_HOSTPORT=$SERVICE_HOST:9292
   
   KEYSTONE_AUTH_HOST=$SERVICE_HOST
   
   KEYSTONE_SERVICE_HOST=$SERVICE_HOST
   
   ENABLED_SERVICES=n-cpu,n-net,n-api,c-sch,c-api,c-vol

    

3. Add some variables which would use during install openstack
   Copy following info to a shell file:

   #!/bin/bash
       
   export OS_AUTH_URL=http://controller'ip:5000/v2.0 
   export OS_TENANT_ID=69eaaebc44174418956d4dd104f2be76
   export OS_TENANT_NAME="admin"
   
   export OS_USERNAME="admin"
   export OS_PASSWORD=123456
   export OS_VOLUME_API_VERSION=2

    

   Update the OS_AUTH_URL with your controller node ip. And use command:mysql -uroot -p123456 -e "select * from keystone.project;"  to get OS_TENANT_ID and update it, then run this shell

4. Run devstack to install openstack: ./stack.sh

Configure migrations

1. Host name configuration
   Add three server's host names to /etc/hosts to be sure that every node can ping successful to others

   server's IP   server's host name

    

2. Configure /etc/sysconfig/iptables file to allow libvirt listen on TCP port 16509 and add a record accepting KVM communication on TCP port within the range from 49152 to 49261
   Note: ubuntu does not have sysconfig folder under etc, need create it manually.

   -A INPUT -p tcp -m multiport --ports 16509 -m comment --comment "libvirt" -j ACCEPT
   
   -A INPUT -p tcp -m multiport --ports 49152:49216 -m comment --comment "migration" -j ACCEPT

    

3. Configuration libvirt
   Enable libvirt listen flag at /etc/sysconfig/libvirtd file: add

   LIBVIRTD_ARGS="-listen"

    

   Configure /etc/libvirt/libvirtd.conf:

   listen_tls = 0
   listen_tcp = 1
   auth_tcp = "none"

    

   Configure /etc/init/libvirt-bin.conf, modify exec /usr/sbin/libvirtd  to exec /usr/sbin/libvirtd -l
   Configure /etc/libvirt/qemu.conf,  modify security_driver="none"

   Restart libvirt, after executing the command, ensure that libvirt is successfully restarted: sudo stop libvirt-bin && sudo start libvirt-bin

   ps -ef |grep libvirt
   

4. Nova configuration
   Modify:force_config_drive = None(value from always to None) 

   Add:live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE  (under default)

5. Restart service on each node
   screen -s stack -X quit
   ./rejoin-stack.sh

Live migration test

Before start live migration test, to be sure that the nova compute service are ready

      

 

1. create instance and attach volume
   nova boot --flavor=flavor_id --image=image_id instance_name
   nova volume-attach vm_id volume_id auto
   

   +----------+--------------------------------------+
   | Property | Value                                |
   +----------+--------------------------------------+
   | device   | /dev/vdb                             |
   | id       | 3deb171f-8d61-4df9-8e6e-29d86575e78e |
   | serverId | bb12f3b0-c17e-49f7-b482-255258931eb4 |
   | volumeId | 3deb171f-8d61-4df9-8e6e-29d86575e78e |
   +----------+--------------------------------------+

2. How to mount the volume which attached to the instance
   1> execute command to ssh to the instance, for example: ssh cirros@10.0.0.2
   2> execute command to list the partition tables for the installed devices: sudo fdisk -l
        see:Disk /dev/vdb......
   3> create a filesystem on the  device: sudo mkfs.ext4 /dev/vdb
   4> create a dictionary and mount the volume:
        sudo mkdir /data
        sudo mount /dev/vdb /data/
   5> then we can create folder or file on it
   
   
3. nova show instance_id
4. nova live-migration --block_migrate vm_id target_server_hostname
5. check the host before and after migration
   

Issue summary

1. n-api could not start when install openstack with devstack
Solution: Restart controller node and rejoin the service, then run unstack and stack on compute node

2. Get error:libvirtError: operation failed: Failed to connect to remote libvirt URI qemu+tcp://computer247/system: Unable to resolve address 'computer247' service '16509': Name or service not known
Solution: check the iptables in configure migration step 2.

3. Live Migration failure: Invalid value '4-7,12-15' for 'cpuset.cpus': Invalid argument
Solution: Check the CPU info in your compute nodes, if there have different architecture of cpus, it would be fail to migrate

4. Compute node's nova-cpu service could not enable:
libvirt-bin service, then run command in both compute node and controller node:
screen -s stack -X quit
./rejoin-stack.sh

5. Get error:

Solution: It's about nova configuration, check /etc/nova/nova.conf:
Modify:force_config_drive = None(value from always to None)      Add:live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE  (under default)
then rejoin stack in compute node

6. Getting error:screen-n-cpu.2015-06-02-184950.log:2015-06-04 00:16:45.797 ERROR nova.virt.libvirt.driver [-] [instance: d1b56987-4691-4259-897f-ba4ce3e71260] Live Migration failure: Failed to open file '/dev/disk/by-path/ip-9.115.246.45:3260-iscsi-iqn.1986-03.com.ibm:2145.v7k41.node1-lun-4': No such file or directory
Solution:  No solution currently by me, it is a bug and would be fixed

7. Gettting error when rejoin stack:2015-06-09 14:02:33.154 TRACE cinder.openstack.common.threadgroup ArgumentError: Could not parse rfc1738 URL from string ''"

Solution: modify the node sql_connection  in /etc/cinder/cinder.conf and /etc/nova/nova.conf with( It has no value as default): take following as example
/etc/cinder/cinder.conf:  sql_connection = mysql://root:123456@9.115.246.185/cinder?charset=utf8
/etc/nova/nova.conf:   sql_connection = mysql://root:123456@9.115.246.185/nova?charset=utf8