文档章节

Newton版Openstack创建云主机失败

哓竹
 哓竹
发布于 2017/06/06 11:15
字数 1648
阅读 473
收藏 0

1.错误背景

安装了N版Openstack Ha,在创建云主机时源选择镜像并创建新卷时,云主机创建失败。检查发现,卷其实创建成功了并且可用,可以通过创建的卷启动云主机。使用cirros测试镜像,不会出现问题,使用CentOS7.3会出现这种报错,不仅同时创建2个,就连一次只创建一个也会报错。

2.检查日志

2017-06-06 09:30:28.513 665097 WARNING nova.virt.block_device [req-c5016b48-399b-44f8-adea-a86421f1e129 c49d86f8a39148bb809d2da44811ca3b f19126d9035d4209b363386e8e267129 - - -] Failed to delete volume: dd04f3d2-63cb-45f2-9a31-9ecc523bd4
0f due to Invalid input received: Invalid volume: Volume status must be available or error or error_restoring or error_extending and must not be migrating, attached, belong to a group or have snapshots. (HTTP 400) (Request-ID: req-0bd8e
4b4-6989-4c34-a623-20d2bff98816)
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [req-c5016b48-399b-44f8-adea-a86421f1e129 c49d86f8a39148bb809d2da44811ca3b f19126d9035d4209b363386e8e267129 - - -] [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94] Instance faile
d block device setup
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94] Traceback (most recent call last):
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1586, in _prep_block_device
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     wait_func=self._await_block_device_map_created)
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 514, in attach_block_devices
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     map(_log_and_attach, block_device_mapping)
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 512, in _log_and_attach
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     bdm.attach(*attach_args, **attach_kwargs)
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 404, in attach
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     self._call_wait_func(context, wait_func, volume_api, vol['id'])
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 363, in _call_wait_func
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     {'volume_id': volume_id, 'exc': exc})
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     self.force_reraise()
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     six.reraise(self.type_, self.value, self.tb)
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/nova/virt/block_device.py", line 353, in _call_wait_func
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     wait_func(context, volume_id)
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1258, in _await_block_device_map_created
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     volume_status=volume_status)
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94] VolumeNotCreated: Volume dd04f3d2-63cb-45f2-9a31-9ecc523bd40f did not finish being created even after we waited 189 seconds or 61
attempts. And its status is downloading.
2017-06-06 09:30:28.514 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]
2017-06-06 09:30:28.638 665097 WARNING nova.scheduler.client.report [req-c5016b48-399b-44f8-adea-a86421f1e129 c49d86f8a39148bb809d2da44811ca3b f19126d9035d4209b363386e8e267129 - - -] No authentication information found for placement API
. Placement is optional in Newton, but required in Ocata. Please enable the placement service before upgrading.
2017-06-06 09:30:28.638 665097 WARNING nova.scheduler.client.report [req-c5016b48-399b-44f8-adea-a86421f1e129 c49d86f8a39148bb809d2da44811ca3b f19126d9035d4209b363386e8e267129 - - -] Unable to refresh my resource provider record
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [req-c5016b48-399b-44f8-adea-a86421f1e129 c49d86f8a39148bb809d2da44811ca3b f19126d9035d4209b363386e8e267129 - - -] [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94] Build of insta
nce 30e8a024-d1c0-4a64-aae7-a830e8c6be94 aborted: Block Device Mapping is Invalid.
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94] Traceback (most recent call last):
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1783, in _do_build_and_run_instance
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     filter_properties)
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1943, in _build_and_run_instance
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     'create.error', fault=e)
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     self.force_reraise()
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     six.reraise(self.type_, self.value, self.tb)
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1908, in _build_and_run_instance
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     block_device_mapping) as resources:
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     return self.gen.next()
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2071, in _build_resources
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]     reason=e.format_message())
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94] BuildAbortException: Build of instance 30e8a024-d1c0-4a64-aae7-a830e8c6be94 aborted: Block Device Mapping is Invalid.
2017-06-06 09:30:28.639 665097 ERROR nova.compute.manager [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94]
2017-06-06 09:30:29.364 665097 INFO nova.compute.manager [req-c5016b48-399b-44f8-adea-a86421f1e129 c49d86f8a39148bb809d2da44811ca3b f19126d9035d4209b363386e8e267129 - - -] [instance: 30e8a024-d1c0-4a64-aae7-a830e8c6be94] Took 0.72 secon
ds to deallocate network for instance.

3.错误分析:

底层存储Ceph创建新卷的时候速度太慢,导致nova连接卷超时或者超出连接次数 两个解决办法: 一个是提高ceph读写速度(增加ssd作为journal),条件不允许,此方法作罢。 另一个是寻找是否有相关参数设定连接时间或者连接次数。

4.源码查看

错误信息中看到了_await_block_device_map_created 找到了对应的函数

def _await_block_device_map_created(self, context, vol_id):
        # TODO(yamahata): creating volume simultaneously
        #                 reduces creation time?
        # TODO(yamahata): eliminate dumb polling
        start = time.time()
        retries = CONF.block_device_allocate_retries
        if retries < 0:
            LOG.warning(_LW("Treating negative config value (%(retries)s) for "
                            "'block_device_retries' as 0."),
                        {'retries': retries})
        # (1) treat  negative config value as 0
        # (2) the configured value is 0, one attempt should be made
        # (3) the configured value is > 0, then the total number attempts
        #      is (retries + 1)
        attempts = 1
        if retries >= 1:
            attempts = retries + 1
        for attempt in range(1, attempts + 1):
            volume = self.volume_api.get(context, vol_id)
            volume_status = volume['status']
            if volume_status not in ['creating', 'downloading']:
                if volume_status == 'available':
                    return attempt
                LOG.warning(_LW("Volume id: %(vol_id)s finished being "
                                "created but its status is %(vol_status)s."),
                            {'vol_id': vol_id,
                             'vol_status': volume_status})
                break
            greenthread.sleep(CONF.block_device_allocate_retries_interval)
        raise exception.VolumeNotCreated(volume_id=vol_id,
                                         seconds=int(time.time() - start),
                                         attempts=attempt,
                                         volume_status=volume_status)

有两个参数block_device_allocate_retries和block_device_allocate_retries_interval,看名字可以知道一个是尝试次数,一个是尝试间隔

5.解决经过

参数默认值

block_device_allocate_retries_interval=3
block_device_allocate_retries=60

先修改间隔
block_device_allocate_retries_interval=10

重启nova-compute

systemctl restart openstack-nova-compute.service

再以同样的方式创建一台云主机,发现可以创建成功。

6.总结

Newton版本创建云主机可以同时批量从云硬盘创建云主机,相比先前自己用的kilo版要方便许多,对于一次创建多台,上面两个参数可能还需调整,多次尝试应该会找到合适的参数值。

参考了http://blog.csdn.net/onlyellow/article/details/51200267, 我们的环境差不多,对于haproxy的转发方式以及galera的参数调整需要进一步测试才能知道。

© 著作权归作者所有

共有 人打赏支持
哓竹
粉丝 4
博文 47
码字总数 52741
作品 0
朝阳
运维
私信 提问
OpenStack推出最新版本Newton,显著提升单一云平台 对虚拟化、裸机和容器的管理

2016年10月10日-北京- OpenStack社区日前发布其命名为“Newton”的第14个版本。OpenStack是用于构建云的部署最广泛的开源软件。此次推出的新功能包括:Ironic裸机开通服务,Magnum容器编排集...

玄学酱
05/11
0
0
openstack之(一)基础环境

Openstack的概述Openstack是一个由NASA(美国国家航空航天局)和Rackspace合作研发并发起的,以Apache许可证授权的自由软件和开放源代码项目。 Openstack是一个开源的云计算管理平台项目,由...

runs0517
05/10
0
0
2017新年OpenStack南京Meetup分享会

今年10月,OpenStack最新版本Newton发布,在社区公布的代码贡献名单中,中国23家企业组团上榜,中国科技军团已经成为全球OpenStack技术的中坚力量。2016年作为OpenStack在中国企业落地的元年...

Alaise
2016/12/20
98
0
OpenStack centos版安装(二)

在OpenStack第一天文档翻译后,丁丁努力坚持每晚抽时间翻译OpenStack官方网站提供的安装与部署指南,本文翻译自官方安装与部署指南的前四章内容,前四章内容主要是对OpenStack整体安装步骤以...

蓝狐乐队
2014/04/18
0
0
openstack项目中遇到的各种问题总结 其三(命令汇总)

三、注意事项 3.1、挂载目录是需要注意的事项 把分区挂载到目录下的操作要谨慎小心 对于存放系统重要文件的目录不要挂载,对于目录下有重要文件的目录需要做备份,因为挂载过程中会把目录清空...

断臂人
05/06
0
0

没有更多内容

加载失败,请刷新页面

加载更多

微服务分布式事务实现

https://www.processon.com/view/link/5b2144d7e4b001a14d3d2d30

WALK_MAN
今天
2
0
《大漠烟尘》读书笔记及读后感文章3700字

《大漠烟尘》读书笔记及读后感文章3700字: 在这个浮躁的社会里,你有多久没有好好读完一本书了? 我们总觉得自己和别人不一样,所以当看到别人身上的问题时,很少有“反求诸己”,反思自己。...

原创小博客
今天
3
0
大数据教程(9.5)用MR实现sql中的jion逻辑

上一篇博客讲解了使用jar -jar的方式来运行提交MR程序,以及通过修改YarnRunner的源码来实现MR的windows开发环境提交到集群的方式。本篇博主将分享sql中常见的join操作。 一、需求 订单数据表...

em_aaron
今天
3
0
十万个为什么之什么是resultful规范

起源 越来越多的人开始意识到,网站即软件,而且是一种新型的软件。这种"互联网软件"采用客户端/服务器模式,建立在分布式体系上,通过互联网通信,具有高延时(high latency)、高并发等特点...

尾生
今天
3
0
Terraform配置文件(Terraform configuration)

Terraform配置文件 翻译自Terraform Configuration Terraform用文本文件来描述设备、设置变量。这些文件被称为Terraform配置文件,以.tf结尾。这一部分将讲述Terraform配置文件的加载与格式。...

buddie
今天
5
0

没有更多内容

加载失败,请刷新页面

加载更多

返回顶部
顶部