Diagnostics and Troubleshooting

SCM Controller Services

All of the following services should be running on the SCM controller with the given ports, and writing to the given logs.

The Non HA Port is used by services in single-controller setups. In dual-controller setups, HAProxy listens on the service’s normal (non HA) port and forwards to the service listening at the HA port.

You can check if a port is listening with Linux commands like:

# lsof -i :port
Service Description Non HA Port HA Port Log
Scyld Cloud Portal SCM Web Dashboard 443 7690 /var/log/httpd/cloudportal/ /var/www/wsgi/cloudportal/cloudportal.log
Scyld Cloud Auth SCM Authentication and Authorization 443 7691 /var/log/httpd/cloudauth/ /var/www/wsgi/cloudauth/cloudauth.log
Scyld Cloud Accountant SCM Accounting 443 7692 /var/log/httpd/cloudaccountant/ /var/www/wsgi/cloudaccountant/cloudaccountant.log
Scyld Cloud Controller SCM-OpenStack Interface 443 7693 /var/log/httpd/cloudcontroller/ /var/www/wsgi/cloudcontroller/cloudcontroller.log
Redis In-memory database 6379   /var/log/redis/redis-server.log
Haproxy (if present) HA proxy service 8080 (Stats)   /var/log/haproxy.log

OpenStack Services

OpenStack services are run inside Docker containers running on each OpenStack controller.

To check the health of the Docker containers on a controller:

# docker ps -a
[root@cc1-d ~]# docker ps
CONTAINER ID        IMAGE                                                              COMMAND             CREATED             STATUS              PORTS               NAMES
9e97dbdf37a9        k4e.scm:4000/kolla/centos-binary-grafana:3.0.2                     "kolla_start"       2 weeks ago         Up 2 weeks                              grafana
3d0d128f4b32        k4e.scm:4000/kolla/centos-binary-gnocchi-statsd:3.0.2              "kolla_start"       2 weeks ago         Up 2 weeks                              gnocchi_statsd
16e1bbb9f94d        k4e.scm:4000/kolla/centos-binary-gnocchi-metricd:3.0.2             "kolla_start"       2 weeks ago         Up 2 weeks                              gnocchi_metricd
2f137db7e998        k4e.scm:4000/kolla/centos-binary-gnocchi-api:3.0.2                 "kolla_start"       2 weeks ago         Up 2 weeks                              gnocchi_api
1fc44027a3eb        k4e.scm:4000/kolla/centos-binary-horizon:3.0.2                     "kolla_start"       2 weeks ago         Up 2 weeks                              horizon
ee806258eeca        k4e.scm:4000/kolla/centos-binary-heat-engine:3.0.2                 "kolla_start"       2 weeks ago         Up 2 weeks                              heat_engine
8c48feaa647a        k4e.scm:4000/kolla/centos-binary-heat-api-cfn:3.0.2                "kolla_start"       2 weeks ago         Up 2 weeks                              heat_api_cfn
b03f7cf87248        k4e.scm:4000/kolla/centos-binary-heat-api:3.0.2                    "kolla_start"       2 weeks ago         Up 2 weeks                              heat_api
ee130fc0fbf9        k4e.scm:4000/kolla/centos-binary-cinder-backup:3.0.2               "kolla_start"       2 weeks ago         Up 2 weeks                              cinder_backup
1152a577e92a        k4e.scm:4000/kolla/centos-binary-cinder-volume:3.0.2               "kolla_start"       2 weeks ago         Up 2 weeks                              cinder_volume
cf6a9c0e015d        k4e.scm:4000/kolla/centos-binary-cinder-scheduler:3.0.2            "kolla_start"       2 weeks ago         Up 2 weeks                              cinder_scheduler
10c43d513d5b        k4e.scm:4000/kolla/centos-binary-cinder-api:3.0.2                  "kolla_start"       2 weeks ago         Up 2 weeks                              cinder_api
5c4a5573e3a2        k4e.scm:4000/kolla/centos-binary-neutron-metadata-agent:3.0.2      "kolla_start"       2 weeks ago         Up 2 weeks                              neutron_metadata_agent
a97881bd17f2        k4e.scm:4000/kolla/centos-binary-neutron-lbaas-agent:3.0.2         "kolla_start"       2 weeks ago         Up 2 weeks                              neutron_lbaas_agent
d106b0e253ca        k4e.scm:4000/kolla/centos-binary-neutron-l3-agent:3.0.2            "kolla_start"       2 weeks ago         Up 2 weeks                              neutron_l3_agent
7530545a141f        k4e.scm:4000/kolla/centos-binary-neutron-dhcp-agent:3.0.2          "kolla_start"       2 weeks ago         Up 2 weeks                              neutron_dhcp_agent
a8d52deb9ed2        k4e.scm:4000/kolla/centos-binary-neutron-openvswitch-agent:3.0.2   "kolla_start"       2 weeks ago         Up 2 weeks                              neutron_openvswitch_agent
0e3077090730        k4e.scm:4000/kolla/centos-binary-neutron-server:3.0.2              "kolla_start"       2 weeks ago         Up 2 weeks                              neutron_server
0f3007c497a1        k4e.scm:4000/kolla/centos-binary-nova-conductor:3.0.2              "kolla_start"       2 weeks ago         Up 2 weeks                              nova_conductor
9e5461eab29c        k4e.scm:4000/kolla/centos-binary-nova-spicehtml5proxy:3.0.2        "kolla_start"       2 weeks ago         Up 2 weeks                              nova_spicehtml5proxy
a7e1e000587d        k4e.scm:4000/kolla/centos-binary-nova-scheduler:3.0.2              "kolla_start"       2 weeks ago         Up 2 weeks                              nova_scheduler
fe20eb20c8d5        k4e.scm:4000/kolla/centos-binary-nova-consoleauth:3.0.2            "kolla_start"       2 weeks ago         Up 2 weeks                              nova_consoleauth
da68b9764cd4        k4e.scm:4000/kolla/centos-binary-nova-api:3.0.2                    "kolla_start"       2 weeks ago         Up 2 weeks                              nova_api
aac9176c907e        k4e.scm:4000/kolla/centos-binary-glance-api:3.0.2                  "kolla_start"       2 weeks ago         Up 2 weeks                              glance_api
0a096c67003e        k4e.scm:4000/kolla/centos-binary-glance-registry:3.0.2             "kolla_start"       2 weeks ago         Up 2 weeks                              glance_registry
37202d5529f7        k4e.scm:4000/kolla/centos-binary-keystone:3.0.2                    "kolla_start"       2 weeks ago         Up 2 weeks                              keystone
11dd5af66a86        k4e.scm:4000/kolla/centos-binary-rabbitmq:3.0.2                    "kolla_start"       2 weeks ago         Up 2 weeks                              rabbitmq
7e65e2a8825b        k4e.scm:4000/kolla/centos-binary-mariadb:3.0.2                     "kolla_start"       2 weeks ago         Up 2 weeks                              mariadb
cca6cdc74245        k4e.scm:4000/kolla/centos-binary-memcached:3.0.2                   "kolla_start"       2 weeks ago         Up 2 weeks                              memcached
796247e2a783        k4e.scm:4000/kolla/centos-binary-kibana:3.0.2                      "kolla_start"       2 weeks ago         Up 2 weeks                              kibana
e0d6f0a0c25e        k4e.scm:4000/kolla/centos-binary-keepalived:3.0.2                  "kolla_start"       2 weeks ago         Up 2 weeks                              keepalived
9abb6ca64f2d        k4e.scm:4000/kolla/centos-binary-haproxy:3.0.2                     "kolla_start"       2 weeks ago         Up 2 weeks                              haproxy
cb4f91e02914        k4e.scm:4000/kolla/centos-binary-elasticsearch:3.0.2               "kolla_start"       2 weeks ago         Up 2 weeks                              elasticsearch
4cf2280d2854        k4e.scm:4000/kolla/centos-binary-ceph-rgw:3.0.2                    "kolla_start"       2 weeks ago         Up 2 weeks                              ceph_rgw
c34e2b58edba        k4e.scm:4000/kolla/centos-binary-ceph-osd:3.0.2                    "kolla_start"       2 weeks ago         Up 2 weeks                              ceph_osd_0
c0349b4d3650        k4e.scm:4000/kolla/centos-binary-ceph-mon:3.0.2                    "kolla_start"       2 weeks ago         Up 2 weeks                              ceph_mon
afa384731c7b        k4e.scm:4000/kolla/centos-binary-cron:3.0.2                        "kolla_start"       2 weeks ago         Up 2 weeks                              cron
e3003c071f19        k4e.scm:4000/kolla/centos-binary-kolla-toolbox:3.0.2               "kolla_start"       2 weeks ago         Up 2 weeks                              kolla_toolbox
c7eee2e64766        k4e.scm:4000/kolla/centos-binary-heka:3.0.2                        "kolla_start"       2 weeks ago         Up 2 weeks                              heka

Log files for each service are available from Docker by running docker logs SERVICE_NAME or under /var/lib/docker/volumes/kolla_logs/_data/SERVICE_NAME.

A web dashboard for logs, Kibana, is also available at the same hostname that provides the Horizon dashboard at port 5601.

Ceph

To get a quick Ceph status check, run ceph health on a controller. You should get this response:

# ceph health
HEALTH_OK

If the Ceph noout option has been set, you’ll get this response instead:

# ceph health
HEALTH_WARN noout flag(s) set

More details are available with ceph -s:

# ceph -s
    cluster 8c213314-82f6-44ea-bcdc-f8d44a93186b
     health HEALTH_WARN
            noout flag(s) set
     monmap e1: 4 mons at {cc1-d=10.10.4.1:6789/0,vmhost1-d=10.10.4.3:6789/0,vmhost2-d=10.10.4.4:6789/0,vmhost3-d=10.10.4.5:6789/0}
            election epoch 76, quorum 0,1,2,3 cc1-d,vmhost1-d,vmhost2-d,vmhost3-d
     osdmap e42: 4 osds: 4 up, 4 in
            flags noout
      pgmap v19117: 384 pgs, 3 pools, 13272 MB data, 1674 objects
            26745 MB used, 3678 GB / 3704 GB avail
                 384 active+clean
  client io 11374 kB/s rd, 9661 B/s wr, 188 op/s