{"id":886,"date":"2014-11-05T15:37:20","date_gmt":"2014-11-05T20:37:20","guid":{"rendered":"https:\/\/www.softwareab.net\/wordpress\/?p=886"},"modified":"2017-09-01T09:16:59","modified_gmt":"2017-09-01T14:16:59","slug":"openstack-debugging-neutron-dhcp","status":"publish","type":"post","link":"https:\/\/www.softwareab.net\/wordpress\/openstack-debugging-neutron-dhcp\/","title":{"rendered":"Debugging Neutron DHCP Failures on OpenStack"},"content":{"rendered":"

Hi All. Today’s adventure touches on a common OpenStack Neutron theme: “I just created a VM. It’s running. It says it has an IP address. And I can’t ping it!!” Read on for tips on debugging Neutron DHCP Failures on OpenStack that we use in our lab.<\/p>\n

<\/p>\n

The Problem with Neutron<\/h3>\n

Here is the deal: I created a VM against a standard network and it had an assigned IP as follows:<\/p>\n

\r\n[l.abruce@co1 rc_scripts(lvosksclu100-rc-admin)]$ nova --os-tenant-name=infrastructure show testnew<\/strong>\r\n+--------------------------------------+----------------------------------------------------------------+\r\n| Property                             | Value                                                          |\r\n+--------------------------------------+----------------------------------------------------------------+\r\n[...output cut...]<\/em>\r\n| infrastructure-net network           | 10.0.0.10                                                      |<\/strong>\r\n[...output cut...]<\/em>\r\n+--------------------------------------+----------------------------------------------------------------+\r\n<\/pre>\n
You see that, right? 10.0.0.10? That’s expected as I’m using per-tenant private networking. But on the actual running VM, I had a teensy problem: DHCP was timing out. Even though OpenStack thought my VM had that 10.0.0.10 IP address, my running VM didn’t actually get it.<\/p>\n
Bummer.<\/p>\n
So let’s look at what happens. In my setup (using Neutron GRE) the flow is:<\/p>\n
\r\n**PHYSICAL Nova Compute HOST***\r\nVM -> qbridge host -> tap device -> br-int -> br-tun\r\n\r\n-> [physical switch]<\/em> ->\r\n\r\n**Neutron Controller (running as a VM)**\r\nbr-tun -> br-int -> [DHCP namespace] -> DHCP process (dnsmasq)\r\n<\/pre>\n
The problem could be happening at any point in the mix, but I have other VMs that I can start \/ restart just fine. So let’s look explicitly at the DHCP server.<\/p>\n
Locate the DHCP Namespace<\/h3>\n
Notice above where I used [DHCP namespace]<\/code> as an identifier. To permit network isolation, Neutron creates a separate network namespace<\/a> so that each per-tenant network can have its own router, its own DHCP server, and so on. To locate this namespace, first find out your network ID; here’s a command for you:<\/p>\n
\r\n[l.abruce@co1 rc_scripts(lvosksclu100-rc-admin)]$ nova network-list | grep \" infrastructure-net \"<\/strong>\r\n| 88513920-091b-4bee-a47c-5636593c69f5 | infrastructure-net     | None |\r\n<\/pre>\n
So my network ID is 88513920-091b-4bee-a47c-5636593c69f5<\/code>. Armed with that information, let’s go to the Neutron Controller node and look for the DHCP namespace:<\/p>\n
\r\n[root@lvosneutr100 ~]# ip netns | grep 88513920-091b-4bee-a47c-5636593c69f5<\/strong>\r\nqdhcp-88513920-091b-4bee-a47c-5636593c69f5\r\n<\/pre>\n
Woohoo, we found it. Now let’s look at the network interfaces associated with that namespace:<\/p>\n
\r\n[root@lvosneutr100 ~]# ip netns exec qdhcp-88513920-091b-4bee-a47c-5636593c69f5 ip a<\/strong>\r\n36: tap315b3aa8-c3:  mtu 1500 qdisc noqueue state UNKNOWN \r\n    link\/ether fa:16:3e:86:d8:5a brd ff:ff:ff:ff:ff:ff\r\n    inet 10.0.0.3\/24 brd 10.0.0.255 scope global tap315b3aa8-c3\r\n    inet6 fe80::f816:3eff:fe86:d85a\/64 scope link \r\n       valid_lft forever preferred_lft forever<\/em>\r\n<\/pre>\n
We’ve got our DHCP namespace and we know the network interface, so we’re closing in on the problem.<\/p>\n
Watch DHCP Traffic<\/h3>\n
Now, we don’t just want to watch *all* DHCP traffic, only the traffic specific to our failing tenant network (with the failing VM on it). Let’s go over what we know so far:<\/p>\n