OpenStack: Fix “Missing” External IPs in Neutron

Have you ever assigned a “floating” IP on your external network to an OpenStack VM…but not been able to ping it? Read our article for info on how we workaround the problem!

Here’s the scenario: you’ve just created a VM and you’ve just allocated a “floating” IP from your external network. You see that the external IP is properly associated:

[l.abruce@co1]$ nova --os-tenant-name=foobar list
+--------------------------------------+------+--------+-------------------------------------+
| ID                                   | Name | Status | Networks                            |
+--------------------------------------+------+--------+-------------------------------------+
| 39612446-a2b9-4b6b-ab25-08af0d3a0f85 | vm1  | ACTIVE | foobar-net=10.0.0.40, 172.20.132.33 |
| a288bdce-a0c4-4ba8-b6d7-28274ca47fa1 | vm2  | ACTIVE | foobar-net=10.0.0.3, 172.20.132.23  |
+--------------------------------------+------+--------+-------------------------------------+

In the above, we see that the vm2 has internal IP of 10.0.0.40 and external IP 172.20.132.33.

The problem? You try to ping that IP address…and, nothing. Nada. Zilch. Zero.

The solution? Normally, a lot of frantic searching on Google, mass restarts of Neutron services, and serious hand-wringing.

But no more! While the *root cause* is that Neutron notification via AMQP is failing to catch the changed IP, we won’t solve that now (or World Hunger, for that matter). But we will get you back up and runnning; no need to restart services, interrupt clients, etc.

Here’s the process:

  1. Find the Router ID. For the tenant in question, get the ID of the actual Router object you created:
    [l.abruce@co1 rc_scripts(lvosksclu100-rc-admin)]$ neutron --os-tenant-name=foobar router-list | grep foobar
    | 418f9a8d-b994-44eb-86b2-c5fa0ea9d732 | foobar-router | {"network_id": "cab649b6-c4b2-4da5-af58-15319d244abf", "enable_snat": true} |
    

    The Router ID is 418f9a8d-b994-44eb-86b2-c5fa0ea9d732.

  2. Check the Router Configuration on your Neutron Controller. We logon to our system, become root, and check out the current settings using these commands.
    • First, verify you can hit the VM on its internal IP. If you can’t…this article won’t help you (you have a different issue or DHCP failure):
      [root@neutron ~]# ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 ping -c 4 10.0.0.40
      PING 10.0.0.40 (10.0.0.40) 56(84) bytes of data.
      64 bytes from 10.0.0.40: icmp_seq=1 ttl=64 time=1.80 ms
      64 bytes from 10.0.0.40: icmp_seq=2 ttl=64 time=0.761 ms
      64 bytes from 10.0.0.40: icmp_seq=3 ttl=64 time=0.618 ms
      64 bytes from 10.0.0.40: icmp_seq=4 ttl=64 time=0.643 ms
      
      --- 10.0.0.40 ping statistics ---
      4 packets transmitted, 4 received, 0% packet loss, time 3003ms
      rtt min/avg/max/mdev = 0.618/0.956/1.802/0.491 ms
      
    • Assuming that you have connectivity to the internal IP, now get all of the *external* IP addresses bound to that Router:
      [root@neutron ~]# ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 ip a
      13: qg-ad0b02aa-38:  mtu 9000 qdisc noqueue state UNKNOWN 
          link/ether fa:16:3e:f9:ec:24 brd ff:ff:ff:ff:ff:ff
          inet 172.20.132.8/18 brd 172.20.191.255 scope global qg-ad0b02aa-38
          inet 172.20.132.23/32 brd 172.20.132.23 scope global qg-ad0b02aa-38
          inet6 fe80::f816:3eff:fef9:ec24/64 scope link 
             valid_lft forever preferred_lft forever
      46: qr-d7b86f4d-90:  mtu 1500 qdisc noqueue state UNKNOWN 
          link/ether fa:16:3e:8b:99:e3 brd ff:ff:ff:ff:ff:ff
          inet 10.0.0.1/24 brd 10.0.0.255 scope global qr-d7b86f4d-90
          inet6 fe80::f816:3eff:fe8b:99e3/64 scope link 
             valid_lft forever preferred_lft forever
      54: lo:  mtu 16436 qdisc noqueue state UNKNOWN 
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
          inet 127.0.0.1/8 scope host lo
          inet6 ::1/128 scope host 
             valid_lft forever preferred_lft forever
      

      The above shows that we have two “external” addresses (172.20.132.8 and 172.20.132.23) bound to this Router. The 172.20.132.8 is the IP assigned to the Router object itself, the 172.20.132.23 is for another running VM…so – we are missing 172.20.132.33!

    • Next, get the iptables information for that Router:
      [root@neutron ~]# ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 iptables -S -t nat
      -P PREROUTING ACCEPT
      -P POSTROUTING ACCEPT
      -P OUTPUT ACCEPT
      -N neutron-l3-agent-OUTPUT
      -N neutron-l3-agent-POSTROUTING
      -N neutron-l3-agent-PREROUTING
      -N neutron-l3-agent-float-snat
      -N neutron-l3-agent-snat
      -N neutron-postrouting-bottom
      -A PREROUTING -j neutron-l3-agent-PREROUTING 
      -A POSTROUTING -j neutron-l3-agent-POSTROUTING 
      -A POSTROUTING -j neutron-postrouting-bottom 
      -A OUTPUT -j neutron-l3-agent-OUTPUT 
      -A neutron-l3-agent-OUTPUT -d 172.20.132.23/32 -j DNAT --to-destination 10.0.0.3 
      -A neutron-l3-agent-POSTROUTING ! -i qg-ad0b02aa-38 ! -o qg-ad0b02aa-38 -m conntrack ! --ctstate DNAT -j ACCEPT 
      -A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697 
      -A neutron-l3-agent-PREROUTING -d 172.20.132.23/32 -j DNAT --to-destination 10.0.0.3 
      -A neutron-l3-agent-float-snat -s 10.0.0.3/32 -j SNAT --to-source 172.20.132.23 
      -A neutron-l3-agent-snat -j neutron-l3-agent-float-snat 
      -A neutron-l3-agent-snat -s 10.0.0.0/24 -j SNAT --to-source 172.20.132.8 
      -A neutron-postrouting-bottom -j neutron-l3-agent-snat
      

      Look for the values associated with the external IPs, we’ll add a few more entries.

  3. Add the missing external IP address to your Router object:
    [root@neutron ~]# ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 ip a add 172.20.132.33/32 broadcast 172.20.132.33 scope global dev qg-ad0b02aa-38
    

    You get no output from that command unless you have a failure. Let’s verify:

    [root@neutron ~]# ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 ip a show
    13: qg-ad0b02aa-38:  mtu 9000 qdisc noqueue state UNKNOWN 
        link/ether fa:16:3e:f9:ec:24 brd ff:ff:ff:ff:ff:ff
        inet 172.20.132.8/18 brd 172.20.191.255 scope global qg-ad0b02aa-38
        inet 172.20.132.23/32 brd 172.20.132.23 scope global qg-ad0b02aa-38
        inet 172.20.132.33/32 brd 172.20.132.33 scope global qg-ad0b02aa-38
        inet6 fe80::f816:3eff:fef9:ec24/64 scope link 
           valid_lft forever preferred_lft forever
    [...output cut...]
    

    We have our missing external IP address…now let’s setup routing for it.

  4. To setup routing, we issue a few simple commands in the Router object’s context.
    • First, map (NAT) outgoing calls from the internal IP address to the external IP:
      ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 iptables -t nat -A neutron-l3-agent-OUTPUT -d 172.20.132.33/32 -j DNAT --to-destination 10.0.0.40
      
    • Next, map incoming calls to the external IP to the internal IP address:
      ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 iptables -t nat -A neutron-l3-agent-PREROUTING -d 172.20.132.33/32 -j DNAT --to-destination 10.0.0.40
      
    • Finally, setup source NAT from the internal IP to the external IP:
      ip netns exec qrouter-418f9a8d-b994-44eb-86b2-c5fa0ea9d732 iptables -t nat -A neutron-l3-agent-float-snat -s 10.0.0.40/32 -j SNAT --to-source 172.20.132.33
      

And – that is it! Now let’s verify we can get to the external IP from any other node (like your local laptop):

[l.abruce@co1]$ ping -c 4 172.20.132.33
PING 172.20.132.33 (172.20.132.33) 56(84) bytes of data.
64 bytes from 172.20.132.33: icmp_seq=1 ttl=62 time=5.90 ms
64 bytes from 172.20.132.33: icmp_seq=2 ttl=62 time=11.2 ms
64 bytes from 172.20.132.33: icmp_seq=3 ttl=62 time=13.0 ms
64 bytes from 172.20.132.33: icmp_seq=4 ttl=62 time=2.17 ms

--- 172.20.132.33 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3006ms
rtt min/avg/max/mdev = 2.175/8.077/13.035/4.300 ms

Success!

Team-oriented systems mentor with deep knowledge of numerous software methodologies, technologies, languages, and operating systems. Excited about turning emerging technology into working production-ready systems. Focused on moving software teams to a higher level of world-class application development. Specialties:Software analysis and development...Product management through the entire lifecycle...Discrete product integration specialist!

Posted in Network, OpenStack, Teknocratica Tagged with: , , , ,
One comment on “OpenStack: Fix “Missing” External IPs in Neutron
  1. I have been looking for this solution for months! See OpenStack Questions.

    Many thanks for a very clear and “obvious” solution. I was on the point of removing OpenStack and trying DevStack, in the faint hope …

    Go not gentle in to the cold, dark night
    – D. Thomass —- Happpy 100 Dylan.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

Human Verification: In order to verify that you are a human and not a spam bot, please enter the answer into the following box below based on the instructions contained in the graphic.