I know network ‘products’ – but I don’t really know networks. So I am a Layer 2 kinda guy and everything Layer 3 is black magic to me.
Nonetheless it was time to do some with it and jump into the deep end. I do have the VCP6-NV, but I JUST about passed with a lot of luck, so I still have a lot of catching up to do.
Installing NSX is the easy part, especially East <> West, but when it comes to North <> South – a whole other world awaits.
I am working currently on a project which involves NSX in a Mutli-Site / vCenter environment. So I thought I am taking this as opportunity to play with OSPF in my own lab, rather than in a customer’s environment (yea I know, boring ey 😛 ).
So it was time to order some stuff and start asking stupid questions (Thanks Myles and Sam for not running away). Also worth watching is Iwan’s Youtube Series about the VCIX-NV.
Whilst it is based on an older version of NSX (IS-IS doesn’t even exist in NSX anymore) – but it still helped a ton.
My broadband router doesn’t support anything but .. well internet. Doesn’t even do static routes, so I needed something else.
The Ubiquiti Edge Router X was perfect. In a nutshell, I am going from my ADSL Broadband Router, to the EdgeRouter X to NSX to vRealize Automation.
The diagram below doesn’t include vRealize Automation – but this is at least the network.
(yay, first diagram on newly acquired whiteboard)
Here I just want to go through the installation. Well, ‘go through’. As you guys might know – my articles usually involve a heck of a lot of screenshots taken at every step on the way.
But this took me a while to get working and I really only have the end result. But – I am sure once you see the screenshots, you can get it done in your own lab 🙂
So I kinda expect people to understand and know NSX already.
Let’s start at the bottom. As this is a lab – I only have one controller – I am skint when it comes to resources 🙂
Here I got one Transport Zone – cunningly called – Transport Zone
Then I have a bunch of Logical Switches. As you can see, I got 4 Switches here. Diagram shows three – but heh – same difference 🙂
I will be using a Logical Switch also for the transit between ESG and DLR – you shall see more in a bit.
Logical Distributed Router
The LDR is as small as it comes – no HA
In terms of interfaces you can see that I have one Uplink and a couple Internal interfaces. You will see the by the naming that I have attached only the Logical Switches to the Internal LDR interfaces. I don’t need a gateway on my transit network (as an internal interface that is).
Still with me ? Good .. let’s continue
Under Routing > Global Configuration you can see that I have configured my ERx as Gateway. The problem is my 1st entry point – the Broadband Router – it doesn’t do anything dynamic – no OSPF – or even static – no static routes – so I need still a way out because my VMs need to be able to get out to the internet and the ADSL router has no way of knowing those Logical Switch Subnets.
Under Dynamic Routing Configuration I select the Uplink as Router ID
Now to the actual OSPF part. Obviously you need to enable OSPF .. What IPs are those ?
- Protocol Address
- This is a new IP in the range of your Transit Network
- Forwarding Address
- That is the internal interface of your LDR
- Area Definition
- This can be anything really. Here I changed the default ‘0’ to ‘100’. As you can see in my diagram, I use Area ‘0’ between ER X and ESG so I leave the area between ESG and LDR separate. ‘100’ it is 😉
- Area to Interface Mapping
- Now map the area just created / changed to the uplink. Important: Make sure your MTU is correctly configured across the board.
If further down the line you notice that OSPF is not working and is stuck in status ExchangeStart/DR, then you might still have an MTU mismatch.
When configuring the mapping, you can select to ignore the MTU
But ideally you fix the MTU, rather just using a plaster 🙂
Last thing to do with the LDR is the Route Distribution .
If you don’t do this – you will end up with an OSPF Neighbour, but no routes are actually being advertised.
Also make sure you allow all connected interfaces.
Now moving further up the chain.
Edge Service Gateway (ESG)
Same thing, just a compact will do for me
The Uplink is connected to the same network as my EdgeRouter X, whilst the Internal interface is connected to the Transit Network
At this stage your Transit Network should have three IPs used
- 172.5.0.1
- LDR
- 172.5.0.2
- ESG
- 172.5.0.3
- OSPF Protocol IP
Under Routing > Global Configuration once again I configured the gateway, to provide a way out and configured the Uplink as Router ID under the Dynamic Routing Configuration and of course enabled OSPF
Under OSPF you then plumb those two OSPF areas together. As you can see in the diagram, the Area between ESG and LDR is Area 100 and the Area between ESG and the Edge Router X is Area 0
So again enable OSPF and add / change the Area 100. Area 0 and 51 are there by default (I won’t be using 51).
Once the new Area has been configured – map the areas to the appropriate uplinks.
- Uplink (Going to Edge Router X)
- Area 0
- Internal (Going to LDR)
- Area 100
Under Route Redistribution once more enable OSPF and allow all connected ports
At this stage you should be able to confirm OSPF is working. SSH to the ESG and run the following command
show ip ospf neighbors
This should now show the OSPF neighbors, in my example the Transit network configured.
Remember, 172.5.0.1 is the ‘External’ interface of the LDR whilst the IP 172.5.0.3 is the OSPF Protocol IP
Note: You can see a second neighbor in the screenshot – 172.16.0.1 – I have already configured my Edge Router X and it can therefore see the Area 0 already.
Next check the subnets. As mentioned, I expect the following
- 172.1.0.0/24
- 172.2.0.0/24
- 172.3.0.0/24
- 172.4.0.0/24 (not in diagram but configured)
Run the following command
show ip route
You can see that the subnets are indeed peered (Again, including the subnet from Area 0 which I have already configured)
Edge Router X
This one is straight forward actually.
- Router ID
- Can be anything – I just use the IP of the internal interface
- Redistribute Connected
- Same as with the ESG / LDR – want to make sure routes are distributed across the board.
- Areas
- Here enter the internal subnet of the Edge Router X and the Area 0 (Internal Edge Router x <> External ESG).
- Interfaces
- Against, same deal. My Edge Router X is configured so every lan port is bridged to a logical switch. Mapping the switch now means it covers all interfaces, not just a particular one.
Using the same commands on the ESG should now show the additional routes and neighbor.
The Dashboard should also now show additional OSPF routes
So the routing table
vRealize Automation
So I talked about vRA as well.
First of all, you need to explain to vRA that there is NSX kicking about.
How to do integrate NSX into vRA can be found in my previous article HERE.
Here I want to do use, and demonstrate, NSX in multiple ways.
- Deploy a Virtual Machine into any of the Logical Switch Subnets
- Add a possible selection to the blueprint
- Deploy a set of Virtual Machines with a load balancer
- Create a NATed Network Profile
- Create a Routed Network Profile
I won’t go through the whole end to end setup for vRA and its components. You can find plenty of articles on my website. I want to keep this short and sweet (actually, won’t be short, but sweet).
Deploy a Virtual Machine into any Logical Switch Subnet (With Selection)
First, I have created two Network Profiles of type External. VMs are essentially plumbed straight into said network and these profiles have the relevant subnets configured (including IP range dedicated to vRA).
Example:
Under the Reservation itself, you can see that I configured those two Logical Switches with the relevant Network Profiles – from vRA perspective, Logical Switches are just Port Groups anyway.
When you create a Blueprint, you can add one of those networks, both or neither. But we want both – as selection, not at the same time.
To achieve this we are using Custom Properties. The property we need is
VirtualMachine.Network0.Name
The number ‘0’ determines the network for the first network card. My Templates only got one NIC, so I use this.
So let’s create a Custom Property
- Name
- Above mentioned Property
- Label
- Can be anything
- Data Type
- String
- Required
- Yes, otherwise if a network is not chosen, it will be selected in RR
- Displayed As
- Dropdown
- Predefined Values
- Name: This appears in the Blueprint and should be descriptive
- Value: This will be the name of the network – in this case the PortGroup
Next we need a Property Group. You cannot add network properties directly onto a VM object.
Now add the newly created property group to the blueprint. Which network you attached to the VM doesn’t matter – the property will overwrite it where required.
Now you can select the network required when requesting the blueprint
The VMs should now deploy into the correct Logical Switch and an IP be allocated accordingly
Deploy Virtual Machines with a Load Balancer
First ensure you configured a Reservation Policy for the Edges and the Transport Zone
Then the Blueprint needs to have multiple machines obviously – here I selected 2 – and drag an On Demand Load Balancer onto the canvas.
I keep it simple here for testing – both VM network and VIP network are in the same network and the VIP must be part of the IP pool configured in the network profile.
You don’t need to enter a VIP manually – you can leave the field empty and it will just randomly select one from the pool.
Once deployed you got your two VMs and Load Balancer
In your Web Client you can then see and administrate the Edge – including the pre-configured pool selected in the Blueprint.
Deploy a VM with a NATed Profile
Here you can see I have created a NAT Profile. 172.1.0.0/24 (which is a Logical Switch) will be the ‘Public’ interface of the VM, whilst 192.162.0.0/24 is the ‘Private’ range.
And of course you need to configure an IP Range (Network Range).
Now rather than dragging an existing network onto the canvas, drag a On-Demand NAT Network onto the canvas and connect it to the VM.
When deployed, you can see the Edge is deployed with its interfaces based on the blueprint
And the VM is connected to the internal (NATed) Edge interface
Once again you can then see the Edge in the Web Client with the pre-configured NAT rules
Create a Routed Network Profile
Almost done 🙂
That takes a bit more work and requires a working NSX environment – ok, they all do, but in this instance even more so.
Let’s recap a few things.
The IP of my ESG here is the 172.5.0.2 / 255.255.255.248
A routed network requires an External Network as well as a Routed Network.
Let’s do this step by step
So here the External Network – You can see I have added the right subnet and as Gateway the IP of the ESG on the transit network (refer to the diagram if needed)
You don’t need to add a range.
Now add a Routed Network Profile
Set the following
- External Network Profile
- External Profile just created (the one attached to the Transit network with the Transit Gateway)
- Subnet Mask
- This is the subnet mask of the Transit Network
- Range Subnet Mask
- This is the subnet mask used by the VMs – here I am using a /29 (no idea why, is testing anyway) – so it can be a /24 or whatnot – up to you and your environment.
- Base IP
- Essentially this becomes the gateway once the Logical Switch is being created – Here I am using 192.168.70.1
Click Generate Network Range – this will now create the required IPs given to the VMs attached to the Logical Switches. Here you can see I have a range of 192.168.70.1 – 6 (.1 is already taken by the gateway)
Now move over to the Reservation and the Network tab.
First, ensure your newly created External Network is attached to its portgroup – or like in my case – Virtual Wire – of the Transit Network
Then scroll down. Select your Transport Zone and Security Groups if you got any (I don’t want them by default in a Security Group …. yet)
Scroll down even further. Under Routed Gateways select your LDR, its uplink and the newly created External Network for the transit.
Next step – Blueprint. If you already have a blueprint and you want to retro-fit it with NSX stuff, click the Settings icon
Select your Transport Zone and Reservation Policy part of the Transit Network / LDR etc. If you follow this article, you probably did that already in Step #1
Now just like any other network object on the canvas (i.e. Existing Network) add a On Demand Routed Network and connect it to your Virtual Machine Object.
Now you should be able to go ahead and deploy a test VM.
Remember, I had a range of 192.168.70.1/29 configured.
So far so good … VM got an IP from said range.
Now can I ping ? Oh yea – right from my Mac Book – which is using the EdgeRouter X as Gateway
Checking if I get out of the VM
Routing table and OSPF DB on ESG
And for good measure – the Edge Router X
There is of course a lot more you can do with NSX in vRA – but this is the hardest bit I think …
Happy Routing 🙂