Cisco and Networking

Post regarding Cisco and general networking issues and their fixes.

Kemp Load Master and Palo Alto Firewall - Random Packet Drops and Disconnections

posted 18 Jul 2020, 04:21 by Tristan Self

We had a rather irritating issue whereby we were seeing intermittent packet drops and connection failures on our Kemp Load Master.

The Kemp Load Master sat inbetween a Palo Alto Firewall within a DMZ zone. Client connections from the Internet would be directed to the Kemp Load Master in the DMZ, which would then make the onward connection to the internal Microsoft Exchange Server cluster, ADFS servers, Shibboleth servers, and other services offered.

We started to see these issues logs on the Kemp Load Master:

Jul 18 11:42:58 kemp-lb-01.domain.co.uk vsslproxy: reencrypt(116) - connect failed to xxx.xxx.xxx.xxx:443 (errno 110) 

The (116) means the Virtual Service on the Kemp and this also ties to one of the back end real servers providing service, as you can see connections were failing to any of the real servers.

https://kemptechnologies.com/faq/troubleshoot/

According to Kemp, this means the connection from the Kemp was being interrupted when on its way to the internal real server.

We also had sporadic issues from some clients, many worked fine, but some were reporting slowness or were finding their connections would drop from time to time. Specifically it was Outlook Web Access, Outlook, Outlook for Mac, other Exchange EWS clients and ActiveSync clients that were seeing the issues. Other services published via the Kemp Load Master were seeming unaffected.

Solution


The issue and fix is described in the above articles.

Essentially, the "Challenge-Ack" mechanism was not being handled by the Palo Alto firewall, leading to the connecting being reset.

  • The following counters were observed.
                              tcp_drop_packet                        1        0 warn      tcp       pktproc   packets dropped because of failure in tcp reassembly
              tcp_drop_out_of_wnd                    2        0 warn      tcp       resource  out-of-window packets dropped
  • Basically the  TCP connection between both client and server enters into a hung state. In other words, the client keeps on trying to establish a new connection while the server continues to respond with a challenge ACK.
To fix the issue, all that was required was to turn on "allow-challenge-ack" on the Palo Alto Firewall:
>configure
#set deviceconfig setting tcp allow-challenge-ack yes
#commit
#exit
>

Note that this only affects PanOS 8.0.7 onwards, from this version onwards this functionality is added, but turned off. You must manually turn "allow-challenge-ack" to yes. Earlier versions of PanOS this was not an issue.


HP BladeSystem Flex 10 Ethernet Switch Configuration

posted 19 Aug 2016, 07:46 by Tristan Self

Had to do a bit of configuration on an HP c7000 blade chassis the other day. There was a need to have two separate uplink pairs going to two different pairs of switches from the single chassis. I needed to provide both uplink paths to each host within the chassis (they were ESXi hosts).

Here is the configuration to do this for one of the uplink pairs, for the other uplink pair you just do the same thing, but applying the server-port-map-range to different physical ports on the blade. Each blade has 8 pNICs (4 on the LOM and 4 on a mezzanine card).

In this example ports 3 and 4 are going to be used to provide active/standby paths through the Flex-10 fabric to the physical network beyond.

add uplinkset VLAN_Trunk_1 ConnectionMode=Failover
add uplinkport enc0:1:X4 UplinkSet=VLAN_Trunk_1 Speed=Auto Role=Primary
add uplinkport enc0:2:X4 UplinkSet=VLAN_Trunk_1 Speed=Auto Role=Secondary

add network vlan1 -quiet UplinkSet=VLAN_Trunk_1 VLanID=552 NativeVLAN=Enabled NAGs=Default
set network vlan1 SmartLink=Enabled

add network vlan2 -quiet UplinkSet=VLAN_Trunk_1 VLanID=555 NAGs=Default
set network vlan2 SmartLink=Enabled

add network vlan3 -quiet UplinkSet=VLAN_Trunk_1 VLanID=554 NAGs=Default
set network vlan3 SmartLink=Enabled


If you want vlan1 to be untagged to the host where these 3 vlans are trunked to the host on ports 2 and 3, you need an extra bit of configuration:

set server-port-map serverprofile1:3 vlan1 untagged=true
set server-port-map serverprofile1:4 vlan1 untagged=true





Cisco ASA Firewall - DMZ to Inside Access

posted 15 Jul 2015, 09:13 by Tristan Self   [ updated 15 Jul 2015, 09:26 ]

On a Cisco ASA firewall you will probably want to use the DMZ for servers that are web facing, and also restrict/deny any access they have to the internal network. The idea being that a connection to a web server say in your DMZ would get into the DMZ, and if another connection is required e.g. for a database lookup, it would be made back through the firewall between the DMZ and inside to the internal server.

When a firewall is brand new and un-configured, there is an implicit incoming rule on the interface saying "any less secure networks" so this means that if you have configured the interface security levels as follows:

outside = 0
inside = 100
dmz = 50

then the following is true:

A host on the inside (internal network) can access anything on the dmz or outside (internet) using the default implicit rules.
A host on the dmz (DMZ network) can access anything on the outside (internet) but not the inside (internal network) using the default implicit rules. Because it only allows "any less secure networks".

But at the point of adding your own rules to the ACL this implicit rule is removed and then an implicit deny is added to the end, and then only your traffic is allowed.

So onto the point of this article. A scenario to explain it all.


If you have a host on your DMZ network and you want it to only be allowed to access the internet on HTTP or HTTPS (i.e. DMZ to outside) and allowed access to the internal network on TCP/1433 (SQL) (i.e. DMZ to inside) you need to setup the commands below.

The rule set is basically split into 4 sections below to allow traffic to the inside host from the DMZ, traffic from the DMZ to the internet, block all other traffic from the DMZ to the inside and block anything else.

1. Allow specific traffic from the DMZ to the inside.access-list dmz_access_in line 1 extended permit tcp host 192.168.101.50 object inside-network eq sqlnet
2. Deny all other traffic from the DMZ to the inside.access-list dmz_access_in line 2 extended deny ip host 192.168.101.50 inside-network
3. Allow specific traffic from the DMZ to the outside.

access-list dmz_access_in line 3 extended permit tcp host 192.168.101.50 any4 eq http

access-list dmz_access_in line 4 extended permit tcp host 192.168.101.50 any4 eq https

4. Block Everything else.access-list dmz_access_in line 5 extended deny ip any any

Cisco ASA NAT Exemption (post version 8.3)

posted 13 Jul 2015, 08:52 by Tristan Self

The changes to the ASA IOS post version 8.3 changes the way that NAT works. NAT Exemption is normally used to disable translation for certain addresses e.g. for VPN tunnelling.

So for this example below you create an access-list containing the IP addresses that are to be exempted from NAT. So say these are the site to site VPN addresses where 192.168.0.0/24 is the A end, and 172.31.0.0 255.255.255.0 is the B end.

 
# access-list NAT_EXEMPT extended permit ip 192.168.0.0 255.255.255.0 172.31.0.0 255.255.255.0
# nat (inside) 0 access-list NAT_EXEMPT

This basically says traffic going through the firewall from 192.168.0.0/24 to 172.31.0.0/24 through the firewall should not be NATTed.

Cisco ASA Static NAT (pre version 8.3)

posted 13 Jul 2015, 06:35 by Tristan Self

Pre version 8.3 to statically NAT an internal host to an external IP address you would use the following:

 static (inside,outside) 212.219.63.195 192.168.100.10 netmask 255.255.255.255

This then means......

inside = the source interface for the NAT connection (assuming going from the inside to the outside)
outside = the translated interface for the NAT connection
212.219.63.195 = External IP address to which the internal host is translated to on the outside.
192.168.100.10 = Internal IP address from which is the original source of the traffic.
network mask 255.255.255.255 = Specfies that it is a single host, change this to 255.255.255.0 for a range of addresses for example.

Cisco ASA Dynamic NAT Example (Post version 8.3)

posted 10 Jul 2015, 08:45 by Tristan Self   [ updated 13 Jul 2015, 08:29 ]

Okay a basic example with a Cisco ASA for the NAT rules. Lets say there is PC1 on the inside network that you want to have internet access. The IP address 192.168.1.100 can't just connect to the Internet using that IP address as its not routable, so instead we need to translate it. 

In this example the PC1 is going to be translated from the IP address 192.168.1.100 to 210.123.243.12 (the IP address of the outside interface of the Cisco ASA firewall), to make its outgoing connection. On the way back in the translation is made in reverse, so the traffic coming back from the webhost goes back to 201.123.243.12, a dynamic translation is in place (in memory) on the firewall that translates this back to the original IP address 192.168.1.100.

Now thats fine for one PC but what about a whole network of PCs sharing one IP address, you can't have this for each you'd run out of IP addresses on the outside interface. So instead, you add a port number (hence the dynamic NAT or PAT, port address translation), this ensures that each individual translation can be uniquely idenified by a port number so that it can find its way back. Right lets go through that example above again with PAT.

PC1 is going to be translated from the IP address 192.168.1.100 to 210.123.243.12:1234 (the IP address of the outside interface of the Cisco ASA firewall), to make its outgoing connection. On the way back in the translation is made in reverse, so the traffic coming back from the webhost goes back to 201.123.243.12:1234, a dynamic translation is in place (in memory) on the firewall that translates this back to the original IP address 192.168.1.100.

PC1 (192.168.1.100/24) -----> 192.168.1.1/24 (inside) <-----> ASA Firewall <-----> 201.123.243.12/27 (outside) --------> WebHost 84.12.45.124/28

NAT RULE

Okay so how do you configure this, well you create a network object for the internal range (i.e. the range PC1 is in) so 192.168.1.0/24, then you put the NAT rule within it so something like this:

 object network inside-network
    subnet 192.168.1.0 255.255.255.0
    nat (inside,outside) dynamic interface
!

Okay so what does this mean? Well you create the object network inside-network then say its the subnet 192.168.1.0/24. At this point you then apply the dynamic NAT (PAT) rule to the network object that says any traffic hitting the inside is translated to the outside using dynamic NAT (PAT) on the firewall's outside interface IP address. Of course here if you have more than 1 IP address on the outside firewall port you could use one of these in the rules instead.

inside-network = the internal network object name

subnet 192.168.1.0 255.255.255.0 = the subnet definition

nat (inside,outside) = the source interface (i.e. inside) and the destination interface (i.e. outside) of the translation

dynamic = specifies that this is a dynamic NAT rule as opposed to a static NAT rule.

interface = specifies to use the outside interface's IP address 201.123.243.12 in this example, here you could substitute a specific IP address if you didn't want to use the firewall's interface IP.

ACL RULE

You'll also need an access control list rule to allow this out, so a quick example that would allow port 80 HTTP out to any IPv4 address on the internet would be something like this:

 access-list acl-in extended permit tcp object inside-network any4 eq 80

acl-in =  Means the ACL on the inside firewall (so traffic coming into the inside interface going to the outside or dmz interfaces).

tcp = Means the type of traffic TCP or UDP, in this case TCP for TCP/80 HTTP.

inside-network = Means the source network or host (as an object in this case).

any4 = Means the destination network or host (in our case this is any host on the Internet).

80 = Means the destination port on the remote host (in our case HTTP port TCP/80).


Pre version 8.3

Create the pool for the outside interface:
 global (outside) 1 interface

Now create the dynamic NAT (PAT) rule for all inside traffic to the outside

nat (inside) 1 192.168.1.0 255.255.255.0

This is all you need from a NAT point of view, now you can create an ACL to allow out the traffic.

Enable Wake On Lan Across VLAN on Cisco Network

posted 6 Feb 2015, 04:47 by Tristan Self

Wake on LAN packets do not traverse routers (normally) so you need to configure your network to do this. There are some security implications you need to be aware of first, Cisco's document on it is a good read: http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3750-series-switches/91672-catl3-wol-vlans.html

Also download Solarwinds Free Wake On LAN tool, its a good one to use to test this, it also has a command line to it as well.

So in this example we have the Wake On LAN controller PC on the IP address 172.26.40.22 in VLAN 240. The PC(s) we want to switch on are in the VLAN 244 subnet. We need to configure our core switch/L3 router with the below. This will allow the controller PC to send a request to the subnet to wake up the PC.

access-list 150 permit udp host 172.26.40.22 any eq 7
!
ip forward-protocol udp 7
!
interface vlan 240
 ip helper-address 172.26.47.255
!

interface vlan 244
 ip directed-broadcast 150
!

You need to ensure that the WOL tool you are using will send on udp port 7, if it doesn't the IP directed won't match.

Set the TFTP Source Interface Cisco Switch

posted 18 Jul 2014, 03:21 by Tristan Self

This was an interesting one, I was trying to download a IOS image to a switch but the connection was being denied.

It turned out the connection was coming from a VLAN interface on the switch that did not have access to the TFTP server.

You can force the VLAN interface used with this command:

ip tftp source-interface vlan 5

Where the VLAN 5 is the interface from which the want the traffic to emerge.

Cisco ASA https:// Page Cannot be Displayed

posted 21 May 2014, 02:18 by Tristan Self   [ updated 21 May 2014, 02:20 ]

Okay here's a weird one, we had two identical clusters of firewalls running:
 
ASA IOS: 9.0(2)
ASDM: 7.1(2)
 
One of the firewall clusters you could access https:// to get to the ASDM with no problem, the other one, you access you get page cannot be displayed with some error about SSL not working.
 
Firstly I ran this command:
 
# show run all ssl
ssl server-version any
ssl client-version any
ssl encryption des-sha1
Note the bottom line in green, this only has one type of encryption strangely you'll find this works on Windows XP and not Windows 7. You need to run this command from the command line to enable it:
 
# ssl encryption rc4-sha1 aes128-sha1 aes256-sha1 3des-sha1
 
Now if you try to access https://<ip of firewall> you should find it works. Note you also need to be sure you are running the correct version of Java too.

Cisco 3750 and 3750e in a Stack

posted 30 Jan 2014, 04:18 by Tristan Self

Here's something that came up, we are replacing a few switches and want to replace it will a couple of 3750s and a 3750e in a stack.

The issue was that the 3750e has a different IOS to the 3750s but still must be the same version. After some playing around I found these versions work together.

c3750e-ipbasek9-mz.122-55.SE8.bin
c3750-ipbasek9-mz.122-55.SE8.bin

This is the latest version that the 3750 (v1) supports, it won't support the 15.0 IOS because the flash isn't big enough. The 3750e can't run anything lower than this because its firmware is not compatible.

But this has allowed me to get two 3750s and a 3750e working in the same stack with no problems.

1-10 of 22