When PathPing is run, the first results you see list the route as it is tested for problems. This is the same path that is shown via Tracert. PathPing then displays a busy message for the next 125 seconds (this time varies by the hop count, requiring 25 seconds per hop). During this time PathPing gathers information from all the routers previously listed and from the links between them. At the end of this period, it displays the test results.
The two rightmost columns — "This Node/Link Lost/Sent=%" and "Address" — contain the most useful information. The link between 172.16.87.218 (hop 1), and 192.168.52.1 (hop 2) is dropping 13 percent of the packets. All other links are working normally. The routers at hops 2 and 4 also drop packets addressed to them (as shown in the "This Node/Link" column), but this loss does not affect their forwarding path.
The loss rates displayed for the links (marked as a "|" in the rightmost column) indicate losses of packets being forwarded along the path. This loss indicates link congestion. The loss rates displayed for routers (indicated by their IP addresses in the rightmost column) indicate that those routers' CPUs or packet buffers might be overloaded. These congested routers might also be a factor in end-to-end problems, especially if packets are forwarded by software routers.
Find more information about PathPing, including switches (command line options) for the utility.
3.6.6 Nslookup
The Nslookup tool is used to query a Domain Name Service for IP address and host names. This is useful as client-side DNS failures give a false positive for a connectivity problem while server-side failures can cause sluggish service connection times.
When you start Nslookup, it shows the host name and IP address of the DNS server that is configured for the local system, and then display a command prompt for further queries. If you type a question mark, Nslookup shows all available commands. You can exit the program by typing exit.
To look up a host's IP address using DNS, type the host name and press Enter. Nslookup defaults to using the DNS server configured for the computer on which it is running, but you can focus it on a different DNS server by typing server < name> (where < name> is the host name of the server you want to use for future lookups). Once another server is specified, anything entered after that point is interpreted as a host name.
3.6.7 Netstat
Netstat shows the state of the active network connections on a host. This is very important information to find for a variety of reasons. For example, when verifying the status of a listening port on a host or to check and see what remote hosts are connected to a local host on a specific port. It is also possible to use the netstat utility to determine which services on a host that is associated with specific active ports.
View more information on using Netstat and its options/switches, or type "netstat /?" at the command prompt.
3.6.8 Arp
The Address Resolution Protocol (ARP) is a communications protocol used for resolution of Internet layer addresses into link layer addresses, a critical function in the Internet protocol suite. ARP was defined by RFC 826 in 1982 and is Internet Standard STD 37. ARP is also the name of the program for manipulating these addresses in most operating systems.
In Microsoft Windows Arp allows you to view and modify the ARP cache. If two hosts on the same subnet cannot ping each other successfully, try running the arp -a command on each computer to see whether the computers have the correct media access control (MAC) addresses listed for each other. You can use Ipconfig to determine a host's correct MAC address.
You can also use Arp to view the contents of the ARP cache by typing arp -a at a command prompt. This displays a list of the ARP cache entries, including their MAC addresses.
Find more information about the Arp command.
3.6.9 Route
In MS Windows, the 'Route' command is used to view and modify the IP routing table. Route Print displays a list of current routes that the host knows. Route Add adds routes to the table. Route Delete removes routes from the host's routing table.
Find more information about the Route command.
3.6.10 Telnet
Telnet (short for TELetype NETwork) is a network protocol used to provide a command line interface for communicating with a device. Telnet is used most often for remote management but also sometimes for the initial setup for some devices, especially network hardware like switches or access points. As a troubleshooting tool, Telnet can be useful to test connectivity to a remote host. Telnet provides a bidirectional interactive text-oriented communication facility using a virtual terminal. In MS Windows, you will need to enable the Telnet Client through the Control Panel. To do this, go to the Programs and Features section of Control Panel and click on 'Turn Windows features on or off'. From the Windows Feature window, selecting Telnet Client and then clicking OK will enable Telnet.
Find more information about the Telnet command.
3.6.11 Other Utilities
In addition to those described above, there are a wide range of utilities available for multiple operating systems. Some are available for either Windows or Unix/Linux family and some are available for both. The list below includes some that may be useful.
Pchar
- Based on PathChar
- Measures Network performance on a per-hop and total path basis
- IPv4 and IPv6
- Useful in isolating network problems
Netcat (nc)
- Similar in operation to telnet
- Tests application connectivity
- Can test TCP and UDP services
- Note - There are many other applications that provide this type of functionality. Examples include puTTY and Tera Term. The selection of one over the other is strictly a personal preference.
Host - Unix tool similar in functionality to the NSLookup Windows command
Dig - Linux tool similar in functionality to the NSLookup Windows command
3.7 Network troubleshooting application suites
This section briefly describes a number of application suites that extend the network trouble-shooters toolkit and can be essential for the efficient and successful operation of the corporate network.
3.7.1 Network monitoring and discovery tools
Network monitoring tools can either be a component of your NMS or a separate utility. In either case, a network monitoring tool is used to record and analyse the characteristics within its configured network. Network monitoring tools can monitor for network performance as well as network outage and device resource use. They typically aggregate multiple network devices into a single user interface for cross-device analysis. Some features in a network monitoring tool that are critical for the troubleshooting process are:
- multiple device capability
- traffic graphing support
- device resource use monitoring
- alerting and notification via multiple mediums
- SMS/text messaging support
- SNMP management
- traffic analysis
- built-in traffic filters and aggregators
Sometimes part of the network monitoring system and sometimes a separate application, Network Discovery tools can scan the network for known device heuristics. When a device heuristic is found at a particular address, the network discovery tool logs the location and device type.
Numerous Network Discovery software solutions exist and each has a specific mechanism for seeking out devices: by IP address, MAC address, SNMP response, DNS entry, or even individual switch port on switching devices. Some features useful in a network discovery tool are:
- NMS integration
- multiple IP range entry
- fast scanning
- device heuristic databases with SNMP
- switch port mapping
- data export to common file formats
3.7.2 SNMP trapping
SNMP trap receiving tools are out-of-band tools that can receive, analyse, and display low-level trap information from an SNMP-enabled device for purposes of troubleshooting and SNMP analysis outside the NMS. SNMP trap editing tools allow for the editing of trap templates to customize NMS response when traps occur. These tools incorporate some needed features for advanced SNMP manipulation, such as:
- data export to common file formats
- trap manipulation
- tree view
- trap mimicking and simulation
SNMP trapping is commonly an integral part of the Network Monitoring system or tool.
3.7.3 MIB browsers
Management Information Bases (MIBs) are databases of characteristics about network devices. Those databases are released by the manufacturer and house readable and writeable information about the configuration and status of the network device. A MIB Browser is a specialized tool that can peer into the data inside a MIB and pull out relevant Object ID (OID) information. Remember that OIDs are little more than strings of numbers used as unique addresses for device data. A good MIB Browser will include a pre-populated database of known OIDs and their related data. It will also enable the ability to "walk the MIB tree," gathering all known data from that MIB and presenting it to the administrator.
The real power of an effective MIB Browser is in its ability to view and search the MIB for relevant information and allow the administrator to modify and customize that information as necessary. A good MIB Browser will typically include:
- remote device support
- large database of known OIDs
- view/search/walk via tree-view
- editing functions
- reading/writing support
- multiple-device support
MIB Browsers are primarily used as customization tools for the SNMP-enabled devices plugged into your NMS.
3.7.4 Attack identification and simulation
Administrators unfamiliar with the changes in a network's functionality during an external attack situation will be unprepared for fending off that attack once it occurs. Attack identification and simulation tools enable the administrator to identify when common network attacks occur such as broadcast storms, cache poisoning, replay attacks, and so on. They also allow for the simulation of such attacks upon a network to monitor and analyse the behaviour of that network as well as to assist in preparing the network against a real attack by an outside attacker.
Attack identification tools such as network intrusion detection systems and network intrusion protection systems can be complicated to install and manage due to the prevalence of false positives and false negatives such systems can generate. Features of interest in either type of tool include:
- performance monitoring elements
- identification databases with real-time update
- multiple attack profiles
- dictionary and brute force capabilities
- network device security checks
- port scanning
- network jamming
- remote TCP resetting
It is important to note that such tools have the capability of inhibiting the successful operation of the network so must be used with great care and by technicians who are fully skilled in their operation.
3.7.5 Network Packet Sniffers
Network Packet Sniffers are application programs that can analyse what is actually happening on the wire by intercepting and logging traffic on a network. A Packet Sniffer (also known as a Network Analyser or Protocol Analyser) is useful for measuring performance and connectivity, and can help to establish a network performance baseline.
Packet sniffers work by intercepting and logging network traffic that they can 'see' via the wired or wireless network interface that the packet sniffing software has access to on its host computer.
On a wired network, what can be captured depends on the structure of the network. A packet sniffer might be able to see traffic on an entire network or only a certain segment of it, depending on how the network switches are configured, placed, etc. On wireless networks, packet sniffers can usually only capture one channel at a time unless the host computer has multiple wireless interfaces that allow for multichannel capture.
Once the raw packet data is captured, the packet sniffing software must analyse it and present it in human-readable form so that the person using the packet sniffing software can make sense of it. The person analysing the data can view details of the 'conversation' happening between two or more nodes on the network.
Network technicians can use this information to determine where a fault lies, such as determining which device failed to respond to a network request.
Packet sniffers can be very useful when analysing network problems. There are numerous hardware solutions available on the market, as well as many software applications including freeware. This document does not describe, promote, or endorse any particular solution.
General information on packet analysers, as well as specific information about particular brands, can be found easily through an internet search engine.
3.8 Speed test tools
A very easy test that can be used to both determine the internet bandwidth available to a specific host and to determine the quality of an internet connection is the use of one of the many online speed tests tools. Whilst there are other, more detailed, methods that can be used to determine the speed or capacity of the network connection between your network and a given destination, online speed check utilities are an easy and quick way to obtain a good idea of how the network is performing, between your end site and that used by the chosen speed check tool. These tools can be useful when measuring how long it is going to take to upload or download information from a local to remote host. The measurement given in the speed test results can also be used to determine whether the connection is offering the amount of bandwidth that was purchased from the internet provider. Some online speed test tools can provide an indication of the quality of the connection by measuring the ping response times and jitter amounts over a short period of time. This information can be used to determine a likelihood of how well the measured connection will deal with certain types of high demand traffic like Voice over IP (VoIP) or gaming. When using any of these tools it is important to consider the following:
- Keep in mind that some amount of bandwidth difference is expected between the quoted bandwidth purchased and the measured bandwidth.
- The results of free online speed test tools may not be used as an alternative to bandwidth measurements (or other contracted service levels) given for your connection to the HSCN.
- Be clear on how the online speed test obtains and presents its results.
- Bear in mind that these are 'snapshot' views and the time they were run may be significant in terms of how busy your network (and internet connection) is.
Online speed test tools are generally an 'indicator' of performance and bandwidth and can be useful as part of the network trouble-shooters toolkit. Popular online tools include:
3.9 IP address management
This section describes a set of tools to specifically deal with the management and maintenance of IP addresses. The tools discussed in the following sections are designed to assist with that process of managing the scope of IP addresses on your network.
3.9.1 Subnet and IP calculator
A subnet and IP calculator can be used to ensure a correct IP address selection and with this a correct IP address configuration. The applications/tools available vary in functionality. An IP subnet mask calculator generally enables subnet network calculations using network class, IP address, subnet mask, subnet bits, mask bits, maximum required IP subnets and maximum required hosts per subnet. Results of the subnet calculation usually provide the hexadecimal IP address, the wildcard mask, for use with ACL (Access Control Lists), subnet ID, broadcast address, the subnet address range for the resulting subnet network and a subnet bitmap. Variants (or other functionality) may include CIDR network calculations using IP address, subnet mask, mask bits, maximum required IP addresses and maximum required subnets and give the wildcard mask, CIDR network address (CIDR route), network address in CIDR notation and the CIDR address range for the resulting CIDR network etc.
As with other applications in this space, they are many and varied with free and paid applications available for download as well as online tools that are generally free to use.
3.9.2 DHCP
Interestingly, although the automatic assignment of addresses through the Dynamic Host Configuration Protocol (DHCP) is considered a network function, its administration is usually done by systems administrators. This is usually the case with small and medium-sized networks because the server that handles the DHCP service resides not on a network device but instead on a server.
However, the management of DHCP scopes can leak into the role of the network administrator in situations in which DHCP scopes fill up. In networks with many DHCP scopes at high utilization, when the scope fills to 100%, users interpret the resulting lack of network connectivity as a network problem. In those situations, the network administrator is often the first to be called in to troubleshoot the problem.
Including DHCP scope monitoring tools in your network administrators' toolset can help in these situations as full scope problems are difficult to track down using other tools. When considering a DHCP scope monitoring tool, look for one with capabilities that include:
- tabular user interface
- support for BIND and Windows-based DHCP
- alerting and notification
- visual identification of full and near-full scopes
Problems associated with full and nearly-full DHCP scopes can be a troubleshooting nightmare. This is because the client error messages associated with a full DHCP scope in many OSs are unclear. Also, the resolution to the problem is often a re-segmenting of the network to add new subnets. It is for this reason many networks utilize multiple full Class C networks for workstation networks.
If you are having issues with full or nearly-full scopes due to machines that repeatedly come on and off the network, consider reducing the DHCP lease time to a very short amount of time before re-segmenting the network. DHCP renewal traffic is very minimal on today's networks and the added traffic from the increased number of DHCP lease renewals should not significantly impact network performance.
3.9.3 IP address management tools
Where the intersection of the systems and the network administrator can cause difficulty is in the management of available IP addresses for subnets not serviced by DHCP. In typical networks, these subnets often house the network servers and server infrastructure. Because servers are critical components of the network, management of their IP address space is important to ensuring their uptime and availability.
In early networks, systems administrators often use a "ping and pray" approach to finding an available IP address on a server subnet. In this approach, they ping various addresses on the server subnet and look for the first one that does not respond. They then configure the new server with that IP address and "pray" that it wasn't in use by a server experiencing an extended outage. In dynamic situations with servers going up and down for extended periods, this can be especially problematic.
A better approach to using "ping and pray" is to incorporate an IP address management tool that monitors for use of IP addresses in critical subnets. The tool can store the last known-connected device for each IP as well as notify the administrator how long that IP address has either been used or has gone unused. When looking for such a tool, consider features such as:
- forward and reverse DNS lookups
- data export to common file formats
- active monitoring
- database storage
- SNMP support