Network Monitor Finally Comes Out Of Hiding

By watching a number of rings longer term, we noticed that this phenomena occurred every few minutes, but each time the burst was send to a different workstation. The NetBIOS Name Recognized packet contained the name "NTCORP," the main domain name of which most users were members. The 16th character of the name (also called the suffix) was hex 1E, a group name used to elect a master browser. Early in the trace, the workstation broadcast a Find Name of NTCORP with a suffix of hex 1D, which is used to access a master browser. The Primary Domain Controller, which functions as a master browser by default-master election criteria, responded. According to our Sniffer, the workstation then attempted what looked like a \MAILSLOT\BROWSE SMB conn ection to the master browser, which never responded. This apparently caused the workstation to send out a Find Name of NTCORP with the suffix 1E. The Find Name was broadcast as an IPX Type 20 (also known as an IPX WAN broadcast). Luckily, there were no routers running in parallel, so there was never a problem of more than one Type 20 packet appearing on each remote ring. Unfortunately, every NT workstation is a potential browser by default, and responds to the hex 1E elect master browser request, generating more than 6,000 return packets! Why 6,000? Call it classic NetBIOS broadcasting; in the Microsoft implementation, the workstation sends out three Find Name requests, even if replies were initially received. We were hoping that Network Monitor might provide us with more clues to this problem, especially with the \MAILSLOT\BROWSE packet, something we've always seen a lot of in Microsoft networking. Enter Network Monitor Viewing the same packet traces on NetMon gave us insight into the problem. Using the detail display let us drill down on any given layer of protocol by expanding or collapsing details as needed. We like this methodology for displaying packet detail, which was actually pioneered in NetMon and later adopted by Sniffer Basic (formerly NetXRay). Although we found the NetBIOS Find Name packets decoded similarly by Sniffer and NetMon, the mysterious \MAILSLOT\BROWSE packets in the Sniffer were decoded as Get Backup List Request browse packets in NetMon (see screenshot, below). We now knew that this was part of the Microsoft Browse protocol at work, as described by Microsoft. When using IPX, a workstation first locates the master browser using the NetBIOS Find Name NTCORP hex 1D IPX broadcast, then asks for a list of backup browsers (usually requesting up to four because the master browser maintains a list of one backup browser for every 32 workstations in a domain). The workstation then establishes a session with a backup browser s erver to get the actual resource information. The problem in our case was the master browser never responded to the Get Backup List Request packets, prompting the workstation to broadcast the Elect Master Browser packet, which in turn, generated the 6,000-plus packet response. These response storms overwhelmed the switches, causing them to drop packets. With enough successive response storms, other packets were caught in the storm and dropped, with some unlucky users losing their print connections. Having identified the root of the problem, we now had to complete the analysis and come up with potential solutions. At the end of the response-packet storm, the workstation sent out another\MAILSLOT\BROWSE as decoded by the Sniffer. NetMon decoded this packet as a Microsoft Browse Force Election packet. Immediately following this packet was yet another burst of hundreds of NT workstations (potential browsers) participating in the election process. Eventually, the PDC (Primary Domain Controller) won the elect ion and the network settled down for awhile. There were a number of possible solutions, such as removing the NWLINK driver from the NT workstations and relying instead on Novell's Client 32 for IPX (because there were still NetWare services the NT workstations had to access). Another solution was to prevent NT workstation from being a potential browser (via a registry setting), and so not participate in the browser elections. Our solution depended on another important piece of information about the Microsoft Browse protocol. Microsoft documents the fact that every IP subnet has its own master browser. What it doesn't say is that for IPX, there is one master browser for all of IPX. For both protocols, there is a backup browser for every 32 workstations. But bear in mind that a workstation will always first request a list of backup browsers from the master browser. Bottom line: The PDC was overwhelmed as a master browser. Fortunately, no IPX resources needed to be accessed over the backbone. Because th e PDC was located on the far side of the backbone, IPX Type 20 WAN broadcasts were simply filtered out at the router. This forced a BDC (Backup Domain Controller) that's located on the near side of the router in a workgroup to function as a master browser. Subsequent packet traces showed that the BDC responded just fine to IPX browse requests, and we no longer had the brief, but intense browse response packet storms. Not only that, but the users were no longer dropping their print connections.