ON THE WIRE / BILL ALDERSON AND J. SCOTT HAUGDAHL

File Reading One Byte At A Time

Q: We have recently ported a Unix application to DOS for use on our Ethernet LAN. The application runs great except when loading files. Then, the application appears to grind to a halt. We noticed that we nearly maxed out our file server's CPU utilization when we ran the ported application. Our network file servers are NetWare 3.x, and response time is fine with our standard suite of off-the-shelf DOS and Windows applications. The application runs slow even if the workstations and server are on the same physical Ethernet.

Scott: The raw transmission rate of an 802.3/Ethernet LAN is 10 million bits per second or 1,250,000 bytes per second. Unfortunately, even the fastest Pentium and RISC-based workstations and servers often transfer data at a far lower rate for a zillion different reasons.

Bill: In this case, a protocol analysis study of our client's application in action quickly revealed the problem--the application was sequentially reading a 1-MB file one byte at a time!

Scott: We have seen this problem many times. Once we saw an application sequentially writing to a file four bytes at a time! These applications generate lots of little frames and beat up the network, not to mention the file server.

Bill: Generally speaking, the larger the frame size, the better. To see why this is the case, we need to take a look at the overhead that is present in every Ethernet frame. First, there is the mand atory 9.6 microsecond interframe spacing or wait time, which equates to 96 bits or 12 bytes of overhead. Add another 8 bytes for the preamble and 4 bytes for Cyclic Redundancy Check (CRC), and we have 24 bytes of mandatory overhead in every frame.

Scott: Further, we have protocol overhead inside each frame. Since we are typically only interested in the data that makes it to the application, we often base our benchmarks on throughput at the application layer. Here, frame sizes can have a dramatic impact on throughput, due to protocol header overhead.

Bill: As an example, on a typical NetWare Ethernet frame, a response from a file server has the 24 bytes of frame overhead, plus 52 bytes of protocol headers. If our application were to read only one byte from a file, we would also have to pad the data to meet the minimum 64-byte Ethernet frame requirement, creating another 8 bytes of overhead. Thus our total overhead for transferring 1 byte of application data in a single frame is 24 + 52 + 8, which equals 84 bytes, or 840 percent.

Scott: As we increase the amount of application data inside the frame, we dramatically reduce the overhead, as illustrated in the table on the next page. The table, based on data collected by a protocol analyzer, reflects the percentage of overhead and number of frames generated on an Ethernet to transfer a 1 MB file, and factors in each read request packet. The counts are based on actual NetWare file transfers without using the packet burst protocol. (Packet burst only works for applications requesting large block sizes. This particular application was written such that unknown to the programmer, data was being returned from the server only one byte at a time.)

Bill: So why was this particular application behaving this way?

Scott: Believe it or not, because it asked for it that way. We went back to the Unix programmers, and they thought they could simply port their C source code to DOS and recompile. The newly compiled application worked pretty well locally, but it was a different story over the network. Further analysis of the application source code revealed a pipe-like function that was reading and scanning characters, one at a time.

Bill: Aha! In the Unix version of the application, a block of data is returned to the application even if it only asks for one byte, whereas DOS implicitly returns a single character! On a local DOS machine, the single character comes from a disk or cache, but on a network, that disk/cache is located on the file server; thus every character generates two frames--a request and response frame on the network.

Scott: From the table we can see that in the end, reading the 1-MB file one byte at a time resulted in 128 million bytes of network bandwidth consumed on the network and generation of two million frames.

Bill: Once we got the programmers to rewrite the code by having the application read the same data 1,024 bytes at a time, the result was 1,119,141 bytes of bandwidth consumed and only 1,953 frames generated--a huge savings in bandwidth, not to mention a much faster response time to the end user!

Scott: Once again, large packets save the day.

Bill and Scott are principals of Pine Mountain Group and spend their time troubleshooting and optimizing large networks, training end users in protocol analysis, and developing tools to allow users to make better use of their protocol analyzers. They can be reached at otw@pmg.com.