Logs and Analysis

To develop a web site effectively, you will need to regularly analyze the web site's log files, which contain data on everyone who accesses the site. From it you can determine, the number of requests made, the identities (IP addresses) of the clients and the pattern of hyperlinks that are followed across the web site. While small scale information can be gained by manually viewing the log files, this technique is not appropriate for finding large-scale trends. Each request for a page creates 60 bytes or so of data that is added to the log file ó more if images are requested along with the pages, which is usually the case. Multiplying this number by, say, 200 daily page requests means that roughly 50-60 kilobytes of data added to the log each day. Therefore, manual viewing is in reality restricted to small samples of the logs.

 

To automatically analyze the complete logs, we will be using Analog, a small yet powerful program which is configurable, scalable and free. It is currently the most popular log file analysis program on the web (a 25% market share according to a GVU report at http://www.gvu.gatech.edu). It will be configured to produce separate reports for each virtual host, and update them each morning, and the reports will only be read by authorized people.

Manual Logfile Analysis

While manual analysis will not be suitable for viewing overall trends, it allows you to interpret the logs with human intelligence. For example, if you notice lots of visitors are requesting one page then leaving, you may want to investigate ways of encouraging them to stay on your site. Do you provide links to other relevant pages? Are they arriving directly into a frame and being trapped with no links out? Are your pages so large, or your connection so slow, they are giving up waiting and leaving the site?

 

You will have chosen where to place your logs when editing httpd.conf. Simply open one in an editor and concentrate on a small section. Below is an extract from access_log on my machine (with the IP addresses replaced by dummy ones):

 

 

The first number in each line is the IP address of the client. By following an IP address through the log, you can find the path an individual visitor took through your site. (Office networks and ISPs such as AOL employing proxies represent around 25% of web traffic, and can cause a single user to appear to come from multiple IP addresses, or allow users to receive some pages without them appearing in your logs. This technique remains accurate the remainder of the time, and is normally accurate even during access via a proxy server, assuming there are not multiple caches. However there is as yet no way round this growing problem).

 

There follows the date and time, followed by the requested filename and the version of HTTP in double quotes. A single slash (/) here represents a directory request, which usually returns index.html. The number immediately following the request is the HTTP success code which is either 200 or 304 as shown above. Any unsuccessful requests, i.e. producing) 403 (Access forbidden) or 404 (File not found) codes go into the error_log file.

 

The next field is the referrer, which in all of the above log entries is http://www.trampolining.net/. The identity of the referrer depends on what file is being logged at the time. In the case of images, the referrer is simply the page that contains the image, but in the case of pages, it is the page the browser was previously viewing ó this gives a good idea where your visitors are coming from. The final pieces of information are the browser and the version of operating system.

 

As you can see, each page can generate many lines of log so to make this kind of following easier, we can cut out some of the unwanted information. To follow the path of just one client, type:

 

 

This will display only log entries created by the client with IP address 231.231.231.231.

 

There are many log file entries corresponding to images, which are often of little interest. To view only page entries, type:

 

 

You can even view page requests from a single client:

 

 

A final technique allows you to watch current requests in real time. This command is:

 

 

You can make this easier to read by removing the image requests and displaying only page requests:

 

Automatic Analysis

This is the vehicle by which we will obtain an overview of our system's usage. Installation of analog is quite simple:

 

q     Download Analog from http://www.statslab.cam.ac.uk/~sret1/analog/ to /usr/local/analog/.

q     Change to the /usr/local/analog directory.

q     Open the analhead.h file for editing and change ANALOGDIR to /usr/local/analog/.

q     Type make.

 

We also need to prepare a directory for the reports and populate it with the necessary images:

 

q     Type mkdir /home/www/trampolining.net/analog.

q     Copy /usr/local/analog/images/* to /home/www/trampolining.net/analog.

 

That's it ‑ Analog is ready for use.

 

Analog is set up using configuration files; the default is analog.cfg which we will edit now, and later on we will create an additional configuration file for each virtual host.

 

LOGFORMAT specifies the format of log used. Analog natively supports the Apache formats COMBINED and COMMON. LOGFILE tells Analog where to look for the access log.

 

 

HOSTNAME specifies the name to put at the top of the report.

 

 

Remember we told Apache not to resolve IP addresses? This little section tells Analog to resolve them, but is much more efficient because addresses are only resolved once, and then written to the cache file specified in DNSFILE. DNSGOODHOURS is the number of hours to trust an entry in the cache file, DNSBADHOURS is the number of hours to wait before attempting to resolve a bad IP address again. DNS WRITE tells Analog to try to resolve unknown IP addresses, then write them to the dnsfile.txt file. The alternative command DNS READ would tell Analog to skip IP addresses which didn't exist in the dnsfile.txt file, thus saving time. On the first run, Analog will complain about dnsfile.txt not existing ó ignore it, Analog will create it.

 

 

This directive tells Analog where to create the report.

 

 

HOSTEXCLUDE directives tell Analog to ignore accesses from a certain IP address or hostname. This allows you to report what your visitors do, without being influenced by your own visits! In this example I exclude all page accesses from Cambridge University using Cambridge's IP allocation, and exclude all accesses from York University using the resolved hostnames.

 

 

If your web site contains pages with other extensions than .htm or .html, for example JSPs or .shtml, you will need to add them here to include them in the page counts, otherwise Analog will assume them to be images.

 

 

Save your completed file as analog.cfg then type ./analog (or ./analog +g/other-config-file.cfg if you have an additional config file). If all goes well, you should get a report like this:

 

 

You will need to create a configuration file for each virtual host, and save it with a different filename, e.g. trampolining_net.cfg. Finally we are ready to schedule Analog to run each morning. We do this using a cronjob, a Linux feature that allows tasks to be run at regular times. We will need to create a separate task for each report, and run them at different times to prevent multiple simultaneous Analog processes clashing.

 

The set up of cronjobs requires you to use the vi editor, which is explained in Appendix B. This is what you type at the vi command prompt;

 

 

The cronjob is now set up to run Analog at 0:55 a.m. each morning, which will write to the default configuration file, analog.cfg), and again at 1.55 a.m. to run Analog with the configuration file trampolining_net.cfg.

 

Our final task is to protect the reports from unwelcome visitors. To do this, we will create a directory container for the /home/www/trampolining.net/analog/ directory in the primary server section of httpd.conf:

 

 

This will deny the contents of www.trampolining.net/analog to anyone except the owner of IP address 123.123.123.123.

 

 

Deploying an FTP Server

While FTP servers are less prevalent in the current web browser driven Internet, they are still the primary method of distributing very large files and maintaining large stores of files. This section will demonstrate how to deploy an effective anonymous FTP server which modern web browsers will be able to access directly. As we said right at the beginning of the chapter, we will be installing the server developed by Washington University, WU-FTP, which can be downloaded from http://www.wu-ftpd.org. For more information on WU-FTP, look up http://www.landfield.com/wu-ftpd/.

Installing WU-FTP

To install WU_FTP, you will need to carry out the following procedure:

 

1.    Download WU-FTP and extract to /usr/local

2.    Type ./build CC=gcc lnx ó Note that to build the ftpd daemon, you might have to install the byacc utility first, which contains the yacc parser.

3.    Type ./build install

4.       We need tell Linux to use WU-FTP for FTP requests by editing /etc/inetd.conf. Look for a line beginning ftp, and make sure it is uncommented. Then edit it to look like this:

ftp stream tcp nowait root /usr/local/wu-ftpd ftpd -laio

5.    Type ps -uax | grep inetd, which will produce a listing of system processes with the word inetd in the title. You should get output like this:

root       354  0.0  0.4  1252  528 ?        S    Oct21   0:00 inetd
root     19048  0.0  0.3  1152  440 pts/2    S    19:31   0:00 grep inetd

The first of the two is of importance to us (the second merely being the search just carried out). What the listing does is provide us with the process ID (PID) which is 528 in the above case.

6.    Restart inetd by typing kill -HUP PID, where PID is the process ID listed from step 5.

 

The latest download of WU-FTP comes with a configure script. It can be installed, from the wu-ftp-version directory, using the ./configure, make, make install sequence of commands as in the other installations in this chapter.

 

There we have it! The Washington University ó File Transfer Protocol daemon is installed and ready for action! We can check the installation by typing ftp www.trampolining.net, or whatever your hostname/IP address is. You should be presented with a login screen, and you will be able to log in using a standard Linux user account and password set up on your system.

 

Configuring WU-FTP

To provide access to the general public we need to allow anonymous access. Before doing this, we need to create a safe directory for anonymous users, which will appear to them as the root of the FTP server. This prevents anonymous users browsing around your machine to obtain private information! We also need to create an user account for anonymous FTP users to use.

Creating an FTP directory

We will create our FTP directory in /home/ and adopt a traditional directory structure:

 

 

The first, /home/ftp, will be the root directory of our anonymous FTP server. /home/ftp/bin will contain links to commands we want to allow FTP users to use, in particular ls (to list the contents of a directory) and cd (to change directory). /home/ftp/etc is present to hold a password file if necessary and /home/ftp/pub/ is the public directory which contains the files we are making available.

 

All directories and files within this structure should be owned by root, and none of them should have Group or All write permissions. This will prevent the user editing any of the files ó by editing the contents of /home/ftp/bin/, a user could execute any code on your machine. All the directories should have All read and execute permissions, to allow users to enter the directory (execute permission) and read the contents (read permission). Finally, all the files contained should have All and Group read permissions only ó this will allow users to download files, but not change or execute them on your server.

 

You may require the creating of yet another directory, as follows:

 

 

This directory is special in that it is available for users to upload files to. For this reason, it must have Group and All write permissions and but not Group and All read permissions which will prevent users viewing the contents of this directory. While this is the standard way to implement two-way FTP access, it does pose a security risk ó users could potentially upload illegal files and use your server to store them. It is a serious policy decision whether or not to provide this service ó if you do, be sure to set a umask to prevent uploaded scripts being executed. A slightly more secure system involves removing All write permissions from this directory too, then creating subdirectories with full read, write and execute permissions ó these can then be accessed by 'trusted users'. Anyone you have not told the location of these folders to should be unable to find them, since /home/ftp/incoming cannot be listed ó there are no read permissions for All.

 

To summarize, this is how I suggest that you set the access permissions for your FTP site:

 

Configuring Linux for WU-FTP

The most important change is to modify the main Linux /etc/passwd file to ensure the anonymous FTP user is limited to /home/ftp/pub. Open the file for editing, you should see a listing like this:

 

 

If no FTP user exists, use the root command adduser to add ftp. The important line begins with ftp, which contains the user settings for FTP User. Note there is no entry after the final colon. This ensures no command shell is made available to the FTP User. To force /home/ftp/ to be treated as root directory, we edit this line slightly, adding a decimal point where we want the user to be rooted. The final /pub ensures they are initially placed in that directory:

 

 

Finally, we need to create a set of configuration files for WU-FTP in /etc. Luckily there is no need to create them by hand, as WU-FTP distribute a default set with the program, which will prove fine for our anonymous server. We will copy these default files to /etc:

 

 

We can implement an extra security touch. In /home/ftp/ type:

 

 

There are some final modifications which are not strictly necessary but make anonymous access that little bit easier. Hard linking /home/ftp/bin/ls to point to /bin/ls will allow clients to list the directory through FTP. Make sure that the owner is root and it has group, owner and all execute permissions only. Copying /etc/passwd and /etc/netconfig into /home/ftp/etc/ will provide the replace the user and group IDs for each file and folder with their corresponding names. However these files contain far too much sensitive information and need editing. Only groups and users owning files within the FTP directory should be left in, and password information should be left out ó there should just be an x after the user name, not a random string of characters. Anonymous access should now be available.

Making your Servers Persistent

In the event that your Linux machine crashes or loses power, the priority is to get the machine serving requests as quickly as possible. This can be eased greatly if the system has been designed to recover from a crash ‑ if the services start themselves on boot up, it can save a great deal of time trying to remember what needs to be started!

 

There are two main services that need special attention in order to enable autostart. First, we need to make sure the network is ready for requests. The ifconfig utility must be configured for each virtual host. We can make this automatic by editing /etc/rc.d/rc.local, or the equivalent file for your Linux distribution. At the end of this file we append all the commands we used originally when we set up the virtual hosts:

 

 

The other service we need to start is the Apache web server. Again we will start this by appending the setup command to a boot file. Editing the boot script of the machine is a simple way to do this. You could create a startup script in init.d (called apache) and link to it from S20apache in rc2.d. A sample file follows:

 

 

You can create the symbolic link by changing directory to rc<n>.d (where <n> is your runlevel - usually 3, but you might also want to create one in rc5.d if you use a graphical login.) Create the link by entering ln -s /etc/rc.d/init.d/apache /etc/rc.d/rc3.d/S20apache.

Summary

In this chapter you have learned how to install the highly popular Apache web server and configure it to meet your requirements and set up virtual hosts. You were shown how to install and configure the ApacheJServ servlet as well as how to modify your Apache configuration to make use of SSI and CGI. Other newer powerful technologies such as mod_perl and JSP were also briefly discussed. You were also instructed in the setting up and configuration of one of the main open source FTP applications WU-FTP.

 

In addition to setting up the servers, this chapter also covered an important administrative task, namely the analysis of the server logs files, with some discussion on manual analysis using command line tools and automatic analysis using the free Analog tool. Finally you learnt some tips on server persistence ó by making minor alterations to system files you can restart Apache on reboot and have it ready to receive requests.

 

For a discussion on the advanced configuration of Apache, and for other information on Apache itself, ApacheJServ and JSP, see Professional Apache.

References

Web

The Apache home page:

http://www.apache.org/

 

Security bulletins for Internet services:

http://www.cert.org

 

Java Servlets Page:

http://java.apache.org

 

WU-FTP's web site:

http://www.wu-ftpd.org

 

More information on WU-FTP:

http://www.landfield.com/wu-ftpd/

 

Analog logfile analyzer site:

http://www.statslab.cam.ac.uk/~sret1/analog/

 

Jakarta Development site:

http://jakarta.apache.org

HOWTOs

Details on how to set up web servers and clients:

WWW-HOWTO

 

How to set up a multi-purpose web server:

Apache SSL PHP/FI frontpage mini-HOWTO

Books

Peter Wainwright, Professional Apache, Wrox Press, ISBN 1861003021

Danny Ayers et al, Professional Java Server Programming, ISBN 1861002777

©1998 Wrox Press Limited, US and UK..