To authenticate any service in a Linux system against a Microsoft Windows domain, you need a PAM module that is not available from the Red Hat 7.2 Linux installation CD. Download it from ftp://ftp.samba.org/pub/samba/pam_smb/. As of this writing, version 1.1.6 is the latest. Put it on your Linux system, then run the following commands:
tar zxvf pam_smb-1.1.6.tar.gz
cd pam_smb
./configure
make
cp pam_smb_auth.so /lib/security/
You'll need to edit the file /etc/pam_smb.conf so it looks like this:
DOMAIN
PDC
BDC1
BDC2
DOMAIN is the name of the Windows domain, PDC is the NetBIOS name of the primary domain controller. The BDC lines, the NetBIOS names of backup domain controllers, are optional. In fact, any Windows NT or 2000 server can be used here, as long as it is in the same domain. By using the PDC and BDCs, you can be sure there is a server always up and responding. Finally, edit the file /etc/hosts and add the following lines:
192.168.1.1 pdc.domain.net pdc
192.168.1.2 bdc1.domain.net bdc1
192.168.1.3 bdc2.domain.net bdc2
The IP addresses you use are those of your PDC and BDCs, and the FQDNs (Fully Qualified Domain Names, pdc. domain.net) are the actual DNS names of those computers. The final name for each line is the NetBIOS name.
This last step may not be necessary if the NetBIOS names match the DNS host name for each domain controller and if the Linux server resolves (via DNS) the domain controllers¹ names without using the fully qualified domain name. In other words, if the PDC¹s NetBIOS name is PDC, then on the Linux server you should be able to type ping pdc and get a ping response. If not, you must edit /etc/hosts as described above. Then running ping pdc should work, as should pam_smb.
See us1.samba.org/samba/ftp/docs/textdocs/ENCRYPTION.txt for more information on how the challenge-response authentication in a Windows domain works.