Branch Office Management

Money: How much can we spend?

Resources: What's available in terms of equipment and people?

Time: How much time do we have to deliver?

In our experience, this threesome must always be in balance. How does this relate to managing remote sites? Here's an example: In calculating remote-branch support costs, you consider the obvious extended infrastructure expenses, but also conduct a risk-assessment analysis, including a review of failure points and how downtime will impact the business. You may find that after reaching a service-level agreement, driven by the business costs of downtime, the budget is inadequate to support the project. Time to send up a flare.

[Lesson learned] Things aren't always as cheap as they appear. A company decided to implement VPNs across the Internet for all its remote locations. The driver was WAN cost savings. Sounded great on paper, until the first ISP network outage. With no backup network, the company was dead in the water. Management fumed that IT had failed to assess the probability and cost of downtime, or to propose ways of mitigating the risk. Providing such information up front would have spared everyone a lot of trouble.

People are always the most expensive component of any IT endeavor. The decision of who to use and where is a difficult one--76 percent of our poll respondents said availability of local technical support is their biggest challenge. When determining personnel needs for remote sites, consider:

Geographic location and time zones: Can a centralized support group cover the distance and time-of-day variations?

Language differences: Even if you can find people to cover 24 hours at a central location, are there language barriers?

Business requirements: How can you best meet special business-unit needs? For example, if a remote warehouse is printing pick slips for shipping, it will need reliable on-site printer support.

Standardization: Are corporate and industry standards defined and adhered to? Streamlining support means minimizing variables. Are multiprotocol environments truly necessary?

Architecture: The expertise required to support complex architectures may dictate additional support, so simplify whenever possible. Are you planning for redundancy in critical components of your architecture? Will you define the applications and support them with the necessary architecture, or is it better to select applications that integrate nicely with your existing environment?

Convergence: Is your data group separate from your voice group? Are you using separate facilities for each? Are you well-positioned for VoIP (voice over IP), potentially a huge cost saver? Corporate culture and political gamesmanship can throw up roadblocks to effective remote-site support.

[Lesson learned] Share facilities to minimize costs. We worked with a company in which data was handled by the IT group and phones by the facilities group. Separate lines were provisioned and managed by each. Some remote sites actually procured their own lines locally. By combining the lines and renegotiating a single large contract, the company saved more than $1.5 million the first year in communications costs alone--all without changing the technologies. Simply sharing the channels more efficiently on the T1 lines and managing them from a single point provided huge cost savings.

[Lesson learned] Creative solutions can span technology hurdles and political camps. A company needed X.25 communications to deal with remote sites in technology-challenged areas. The network group had two factions: One favored more progressive IP-based systems, while the other insisted on tried-and-true X.25 systems. Arguments broke out. The creative and simple solution was to tunnel IP through the X.25 network where necessary to provide reliable transport and simplify transition to IP-based systems.Long-haul communications services are one of the major components--and costs--of tying in remote sites. Without these services, there would be no communications, system access or real-time operations, so contingency planning in this area is crucial.

We've heard many newcomers to the networking profession claim that a VPN is all you need to connect remote sites. Wrong. Although a VPN may seem cost-efficient, without the proper business analysis and service-level agreement, you're operating on the lunatic fringe. Responsibility and accountability go hand in hand--your company's business (and your job) should not depend on a network for which no one is directly responsible. What levels of service can you commit to for Internet-based systems?

There's no way around it: IT managers supporting remote sites must do their homework on WAN connectivity. Address the potential impacts of design selections. Analyze traffic to identify current network-traffic flows and potential bottlenecks. Examine migration paths to newer technologies and VoIP convergence and factored them into the equation. Communications pipe sizing could lead you to a different technology. Mesh, hub-spoke and hub-hub-spoke models are all valid and driven by business requirements and the traffic flows they generate. Traffic analysis will also provide an opportunity for you to tighten the belt and ensure there's no unnecessary traffic traversing the WAN.A hub-hub-spoke model is ideal for companies with overseas and/or regional traffic flows. Say manufacturer based in the United States has a multilingual, multinational network based in England into which all European Union countries are tied. Orders and consolidation information are relayed to U.S. systems for stock replenishment in the British central warehouse. A much smaller long-haul communications pipe is necessary between the British and U.S. sites--hence, the hub-hub-spoke model. Because the England-to-United States link is the most expensive, long-haul communications are minimized and special requirements (multilingual) are handled at the second hub.

[Lesson learned] Know before you go; look before you leap. We worked with a company that had outgrown its single building and decided to move the engineering/development group to a new site a couple of miles away. T1 lines were ordered, with bandwidth allocated based on an IT manager's gut feeling rather than on usage statistics. Day 1 was a nightmare: The bandwidth allocated was inadequate, and the engineering teams were rightly upset. Business came to a halt. The network team should have been allowed to monitor LAN traffic from the groups involved to determine the bandwidth necessary to maintain service levels across the WAN. Amazingly, the manager chalked it up to "working out the bugs" and wasn't disciplined. We would have considered that a career-altering decision.

[Lesson learned] Rush, and you'll go nowhere fast. One company was in such a hurry to complete its systems conversion, it chose dial-on-demand ISDN lines rather than wait for leased lines. Routers were set to establish connections on an as-needed basis. After implementation, the central network staff failed to monitor outgoing connections, and some keep-alive traffic caused unnecessary connections to be established every five minutes. When the bill arrived, it exceeded all estimates. In the rush to "get things done," costly router-programming oversights were made. Even worse, they weren't caught until flags went up in accounting.

Don't forget backup facilities when you're performing capacity planning for new lines or adding capacity to existing lines. And don't assume anything: If you're using a single carrier to provide both your primary and backup communications, get full disclosure on the circuits behind the scenes.[Lesson learned] Redundant to what point? A nationwide company with centralized operations at corporate had multiple T1 lines for voice. One line was designated as backup for voice and data communications. Premises equipment was programmed to route traffic to the backup automatically when there were faults on the main circuit. Then, one day, it happened: A major T3 line went down. The premises equipment promptly changed over to the backup circuit ... which was also dead. The circuits terminated on the same T3, so the only real redundancy was in the local loop.

When managing remote sites, the less planning and streamlining done up front, the greater the need for IT staff intervention later. Some companies see remote-control software as the silver bullet for managing remote branches with limited manpower. But it's no replacement for proper planning.

Are end users well-trained? Are effective patch-management systems in place? Are users allowed to alter their own PC configurations? All these factors can affect how much those users will rely on your support staff.

So what's the best way to physically support remote sites? It boils down to time, money and resources again. Most organizations put their emphasis on a central IT staff, which costs less and uses fewer resources, but this group can become bogged down in unnecessary--and time-eating--firefights that could have been avoided with better training, standardized systems and controls. Some alternatives to a centralized support staff model:

Outsourcing: Although usually the most costly option, outsourcing IT support may be a good fit if some of your sites are in hard-to reach locations or require 24/7 support, or if you have a small IT staff. In fact, more than 31 percent of respondents to our survey said they use outside technology support. Ask vendors about the scope of their support activities, their geographic reach and coverage, the quality and consistency of service between vendor offices, response time, and blanket contracts for all sites. Note that some vendors may have multisite presence, but their offices are independently run with wide variances in scope, service quality and response time.

Business-unit liaisons: If business-unit managers are amenable to sharing personnel in exchange for speedy response, this might be a viable option. Technically savvy end users--whom we call IT champions--are given in-depth training by the IT group. They're then designated as central contact points and the first line of basic troubleshooting support for their areas.

On-site IT: Depending on the architecture and size of the remote office, this might be the most viable solution. Although pricey, it's still more cost-effective than outsourcing, in most instances. The challenges are recruiting and managing long distance. Candidates must be well-disciplined, well-rounded and highly motivated. More than 43 percent of our poll respondents said they base IT support staff in branches.

One of the most surprising statistics from our survey is that more than 34 percent of respondents have no support, or only ad hoc support furnished by non-IT staff members, at remote branches. This neglect can lead to security problems and ineffective system use because non-IT staff will spend inordinate amounts of time on problems unrelated to their jobs, hurting productivity.

Linchpin of Support: Standards

Although not a direct cost consideration, the efficiencies you'll achieve--or not--in remote deployment and support hinge on company standardization. Nearly 45 percent of our poll respondents said standards compliance is the biggest challenge for remote branches. With standards in place, you can change to another vendor for any reason without worrying about knowledge-transfer, and support staff will be interchangeable.And if PC hardware is standardized, hot spares can be maintained and swapped out by support staff to minimize turnaround times. Stock can be maintained at each remote site or centrally. Standardizing on a limited number of software configurations means disk-imaging software can be used to revert back to a working configuration quickly. A rollback can even be performed remotely overnight to avoid shipping costs and potential equipment damage.

In the same vein, if you have components that take a long time to order or install, an on-site spare should be kept ready. Also consider maintaining an "IT First Aid Kit" that contains common repair stock, such as power supplies.

[Lesson learned] Sometimes, the pieces absolutely, positively won't be there overnight. A nationwide company subscribed to the "ship it overnight" philosophy to route hot-spare PCs to replace troublesome remote machines. In one instance, three different shipments were made before the internal customer received a working replacement. Despite careful and appropriate packing by the IT team, PCs arrived damaged--in one case, manhandled to the point that most of the internal cards had worked loose from the motherboard! It took four working days before the user was back online. Consider keeping spares at each remote location, physical security and budget permitting.

In our survey, when we asked readers to rank remote-branch management tools in order of importance, documentation came in dead last out of 12 areas. That's unfortunate, because it's so easy--and inexpensive--to produce documentation, especially compared with the difficulty and cost of learning about a remote site. For example, central-office personnel may have never set foot in the branch they're supporting. Detailed information about the site, including reference materials, photos, diagrams, floor plans, communications outlets, wiring details and equipment lists, is invaluable. Products like Netviz can help organize and assemble this information, but it requires some effort.

Next to creating policy and procedure documentation, the most important component is information dissemination and company buy-in. In terms of remote branches, you must have ardent support, or it's a losing battle. To win this support, ensure that branches are given the same attention as local corporate departments.Data Storage and Backup

Regardless of your applications and technology platforms, you must protect company knowledge and personnel investments with solid backup operations. You can accomplish this mission in several ways--for example, you can use centralized or decentralized servers. To determine what will work best for you company, answer the where, when and how questions:

Where: Are you pulling or pushing data across communication lines? Are you using automated libraries or does the staff perform backups manually?

When: Do you have a downtime window big enough to support your approach? Time may dictate technology requirements and backup strategies.

How: Can you do real-time backups? Do you need to stage the data offline for backup? Data sizes and staffing hours also will influence how you approach data-replication strategies and time frames.Again, the only costs greater than the communications component are human resources. Staffing levels and support for remote data stores and servers need to be balanced with application and infrastructure costs. In some instances, archaic LAN applications will drive the need for more remote servers to keep communications-line costs down and response times healthy.

Stay Safe

Last but certainly not least is security. Your remote branches are extensions of your corporate infrastructure, and you must watch them just as closely. For example, how will you handle communications from the remote site to the outside world for voice and/ or data? One option is to route them through the central office and use a single access point/firewall. But this puts further strain on WAN bandwidth. Another choice is to place additional access points/firewalls at the remote branches. Carefully analyze traffic and weigh remote-support, security and cost considerations to arrive at the right decision for your company.

Of special note are home users. Often connected via VPNs, their remote PCs become an integral part of your infrastructure. Base security guidelines must be established before you accommodate this type of connection.

Your security defense must be nimble and quick. In addition to security-patch tools, you need configuration-management tools to close potential holes, as well as intrusion-detection tools to tell you of abnormal activity or device presence in the remote branch.The scope of managing remote sites is all-encompassing and must systematically address almost all the functional IT areas. Here are the Top 11 steps to remote-support success:

1. Formulate--and follow--a road map detailing corporate system migrations, obsolescence and enhancements.

2. Implement corporate standards.

3. Remember that hidden costs will be directly inverse to the level of efficiency and standardization built into systems and infrastructures.

4. Document, distribute, communicate and adhere to policies and procedures.5. Include remote site-support reviews in every IT decision.

6. Communicate. The decisions made in many areas determine how smoothly remote branches can be managed, so work together. Reach out early and often to business units to determine their requirements, future enhancements and loads.

7. Weave security and data protection into all aspects of planning.

8. Plan for the high-profile costs of communications and staff. These must be planned up front, with detailed analysis and design considerations.

9. Review geographic locations with vendors and suppliers to determine available technologies and resources.10. Take your time. Remote-site implementations must be well-thought-out. Take time to gather requirements and usage statistics from current systems and infrastructure. Factor in technology enhancements and loads--VoIP, for example.

11. Plan, review, plan, monitor, plan, project, plan, execute and plan some more. The IT landscape is constantly changing with shifting business and technology decisions. Keep abreast of them.

Richard J. Brown is an independent systems consultant and owner of Business Applications Ltd., a technology consultancy and provider of Web hosting and e-mail services. He has worked with and managed corporate systems groups for more than 25 years in the high-tech, finance and manufacturing fields. Write to him at [email protected].

Because remote branches are an extension of your organization's infrastructure, you must have solid IT practices in place to effectively manage and support them. Systemic problems only get worse when extrapolated to remote sites. On the upside, economies of scale and benefits of standardization apply when corporate systems expand to include branch offices.

Before making implementation plans, conduct a careful analysis and needs assessment with your business units. In some instances, you'll need to augment or alter your corporate systems approach. Because communications facilities and human resources are major cost factors, you'll achieve the greatest efficiencies by facilitating the work of support staff, whether central, on-site or contracted. Standardize hardware, software platforms and tools; adopt solid corporate policies and standard operating procedures; and give your IT staff and internal customers a clear understanding of their roles and functions.In "Technologies Bridging the Gap", we explore at two technologies that are crucial to keeping your remote house in order--network-protocol analyzers and remote-access and control software--and compare commercial offerings with their open-source counterparts. For both product groups, we were pleasantly surprised at how well the commercial and open-source feature lists compared. Far from being underpowered and limited to just a few options, the open-source products held their own, especially for shops with strong networking know-how. Large sites with extensive feature requirements may need to cough up the cash for commercial offerings.

Those with remote locations spread across the nation or the world face additional challenges. In our reader survey, 60 percent of the respondents said they have national branch locations, while 10 percent indicated they have international offices and 29 percent reported having a mix of both. Given the prevalence of remote offices, we've pulled together some best practices for managing far-flung locations.

National Challenges

Different service levels: Be very careful when looking to contract outside vendors for national networking and break/fix support. Some companies are merely franchise operators, and there's no consistency of service from one location to another. Of course, consistency and quality of service can also vary widely for company-owned service providers, depending on the pool of resources they have to draw on. We've seen technicians with widely differing knowledge bases billed at the same rate. If you can spare the time to locate them, best-of-breed service companies in each area might make more sense.

Regional coverage: Don't assume anything. Technology readily available at your corporate headquarters may not be available in your remote branches, and certain providers may not have a presence close enough to realistically support them. Deliver a list of all your locations to the vendors and ask for their nearest facilities and capabilities, including guaranteed response times. In some instances, you may be forced to implement less-than-ideal infrastructure to tie in some locations.

Time zones: If you opt for central helpdesk coverage, this may entail staggering the workday for critical staff. If remote systems are to be backed up, or if you have batch processes running, the operating window will be further constrained.

International Challenges

Even more time zones for central support: International time variances can be severe. If you choose centrally located support, be prepared for unconventional staffing hours based on remote-site locations. It may be difficult to recruit strong talent for odd-hour coverage.

Language barriers: Native-language communications are always the most effective. Standardize on a single language or hire multilingual administration and support personnel.

Cultural barriers: Internationally, there are huge differences between business approaches, work ethics and attitudes. Some are overt and easy to identify, like monthlong summer vacations, while others are covert and work against you behind the scenes. Try to understand the cultural differences and plan for their impact.

Standardization: Standards for telephone equipment, electricity and other factors must be considered. Make sure your equipment vendors support a wide range of country standards.

Local business practices and regulations: Some countries have nationalized services, and procurement could take significantly longer than what you're used to. Also, the unsavory practice of bribing a local supplier for equipment or services may be considered the norm in some locales.

Now that everything's in place, you can sit back and have a latte, right? Wrong. You must keep tabs on branch office compliance, specifically:

Change management: Do you have a methodology for pushing out changes to base configurations, new applications and security patches? Do you know when system revision levels aren't where they need to be?

Systems monitoring: Do you have a warning system to inform you when services that affect remotely deployed systems, like servers, communication lines and remote firewalls, go down?

Device management: Do you have methods for controlling critical remote devices, including power management?

Performance: Do you have tools in place to gauge the performance of your communication lines and remote LANs?

Metrics: Do you have a reporting system to determine your capacity status, as well as to measure growth and project how long it will take to reach capacity?

Lesson learned: Computers sometimes speak in foreign tongues.A large international corporation was building a hub/spoke system for most of the European Union nations. The hub was located in England, and the central support personnel were primarily English-speaking. The company hired contractors for each country, instructing them to install English on each server. When the German contractor got on-site, the local personnel convinced him he must be mistaken: Why wouldn't he install German on the German server? You can imagine the English team's surprise when it first attempted to remotely administer the server.

Lesson learned: You can't always get what you want, WAN you want.

A multinational corporation was working against the clock to build an international network. Unable to install fixed lines quickly enough, it opted for an ISDN network, which it believed would allow quick implementation (four to five days in most countries) and buy time for a longer-term solution with adequate bandwidth. This worked out fairly well in most countries. But when the company approached Telnor in Spain, the nationalized PTT said the ISDN switch was full and would not be available until someone relinquished service. Fortunately, the company was able to talk local IT folks from a sister unit into letting it share some WAN capacity as a stopgap measure.