On Location: MedicAlert

As you stand at the front of MedicAlert headquarters in Turlock, Calif., it seems an unlikely place to find bleeding-edge technology. The lobby of the nonprofit organization features photos of the company's founders from 1953 and medical ID bracelets that have been in circulation for decades. Its "wall of stars" includes such celebrity endorsers as the two protagonists of the 1970s TV hit CHiPs and a marginal NFL quarterback who last saw action in 1976. Top executives still come to work in jacket and tie.

In the back of the building, however, a revolution is quietly taking place. In their 1980s-vintage cubicles, a group of developers is creating Web services applications that would make many Fortune 500 software groups jealous. IT engineers have constructed an SOA (service-oriented architecture) environment capable of supporting multiorganizational interfaces that most industries are still only dreaming of. We've covered Web services pioneers before, but few companies have attempted an SOA architecture that could interface with multiple partner applications.

Doing Web services for specific apps is different from building an SOA architecture that can support multiple Web services. These SOA services will completely alter the company's 53-year-old business model--and maybe even change the way the medical industry handles patient records.

In a nutshell, MedicAlert is transforming itself from an organization that has specialized in medical ID bracelets for five decades into an organization that could lead the charge in storing and maintaining personal medical information. Using its huge repository of customer-managed data on health and treatment history, MedicAlert is developing services that will let consumers collect and maintain their own patient profiles, rather than trying to navigate the maze of records systems maintained by doctors, hospitals and insurance companies."Where we used to primarily store information that would be valuable in time of emergency, we've expanded that in such a way that individuals can now store [information on] their medications, their doctor visits, their hospital visits, some digital images--anything related to their personal health," says Paul Kortschak, president and CEO of MedicAlert. "So we're no longer just for people who have a medical condition--we're for healthy people who'd like a safe and secure location to store their medical information."

MedicAlert's newest product, E-HealthKey, illustrates the organization's new direction. Essentially a keyfob-sized memory stick, E-HealthKey stores critical medical information, such as health conditions, medications and treatment history, on a device that can be accessed by any emergency response technician or doctor with a USB port--no special software required. Today, MedicAlert is using Web services to synchronize the data between its repository and E-HealthKeys in the field, which customers can update anytime using their home PCs. To date, the MedicAlert database is accessible only to customers, though part of the Web services project will be to make that data uploadable to doctors' offices and by EMTs. For now, the E-HealthKey works better as a standalone device because it's not a given that an EMT would have a Web connection at any location, but one could read the data on any laptop with a USB port.

MedicAlert executives and software developers envision a day when personal medical information may be transferred directly from the repository to applications at any health-service provider or insurance company, at the patient's request. And eventually, they hope, caregivers and insurance companies might upload data directly to a patient's MedicAlert file over Web services, adding more accurate and detailed records to the customer-maintained data in the repository. MedicAlert executives were understandably reticent about giving details on these future plans, not wanting to give too much information away to competitors.

"But if you know anything about Web services, you can see where we're going," says David Harrington, vice president and CTO at MedicAlert. "The potential for linking data and applications is huge."

From an industry perspective, MedicAlert's timing couldn't be better. HIPAA (Health Information Portability and Accountability Act) has made consumers more aware than ever of their right to control their medical information and the difficulties associated with collecting and securing records from doctors, hospitals, labs and insurance companies. From both a regulatory perspective and the consumer-market perspective, then, the medical industry seems ready for MedicAlert's proposition.Unfortunately, the same cannot be said for Web services and SOA, which have been plagued by a lack of products, standards and implementation guidelines, which we discuss in more detail in "On the Road to Discovery". The gist of the problem is that, though much attention has been paid to SOA, and there are some SOA products, there's no commonly accepted template for building an SOA architecture. MedicAlert had to create one from scratch, which wasn't easy because the medical industry hasn't created any standards for Web services calls to medical records. Although MedicAlert developers read every reference book and specification they could locate, they often found themselves on their own. Even now, with the essential elements of the architecture in place, MedicAlert can't be sure whether its plan to link with other organizations' applications will work, especially since the medical industry is notorious for its slow adoption of emerging technology.

"We think what we've done is flexible enough to adapt to whatever develops in the industry," says Jorge Mercado, SOA architect at MedicAlert. "But we know there are a lot of 'ifs' still out there."

How It All Began

MedicAlert's path to Web services has been long and not always straight. It started in 1999, when Harrington, then an independent IT consultant working for MedicAlert, began thinking about the potential for the newly emerging XML standard, which promised to enable transfer of data between heterogeneous applications over IP. MedicAlert had a database of personal medical information from more than 1 million customers. Why not enable exchange of that data using XML?

In January 2000, Harrington pitched his idea to top executives at MedicAlert. The original proposal, which Harrington shared with Network Computing, is full of the dotcom terminology so popular in 2000, but it isn't so far from the business model that the company is adopting now: A common, trusted third-party repository accessible to patients, providers and insurance companies via the Web. So why did it take so long?"When I made this proposal in 2000, they all yawned," Harrington recalls. "They weren't the most forward-thinking group, let's just put it that way, and they're all gone now. I think they hired me to come in and tell them how to fire the entire IT department and outsource everything to IBM or somebody, so that they wouldn't have to worry about it anymore. But I didn't; I didn't feel the same way. I said, 'No, actually, I think you've got something here.' And I held on until we finally got a good management team."

It was a long wait. After making some recommendations on how to reorganize the IT organization, Harrington was hired as full-time CTO in October 2001. MedicAlert went through some tough times--several top officers were convicted of embezzling from the company in 2004--and nobody outside IT made the time to listen to Harrington's ideas. Then, he got permission from an interim CEO to start a preliminary Web services project. And when new president and CEO Paul Kortschak arrived, it was a whole new ballgame.

"Paul's great, he's very tech-savvy," Harrington says. "When Paul arrived in April 2004, I got the management team to take what I was proposing seriously. And they said, 'Yeah, that's the direction we want to move in.'" Since then, Harrington and Kortschak have worked together on the technology vision for the company; Kortschak sometimes takes Harrington to tech shows and demonstrations that might offer new opportunities for MedicAlert.

It's a Go. Now What?

Once he got the go-ahead to get started, Harrington looked for help. MedicAlert is primarily a Windows shop, so he knew that .Net would be the order of the day. He hired Jorge Mercado, a Microsoft consultant with a background in medical-compliance projects, who hired Tim Freeman, a software developer with the Health Plan of San Joaquin. The team was taking shape, but its challenges had just begun."One of the things with technology, especially Microsoft technology, is that it's easy and affordable to get something up and running really fast--and oftentimes, if you do that, you'll do it wrong," Mercado says. "We wanted to sort of tie our hands behind our backs on purpose and say, 'We're not doing anything for 12 months, until we really understand what it is that we have to do.'" So the group spent many of its early days reading, building prototypes and experimenting before it began production work.

One of those prototypes was a application demoed on Capitol Hill in conjunction with several other companies; the context was a presentation to some government agencies and Congressional staff to prove that medical data could be stored virtually and then called up over the Internet. "There are other ways to do that, but Web services was really the best," Mercado says. The team put together a demo application in about two weeks, with little regard for SOA, standards or security.

"After we did that demo, we realized, 'Wow, there's a lot of potential here.' And we also realized, 'Wow, we really screwed this up.' I wanted to make those mistakes, because I wanted to see what things we were doing wrong so that we could do them better. Then we saw that there was this whole SOA movement going on, and we realized that that was the way we wanted to go."

Making SOA a Reality

Around that time, MedicAlert's top executives began talking about a portable data device for its customers to carry, the device now known as E-HealthKey. Once again, the team had found an application for Web services--synchronization of data between the MedicAlert repository and the USB device. "We presented a plan in January, and we released the product in August," Harrington recalls. "We didn't even start writing code until March. It was really a remarkable effort, considering we hadn't done any SOA before. But who had?"The E-HealthKey project served as a starting point for MedicAlert's full-scale SOA architecture, which now provides the framework for several services, both internal and external. Today, the company has Web services for order processing, financials and customer demographics, and it is working on a number of new services that will be implemented over time. All these services were constructed using a common architecture, and the team hopes that at least some services will be reusable for multiple processes or applications.

"We have a Web service that manages demographic data, so if the customer changes a name or address, that service is going to be called to manage that process," Mercado says. "In and of itself, that's not very useful--there are other ways to make that change. But it becomes useful and powerful if you can reuse that service as part of abstracted business processes or business control processes. What we want to do is create a whole series of services that, together, operate an entire business process. That's why SOA is important--you're not just writing individual apps, you're enabling the whole business. We're not there yet, but that's where we want to go."

Mistakes Along the Way

MedicAlert's odyssey toward SOA has featured many misadventures and dead ends, as its development team will be the first to tell you. We asked them to share, in hopes of helping readers who find themselves doing SOA projects of their own.

One of the team's most frequently mentioned missteps was its early focus on Microsoft's Enterprise Development Reference Architecture (EDRA), an SOA development package that includes Microsoft's earlier Application Blocks as well as a service-oriented reference architecture. EDRA is supposed to make it easier for organizations to develop an SOA environment quickly, and MedicAlert's IT staff did its best not only to implement EDRA but to make all its Web services compliant with the architecture defined there."We became EDRA experts, because we really were convinced that that was the way to go," says Tim Freeman, a senior developer on the MedicAlert SOA project. "But what we know now is that EDRA may not be the best choice, and it wasn't necessary for us to be so rigid with it."

EDRA was overly complex, and yet it didn't work for the services they were trying to build. The team has since dropped EDRA ("Everybody was very happy that day," Freeman says) and uses a variety of development strategies, and its own service-oriented architecture.

One thing that the team considers a success is its small-group approach to development, both for the SOA environment and for the applications and services that run on top of the architecture. "You hear about these big companies doing SOA projects with dozens of developers; I just don't know how they do it," Mercado says. "You're much better off starting with a small team, with a service or application that isn't business-critical, so that you can make your mistakes and not cause problems for anyone."

The MedicAlert software development team also keeps re-learning the lesson that off-the-shelf is better than do-it-yourself. "In terms of our applications, we are about 60-40--60 percent is what we build ourselves and 40 percent is what we buy," Mercado says. Essentially, the group wants to develop custom apps and services only for its custom Web processes--they don't write their own financial or office apps, for example.

"We want to focus on writing apps to support our repository, and that's it," Mercado says. "We don't want to be in the business of writing manufacturing and fulfillment code, which we've done in the past. We're putting in more off-the-shelf stuff that will take us closer to 50-50. We don't have to reinvent the wheel. If there's a particular business process that we're doing a certain way, and only custom code will satisfy those requirements, why not change the business process so that we can buy something off the shelf? A lot of the developers hate me because I always tell them, 'Figure out a way to do this without writing any code.'"Another lesson that the team has learned is that while development of the SOA architecture might be in the hands of just a few people, its impact on the rest of the IT environment--indeed, the entire business environment--can be broad.

"There are only three of us working on the architecture, but everyone is involved," Mercado says. "We have developers who are creating Web services--they may not be concerned with how that service works in the greater scheme of things, but they crank out code for a Web service, and they make sure that it takes a request in and generates a response. There are also guys on the systems side who, because now we're dealing with Web services, need to retrofit their boxes differently, secure them differently, and maintain and manage them differently. But they're not looking at it from an SOA perspective. It's just another new way of doing things."

The SOA team provides guidelines on how hardware should operate and what performance and capacity requirements are necessary to support Web services.

SOA Project Report

MedicAlert has finished the "crawling" phase of SOA and is now "walking," Mercado says. "There are four things to achieve within service orientation: identity access management, Web services management, business process integration and entity aggregation. The hard part is figuring out which of those four you can do first. We went after Web services management and BPI, because those were the two areas where we had some real problems in our legacy systems."What BPI means to us is taking all of your autonomous services and making them work together without knowing about each other. It's a set of process-control models," Mercado continues. "If you have a process model that describes a medical information repository update, it outlines the process that the customer will follow to update their information, and the rules that dictate what services get called under what conditions. And with SOA, we hope to leverage that same process model when other applications come on board in the future, instead of having to code it more than once."

After creating some of its first Web services, MedicAlert quickly learned that it needed another key component: management.

"We're all used to things like [Hewlett-Packard] OpenView, where if a machine dies, it turns red, or if there's a problem, it turns yellow," Mercado observes. "But there isn't anything like that built into Web services, so that's why we went out and got management tools from AmberPoint. We absolutely have to know what's going on with the Web service. If we don't know, then it might as well not even be there." MedicAlert is using several Web services management tools from AmberPoint that help the company monitor the status of each service and pinpoint potential trouble spots.

MedicAlert is deploying identity access-management functionality, but Mercado declined to give details, citing security concerns. Entity aggregation is last on the company's priority list: "It's something we will need to deal with once we have a more mature SOA," he says.

Payback?How much has the Web services-SOA project cost MedicAlert? Looking at what the company deployed specifically for the rollout of E-HealthKey, Harrington estimates the total cost was just more than $1 million: $150,000 in software, $400,000 in hardware and about $500,000 in labor. The hardware costs were primarily the infrastructure upgrade the company undertook along with the Web services project.

It replaced aging hubs and routers, for example, with a modern, load-balanced environment that is more capable of handling traffic coming from the Web. Much of the labor cost came from having to build an SOA architecture from scratch, in an industry where there were no simple templates or guidelines for doing it. But MedicAlert officials feel that the time and money were well spent, because the company is developing new services that will follow the same SOA architecture.

Sudden Impact

MedicAlert's decisions, on both a business and a technical level, have had a profound effect on its future IT infrastructure plans. On the business side, the company is migrating its marketing efforts from a focus on those with a critical medical condition--who need the MedicAlert bracelet--to a broader, healthier audience that is looking for a safe, convenient place to store medical records.

"We're already talking with some corporations that may want to provide the MedicAlert service as a health benefit for their employees," Ramesh says. "That could improve their retention of employees, and it could increase the emergency treatment efficiency for those employees. And that might lower companies' insurance rates, just as a good driver discount might lower you car insurance rates."If the strategy is successful, however, MedicAlert may no longer be dealing with just 4 million bracelet owners, but a customer base that is larger by an order of magnitude. With all of those customers doing updates via Web services--and eventually, with caregivers and insurance companies interfacing with the repository--an infrastructure buildout has become an essential part of the company's Web services plan. The buildout is important on the technical side, too, because the organization plans to eventually support application-to-application traffic as well as conventional customer-to-repository queries (for more on the infrastructure upgrade, see "On the Road to Discovery").

The new business model will also change some of the business processes that IT is attempting to map in its Web services. For example, marketing and sales are moving from a strategy that focuses primarily on those with critical medical conditions--a relatively small audience--to a strategy that could potentially attract any consumer of health services. The manufacturing department is spending less time engraving bracelets and more time handling E-HealthKeys or RFID cards. MedicAlert's large call center is answering more computer-related questions as more users update their information via the Web or via E-HealthKey.

"We had done a lot of training with our call-center people on how to navigate through the E-HealthKey screens, but the questions we were getting initially were a lot simpler than that," says Brian Sato, vice president of operations at MedicAlert. "People who didn't understand that they were supposed to take it out of the box. People who plugged it in but never turned their PC on. People who thought all their medical information would already be loaded on the E-HealthKey when they received it. We had to go through a period of Computer 101 training with our customers, and that was new to us."

More Initiatives

E-HealthKey is not MedicAlert's only new service. Over the last year or so, the company has introduced a number of new offerings, including Signature Genetics, a service that tests an individual's DNA to help locate potential problems with metabolizing certain drugs. The data enables doctors to customize prescriptions according to the patient's genetic makeup, and it can be helpful in identifying drug combinations that might cause potentially harmful interactions.The company is also offering the Health Enhancement System, a remote monitoring tool that lets MedicAlert check an individual's key vital signs from a distance at regular intervals. If the individual fails to respond, or if any of the vital signs fall outside the prescribed norm, a MedicAlert clinician is notified so that the appropriate steps can be taken.

And, MedicAlert is doing a pilot test on RFID technology that would let customers put much of the same information they would put into E-HealthKey into an RFID chip on a credit card that they could carry with them at all times. In case of an emergency, the data on the card would be available to emergency health technicians through an RFID card reader.

So far, the only one of these services to employ Web services is E-HealthKey, but it is possible that the RFID technology would also use Web services processes to aid with information updates in the future.

"We have a lot more SOA-based services ideas in the pipeline, and I'd love to tell you about them, but I don't want to get killed," Mercado says. "Come back in six to nine months, and we'll hopefully be able to tell you more."

David HarringtonVice President and CTO
At Work: Responsible for all of MedicAlert's software development, network infrastructure and technical operations

At Home: 59 years old. Married, one son

Hobbies: Travel to Ireland

Alma Mater: University of Kansas; B.S. in Mathematics

HOW HE GOT HERE: 2001 to present: VP and CTO, MedicAlert2000-2001: Independent consultant

MOUTHING OFF: Best part of implementing Web services and SOA: "It gives us the connectivity we need to reach our business goals."

Worst part: "It's an evolving environment, and there are no experts. You're flying blind most of the time."

Greatest business challenge: "The previous management here had an almost willful ignorance of IT, yet they blamed IT for almost any problem. My biggest challenge is to demystify IT so that it's seen as just another business tool."

Most misunderstood aspect of my job: "That technology can't do everything. People have seen so much development in technology in recent years that they now think anything is possible in zero time."If I had a bigger IT budget, I would: "Hire more people."

My dream job: "Airline pilot."

When I retire, I will: "Move to Ireland."

What most people don't know about their personal medical information: "Most people don't keep a record of what conditions they've had, and medical records are not made friendly to the consumer. That's what we're trying to change."

What most colleagues don't know about me: "I was almost a journalist. I finished a master's program in journalism when I was in college."In my car stereo right now: "Bruce Springsteen. Only it's not a CD--it's on my iPod."

Jason Krohn
Manager of Software Engineering

At Work: Responsible for managing and organizing MedicAlert's software development projects

At Home: 30 years old. Married, no children. Hobbies: Gaming, book collecting

HOW HE GOT HERE: 2005 to present: Manager of software engineering, MedicAlert
2000-2005: Programmer, National Health PlansMOUTHING OFF: Best part of implementing Web services and SOA: "It makes us more responsive to the business."

Worst part: "Learning SOA. We had to make a lot of mistakes to get to where we were going."

Greatest business challenge: "Migrating to new technology environments while everything is still in flight."

Most misunderstood aspect of my job: "There's a lot of operational work that needs to be done that no one ever sees."

If I had a bigger IT budget, I would: "Hire more people."If I had the Web services/SOA project to do over again, I would: "Focus on the smaller services first."

My dream job: "Book editor."

When I retire, I will: "Move to the beach and read."

The best part about working at MedicAlert: "It's a true team environment."

What most people don't know about their personal medical information: " Many customers have found they have fewer doctor visits and are healthier because they have their medical information [available] when they need it."What most colleagues don't know about me: "I raise dogs in my free time."

Worst moment of downtime: "In another job I had a technician who pointed a Unix drive to the wrong partition. We were down about a week."

Favorite gadget: "My Nikon D-70 camera."

Tim Wilson is Network Computing's editor, business technology. His background includes four years as an IT industry analyst and more than 14 years as a journalist specializing in networking technology. Write to him at [email protected].

Our ninth "On Location" documentary-style case study takes us to Turlock, Calif., which is nestled midway between Sacramento and Fresno in the San Joaquin Valley. Turlock is home to MedicAlert, a company best known for its metal bracelets and other jewelry, worn by 4 million people to alert medical personnel to allergies and other serious health conditions. Until about two years ago, the 50-year-old company was content to quietly serve that population, but everything changed when CEO Paul Kortschak came on board.Kortschak liked CTO David Harrington's ambitious plan to use SOA (service-oriented architecture) and Web services to build on MedicAlert's trusted reputation and huge repository of health data. The twist: Most companies successfully deploying SOA (and there still aren't many) are using it internally. In contrast, MedicAlert is aiming for an outward-facing architecture that can link to systems in a variety of health-care settings.

We can't help admiring how MedicAlert's IT team jumped headfirst into a full-blown SOA project at a time when few resources existed and, after much trial and error, came through relatively unscathed and with a workable product--the E-HealthKey, a USB key-fob device on which people can store their medical histories in a form easily readable by anyone with a PC and USB port. The premise: Anything that helps eliminate the torturous task of retrieving records stored in doctors' offices, hospitals and HMOs can't help being a hit. Sounds like good medicine to us.

Visit our On Location home page for more on this and previous stories.