Understanding SOA Governance

SOA governance is ultimately about control--laying down the law on process and policy, then enforcing it within your organization to ensure IT delivers quality applications and services that meet business needs. Implemented correctly, SOA governance products can facilitate and automate that enforcement, while ensuring adherence to an architectural standard for your SOA initiative, allowing the reuse of services across projects using metadata management, and decreasing the number of defects in deployed services by enforcing testing and documentation policies.

All worthy goals, but "experts" and vendors are tossing the term governance around like crooked Olympics judges hand out perfect 10s. We don't blame you if you're tired of hearing the word. It's overused, overhyped, and under-explained. In many ways, this is to be expected: Governance is about process and policy, which are specific to each organization. But there are some constants.

First, SOA governance should start early--in the requirements-gathering phase, with documentation stored as artifacts in the repository to verify alignment with business goals. Governance covers the SDLC (software development lifecycle) and run-time policies of everything--including deployment of, and access to, the services that make up your SOA. Let's say you're a developer starting a new SOA project. You probably have a directory on a network share, a spreadsheet, perhaps even a document-management system where you're storing the project charter and initial requirements document. SOA governance starts with these documents, so you'll want to create a project and store them in the SOA repository for future reference.

SOA registry/repository products focus on metadata management--the metadata collected about vital documents lets developers, managers and business analysts search and locate them during the development and deployment phases of the project.Another constant: From your developers' viewpoint, governance is widely seen as an impediment--documentation and metadata collection especially are considered tedious at best, no better than "busy work" in many cases. Easing the process can only help.

Two Timers

There are two parties within SOA governance: design time and run time. Both are necessary, and both are part of a holistic governance strategy intended to aid--not impede--your SOA initiative. Design-time governance focuses on the development phase and is primarily an internal concern, while run-time governance deals with policies that regulate access, security and the performance of services that may be consumed by internal and external clients.

The available SOA registry/repository products deal with design-time governance quite well, but their run-time aspects have a long way to go before we'd consider them enterprise-ready. Interoperability with the rest of your SOA infrastructure--security devices to enforce security policies and access control, management points to enforce SLAs--is not up to par, though partnerships between registry/repository vendors and other SOA infrastructure vendors are popping up like dandelions. These partnerships will eventually result in better interoperability and enable the exchange of policies between product sets for better run-time governance. But that will take time; we don't expect to see many truly interoperable implementations until next year.

Let's focus on what is ready now: Design-time governance focuses on the SDLC, providing a framework in which service developers can discover, deploy, document and reuse services in a collaborative environment. Validation, de-duplication, versioning, and enforcing processes like business and technical reviews are within the realm of design-time governance products.

Note that SOA governance products do not, and should not, take the place of conventional version control. Rather, they should work with version-control systems, which typically have been concerned only with organization and storage of artifacts--they do not deal with metadata or offer search capabilities based on metadata, and on their own they don't provide validation and process-management capabilities. Version control is just storage; the onus of managing what's in the version-control system is still left to a person. In contrast, management of artifacts in the registry/repository can be largely automated and includes a wider array of functionality.

Run-time governance focuses on controlling deployment through approval processes and on applying run-time access-control policies to services. Say a developer has completed development and testing of a service and submitted the service for approval. After the appropriate people have ensured that the service meets organizational policies, the registrar (a common term for the "super admin" of the registry/repository) pushes a button and, voilà, the service is now considered "published."

In a fully implemented governance scenario, the service WSDL and any associated run-time policies--specifying, for example, access control; security options, such as encryption or digital signature requirements; and SLA policies--would also be pushed to the appropriate SOA infrastructure product for run-time enforcement. The alternative to this method, much preferred by all vendors, is that SOA infrastructure products query the registry/repository and retrieve the appropriate run-time policies on their own. It'll be some time before we see that functionality across the market, however.

Get a GrepAt the heart of any governance initiative is the registry/repository, or as pundits like to say, "regrep." What started out as an initiative to provide cataloging of services through implementation of a UDDI (Universal Description, Discovery and Integration) spec has grown beyond the bounds of a simple digital yellow pages. The registry, focused on metadata management, has joined forces with the repository, focused on storage of metadata, to provide a single source of record for the services that comprise a SOA. The market is immature, however. We looked at some early versions while working on this article, before we were rudely interrupted by flooding in our lab.

There's still not universal agreement regarding what the core features of a registry/repository should be, nor on the role UDDI plays in governance. We do see growing consensus in the vendor community that UDDI should be used for integration and interoperability, and little else. The registry provides an accessible interface to search a catalog of services, assets or artifacts, while the repository is responsible for storing, categorizing and validating services and associated artifacts as they are published. Together they provide a powerful framework for enforcing policy and process at both ends of the SDLC.

Further confusing the market is the split between UDDI- and ebXML-based implementations (see "The (Non)Competing Standards". Both are registry standards and provide interfaces for searching and publishing, the primary purposes of a registry. But UDDI, which came first, defines a much narrower underlying storage model in that it's focused on storing SOA-related artifacts. The specification, and API through which developers interact with UDDI-based registries, is less extensible and more rigid than its ebXML counterpart.

The ebXML spec, on the other hand, was designed without concern for what might be managed within a registry based on ebXML, focusing instead on providing implementers with flexibility and extensibility, which can then be passed on to users. Time and innovative efforts have eclipsed most of the differences between implementations of registries/repositories based on these standards as far as the user is concerned, but the standards themselves are not changing drastically through revisions. Although functionality is not hindered by a vendor's decision to use one standard over the other, we find that the two often seem pitted against each other anyway. Infravio and Sun Microsystems use ebXML--in fact Sun's implementation runs in tandem with the open-source ebXML registry, freebXML. Standing on the other side of the fence with UDDI-based offerings are the jointly developed Fujitsu-Software AG CentraSite and Systinet 2 from Systinet (recently acquired by Mercury Interactive). Systinet's extensive partnerships and OEM agreements have made it the darling of the wider ISV SOA community.

Somewhere in the middle sit Flashline and LogicLibrary. While Systinet, Infravio and Sun have built their registries/repositories around the registry, Flashline and LogicLibrary have taken a completely different approach. Both offerings have limited support for UDDI as an interface, but both are design-time focused and developer-oriented. Flashline and LogicLibrary are repositories first, registries second. This is a major departure from the other vendors' offerings, which were registries first and repositories second. Clearly, Flashline and LogicLibrary view repository/governance features as more important than the registry aspects. Although both provide registry-like functionality, they've hamstrung their UDDI implementations, preferring proprietary integration through partnerships and code over standards-based integration--which is where most of the SOA community is headed.

LogicLibrary, for example, integrates only with UDDI-compliant registries for publishing assets; there's no direct UDDI interface into its repository. Flashline does offer a UDDI interface to allow queries into its product and to ease integration with other SOA infrastructure products, but it does not let other products use the publishing interfaces of UDDI to submit new assets. Still, because both are so repository focused, they do a much better job of governing the design-time process in terms of integration with IDEs, approval processes, access control and gathering reuse metrics.

Finally, it's important to remember that pure UDDI registries and those bundled with SOA products, such as Registry Manager from SOA Software, are just that--registries. These products are little more than catalogs, with no governance features or real control over their query, publishing and subscription options. A registry is only one slice of the governance pie, and a small one at that.

Ultimately, the registry/repository is about enabling and encouraging, not imposing, governance within your specific organizational structure. That's why many IT groups find vendors reluctant to dig into details about what their offerings do--each product can, and must, be tailored to a specific organization and its processes. There is no single "out of the box" registry/repository that can solve every facet of your governance needs. Registry/ repository products are about providing interfaces into frameworks that offer a set of services designed to be customized. This will strike some as a cop-out, but look what happened to ERP/CRM vendors: They lost big because they forced process on customers. Business re-engineering was a disaster, and the reg/rep guys know it. And, while the registry/repository certainly provides a framework within which developers, architects, and business-process and service owners can collaborate to ensure compliance with organizational policies, whether those policies are followed and enforced is up to the individuals who interact with the system on a daily basis. You can lead a horse to water, but you can't make it drink.Governance in Action

Four primary functions needed for SOA governance are provided by the registry/repository: validation, cataloging, relationship management and collaborative process/policy enforcement. All play vital roles in enforcing standards, enabling reuse and increasing the quality of services. Although other features, such as versioning and federated query capabilities, are nice to have, without these primary functions you'll find it difficult to implement your governance initiative.

» Validation provides an automated mechanism to ensure that services and their associated artifacts--WSDL, XSD (XML Schema Definition), XSLT (Extensible Stylesheet Language Transformation)--conform to organizational standards. Upon submission of an artifact to the registry/repository, an automatic validation process can be spawned to verify that namespaces are consistent and artifacts are well-formed and adhere to known standards, for example. The process can even be configured to run WS-I Basic Profile checks against WSDL.

Submission of artifacts can trigger a workflow to start an approval process that includes reviews of code, documentation and adherence to corporate standards as well as an evaluation of how well the service fulfills its stated business requirements. Most registries/repositories provide plug-in functionality for validation services that let you extend existing checks to include organization-specific validation items. If you can code the validation service, it can generally be incorporated into the registry/repository. Validation's role within SOA governance is to ensure adherence to corporate standards and increase the overall quality of service by not allowing poorly written artifacts to be submitted for corporate use.

Even if you can't code an organization-specific validation service, the registry/repository decreases the time it takes to manually determine whether reusable services exist that implement given business functions. Architects should be performing reviews of new services to determine whether they duplicate existing functionality (you are, right?) and the registry/repository eases this process by revealing policy data attached to assets or artifacts. With specific policies attached, architects can review new assets with an eye toward not only meeting IT standards, but organizational policies and goals as well.These tasks are time-consuming, but they can be streamlined by using a registry/repository in which the right metadata has been collected. SOA architects can query the registry/repository using business or technical terminology that describes the service in question, and be presented with all services that might match the parameters. In time, developers should learn to routinely perform this task before diving into development, which in turn will shorten the development cycle and result in greater IT, and therefore business, agility.

» Cataloging services provide a mechanism for collecting metadata about a service; this metadata is available for searching by users of the registry/repository as they're starting a project, to determine whether or not services exist that can be reused. Cataloging is the heart of metadata management: Automated cataloging services can pull data from documents like WSDL and XML schema, even code, if you choose to use the repository to store that as well.

Automatically collecting metadata relieves developers of the need to enter this data manually, an onerous task that is often left until the last minute or avoided. By automatically collecting a larger percentage of the metadata, cataloging services encourage adherence to this requirement by making it less tedious. Most registry/repository products do allow configuration of required fields, however, to ensure that a minimum amount of metadata is collected from the submitter to provide the necessary information on which the asset can be categorized and later searched.

So, does this automated metadata gathering rely on developers performing certain tasks while creating artifacts? Maybe. For some types of artifacts, like WSDL and defined XML standards, it's easy enough to automate the process because the format is well-defined and known. For other automated categorization/validation processes, it may be necessary to require that certain data be present in the documents so that cataloging/ validation can be automated. So, cataloging itself isn't a magic bullet, but for well-defined types it does remove the onus from the developer to instrument the document for automatic cataloging.» Relationship management is important because it enables features like impact analysis. An impact analysis is particularly useful in governing the run-time aspects of your SOA because it offers visibility into the effect of a change in one service or artifact across your infrastructure. Relationship management--through manual or automated cataloging processes--determines the inter-relationships among artifacts and services. XSD files are often imported by WSDL documents, which are in turn tied to services. Reuse of XML schema provides consistency across services, and is a best practice. But if the XML schema changes, it can be a nightmare to determine which services are affected. Impact analysis uses the relationships among artifacts to determine which services and artifacts will be affected, which in turn provides project managers with a better understanding of the time and effort necessary to enact such a change.

» Process and policy control is perhaps the most difficult of the four registry/repository functions to implement, primarily because process and policy are both unique to the organization. Even if Company A and Company B both subscribe to agile programming methodologies, the actual processes required by each will likely be substantially different. Who reviews code? When? How often? Are the unit tests run by developers (your developers are doing unit testing, right?) also run by QA engineers before the service is considered complete?

As we mentioned, SOA governance vendors learned from the mistakes made in other technology areas that dealt primarily with process: Early ERP and CRM implementations subscribed heavily to the "business process reengineering" dogma and sought to force organizations to change their processes to match the product, rather than adapting to individual businesses' requirements. Registry/repository products generally support enforcement of some kind of SDLC approval process, with varying degrees of control and customization. The best solutions provide a customizable mechanism for codifying your organization's review process that ensures that artifacts and services published to the registry/repository for use by other developers adhere to corporate standards, whatever they may be. Clearly, you don't want developers publishing services and artifacts willy-nilly into the registry/repository with no control over whether they're suitable for use in the wider corporate environment. The registry/repository is designed to prevent this from occurring by imposing some measure of control over the publication process.

Process control and enforcement must be collaborative and automated, and developers and management alike must know when the status of services and artifacts change so they can act on those changes in a timely manner. A subscription-based model is essential; it can immediately notify subscribers when artifacts are published or modified. Letting business analysts subscribe to services or artifacts relevant to their projects is also important--it empowers them with the information they need to gauge the effectiveness of IT in serving their needs while cutting down on status meetings, where developers are pulled from their keyboards to present progress updates.

Although the registry/repository won't supply the detailed status reports of an IT project- or portfolio-management product, it can provide a view of what progress is being made on a project as artifacts are submitted, reviewed and published. If that knowledge reduces the time developers spend in meetings, they're spending more time developing services.The Run-Time Story

Governance focuses heavily on the design-time aspects of SOA implementations, but there is a run-time side to this story. All the products in this space provide UDDI-compliant interfaces, which let them integrate with the rest of a SOA infrastructure. SOA management suites, SOA security gateways, even enterprise service buses can all integrate via UDDI with the registry/repository to pull the WSDL documents necessary for these products to perform their functions. By using the registry/ repository as the authoritative source of record for these documents, the rest of your SOA infrastructure keeps synchronized and up to date.

In fact, the subscription and notification capabilities of the registry/repository offer a good way to keep administrators of the disparate pieces of your SOA infrastructure informed about changes to existing services or on-boarding of new services. Problems that might occur due to changes in interfaces can easily be identified--if administrators are informed. This saves time tracking down and resolving issues that arise from a potentially volatile environment and thus increases service availability.

Ultimately, the registry/repository should also be responsible for managing the run-time policies used by SOA infrastructure products that describe security and access rights, but that integration is only beginning to appear in products today. WS-Policy and its profiles appear to be the protocol by which these policies will be exchanged in an interoperable way, but WS-Policy has not yet reached standard status. That means that support is not yet a checkbox feature for registry/repositories--or any other SOA infrastructure product--and vendors are left to their own devices to support exchange of policy. That support is limited, but there is some promising activity occurring in the standards bodies and in the vendor community (partnerships and integration); we expect that within two years, the registry/repository will be able to exchange policy with other SOA infrastructure elements and take on its ultimate role: As a centralized policy management repository.

Our advice: Choose a vendor with a good track record of adopting standards. Meantime, look for vendors that maintain good partnerships with other SOA infrastructure vendors to ensure this functionality.

We call this aspect of the registry/repository "run-time" but we don't suggest using the registry/repository as a run-time service discovery engine, especially in a high-volume scenario. Policy and service descriptions, in the form of WSDL documents, should be managed in the registry/repository, but the exchange of both between your SOA infrastructure and the registry/repository should be considered in the same light as deployment of applications to production. The exchange should occur on a regularly scheduled basis, but the registry/repository shouldn't be involved in every client request made to a service. Similarly, the feedback mechanisms that allow SOA infrastructure products to feed the registry/repository with service metrics that aid in calculating the reuse of services should not necessarily be real-time processes.

Lori MacVittie is a Network Computing senior technology editor working in our Green Bay, Wis., labs. She has been a software developer, a network administrator and a member of the technical architecture team for a global transportation and logistics organization. Write to her at [email protected].

The Non Competing StandardsAlthough both the UDDI and ebXML technical committees are sponsored by OASIS, the two standards are different in their approaches to metadata management.

UDDI is increasingly viewed as a mechanism to integrate registries rather than as the principal upon which the registry/ repository is founded. Existing ebXML registries/ repositories support UDDI 3.0 interfaces using SOAP primarily for inquiries, but in many cases for publishing too. UDDI lacks the extensibility of ebXML in terms of its ability to support objects and artifacts outside the SOA. Don't confuse the spec's inflexibility with the capabilities found in UDDI registries/repositories, however--vendors have worked around the spec's inherent limitations through proprietary extensions and their registries/ repositories are no less flexible than their ebXML-based competitors in terms of artifact and metadata management.

A path should be chosen based on both how well the product suits your governance needs and on its ability to provide the level of enforcement required for your organizational processes and policies. Whether the registry/repository is based on UDDI or ebXML is not a valid criterion for choosing a governance solution, as both sets of available products support a UDDI interface, meaning they can interoperate with the rest of your SOA intermediaries, and both specifications are open standards with strong support communities.

» UDDI 3.0

» ebXML 3