Service & API Catalog

The steady growth of DevOps has leveraged fairly technical concepts into major building blocks of modern IT strategy and even business models. Product teams use Microservices and well-aligned APIs to split work into domains, allowing them both for rapid advancement and low coupling. Business teams seek to unlock new business models & look for API monetization, while Enterprise Architects face the challenge to ensure governance and drive integration between self-developed software, SaaS and no-code/low-code developments.

Development speed and ingenuity can now be best achieved by empowering autonomous teams. But in fast-scaling development organizations and service landscapes, increasing complexity can quickly become a burden. Therefore, managing DevOps complexity is critical to driving development efficiency and software reliability.

Key starting points here are a clear and up to date overview of an organization's Services and the APIs that connect them which can then be expanded to capture further dependencies, both technical (used Libraries, Licenses and more) and business such as Team Ownership. In the following we will break down the LeanIX VSM approach to Service Catalogs and API Catalogs from the basic benefits, what's in it for the different types of benefactors as well as a start to setting things up:

📘

TLDR: Complexity around modern software cannot be handled without automation

Cloud computing, microservices, APIs, small empowered teams & the advance of no-code / low-code introduces a highly dynamic landscape. Instead of wasting tedious time with manual processes or fire fighting, automating the documentation of developed software and dependencies frees time & helps product leaders to increase the maturity of their engineering organization.

👍

TLDR: How VSM can help?

VSM ships with a well-established out-of-the-box Data Model for Product IT, fueled by ready-to-use & extensible Integrations into Business, Development & Operations. Easy to configurable collaboration functionality like the Portals can be used to provide tailor-made catalogues for various stakeholders, while our Reports helps product leaders to get a quick overview. All discovered artifacts can be used in Diagrams as well.

What's in there for me?

As a member of a product team: Save time by automated documentation, get fast access to information & people by a well-maintained knowledge base.

As a product leader: Get a big picture of Teams & Technologies in your organization, help teams to streamline governance & foster knowledge exchange

As an enterprise architect: Automate the intake of self-developed software to boost governance (e.g. am I allowed to use a Cloud Service in this region) & technology standardization use cases.

As an engineering leader: Facilitate new team members onboarding by providing them with a centralized knowledge base while promoting reusability of existing software components.

Tangible Outcomes - Service Catalog:

The Basics:

In a diverse, fast changing landscape of services, be they microfrontends, microservices or different setups clear and easy to access transparency is key to control complex, understand dependencies and take action where needed. With the VSM Service Catalog the necessary level of documentation is automated through integration with tools already widely used in engineering - from code repos over CICD pipelines to various engineering management tools - to minimize documentation overhead and ensure information is reliably up to date.

The Outcomes:

A baseline of services:

LeanIX VSM brings the different services and technologies teams develop into an easy to access catalog - allowing access to an unlimited number of users for convenient knowledge sharing as well as links to jump-off points with further, more technical information (navigating into the relevant repo for example).

Team Responsibilities:

This catalog can be extended with responsibility allocation, clearly defining not only what is available and being worked on but by whom and what their roles are - key in a large, diversified organisation to quickly identify stakeholders and experts for questions and projects.

👍

Hint:

The information on teams can be further expanded upon in our Engineering Efficiency case, which adds metrics on performance and efficency.

Service dependencies:

In terms of discovering vulnerabilities and ensuring compliance, it is not unusual for some organizations who internally develop their own software to face challenging situations such as the following:

• A vulnerability in a library ubiquitously used across your microservices (e.g. Log4j) exposes your organization to attackers who could easily break into your systems to steal data and execute arbitrary code. Whether a specific component relies on the affected library is tribal knowledge, but there's a high chance that some of your systems handling personal and financial data rely on this library.

• When building software, it is fairly common for your developers to use open source components and libraries. Despite their efforts to ensure compliance, your legal/technical teams have a hard time to keep up with the complexity of your environment and the speed at which it evolves. For example, a recent terms and conditions change in one of the open source licenses used by your teams (e.g. Elasticsearch and Kibana change from Apache v2 license to SSPL in 2021) may put your organization at risk of being forced to release its intellectual property to everyone.

• One of your security scanning tools sends an alert about a DoS attack affecting a package (e.g. a XML parser) used across various of your projects. The tool's security report provides you with extensive details about the attack, its severity and list of affected projects. The information you have at hand is however only one piece of the whole puzzle to execute a mitigation plan and track its progress.

To tackle these challenge the catalog can be further extended to capture the dependencies of the technology you built - making insights into queries such as where certain libraries or licenses are used easily available.

LeanIX VSM then links software, dependencies and teams to enable engineers and architects to automate and streamline governance, and to identify and mitigate security vulnerabilities. This transparency created by VSM allows organizations to:

• Provide guidance on governance and compliance by setting standards and identifying services and vulnerabilities that do not meet defined requirements.
• Query the VSM inventory and find out which services are affected by a security vulnerability, their dependencies on other services, and which teams are responsible for them.

Tangible Outcomes - API Catalog:

To give engineers and developers quick and straightforward access to information around all the different APIs available in their organization the current best practice is to provide an API Catalog / Portfolio – all the relevant info on APIs, their functionality, availability status, owners, etc. and only that information free from further clutter – to speed up and ease development work by cutting down on time spend searching for documentation. VSM provides this in the form of one or more freely configurable API Catalogs, displaying the relevant API details, filter- and searchable, with baseline information and jump-off points to deeper sources:

API Visualizations:

To give both engineers and management an easy way to understand the API landscape they are working in or supervising simple and clear visualizations are essentially – at a glance they should allow you to easily understand dependencies, data flows and performance indicators without the need to crawl individual documentation.

VSM delivers these insights with just a couple of key reports:

API landscapes deliver a high-level overview of your API inventory, grouped by key categories (which APIs are used in which Product?) and color-coded to highlight relevant KPIs (which of my APIs are publicly available?). Fully configurable these allow lightweight reporting for top-level analytics .

What do I need?

Buy-in from product leadership (at least in your sub-organization) to boost transparency & knowledge sharing based on a central catalog

Access to at least one of the systems described at Discovery - Overview. We highly recommend starting with out-of-the-box integration to show results fast.

❗️

Don't shoot the messenger

Transparency can be painful. Reinsure yourself that you have the necessary buy-in, and be prepared to learn that establishing central standard might be tedious - yet, it's the base for every following optimization

How do I get there - Service Catalog:

👍

Make sure to try it yourself before continuing

It takes only a few minutes to try it yourself - make sure to have an overview of the system before moving on.

In the following, we will show a high-level walkthrough on setting up a Service Catalog based on a Github connection - make sure to read the User Guides to understand details.

To create an inventory of your services and the corresponding responsibilities, you start setting up a connection to your GitHub organization and read out all self-developed services based on your repositories' names and descriptions. If you use team entities in GitHub you can understand the responsibilities through the assigned teams or you can ask your developers to create GitHub topics with the team name, such as team-<team-name>.

Our out-of-the-box repository integration will do all the heavy lifting, all you need to provide is an API token. Learn more about our GitHub Repository Integration.

📘

Not using GitHub?

Discover your services and responsibilities with our out-of-the-box CI/CD or Public Cloud integrations.

Surfacing microservices and team responsibilities

Based on the Microservice Fact Sheet and the related Team Fact Sheet you can already understand who is owning which services. If you define a Domain-Model or Products in your organization you can map both Domain and Product Fact Sheets to your Microservice Fact Sheet to understand: Which Product/Domain is supported by what services and who owns those?

Based on automation, views like this service maintain accurate overviews and provide developers with a central knowledge base to look up service-related information and navigate your service landscape.

Enriching services with deployments & dependencies

Going into more depth of your service landscape, you can now provide additional information on how those services are being developed and deployed. This information is stored on the Deployment Fact Sheet to understand how a particular service is built and deployed at a specific point in time.

Both the Deployment and Library Fact Sheet are populated automatically through our CI/CD integration. You can leverage one of our native plugins for Jenkins, GitHub Actions or Azure Pipelines or hook any other pipeline up to our CI/CD API.

📘

Need more data? Customize our CI/CD integration

The CI/CD integration can also reinforce or even replace the aforementioned repository integration (e.g. with GitHub). Through a yaml manifest we allow you to add a variety of additional information about your microservices and team responsibilities.

VSM shows you the current live deployment while also allowing you to go back and see historical deployments. With this mechanism, you can understand what software libraries were introduced to a certain service at a certain point in time and what dependencies are currently deployed in production.

Understanding service infrastructure & architecture drift

To make the picture on our microservice catalog complete we also want to visualize how our services are running on production infrastructure. Complex pipeline setups with multiple developments and test stages can become overwhelming and it is easy to lose track of what build of your service is actually currently running in production and how they are configured.

Through our Kubernetes integration, we provide you a simple mechanism to understand what version of your service is running on which cluster. We deploy a simple agent to your cluster that scans your K8s objects and sends the information directly to VSM.

📘

No K8s? No problem!

With our existing customers, we have also worked on visualizing more traditional infrastructure setups. We are happy to support you!

The Compute Environment Fact Sheet displays your Clusters with their configuration and links them to the Fact Sheets of the microservices that they run. Additionally we add detailed information about your K8s configurations.

Additionally we create a Deployment Fact Sheet for any K8s deployment that we track on your cluster. By holding this against the Deployment Fact Sheet from your CICD build & deployment we can detect any infrastructure drift that might occur.

How do I get there - API Catalog:

To ensure all the insights above are consistently up to date automated discovery is a necessity – VSM supports a wide variety of integration options to ensure data can be collected from various systems your organization might be using, ranging from the consolidation of data from multiple API management systems (e.g. Mulesoft Anypoint, AWS API Gateway, etc.) as well as discovery from API specifications (OpenAPI, Swagger, etc.).

Depending on your current toolchain, VSM can be set up to support two scenarios:

API Management in place

An API Management tool is great to capture technical dependencies, but often struggles to:

• Embrace the domain or business context
• Create a clear linkage to responsibilities
• Provide easy entry points for non-experts
• Provide dynamic, graphical capabilities to explore impacts & dependencies

Those four points are addressed by VSM and can justify the co-existence of both toolings. As API Management solutions naturally provide open APIs as well, a connection should be fairly simple.

No API Management in place

In this case, adding to the points above VSM can support setting up an API catalog without a management tool. Approaches can be:

• Manual documentation (obviously only recommended as a stop-gap solution)
• Discovery of APIs based on CI/CD, e.g. by extracting Swagger definition files

📘

Information

Advanced use cases for product leaders include the import key metrics to VSM to evaluate your APIs in terms of performance and operability - see the following example. Also, we see customers leverage our Integration API to integrate with service meshes like https://istio.io/ - please get in touch with [email protected] to discuss options