How to add observability into microservice architecture?

When we need microservice?

Microservice architecture distribute the business logics in different server.
Following are the benefit of microservice

1. Decomposition the service responsibility
2. Partial service redeployment
3. Scale the partial services according to the workload For example: In machine learning system, ml model usually require specific lib, hardware and more cpu resource. Putting ml model with business logic part into same service probably hurt other services performance.

The problems of microservice system

The services are distributed on different server, we have log files, cpu and memory metrics, IO latency to look up in multiple places. Compare to microservice system, monolithic is relative simple, inspection only required at single point.

The concepts of observability

Tracing

The painpoints of microservice architecture is we if don’t have metadata for aggregation, we need to trace logs, metrics in distributed places. Passing correlation ids between the services is key for aggregating distributed informations

Logging

Every log / request time interval in the downstream services, databases should be exported to backend collector, for later lookup.

Metrics

Recording the resource status of services. ex: CPU, memory, IO latency.

Monitor

Tracking the metrics for ensuring the status is keeping in some reasonable threshold, if anomaly is detected then send the alert

1. Keep tracking on the specific resource metrics (CPU, memory, IO…).
2. Fake event testing, ex: Build experiment should finish in 1 min

Observability Architecture Example