Service Communication — Microservices Interview

Target: Senior Engineer · Engineering Lead · Pre-Architect Focus: gRPC vs REST, reliability patterns, API versioning, service mesh

Q: How do you ensure reliable inter-service communication with retries and timeouts?

Why interviewers ask this: Network failures are inevitable. Tests understanding of timeout strategies, exponential backoff, and when retries are safe.

Answer

The problem: Networks are unreliable. A 99.9% reliable service calling 10 services = 99% overall reliability (all succeed). Each retry risks cascading failures.

Solution hierarchy:

Failure handling order:
1. Timeout — don't wait forever (2-5 sec typical)
2. Detect failure → fast-fail
3. Retry with backoff (exponential: 100ms → 200ms → 400ms)
4. Circuit breaker — stop retrying if service is down
5. Fallback — return degraded response

Spring Boot example:

@Service
public class PaymentClient {

    @Retry(name = "payment", fallbackMethod = "paymentFallback")
    @CircuitBreaker(name = "payment")
    @TimeLimiter(name = "payment")
    public CompletableFuture<PaymentResponse> charge(PaymentRequest req) {
        return CompletableFuture.supplyAsync(() ->
            webClient.post()
                .uri("http://payment-service/charge")
                .bodyValue(req)
                .retrieve()
                .bodyToMono(PaymentResponse.class)
                .timeout(Duration.ofSeconds(2))
                .block()
        );
    }

    public CompletableFuture<PaymentResponse> paymentFallback(
            PaymentRequest req, Exception ex) {
        // Return degraded response
        return CompletableFuture.completedFuture(
            PaymentResponse.pending(req.orderId)
        );
    }
}

Configuration:

resilience4j:
  retry:
    instances:
      payment:
        max-attempts: 3
        wait-duration: 100ms
        exponential-backoff-multiplier: 2
        retry-exceptions:

          - java.net.ConnectException
          - java.io.IOException

  timelimiter:
    instances:
      payment:
        timeout-duration: 2s

Backoff strategy:

Attempt 1: immediate
Attempt 2: wait 100ms
Attempt 3: wait 200ms
Total max wait: 300ms before failing

Common Mistake

Retry on any exception = disaster. SocketTimeoutException → safe to retry. PaymentAlreadyProcessedException → fail immediately. Design idempotent operations before enabling retries on mutations.

Q: When should you use gRPC vs REST for inter-service communication?

Why interviewers ask this: Tech choice has cascading implications for performance, debugging, and team expertise. Tests architectural trade-off thinking.

Answer

Criteria	REST	gRPC
Serialization	JSON (text)	Protocol Buffers (binary)
Size	Large (~200 bytes)	Small (~50 bytes) — 4x smaller
Speed	Slower parsing	Fast binary parsing
Protocol	HTTP/1.1 (one request at a time)	HTTP/2 (multiplexed)
Streaming	No native support	Bidirectional streaming
Debugging	Easy (curl, browser)	Harder (need grpcurl)
Ecosystem	Mature, widely supported	Growing but less mature
Browser clients	✅ Native	❌ Requires gRPC-Web proxy
Use case	Public APIs, third-party clients	Internal service-to-service

Performance comparison:

Service A → Service B (processing 10,000 requests)

REST:

- Request size: 200 bytes × 10k = 2 MB
- Response size: 300 bytes × 10k = 3 MB
- Total: 5 MB over network
- Latency: ~50 ms per request (JSON parsing)

gRPC:

- Request size: 50 bytes × 10k = 500 KB
- Response size: 75 bytes × 10k = 750 KB
- Total: 1.25 MB over network (-75%)
- Latency: ~5 ms per request (binary parsing, HTTP/2)
- 10x faster per request

gRPC service definition:

syntax = "proto3";

service OrderService {
  rpc GetOrder(OrderId) returns (Order);
  rpc CreateOrder(OrderRequest) returns (OrderResponse);
  // Bidirectional streaming
  rpc ProcessOrders(stream Order) returns (stream OrderStatus);
}

message Order {
  string id = 1;
  int64 amount = 2;
  string status = 3;
}

gRPC + Spring Boot:

@GrpcService
public class OrderServiceImpl extends OrderServiceGrpc.OrderServiceImplBase {

    @Override
    public void getOrder(OrderId request, 
                        StreamObserver<Order> responseObserver) {
        Order order = orderService.findById(request.getId());
        responseObserver.onNext(order);
        responseObserver.onCompleted();
    }
}

Recommendation:

Use REST — Public APIs, third-party clients, simplicity required
Use gRPC — Internal service-to-service (10+ services), high throughput, streaming needs
Hybrid — REST for edge (API Gateway) → gRPC internally

Q: How do you implement API versioning without breaking clients?

Answer

Three versioning strategies:

Strategy	Example	Pros	Cons
URL path	`/v1/orders`, `/v2/orders`	Explicit, works with proxies	Duplicate code, routing complexity
Header	`Accept: application/json; version=2`	Single codebase, clean URLs	Less discoverable, cache issues
Query param	`/orders?api_version=2`	Flexible, easy to test	Cache-key issues, non-standard

Best practice — URL path with header fallback:

@RestController
@RequestMapping("/api")
public class OrderController {

    @GetMapping(
        "/v1/orders/{id}",
        produces = "application/json"
    )
    public OrderV1 getOrderV1(@PathVariable String id) {
        Order order = orderService.findById(id);
        // Map internal model to V1 (no new fields)
        return OrderV1.from(order);
    }

    @GetMapping(
        "/v2/orders/{id}",
        produces = "application/json"
    )
    public OrderV2 getOrderV2(@PathVariable String id) {
        Order order = orderService.findById(id);
        // Map to V2 (includes new fields, e.g., "estimatedDelivery")
        return OrderV2.from(order);
    }
}

API evolution best practices:

Add fields, never remove — Old clients ignore unknown fields
Default old fields — Always include fields from V1 in V2
Deprecation timeline — Announce 6–12 months before retiring API version
Semantic versioning — MAJOR.MINOR.PATCH (v1, v2, v1.1)

Deprecation example:

2024-01: Release /v2/orders (new field: tracking_id)
2025-01: Announce deprecation of /v1/orders
2025-07: Retire /v1/orders (clients must migrate)

Q: Should you implement a service mesh (Istio, Linkerd)?

Why interviewers ask this: Service mesh is a major operational investment. Tests cost/benefit thinking and maturity assessment.

Answer

Service mesh = sidecar proxies + control plane that handle:

Retries, circuit breaking, timeouts (without code changes)
Load balancing, traffic splitting (canary deployments)
mTLS encryption between all services
Distributed tracing, observability

Decision matrix:

Organization Stage	Use Service Mesh?	Why
< 5 microservices	❌ No	Overkill — use libraries (Resilience4j)
5-20 microservices	⚠️ Maybe	Only if you have Kubernetes + experienced ops team
20-50 microservices	✅ Yes	Centralized policy enforcement pays off
50+ microservices	✅ Definitely	Manual resilience in each service becomes unmaintainable

Tradeoffs:

Aspect	Pro	Con
Resilience	Automatic circuit breakers, retries in proxy	Added complexity, learning curve
Observability	Automatic tracing, metrics without code changes	More infrastructure to operate
Overhead	Centralized policy, no library duplication	~10% latency penalty, memory per pod
Debugging	Flow is visible in mesh	More tools to learn (istioctl)

Recommendation:

Start with Resilience4j libraries (simpler, less overhead)
Migrate to service mesh when you have 15+ services AND a dedicated platform team
Use managed service mesh (AWS App Mesh, Google Anthos) to reduce operational burden

Diagram — Complete Service Communication Architecture

graph LR
    Client["Client"]
    Gateway["API Gateway\n· REST /v2/\n· Rate limit"]
    SvcA["Service A"]
    SvcB["Service B"]
    Mesh["Service Mesh · Istio\n· mTLS\n· Circuit breaker\n· Tracing"]

    Client -->|REST\nHTTP/1.1| Gateway
    Gateway -->|gRPC\nHTTP/2| SvcA
    SvcA -->|gRPC\nHTTP/2| SvcB
    Mesh -.->|Sidecar proxy\nretry + circuit breaker| SvcA
    Mesh -.->|Sidecar proxy\nretry + circuit breaker| SvcB

    style Gateway fill:#4ecdc4
    style SvcA fill:#51cf66
    style SvcB fill:#ffe066
    style Mesh fill:#9b59b6

Advanced API Design

Q: What are the core best practices for designing a production-grade REST API?

Why interviewers ask this: REST is ubiquitous but poorly designed REST APIs cause breaking changes, client confusion, and performance problems. Tests pragmatic API craft beyond just "use HTTP verbs correctly".

Answer

HTTP methods and idempotency contract:

Method	Safe?	Idempotent?	Use Case
`GET`	✅	✅	Read resource
`HEAD`	✅	✅	Read headers only
`PUT`	❌	✅	Full replace — same result if repeated
`PATCH`	❌	❌ (by default)	Partial update — use idempotency key
`DELETE`	❌	✅	Delete — 200 or 404 on repeat
`POST`	❌	❌	Create — must add idempotency key for safety

Idempotency key pattern for non-safe operations:

// Client sends: POST /orders + Idempotency-Key: uuid-1234
@PostMapping("/orders")
public ResponseEntity<Order> createOrder(
        @RequestHeader("Idempotency-Key") String idempotencyKey,
        @RequestBody OrderRequest req) {

    // Check if this key was already processed
    Optional<Order> cached = idempotencyStore.get(idempotencyKey);
    if (cached.isPresent()) {
        return ResponseEntity.ok(cached.get());  // Return same response, no double-process
    }

    Order order = orderService.create(req);
    idempotencyStore.store(idempotencyKey, order, Duration.ofHours(24));
    return ResponseEntity.status(201).body(order);
}

HTTP status codes — use them precisely:

Code	Meaning	When to Use
`200 OK`	Success with body	GET, PUT, PATCH response
`201 Created`	Resource created	POST creating a new resource
`202 Accepted`	Async processing started	Long-running job queued
`204 No Content`	Success, no body	DELETE
`400 Bad Request`	Client sent invalid data	Validation error
`401 Unauthorized`	Not authenticated	Missing/invalid token
`403 Forbidden`	Authenticated but not authorised	Wrong role/scope
`404 Not Found`	Resource doesn't exist
`409 Conflict`	State conflict	Duplicate creation, optimistic lock fail
`422 Unprocessable Entity`	Semantically invalid	Business rule violation
`429 Too Many Requests`	Rate limited	Add `Retry-After` header
`503 Service Unavailable`	Downstream down	Circuit open

Pagination — always paginate collection endpoints:

// Cursor-based pagination (preferred for large datasets)
GET /orders?after=ord-cursor-xyz&limit=20

// Response includes navigation cursors
{
  "data": [...],
  "pagination": {
    "limit": 20,
    "next_cursor": "ord-cursor-abc",  // Opaque — don't expose DB offset
    "has_more": true
  }
}

// Spring Boot example
@GetMapping("/orders")
public PagedResponse<Order> getOrders(
        @RequestParam(required = false) String after,
        @RequestParam(defaultValue = "20") int limit) {

    List<Order> orders = orderRepository.findAfterCursor(after, limit + 1);
    boolean hasMore = orders.size() > limit;
    String nextCursor = hasMore ? orders.get(limit - 1).getId() : null;
    return new PagedResponse<>(orders.subList(0, Math.min(limit, orders.size())),
                               nextCursor, hasMore);
}

Why cursor over offset pagination:

Offset: GET /orders?page=5&size=20
  Problem: INSERT during pagination shifts rows → page 5 shows wrong data
  Problem: OFFSET 100 = DB scans and discards 100 rows (expensive at scale)

Cursor: GET /orders?after=last-seen-id&limit=20
  ✅ Stable — inserts don't affect cursor position
  ✅ O(1) — WHERE id > cursor uses index directly
  ❌ Can't jump to arbitrary page (rare need)

Content negotiation:

// Support multiple response formats via Accept header
@GetMapping(value = "/orders/{id}",
    produces = { MediaType.APPLICATION_JSON_VALUE, "application/vnd.api+json" })
public Order getOrder(@PathVariable String id) { ... }

// Versioning via media type (preferred for major breaking changes)
@GetMapping(value = "/orders/{id}",
    produces = "application/vnd.orders.v2+json")
public OrderV2 getOrderV2(@PathVariable String id) { ... }

Architect Insight

Use 202 Accepted + a job ID for any operation taking > 500ms. Return GET /jobs/{id} for the client to poll status. This prevents gateway timeouts, enables retries, and gives clients progress visibility — much better than holding the HTTP connection open.

Q: When would you choose GraphQL over REST or gRPC? What are the trade-offs?

Why interviewers ask this: GraphQL is increasingly common in BFF (Backend for Frontend) layers. Tests understanding of when client-driven queries add value versus complexity.

Answer

GraphQL lets clients specify exactly what data they need in a single query — solving over-fetching and under-fetching problems common with REST.

Criteria	REST	gRPC	GraphQL
Data shape	Fixed per endpoint	Fixed per proto	Client-defined per query
Over-fetching	Common (`GET /users` returns all fields)	Reduced (proto fields)	Eliminated — client selects fields
Under-fetching	N+1 requests for related data	Multiple calls	Single query for nested data
Performance	Good	Best (binary, HTTP/2)	Overhead from query parsing; use DataLoader
Type safety	No (OpenAPI optional)	Strong (protobuf)	Strong (schema-first)
Browser clients	✅ Native	❌ Needs proxy	✅ Native (POST /graphql)
Caching	✅ HTTP cache (GET)	Manual	❌ POST requests, manual persisted queries
Best for	Public APIs, CRUD	Internal high-throughput	BFF, aggregation layers, mobile clients

Spring Boot GraphQL example:

// Schema
// type Query { order(id: ID!): Order }
// type Order { id: ID!, status: String!, customer: Customer! }
// type Customer { id: ID!, name: String!, email: String! }

@Controller
public class OrderGraphQLController {

    @QueryMapping
    public Order order(@Argument String id) {
        return orderService.findById(id);
    }

    @SchemaMapping(typeName = "Order", field = "customer")
    public Customer customer(Order order) {
        // DataLoader batches N customer lookups into 1 query
        return customerService.findById(order.getCustomerId());
    }
}

N+1 problem — always use DataLoader in GraphQL:

// Without DataLoader: 1 query for orders + N queries for customers
// With DataLoader: 1 query for orders + 1 batch query for all customers

@Bean
public BatchLoaderRegistry batchLoaderRegistry() {
    return registry -> registry.<String, Customer>forTypePair(String.class, Customer.class)
        .registerMappedBatchLoader((customerIds, env) ->
            customerService.findAllByIds(customerIds)  // Single DB call
                .collectMap(Customer::getId)
        );
}

When to choose GraphQL:

✅ Use GraphQL when:
  - Mobile clients need bandwidth-optimised responses (request only needed fields)
  - BFF layer aggregates data from 5+ microservices into one response
  - Rapid product iteration — clients evolve queries without API changes
  - Complex nested entity graphs (social graphs, product catalogues with variants)

❌ Avoid GraphQL when:
  - Simple CRUD services (REST is simpler and more cacheable)
  - High-performance internal service calls (use gRPC)
  - Team unfamiliar with schema design and DataLoader patterns
  - Public rate-limited APIs (POST semantics break HTTP caching)

Architect Insight

A common architecture is REST or gRPC for internal service-to-service communication, and GraphQL only at the BFF/API Gateway layer facing the client. This keeps internal APIs simple and lets the edge layer optimise for client needs independently.