C#, DI, middleware, EF Core, async/await and Web APIs. Open a question to read the answer.
The Common Language Runtime (CLR) is the virtual machine that executes .NET code. The C# compiler does not emit native machine code; it emits Common Intermediate Language (IL) plus metadata into an assembly. At run time the CLR loads that assembly, verifies it, and the Just-In-Time (JIT) compiler (RyuJIT) translates each method’s IL into native code the first time the method is called. The CLR also provides the garbage collector, type safety, exception handling, and the thread pool that ASP.NET Core runs on.
Modern .NET (8, 9 and the current LTS, .NET 10, released November 2025) uses tiered compilation: a method is first jitted quickly with minimal optimisation (tier 0), call counts are tracked, and hot methods are re-jitted at tier 1 with full optimisation. Dynamic PGO (on by default since .NET 8) feeds observed branch and type information into that tier-1 recompile, so devirtualisation and inlining decisions are driven by real traffic. ReadyToRun (R2R) precompiles IL to native at publish time to cut startup, and Native AOT removes the JIT entirely for self-contained binaries.
This pipeline shows how source becomes running code.

Interview takeaway: the JIT is why a freshly started ASP.NET Core service is slower for its first few seconds and why warm-up or R2R matters for latency-sensitive deployments.
Apply. In a payments API running in Kubernetes, the runtime decisions I make are about startup and steady state. I publish with <PublishReadyToRun>true</PublishReadyToRun> so pods pass readiness probes quickly after a scale-out, keep tiered compilation on so steady-state code still gets PGO-optimised, and set <TieredPGO>true</TieredPGO> explicitly in the csproj so nobody disables it by accident. For a small CLI tool shipped as a sidecar I use Native AOT instead, accepting the reflection restrictions. I pin the runtime with a global.json and a rollForward policy so CI and production agree on the exact SDK.
Test. Startup and warm-up are measured, not assumed: a BenchmarkDotNet suite covers hot serialisation paths, and a k6 load test fires traffic at a cold pod and records p99 for the first 60 seconds versus minute five. The CI pipeline runs dotnet publish with the production flags so trimming or AOT breakage (missing reflection metadata) fails the build, not the release.
Troubleshoot. Symptoms: slow first requests after every deployment, or a method that is mysteriously slow only in production. I check DOTNET_TieredCompilation / DOTNET_TieredPGO environment variables that an ops template may have set, inspect dotnet-counters monitor for jit-methods-jitted and time-in-jit, and use dotnet-trace collect --profile cpu-sampling with PerfView to see whether time is in tier-0 code. For AOT, the ILC warnings list exactly which reflection call cannot be resolved.
An important consideration: R2R code is lower quality than tier-1 JIT output, so disabling tiered compilation to “simplify” things usually makes steady-state throughput worse.
Managed memory means the CLR, not the developer, allocates objects on the managed heap and reclaims them when no live reference remains. The garbage collector is generational and tracing: new objects land in gen 0; survivors are promoted to gen 1 and then gen 2. Gen 0 collections are frequent and cheap because most objects (request DTOs, strings, closures) die young; gen 2 collections are rare and expensive because they walk the whole heap. Objects of 85,000 bytes or more go straight to the Large Object Heap (LOH), which is collected with gen 2 and is only compacted on demand.
Key behaviours that matter in ASP.NET Core:
IDisposable and using exist because the GC knows nothing about unmanaged resources (sockets, file handles, DB connections); finalizers are a safety net, not a strategy.DOTNET_GCHeapHardLimit or GCHeapHardLimitPercent can cap it further. .NET 8 added DATAS (dynamic adaptation to application sizes), which is on by default from .NET 9 and resizes the number of server heaps with load.The generations and promotion path look like this.

Apply. For an order-ingestion service on 2-vCPU pods I choose the GC mode deliberately: Server GC with DATAS so heaps shrink when traffic drops, and GCHeapHardLimitPercent set so the runtime never fights the container limit. Code-level choices follow the generational model: rent buffers from ArrayPool<byte>.Shared instead of allocating 1 MB arrays per request (which would hit the LOH), stream large responses rather than building them in memory, and avoid caching per-request objects in static dictionaries, which is how short-lived objects turn into gen 2 garbage.
Test. Allocation is a measured regression, not a feeling. BenchmarkDotNet with [MemoryDiagnoser] tracks bytes allocated per operation for the hot serialisation path and fails the build if it grows. A soak test runs the service at steady load for an hour while dotnet-counters records gc-heap-size, gen-2-gc-count and time-in-gc; a flat heap and a stable gen 2 rate mean no leak. Unit tests assert that IDisposable wrappers (a HttpResponseMessage, a DbContext) are disposed by using a counting fake.
Troubleshoot. The classic symptom is a pod that is OOMKilled every few hours. I take a dotnet-gcdump (or dotnet-dump collect) from a live pod and compare two snapshots in Visual Studio or dotnet-dump analyze with dumpheap -stat; the type whose count grows between snapshots is the leak, and gcroot shows who holds it. Common culprits: event handlers never unsubscribed, a singleton holding a List<T> that only grows, MemoryCache entries without size limits, and captured HttpContext in a background task. High time-in-gc with a stable heap instead points to LOH churn; dumpheap -min 85000 finds the big arrays.
An important consideration: GC.Collect() in production code is almost always a bug; it forces a blocking gen 2 and hides the real allocation problem.
A value type (struct, enum, and the built-ins such as int, double, bool, DateTime, Guid) holds its data directly: a variable is the value, assignment copies it, and two variables never alias the same storage. A reference type (class, record class, interface, delegate, string, arrays) is a pointer to an object on the managed heap: assignment copies the reference, so two variables can observe the same object and null is a legal value.
| Aspect | Value type | Reference type |
|---|---|---|
| Storage | Inline where declared (stack, register, or inside the containing object) | Heap object; variable holds a reference |
| Assignment | Copies the whole value | Copies the reference only |
| Equality default | Field-by-field (record struct generates it) | Reference identity unless overridden |
| Nullability | Needs Nullable<T> (int?) | Nullable by design; nullable reference types add compiler analysis |
| GC cost | None by itself; boxing allocates | Every instance is a GC allocation |
Where it shows up in ASP.NET Core: DateTimeOffset, Guid, decimal are structs, so passing them is cheap and there is nothing to dispose or null-check. Span<T> and ReadOnlySpan<T> are ref structs that cannot escape to the heap, which is exactly what makes them safe for zero-copy parsing. The common bug is mutating a struct through a copy:
public struct Money { public decimal Amount; }
var prices = new List<Money> { new() { Amount = 10m } };
prices[0].Amount = 20m; // CS1612: indexer returns a copy, not a reference
var fixedPrices = new Money[1];
fixedPrices[0].Amount = 20m; // arrays expose the element itself; this worksPrefer readonly record struct for small immutable values and class / record for anything with identity or a lifecycle.
Apply. In a pricing service I model identifiers and amounts as readonly record struct types (OrderId, Money) so they are allocation-free, compared by value, and impossible to confuse with a raw Guid at a call site. Entities with identity (Order, Customer) stay classes because EF Core tracks them by reference. For high-volume parsing of CSV uploads I use ReadOnlySpan<char> and Utf8Parser to avoid creating a string per field. Generic collections (List<int>, Dictionary<Guid, Order>) are used precisely because they avoid boxing; non-generic ArrayList, Hashtable or object parameters are rejected in review.
Test. Equality semantics are tested explicitly: Assert.Equal(new Money(10m, "EUR"), new Money(10m, "EUR")) for the struct, and a test proving two Order instances with the same data are not equal unless the domain says so. BenchmarkDotNet with [MemoryDiagnoser] guards the parser’s “0 B allocated” claim. Nullable reference types are enabled (<Nullable>enable</Nullable>) with warnings as errors, so a missing null check is a compile failure rather than a test.
Troubleshoot. A NullReferenceException in production almost always traces back to a reference type arriving where a value was assumed, typically a deserialised DTO with a missing property; enabling required members (C# 11) makes the binder fail fast instead. Unexpected allocations in a profiler (dotnet-trace, PerfView’s allocation view) with System.Int32 or Boxed ValueTuple as the type point to boxing through interfaces, string.Format with struct arguments, or LINQ over IEnumerable<object>. A struct “not updating” is the copy-semantics bug; the fix is to make it a class or return by ref.
An important consideration: large mutable structs (over roughly 16 to 24 bytes) are copied on every pass and often cost more than a heap allocation would.
async / await is compiler-generated continuation passing. A Task (or Task<T>) is a promise of a future result; await registers the rest of the method as a continuation and returns control to the caller immediately if the task is not yet complete. No thread is blocked while the awaited I/O (database call, HTTP request, file read) is in flight, which is what lets an ASP.NET Core server handle thousands of concurrent requests with a few dozen threads.
Essential facts:
async method into a state machine; each await on an incomplete task is a suspension point. If the task is already complete, execution continues synchronously with no scheduling cost.SynchronizationContext, so continuations resume on any thread-pool thread and ConfigureAwait(false) is unnecessary in application code (still recommended in libraries).ValueTask<T> avoids allocating a Task when the result is usually available synchronously (cache hits, buffered reads), but may be awaited only once.async void is only for event handlers; exceptions from it crash the process because nobody observes them.CancellationToken should flow through every async call; ASP.NET Core supplies HttpContext.RequestAborted.app.MapGet("/orders/{id:guid}",
async (Guid id, OrdersDb db, CancellationToken ct) =>
{
var order = await db.Orders
.AsNoTracking()
.FirstOrDefaultAsync(o => o.Id == id, ct); // thread released here
return order is null ? Results.NotFound() : Results.Ok(order);
});The thread that started the request is returned to the pool at the await and a possibly different thread finishes the response.
The diagram shows what the compiler-generated state machine does at each await.

Apply. The rule in an order service is “async all the way”: controllers, application services, EF Core calls and HttpClient calls all return Task, and every method takes a CancellationToken that originates from HttpContext.RequestAborted so an abandoned browser tab does not keep a SQL query running. Fan-out work (fetching inventory and pricing for a basket) uses Task.WhenAll rather than sequential awaits. CPU-bound work is not wrapped in Task.Run inside request handlers; it either runs inline or is queued to a background Channel<T> consumer.
Test. Unit tests use xUnit async Task test methods and assert cancellation behaviour with a pre-cancelled CancellationTokenSource, expecting OperationCanceledException. For concurrency bugs I write a test that awaits the same ValueTask twice to prove the guard throws, and tests for WhenAll paths assert that one failed task surfaces as AggregateException handling. Analyzer rules (CA2007, CA2012, VSTHRD package, and AsyncFixer) run in CI so .Result, .Wait() and un-awaited tasks fail the build.
Troubleshoot. The signature of sync-over-async is throughput that collapses under load while CPU stays low: dotnet-counters shows threadpool-queue-length climbing and threadpool-thread-count creeping up by one or two per second (hill-climbing injection). A dotnet-dump with clrstack -all or pstacks shows dozens of threads blocked in Task.Wait or GetResult. The fix is to remove the blocking call and make the chain async; in the meantime DOTNET_ThreadPool_MinThreads buys time. Fire-and-forget tasks that swallow exceptions are found by subscribing to TaskScheduler.UnobservedTaskException and logging.
An important consideration: await inside a lock is a compile error, but SemaphoreSlim.WaitAsync is the right primitive and must always be released in a finally.
The .NET thread pool is a process-wide set of worker threads plus a separate set of I/O completion threads. Every Task.Run, timer callback, async continuation and Kestrel request is a work item queued to it. Because the pool is shared, ASP.NET Core does not create a thread per request; a few dozen threads service thousands of in-flight requests as long as those requests await rather than block. The minimum thread count defaults to the processor count, and the pool injects more threads using a hill-climbing algorithm when work items wait too long, roughly one or two new threads per second, which is why blocked threads cause slow degradation rather than an immediate failure. .NET 6 and later also detect blocking on tasks and inject threads faster for that specific case.
Concurrency primitives, from most to least preferred in ASP.NET Core code:
Channel<T> for producer/consumer handoff to background workers.ConcurrentDictionary<K,V>, ConcurrentQueue<T> for shared caches and queues.SemaphoreSlim to throttle outbound calls (it has WaitAsync, so it composes with async).lock (and the .NET 9 System.Threading.Lock type) for short, synchronous critical sections only.Parallel.ForEachAsync for CPU-bound fan-out in batch jobs, not in request handlers.Request-scoped state must not be stored in static fields or singletons; it is shared across all concurrent requests and becomes a data race. AsyncLocal<T> is the mechanism behind HttpContextAccessor and Activity.Current for flowing per-flow values across awaits.
Apply. In a notification service that must call a third-party SMS gateway limited to 50 concurrent connections, I wrap the client in a SemaphoreSlim(50) and await it, so excess requests queue without consuming threads. Inbound events are written to a bounded Channel<Notification> and drained by a BackgroundService, which decouples HTTP response time from delivery time and gives natural backpressure (BoundedChannelFullMode.Wait). Shared idempotency keys live in a ConcurrentDictionary<string, DateTimeOffset> with periodic eviction. Nothing request-specific is kept in a singleton; anything that must be shared is immutable or guarded.
Test. Concurrency tests are deliberately adversarial: a test starts 1,000 tasks hitting the idempotency store with 10 distinct keys and asserts exactly 10 side effects happened. Another test uses a TaskCompletionSource to hold the semaphore and asserts that the 51st call is still pending. Load tests in k6 push past the expected concurrency so thread injection behaviour is observed before production does it.
Troubleshoot. Thread-pool starvation presents as rising latency with low CPU. dotnet-counters monitor System.Runtime shows threadpool-queue-length growing and threadpool-thread-count climbing slowly; dotnet-stack report or a dump with clrstack -all shows what the threads are stuck on, usually .Result on an HTTP call or a synchronous SqlConnection.Open. Deadlocks in a lock are visible the same way with syncblk. A race corrupting a shared Dictionary<,> throws InvalidOperationException: Operations that change non-concurrent collections must have exclusive access and is fixed by ConcurrentDictionary or by removing the shared state.
An important consideration: raising ThreadPool.SetMinThreads masks starvation rather than fixing it and increases context switching once the real problem is removed.
ASP.NET Core ships a built-in IoC container (Microsoft.Extensions.DependencyInjection) in which every registration has one of three lifetimes that decide when the container creates and disposes an instance.
| Lifetime | Instance created | Typical use |
|---|---|---|
| Transient | Every time it is requested | Lightweight, stateless helpers, validators, mappers |
| Scoped | Once per scope; in ASP.NET Core a scope is one HTTP request | DbContext, unit of work, per-request caches, current-user accessor |
| Singleton | Once per application; disposed at shutdown | Configuration, HttpClient handlers, caches, loggers, thread-safe clients |
The diagram shows how each lifetime maps onto the root provider and per-request scopes, including the captive-dependency mistake.

The one rule interviewers probe is the captive dependency: a singleton must not depend on a scoped or transient service, because the container would resolve that dependency once and hold it forever. A singleton holding a scoped DbContext means every request shares one context, which is both a threading bug and a memory leak. The container throws at startup in Development because ValidateScopes is enabled there, but not in Production unless UseDefaultServiceProvider(o => o.ValidateScopes = true) is set.
builder.Services.AddSingleton<IClock, SystemClock>();
builder.Services.AddScoped<IOrderRepository, EfOrderRepository>();
builder.Services.AddTransient<IValidator<CreateOrder>, CreateOrderValidator>();
builder.Services.AddDbContext<OrdersDb>(o => o.UseNpgsql(conn)); // scopedDisposal follows the lifetime: scoped and transient IDisposable services resolved from a scope are disposed when the scope ends, singletons at host shutdown. Background services, which are singletons, create their own scope with IServiceScopeFactory.CreateScope() to use scoped services correctly.
Apply. In an order service the DbContext, the current-tenant accessor and the unit of work are scoped so one request sees one consistent transaction. The IPaymentGatewayClient is a typed HttpClient (transient wrapper over a pooled handler), the in-memory product cache and the IClock are singletons, and FluentValidation validators are transient. The BackgroundService that drains the outbox injects IServiceScopeFactory, opens a scope per batch, resolves the repository from that scope, and disposes it; it never injects the DbContext directly. ValidateScopes and ValidateOnBuild are turned on in every environment, not just Development, so a misregistration crashes at startup rather than at 3 a.m.
Test. A startup test builds the real WebApplicationFactory<Program> and resolves every registered service type from a scope; ValidateOnBuild plus that loop catch missing registrations and captive dependencies in CI. Behavioural tests for scoped services assert that two resolutions inside one scope return the same instance and two scopes return different ones. Replacing registrations in tests uses ConfigureTestServices with RemoveAll<T> before adding the fake.
Troubleshoot. Symptoms map cleanly to lifetime mistakes. “A second operation was started on this context instance before a previous operation completed” means a DbContext is shared across threads, almost always a captive dependency in a singleton or a static. “Cannot access a disposed object” after a request completes means a scoped service was captured by a fire-and-forget task. “Cannot resolve scoped service from root provider” means someone called app.Services.GetRequiredService<T>() instead of creating a scope. The fix is always to shorten the consumer’s lifetime or to inject a factory.
An important consideration: transient IDisposable services resolved from the root provider are never disposed until shutdown, which is a slow leak that only shows up in long-running processes.
The middleware pipeline is the ordered chain of components that every HTTP request passes through in ASP.NET Core. Each middleware receives the HttpContext and a RequestDelegate next; it can act before calling next, short-circuit by not calling it, and act again after next returns as the response flows back out. The order in Program.cs is the order of execution, which is why exception handling goes first and endpoint execution goes last.
The request travels inward and the response travels outward through the same components.

Framework middleware is added with Use* methods and custom logic can be inline or a class:
app.Use(async (ctx, next) =>
{
ctx.Response.Headers.Append("X-Request-Id", ctx.TraceIdentifier);
await next(ctx); // everything after runs on the way out
});
app.UseExceptionHandler();
app.UseHttpsRedirection();
app.UseAuthentication();
app.UseAuthorization();
app.MapControllers();UseRouting and UseEndpoints are added implicitly by WebApplication at the start and end of the pipeline. Map("/path", ...) branches the pipeline by path and UseWhen branches by predicate while rejoining the main pipeline. Endpoint filters (minimal APIs) and MVC action filters run inside the endpoint and are therefore not middleware, which matters when deciding where cross-cutting logic belongs.
Apply. In a multi-tenant API I write a TenantResolutionMiddleware that reads the X-Tenant header or subdomain, loads the tenant into a scoped ITenantContext, and short-circuits with 400 if it is missing; it sits after authentication (so it can also check the token’s tenant claim) and before authorization policies that depend on it. Correlation-ID propagation, request logging with timing, and a maintenance-mode switch are also middleware because they need to see every request including static files and health checks. Anything that needs the bound endpoint (rate limit policy names, authorization metadata) is placed after UseRouting, which WebApplication adds for me at the start.
Test. Middleware is unit-tested in isolation with a DefaultHttpContext and a RequestDelegate stub that records whether it was called; that proves both the short-circuit and the pass-through paths. Ordering is tested end-to-end with WebApplicationFactory: a request without the tenant header must return 400 even for an authenticated user, and a request to /healthz must bypass tenant resolution. A test also asserts the X-Request-Id header is present on error responses, which proves the header middleware runs outside the exception handler.
Troubleshoot. “Headers are read-only, response has already started” means a middleware tried to set a header after next completed and the body was flushed; use ctx.Response.OnStarting(...) instead. Authorization that never triggers usually means UseAuthorization was placed before UseAuthentication or before routing. CORS failures that appear only on real browsers trace to UseCors placed after an endpoint that short-circuits preflight. To see the real order I enable Microsoft.AspNetCore logging at Debug, which prints each middleware entering and leaving, or inspect the IApplicationBuilder registration order in a startup test.
An important consideration: middleware that reads the request body must call EnableBuffering() first or the model binder downstream will see an empty stream.
Routing maps an incoming URL and HTTP method to an endpoint: a minimal API handler, a controller action, a Razor page, a SignalR hub or a gRPC service. It is endpoint routing, meaning UseRouting selects the endpoint early in the pipeline and attaches it to HttpContext, so later middleware (authorization, CORS, rate limiting) can read the endpoint’s metadata before the handler executes. UseEndpoints (implicit in WebApplication) actually invokes it.
Route templates combine literals, parameters and constraints:
/orders/{id:guid} — typed parameter; a non-GUID yields 404, not 400./files/{*path} — catch-all./reports/{year:int:min(2000)} — multiple constraints./items/{id?} and /{page=1} — optional and default values.Attribute routing on controllers ([Route("api/[controller]")], [HttpGet("{id}")]) and MapGet / MapPost on minimal APIs compile to the same endpoint data structures. Conventional routing (MapControllerRoute with {controller=Home}/{action=Index}) remains for MVC views.
var orders = app.MapGroup("/api/v1/orders").RequireAuthorization();
orders.MapGet("/", ListOrders);
orders.MapGet("/{id:guid}", GetOrder).WithName("GetOrder");
orders.MapPost("/", CreateOrder);
// LinkGenerator resolves "GetOrder" to /api/v1/orders/{id} for Location headersMatching is deterministic: literals beat parameters, constrained parameters beat unconstrained, and two endpoints with the same precedence throw AmbiguousMatchException at request time. Route values flow into model binding, and LinkGenerator builds URLs from endpoint names so paths are never hard-coded in responses.
Apply. For a public catalogue API I group endpoints with MapGroup("/api/v{version:apiVersion}/products") so versioning, authorization and OpenAPI tags are declared once. Resource identifiers are constrained ({sku:regex(^[A-Z0-9-]{{4,20}}$)}) so malformed input never reaches the handler, and a custom IRouteConstraint validates tenant slugs against a cached list. Every MapPost returns Results.CreatedAtRoute("GetProduct", new { sku }, dto) so the Location header is generated by LinkGenerator, which keeps links correct when a prefix changes behind a gateway (UsePathBase handles the gateway prefix).
Test. Route tests run against WebApplicationFactory: a theory enumerates URLs and expected status codes (/api/v1/products/ABC-1 is 200, /api/v1/products/abc is 404 because the constraint fails, /api/v1/products/ without auth is 401). A startup test enumerates EndpointDataSource.Endpoints and asserts there are no duplicate route patterns with the same HTTP method, which catches ambiguity before a user does. Link generation is tested by asserting the Location header of a create response resolves to a 200.
Troubleshoot. A 404 that “should” match is diagnosed by enabling Microsoft.AspNetCore.Routing at Debug; the log shows candidate endpoints and why each was rejected (usually a constraint or the HTTP method). AmbiguousMatchException in production lists the competing endpoints; the fix is a more specific template or an [Order]. Behind a reverse proxy that strips a prefix, routes fail until UsePathBase("/catalog") is added before routing and ForwardedHeaders are configured. Catch-all parameters that swallow API routes are moved after specific endpoints or given a constraint.
An important consideration: route constraints are for disambiguation, not validation; a failing constraint produces 404, so business validation belongs in the handler where it can return 400 with details.
Both are ways to define HTTP endpoints on the same routing and hosting infrastructure. Minimal APIs (since .NET 6) register handlers directly with MapGet / MapPost lambdas or method groups and rely on compile-time or reflection-based parameter binding. Controllers (MVC/Web API) are classes deriving from ControllerBase with attribute-routed action methods and a rich filter pipeline.
| Aspect | Minimal APIs | Controllers |
|---|---|---|
| Definition | Lambdas or static methods on WebApplication / RouteGroupBuilder | [ApiController] classes with action methods |
| Cross-cutting | Endpoint filters, MapGroup conventions | Action/result/exception filters, model-state auto 400 |
| Binding | Explicit; [FromBody] inferred for complex types; TryParse / BindAsync | Full IModelBinder system, ModelState, IValidatableObject |
| Native AOT | Supported via Request Delegate Generator (.NET 8+) | Not AOT-compatible |
| Fit | Microservices, small focused APIs, high-throughput, AOT | Large APIs with many conventions, views, legacy filters |
Performance differences are small on .NET 8 and later; the real distinction is ceremony versus structure. A minimal API endpoint and its equivalent controller:
// Minimal API
app.MapPost("/orders",
async (CreateOrder cmd, IOrderService svc, CancellationToken ct) =>
Results.Created($"/orders/{await svc.CreateAsync(cmd, ct)}", null));
// Controller
[ApiController, Route("orders")]
public sealed class OrdersController(IOrderService svc) : ControllerBase
{
[HttpPost]
public async Task<IActionResult> Create(CreateOrder cmd, CancellationToken ct)
=> Created($"/orders/{await svc.CreateAsync(cmd, ct)}", null);
}Since .NET 7 minimal APIs have filters, route groups and typed results (Results<Ok<T>, NotFound>), closing most of the feature gap except built-in model validation, which .NET 10 adds through AddValidation() with data annotations.
Apply. I pick per service, not per company. A new inventory microservice with 20 endpoints uses minimal APIs organised by feature: each feature folder has a static MapInventoryEndpoints(this RouteGroupBuilder) extension, handlers are static methods (not lambdas) so they are unit-testable, and typed results (Results<Ok<StockDto>, NotFound, ValidationProblem>) document responses for OpenAPI. An existing back-office API with 200 actions, custom action filters and IModelBinder implementations stays on controllers; rewriting it buys nothing. Where the two meet in one host, both are mapped and share the same middleware, auth and ProblemDetails configuration.
Test. Static minimal API handlers are called directly in unit tests with fake services and the typed result is pattern-matched (Assert.IsType<NotFound>(result.Result)), no HTTP involved. Controllers are tested the same way by constructing the controller with mocks. Integration tests via WebApplicationFactory are identical for both styles, which is the point: the HTTP contract, not the implementation style, is what the tests lock down. A snapshot test of the generated OpenAPI document catches an endpoint whose response metadata changed.
Troubleshoot. With minimal APIs the common failure is binding: “Body was inferred but the method does not allow inferred body parameters” on a GET, or a complex type that silently binds to null because it has no TryParse / BindAsync. [AsParameters] fixes query-object binding. In controllers the equivalent symptom is ModelState invalid with no explanation, usually because [ApiController] inferred [FromBody] on the wrong parameter. Missing validation on minimal APIs before .NET 10 is solved with an endpoint filter that runs FluentValidation, or by enabling AddValidation() on .NET 10.
An important consideration: mixing both styles in one service is fine technically but confuses teams; the pipeline, filters and error format should be configured once so behaviour is identical regardless of style.
Model binding converts raw HTTP data (route values, query string, headers, form fields, JSON body) into the typed parameters of an endpoint; validation then checks that the bound object satisfies declared rules before the handler runs.
Binding sources and their precedence in controllers: route data, then query string, then form fields; the body is bound only with [FromBody] or when [ApiController] infers it for complex types. Minimal APIs infer similarly: route and query for simple types, body for complex types, and services from DI. Custom types participate by implementing static bool TryParse(string, out T) or static ValueTask<T?> BindAsync(HttpContext). JSON uses System.Text.Json by default, which is case-insensitive for property names in web defaults and rejects trailing commas and comments unless configured.
Validation in controllers uses System.ComponentModel.DataAnnotations attributes plus IValidatableObject; [ApiController] turns an invalid ModelState into an automatic 400 ValidationProblemDetails. Minimal APIs gained the same built-in behaviour in .NET 10 with builder.Services.AddValidation(); earlier versions use an endpoint filter or FluentValidation.
public sealed record CreateOrder(
[property: Required, StringLength(50)] string CustomerRef,
[property: Range(1, 1000)] int Quantity,
[property: Required, EmailAddress] string NotifyEmail);
// Invalid body produces RFC 9457 output:
// { "title": "One or more validation errors occurred.", "status": 400,
// "errors": { "Quantity": ["The field Quantity must be between 1 and 1000."] } }Binding failures and validation failures are distinct: a non-numeric Quantity is a binding error that still appears in ModelState; a Quantity of 0 binds fine and fails validation. Both end up in the same 400 response.
Apply. In a checkout API, request contracts are sealed record DTOs with required members so a missing field fails deserialisation instead of arriving as null. Data annotations cover shape (lengths, ranges, formats) and FluentValidation handles rules that need services, such as “the coupon code must exist and be active”, registered as scoped validators invoked from an endpoint filter. Domain identifiers like OrderId implement TryParse so they bind straight from the route. JsonSerializerOptions are configured once (JsonStringEnumConverter, UnmappedMemberHandling.Disallow for strict contracts) so unknown properties are rejected rather than ignored.
Test. Validation rules are unit-tested against the validator class with TestValidate, asserting the exact property and error code, which is faster than HTTP. Contract tests through WebApplicationFactory post deliberately malformed JSON (wrong type, missing required, extra field) and assert the 400 body matches the ProblemDetails shape and the errors keys, because clients depend on those keys. A happy-path test asserts the bound object equals the expected record, catching case-sensitivity and enum-serialisation regressions.
Troubleshoot. A property that is always null or default means the binder could not find a source: check the JSON casing, a missing public setter or init, or a constructor whose parameter name differs from the property. A 415 means the Content-Type header is not application/json. “The JSON value could not be converted” with a path points to an enum sent as a string without the converter, or a DateOnly / TimeOnly format issue. Validation that “does not fire” on minimal APIs before .NET 10 is because there is no automatic validation; the filter is missing. Request logging at Information for Microsoft.AspNetCore.Mvc.ModelBinding shows each parameter’s chosen source.
An important consideration: validation attributes run only on the top-level object unless the nested type is also annotated or [ValidateComplexType] -style recursion is configured; nested DTOs are a frequent blind spot.
Authentication establishes who the caller is by validating credentials (a cookie, a bearer token, a client certificate) and producing a ClaimsPrincipal. Authorization decides what that principal may do, evaluated per endpoint against policies, roles or resource-based rules. In ASP.NET Core they are separate middleware: UseAuthentication() populates HttpContext.User from the registered scheme’s handler; UseAuthorization() reads the endpoint’s [Authorize] metadata and runs IAuthorizationService. The order matters, and both must sit after routing.
The diagram shows where each middleware runs and which failure produces which status code.

| Aspect | Authentication | Authorization |
|---|---|---|
| Question | Who are you? | Are you allowed to do this? |
| Failure status | 401 Unauthorized (challenge) | 403 Forbidden |
| Configured via | AddAuthentication().AddJwtBearer()/.AddCookie() schemes | AddAuthorizationBuilder().AddPolicy(...) |
| Output | ClaimsPrincipal on HttpContext.User | Allow/deny for an endpoint or resource |
| Extension point | AuthenticationHandler<TOptions> | IAuthorizationHandler, IAuthorizationRequirement |
Policies compose requirements, and a requirement handler can inspect the resource:
builder.Services.AddAuthorizationBuilder()
.AddPolicy("CanRefund", p => p
.RequireAuthenticatedUser()
.RequireClaim("scope", "payments.refund")
.AddRequirements(new SameTenantRequirement()));
app.MapPost("/refunds", Refund).RequireAuthorization("CanRefund");Multiple schemes can coexist (cookies for the browser UI, JWT bearer for the API) with a policy scheme or [Authorize(AuthenticationSchemes = ...)] selecting which applies. FallbackPolicy makes every endpoint require authentication unless it is marked [AllowAnonymous], which is the safer default for an API.
Apply. For a B2B invoicing API I configure JWT bearer against the company identity provider and set a FallbackPolicy requiring an authenticated user, so a new endpoint is secure by omission. Authorization is policy-based, never role-string checks in handlers: "invoices.read" / "invoices.write" scope policies plus a resource-based IAuthorizationHandler that confirms the invoice’s TenantId matches the token’s tid claim, invoked with IAuthorizationService.AuthorizeAsync(user, invoice, "OwnsInvoice") after the entity is loaded. Claims transformation (IClaimsTransformation) adds internal permissions from the database once per request so policies stay declarative.
Test. Integration tests register a test authentication handler via ConfigureTestServices that reads claims from a request header, letting a test say “act as tenant A with scope invoices.read” without a real token. Theories cover the matrix: no token yields 401, token without scope yields 403, correct scope on another tenant’s invoice yields 403, and the happy path yields 200. The FallbackPolicy is proven by a test that enumerates EndpointDataSource and asserts every endpoint has either an authorization or an [AllowAnonymous] attribute. Policy handlers are unit-tested with a fabricated AuthorizationHandlerContext.
Troubleshoot. A 401 with a valid-looking token is diagnosed from Microsoft.AspNetCore.Authentication Debug logs, which print the exact reason (audience mismatch, expired, signature key not found). A 403 where the claim “is there” is usually the claim type: the handler maps scope to a space-separated single claim or renames role to the long schema URI unless MapInboundClaims = false. A 200 on an endpoint that should be protected means UseAuthorization is missing or the endpoint was mapped before it in a Map branch.
An important consideration: authorization decisions that depend on database state must be re-evaluated per request; caching a ClaimsPrincipal across requests is a privilege-escalation bug.
A JSON Web Token (JWT) is a signed, base64url-encoded set of claims (sub, iss, aud, exp, scopes) that an API can validate locally without calling the issuer, using the issuer’s public keys. OpenID Connect (OIDC) is the authentication protocol layered on OAuth 2.0 that defines how a client obtains those tokens from an identity provider (Entra ID, Keycloak, Auth0, Duende IdentityServer) through flows such as authorization code with PKCE, and how discovery (/.well-known/openid-configuration) and key rotation (JWKS) work.
In ASP.NET Core the two sit on different sides:
AddJwtBearer(): it downloads the discovery document, caches the signing keys, and validates signature, issuer, audience, lifetime (with a default 5-minute clock skew) on every request. No session state is kept.AddOpenIdConnect() plus AddCookie(): OIDC runs the browser redirect dance once, then the resulting identity is stored in an encrypted cookie for subsequent requests.The authorization code flow with an SPA or server app and a downstream API looks like this.

builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddJwtBearer(o =>
{
o.Authority = "https://login.example.com/realms/shop";
o.Audience = "orders-api";
o.TokenValidationParameters.ValidateIssuerSigningKey = true;
});Access tokens are for the API, ID tokens describe the user to the client; APIs must reject ID tokens by checking the audience.
Apply. In a customer portal plus API, the Blazor Server front end uses AddOpenIdConnect with authorization code plus PKCE, stores tokens in the cookie ticket, and calls the API through a typed HttpClient whose DelegatingHandler attaches the access token and refreshes it with the refresh token when exp is near. The API uses AddJwtBearer with Authority and Audience, MapInboundClaims = false so claim names stay short, and RequireHttpsMetadata = true in all non-local environments. Token lifetime is short (10 minutes) and the API never persists tokens; machine-to-machine callers use client credentials with their own audience.
Test. Unit-level tests mint tokens with a test RSA key through JsonWebTokenHandler and point the bearer options at a fake JWKS served by WebApplicationFactory, so expiry, wrong audience, wrong issuer and tampered signature are each asserted to return 401 with the correct WWW-Authenticate error. For the OIDC client I run Keycloak in a Testcontainers container in CI and drive the full redirect flow with Playwright, including the refresh path by shrinking the token lifetime to 30 seconds. Clock skew tests use a TimeProvider injected into validation.
Troubleshoot. IDX10214: Audience validation failed means the client requested the wrong scope/resource and received a token for a different API. IDX10501: Signature validation failed after a key rotation means the JWKS cache is stale; the handler refreshes automatically on unknown kid but AutomaticRefreshInterval can be shortened. Correlation failed on the OIDC callback is almost always SameSite cookie policy or a mismatch between the redirect URI and the public hostname behind a proxy, fixed with ForwardedHeaders. Unexpected 401 on a healthy token with Debug logs silent points to the Authorization header being stripped by a gateway.
An important consideration: validating only the signature is not enough; issuer, audience, lifetime and typ must all be checked, and ID tokens must never be accepted as access tokens.
Change tracking is how a DbContext knows what to write when SaveChanges is called. Every entity materialised by a tracked query is recorded in the context’s identity map with a snapshot of its original values and a state (Unchanged, Added, Modified, Deleted, Detached). On SaveChanges, EF Core compares current values to the snapshot (snapshot change detection), builds the minimal UPDATE / INSERT / DELETE statements, wraps them in a transaction and resets the states.
Consequences that matter:
AsNoTracking() (or UseQueryTrackingBehavior(NoTracking) as the context default). AsNoTrackingWithIdentityResolution() is the middle ground that deduplicates without tracking.DbContext is a unit of work and is registered scoped, one per request. Tracked entities must not outlive the request or be shared across threads.Attach (state Unchanged, then mark properties) or Update (all properties Modified), which is how disconnected web updates work.The diagram shows the entity states the change tracker moves between.

var order = await db.Orders.Include(o => o.Lines)
.SingleAsync(o => o.Id == id, ct); // tracked
order.Status = OrderStatus.Paid; // snapshot now differs
order.Lines.Add(new OrderLine { Sku = "X1", Qty = 2 }); // Added
await db.SaveChangesAsync(ct);
// UPDATE Orders SET Status=@p0 WHERE Id=@p1; INSERT INTO OrderLines ...ExecuteUpdateAsync / ExecuteDeleteAsync (EF Core 7+) bypass tracking entirely for bulk set-based changes.
Apply. In an order service the context default is QueryTrackingBehavior.NoTracking, because most endpoints are reads projected straight to DTOs with Select, and only command handlers opt in with AsTracking() when they intend to modify an aggregate. Each command loads the aggregate with the includes it needs, mutates the domain objects, and calls SaveChangesAsync once; the repository never exposes Update. For disconnected edits from a PUT, I load the current row, map the allowed fields onto it, and let snapshot detection write only the changed columns, rather than attaching the client’s object and trusting every property. Bulk status changes use ExecuteUpdateAsync so no entities are materialised.
Test. Integration tests run against the real provider (PostgreSQL in Testcontainers), not the in-memory provider, which does not enforce tracking semantics faithfully. A test modifies an aggregate and asserts db.ChangeTracker.Entries().Count(e => e.State == EntityState.Modified) == 1, then uses ToQueryString() or the logged SQL to assert only the intended column is in the UPDATE. A regression test asserts that a read endpoint’s query is not tracked by checking ChangeTracker.Entries() is empty after it runs.
Troubleshoot. “The instance of entity type cannot be tracked because another instance with the same key value is already being tracked” means two objects for one row entered the same context, typically a tracked query followed by Attach of a DTO-built entity; fix by querying first or using AsNoTracking. A slow read endpoint with high gen 0 allocation often shows tracking overhead in a profiler, resolved by AsNoTracking and projection. Memory growth in a long-lived context (a background job reusing one DbContext for a million rows) is the identity map growing without bound; batch and recreate the context, or call ChangeTracker.Clear().
An important consideration: owned types and collections that are replaced wholesale (order.Lines = newList) are seen as delete-all-then-insert, not a diff; modify the existing collection instead.
Migrations are EF Core’s versioned schema-change mechanism. Each migration is a C# class with Up and Down methods generated by diffing the current model against a model snapshot stored in the project; the database tracks which migrations have been applied in the __EFMigrationsHistory table. Developers run dotnet ef migrations add AddOrderStatus, review the generated operations, and apply them with dotnet ef database update, a generated SQL script, or a migration bundle.
Key mechanics:
ModelSnapshot.cs) is the source of truth for diffs, so it is committed and merge conflicts in it are real conflicts.migrationBuilder.Sql("UPDATE ...") is added by hand when a column split or backfill is needed.dotnet ef migrations script --idempotent produces a script safe to run on any database state, which is what most CD pipelines deploy.dotnet ef migrations bundle creates a self-contained executable that applies pending migrations without the SDK, convenient for containers.Database.MigrateAsync() at application start works for single-instance apps but is risky with multiple replicas starting simultaneously.dotnet ef migrations add AddOrderStatus --project src/Orders.Infrastructure \
--startup-project src/Orders.Api
dotnet ef migrations script --idempotent -o artifacts/migrate.sql \
--project src/Orders.Infrastructure --startup-project src/Orders.ApiThe generated Up for a new non-nullable column includes a default so existing rows are valid:
migrationBuilder.AddColumn<int>(
name: "Status", table: "Orders", nullable: false, defaultValue: 0);Migrations describe schema history; they are not a substitute for a DBA’s review on large tables, where a naive ALTER can lock a production table for minutes.
Apply. In a service deployed as several replicas I never run MigrateAsync from the web process. The CD pipeline builds a migration bundle, runs it as a Kubernetes Job (or an init step gated to one instance) before the new pods roll out, and the pipeline fails if the bundle fails. Every schema change is expand/contract: add the nullable column or new table first, deploy code that writes both, backfill with a hand-written Sql() in batches, then a later migration tightens constraints and drops the old column. Destructive operations in Up require a reviewer to see the generated SQL in the PR; dotnet ef migrations script output is attached as an artifact.
Test. A CI step runs dotnet ef migrations has-pending-model-changes (EF Core 8+) so a model edit without a migration fails the build. An integration test applies all migrations to an empty PostgreSQL container, then runs Down for the latest and Up again, proving reversibility. Another test applies the idempotent script to a database restored from a production-like snapshot in a staging pipeline and times each statement; anything over a threshold on a large table is flagged for an online-DDL approach.
Troubleshoot. “The model backing the context has changed” or a snapshot merge conflict is fixed by removing the two conflicting migrations, re-adding one on the merged model, and regenerating the snapshot. A migration that times out on ALTER TABLE is replaced with a provider-specific online operation (CREATE INDEX CONCURRENTLY, or batched updates) using Sql() and suppressTransaction: true where required. Two replicas racing to migrate show as a __EFMigrationsHistory duplicate key or a deadlock; the fix is the single Job model. A missing default on a new non-nullable column fails on existing rows and is corrected by editing the migration, not the model.
An important consideration: the migration’s Up is C# and can drift from what the model snapshot says if hand-edited carelessly; always regenerate rather than patch when in doubt.
Deferred execution means a LINQ query is a description of work, not a result. Where, Select, OrderBy, Skip, Take and most other operators build up an expression (or an iterator chain) and do nothing until something enumerates it: a foreach, ToList(), ToArray(), Count(), First(), Any(), or await ToListAsync() in EF Core. Each enumeration re-executes the whole query against the current data.
Two flavours exist:
IEnumerable<T>): operators are lazy iterators (yield return), executed in memory, streaming one element at a time.IQueryable<T> (EF Core, LINQ providers): operators build an expression tree that the provider translates to SQL when enumerated, so filters added before enumeration become part of the WHERE clause.var pending = db.Orders.Where(o => o.Status == OrderStatus.Pending); // no SQL yet
if (customerId is not null)
pending = pending.Where(o => o.CustomerId == customerId); // still none
var page = await pending.OrderBy(o => o.CreatedAt)
.Skip(40).Take(20)
.ToListAsync(ct); // one SQL queryThree pitfalls follow directly from the model. Enumerating twice runs two queries (or recomputes an expensive in-memory projection). Captured variables are read at enumeration time, so changing customerId after building but before enumerating changes the result. Switching from IQueryable<T> to IEnumerable<T> (an explicit cast, or AsEnumerable(), or calling a method EF cannot translate) silently moves every later operator into memory, and the database returns the whole table. The deliberate counterparts are ToList() to materialise exactly once and AsEnumerable() when client-side evaluation is actually intended.
Apply. In a reporting API I exploit deferral to compose queries from optional filters: a SearchOrdersQuery handler builds an IQueryable<Order> step by step (status, date range, customer, free text) and only the final Select to a DTO plus ToListAsync touches the database, producing one SQL statement with just the needed columns. Specification objects return Expression<Func<Order, bool>> so they compose server-side. Repository methods return IQueryable only within the application layer; anything that crosses a boundary (controller, response) is materialised first, because a live query escaping the DbContext scope is an accident waiting to happen.
Test. I assert translation, not just results. A test calls query.ToQueryString() and checks the SQL contains WHERE and LIMIT and does not contain a client-side projection. Integration tests against PostgreSQL in Testcontainers count executed commands via a DbCommandInterceptor and assert exactly one for the search endpoint. For in-memory pipelines a test enumerates an IEnumerable<T> returned by a method twice and asserts the underlying source was read once, which forces the implementation to materialise where it should.
Troubleshoot. InvalidOperationException: The LINQ expression could not be translated (EF Core 3+ fails fast) means a C# method or pattern has no SQL equivalent; move it after AsEnumerable() deliberately or rewrite with EF.Functions. “There is already an open DataReader associated with this Connection” arises from nested enumeration of two deferred queries on one context without MARS; materialise the outer one. ObjectDisposedException on a DbContext means an IQueryable escaped the request scope and was enumerated later. A report that is unexpectedly slow with a full-table SELECT in the EF log is the silent IEnumerable switch; find the first operator after the cast.
An important consideration: Count() followed by ToList() on the same IQueryable is two round trips; fetch the page and the total together or decide the total is not needed.
The N+1 problem is one query for a list of parents followed by one query per parent for its children, so 100 orders become 101 round trips. In EF Core it appears whenever navigation properties are accessed after the fact, most sharply with lazy loading (UseLazyLoadingProxies), where every order.Lines access silently queries. Without lazy loading, the navigation is simply null, which is a different bug but not a performance one.
The tuning toolbox, in the order I reach for it:
Select into a DTO with only the needed columns and child collections; EF Core generates a single query or a small fixed number of queries regardless of N.Include / ThenInclude when the full entity graph is needed. For multiple collection includes, AsSplitQuery() avoids the cartesian explosion of one giant join.Include(o => o.Lines.Where(l => l.Qty > 0)).EF.CompileAsyncQuery) for very hot paths.HasIndex) matched to the actual WHERE / ORDER BY columns.ExecuteUpdateAsync for set-based writes; MaxBatchSize for inserts.var summaries = await db.Orders
.Where(o => o.CustomerId == customerId)
.Select(o => new OrderSummary(
o.Id,
o.CreatedAt,
o.Lines.Sum(l => l.Qty * l.UnitPrice), // computed in SQL
o.Lines.Count))
.AsNoTracking()
.ToListAsync(ct);
// One SELECT with correlated subqueries, no matter how many ordersDiagnostics come from the provider’s own logging (LogTo, EnableSensitiveDataLogging in development), ToQueryString(), and the database’s plan (EXPLAIN ANALYZE). The warning MultipleCollectionIncludeWarning is EF Core telling you a split query is probably warranted.
Apply. In an e-commerce catalogue I ban lazy-loading proxies outright and make AsNoTracking the default; every read endpoint projects to a DTO and every command handler loads exactly the aggregate it mutates with explicit includes. Hot endpoints (GET /products/{id}) use compiled queries and a model-defined covering index. Paged lists use keyset pagination (WHERE (CreatedAt, Id) < (@lastAt, @lastId)) instead of Skip, which degrades linearly. Search uses EF.Functions.ILike or full-text functions so filtering stays in SQL. The CI build enables ConfigureWarnings(w => w.Throw(RelationalEventId.MultipleCollectionIncludeWarning)) so cartesian-explosion queries fail tests rather than slow production.
Test. A DbCommandInterceptor registered in integration tests counts commands per HTTP request; a test loads a customer with 50 orders and asserts the count is 1 (or 2 for a split query), which is the only reliable N+1 regression test. Query plans for the top ten queries are captured against a production-sized dataset in a nightly job and compared for sequential scans. BenchmarkDotNet measures the projection path’s allocations so an accidental Include of a large navigation shows up as a bytes-per-op jump.
Troubleshoot. The symptom is an endpoint whose latency grows with the size of the result set, and a database that shows hundreds of identical short queries differing only by parameter. EF Core logs at Information with Microsoft.EntityFrameworkCore.Database.Command show the repeated statements; OpenTelemetry traces show a request with 100 child spans. The fix is projection or Include. A single slow query instead gets EXPLAIN ANALYZE: a missing index, a parameter sniffing issue, or an OR across navigations that the provider turned into a LEFT JOIN chain. AsSplitQuery resolves result-set bloat when a query with three collection includes returns megabytes of duplicated parent columns.
An important consideration: Include on a collection in a paged query multiplies rows before Skip/Take unless EF Core can push the paging into a subquery; always check ToQueryString() for paged queries with includes.
Microsoft.Extensions.Logging is the logging abstraction: code depends on ILogger<T>, writes structured messages with named placeholders ("Order {OrderId} shipped in {Elapsed}ms"), and providers (console, Serilog, OpenTelemetry) decide formatting and destination. Levels (Trace to Critical) and category filters are configured per namespace in appsettings.json, and [LoggerMessage] source generators produce allocation-free log methods for hot paths. Scopes (BeginScope) attach ambient properties such as a correlation ID to every message inside them.
OpenTelemetry standardises the other two signals, traces and metrics, and .NET implements it natively: System.Diagnostics.Activity is an OpenTelemetry span, and System.Diagnostics.Metrics.Meter is an OpenTelemetry instrument. ASP.NET Core, HttpClient, EF Core, and SQL clients emit spans and metrics automatically; the OpenTelemetry.* packages collect them and export via OTLP to a collector or backend (Jaeger, Tempo, Azure Monitor, Honeycomb). Trace context propagates across services through the W3C traceparent header without application code.
builder.Logging.AddOpenTelemetry(o => o.IncludeScopes = true);
builder.Services.AddOpenTelemetry()
.ConfigureResource(r => r.AddService("orders-api"))
.WithTracing(t => t.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddEntityFrameworkCoreInstrumentation()
.AddOtlpExporter())
.WithMetrics(m => m.AddAspNetCoreInstrumentation()
.AddRuntimeInstrumentation()
.AddOtlpExporter());Logs emitted inside a request automatically carry TraceId and SpanId, which is what links a log line to a trace in the backend. .NET Aspire’s ServiceDefaults project wraps this configuration for local dashboards.
Apply. Across an order platform of six services I ship a shared ServiceDefaults library that configures OTLP export, resource attributes (service.name, service.version, deployment.environment) and W3C propagation identically. Business operations get their own spans via an ActivitySource ("orders") with tags like order.id and payment.provider, and domain counters (orders_created_total, payment_latency) via a Meter. Logging uses [LoggerMessage] definitions with event IDs so messages are queryable, request logging is one summary line per request (no body logging), and PII is redacted via Microsoft.Extensions.Compliance.Redaction. Sampling is parent-based at 10% in production with tail sampling in the collector keeping all error traces.
Test. Unit tests capture logs with a FakeLogger<T> (Microsoft.Extensions.Diagnostics.Testing) and assert the event ID and structured state, not the rendered string. Spans are tested with an ActivityListener or the in-memory exporter, asserting a POST /orders request produces a span named after the route with http.response.status_code set and a child orders.create span. In CI a smoke test runs the service with an OTLP collector container and asserts traces arrive, catching exporter misconfiguration before deployment.
Troubleshoot. Missing traces in the backend: check OTEL_EXPORTER_OTLP_ENDPOINT, enable the OpenTelemetry self-diagnostics file, and look for Export failed messages. Broken trace continuity between services is usually a custom HttpClient created without the factory (no propagation) or a message bus where traceparent is not copied into message headers. Log volume blowing up costs is addressed by category filters (Microsoft.AspNetCore: Warning) and by moving per-item logs to Debug. High CPU in logging points to string interpolation in log calls instead of placeholders; analyzers CA1848 and CA2254 flag it.
An important consideration: a span per database row or a log line per loop iteration is cardinality abuse; aggregate with metrics and keep spans at operation granularity.
The options pattern binds a section of configuration (from appsettings.json, environment variables, user secrets, Key Vault, command line, in that layered order) to a strongly typed class and exposes it through DI. Code depends on IOptions<PaymentOptions> instead of reading IConfiguration["Payment:Timeout"] strings everywhere, which gives compile-time names, defaults, and validation.
Three interfaces differ in lifetime and reload semantics:
| Interface | Lifetime | Reload behaviour | Use |
|---|---|---|---|
IOptions<T> | Singleton | Computed once; never changes | Settings fixed for process lifetime |
IOptionsSnapshot<T> | Scoped | Recomputed per request if source changed | Per-request consistent view of reloadable config |
IOptionsMonitor<T> | Singleton | CurrentValue live; OnChange callback | Singletons and background services that must see updates |
Registration, validation, and named options:
builder.Services.AddOptions<PaymentOptions>()
.BindConfiguration("Payment")
.ValidateDataAnnotations()
.Validate(o => o.Timeout > TimeSpan.Zero, "Timeout must be positive")
.ValidateOnStart(); // fail fast at host start, not first use
public sealed class PaymentOptions
{
[Required, Url] public string BaseUrl { get; init; } = "";
public TimeSpan Timeout { get; init; } = TimeSpan.FromSeconds(10);
}reloadOnChange: true on the JSON provider plus IOptionsMonitor<T> lets feature toggles change without a restart. Named options (Configure<T>("Stripe", ...), IOptionsMonitor<T>.Get("Stripe")) support several instances of one shape. IConfigureOptions<T> / IPostConfigureOptions<T> let libraries adjust values after binding. The .NET 8 IOptions source generator ([OptionsValidator]) produces the validation code without reflection, which Native AOT requires.
Apply. In a payments service every external integration (StripeOptions, FraudCheckOptions, OutboxOptions) is a sealed options class bound with BindConfiguration, annotated with data annotations plus a [OptionsValidator] partial, and registered with ValidateOnStart(), so a missing API URL kills the pod at startup where the orchestrator reports it instead of returning 500s at 2 a.m. Feature flags use IOptionsMonitor<FeatureOptions> in singletons and IOptionsSnapshot<T> in request-scoped handlers, so one request sees one consistent set. Secrets never live in options classes bound from JSON files; they come from the Key Vault provider in the same section so the shape is unchanged across environments.
Test. Options classes are tested by building a ConfigurationBuilder from an in-memory dictionary, binding, and asserting values and validation failures (OptionsValidationException with the expected message). A host startup test loads each environment’s appsettings.{env}.json through the real Program with ValidateOnStart and asserts the host builds, catching a typo in a production-only file. Reload behaviour is tested by writing a temp JSON file, building configuration with reloadOnChange, modifying the file, and awaiting the OnChange callback with a timeout.
Troubleshoot. All-default values after deployment mean the section name or casing differs between the class and the file, or an environment variable uses : instead of __ as the separator in Linux containers. OptionsValidationException at startup names the property and rule. Changes not picked up in a singleton mean it injected IOptions<T> instead of IOptionsMonitor<T>; changes not picked up at all in Kubernetes mean the ConfigMap is mounted via subPath, which breaks file-watch notifications. IConfigurationRoot.GetDebugView() prints every key and which provider supplied it, which resolves most precedence arguments in a minute.
An important consideration: IOptionsSnapshot<T> cannot be injected into singletons (it is scoped), and the container will throw at startup with scope validation on; the fix is IOptionsMonitor<T>.
ASP.NET Core offers caching at several layers, each trading freshness for cost.
IMemoryCache): per-process dictionary with absolute/sliding expiration, priorities and size limits. Fast, but not shared between replicas and lost on restart.IDistributedCache): byte-array API backed by Redis, SQL Server or NCache; shared across instances, survives restarts, costs a network hop and serialisation.Microsoft.Extensions.Caching.Hybrid): combines an in-process L1 with a distributed L2, adds stampede protection (concurrent requests for one key share a single factory call), tag-based invalidation, and a GetOrCreateAsync API that replaces the hand-written get-check-set pattern.AddOutputCache, .NET 7+): caches whole HTTP responses on the server by policy, varies by query/header/route, supports eviction by tag, and works regardless of client cache headers.Cache-Control headers; it is largely superseded by output caching.ETag, Cache-Control) let browsers and CDNs cache without touching the server.The diagram shows the lookup order of HybridCache and where stampede protection applies.

builder.Services.AddHybridCache();
public sealed class ProductReader(HybridCache cache, CatalogDb db)
{
public ValueTask<ProductDto?> GetAsync(string sku, CancellationToken ct) =>
cache.GetOrCreateAsync($"product:{sku}",
async token => await db.Products.Where(p => p.Sku == sku)
.Select(p => ProductDto.From(p))
.FirstOrDefaultAsync(token),
new HybridCacheEntryOptions { Expiration = TimeSpan.FromMinutes(5) },
tags: ["products"], cancellationToken: ct);
}Invalidation is the hard part: cache.RemoveByTagAsync("products") after a catalogue import is far more maintainable than tracking individual keys. Caching per-user data must include the user in the key, and sensitive data should never go in output cache without a vary-by-user policy.
Apply. In a product catalogue API serving a storefront, HybridCache fronts product lookups with Redis as L2 so twelve replicas share a warm cache and a restart does not stampede the database. Entries carry tags per category, and the admin import endpoint calls RemoveByTagAsync for the affected categories rather than flushing everything. Public list endpoints use output caching with a 30-second policy varied by query string, which absorbs bot traffic; authenticated endpoints are excluded from output caching entirely. Responses carry ETags derived from a row version so the storefront’s CDN serves 304s. Cache keys include a schema version prefix so a DTO shape change does not deserialise stale payloads after a deploy.
Test. Unit tests inject a real HybridCache with an in-memory L2 and assert that two concurrent GetOrCreateAsync calls invoke the factory once, proving stampede protection. Invalidation tests write, tag-evict, and assert a miss. Integration tests with Redis in Testcontainers verify the serialised shape round-trips and that a key written by one WebApplicationFactory instance is visible to a second one. Output caching is tested by asserting the second response carries the Age header and the backing service’s call count did not increase.
Troubleshoot. Stale data after an update is nearly always a missing invalidation path (a bulk update via ExecuteUpdateAsync that bypassed the service that evicts). Redis RedisTimeoutException under load points to oversized values or a synchronous GetString on the hot path; check redis-cli --bigkeys and payload sizes. Memory growth from IMemoryCache means no SizeLimit and no Size on entries. Hit ratio is tracked via a Meter counter for hits and misses; a ratio collapse after a deploy usually means the key format changed. Output cache serving one user’s data to another is a missing VaryByValue on the user claim.
An important consideration: caching hides database problems until the cache is cold; load tests must include a cold-cache phase.
IHttpClientFactory solves two problems with HttpClient. Creating a new client per request exhausts sockets, because disposing a client does not immediately release the connection (TIME_WAIT). Keeping one static client forever means DNS changes are never observed, because the underlying handler caches connections to the old IP. The factory pools HttpMessageHandler instances, rotates them every two minutes by default (SetHandlerLifetime), and hands out lightweight HttpClient wrappers that are safe to create and drop freely.
It also provides a configuration surface:
AddHttpClient("github", c => c.BaseAddress = ...) then factory.CreateClient("github").AddHttpClient<IPaymentsClient, PaymentsClient>() injects a class whose constructor takes the configured HttpClient; this is the usual choice in application code.Microsoft.Extensions.Http.Resilience adds AddStandardResilienceHandler() (retry, circuit breaker, timeout, rate limiter via Polly v8).builder.Services.AddTransient<BearerTokenHandler>();
builder.Services.AddHttpClient<IPaymentsClient, PaymentsClient>(c =>
{
c.BaseAddress = new Uri(builder.Configuration["Payments:BaseUrl"]!);
c.Timeout = TimeSpan.FromSeconds(10);
})
.AddHttpMessageHandler<BearerTokenHandler>()
.AddStandardResilienceHandler();Typed clients are registered transient, so they must not be captured by singletons; inject IHttpClientFactory there instead. Since .NET 6 the alternative for simple cases is one long-lived HttpClient over a SocketsHttpHandler with PooledConnectionLifetime set, which also solves the DNS issue, but the factory remains the standard in ASP.NET Core for its DI integration and handler pipeline.
Apply. In an order service that calls inventory, pricing and a payment provider, each dependency is a typed client with its own BaseAddress, timeout and resilience policy tuned to that dependency (payments gets no retries on POST, inventory gets three). Shared DelegatingHandlers add the bearer token from a token cache, propagate traceparent (automatic) and the tenant header, and apply ProblemDetails parsing on non-success so callers get typed exceptions. Clients are injected into scoped handlers, never singletons; the one singleton that needs HTTP (a background price refresher) uses IHttpClientFactory.CreateClient("pricing") per iteration. UseSocketsHttpHandler is configured with EnableMultipleHttp2Connections for the gRPC-over-HTTP/2 dependency.
Test. Typed clients are unit-tested by constructing them with an HttpClient over a fake HttpMessageHandler that returns canned responses and records requests, asserting URLs, headers and deserialisation. Resilience behaviour is tested the same way: a handler that fails twice then succeeds proves the retry count; a handler that delays proves the timeout. Contract tests use WireMock.Net in CI to replay recorded provider responses, including 429 with Retry-After. A startup test resolves every typed client to catch a missing registration.
Troubleshoot. SocketException: Address already in use or thousands of TIME_WAIT connections (ss -tan | grep TIME-WAIT | wc -l) means someone is new HttpClient() -ing per call; the analyzer CA2000 plus a code search finds it. Calls to a dependency that moved behind a new IP still failing after the DNS change means a static client without PooledConnectionLifetime. TaskCanceledException with no explicit cancellation is the HttpClient.Timeout firing; the OpenTelemetry span shows the duration. Resilience retries amplifying an outage show as 3x request volume on the dependency’s dashboard; the circuit breaker thresholds are then tightened.
An important consideration: the HttpClient.Timeout applies to the whole call including retries unless the resilience pipeline’s per-attempt timeout is lower, and the two must be set consistently.
A hosted service is a singleton that the generic host starts and stops with the application. IHostedService defines StartAsync and StopAsync; BackgroundService is the common base class that runs a long-lived ExecuteAsync(CancellationToken stoppingToken) loop and handles start/stop plumbing. Hosted services run in the same process as Kestrel, share its DI container and configuration, and are started in registration order before the server begins accepting requests (unless BackgroundServiceExceptionBehavior and startup ordering are changed).
Typical uses: draining a queue or Channel<T>, running scheduled jobs with PeriodicTimer, consuming Kafka/Service Bus messages, warming caches, and processing an outbox table.
public sealed class OutboxDispatcher(IServiceScopeFactory scopes,
ILogger<OutboxDispatcher> log) : BackgroundService
{
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
using var timer = new PeriodicTimer(TimeSpan.FromSeconds(2));
while (await timer.WaitForNextTickAsync(stoppingToken))
{
await using var scope = scopes.CreateAsyncScope();
var outbox = scope.ServiceProvider.GetRequiredService<IOutbox>();
try { await outbox.DispatchPendingAsync(stoppingToken); }
catch (Exception ex) when (ex is not OperationCanceledException)
{ log.LogError(ex, "Outbox dispatch failed"); }
}
}
}Rules that make them production-safe: create a DI scope per unit of work because the service itself is a singleton; honour stoppingToken so SIGTERM leads to a clean exit within HostOptions.ShutdownTimeout (default 30 seconds since .NET 8); and catch exceptions inside the loop, because since .NET 6 an unhandled exception in ExecuteAsync stops the whole host by default. For heavy or multi-node scheduling, Quartz.NET or Hangfire add persistence and distributed locking on top of this primitive.
Apply. In an order service I run three hosted services: an outbox dispatcher (above), a Service Bus consumer that writes to a bounded Channel<T> consumed by N worker loops started in ExecuteAsync with Task.WhenAll, and a nightly reconciliation job using PeriodicTimer plus a database advisory lock so only one of the replicas runs it. Each unit of work gets its own DI scope and its own Activity so traces are per message, not per process lifetime. Work is idempotent and checkpointed (message lock renewal, outbox row status) because a pod can be killed between steps. HostOptions.ShutdownTimeout is raised to match the longest unit of work, and the Kubernetes terminationGracePeriodSeconds is set above it.
Test. BackgroundService logic is pulled into a testable ProcessOnceAsync method so unit tests call one iteration with fakes and a pre-cancelled token. Host-level tests use WebApplicationFactory with the real service registered and a Testcontainers broker, publish a message, and poll the database with a timeout for the side effect. Graceful shutdown is tested by starting the host, enqueuing slow work, calling StopAsync, and asserting the work completed and no TaskCanceledException leaked. A test also asserts the service survives a thrown exception in one iteration and processes the next.
Troubleshoot. A service that silently stopped is the .NET 6+ default: an unhandled exception terminated the host, or BackgroundServiceExceptionBehavior.Ignore was set and it died quietly; logs at Critical from Microsoft.Extensions.Hosting show which. Work “stuck” with growing queue depth and no errors usually means a synchronous blocking call starving the pool, found with dotnet-stack. “Cannot access a disposed object” means a scoped DbContext was injected into the singleton service. Duplicate processing across replicas is a missing distributed lock. Slow deployments with SIGKILL after 30 seconds mean stoppingToken is not being checked inside the loop.
An important consideration: hosted services start before Kestrel is listening, so a slow StartAsync (not ExecuteAsync) delays readiness; do heavy initialisation inside ExecuteAsync.
Exception handling in ASP.NET Core happens at the outermost middleware so every downstream failure becomes a well-formed HTTP response. UseExceptionHandler() catches unhandled exceptions, and since .NET 8 it delegates to registered IExceptionHandler implementations, which run in order until one returns true. UseDeveloperExceptionPage (automatic in Development) shows stack traces and must never reach production. UseStatusCodePages fills in bodies for bare 4xx/5xx responses.
ProblemDetails is the RFC 9457 (formerly 7807) JSON shape for errors: type, title, status, detail, instance, plus extensions. AddProblemDetails() registers IProblemDetailsService, which the exception handler, status code pages and [ApiController] validation all use, so clients see one format for every failure. ValidationProblemDetails adds the errors dictionary.
builder.Services.AddProblemDetails(o => o.CustomizeProblemDetails = ctx =>
ctx.ProblemDetails.Extensions["traceId"] = ctx.HttpContext.TraceIdentifier);
builder.Services.AddExceptionHandler<DomainExceptionHandler>();
app.UseExceptionHandler();
app.UseStatusCodePages();
public sealed class DomainExceptionHandler : IExceptionHandler
{
public async ValueTask<bool> TryHandleAsync(HttpContext ctx, Exception ex,
CancellationToken ct)
{
if (ex is not DomainException d) return false; // let others handle
ctx.Response.StatusCode = StatusCodes.Status409Conflict;
await ctx.Response.WriteAsJsonAsync(new ProblemDetails
{ Title = d.Code, Detail = d.Message, Status = 409 }, ct);
return true;
}
}Exceptions are for exceptional paths; expected outcomes (not found, validation, conflict) are better returned as typed results (Results.Problem, TypedResults.ValidationProblem) without throwing, which avoids the cost of unwinding and keeps handlers explicit. The handler is also where OperationCanceledException from a disconnected client is turned into a 499-style no-op rather than a 500.
Apply. In a public orders API every error response is ProblemDetails with a stable type URI (https://api.example.com/problems/insufficient-stock) documented in the OpenAPI spec, so clients branch on type, not on text. An IExceptionHandler chain maps domain exceptions to 409/422, DbUpdateConcurrencyException to 409 with the current ETag, and OperationCanceledException when RequestAborted is set to a silent abort; anything else falls to the default 500 with traceId and no exception message. Handlers return TypedResults.Problem for expected failures instead of throwing. The CustomizeProblemDetails hook stamps traceId and instance on every response, and the Development environment is the only one with detail containing the exception message.
Test. Each mapping is covered by an integration test that triggers the exception through a real endpoint (a stub service throwing InsufficientStockException) and asserts status, Content-Type: application/problem+json, type, and that detail does not contain a stack trace in Production configuration. A test for the 500 path asserts the response contains traceId and that the FakeLogger recorded one Error with the exception attached, which is the only place the real message should appear. The IExceptionHandler classes are also unit-tested directly with a DefaultHttpContext.
Troubleshoot. A plain-text 500 with no body means UseExceptionHandler is after the failing middleware or AddProblemDetails is missing. “Response has already started” from the handler means the endpoint wrote headers before throwing (streaming), so the handler cannot rewrite; the fix is to validate before streaming. Error rates spiking without logs point to exceptions swallowed in a catch that returns null; a search for empty catch blocks and the CA1031 analyzer find them. Client-reported traceId values are looked up directly in the tracing backend, which is the whole point of stamping them.
An important consideration: the exception handler runs once per request and cannot fix a response whose body has been partially flushed, so streaming endpoints need their own error protocol.
In .NET, performance work is mostly allocation work. Every heap allocation costs a little CPU, but the real price is paid later in garbage collection: gen 0 collections pause briefly, promotions copy memory, and LOH allocations trigger gen 2. A web service that allocates 10 KB per request at 5,000 requests per second churns 50 MB per second through gen 0, and that shows as time-in-gc and p99 latency jitter. Reducing allocation on hot paths is therefore the highest-yield optimisation after fixing the obvious I/O problems.
Techniques the runtime and BCL make available:
Span<T> / ReadOnlySpan<T> and Memory<T> for slicing without copying; stackalloc for small scratch buffers.ArrayPool<T>.Shared and MemoryPool<T> to rent large buffers instead of allocating.System.Text.Json source generators and Utf8JsonReader for serialisation without reflection or intermediate strings.string.Create, ValueStringBuilder patterns, and interpolated string handlers to avoid temporary strings.readonly struct / record struct for small values; in parameters to pass them without copying.ValueTask on hot async paths that usually complete synchronously; IAsyncEnumerable<T> for streaming.[LoggerMessage] and FrozenDictionary / FrozenSet (.NET 8) for read-heavy lookups.System.IO.Pipelines for high-volume network parsing.// Parse "orderId:lineNo" without allocating substrings
static (int OrderId, int LineNo) ParseKey(ReadOnlySpan<char> key)
{
int sep = key.IndexOf(':');
return (int.Parse(key[..sep]), int.Parse(key[(sep + 1)..]));
}Measurement comes first: BenchmarkDotNet with [MemoryDiagnoser] for micro-level numbers, dotnet-counters for GC and thread-pool health, and dotnet-trace /PerfView for CPU and allocation stacks. Framework-level gains arrive with each runtime version (dynamic PGO, improved JIT loop optimisations, faster Dictionary and LINQ), so staying on the current LTS is itself a performance task.
Apply. In a market-data gateway handling 20,000 messages per second, the hot path is designed around zero allocation per message: frames are read with PipeReader, parsed with Utf8JsonReader into readonly record struct values, routed through a Channel<T> with SingleReader, and serialised to clients with a System.Text.Json source-generated context into a rented IBufferWriter<byte>. Lookups use FrozenDictionary<string, Symbol> built once at startup. Logging on that path is [LoggerMessage] at Debug with IsEnabled guards. Everything outside the hot path (admin endpoints, config reload) is written plainly, because readability wins where allocation does not matter.
Test. A BenchmarkDotNet project runs in CI nightly with [MemoryDiagnoser] and a BenchmarkDotNet.Analyzers threshold; a change that moves ParseFrame from 0 B to 48 B per op fails the pipeline. Load tests with k6 or NBomber run at target throughput while dotnet-counters records gen-0-gc-count, gen-2-gc-count, time-in-gc, and alloc-rate; the acceptance criterion is gen 2 count near zero and time-in-gc under 5%. p99 latency is compared release-over-release in the same environment.
Troubleshoot. A latency spike correlating with gen-2-gc-count increments is LOH churn or mid-life objects; dotnet-gcdump plus dumpheap -stat shows which types. High alloc-rate with low gen 2 is pure gen 0 churn: dotnet-trace collect --providers Microsoft-DotNETCore-SampleProfiler plus the GCAllocationTick event in PerfView shows the allocating call stacks, typically LINQ in a loop, string.Split, closures capturing per-request state, or boxing through an object parameter. CPU hot spots without allocation get a sampling profile and often turn out to be regex without RegexOptions.Compiled or the source generator, or DateTime.Now instead of UtcNow.
An important consideration: optimising without a benchmark that reproduces the production profile frequently makes code harder to read for no measurable benefit; measure, change one thing, measure again.
The question conflates two levels; both are standard in ASP.NET Core projects and the distinction is what is in-process and what is real.
Unit tests exercise one class in isolation with dependencies replaced by fakes or mocks. They use xUnit (default in templates), NUnit or MSTest, with NSubstitute or Moq for mocks and FluentAssertions (or xUnit’s built-in asserts) for readability. Targets are domain logic, validators, mapping, policy handlers, and minimal API static handlers invoked directly. They run in milliseconds and need no network.
Integration tests boot the real application with WebApplicationFactory<Program> from Microsoft.AspNetCore.Mvc.Testing. The app runs in-memory with the full DI container, middleware, routing, model binding and authorization; HttpClient calls go through the pipeline without a socket. ConfigureTestServices swaps external dependencies (a payment gateway) for fakes while keeping the database real via Testcontainers (PostgreSQL, SQL Server, Redis in Docker). The in-memory EF provider is not a substitute, because it ignores constraints, transactions and SQL translation.
public sealed class OrdersApiTests(ApiFactory factory) : IClassFixture<ApiFactory>
{
[Fact]
public async Task Create_returns_201_with_location()
{
var client = factory.CreateClient();
var response = await client.PostAsJsonAsync("/api/orders",
new { customerRef = "C-1", quantity = 2 });
Assert.Equal(HttpStatusCode.Created, response.StatusCode);
Assert.StartsWith("/api/orders/", response.Headers.Location!.PathAndQuery);
}
}ApiFactory derives from WebApplicationFactory<Program>, implements IAsyncLifetime to start the container, and overrides ConfigureWebHost to point the connection string at it. The pyramid stays honest: many unit tests, a focused set of integration tests per endpoint, and a handful of end-to-end tests against a deployed environment.
Apply. In an order service the test solution has three projects: Orders.UnitTests for domain and handlers with no I/O, Orders.IntegrationTests with a shared ApiFactory fixture that starts PostgreSQL and Redis once per test class collection via Testcontainers and applies migrations, and Orders.ContractTests that verify the OpenAPI document and consumer-driven contracts. Authentication is replaced by a test scheme that reads claims from headers. Each integration test runs inside a Respawn database reset so tests are independent and can run in parallel across collections. External HTTP dependencies use WireMock.Net, not mocks of HttpClient, so serialisation is exercised too.
Test. Test quality is itself tested: Stryker.NET mutation testing runs weekly on the domain project and a mutation score drop blocks the merge, which catches assertions that do not actually assert. Coverage via coverlet is reported but not gated; a gate on the critical pricing module is at the line level. Flaky tests are quarantined with a [Trait("Flaky")] and tracked, never retried silently. dotnet test --blame-hang-timeout 5m is on in CI so a deadlocked test produces a dump instead of a cancelled job.
Troubleshoot. A WebApplicationFactory that fails with “The entry point exited without ever building an IHost” means Program is top-level statements without the public partial class Program { } marker, or the host is built differently in tests. Tests passing locally and failing in CI with SqlException: timeout are usually Testcontainers waiting strategy problems; use WithWaitStrategy on the port and a health command. Cross-test contamination appears as order-dependent failures and is fixed by Respawn or a per-test schema. Slow suites are profiled with dotnet test --logger trx timings; the usual cost is a container per test class instead of per collection.
An important consideration: integration tests that mock the repository test nothing of value; the database and the serialiser are the parts that break, so they must be real.
API versioning lets a service evolve its contract while existing clients keep working. The Asp.Versioning.Http and Asp.Versioning.Mvc packages (the successor to Microsoft.AspNetCore.Mvc.Versioning) add version awareness to routing, controllers, minimal APIs and OpenAPI generation.
Versions can be read from several places; the package supports them individually or combined:
| Strategy | Example | Trade-off |
|---|---|---|
| URL segment | /api/v2/orders | Explicit, cache-friendly; routes duplicate |
| Query string | /api/orders?api-version=2.0 | Default of the package; easy for browsers |
| Header | X-Api-Version: 2.0 | Clean URLs; invisible in logs and caches |
| Media type | Accept: application/json;v=2 | Purest REST; hardest for clients |
builder.Services.AddApiVersioning(o =>
{
o.DefaultApiVersion = new ApiVersion(1, 0);
o.AssumeDefaultVersionWhenUnspecified = true;
o.ReportApiVersions = true; // api-supported-versions header
o.ApiVersionReader = new UrlSegmentApiVersionReader();
}).AddApiExplorer(o => o.GroupNameFormat = "'v'VVV");
var set = app.NewApiVersionSet().HasApiVersion(1.0).HasApiVersion(2.0)
.HasDeprecatedApiVersion(0.9).Build();
app.MapGet("/api/v{version:apiVersion}/orders", ListOrdersV1)
.WithApiVersionSet(set).MapToApiVersion(1.0);
app.MapGet("/api/v{version:apiVersion}/orders", ListOrdersV2)
.WithApiVersionSet(set).MapToApiVersion(2.0);Versioning policy matters more than mechanism: additive changes (new optional fields, new endpoints) do not need a new version; a new major version is reserved for breaking changes such as removing a field or changing semantics. Deprecated versions are announced via the api-deprecated-versions header and a Sunset date, then removed on schedule.
Apply. For a partner-facing orders API I use URL-segment versioning because partners log and cache by URL, with ReportApiVersions on so every response advertises supported and deprecated versions. Internally there is one set of domain handlers; versions differ only in the request/response contracts, so v1 and v2 endpoints map DTOs to the same command. A version is created only for breaking changes; adding fields ships in-place. Each deprecated version gets a Sunset header with a date six months out, an entry in the changelog, and a dashboard panel of calls by version so the team knows when it is safe to delete. OpenAPI documents are generated per version group so partners see exactly their contract.
Test. Contract tests pin each version’s response schema: a snapshot of the v1 JSON for a known order is committed, and any diff fails the build, which prevents accidental breaking changes to the old version while developing v2. Routing tests assert /api/v3/orders returns 400 with UnsupportedApiVersion ProblemDetails rather than 404, that an unversioned call hits the default, and that the api-supported-versions header lists exactly the configured versions. The generated OpenAPI documents per version are diffed against the previous release with openapi-diff and classified as breaking or not.
Troubleshoot. Every request returning UnsupportedApiVersion after a refactor usually means an endpoint was moved out of its ApiVersionSet or the route constraint name changed from apiVersion.
Clients on a version that “disappeared” typically hit a reverse proxy rewrite that stripped the segment. Duplicate OpenAPI operation IDs across versions come from GroupNameFormat not being set. Partners still calling a sunset version are identified from the version-by-caller metric, and the sunset date is enforced only after direct outreach.
An important consideration: versioning the URL does not version the database or the domain; if v2 needs a new persistence model, the migration strategy must be designed independently of the HTTP contract.
REST is the architectural style of resource-oriented HTTP APIs: nouns in URLs, verbs via HTTP methods, status codes for outcomes, and stateless requests. OpenAPI is the machine-readable specification (currently 3.0 and 3.1, JSON or YAML) that describes such an API: paths, operations, parameters, schemas, security schemes, and response codes. Swagger is the historical name of the spec (OpenAPI 2.0 was “Swagger 2.0”) and today refers to the tooling around it, chiefly Swagger UI for interactive docs and Swashbuckle, the long-standing ASP.NET Core generator.
In .NET 9 and later the template uses the built-in Microsoft.AspNetCore.OpenApi package, which generates the document from endpoint metadata at run time (or at build time for Native AOT) without Swashbuckle; Swashbuckle remains supported and widely used, and NSwag is the other mature option. A UI is added separately: Swagger UI, Scalar, or Redoc. .NET 10 adds OpenAPI 3.1 as the default version and can generate the document at build time for use in CI.
builder.Services.AddOpenApi(); // built-in generator
app.MapOpenApi(); // serves /openapi/v1.json
if (app.Environment.IsDevelopment())
app.MapScalarApiReference(); // or UseSwaggerUI pointing at it
app.MapPost("/orders", CreateOrder)
.WithName("CreateOrder")
.WithSummary("Create an order")
.Produces<OrderDto>(StatusCodes.Status201Created)
.ProducesValidationProblem()
.WithTags("Orders");The document is only as good as the metadata: Produces, ProducesProblem, typed results (Results<Created<OrderDto>, ValidationProblem>), XML doc comments, and [Description] attributes feed it. Document transformers (AddDocumentTransformer) inject security schemes and servers. Generated clients (NSwag, Kiota, openapi-generator) consume the document so the spec is the contract, not an afterthought.
Apply. For a partner API the OpenAPI document is a build artifact, not a Development-only page. The pipeline runs the build-time generator (Microsoft.Extensions.ApiDescription.Server) to emit openapi/v1.json, commits a copy to the repo, publishes it to the developer portal, and uses it to generate the TypeScript and C# SDKs with Kiota so partners never hand-write models. Endpoints use typed results so response codes are inferred, and a document transformer adds the OAuth2 security scheme, server URLs per environment, and the ProblemDetails type catalogue. Internal admin endpoints are excluded with ExcludeFromDescription(). Swagger UI/Scalar is served only in non-production or behind authentication.
Test. A test loads /openapi/v1.json through WebApplicationFactory and validates it with Microsoft.OpenApi.Readers, asserting zero diagnostics. The committed spec is diffed against the generated one in CI (openapi-diff or oasdiff) and a breaking change on an existing version fails the build with the exact path listed. Example requests in the document are executed by a Schemathesis or Dredd run in the nightly pipeline, which fuzzes each operation against the real service and catches undocumented 500s. The generated client is compiled in CI so a schema change that breaks client generation is visible immediately.
Troubleshoot. A schema showing object with no properties for a DTO means the type is not reachable by the serializer metadata (an object return type, or IResult without Produces). Duplicate schema names from two OrderDto classes in different namespaces are fixed with a schema-ID transformer. Polymorphic responses missing variants need [JsonDerivedType]. A document that differs between local and CI usually comes from environment-specific transformers reading configuration. Partners complaining the UI “does not send the token” is a missing AddSecurityRequirement transformer.
An important consideration: the document describes what the code exposes, so writing the spec first and hand-maintaining it separately guarantees drift; generate it and test it.
CORS (Cross-Origin Resource Sharing) is a browser mechanism, not a server security control. A page on https://shop.example.com calling https://api.example.com is cross-origin, and the browser blocks the response unless the API returns Access-Control-Allow-Origin matching the page’s origin. Non-simple requests (JSON bodies, custom headers, credentials) first trigger a preflight OPTIONS request that must be answered with the allowed methods and headers. ASP.NET Core implements this with AddCors policies and UseCors, which must run after routing and before authorization so preflights (which carry no token) are answered.
builder.Services.AddCors(o => o.AddPolicy("Storefront", p => p
.WithOrigins("https://shop.example.com", "https://admin.example.com")
.WithHeaders("Authorization", "Content-Type")
.WithMethods("GET", "POST", "PUT", "DELETE")
.AllowCredentials()
.SetPreflightMaxAge(TimeSpan.FromHours(1))));
app.UseCors("Storefront");AllowAnyOrigin() with AllowCredentials() is rejected by the framework because it would let any site act as the user. CORS protects users of browsers; it does nothing against curl, so authorization is still required on every endpoint.
Security headers harden what the browser does with responses. ASP.NET Core sets none of these by default except HSTS via UseHsts():
Strict-Transport-Security (HSTS): force HTTPS for a year; UseHsts() plus UseHttpsRedirection().Content-Security-Policy: whitelist script/style sources; the primary XSS mitigation for Razor and Blazor apps.X-Content-Type-Options: nosniff, X-Frame-Options: DENY or CSP frame-ancestors, Referrer-Policy, Permissions-Policy.Secure, HttpOnly, SameSite=Lax or Strict via CookiePolicyOptions.These are added with a small middleware or the NetEscapades.AspNetCore.SecurityHeaders package; for pure JSON APIs CSP matters less, but nosniff and HSTS still apply.
Apply. For a storefront SPA plus API, the API’s CORS policy lists the exact production and staging origins from configuration (never wildcards), allows credentials because the SPA uses a cookie-backed BFF, and sets SetPreflightMaxAge to cut OPTIONS traffic. UseCors sits between UseRouting and UseAuthentication. A security-headers middleware adds HSTS with includeSubDomains and preload, nosniff, Referrer-Policy: strict-origin-when-cross-origin, and a nonce-based CSP for the Razor-rendered pages; the pure JSON endpoints get a restrictive Content-Security-Policy: default-src 'none' and X-Frame-Options: DENY. Cookies are Secure, HttpOnly, SameSite=Lax. The reverse proxy is configured not to duplicate these headers.
Test. Integration tests send an OPTIONS preflight with Origin and Access-Control-Request-Method and assert the exact Access-Control-Allow-* headers for an allowed origin and their absence for https://evil.example. A header test asserts every response, including 404s and ProblemDetails, carries the security header set, which catches a middleware placed after the exception handler. Playwright end-to-end tests run the SPA against the API with CSP in report-only mode first; violations are collected from the report-to endpoint and fixed before enforcing. Mozilla Observatory or securityheaders.com scans run against staging in the pipeline.
Troubleshoot. “No ‘Access-Control-Allow-Origin’ header is present” in the browser but the policy looks right: the request failed before CORS ran (a 500 or a 401 on the preflight because UseCors is after authorization), or the origin has a trailing slash or different port. Preflights failing with 405 mean a proxy or a [HttpPost] -only route handled OPTIONS. CSP breaking the app after a deploy shows in the browser console with the blocked URI; inline scripts need the nonce. Duplicate Access-Control-Allow-Origin values occur when both the proxy and the app add them.
An important consideration: CORS misconfiguration fails only in browsers, so API tests with HttpClient alone will never catch it; preflight tests must send the real headers.
Health checks are HTTP endpoints an orchestrator or load balancer polls to decide whether an instance should receive traffic or be restarted. Microsoft.Extensions.Diagnostics.HealthChecks provides AddHealthChecks(), an IHealthCheck interface returning Healthy, Degraded or Unhealthy, and MapHealthChecks to expose results. The AspNetCore.HealthChecks.* community packages add ready-made checks for PostgreSQL, SQL Server, Redis, RabbitMQ, Azure Service Bus and URLs.
Kubernetes distinguishes three probes, and the checks should be tagged accordingly:
| Probe | Question | What to check |
|---|---|---|
| Liveness | Is the process stuck? | Nothing external; just that the host responds |
| Readiness | Can it serve traffic now? | Database, cache, message broker, warm-up complete |
| Startup | Has it finished booting? | Migrations applied, caches primed; disables liveness until true |
builder.Services.AddHealthChecks()
.AddNpgSql(conn, tags: ["ready"])
.AddRedis(redisConn, tags: ["ready"])
.AddCheck<OutboxLagCheck>("outbox", tags: ["ready"]);
app.MapHealthChecks("/healthz/live", new HealthCheckOptions
{ Predicate = _ => false }); // process alive only
app.MapHealthChecks("/healthz/ready", new HealthCheckOptions
{ Predicate = c => c.Tags.Contains("ready") });The flow from probe to pod state looks like this.

Putting a database check on liveness is the classic mistake: a database outage would restart every pod in a loop instead of simply taking them out of rotation.
Apply. In an order service on Kubernetes, liveness is a bare endpoint with Predicate = _ => false, readiness checks PostgreSQL, Redis and a custom OutboxLagCheck that reports Degraded when the outbox backlog exceeds a threshold, and a startup probe waits for the warm-up hosted service to set a flag. Readiness results are written as JSON via UIResponseWriter for humans, but the probe only reads the status code. On SIGTERM an IHostApplicationLifetime.ApplicationStopping hook flips readiness to Unhealthy and the host waits a few seconds before shutting down Kestrel, so the load balancer stops routing before connections are refused. Health endpoints are excluded from authentication, rate limiting, request logging and output caching, and are not exposed through the public ingress.
Test. Integration tests assert /healthz/live returns 200 even when the PostgreSQL container is stopped, and /healthz/ready returns 503 in that case and 200 after restart, which proves the tags are right. The OutboxLagCheck is unit-tested with a fake repository returning various backlogs. A chaos test in staging kills Redis and asserts pods leave the Service endpoints but are not restarted. Probe timing (initialDelaySeconds, periodSeconds, failureThreshold) is reviewed in the Helm chart test against measured startup time.
Troubleshoot. CrashLoopBackOff after an infrastructure incident means an external dependency is on liveness; move it to readiness. A pod that is Running but receives no traffic shows readiness failing in kubectl describe pod; the JSON body names the failing check. Readiness timeouts under load mean the check itself is heavy (a SELECT COUNT(*) on a big table); checks must be cheap and have their own short timeout via HealthCheckRegistration.Timeout. Health endpoints flooding logs are filtered with UseWhen or a log filter on the path.
An important consideration: a readiness check that fails when a non-critical dependency is down turns a partial degradation into a full outage; report Degraded for optional dependencies and reserve Unhealthy for those the service cannot function without.
Kestrel is the cross-platform web server built into ASP.NET Core. It is in-process, written in managed code on top of System.IO.Pipelines and SocketsHttpHandler -style networking, and supports HTTP/1.1, HTTP/2 and HTTP/3 (QUIC), TLS, Unix domain sockets and named pipes. It is production-grade and is what runs in every container; IIS (Windows) hosts it in-process via the ASP.NET Core Module, and HTTP.sys is the alternative Windows-only server for kernel-mode features like port sharing and Windows authentication.
A reverse proxy (NGINX, Envoy, YARP, Azure Application Gateway, an ingress controller) sits in front of Kestrel to terminate TLS, load-balance across replicas, serve static content, apply WAF rules and provide a single public hostname. Kestrel does not need one for security, but almost every deployment has one for operational reasons. YARP (Yarp.ReverseProxy) is Microsoft’s proxy library that runs on Kestrel itself, used for API gateways and BFFs.
The deployment topology typically looks like this.

Because the proxy terminates TLS, Kestrel sees plain HTTP from the proxy’s IP, so the app must trust X-Forwarded-For, X-Forwarded-Proto and X-Forwarded-Host via UseForwardedHeaders (with KnownProxies / KnownNetworks) so Request.Scheme, redirect URLs, and OIDC callbacks are correct. Kestrel is configured in appsettings.json under Kestrel:Endpoints or in code (ConfigureKestrel): limits such as MaxRequestBodySize (30 MB default), MaxConcurrentConnections, KeepAliveTimeout and RequestHeadersTimeout protect against slow-client attacks. Since .NET 8 container images listen on port 8080 as a non-root user.
Apply. In Kubernetes with an NGINX ingress terminating TLS, each service’s Kestrel listens on plain HTTP 8080 (the .NET 8+ image default) and HTTP/2 without TLS for gRPC via Protocols = HttpProtocols.Http2 on a second port. UseForwardedHeaders is the first middleware, with KnownNetworks set to the cluster CIDR so a client cannot spoof X-Forwarded-For. MaxRequestBodySize is lowered to 2 MB globally and raised per endpoint for the document upload route with [RequestSizeLimit]. AddServerHeader = false removes Server: Kestrel. Where a BFF is needed, YARP runs as its own service with route-level transforms adding the user’s access token to downstream calls.
Test. Integration tests send requests with X-Forwarded-Proto: https from a trusted address and assert Request.Scheme is https and that a redirect’s Location uses the public host; a test from an untrusted address asserts the header is ignored. Load tests via the ingress compare p99 to direct-to-pod to quantify proxy overhead. A test posts a 3 MB body and expects 413 ProblemDetails. For HTTP/2 gRPC, a test client with HttpVersionPolicy.RequestVersionExact proves the unencrypted HTTP/2 endpoint negotiates correctly.
Troubleshoot. OIDC redirects going to http:// or to the pod’s internal hostname is the signature of missing or untrusted forwarded headers; Microsoft.AspNetCore.HttpOverrides Debug logs say “Unknown proxy” with the IP to add. BadHttpRequestException: Request body too large is the size limit. 502s from the proxy under load with Kestrel idle usually mean the proxy’s keep-alive timeout exceeds Kestrel’s KeepAliveTimeout (130 seconds default), so Kestrel closed the connection the proxy was about to reuse; align them. HTTP/2 gRPC failing with HTTP_1_1_REQUIRED means the proxy downgraded the backend protocol; enable grpc_pass or HTTP/2 backend in the ingress annotations. Slow clients holding connections are visible in kestrel-current-connections counters and bounded by MinRequestBodyDataRate.
An important consideration: trusting forwarded headers from any source turns client IP based rate limiting and audit logs into spoofable data; KnownProxies or KnownNetworks must always be set.
Rate limiting caps how many requests a caller may make in a window, protecting the service and downstream dependencies from bursts, abuse and runaway clients. Since .NET 7 it is built in: Microsoft.AspNetCore.RateLimiting provides middleware (UseRateLimiter) driven by System.Threading.RateLimiting algorithms, with policies attached to endpoints or groups via RequireRateLimiting("name") and [EnableRateLimiting] / [DisableRateLimiting].
The four built-in algorithms:
| Algorithm | Behaviour | Fit |
|---|---|---|
| Fixed window | N permits per window; resets on boundary | Simple quotas; bursty at boundaries |
| Sliding window | Window split into segments; smoother than fixed | Public API quotas |
| Token bucket | Tokens replenish at a rate; bursts up to bucket size | Allow short bursts, bound average |
| Concurrency | At most N in flight; releases when request completes | Protect a slow downstream or CPU-heavy endpoint |
Partitioning decides who is limited: per API key, per user, per IP, or global. Requests beyond the limit are rejected (default 503, usually set to 429) or queued up to QueueLimit.
builder.Services.AddRateLimiter(o =>
{
o.RejectionStatusCode = StatusCodes.Status429TooManyRequests;
o.AddPolicy("per-key", ctx => RateLimitPartition.GetTokenBucketLimiter(
ctx.Request.Headers["X-Api-Key"].ToString(),
_ => new TokenBucketRateLimiterOptions
{
TokenLimit = 100, TokensPerPeriod = 20,
ReplenishmentPeriod = TimeSpan.FromSeconds(1), QueueLimit = 0
}));
o.OnRejected = (ctx, _) =>
{
ctx.HttpContext.Response.Headers.RetryAfter = "1";
return ValueTask.CompletedTask;
};
});
app.UseRateLimiter();
app.MapGet("/search", Search).RequireRateLimiting("per-key");The built-in limiters are in-memory and per instance; with several replicas the effective limit multiplies unless the limit is enforced at the gateway or backed by a distributed store (Redis-based implementations exist as community packages).
Apply. For a public search API behind an ingress running six replicas, I split responsibility: the gateway enforces the coarse per-API-key quota (shared state, 429 with Retry-After), and each ASP.NET Core instance enforces local protection, a concurrency limiter of 20 on the expensive /search endpoint so a burst cannot saturate Elasticsearch, and a token bucket per authenticated user on write endpoints. Partition keys come from the validated user ID or API key, never the raw IP, because the proxy would make every caller share one IP unless forwarded headers are trusted. Health endpoints are [DisableRateLimiting], and rejections emit a metric tagged by policy.
Test. Unit-style tests use WebApplicationFactory with a shrunk policy (limit 2 per second) and send three requests in a tight loop, asserting two 200s, one 429 with Retry-After, and then a 200 after waiting. For the concurrency limiter, a test holds two requests open with a TaskCompletionSource in the fake search service and asserts the third gets 429 immediately.
Troubleshoot. Every caller being limited together means the partition key resolved to an empty string or to the proxy IP; log the key in OnRejected to confirm. Limits that are “six times too generous” are the per-instance nature of in-memory limiters. 503 instead of 429 is the default RejectionStatusCode. Requests hanging rather than failing means QueueLimit is non-zero with a long window; set QueueLimit = 0 for interactive APIs.
An important consideration: rate limiting middleware runs after routing, so the policy applies to the matched endpoint; requests that do not match any route (404s) are not limited and still need protection at the proxy.
SignalR is ASP.NET Core’s library for server-to-client push over a persistent connection. A hub is a server class whose public methods clients call, and which calls methods on connected clients (Clients.All, Clients.Group("orders-42"), Clients.User(userId), Clients.Caller). Transport is negotiated automatically: WebSockets when available, falling back to Server-Sent Events or long polling, so the application code never deals with the socket. Messages use a JSON protocol by default or MessagePack for compact binary.
public sealed class OrderHub : Hub
{
public Task Watch(Guid orderId) =>
Groups.AddToGroupAsync(Context.ConnectionId, $"order-{orderId}");
}
// From an application service, outside the hub:
public sealed class OrderNotifier(IHubContext<OrderHub> hub)
{
public Task StatusChanged(Guid orderId, string status) =>
hub.Clients.Group($"order-{orderId}")
.SendAsync("statusChanged", new { orderId, status });
}Clients exist for JavaScript/TypeScript, .NET, Java and Blazor; strongly typed hubs (Hub<IOrderClient>) replace magic method names with an interface. Authentication reuses the app’s schemes (bearer tokens are passed as a query string for WebSockets, which the handler supports via OnMessageReceived), and [Authorize] on a hub or method works as on controllers.
Hubs are transient and hold no state between calls; per-connection state goes in Context.Items or groups. Scaling beyond one server requires a backplane so a message sent on one instance reaches clients connected to another: the Redis backplane (AddStackExchangeRedis) or Azure SignalR Service, which also offloads the connections themselves. Sticky sessions are required at the load balancer unless WebSockets are guaranteed, because the negotiate request and the connection must hit the same instance.
Apply. For a logistics dashboard showing live shipment status to dispatchers, the API publishes status events to an OrderHub via IHubContext<OrderHub, IOrderClient> from the outbox dispatcher, so updates are sent exactly once after the database commit rather than from inside the request. Clients join groups per depot, which keeps fan-out bounded and authorization simple: the Watch method checks the user’s depot claim before adding to the group. The service runs three replicas with a Redis backplane and the ingress configured for WebSockets with sticky sessions as a fallback. MessagePack protocol cuts bandwidth for the high-frequency position stream, and MaximumReceiveMessageSize is lowered to 32 KB.
Test. Hub methods are unit-tested by instantiating the hub with a mocked IHubCallerClients<IOrderClient> and HubCallerContext, asserting the group add and the authorization branch. Integration tests use WebApplicationFactory with HubConnectionBuilder pointed at factory.Server.CreateHandler(), connect two clients, trigger a status change through the HTTP API, and assert only the subscribed client receives statusChanged within a timeout. The backplane path is covered with two factory instances sharing one Redis container. A load test with the Microsoft.Azure.SignalR crankier-style tool or a custom .NET client farm validates 5,000 concurrent connections per pod.
Troubleshoot. Clients missing updates intermittently on multiple replicas is the missing backplane or a backplane with a different channel prefix per instance. Connections falling back to long polling (visible in the negotiate response and Microsoft.AspNetCore.SignalR Debug logs) point to a proxy not upgrading WebSockets. 401 on connect with a valid token means the token is in the query string and OnMessageReceived is not reading access_token. Memory growth per connection points to a ConcurrentDictionary of connection IDs never cleaned in OnDisconnectedAsync. “Connection closed with an error” after 30 seconds is the ingress idle timeout being shorter than SignalR’s KeepAliveInterval (15 seconds) times two; raise the proxy timeout.
An important consideration: SignalR delivery is at-most-once with no persistence; a client that reconnects must refetch current state over HTTP, so the push is a notification, not the source of truth.
gRPC is a contract-first RPC framework using Protocol Buffers for compact binary serialisation and HTTP/2 for multiplexed, streaming transport. In .NET the Grpc.AspNetCore package hosts services on Kestrel and Grpc.Net.Client provides the client; the .proto file is compiled by Grpc.Tools at build time into C# message classes, a service base class and a client, so request and response types are shared without hand-written DTOs.
Four call types exist: unary, server streaming, client streaming and bidirectional streaming, all expressed in the .proto and surfaced as Task<T>, IAsyncStreamReader<T> and IServerStreamWriter<T> in C#.
public sealed class InventoryService(IStockRepository repo)
: Inventory.InventoryBase
{
public override async Task<StockReply> GetStock(StockRequest request,
ServerCallContext context)
{
var level = await repo.GetAsync(request.Sku, context.CancellationToken)
?? throw new RpcException(new Status(StatusCode.NotFound, request.Sku));
return new StockReply { Sku = request.Sku, Quantity = level };
}
}
// Program.cs
builder.Services.AddGrpc();
app.MapGrpcService<InventoryService>();Compared with JSON over HTTP/1.1, gRPC gives smaller payloads, lower latency, strict schemas with backward-compatible evolution rules (field numbers, never reuse), deadlines and cancellation propagated end to end, and first-class streaming. It is the natural choice for service-to-service calls inside a cluster. Its constraints: browsers cannot speak native gRPC (gRPC-Web with a proxy or Grpc.AspNetCore.Web is the workaround), it needs HTTP/2 end to end (or HTTP/3), and human debugging requires tooling (grpcurl, grpcui) rather than a browser. gRPC JSON transcoding (.NET 7+) exposes the same service as REST+JSON from annotations in the .proto, giving both interfaces from one implementation. Client-side, AddGrpcClient<T> integrates with IHttpClientFactory, resilience handlers and OpenTelemetry propagation.
Apply. Between the order service and the inventory service inside the cluster, I use gRPC unary calls for reservations with a 500 ms deadline set by the caller (CallOptions with Deadline), and a server-streaming call for stock-level subscriptions replacing a polling loop. The .proto files live in a shared contracts repository published as a NuGet package containing only the generated code, so both sides compile against the same version; evolution follows protobuf rules (add fields, never renumber, reserve removed numbers). Clients are registered with AddGrpcClient<Inventory.InventoryClient> plus AddStandardResilienceHandler with retries only on Unavailable. Kestrel exposes HTTP/2 without TLS on an internal port because the mesh handles mTLS. Public consumers get JSON transcoding on the same service.
Test. Service implementations are unit-tested by calling the overridden method with a TestServerCallContext and fake repositories, asserting the RpcException status code on not-found. Integration tests use WebApplicationFactory and create a GrpcChannel over factory.Server.CreateHandler(), which runs the full interceptor and auth pipeline in-process. Streaming is tested by consuming ResponseStream.ReadAllAsync() with a cancellation token and asserting the sequence. A contract check runs buf breaking against the previous tag of the .proto package in CI so a wire-breaking change fails the build.
Troubleshoot. StatusCode=Unavailable, "An HTTP/2 connection could not be established" is HTTP/1.1 only on the server or a proxy downgrading; verify Protocols = Http2 and the ingress backend protocol. DeadlineExceeded on the client with the server reporting success indicates clock skew or a deadline shorter than the dependency’s p99; the OpenTelemetry trace shows where the time went. Large messages failing with ResourceExhausted hit the 4 MB MaxReceiveMessageSize; stream instead of raising it. Client-side load balancing not spreading across pods happens because one HTTP/2 connection carries all calls; enable EnableMultipleHttp2Connections or use a headless Service with the gRPC client’s round_robin resolver.
An important consideration: gRPC retries must be scoped to idempotent methods; retrying a non-idempotent Reserve on Unavailable can double-book stock when the first call actually succeeded.
Resilience is the practice of keeping a service correct and responsive while its dependencies are slow, flaky or down. Polly is the .NET library that packages the standard patterns as composable strategies in a pipeline; Polly v8 (2023) rewrote the API around ResiliencePipeline, and Microsoft.Extensions.Resilience and Microsoft.Extensions.Http.Resilience integrate it with DI, options and HttpClientFactory.
The strategies and what each defends against:
The diagram shows the order in which the pipeline in the example wraps a single outbound call.

builder.Services.AddHttpClient<PricingClient>(c =>
c.BaseAddress = new Uri("https://pricing.internal"))
.AddResilienceHandler("pricing", p => p
.AddTimeout(TimeSpan.FromSeconds(5)) // total
.AddRetry(new HttpRetryStrategyOptions
{
MaxRetryAttempts = 3, BackoffType = DelayBackoffType.Exponential,
UseJitter = true
})
.AddCircuitBreaker(new HttpCircuitBreakerStrategyOptions
{
FailureRatio = 0.5, SamplingDuration = TimeSpan.FromSeconds(30),
MinimumThroughput = 20, BreakDuration = TimeSpan.FromSeconds(15)
})
.AddTimeout(TimeSpan.FromSeconds(1))); // per attemptAddStandardResilienceHandler() applies exactly this shape with sensible defaults. Order matters: strategies execute outermost first, so total timeout wraps retry, which wraps the circuit breaker and the per-attempt timeout. Resilience is a complement to, not a replacement for, idempotency keys, deadlines propagated across hops, and graceful degradation designed into the product.
Apply. In a checkout service I tune pipelines per dependency rather than applying one default. Pricing (read-only, fast) gets the standard handler plus hedging after 200 ms against a second replica. The payment provider gets no automatic retries on POST /charge; instead the request carries an idempotency key and the client retries only on a network-level failure before the request was sent, or by querying the charge status. The fraud service gets a circuit breaker and a fallback that marks the order “pending review” rather than failing checkout. Breaker state changes are exported as metrics so operators see opens and closes.
Test. Each pipeline is tested in isolation with a fake HttpMessageHandler scripted to fail N times then succeed, asserting attempt count and total elapsed time using Polly’s FakeTimeProvider so backoff is simulated instantly. Breaker tests drive 20 failures and assert the 21st call throws BrokenCircuitException without touching the handler, then advance time past BreakDuration and assert a probe goes through. Integration tests with WireMock.Net inject 503s and latency and assert the user-facing outcome (pending review, not 500).
Troubleshoot. A dependency outage turning into a self-inflicted flood is retries without a breaker, or a breaker with MinimumThroughput so high it never trips; the dependency’s request-rate graph tripling at the incident start confirms it. Retries on POST producing duplicate orders is a missing idempotency key, not a Polly setting. BrokenCircuitException leaking to users as 500 means no fallback and no mapping to a 503 ProblemDetails.
An important consideration: retries multiply load exactly when the dependency is weakest, so every retry policy should be justified by a measured transient failure rate, with jitter and a breaker always present.
A distributed transaction tries to make writes to two independent resources (a database and a message broker, or two databases) atomic. The classic mechanism is two-phase commit via a coordinator (MSDTC with TransactionScope on .NET Framework). In .NET Core and modern .NET, distributed transactions are effectively unavailable: TransactionScope spanning two connections throws PlatformNotSupportedException on Linux, Windows-only support returned in .NET 7, and brokers such as RabbitMQ, Kafka and Azure Service Bus do not participate in 2PC at all. Microservice architectures therefore avoid them by design.
The transactional outbox pattern replaces the distributed transaction with a local one. The service writes its business rows and a row describing the event to publish into the same database in one transaction. A separate dispatcher reads unpublished outbox rows, publishes them to the broker, and marks them sent. The database guarantees the event exists if and only if the business change committed; the dispatcher guarantees eventual publication, with at-least-once delivery, so consumers must be idempotent (an inbox table of processed message IDs is the mirror pattern).

Libraries such as MassTransit, NServiceBus, Wolverine and CAP implement outbox and inbox on top of EF Core so the pattern is a configuration switch rather than hand-written code. Where a multi-step business process spans services, a saga (orchestrated or choreographed, with compensating actions) replaces the atomicity a distributed transaction would have offered.
Apply. In an order service on PostgreSQL publishing to Azure Service Bus, the PlaceOrder handler adds the Order aggregate and an OutboxMessage (ID, type, JSON payload, occurred-at, trace context) to the same DbContext and calls SaveChangesAsync once, so one transaction covers both. A BackgroundService dispatcher polls with FOR UPDATE SKIP LOCKED in batches of 100, publishes with the outbox ID as the broker MessageId for duplicate detection, marks rows sent, and deletes rows older than seven days. Consumers in other services keep an inbox table keyed by MessageId and process inside a transaction with the inbox insert, which makes them idempotent. The cross-service reserve-charge-confirm flow is a MassTransit saga with compensation, not a distributed transaction.
Test. An integration test places an order with the broker unreachable and asserts the order and outbox rows exist and the HTTP call succeeded, which is the whole point of the pattern. A second test starts the broker (Testcontainers), runs one dispatcher iteration, and asserts the message arrived and the row is marked. Idempotency is tested by delivering the same message twice to a consumer and asserting one side effect. A saga test drives the payment-failed path and asserts stock was released.
Troubleshoot. Events missing downstream while orders exist means the dispatcher stopped (hosted-service exception killed it) or the outbox write was outside the transaction (a second SaveChanges or a separate context). Duplicate downstream effects reveal a consumer without an inbox or a dispatcher that publishes then crashes before marking; the broker’s duplicate detection window helps but the inbox is the real fix. A growing outbox table with rising OutboxLagCheck readiness degradation points to broker throttling or a poison message blocking the batch; move it to a dead-letter status after N attempts.
An important consideration: the outbox trades atomicity for eventual consistency, so the UI and downstream services must tolerate a delay of seconds and must not assume the event has already been processed when the HTTP response returns.
Containerising a .NET application means packaging the published output and the runtime into an OCI image so it runs identically on any host. Microsoft publishes base images on mcr.microsoft.com/dotnet: sdk for building, aspnet for running web apps, runtime for console apps, runtime-deps for self-contained or Native AOT publishes, in Debian, Ubuntu, Alpine and distroless (-chiseled) variants. Since .NET 8 the images run as a non-root app user on port 8080 by default.
Two ways to build an image:
sdk stage, copy the output into the slim aspnet stage. Layer caching works best when *.csproj files are copied and restored before the rest of the source.dotnet publish -t:PublishContainer (built into the SDK since .NET 7/8) builds an image without a Dockerfile or Docker daemon, with ContainerRepository, ContainerImageTag and base image set in the csproj; .NET Aspire uses this for its deployments.FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY Directory.Build.props Directory.Packages.props ./
COPY src/Orders.Api/Orders.Api.csproj src/Orders.Api/
RUN dotnet restore src/Orders.Api/Orders.Api.csproj
COPY . .
RUN dotnet publish src/Orders.Api -c Release -o /app --no-restore
FROM mcr.microsoft.com/dotnet/aspnet:10.0-noble-chiseled
WORKDIR /app
COPY --from=build /app .
EXPOSE 8080
ENTRYPOINT ["dotnet", "Orders.Api.dll"]Runtime behaviour inside a container differs in ways that matter: the GC and thread pool read cgroup CPU and memory limits (so a 0.5 CPU limit means one thread pool thread and workstation GC unless overridden), DOTNET_ environment variables configure the runtime, configuration keys use __ as the section separator, and SIGTERM triggers graceful host shutdown. Chiseled images have no shell, so diagnostics tools must be attached from a sidecar or the image must include dotnet-monitor.
Apply. For an order API I use the multi-stage Dockerfile above with the chiseled Ubuntu base, because it has no shell or package manager and roughly a third of the CVE surface of the full image. Central Package Management files are copied first so the restore layer is cached across most commits. The image is built once in CI, tagged with the git SHA, scanned with Trivy, signed with cosign, and promoted through environments unchanged; configuration differences come from environment variables and mounted secrets, never from rebuilding. Resource requests and limits are set in the Helm chart with at least 1 CPU so Server GC is selected, DOTNET_GCHeapHardLimitPercent=75, and ASPNETCORE_FORWARDEDHEADERS_ENABLED=true. A dotnet-monitor sidecar provides dumps and traces on demand without tools in the app image.
Test. The pipeline runs the integration suite against the built image, not the source, using Testcontainers to start the image alongside PostgreSQL, so a missing native dependency or a wrong ENTRYPOINT fails before deployment. A structure test asserts the user is non-root, the port is 8080 and no secrets are baked into layers. docker scout or Trivy gates on critical CVEs.
Troubleshoot. A container that exits immediately with no logs: run with --entrypoint overridden to list /app or check dotnet Orders.Api.dll is the right assembly name; for chiseled images use docker run --rm -it ... /bin/sh on the non-chiseled variant to compare. Globalization Invariant Mode exceptions on Alpine mean ICU is missing; install icu-libs or set InvariantGlobalization. OOMKilled with a heap far below the limit is non-GC memory (native buffers, many threads); DOTNET_GCHeapHardLimit must leave headroom. Configuration not applied is Payment:Timeout written with a colon instead of Payment__Timeout in the env var.
An important consideration: image tags like aspnet:10.0 move as patches ship, so pinning to a digest or rebuilding regularly is required to actually receive security fixes while keeping builds reproducible.
Azure Functions is a serverless host that runs your code in response to triggers (HTTP, queue, timer, Service Bus, Blob, Event Hubs, Cosmos DB change feed). The isolated worker model runs your .NET code in a separate process from the Functions host, communicating over gRPC. It is the only model supported for .NET 9 and later and for current .NET LTS releases. The older in-process model, which loaded your assembly into the host and tied you to the host’s .NET version, ends support in November 2026, so new work should always use isolated.
What isolated gives you:
Program.cs with HostBuilder / FunctionsApplication, so DI, configuration, logging and middleware work exactly as in a normal generic-host app.IFunctionsWorkerMiddleware) for cross-cutting concerns such as exception handling and correlation.ConfigureFunctionsWebApplication), which lets HTTP triggers use HttpRequest and IActionResult instead of the host’s own request types.var builder = FunctionsApplication.CreateBuilder(args);
builder.ConfigureFunctionsWebApplication();
builder.Services.AddApplicationInsightsTelemetryWorkerService()
.ConfigureFunctionsApplicationInsights();
builder.Services.AddSingleton<IOrderProcessor, OrderProcessor>();
builder.Build().Run();
public sealed class OrderFunctions(IOrderProcessor processor)
{
[Function("ProcessOrder")]
public Task Run(
[ServiceBusTrigger("orders", Connection = "Bus")] OrderMessage message,
CancellationToken ct) => processor.HandleAsync(message, ct);
}Hosting plans change behaviour: Consumption and Flex Consumption scale to zero and out automatically with cold starts, Premium keeps pre-warmed instances, and Dedicated runs on an App Service plan. Durable Functions adds orchestration for long-running workflows. Triggers are at-least-once, so function bodies must be idempotent.
Apply. In an order platform, a Service Bus-triggered function processes payment confirmations on the Flex Consumption plan: it scales with queue depth, authenticates to Service Bus and Key Vault through a managed identity (identity-based connection, no connection strings), and delegates immediately to an application service so the function class is a thin adapter. Idempotency is enforced with an inbox check on the message ID because delivery is at-least-once. Failures beyond MaxDeliveryCount land in the dead-letter queue, with a separate function alerting on it. A timer-triggered reconciliation uses [TimerTrigger] with RunOnStartup = false and a blob lease so only one instance runs it.
Test. Business logic is tested without the Functions runtime because the function body just calls a service. The thin function class gets a unit test with a fake service and a stub FunctionContext. For integration, Azurite emulates storage and the Service Bus emulator (a container) provides queues; a test starts func start or the host in-process, sends a message and polls for the database side effect. Middleware is tested by invoking it with a fake context and next delegate. Load tests send 10,000 messages and check scale-out and processing latency in Application Insights.
Troubleshoot. A function that “is not found” after deployment is usually a missing FUNCTIONS_WORKER_RUNTIME=dotnet-isolated setting or a worker extension package not referenced in the csproj. Duplicate processing means a non-idempotent handler combined with lock expiry; raise the lock duration or renew it. Slow first requests on Consumption are cold starts; fix with Flex Consumption always-ready instances or Premium pre-warmed workers. Logs missing from Application Insights are the classic isolated mistake of not calling ConfigureFunctionsApplicationInsights, plus the default logging filter that suppresses Information from the worker; set Logging:LogLevel in host.json and in code. Host crashes on startup appear in the Log Stream and Kudu diagnostics.
An important consideration: a scale-out function can overwhelm a downstream database, so cap concurrency with maxConcurrentCalls and the plan’s maximum instance count.
Secrets (database passwords, API keys, signing keys, client secrets) must never live in source control, container images or plain appsettings.json. The principle is layered: the least possible number of long-lived secrets, kept in a managed store, delivered to the process at run time, accessed with an identity rather than another secret.
The .NET tooling by environment:
dotnet user-secrets stores values in a per-user file outside the repository, loaded automatically by the default host in the Development environment. It is not encryption, only separation from the repo.Azure.Extensions.AspNetCore.Configuration.Secrets adds Key Vault as a configuration provider, so secrets bind into IOptions<T> like any other value; -- in secret names maps to the : section separator.DefaultAzureCredential (workload identity federation in Kubernetes, OIDC in CI), so no bootstrap secret exists.if (!builder.Environment.IsDevelopment())
{
builder.Configuration.AddAzureKeyVault(
new Uri(builder.Configuration["KeyVault:Uri"]!),
new DefaultAzureCredential(),
new AzureKeyVaultConfigurationOptions
{ ReloadInterval = TimeSpan.FromMinutes(30) });
}Related protections: ASP.NET Core Data Protection keys (used for cookies and antiforgery) must be persisted to shared storage and protected with a key in the vault, or every replica and every restart invalidates sessions. Prefer passwordless connections (Entra ID tokens for Azure SQL and PostgreSQL), short-lived tokens over static keys, and automatic rotation. Secrets must also be kept out of logs, crash dumps and exception messages.
Apply. For an order API on AKS, the pod uses workload identity to reach Key Vault and Azure SQL; the connection to SQL uses an Entra access token, so there is no database password at all. The remaining secrets (payment provider API key, webhook signing secret) live in Key Vault, loaded through the configuration provider with a 30-minute reload and bound to validated options. Data Protection keys are stored in blob storage and encrypted with a Key Vault key, shared by all replicas. CI authenticates to Azure with OIDC federation, so the pipeline holds no stored credentials. Rotation is scheduled, with two webhook secrets valid during the overlap.
Test. A repository scan with gitleaks or GitHub secret scanning runs on every PR and as a pre-commit hook, and the history is scanned once at onboarding. A startup test asserts that options with ValidateOnStart fail when a required secret is absent. An integration test confirms that logging a configuration object redacts sensitive properties ([LogProperties] with redaction or [SensitiveData] handling). Rotation is rehearsed in staging: rotate, observe zero failed requests.
Troubleshoot. Azure.Identity.CredentialUnavailableException or a Key Vault 403 on startup means the identity is not assigned the Key Vault Secrets User role, the federated credential subject does not match the service account, or a firewall blocks the vault; the exception lists each credential in the DefaultAzureCredential chain and why it failed. Users logged out after every deploy means Data Protection keys are stored on the container filesystem. A secret found in a commit is treated as compromised: rotate first, then rewrite history.
An important consideration: environment variables are readable by anyone with exec access to the pod and appear in crash dumps, so for the most sensitive values prefer a mounted file or direct vault access with short-lived caching.
CI/CD for .NET automates build, test, package and release. Continuous integration runs on every commit: dotnet restore, build, test with coverage, static analysis (Roslyn analyzers, dotnet format --verify-no-changes), dependency and vulnerability audits (dotnet list package --vulnerable), and building an immutable artifact (a container image or a dotnet publish zip). Continuous delivery/deployment promotes that same artifact through environments with configuration injected per environment. Common platforms are GitHub Actions, Azure DevOps Pipelines and GitLab CI.
Deployment strategies differ in risk, cost and rollback speed:
| Strategy | How it works | Trade-off |
|---|---|---|
| Rolling | Replace instances gradually | Cheap; old and new versions coexist briefly |
| Blue/green | Deploy full new stack, switch traffic | Instant rollback; double capacity during switch |
| Canary | Route 1-10% of traffic to new version, then ramp | Real-traffic validation; needs metrics and routing |
| Feature flags | Ship code dark, enable per user/tenant | Decouples deploy from release; flag debt |
Platform support: Kubernetes rolling updates are the default; Argo Rollouts or Flagger add canary and blue/green with metric-based analysis; Azure App Service has deployment slots with swap and warm-up; Azure Container Apps has revisions with traffic splitting.
- uses: actions/setup-dotnet@v4
with: { dotnet-version: 10.0.x }
- run: dotnet test -c Release --no-build --collect:"XPlat Code Coverage"
- run: dotnet publish src/Orders.Api -c Release -t:PublishContainer
- run: az containerapp update -n orders -g rg --image $IMAGE --revision-suffix $SHADatabase schema changes must be backward compatible with the previous version (expand/contract), because during any strategy both versions run against the same database.
Apply. For an order platform on Kubernetes, the pipeline builds one image per commit, runs unit and Testcontainers integration tests, scans and signs the image, and deploys to staging automatically; production is a canary managed by Argo Rollouts at 5%, 25%, 100% with automated analysis against Prometheus (error rate and p99 latency compared with the stable version). Migrations run as a pre-deploy Job and are always expand/contract. New behaviour ships behind feature flags so a bad feature is disabled without redeploying. Rollback is the previous image tag, which stays valid because the schema is backward compatible. Environment promotion uses the same artifact digest, never a rebuild.
Test. The pipeline itself is tested: a deployment to an ephemeral namespace per PR runs smoke tests against /healthz/ready and a handful of business-critical API calls (synthetic checks) before merge. Rollback is rehearsed in staging regularly, including a version that fails readiness, to prove the rollout aborts. Canary analysis thresholds are tested by deploying a deliberately slow build to staging and verifying the rollout auto-aborts. Database changes are verified by running the previous application version’s integration tests against the migrated schema.
Troubleshoot. A rollout stuck at 5% shows the analysis failing; the Rollout status names the metric, and comparing the canary’s traces against the stable ones usually reveals the cause. Errors only on deploy often mean readiness is too optimistic (traffic arrives before caches warm); add a warm-up and a readiness check that reflects it, and a preStop sleep so in-flight requests drain. Configuration drift appears as “works in staging, fails in production”; diff the rendered Helm values and environment variables. A slot swap that causes a cold start spike means the slot was not warmed with applicationInitialization. Failed migrations during rollout leave the job logs and the __EFMigrationsHistory table as evidence.
An important consideration: automated promotion is only as safe as the signals it watches, so a canary without meaningful error-rate and latency gates is just a slower rolling update.
Production debugging means diagnosing a live or recently failed process without attaching a traditional debugger that would halt it. .NET provides cross-platform diagnostic tools built on the diagnostic IPC channel exposed by every .NET process (a Unix domain socket or named pipe), so they work in containers without being baked into the app.
The main tools:
dotnet-counters: live metrics (CPU, GC, thread pool, request rate) to see what is wrong.dotnet-trace: CPU sampling and event traces, openable in PerfView, Speedscope or Visual Studio, to see where time goes.dotnet-dump: collects a full or heap-only dump (collect) and analyses it on any OS (analyze) with SOS commands.dotnet-gcdump: a lightweight managed heap snapshot without pausing for a full process dump, used to compare memory growth.dotnet-monitor: a sidecar that exposes dumps, traces, metrics and logs over HTTP with authentication, and can trigger collection automatically on rules (CPU above 80% for 30 seconds).DOTNET_DbgEnableMiniDump=1 with DOTNET_DbgMiniDumpType and DOTNET_DbgMiniDumpName makes the runtime write a dump when the process crashes.Frequent SOS commands in dotnet-dump analyze:
> threads list managed threads
> clrstack -all managed stack of every thread
> pstacks grouped stacks; shows many threads in same place
> dumpheap -stat object counts and sizes by type
> gcroot <address> why an object is still alive
> syncblk lock owners and waiters
> dumpasync pending async state machinesA dump is a snapshot of memory, so it contains whatever the process held: connection strings, tokens, customer data. Dumps must be handled as sensitive data, stored encrypted with limited access and deleted after analysis. Symbols and matching runtime versions matter; analysing a Linux dump on Windows works with dotnet-dump analyze or WinDbg with SOS, provided the same runtime’s libmscordaccore is available.
Apply. For an order service on Kubernetes, the images are chiseled with no shell, so I run dotnet-monitor as a sidecar sharing the diagnostic socket via an emptyDir volume, protected by the same OIDC auth as the platform’s admin tools. Collection rules trigger a heap-only dump automatically when gc-heap-size crosses 80% of the container limit and a 30-second CPU trace when CPU exceeds 85%, writing artifacts to a locked-down storage account with a seven-day lifecycle. DOTNET_DbgEnableMiniDump is set so crashes leave a dump on a mounted volume.
Test. A game day exercises the whole process: a staging build with a deliberate leak (a static list) and another with sync-over-async blocking is deployed, and an engineer who did not write them must find each using only dotnet-counters, a gcdump and a dump. The automated collection rules are verified by forcing the trigger conditions. The runbook is tested by confirming the dump downloads, opens in dotnet-dump analyze and that the first three commands (clrstack -all, dumpheap -stat, pstacks) give actionable output.
Troubleshoot. A hung service shows low CPU and a growing threadpool-queue-length; pstacks on a dump reveals dozens of threads blocked in Task.GetAwaiter().GetResult() in one method. A memory leak shows as two dotnet-gcdump files a few hours apart diffed in Visual Studio; the type with growing count is followed with gcroot to the holding static or event. A crash with exit code 134 and no logs is an unhandled exception or stack overflow; the crash dump’s clrstack identifies it.
An important consideration: collecting a full dump freezes the process for seconds to minutes and writes memory-sized files, so take it from one pod that has been drained from the load balancer, not from every replica at once.
Scaling an ASP.NET Core service means increasing the load it can handle either vertically (bigger instance, more cores and memory) or horizontally (more instances behind a load balancer). Horizontal scaling is the default in cloud deployments because it also gives redundancy, but it only works if instances are interchangeable, which means the service must be stateless.
What statelessness requires in ASP.NET Core:
IDistributedCache / HybridCache with Redis for shared data.The usual limits are elsewhere than the web tier: database connections (each replica opens its own pool, so 20 pods times 100 connections can exhaust the server), downstream rate limits, and thread-pool starvation from blocking calls. Scaling out a service with sync-over-async code multiplies the problem instead of solving it.

Autoscaling reacts to metrics: CPU or memory with the Kubernetes Horizontal Pod Autoscaler, request rate or queue length with KEDA, or platform rules on App Service and Container Apps. Queue-based load levelling, caching, read replicas, and AsNoTracking projections reduce the load before adding instances.
Apply. For an order API facing seasonal peaks, I keep the web tier stateless (Redis-backed HybridCache, Data Protection keys in blob storage, uploads to blob storage) and move slow work behind a queue so the HTTP path stays short. The Kubernetes HPA scales on CPU at 60% with a minimum of three pods across zones, while a KEDA scaler on queue length scales the worker deployment separately. Database connections are budgeted: Maximum Pool Size times the maximum pod count stays below the server limit, with PgBouncer in front of PostgreSQL. Read-heavy queries go to a replica through a read-only DbContext.
Test. Load tests with k6 ramp from baseline to three times expected peak and hold, while recording p95/p99 latency, error rate, pod count, database connections and threadpool-queue-length. Failure tests kill a pod and a zone during load and verify no errors reach clients beyond the retry budget. Autoscaling is tested by measuring the time from load increase to new pods ready, which sets the headroom needed because scale-out is not instantaneous.
Troubleshoot. Adding pods without gaining throughput points to a bottleneck downstream: database CPU or connection saturation (pg_stat_activity count at the limit), a rate-limited dependency, or a lock. Latency rising while CPU is low is thread-pool starvation from blocking calls, seen in dotnet-counters. Uneven load across pods happens with long-lived HTTP/2 or gRPC connections pinned to one pod, fixed with client-side load balancing or connection lifetime limits. Users logged out randomly after scale-out means Data Protection keys are per-instance. Scale-in killing in-flight work needs graceful shutdown and a preStop delay.
An important consideration: autoscaling amplifies whatever the bottleneck is, so a service that scales out into a saturated database converts a slow incident into a full outage.
Add your resume and the job description in CrackItX's Settings. When the interviewer asks a .NET / .NET Core / ASP.NET question, Claude suggests an answer using your real projects and the role's requirements, in real time.
See pricing →