Java Services, AWS Platform, and Reliability

This case study covers my contribution to defined services rather than ownership of the wider platform.

Application changes in a platform with many integration points can fail late - at build, dependency resolution, or integration time - rather than in the change itself.

• Every change goes through merge-request review before it can be deployed
• Services must stay compatible with existing consumers and upstream data contracts
• Dependency and build changes must not alter runtime behaviour silently

• Developing Java application and configuration changes within a defined service scope
• Updating build and dependency configuration when versions or transitive conflicts require it
• Integrating services with platform components such as messaging, storage, and relational data
• Raising, reviewing, and revising changes through GitLab merge requests
• Adding or extending tests so integration behaviour is checked before deployment
• Troubleshooting behaviour reported in higher environments and production

• Relying on manual verification in a shared environment after deployment (faster per change, but failures surface later and are harder to attribute)
• Pinning every dependency indefinitely to avoid build churn (stable short-term, but accumulates upgrade risk)

• Transitive dependency upgrades that compile cleanly but change runtime behaviour
• Configuration values present in one environment and absent in another
• Changes that pass in isolation but break a consumer's expectations

• Keeping integration contracts stable while services evolve
• Diagnosing failures that originate in configuration or dependency changes rather than application logic

Application, configuration, and integration changes in my scope ship through reviewed merge requests with build and integration checks, so integration problems are caught before deployment rather than in a shared environment.

• Java
• REST APIs
• PostgreSQL
• GitLab
• GitLab CI

Most of the time I spent on failing changes traced back to configuration or dependency differences rather than application logic, which is why I now check those first.

I would push more contract-level checks into the pipeline so a change that breaks a consumer fails in review rather than in a shared environment.

This case study covers my contribution to a defined service and infrastructure scope rather than ownership of the wider platform.

New services needed a consistent way to ship on AWS - private networking, container runtime, database access, auth - and a consistent way to move data between services asynchronously without each team inventing its own pattern.

• Everything provisioned through code review, not the AWS console
• Services had to fit a hybrid model spanning on-prem and multi-region AWS
• Traffic had to stay on private networking paths
• Event delivery needed explicit failure handling rather than best-effort retries

• Defining service infrastructure in CDK and CloudFormation so the deployment shape is reviewable alongside the code
• Running services on ECS Fargate, with Lambda for event-driven and asynchronous edges
• Using Aurora PostgreSQL as the system of record, with schema changes applied through Flyway migrations
• Exposing services through API Gateway with routing and authentication expressed as infrastructure code
• Keeping traffic on private networking paths using VPC endpoints, PrivateLink, and hybrid connectivity
• Developing event-driven integrations over Kafka/MSK and Debezium change data capture, with EventBridge routing, SQS queues, dead-letter queues for poison messages, and S3 for durable payloads

• Console-driven networking, routing, and auth configuration (faster initially, but invisible to source control and prone to drift)
• Synchronous service-to-service calls instead of events (simpler to trace, but couples availability of the two services)
• Application-level polling instead of change data capture (fewer moving parts, but higher latency and more load on the database)

• Duplicate event delivery requiring idempotent consumers
• Change-data-capture connector restarts replaying from an earlier offset
• Messages landing in a dead-letter queue with no owner watching it
• Cold-start behaviour on Lambda paths at the asynchronous edges

• Keeping the AWS-native slice consistent with on-prem services in the hybrid model
• Making asynchronous integrations safe to retry without duplicating effects
• Encoding routing and auth as infrastructure code without slowing routine changes

Services in my scope ship with a consistent, reviewable deployment shape on AWS, and their event integrations have explicit retry and dead-letter behaviour instead of implicit best-effort delivery.

• AWS
• AWS CDK
• CloudFormation
• ECS Fargate
• AWS Lambda
• API Gateway
• Aurora PostgreSQL
• Flyway
• Kafka / MSK
• Debezium
• EventBridge
• SQS
• S3

Infrastructure that is reviewable in the same diff as the code is easier to reason about than console configuration, especially when the same change has to ship across regions.

I would invest more in a shared construct library so a new service can adopt the platform defaults for networking, routing, and event wiring in a few lines.

This case study covers my contribution to reliability work on defined services rather than ownership of the platform's resilience strategy.

Deployments and regional failure paths existed, but their behaviour under load and during failover was largely assumed rather than observed.

• Exercises had to run without risking production data
• Rollback had to be possible at any point during a release
• Cross-region replication lag had to be accounted for in recovery expectations

• Contributing to blue-green deployment configuration so a release can be shifted back without redeploying
• Tuning ECS Auto Scaling policies against observed load rather than fixed capacity guesses
• Configuring and exercising multi-region disaster recovery on Aurora Global Database
• Executing fault-injection experiments with AWS FIS to observe how services behave when a dependency degrades
• Recording what each exercise showed, including the cases where recovery was slower than expected

• Relying on documented runbooks alone (cheap, but does not verify the system behaves as the runbook assumes)
• In-place rolling deployments only (simpler, but rollback is slower than shifting traffic back)

• Failover completing while replication lag left recent writes behind
• Auto Scaling reacting after the load spike had already caused errors
• Health checks reporting healthy while a downstream dependency was degraded

• Designing experiments that are informative without being risky
• Setting recovery expectations that account for replication lag

Failover and rollback paths for the services in my scope are rehearsed rather than assumed, and the observed behaviour - including slower-than-expected cases - is written down.

• AWS
• ECS Fargate
• ECS Auto Scaling
• Aurora Global Database
• AWS FIS
• CloudFormation

A recovery path that has never been exercised is an assumption. Running the experiment is usually cheaper than discovering the gap during an incident.

I would run fault-injection experiments on a regular schedule rather than around specific changes, so drift in recovery behaviour is noticed early.

This case study covers monitoring work on defined services rather than ownership of the platform's observability stack.

Monitoring was resource-centric: alerts fired on CPU, memory, or queue depth, which did not reliably indicate whether the service was actually failing its consumers.

• Signals had to work across both AWS-native and on-prem service paths
• Alerts had to be actionable enough to page on
• Dashboards had to be usable by someone unfamiliar with the service internals

• Instrumenting services with request, latency, error, and queue-processing metrics
• Defining service-level objectives for availability and latency on the paths consumers depend on
• Building Grafana dashboards that show the objective first and the supporting resource metrics second
• Wiring CloudWatch metrics, logs, and alarms for AWS-managed components such as queues, databases, and Lambda paths
• Alerting on objective burn and on dead-letter growth rather than on raw resource thresholds
• Using these signals during production troubleshooting to narrow a symptom to a layer

• Keeping threshold alerts on resource metrics (simple to configure, but weak signal-to-noise)
• Relying on log searches during incidents only (flexible, but slow and dependent on knowing what to search for)

• Metrics missing for a short window after a deployment, briefly resembling an outage
• Objectives satisfied in aggregate while one consumer path was failing
• Alerts that fired correctly but pointed at a symptom one layer below the cause

• Choosing objectives that reflect consumer expectations rather than convenient numbers
• Keeping dashboards readable as the number of services grew

Services in my scope have dashboards and alerts tied to service-level objectives, and production investigations start from a signal that indicates consumer impact rather than a resource metric.

• Prometheus
• Grafana
• CloudWatch
• SLOs
• AWS

An alert is only useful if someone can act on it. Tying alerts to objectives made it clearer which pages needed a response and which needed a fix in the monitoring itself.

I would connect dashboards more directly to trace data so moving from an objective breach to the failing request path takes fewer steps.

This case study covers migrations I executed on defined services rather than a platform-wide migration programme.

Some infrastructure changes replace resources rather than update them, and some schema changes break running application versions - both can cause downtime if applied in one step.

• Services had to stay available while the change was applied
• Every change had to be reviewable in source control before it ran
• Old and new application versions had to work against the same schema during a rollout

• Reading change sets before applying them, to see which resources would be replaced rather than updated
• Splitting risky changes into stages that are each safe on their own
• Making schema changes backwards-compatible first with Flyway migrations - add, backfill, switch reads, then remove
• Pairing infrastructure changes with blue-green traffic shifts where a resource had to be replaced
• Onboarding services onto the reviewed infrastructure-as-code workflow so future changes follow the same path
• Verifying behaviour with metrics and dashboards during and after each stage

• Applying the full change in one deployment window (fewer steps, but no safe rollback and risk of downtime)
• Taking a short maintenance window (predictable, but unnecessary once changes are staged properly)

• CloudFormation updates that quietly replace a resource and change its endpoint
• Migrations that lock a table long enough to time out requests
• Backfills that must run in batches to avoid replication lag

• Predicting which infrastructure changes cause replacement
• Sequencing schema changes so both application versions keep working

Stack and schema migrations in my scope were applied to running services without downtime, and each stage left a position the change could be rolled back from.

• CloudFormation
• AWS CDK
• Aurora PostgreSQL
• Flyway
• GitLab CI

Reading the change set before applying it is the cheapest step in the whole migration, and it is the one that most often changes the plan.

I would automate a pre-apply report that flags replacement-causing changes and long-locking migrations before a reviewer approves them.