akash-singh.prod
All systems operational Bangalore · IST

I keep production boring.

I'm Akash Singh, a Senior Site Reliability Engineer with 7 years on call for Hadoop clusters, Kubernetes fleets and Kafka streaming platforms. I build self-service infrastructure on AWS with Terraform and GitOps, set SLOs that teams actually use, and take a lot of money off the cloud bill along the way.

Prefer a shell? Open terminal mode, or press ` anywhere.
career.uptime · Q3 2019 → today 29 / 29 quarters shipping
Q3 2019202120232025now
Hover a bar to see the quarter.
phData PharmEasy VMware Allen Digital
$204k+per year in documented AWS savings at Allen Digital
~200microservices on the production app EKS cluster I managed
900+CDC connectors on Kafka Connect and MSK, reconciled from Git
597pull requests across ~60 repos in two years at Allen
cost · reliability · impact

Cost engineering, measured in dollars

Most of these came from migrations that had to stay invisible to users: zero-downtime cutovers, Spot runbooks, and routing changes that nobody noticed except finance.

Annual spend, before → after

USD / yr
Spark on EMR → Spot Instances −53%
after $45kbefore $96k
AWS OpenSearch → self-hosted (Terraform) −56%
after $35kbefore $80k
$0$25k$50k$75k$100k
beforeafter
$5,000+/moVPC consolidation with MSK and EBS rightsizing, zero downtime
$1,500+/moKafka connector networking via same-AZ broker alignment
~$30k/yrElastiCache reserved nodes plus a Compute Savings Plan

Reduction by initiative

% of prior cost
  • OpenSearch self-hostingAllen56%
  • Spark compute on SpotAllen53%
  • Inter-AZ transfer, zone-aware routingAllen~40%
  • ElastiCache RIs + Compute Savings PlanAllen~30%
  • Overall AWS billPharmEasy~25%
  • Groundcover ingestion, high-cardinality labels droppedAllen20%+
  • ElastiCache Redis → ValkeyAllen~20%

The Groundcover change also improved query latency by more than 25%.

flagship · allen digital · 2024–2026

A self-service CDC and event platform, built from zero

Every insert, update and delete from the app databases, plus client events from web, mobile and iPad, lands in ClickHouse through one Kafka backbone. I built it across three EKS clusters, provisioned it in Terraform, and carried the pager for it end to end.

sources
MySQLbinlog
MongoDBoplog
PostgreSQL · SQL ServerWAL · CDC
Event collectorweb, mobile, iPad events
stream · data EKS + MSK
Kafka Connect source connectorsDebezium on Strimzi, 900+ connectors as Git CRs
Amazon MSKmulti-AZ, CDC and event topics
ClickHouse sink connectorsbatching, type mapping, DLQ
warehouse · tools
ClickHouseanalytics warehouse
Airflow + SparkEMR on EKS, spot executors
OpenMetadata · Supersetcatalog, lineage, BI
AirbyteSaaS and ELT sources
SLIs I defined and alert on
freshnessthroughputconsumer lagerror budgeterror rateretriesDLQ volumeprocessing latencyper-tenant success

Infrastructure as code. MSK, RDS/Aurora, IRSA, S3 and EKS through Terragrunt, with reusable MSK, Kinesis, Valkey and self-hosted OpenSearch modules. Everything ships via GitOps: Jenkins builds, ArgoCD syncs, a promotion job moves stage to prod.

Drift control. A Jenkins reconcile job diffs connector names in Git against the live CRs and the Connect REST API, reports orphans, runs dry-run by default and needs prod approval to delete.

ownership · allen digital

Platform, observability and cost, owned end to end

Besides the data platform, I ran the app cluster, owned how the whole org saw its systems, and owned what it paid for them. Most of this is code: alerts, sampling rules and cost tags live in Git and ship through review.

Observability

groundcover · otel · grafana
  • As codeEvery alert, route and contact point in Terraform, applied by Jenkins on each PR. One teams file maps services to owners; prod pages PagerDuty, stage goes to Slack.
  • Signal, not noiseLonger default alert windows to stop flapping, scoped exclusions for scanners and telemetry paths, and per-service overrides instead of global mutes.
  • CardinalityDropped request_id in the OTel collector, capped Kong metric cardinality, and added head-based trace sampling to the shared Go service framework.
  • CoverageKong per-API status and upstream-health metrics; Airflow metrics moved from lossy StatsD/UDP to OTLP; batch-job logs routed to stdout so pipelines stopped being blind.
  • Cost of seeingA weekday ingestion-trend report on the noisiest services by log and trace volume, so growth gets caught before the invoice does.

Cost & capacity

finops · right-sizing
  • CommitmentsElastiCache reserved nodes plus a Compute Savings Plan: ~30% effective discount, ~$30k a year.
  • AttributionSpark pods on EMR-on-EKS tagged with their Airflow DAG and task, so "the EMR bill is high" became "this pipeline costs this much".
  • ComputeSpark on spot, ARM64 Karpenter node pools, and native multi-arch builds in the shared Jenkins library instead of QEMU emulation.
  • Right-sizingThe ElastiCache fleet moved to Valkey in batches, OpenSearch and cache nodes downsized, Superset and Airflow tuned to real load.
  • Build vs buyEvaluated managed autoscaling platforms against Karpenter before committing to one.
  • Early warningBuilt aws-cost-anomaly-alert so spend spikes reach Slack the day they start.

App platform, reliability & automation

eks · istio · gitops · jenkins
  • App clusterRan the production app EKS and its services next to the data platform: onboarding with Helm, ArgoCD, IRSA and CSI secrets, plus probes, autoscaling and replicas spread across nodes.
  • Service meshIntroduced Istio to make service-to-service traffic more reliable and easier to see, and used its zone-aware routing to cut inter-AZ cost.
  • UptimeHardened services against restart storms and slow shutdowns, tuned connection pools, and wrote runbooks for the issues that kept coming back.
  • ProvisioningInfra as reviewed Terragrunt PRs with reusable modules, so a new database, queue or cache was a merge, not a ticket.
  • AccessA Jenkins pipeline for AWS Identity Center access by resource name, with approvals, expiry and automatic clean-up.
  • Recurring fixesSelf-service jobs for pod restarts, DLQ redrives, heap profiles and one-off cluster commands, so repeat issues stopped needing DevOps.

Team release: OTP reliability & cost

aug 2026 · shared credit

OTP volume was far out of line with daily active users. We put bot detection on the public send-OTP API in monitor mode first, confirmed most of the traffic wasn't human, then switched it to block. I was one of five engineers credited in the release, and I helped on the WAF side with the bot-detection rules in front of the API.

  • WAF bot detection in front of the operator, rolled out monitor → block
  • Tighter rate limits on the OTP APIs
  • A multi-operator retry chain, so one failing route falls through to the next
  • WhatsApp as a second OTP channel
~82%of send-OTP requests were bots, now blocked with no impact on real users
~₹1.35 cr/yrannualised savings for the team's release
90% → 97%OTP delivery rate
96%of OTPs delivered within 5 seconds
postmortems · abridged

Incidents I worked on

The ones worth retelling are the failures that didn't look like their cause. Blameless, shortened, and scrubbed of internal names and numbers.

DNSkubernetes

Unrelated services failing together

Impact
Database connections, GitOps sync and several services degraded at the same time.
Cause
One misbehaving pod made its clients retry nonstop, which saturated cluster DNS for everyone.
Fix
Fixed the workload, then pushed for node-local DNS caching to shrink the blast radius next time.
DBpostgres

High CPU with no traffic spike

Impact
Queries slowed down across the board.
Cause
Catalog statistics bloated after a large bulk delete, so query planning itself became the bottleneck.
Fix
Cleaned up the bloat instead of scaling the instance up.
IAMgo operator

Workloads losing AWS access

Impact
New tenant environments couldn't assume their IAM role.
Cause
A shared IRSA role's trust policy grew with every tenant until it hit IAM's size limit.
Fix
Collapsed entries into one wildcard per service account and added an operator sweeper with a grace period, dry-run first.
CDCkafka connect

Connectors failing after a cert rotation

Impact
Some change-data streams stopped with TLS handshake errors.
Cause
A database CA change the connector truststore didn't recognise.
Fix
Targeted config on the failing connectors only, with TLS kept on; the lasting fix is shipping the CA bundle in the image.
SCALEautoscaling

Intermittent 503s from a core service

Impact
A latency-critical dependency returned errors under normal load.
Cause
A vertical autoscaler cut its CPU request based on average use, so it throttled at peak.
Fix
Reverted, and kept latency-critical services out of automatic right-sizing until load-tested.
PERFapplication

Tail latency on one busy endpoint

Impact
Repeated P99 breaches and pressure on a shared cache.
Cause
An N+1 pattern: records were enriched before pagination, multiplying downstream calls.
Fix
Proposed: paginate first, batch the lookups, add bounded parallelism, and fold earlier partial fixes into one change.
changelog

Where I've been on call

Allen Digital09/2024 – 09/2026

Senior DevOps Engineer · App & Data Platforms, Observability, Cost

Data platform

  • Built the Kafka Connect / MSK CDC and event platform into ClickHouse across three EKS clusters.
  • Ran Airflow, Spark on EMR-on-EKS, OpenMetadata (1.9 → 2.0), Superset and a new Airbyte install at ~99.9% uptime.

App platform

  • Managed the production app EKS cluster and its services alongside the data platform, with reliability and uptime as the job.
  • Introduced the Istio service mesh for more reliable, observable service traffic and zone-aware routing.

Reliability

  • Defined SLIs/SLOs and error budgets; on-call owner from source DB to warehouse load.
  • Zero-downtime VPC consolidation, Control Tower migration and OpenSearch 2.19 → 3.2 upgrade.

Automation

  • Automated infra provisioning, AWS access requests and the day-to-day fixes that kept recurring: pod restarts, DLQ redrives, heap profiles.
  • Terragrunt modules and PR-driven plans so new infra was a reviewed change, not a ticket.

Cost

  • OpenSearch $80k → $35k/yr; Spark $96k → $45k/yr; RIs + Savings Plan ~$30k/yr.
  • Zone-aware routing cut inter-AZ transfer ~40%; Groundcover ingestion −20%+.

Hiring

  • Technical panelist for 37+ candidates across three DevOps levels, up to Lead.
VMware08/2023 – 08/2024

Site Reliability Engineer

  • Operated production Kubernetes clusters running 1,000+ vCPUs.
  • Designed Gatling load and performance test plans to find bottlenecks before customers did.
  • Built Dynatrace and Wavefront dashboards and alerts that shortened detection and response.
  • On-call rotation within SLA; CVE triage and remediation; Jenkins CI/CD pipelines.
PharmEasy01/2022 – 08/2023

Site Reliability Engineer

  • Cut AWS costs ~25% through right-sizing, reserved instance planning and cleanup.
  • Wrote an in-house Python deployment bot that removed manual release steps.
  • Ansible + Terraform for on-demand environments, including load-test and monitoring stacks.
  • Ran Prometheus-based observability on Kubernetes; wrote the team's onboarding docs.
phData07/2019 – 01/2022

DevOps Engineer

  • Managed Hadoop/YARN clusters with thousands of nodes, 40+ PB and tens of thousands of daily jobs.
  • Operated 400+ node Cloudera clusters for a Mastercard fintech account through incidents and upgrades.
  • Spark, HDFS, Hive, Impala, Kafka, BDR and CDSW; capacity planning on multi-tenant clusters.
  • 24×7 on-call; Ansible automation to standardise cluster configuration.
side projects

Tools I built because I needed them

stack

What's in the toolbox

# akash.yaml
cloud:          [AWS, EKS, VPC, IAM, KMS, S3, MSK, Kinesis, Lambda, EC2, ELB, ElastiCache, Control Tower]
orchestration:  [Kubernetes, Helm, Docker, Karpenter]
iac_and_ci:     [Terraform, Ansible, Jenkins, ArgoCD, Atlantis, GitOps]
data_streaming: [Kafka, Kafka Connect, CDC, Airflow, Airbyte, Spark, Cloudera Hadoop, ClickHouse, OpenSearch]
datastores:     [MySQL, PostgreSQL, MongoDB, Redis, Valkey, OpenMetadata]
observability:  [Prometheus, Grafana, Groundcover, Dynatrace, Wavefront, Gatling]
sre_practice:   [SLIs/SLOs, error budgets, incident command, capacity planning, on-call ownership]
languages:      [Go, Python, Java, Bash]
education:      B.Tech CSE, Babu Banarasi Das University (2015–2019)
page me (politely)

Want to talk reliability?

Happy to swap notes on SLOs, Kafka, Kubernetes or trimming a cloud bill. LinkedIn is the best way to reach me.