A real-life problem, solved with data engineering, APIs, Python, ClickHouse, and Grafana.

Why I Built This Project

Like most people in India, a large part of my daily spending happens via UPI — small but frequent transactions across multiple merchants, apps, and bank accounts.

While banks and UPI apps show recent transactions, I felt a few gaps in my day-to-day life:

• No single place to see daily spend trends

• No easy way to answer:

  • Where did my money go today?

  • Which merchant do I spend the most on?

• No analytics, just raw transactions

• No historical view that felt visual and actionable

At the same time, I wanted to:

• Solve a real problem from my daily life

• Build something production-like

• Use modern data + infra tooling

• Learn by actually running the system end-to-end

• That’s how the UPI Daily Spend Tracker was born.

High-Level Architecture

At a high level, the system looks like this:

And everything runs on my self-hosted home server (An old intel Mac running 24*7 with the help of Amphetamine app), secured with HTTPS and can only be accessed using Tailscale VPN

Exposing the Project Securely (DuckDNS + HTTPS)

Since this runs on my home server, the first challenge was safe external access.

What I did:

• Used DuckDNS to get a stable DNS name for my home server

• Set up Nginx as a reverse proxy

• Enabled HTTPS using Let’s Encrypt

• Used Tailscale VPN

This gave me:

• A proper DNS name instead of IPs

• Encrypted access to Grafana

• A setup very similar to real-world deployments

This step alone taught me a lot about:

• TLS termination

• Reverse proxying internal services

• Running production-like infra at home

Authenticating with Gmail (Google OAuth)

All UPI alerts land in my Gmail inbox, so Gmail became my source of truth.

Challenges:

• Gmail APIs require OAuth

• Tokens expire

• Refresh tokens must be handled carefully

What I implemented:

• Google OAuth flow to generate:

  • Access token

  • Refresh token

• Securely stored the refresh token

• Automatically refreshed access tokens when needed

Once this was in place, the system could:

• Access Gmail programmatically

• Run without manual intervention

Fetching Only Relevant UPI Emails

Instead of scraping everything, I kept it clean.

Strategy:

• Created Gmail labels for UPI transaction alerts

• Used Gmail API queries to fetch:

  • Only labeled emails

  • Only new emails since last run

This made the ingestion:

• Efficient

• Predictable

• Idempotent

Parsing Emails into Structured Data

UPI alert emails are semi-structured, not clean JSON.

So the parser extracts:

• Transaction amount

• Merchant name

• Account / bank

• Timestamp

• Transaction type (debit / credit)

This part was surprisingly fun — and very realistic:

• Regex + heuristics

• Handling different bank formats

• Guarding against partial failures

At the end of this step, every transaction becomes a clean event.

Storing Data in ClickHouse

For analytics, I chose ClickHouse.

Why ClickHouse?

• Columnar

• Extremely fast for aggregates

• Perfect for time-series analytics

• Very low operational overhead

Schema highlights:

• Date

• Merchant

• Amount

• Account

• Transaction direction

Once data landed here, queries like:

• “Total spend today”

• “Top merchants this month”

• “Daily spend trend”

became trivial and fast.

Visualizing with Grafana

This is where everything comes together. Dashboards include:

• Daily spend bar chart

• Total spend today

• Month-to-date spend

• Merchant-wise breakdown

• Drill-down Feature (My Favorite Part)

Clicking on a day lets me:

• Drill down into per-merchant spend

Instantly answer:

• Where did my money go on this specific day?

This makes the dashboard interactive, not just pretty.

What the Dashboard Achieves; With one glance, I can now: Track spending patterns, Identify high-frequency merchants, Notice anomalies immediately ,Make more conscious spending decisions. And most importantly: I actually use this dashboard daily.

Daily 9:15 PM Spend Summary Email

Dashboards are useful — but they require you to open them. So I added a daily automated spend summary email that goes out every night at 9 PM to me and my wife. A scheduled job: Queries ClickHouse for total spend of the day, Generates a merchant-wise breakdown, Sends a clean summary via Gmail API. This turned the system from a passive dashboard into a proactive financial awareness tool. Now, instead of checking manually, we automatically get a concise snapshot of our daily spending.

Work in Progress: Available Balance via Kafka

One feature I’m actively building: Real-time available balance tracking, balance change events produced to Kafka, Consumers update balance state, Grafana panel shows live available balance. This introduces: Event-driven architecture, Streaming instead of batch, Stateful processing

Still WIP — but intentionally so.

What I Learned from This Project

This project touched almost every layer:

• OAuth & APIs

• Secure networking (DNS, TLS, reverse proxy)

• Data ingestion pipelines

• Parsing messy real-world data

• Columnar analytics databases

• Visualization & drilldowns

• Event-driven design (Kafka)

But the biggest takeaway:

The best projects come from solving your own problems. This was built because I needed it — and that made all the difference.

During peak traffic hours on our production MySQL primary (prod-shard3-db01), we started observing a sudden spike in query latency across multiple application services.
SHOW PROCESSLIST revealed that most active queries were in the states:

|Opening tables | SELECT ... FROM tbl_abc ...... | 
|Opening tables | SELECT ... FROM tbl_lmn ...... |
|Closing tables | SELECT ... FROM tbl_xyz ...... |

and they were staying there unusually long — in some cases over several seconds.
Application performance degraded sharply, with slow responses and intermittent timeouts.

🔍 First Observation: Table Cache Thrashing

Our PMM dashboard “MySQL Table Open Cache Status” confirmed the suspicion —
the Table Open Cache hit ratio dropped to 10%, while misses due to overflows spiked sharply (~60K ops/sec).

In simple terms, MySQL was unable to reuse already-opened tables because its table_open_cache was too small.
As a result, every query was opening and closing table descriptors repeatedly — a heavy overhead, especially under thousands of concurrent connections.

⚙️ Step 1 — First Fix: Increase table_open_cache from 400 → 3000

We initially noticed that the server had the following value:

mysql> SHOW GLOBAL VARIABLES LIKE 'table_open_cache';
+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| table_open_cache   | 400   |
+--------------------+-------+

We increased it to 3000:

SET PERSIST table_open_cache = 3000;

Almost immediately, the “Opening tables” entries in the processlist began to disappear and most queries started executing normally again.

However, in PMM we could still see cache misses and a few overflow spikes, indicating that MySQL was still occasionally running out of cached table descriptors during high load.

⚙️ Step 2 — Final Fix: Increase table_open_cache to 8192

We further bumped the value to 8192:

SET PERSIST table_open_cache = 8192;

This completely stabilized the cache behavior — the Table Open Cache Hit Ratio went to 100%, and misses and overflows dropped to zero.

Application latency normalized immediately, and the MySQL Opened_tables status variable remained steady verfied from

show global status like 'opened_tables'

🧐 But Why Was table_open_cache So Small Initially?

Here’s where it got interesting.

When we originally had attempted to configure in the my.cnf file below values few weeks back:

table_open_cache = 40000
max_connections  = 10000

MySQL refussed it and logged below warnings in the error log

2025-10-01T20:38:20.318797Z 0 [Warning] [MY-010140] [Server] Could not increase number of max_open_files to more than 10000 (request: 386641)
2025-10-01T20:38:20.318801Z 0 [Warning] [MY-010141] [Server] Changed limits: max_connections: 9190 (requested 10000)
2025-10-01T20:38:20.318804Z 0 [Warning] [MY-010142] [Server] Changed limits: table_open_cache: 400 (requested 40000)

The root cause?

The OS-level open file limit for the MySQL service (mysqld) was capped at 10,000. Since each table and connection consumes file descriptors, MySQL could not honor the higher cache and connection settings, and it automatically reduced table_open_cache down to 400.

root@prod-shard3-db01:/home/pankaj# pid=$(pidof mysqld)
root@prod-shard3-db01:/home/pankaj# cat /proc/$pid/limits | awk 'NR==1 || /open files/'
Limit                     Soft Limit           Hard Limit           Units     
Max open files            10000                10000                files

⚠️ Discovery: The “LimitNOFILE=Infinity” That Never Took Effect

While reviewing configuration, we found that a systemd override had already been created at:

/etc/systemd/system/mysql.service.d/override.conf

with the content:

[Service]
LimitNOFILE=Infinity

At first glance, this should have allowed MySQL to open unlimited file descriptors. That’s when we realized the catch: the MySQL service was never restarted after adding the override file. The LimitNOFILE=Infinity directive was not yet applied to the running systemd service. We verfied this by comparing mysql uptime (mysqladmin status), with the commit timestamp for the change made to the override file.

🔄 Step 3 — Restart to Apply the Systemd Override

After restarting the MySQL service during a maintenance window:

systemctl restart mysql

The new limit took effect successfully:

cat /proc/$(pidof mysqld)/limits | grep "open files"
Max open files: 1048576

Now the OS-level limit comfortably supported our desired table_open_cache and max_connections values.

🧠 Summary

This incident reinforced how MySQL performance tuning is as much about the OS layer as it is about MySQL itself.

A small table_open_cache combined with a tight open_files_limit caused thousands of queries to waste time repeatedly opening and closing tables. A few simple configuration adjustments — increasing table_open_cache and correctly applying the LimitNOFILE=Infinity override — brought the system back to full health.

Original Setup: pt-heartbeat as a Systemd Service

We had pt-heartbeat running as a systemd service on both primary and replica MySQL servers to monitor replication lag.

On the primary, it runs in update mode:

/usr/bin/pt-heartbeat \
  --database=percona --table=heartbeat \
  --sentinel=/tmp/pt-heartbeat-sentinel \
  --log=/var/log/mysql/pt-heartbeat.log \
  --interval=0.01 --update

On replicas, it runs with –monitor.

We defined the log path explicitly using the –log flag and rotated logs using /etc/logrotate.d/mysql, which included a wildcard:

/var/log/mysql/*.log /var/log/mysql/*.err {
    ...
    compress
    size 100M
    rotate 5
    ...
}

The Issue: Disk Space Bloat Due to Orphaned File Descriptors

Over time, we started getting disk space alerts on the root volume (/) of some DB nodes, even though:

/var/log/mysql/pt-heartbeat.log showed 0 bytes, and the .gz rotated files were regularly compressed.

root@prod-mysql-01:/home/pakumar# ls -lah /var/log/mysql/pt-heartbea*

-rw-r--r-- 1 root root 0 Aug 20 00:00 /var/log/mysql/pt-heartbeat.log 
-rw-r--r-- 1 root root 39M Aug 19 23:59 /var/log/mysql/pt-heartbeat.log.1.gz 
-rw-r--r-- 1 root root 56M Jul 4 00:00 /var/log/mysql/pt-heartbeat.log.2.gz 
-rw-r--r-- 1 root root 49K May 22 09:26 /var/log/mysql/pt-heartbeat.log.3.gz 
-rw-r--r-- 1 root root 29M May 20 00:00 /var/log/mysql/pt-heartbeat.log.4.gz 
-rw-r--r-- 1 root root 1.8M Jul 31 2024 /var/log/mysql/pt-heartbeat.log.5.gz

Despite this, a df -h / showed >90% usage, and we noticed space was only freed when we manually restarted the pt-heartbeat service:

systemctl restart pt-heartbeat.service

Root Cause

When logrotate rotates pt-heartbeat.log, it:

• Renames the current log file to .log.1

• Creates a new, empty .log

• Compresses old logs as .gz

However, pt-heartbeat doesn’t reopen the log file automatically. It keeps writing to the old file descriptor — which is no longer linked to a filename on disk.

This results in a classic orphaned inode situation:

• File is “deleted” (renamed and compressed)

• But space is still consumed until the writing process (pt-heartbeat) is restarted

The Fix: Dedicated Logrotate + Systemd Restart

To solve this cleanly and permanently, we made two key changes:

1. Created a separate logrotate config for pt-heartbeat

We moved its log handling out of the MySQL block:

/etc/logrotate.d/pt-heartbeat
bash

```bash
/var/log/mysql/pt-heartbeat.log {
    notifempty
    size 100M
    rotate 7
    compress
    missingok
    copytruncate
    sharedscripts
    postrotate
        if systemctl is-active --quiet pt-heartbeat-primary.service; then
            systemctl restart pt-heartbeat-primary.service
        fi
        if systemctl is-active --quiet pt-heartbeat-replica.service; then
            systemctl restart pt-heartbeat-replica.service
        fi
    endscript
}

1. Excluded pt-heartbeat.log from the MySQL logrotate config

We updated /etc/logrotate.d/mysql:

/var/log/mysql/*.log
! /var/log/mysql/pt-heartbeat.log
/var/log/mysql/*.err

This prevents accidental double-rotation or compression of the same file.

Outcome

After implementing the fix:

• Disk usage immediately dropped from 91% to 34%

• pt-heartbeat.log is now rotated cleanly

• Old log file descriptors are released after every rotation

• No more manual restarts or space alerts

Takeaways

• Long-running services like pt-heartbeat won’t reopen logs unless restarted.

• Logrotate must coordinate with service restarts, especially for tools that don’t support SIGHUP-based log reopening.

• Exclude such logs from broader rotation scripts (like mysql.log) to prevent side effects.

After upgrading some of our production servers to Ubuntu 24.04 (Noble), we began noticing unexpected MySQL restarts during routine package updates.
This was extremely concerning because these restarts happened without any DBA action and caused disruption on live workloads.

On investigation, we found the culprit: a change in the behavior of the needrestart package.

In Ubuntu 24.04, needrestart now restarts services automatically during apt upgrade or unattended-upgrades. Previously, in non-interactive mode, it only reported which services needed restarting but did not actually restart them. This change was meant to improve patch hygiene, but for production databases, it’s dangerous—because MySQL can be bounced in the middle of heavy traffic.

Symptoms We Observed

Here’s what we saw in the MySQL error log:

2025-09-23T06:03:25.902608Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>.
Shutting down mysqld (Version: 8.0.43-34).

There was no corresponding high load, crash, or OOM event.
Instead, in /var/log/messages we saw systemd/dbus activity triggered during package operations:

2025-09-23T06:03:25.424042+00:00 mysql-prod-shard dbus-daemon[2045]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.297' (uid=0 pid=1227540 comm="/usr/bin/gdbus call --system --dest org.freedesktop.PackageKit" label="unconfined")

This confirmed that it was not an internal MySQL crash but an external restart trigger.

We later saw similar patterns across multiple servers whenever unattended upgrades applied security patches.

The Root Cause

• Ubuntu 24.04 ships needrestart with new defaults.
• The /etc/apt/apt.conf.d/99needrestart file included the -m u flag, which forces automatic restarts of affected services during upgrades.
• MySQL was one of those services.

The Fix We Applied

We restored the old behavior by editing:

File: /etc/apt/apt.conf.d/99needrestart

Before

DPKg::Post-Invoke { "if [ -x /usr/sbin/needrestart ]; then /usr/sbin/needrestart -m u; fi"; };

After

DPKg::Post-Invoke { "if [ -x /usr/sbin/needrestart ]; then /usr/sbin/needrestart; fi"; };

This disables the forced auto-restart and brings back the safer reporting-only behavior.

Recommendations

• If you run production databases on Ubuntu 24.04, be aware of this needrestart change.
• Either:
  • Remove the -m u flag (global rollback to old behavior), or
  • Configure /etc/needrestart/conf.d/ overrides to exempt critical services like mysql.service.
• Avoid unattended restarts on DB servers—schedule controlled maintenance windows for patching.

Closing Thoughts

What looked like random MySQL crashes turned out to be a silent policy change in Ubuntu 24.04.
Lesson learned: **always review the upgrade notes and the features introduced before doing an OS upgrade for your critical services

This experience reinforced our principle:
Production DBs must never be restarted automatically by the OS.

Have you faced something similar on Ubuntu 24.04? Let me know in the comments.

Step 1: Spotting the Symptom

Grafana first alerted me that something was off:

The utilization graph climbed steadily until it flatlined at 100%. Something inside MySQL was hammering the disk.

Step 2: OS-Level Debugging

I logged into the Ubuntu server and started with iostat:

iostat -x 1 5

The output showed:

• %util on nvme1n1 / dm-0 → 99-100%

• Very high writes/sec (~1000+) with tiny block sizes (~5 KB)

• Almost no reads happening

This pointed to frequent small write flushes saturating the disk, not big sequential reads/writes.

Step 3: Looking Inside MySQL

Next, I turned to MySQL internals. I checked the InnoDB log flush setting:

SHOW VARIABLES LIKE 'innodb_flush_log_at_trx_commit';

It was set to 1 (the default), which meant:

• InnoDB writes and flushes the redo log to disk at every single transaction commit.
• Guarantees durability (ACID compliance).
• But with lots of small transactions, this results in one fsync per commit → death by IOPS.

This perfectly explained the 5 KB writes and the saturated disk.

Step 4: The Fix

I changed the parameter to 2:

SET persist innodb_flush_log_at_trx_commit = 2;

With this setting, InnoDB still writes logs at every commit but only flushes them to disk once per second.

Result?

Disk utilization plummeted from 100% to almost idle instantly.

Step 5: Why This Works

Here’s the difference:

=1 → Every commit = redo log write + fsync → extremely high IOPS, guaranteed durability.

=2 → Redo log write per commit, fsync once per second → huge IOPS savings, but risk of losing up to 1 second of transactions in case of a crash.

Final Thoughts

This was a good reminder that MySQL’s default settings are tuned for safety, not performance. Knowing how InnoDB’s redo log flushing works, and when to adjust it, can save you from unnecessary I/O bottlenecks.

In my case, simply switching innodb_flush_log_at_trx_commit from 1 to 2 allowed me to bring disk utilization down and continue stress testing without hitting hardware limits.

Goal

Build a script to:

• Take a full backup of MySQL using xtrabackup
• Split and upload it to Amazon S3 in parallel
• Restore the backup to a fresh RDS instance
• Set up GTID-based replication to start syncing with the on-prem master

Pre-requisites

1. AWS CLI Setup

    aws configure
   

2. IAM Role for RDS Ingestion
   • Create a role with AmazonS3ReadOnlyAccess
   • Add this trust policy:

      {
     "Effect": "Allow",
     "Principal": { "Service": "rds.amazonaws.com" },
     "Action": "sts:AssumeRole"
      }
     

3. MySQL Backup User

    CREATE USER 'xtrabackup_user'@'%' IDENTIFIED BY '*****';
    GRANT SELECT, RELOAD, LOCK TABLES, PROCESS, REPLICATION CLIENT, REPLICATION SLAVE, BACKUP_ADMIN ON *.* TO 'xtrabackup_user'@'%';
   

4. GTID Mode Enabled Make sure both RDS and your MySQL source have:

    gtid_mode = ON
    enforce_gtid_consistency = ON
    log_slave_updates = ON
   

The Script

#!/bin/bash
set -euo pipefail

MYSQL_USER="xtrabackup_user"
MYSQL_PASSWORD="****"

S3_BUCKET="your-s3-bucket"
AWS_REGION="us-west-2"
BACKUP_TIMESTAMP="$(date +%F_%H-%M-%S)"
S3_PREFIX="dr-backup"

LOG_DIR="/mnt/PERCONA-BACKUPS/dr_backup"
LOG_FILE="$LOG_DIR/xtrabackup_stream_${BACKUP_TIMESTAMP}.log"
THREADS=12

RDS_INSTANCE_IDENTIFIER="dr-mysql-replica"
RDS_MASTER_USER="root"
RDS_MASTER_PASSWORD="*****"
RDS_ENGINE="mysql"
RDS_ENGINE_VERSION="8.0.41"
RDS_DB_CLASS="db.t4g.large"
RDS_ALLOCATED_STORAGE=200
RDS_INGESTION_ROLE_ARN="arn:aws:iam::<account>:role/<role-name>"
VPC_SECURITY_GROUP_IDS=("sg-123" "sg-456")
DB_SUBNET_GROUP_NAME="rds-subnet"
OPTION_GROUP_NAME="default:mysql-8-0"
DB_PARAMETER_GROUP_NAME="your-param-group"

ONPREM_HOST="10.0.0.10"
ONPREM_PORT=3306
REPL_USER="repl"
REPL_PASSWORD="*****"

exec >> "$LOG_FILE" 2>&1

echo "[1] Taking backup and splitting"
xtrabackup --backup --user="$MYSQL_USER" --password="$MYSQL_PASSWORD" --stream=xbstream \
  --parallel=$THREADS --target-dir=$LOG_DIR | \
  split -d --bytes=10240MB - $LOG_DIR/backup.xbstream

echo "[2] Uploading parts in parallel"
find $LOG_DIR -type f -name 'backup.xbstream*' | \
  xargs -n 1 -P 8 -I {} bash -c 'aws s3 cp "{}" "s3://'"$S3_BUCKET"'/'"$S3_PREFIX"'/" --sse AES256 --region '"$AWS_REGION"

echo "[3] Triggering RDS restore"
aws rds restore-db-instance-from-s3 \
  --db-instance-identifier "$RDS_INSTANCE_IDENTIFIER" \
  --allocated-storage "$RDS_ALLOCATED_STORAGE" \
  --db-instance-class "$RDS_DB_CLASS" \
  --engine "$RDS_ENGINE" \
  --engine-version "$RDS_ENGINE_VERSION" \
  --master-username "$RDS_MASTER_USER" \
  --master-user-password "$RDS_MASTER_PASSWORD" \
  --s3-bucket-name "$S3_BUCKET" \
  --s3-ingestion-role-arn "$RDS_INGESTION_ROLE_ARN" \
  --s3-prefix "$S3_PREFIX" \
  --source-engine "$RDS_ENGINE" \
  --source-engine-version "$RDS_ENGINE_VERSION" \
  --region "$AWS_REGION" \
  --db-subnet-group-name "$DB_SUBNET_GROUP_NAME" \
  --vpc-security-group-ids "${VPC_SECURITY_GROUP_IDS[@]}" \
  --option-group-name "$OPTION_GROUP_NAME" \
  --db-parameter-group-name "$DB_PARAMETER_GROUP_NAME"

echo "[4] Waiting for RDS to become available..."
while true; do
  STATUS=$(aws rds describe-db-instances \
    --db-instance-identifier "$RDS_INSTANCE_IDENTIFIER" \
    --region "$AWS_REGION" \
    --query "DBInstances[0].DBInstanceStatus" \
    --output text 2>/dev/null || echo "not-found")
  echo "  ➤ $STATUS"
  case "$STATUS" in
    "available") break ;; 
    "failed"|"incompatible-restore"|"not-found") exit 1 ;; 
    *) sleep 30 ;;
  esac
done

RDS_ENDPOINT=$(aws rds describe-db-instances \
  --db-instance-identifier "$RDS_INSTANCE_IDENTIFIER" \
  --region "$AWS_REGION" \
  --query "DBInstances[0].Endpoint.Address" \
  --output text)

echo "[5] Setting up replication"
mysql -h "$RDS_ENDPOINT" -u "$RDS_MASTER_USER" -p"$RDS_MASTER_PASSWORD" <<EOF
CALL mysql.rds_set_external_master_with_auto_position('${ONPREM_HOST}', ${ONPREM_PORT}, '${REPL_USER}', '${REPL_PASSWORD}', 0, 0);
CALL mysql.rds_start_replication();
EOF

Key Considerations

• The backup must not be compressed (–compress is not supported by RDS restore)

• Use –slave-info to capture GTID state

• Your RDS parameter group must have gtid_mode=ON

• Use split with –bytes=10240MB to avoid multipart size limits on S3

• Uploading via xargs -P 8 ensures efficient concurrency

Summary

This setup helped us streamline disaster recovery for MySQL into Amazon RDS. The script is production-ready and can be scheduled via cron or integrated into a backup orchestration workflow.

But come Monday morning, we were greeted with a nasty surprise:

Disk space alert : /mnt/DATABASE usage > 90%

A quick investigation revealed the root cause:

The audit log file had grown to 2.1TB in just under 48 hours!

What Went Wrong

The promoted replica had Percona’s Audit Log plugin enabled with the following configuration:

audit_log_policy = ALL
audit_log_handler = FILE
audit_log_rotate_on_size = 0
audit_log_strategy = ASYNCHRONOUS

Let me break this down:

As a replica, this configuration didn’t cause problems — replicas have minimal write traffic.
But once promoted to primary, it started logging every single query from our read-heavy production workloads.

And since rotation was disabled, the log just kept growing… until it ate 2.1TB of disk space.

How We Fixed It (Without Restarting MySQL)

1. Confirmed the audit log file path

mysql -e "SELECT @@datadir;"

The actual file path turned out to be:

/mnt/DATABASE/mysql/audit.log

(with a symlink from /var/lib/mysql/audit.log)

2. Verified that MySQL was still writing to it

sudo lsof /mnt/DATABASE/mysql/audit.log

This showed that the mysqld process still had the file open, so we couldn’t delete it directly.

3. Safely truncated the log file in place

sudo truncate -s 0 /mnt/DATABASE/mysql/audit.log

This instantly freed up 2.1TB without interrupting MySQL.

Since we were using audit_log_strategy = ASYNCHRONOUS, the server didn’t mind — it just kept writing into the now-empty file.

4. (Optional) Rotated logs manually

To force MySQL to start a new log file immediately:

FLUSH AUDIT_LOGS;

How We Prevented This from Happening Again

Enabled audit log rotation:

SET GLOBAL audit_log_rotate_on_size = 1073741824;  -- Rotate at 1GB
SET GLOBAL audit_log_rotations = 5;               -- Keep 5 rotated logs

Added it permanently to /etc/my.cnf:

[mysqld]
audit_log_rotate_on_size = 1073741824
audit_log_rotations = 5

Reduced audit verbosity (optional)

If not all queries need to be logged, consider switching:

SET GLOBAL audit_log_policy = LOGINS;  -- Only log login events

Takeaways

This incident was a great reminder that promoting a replica to primary is not just a replication config change — it comes with a shift in operational responsibilities, including:

• Disk I/O
• Query volume
• Log generation
• Backup behaviors

Always check:

Bonus: Truncate Log File via Cron (Failsafe)

Until you’re confident with log rotation, this cron can be your insurance:

0 3 * * * /usr/bin/truncate -s 0 /mnt/DATABASE/mysql/audit.log

Final Thoughts

We averted a major production impact because we caught it early Monday — but in a different scenario, 2.1TB of audit logs could’ve taken the server offline.

Always treat a replica promotion as a full-blown cutover — not just a read_only = 0.

In this post, I’ll walk you through how I did a MySQL primary promotion using GTIDs. Below is our simple replication topology:

• One primary MySQL instance: prod-db01
• Four replicas: prod-db02, replica-db03, replica-db04, replica-db05 (each potentially having errant GTIDs due to historical promotion/demotion cycles or unclean state)

What is GTID-Based Replication?

GTID (Global Transaction Identifier) ensures each transaction is uniquely identified and executed only once across the replication topology. This helps simplify failover and promotion logic.

Each GTID is a combination of server_uuid:transaction_id, and MySQL keeps track of GTID_EXECUTED and GTID_PURGED sets to manage consistency.

Why Errant GTIDs are a Problem

An errant GTID is a GTID that exists on a replica but not on the current primary. If such a GTID exists, that replica cannot connect to the new primary (after promotion) unless it also has that GTID.

When promoting a new primary, all replicas must have only the GTIDs that also exist on the new primary (or a subset of them).

Step-by-Step: Promoting a New Primary with GTID

Assume the following GTID sets:

prod-db01  (old primary): Executed GTID set: abc:1-12345, def:1-123
prod-db02             : Executed GTID set: abc:1-12345, def:1-123
replica-db03          : Executed GTID set: abc:1-12345, def:1-123, ghi:1-50
replica-db04          : Executed GTID set: abc:1-12345, def:1-123, ijk:1-100
replica-db05          : Executed GTID set: abc:1-12345, def:1-123, lmn:1-500

We’re choosing prod-db02 as the new primary. This means we need to:

1. Inject the missing GTIDs (ghi:1-50, ijk:1-100, lmn:1-500) as empty transactions into prod-db02.
2. Point the other replicas to this new primary.

Step 1: Inject Errant GTIDs into the New Primary

Create a SQL file inject_gtids.sql:

-- inject_ijk.sql
SET GTID_NEXT='ijk:1'; BEGIN; COMMIT;
SET GTID_NEXT='ijk:2'; BEGIN; COMMIT;
-- ... up to ijk:100
SET GTID_NEXT='AUTOMATIC';

You can automate it with a shell script:

#!/bin/bash
uuid=$1
start=$2
end=$3
for i in $(seq $start $end); do
  echo "SET GTID_NEXT='$uuid:$i'; BEGIN; COMMIT;"
done
echo "SET GTID_NEXT='AUTOMATIC';"

Usage:

./generate_gtid_injector.sh ijk 1 100 > inject_ijk.sql
mysql -uroot -p < inject_ijk.sql

Repeat this for ghi:1-50 and lmn:1-500.

Step 2: Configure Replication on Remaining Replicas

On replica-db03, replica-db04, and replica-db05:

STOP REPLICA;
RESET REPLICA ALL;
CHANGE REPLICATION SOURCE TO 
  SOURCE_HOST='prod-db02',
  SOURCE_USER='replica_user',
  SOURCE_PASSWORD='*****',
  SOURCE_AUTO_POSITION=1;
START REPLICA;

Note: Make sure binary log retention is set appropriately to avoid 1236: missing binary log errors.

Step 3: Validate Replication

Run:

SHOW REPLICA STATUS\G

Ensure:

• Replica_IO_Running: Yes
• Replica_SQL_Running: Yes
• Seconds_Behind_Source: 0

Step 4: GTID Sanity Check

Make sure all nodes have compatible GTID sets:

SHOW GLOBAL VARIABLES LIKE 'gtid_executed';

You can use gtid_subtract() or gtid_subset() to compare sets:

SELECT GTID_SUBTRACT('replica_gtids', 'primary_gtids');

Should return an empty set if the replica is in sync.

Lessons Learned

• Always check for errant GTIDs before promotion
• Keep GTID sets as clean and consistent as possible
• Automate injection of GTIDs for repeatable and safe operations
• Use RESET REPLICA ALL when reconfiguring replicas after promotion
• GTID simplifies failover, but you still need to be careful

Final Words

GTID-based replication brings predictability and reliability to MySQL high availability setups. Proper handling of GTID inconsistencies like errant transactions ensures smooth and safe failover without data loss.

This strategy was tested on a live environment with multi-TB datasets and worked reliably across all replicas.

Let me know your thoughts, questions in the comments

Happy Replicating!