My University Runs Their LMS on AWS. Fixing It Costs Less Than Their Hosting Bill. (Part 2: The Fix)

Disclaimer: The fix recommendations in this post are derived entirely from publicly observable behavior documented in Part 1 and standard best practices for Moodle deployments on AWS. None of this comes from inside access to REVA's actual configuration, codebase, or internal systems documentation. I have no knowledge of what their private infrastructure looks like beyond what passive, non-invasive tools revealed. Cost estimates reflect current AWS pricing in the ap-south-1 region as of mid-2025 and will vary with instance selection and usage patterns. If you're implementing any of this: test everything in a staging environment first, run a proper load test before any exam window, and treat this as a starting framework rather than a copy-paste solution. I have no commercial affiliation with any tool, plugin, or service mentioned here.

Same note as Part 1: if this accurately describes your infrastructure and that makes you uncomfortable, the correct response is to fix the infrastructure.

Part 1 ended at roughly 85% confidence. Three EC2 instances behind a Layer 4 NLB doing blind round-robin. No session affinity, no HTTP/2, no keep-alive. Files almost certainly split across three local disks with no shared storage. A database tier that looks like a single RDS instance with nothing in front of it. Good security layer, performance configuration that has the distinct appearance of Moodle install defaults untouched since the server first went live.

This part is where confidence stops mattering. Every fix below works regardless of whether my guess about the RDS instance type was accurate. The behavioral evidence from Part 1 is enough to work from.

One thing needs to be stated before the list, because it changes what order everything should happen in. Enabling sticky sessions is not the first fix. It might not even be the second. It is a patch on a symptom, and a slightly worse patch than it initially appears.

Why "Just Enable Sticky Sessions" Is the Wrong First Move

The obvious reaction to Part 1 is: turn on source IP affinity on the NLB. AWS supports it natively. It's a target group attribute called stickiness.enabled, type source_ip. Five minutes in the console.

Here's the catch. Source-IP stickiness groups traffic by IP, not by student. REVA's campus WiFi and hostel networks sit behind NAT. A single access point can be sharing one public IP across dozens of students, sometimes more. Turn on source-IP stickiness in that environment and every student on Hostel Block C's WiFi gets pinned to whichever EC2 instance happened to handle the first request from that IP. One server absorbs the entire hostel block's exam traffic. The other two sit comparatively idle. You've traded random session failures for a more predictable, equally bad overloaded-server failure.

The real fix isn't to make routing smarter. It's to make routing irrelevant. Any backend should be able to serve any session, so wherever the NLB sends a request, the answer is identical. That requires getting session state off the individual EC2 instances entirely. Once that's done, the NLB can keep round-robining at will and it genuinely doesn't matter which server handles which request.

That's Redis. And it's what should happen first, before anyone opens the load balancer console at all.

Fix 1: Redis for Sessions

Moodle ships with a Redis session handler. No plugin, no third-party code. It's built in. The entire change is four lines in config.php:

$CFG->session_handler_class = '\core\session\redis';
$CFG->session_redis_host = 'your-elasticache-endpoint.cache.amazonaws.com';
$CFG->session_redis_port = 6379;
$CFG->session_redis_acquire_lock_timeout = 120;

Spin up a single ElastiCache for Redis node. A cache.t3.micro handles session data at this scale without needing cluster mode. Point all three EC2 instances at the same endpoint. The session-bouncing problem from Part 1 stops existing. EC2-A writes a session. EC2-B reads it. EC2-C reads it. Same data, same place, every time.

This also kills the 5-second latency spike from the 3am test. That spike was almost certainly a PHP worker stalling on a database session read under internal contention. A Redis session lookup is sub-millisecond. A MySQL session lookup under connection pressure runs at 10ms to 50ms normally, and effectively infinite when the connection pool is exhausted. Moving sessions off the database removes the thing that was intermittently blocking workers at zero load.

Cost: a cache.t3.micro in ap-south-1 runs around $12 to $15 a month. Effort: half a day, most of which is validating failover behavior and running a smoke test, not actually writing config.

Fix 2: The Free Stuff

Three lines in an Nginx config file that have been missing since install day. No new infrastructure, no new spend.

listen 443 ssl http2;
keepalive_timeout 65;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

HTTP/2 collapses the 20 to 30 separate TCP connections a Moodle login page currently opens per student into a single multiplexed connection. Instead of each browser burning through 20-plus SYN/SYN-ACK/ACK cycles just to load the login form, one connection carries all asset requests simultaneously. This is not a minor optimization at scale. When a few hundred to a thousand students load that login page at the same moment, the difference between 25 TCP connections per student and one is the difference between tens of thousands of connection cycles and a few thousand.

keepalive_timeout 65 stops Nginx from tearing down and rebuilding connections between requests from the same client. The connection: close behavior currently in production means each subsequent browser request restarts from scratch. That is not how HTTP is meant to work in 2025 and it never should have shipped this way on a production system serving concurrent exam traffic.

The HSTS line deserves its own paragraph. The existing max-age=0 header is not a neutral placeholder. It is an active instruction telling every browser that has visited the site to discard any stored HSTS policy for this domain. It is working against the TLS hardening that someone clearly put real effort into. Setting it to one year with subdomain coverage is the standard, takes one config line, and undoes what is currently a self-inflicted security regression.

Cost: zero dollars. Effort: under an hour including a restart and a smoke test.

Fix 3: A Connection Pooler

Redis removes session reads and writes from the database. It does nothing about the other queries a single Moodle login fires at MySQL: credential check, dashboard initialization, enrollment lookups, notification counts, recent activity fetch. Something in the range of 10 to 15 queries per login, all sequential, all hitting the same RDS instance.

Without a pooler, every PHP-FPM worker holds its own direct database connection for the full duration of those queries. The max_connections ceiling on RDS becomes the hard limit on how many students can be mid-login simultaneously. Hit that ceiling and MySQL starts queuing connections internally. PHP workers start waiting on connections that are themselves waiting. That's the cascade from Part 1's crash reconstruction, starting at step 7.

Two options. RDS Proxy is the managed path: point it at the existing RDS instance, update PHP's database connection string to point at the proxy endpoint, and AWS handles connection multiplexing and queuing automatically. Pricing scales with the underlying instance's vCPU count, roughly $25 a month for a db.t3.medium-class instance. ProxySQL on a t3.micro is the self-hosted path: about $8 to $9 a month in EC2 costs, more initial setup, one more thing to patch and monitor, and full control over the pooling behavior.

Both paths produce the same result. The database stops receiving 150-plus simultaneous direct connection attempts during an exam login rush and instead receives a managed pool that queues excess requests in front of MySQL rather than letting MySQL queue them internally while PHP workers sit blocked.

Cost: $9 to $25 a month depending on the path. Effort: roughly a day, mostly testing under simulated load rather than setup work.

Fix 4: Files Off Local Disk

Save this one for last, not because it's less important, but because it's the only fix here that touches data migration rather than just configuration.

Right now, pluginfile.php serves every file download by reading from whichever local disk the handling EC2 happens to have, streaming the bytes through PHP as the HTTP response. That PHP worker is occupied for the full duration of the file transfer. It cannot serve other requests until the download completes. And with three instances and no shared storage, a file uploaded on EC2-A may simply not exist on EC2-B or EC2-C, which means some fraction of downloads are probably already silently failing depending on which server the NLB routes the request to.

The correct architecture: files live in S3. When a student requests a download, PHP generates a pre-signed S3 URL, a time-limited cryptographically signed URL granting temporary access to a specific object, and returns a redirect. The browser downloads directly from S3. PHP never touches the bytes. Workers stay free. There is exactly one copy of every file, in one bucket, reachable identically from all three EC2 instances.

Moodle's ecosystem has a maintained plugin for exactly this: tool_objectfs, built by Catalyst IT. It moves the Moodle file pool to S3 with local caching for frequently accessed files and turns downloads into pre-signed redirects. Install the plugin, configure an S3 bucket, run the migration task, verify nothing broke for in-progress submissions. That accounts for most of the effort here, the migration validation rather than the configuration itself.

Cost: S3 storage and transfer for a university's worth of assignment PDFs lands around $5 to $10 a month at this scale. Effort: two to three days, the bulk of which is the initial migration validation rather than plugin configuration.

Fix 5: Basic Rate Limiting

Lower priority than the first four, but worth doing before the next exam window regardless. AWS WAF with a rate-based rule capping requests per IP, plus one of the managed rule groups covering common bot and scanner traffic. This does not fix the architecture. It prevents a retry storm from making an already-strained system worse. A few hundred browsers auto-retrying a timed-out request simultaneously is a self-inflicted load spike layered on top of legitimate traffic, and without any rate limiting, every one of those retries hits PHP directly with no throttling at the perimeter.

Cost: roughly $10 to $15 a month for a WAF web ACL with a handful of rules at this traffic volume. Effort: half a day.

What the Architecture Should Look Like

┌──────────────────────────────────────────────────────┐
│         Students (hundreds to 1000+ concurrent)       │
└──────────────────────┬────────────────────────────────┘
                       │ HTTPS
                       ▼
┌──────────────────────────────────────────────────────┐
│                  AWS WAF (rate-based rule)             │
└──────────────────────┬────────────────────────────────┘
                       ▼
┌──────────────────────────────────────────────────────┐
│        Network Load Balancer -- round robin            │
│     (sessions no longer live on instances,             │
│      so routing decisions stop mattering)              │
└───────────┬──────────────┬──────────────┬─────────────┘
            ▼              ▼              ▼
    ┌─────────────┐ ┌─────────────┐ ┌─────────────┐
    │   EC2 - A   │ │   EC2 - B   │ │   EC2 - C   │
    │ Nginx HTTP/2│ │ Nginx HTTP/2│ │ Nginx HTTP/2│
    │ keep-alive  │ │ keep-alive  │ │ keep-alive  │
    │ PHP-FPM     │ │ PHP-FPM     │ │ PHP-FPM     │
    └──────┬──────┘ └──────┬──────┘ └──────┬──────┘
           └───────────────┼───────────────┘
                           │
              ┌────────────┴────────────┐
              ▼                         ▼
    ┌──────────────────┐     ┌──────────────────────┐
    │  ElastiCache      │     │   RDS Proxy /         │
    │  Redis            │     │   ProxySQL            │
    │  (sessions)       │     │   (pooled conns)      │
    └──────────────────┘     └──────────┬───────────┘
                                        ▼
                             ┌──────────────────────┐
                             │  MySQL RDS instance   │
                             └──────────────────────┘

File storage: S3 via tool_objectfs, pre-signed downloads

The NLB stays in place. Round-robin is fine now because sessions live in Redis, not on individual EC2 instances. Routing decisions are no longer consequential. That's the whole point of fixing the session layer first.

Don't Trust This Until You've Load Tested It

Every fix above is a well-supported inference from standard Moodle behavior and documented AWS characteristics. Well-supported inferences are not proof, especially for a system where I don't know the actual PHP-FPM worker counts, instance sizes, or real query patterns. The fix recommendations are correct in principle. Whether they're correct for this specific setup at this specific scale needs to be verified, not assumed.

Before any of this touches a real exam window, it needs a synthetic load test that replicates the actual failure mode from Part 1: a login rush, not a gradual ramp.

k6 handles this in a few dozen lines:

import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
  scenarios: {
    exam_rush: {
      executor: 'ramping-vus',
      startVUs: 0,
      stages: [
        { duration: '30s', target: 500 },
        { duration: '2m', target: 500 },
        { duration: '30s', target: 0 },
      ],
    },
  },
};

export default function () {
  http.get('https://rulms.reva.edu.in/login/index.php');
  sleep(Math.random() * 2);
}

Five hundred virtual users ramping up over thirty seconds. That's the shape of an exam cohort hitting login at once, not a steady average. Run it against the current setup first. Watch it fall over the same way it does every semester. Then apply fixes one at a time and run it again after each one lands. If response times hold flat under the same synthetic load that currently kills the portal, the fix worked. If they don't, you find out during a test window rather than during someone's actual exam submission.

The load test is not optional. It's the difference between deploying reasonable fixes and deploying verified fixes.

What This Actually Costs

Call it fifty dollars a month, added on top of whatever they're already paying for three EC2s and an RDS instance in Mumbai. Less than a semester's printing budget. Less than what the canteen probably moves in bread pakodas (I prolly contribute heavily to this) on a slow week.

None of this requires new hardware, a new vendor, or a rebuild from scratch. It requires someone with AWS console access setting aside roughly a week of time, spread across however many days makes sense around everything else the IT team has to handle. The security configuration documented in Part 1 already shows this team knows how to make precise, deliberate infrastructure decisions. The TLS hardening, cipher suite selection, header scrubbing, certificate automation, none of that is accidental. It took real knowledge and deliberate effort. This is the same kind of work, pointed at a different layer of the same system.

The performance layer looks like it got skipped the first time, probably under time pressure, probably without someone explicitly flagging what the exam load would look like in practice. That's how these things happen. But it's also why hundreds of students are sitting through delayed exams every semester on AWS infrastructure that, on paper, should have no trouble with this load at all.

If anyone from the REVA LMS department is reading this, I'll be more than happy to help with setting up the new architechture / patching the existing one, all for the low low cost of covering my attendance for the next semester <3