In the dynamic domain of modern web development, constructing infrastructure that is not only functional but also inherently secure, scalable, and cost-effective is paramount. As a leading web development agency, voronkin.com understands that the foundation of any successful digital product lies in its underlying architecture. Far too often, tutorials simplify complex infrastructure concerns, leading to solutions that might work in a sandbox but crumble under the pressures of a production environment. This article delves into a meticulously engineered AWS architecture, provisioned entirely with Terraform, designed to withstand real-world demands for web applications, offering deep insights into design decisions, trade-offs, and crucial lessons learned.

Our focus here isn't just on deploying a basic application; it's about confronting the intricate challenges that platform teams grapple with daily. This includes determining the optimal number of network tiers, safeguarding sensitive data, establishing secure administrative access, and achieving high availability without incurring prohibitive operational costs. The resulting infrastructure provides a fully isolated, auto-scaling, four-tier network on AWS, supporting a simple goal-tracking application. While the application itself is straightforward, the sophistication lies in its dependable infrastructure, offering a blueprint for modern, resilient web development.

Beyond the Basic 3-Tier Architecture: Addressing Security Gaps

Many introductory guides to AWS infrastructure, particularly those using Terraform, often present a simplified 3-tier model. This typically involves segmenting a Virtual Private Cloud (VPC) into public, private, and database subnets. While this provides a rudimentary separation, it frequently collapses critical security distinctions into a single "private" tier. In such configurations, stateless web servers that handle incoming requests and the application servers responsible for business logic and database interaction often reside within the same private subnet, potentially even sharing security groups.

This common approach introduces a significant security vulnerability. If the web tier, which is often the most exposed component to external threats (even behind a load balancer), were to be compromised, an attacker could potentially gain direct access to the same network segment as the application layer. From there, lateral movement towards the database, which holds critical data, becomes a much more achievable objective. This "blast radius" is far too large for production-grade systems, where minimizing the impact of a breach is a top priority for any software engineering team.

To mitigate this risk, a more granular network topology is essential. The principle is clear: network segmentation should enforce strict access rules, ensuring that only authorized components can communicate with each other. Specifically, the database layer should only be accessible by the backend application layer, and the backend layer should only be reachable by the frontend. This philosophy underpins the adoption of a four-tier network design, where each tier is carefully isolated and duplicated across multiple Availability Zones (AZs) to ensure high availability and fault tolerance. This multi-layered defense strategy significantly enhances the overall security posture of the application.

  • Public Subnets: These subnets are directly exposed to the internet via an Internet Gateway. They host essential components like NAT Gateways for outbound internet access from private subnets, the public-facing Application Load Balancer (ALB) that distributes incoming user traffic, and a bastion host for secure administrative access.
  • Frontend Private Subnets: Dedicated to the presentation layer of the application (e.g., a Node.js Express server), these subnets are not directly accessible from the internet. They can only receive traffic from the public-facing ALB, acting as a crucial intermediary.
  • Backend Private Subnets: Housing the core business logic (e.g., a Go API tier), these subnets are even more isolated. They are unreachable from the public internet and can only receive traffic from an internal ALB, which is, in turn, accessed by the frontend tier. This creates a powerful layer of indirection.
  • Database Isolated Subnets: This is the most secure tier, containing sensitive data stores like RDS PostgreSQL instances. These subnets have no direct route to the internet whatsoever and are strictly configured to accept connections only from the security groups associated with the backend application tier.

This additional layer of segmentation, while seemingly a minor addition in terms of Terraform code (just one more aws_subnet resource, a security group, and an internal ALB), profoundly impacts the system's security. A compromise in the frontend tier, instead of potentially exposing the entire database, is now contained, limiting access to a single internal load balancer on a specific port. This significantly reduces the potential damage and complexity of incident response for development teams.

Strategic Deployment of Dual Application Load Balancers (ALBs)

One of the most distinguishing features of this production-grade architecture is the deliberate use of two Application Load Balancers (ALBs) instead of a single one. This decision, while incurring a slight increase in infrastructure cost (approximately $16-20 per month plus LCU charges per ALB), reflects a mature approach to designing highly available and scalable systems, a common pattern in advanced software engineering practices.

The first ALB is public-facing, residing in the public subnets. Its primary role is to accept incoming internet traffic from users and distribute it efficiently across the instances within the frontend Auto Scaling Group. For instance, it might route browser traffic to the Node.js Express tier on port 3000. This is a standard and expected component of any scalable web application.

The critical difference lies in the interaction between the frontend and backend tiers. Instead of the frontend directly calling backend instances via their private IP addresses or relying on a service discovery mechanism like AWS Cloud Map, it communicates with a second, internal-only ALB. This internal ALB, deployed within the backend private subnets, then load-balances requests across the backend Auto Scaling Group, perhaps targeting the Go API tier on port 8080. This architectural choice offers several compelling advantages that far outweigh the marginal cost increase:

  • Consistent Health-Checking and Load Balancing: By placing both frontend and backend tiers behind ALBs, both layers benefit from the same robust health-checking and load-balancing semantics. This ensures that unhealthy instances are automatically removed from rotation, improving overall application reliability and user experience.
  • Enhanced Decoupling and Scalability: The internal ALB acts as a stable, single entry point for the backend service. This means backend instances can scale up or down, fail, or be replaced without the frontend tier needing to be aware of individual instance IPs or changes. This decoupling is vital for microservices architectures and enables independent scaling and deployment of different application components.
  • Simplified Mental Model: Adopting a consistent pattern—"every tier that has more than one instance sits behind an ALB"—streamlines the architectural understanding and reduces cognitive load for developers and operations teams. It avoids the need for different patterns for horizontally scaling tiers, leading to more predictable behavior and easier troubleshooting.
  • Improved Security Posture: The internal ALB provides an additional layer of controlled access. Security groups can be configured to allow traffic only from the frontend ALB, further restricting direct access to backend instances and enhancing the principle of least privilege.

For large-scale web development projects, these benefits translate directly into greater stability, easier maintenance, and improved agility, making the dual ALB strategy a wise investment for any serious production deployment.

Fortifying Data Security with AWS Secrets Manager, Not Environmental Variables

Managing sensitive information, such as database credentials, API keys, and other application secrets, is a cornerstone of secure web development. A common anti-pattern, particularly in less mature projects, involves baking secrets directly into Docker images, configuration files checked into version control, or passing them as plain environment variables during deployment. These methods introduce significant security risks, making secrets vulnerable to exposure through accidental commits, image layer inspection, or process snooping.

This architecture champions a robust approach using AWS Secrets Manager. Here's how it works:

  1. Secure Credential Generation: The RDS master password is not manually entered. Instead, it's generated once during the Terraform apply process using Terraform's random_password resource. This ensures a strong, unique, and complex password (e.g., 16 characters with a curated set of special characters compatible with Postgres connection strings).
  2. Centralized Secret Storage: The generated password, along with other database connection details (username, host, port, database name), is stored as a single JSON blob within AWS Secrets Manager. This centralizes all related credentials under a unique identifier, such as {environment}-{project}-db-credentials. The backend application then only needs to know one Secret ARN (Amazon Resource Name) to retrieve all necessary information.
  3. On-Demand Secret Retrieval: At instance boot-time, the backend's user-data script executes a command to call aws secretsmanager get-secret-value. This retrieves the secret dynamically. The returned JSON is then parsed using a tool like jq, and the individual fields (username, password, etc.) are securely passed as environment variables to the Docker container running the application. This ensures that secrets are never persisted on disk or within the image itself.
  4. Principle of Least Privilege with IAM: Crucially, the EC2 instance's IAM role is granted only the absolute minimum permissions required. It has exactly one Secrets Manager permission: GetSecretValue and DescribeSecret, and this permission is strictly scoped to the ARN of that specific secret. This prevents the instance from accessing other secrets or performing unauthorized actions within Secrets Manager.

This methodology ensures that no plaintext password ever resides within a Dockerfile, a Docker image layer, or a .env file committed to git. It significantly reduces the attack surface for credential theft, aligning with modern cybersecurity best practices for cloud-native applications.

It's important to acknowledge a specific nuance for transparency: while the password is dynamically retrieved, it initially flows through a Terraform variable (var.db_password). Although this variable is marked as sensitive = true to prevent its display in plan output, it still exists within the Terraform state file. For ultimate security, AWS RDS offers a native feature (manage_master_user_password = true) that allows RDS to generate and rotate its own master password, with Terraform never touching the plaintext. The approach detailed here was chosen initially to provide a hands-on understanding of the full credential lifecycle before abstracting it with native cloud features, a valuable learning experience for any developer delving into infrastructure as code.

Resilient Instance Access: Bastion Hosts and SSM for Operational Excellence

Securely accessing instances for troubleshooting, maintenance, or debugging is another critical aspect of production-grade infrastructure. Relying solely on SSH keys distributed to developers, with port 22 open to the internet, is a significant security risk. This architecture implements a deliberately redundant and highly secure approach, combining the traditional bastion host with AWS Systems Manager (SSM) Session Manager.

Every EC2 instance in this stack—whether it's the bastion itself, a frontend server, or a backend server—is provisioned with the same IAM instance profile. This profile includes the AmazonSSMManagedInstanceCore managed policy. This policy alone is sufficient to allow developers to initiate a secure shell session into any instance using aws ssm start-session --target <instance-id>. The key advantages of this SSM-based approach are:

  • No SSH Keys Required: Eliminates the need to manage, rotate, and distribute SSH keys, significantly reducing administrative overhead and the risk of key compromise.
  • No Open Inbound Ports: Session Manager operates without requiring port 22 (SSH) to be open from the internet or even from a bastion host. Communication occurs over an encrypted channel through the SSM agent, which initiates an outbound connection to the SSM service endpoint. This minimizes the network attack surface.
  • Centralized Access Control and Auditing: Access to instances is controlled via IAM policies, allowing for fine-grained permissions based on user roles. All sessions are logged and auditable, providing a clear trail of who accessed what and when, crucial for compliance and security monitoring.

Given the benefits of SSM, one might question the continued presence of a bastion host. The bastion host, also deployed in the public subnet, serves as a hardened, jump box. While SSM is highly effective, a bastion provides a fallback mechanism and can be particularly useful in scenarios where SSM might be temporarily unavailable or for specific tooling that expects traditional SSH access. What's more, for highly sensitive environments or specific compliance requirements, the bastion can act as an additional layer of defense or a dedicated ingress point for specific network segments. Its presence offers a robust redundancy, ensuring operational continuity even in unexpected circumstances. This dual strategy provides maximum flexibility and resilience for managing instance access, a hallmark of sophisticated DevOps practices.

What This Means for Developers

For web development agencies like voronkin.com, and indeed for any professional developer or project team, the principles demonstrated in this robust AWS architecture are not merely theoretical; they represent a fundamental shift in how we approach client projects. Implementing such an infrastructure means we can deliver solutions that are inherently more secure, scalable, and maintainable from day one. This translates directly into tangible benefits for our clients: reduced risk of data breaches, frictionless scalability to meet growing user demands, and lower long-term operational costs due to efficient resource management and automation. For agencies, adopting these sophisticated Infrastructure as Code (IaC) patterns, particularly with Terraform, becomes a significant competitive differentiator. It allows us to accelerate deployment cycles, offer predictable infrastructure provisioning, and guarantee a higher standard of reliability and security for bespoke web applications, enterprise platforms, and AI-driven solutions.

Individual developers and project teams must recognize that modern web development extends far beyond writing application code. A deep understanding of cloud infrastructure, security best practices, and automation tools like Terraform is no longer optional but a core competency. This architecture emphasizes several key takeaways: prioritize network segmentation and the principle of least privilege from the outset, embrace sophisticated secrets management solutions like AWS Secrets Manager to eliminate credential leakage risks, and design for operational resilience through redundant access mechanisms and comprehensive monitoring. Developers should actively invest in learning advanced Terraform patterns, including modularity and state management, and integrate security considerations into every phase of the software development lifecycle, moving security "left" in the DevOps pipeline.

Concretely, we advise our development teams to standardize on modular Terraform configurations for all new projects, creating reusable modules for common components like VPCs, ALBs, and RDS instances. This promotes consistency and speeds up development. Furthermore, integrating tools like HashiCorp Vault or AWS Secrets Manager should be a non-negotiable part of the initial project setup, ensuring that secrets are never hardcoded or exposed. Finally, regularly reviewing and auditing IAM policies and security group rules is essential to maintain a strong security posture. By embedding these practices into our development culture, we ensure that every solution delivered by Voronkin is not just functional, but truly production-ready and future-proof.

Related Reading

Voronkin Web Development specialises in custom software and DevOps solutions — reach out to discuss your next project.