How Insider, Airbnb, and 3M Master AWS Compute: Architecture Spotlights

Airbnb: Securing multi-tenant Kubernetes clusters at scale on EC2 with fine-grained access controls.

The Power of Flexible Compute on AWS

Choosing the right compute services is fundamental to building effective, scalable, and cost-efficient applications in the cloud. Amazon Web Services (AWS) offers a vast array of compute options, but seeing how real companies leverage these services to solve unique challenges provides invaluable insights. In a recent “This is My Architecture” spotlight, AWS showcased innovative compute solutions from Insider, Airbnb, and 3M, each tackling distinct problems with tailored architectures. Let’s dive into their approaches.

1. Insider: Mastering Real-Time Data Ingestion at Scale

• The Challenge: Insider deals with billions of data points daily for its real-time customer experience products. Ingesting, processing, and storing this massive volume reliably is critical.
• The Architecture: Insider leverages Amazon Elastic Kubernetes Service (EKS) as the compute backbone for its data ingestion pipeline. Data flows into EKS pods, which then efficiently write to multiple storage destinations:
• Amazon DynamoDB: For fast key-value lookups.
• Clickhouse: For real-time analytics.
• Amazon S3: As a durable data lake for long-term storage, troubleshooting, and crucially, data replay capabilities.
• Unique Perspective: Insider’s architecture highlights the power of using Kubernetes (via EKS) not just for application deployment, but as a robust data processing engine. Their strategic use of S3 for data replay is a key takeaway for handling potential issues in high-volume data streams without impacting live systems.

2. Airbnb: Securing Multi-Tenant Kubernetes on EC2

• The Challenge: Running multi-tenant Kubernetes clusters efficiently and securely at scale presents significant hurdles, particularly around access control between different pods and services.
• The Architecture: Airbnb runs its large-scale Kubernetes clusters on Amazon EC2 instances. Their unique approach focuses on fine-grained security within the cluster:
• A custom control plane injects specific service account tokens into pods.
• These tokens are used by pods to request temporary credentials from AWS Security Token Service (STS).
• AWS Identity and Access Management (IAM) policies then enforce least-privilege access based on these temporary credentials.
• Unique Perspective: Airbnb demonstrates a sophisticated method for achieving granular security within a multi-tenant Kubernetes environment on AWS. By integrating Kubernetes service accounts tightly with AWS STS and IAM, they ensure that workloads running side-by-side remain securely isolated, a critical requirement for shared infrastructure.

3. 3M: Simplifying HPC Modeling & Securing Sensitive Data

• The Challenge: 3M needed to empower its scientists and engineers to leverage AWS for High-Performance Computing (HPC) modeling without needing deep AWS expertise. Simultaneously, they had to ensure that sensitive data related to novel materials remained confidential until patented.
• The Architecture: 3M built a user-friendly HPC modeling platform abstracting underlying AWS complexity:
• Amazon DynamoDB: Used innovatively for granular access control, ensuring data secrecy based on project or patent status.
• AWS Lambda: Handles backend logic and interacts with other services.
• Amazon Aurora Managed PostgreSQL: Stores metadata and results.
• A web UI provides the simplified interface for users.
• Unique Perspective: 3M’s solution showcases how AWS services can build platforms that democratize access to powerful capabilities like HPC. Their use of DynamoDB specifically for fine-grained, application-level access control tied to intellectual property protection is a compelling pattern for organizations handling sensitive R&D data.

Conclusion: Tailoring AWS Compute to Your Needs

The architectures from Insider, Airbnb, and 3M underscore the flexibility of AWS compute services. Whether facing massive data ingestion, complex security requirements in shared environments, or the need to balance accessibility with data protection for specialized workloads, AWS provides the building blocks. By learning from these real-world examples featured in “This is My Architecture,” organizations can gain practical ideas for designing their own effective and innovative cloud solutions.