AWS Security Find: Unmasking Phantom DNS Queries with Route 53 Resolver Query Logging
Ever wonder what your AWS Fargate tasks are really doing? I recently uncovered a hidden world of DNS requests by enabling Route 53 Resolver query logging. What I found wasn't a malicious attack, but an open-source APM agent probing Google Cloud's metadata service from my AWS environment! Learn how you can use Route 53 Resolver query logging to expose hidden network activity and bolster your AWS security.
Why Bother with Route 53 Resolver Query Logging?
Think of DNS query logging as a network security camera, capturing every domain your resources try to reach. Here's why it's a game-changer:
- Complete DNS Visibility: See every query name, source IP, response code, and timestamp. No more blind spots when troubleshooting network problems in AWS.
- Threat Hunting & Security Insights: Identify potential malware or phishing attempts early. Integrate logs with SIEMs for automated alerting and faster incident response.
- Audit & Compliance: Demonstrate continuous network monitoring, a must for regulated workloads like PCI and HIPAA.
- Faster Troubleshooting: Correlate DNS resolution errors with application failures to diagnose issues quickly. Use Route 53 logs to pinpoint the root cause.
Quick Start: AWS DNS Query Logging & Firewall with CloudFormation
Ready to get started? Use this CloudFormation template to enable DNS query logging and set up a basic DNS Firewall in minutes:
- Paste the template into your AWS CloudFormation console.
- Provide your VPC ID.
- Start capturing all DNS queries to CloudWatch Logs!
Alternatively, download the template as a prepared YAML file from GitHub.
Once deployed, navigate to CloudWatch Logs to examine the DNS queries originating from your VPC.
Understanding Blocked DNS Queries with DNS Firewall
When you implement a DNS Firewall, blocked lookups become clearly visible in your logs. The action field will indicate that a domain was blocked. Below is an example of how a blocked domain appears in CloudWatch Logs or an S3 log file upon enabling Route 53 DNS Firewall:
Level Up: Route 53 DNS Firewall Enhancements
The CloudFormation template gives you a basic DNS Firewall. Enhance it with these features:
- Managed Threat Lists: Automatically block known malicious domains, reducing manual upkeep.
- Custom Allow/Deny Rules: Enforce your organization's approved domain list.
- Real-Time Enforcement: Choose between BLOCK, ALERT, or TRUNCATE responses to specific DNS queries.
Apply these firewall policies at the VPC level for centralized control—no need for per-instance agents!
The Case of the Phantom "metadata.google.internal" DNS Query
My investigation started with debugging flaky outbound calls from an AWS Fargate task. Enabling Route 53 Resolver query logging revealed repeated lookups for metadata.google.internal, a Google Cloud-specific endpoint. This was triggered by an APM agent's auto-detection logic. Because I hadn't explicitly configured the cloud provider in my Elastic APM setup, the agent was attempting to determine the cloud provider by trying the AWS metadata endpoint (which doesn't work on Fargate) and then the Google Cloud endpoint.
Real-World Example: Ubuntu Cloud-Init Probes
Another interesting finding: Ubuntu cloud-init instances querying does-not-exist.example.com to detect DNS interception. While benign, these checks can clutter your logs. This highlights the importance of DNS query logging for understanding and filtering legitimate, but noisy, network behavior.
Best Practices for Route 53 Resolver Query Logging
Here are some tips to maximize the value of your Route 53 setup:
- Automate with IaC: Use CloudFormation or Terraform for consistent, repeatable deployments.
- Centralize Logs: Stream logs to CloudWatch Logs or S3. Integrate with security platforms for automated analysis. Leverage Generative AI to provide automated analysis on aggregated data.
- Tune Policies: Regularly review and whitelist legitimate domains to minimize false positives. Use Git to version control the domain lists.
- Periodic Review: Analyze logs regularly to refine firewall rules and identify new patterns.
Conclusion: Unlock Enhanced AWS Security with DNS Visibility
Route 53 Resolver query logging is a powerful tool for both diagnostics and security. Paired with a DNS Firewall, it gives you the visibility and control needed to prevent unwanted traffic, uncover hidden DNS requests, and strengthen your overall AWS security posture. Turn on logging, explore your logs, and stay secure!
