Unmasking Hidden Threats: AWS Route 53 Resolver Query Logging and DNS Firewall

Worried about hidden network activity within your AWS environment? Debugging flaky outbound calls from AWS Fargate tasks can be a nightmare without the right tools. You need complete DNS visibility to quickly identify the culprit workload. Discover how AWS Route 53 Resolver query logging can bring clarity and control to your AWS network. Not only will you gain visibility into potential network issues, but you'll also be able to swiftly detect and block suspicious activity.

Unveiling the Mystery: My Encounter with Unexpected DNS Queries

While debugging outbound calls from an AWS Fargate task, I noticed repeated lookups for metadata.google.internal.. This endpoint should only exist within Google Cloud VMs, but my workload was running entirely on AWS. Was it a hacker probing my Fargate tasks?

It turned out to be an open-source APM agent auto-detecting the cloud provider by testing the GCP metadata endpoint when the AWS metadata endpoint was unavailable (since AWS Fargate tasks don't expose the EC2 instance metadata endpoint). This "phantom" query, invisible without logging, highlights the importance of enabling Route 53 Resolver query logging.

Why You Need Route 53 Resolver Query Logging

• Complete DNS Visibility: Capture crucial data like query names, source IPs, response codes, and timestamps.
• Security Insights & Threat Hunting: Identify potential malware or phishing attempts early. Feed logs into Security Information and Event Management (SIEM) systems for automated alerting.
• Audit & Compliance: Demonstrate continuous network monitoring for regulated workloads.
• Faster Troubleshooting: Correlate DNS resolution errors with service issues to diagnose application failures more effectively.

Get Started: Enable Query Logging and Basic DNS Firewall with CloudFormation

Here's a quick way to get started using CloudFormation with query logging and a basic DNS Firewall rule group. Simply paste the following code into your stack, provide your VPC ID, and start capturing all DNS queries to CloudWatch Logs:

AWSTemplateFormatVersion: '2010-09-09'
Description: >
  Route 53 Resolver DNS Firewall with VPC-level DNS **query logging**.
  Logs **all** DNS queries for a supplied VPC to CloudWatch Logs.
Parameters:
  VpcId:
    Type: AWS::EC2::VPC::Id
    Description: VPC ID to protect and monitor.
  AssociationPriority:
    Type: Number
    Default: 150
    Description: 'Priority (100-9900) for the VPC firewall association; must be unique within the VPC.'
    MinValue: 100
    MaxValue: 9900
  LogRetentionDays:
    Type: Number
    Default: 30
    Description: Retention (days) for the CloudWatch Logs group.
Resources:
  QueryLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      RetentionInDays: !Ref LogRetentionDays
  BlockedDomains:
    Type: AWS::Route53Resolver::FirewallDomainList
    Properties:
      Name: blocked-domains
      Domains:
        - badexample.com
        - malware.example
        - phishing.test
  FirewallRuleGroup:
    Type: AWS::Route53Resolver::FirewallRuleGroup
    Properties:
      Name: vpc-dns-firewall-rule-group
      FirewallRules:
        - FirewallDomainListId: !Ref BlockedDomains
          Priority: 10
          Action: BLOCK
          BlockResponse: NODATA
  FirewallAssociation:
    Type: AWS::Route53Resolver::FirewallRuleGroupAssociation
    Properties:
      FirewallRuleGroupId: !Ref FirewallRuleGroup
      VpcId: !Ref VpcId
      Priority: !Ref AssociationPriority
      Name: vpc-dns-firewall-association
  DNSQueryLogConfig:
    Type: AWS::Route53Resolver::ResolverQueryLoggingConfig
    Properties:
      Name: vpc-dns-query-logs
      DestinationArn: !GetAtt QueryLogGroup.Arn
  DNSQueryLogAssociation:
    Type: AWS::Route53Resolver::ResolverQueryLoggingConfigAssociation
    Properties:
      ResolverQueryLogConfigId: !Ref DNSQueryLogConfig
      ResourceId: !Ref VpcId
  DNSQueryLogsPolicy:
    Type: AWS::Logs::ResourcePolicy
    Properties:
      PolicyName: route53resolver-query-logs
      PolicyDocument: !Sub |
        {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "Route53ResolverQueryLogsToCloudWatch",
              "Effect": "Allow",
              "Principal": { "Service": "route53resolver.amazonaws.com" },
              "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
              ],
              "Resource": "arn:${AWS::Partition}:logs:${AWS::Region}:${AWS::AccountId}:log-group:${QueryLogGroup}:*"
            }
          ]
        }
Outputs:
  FirewallRuleGroupId:
    Description: ID of the DNS Firewall Rule Group
    Value: !Ref FirewallRuleGroup
  QueryLogConfigId:
    Description: ID of the Resolver Query Log Config
    Value: !Ref DNSQueryLogConfig
  QueryLogGroupName:
    Description: CloudWatch Logs group for DNS query logs
    Value: !Ref QueryLogGroup

Alternatively, you can download it from GitHub. Once deployed, head to CloudWatch Logs to inspect your VPC's DNS queries.

Understanding Blocked DNS Queries

When you implement a DNS Firewall, blocked lookups become clearly visible with an action field. Consider this sample CloudWatch Logs entry:

{
  "version": "1.100000",
  "account_id": "452954105288",
  "region": "us-east-1",
  "vpc_id": "vpc-0b20bb8a5eb28cb99",
  "query_timestamp": "2025-04-27T04:45:23Z",
  "query_name": "badexample.com.",
  "query_type": "A",
  "query_class": "IN",
  "rcode": "NOERROR",
  "answers": [],
  "srcaddr": "10.0.101.123",
  "srcport": "46275",
  "transport": "UDP",
  "srcids": {
    "instance": "i-0cf6fbb4f8960f245"
  },
  "firewall_rule_action": "BLOCK",
  "firewall_rule_group_id": "rslvr-frg-69632fc987684fbd",
  "firewall_domain_list_id": "rslvr-fdl-de20dd498c14dce"
}

Enhancing Security with Route 53 DNS Firewall

The CloudFormation template provides a solid foundation. Enhance it further with:

• Managed Threat Lists: Automatically block known malicious domains.
• Custom Allow/Deny Rules: Enforce approved domains based on your corporate policies.
• Real-Time Enforcement: Choose between BLOCK, ALERT, or TRUNCATE responses.

With policies applied at the VPC level, you avoid the need for per-instance agents.

Decoding the Metadata Mystery: Why the Google Query?

My workload, a Python Flask app using Elastic APM on AWS Fargate, triggered the GCP metadata query. Here's why:

1. The Elastic APM agent attempts to auto-detect the cloud provider if one isn't explicitly configured.
2. It begins by checking for the AWS EC2 metadata endpoint.
3. Since Fargate doesn't support this endpoint, the check fails.
4. The agent then attempts to detect Google Cloud, resulting in the unexpected DNS query.

Since AWS and Azure metadata endpoints are IP-based, they didn't generate any DNS queries.

Real-World Scenario: Why Ubuntu Queries does-not-exist.example.com

Another interesting case: Ubuntu cloud-init uses does-not-exist.example.com. to detect DNS interception. While harmless, these checks can clutter logs, which is why query logging and subsequent filtering is paramount.

Best Practices for Effective DNS Monitoring

• Automate with IaC: Use CloudFormation or Terraform for consistent deployments.
• Centralize Logs: Stream logs to CloudWatch Logs or S3 for analysis and integration with security platforms.
• Tune Policies: Review and whitelist legitimate domains to minimize false positives. Treat domain lists as code and version control them
• Periodic Review: Regularly analyze logs to refine firewall rules and identify new threat patterns.

Conclusion: Elevate Your AWS Security Posture

AWS Route 53 Resolver query logging, combined with a DNS Firewall, provides critical visibility and control over your network. You can prevent unwanted traffic, expose hidden queries, and strengthen your cloud security. Turn on DNS query logging today and transform your DNS into a potent security asset.