Theoretical Foundation: Configure Log Settings in Azure Monitor

1. Initial Intuition

Imagine you manage a large company with hundreds of departments, servers, security cameras, and systems. Each of these systems generates activity logs: who accessed what, when, and with what result. Without a centralized collection strategy, these logs remain scattered across local files, impossible to correlate.

Configuring logs in Azure Monitor is defining the centralized collection system for this company: you specify which sources should send their logs, to which central destination (the company's "central archive"), and for how long these logs should be retained.

The fundamental difference from simply "enabling logs" is that you're configuring a data pipeline: the origin, transmission channel, storage destination, and retention policies. Each of these decisions has a direct impact on cost, security, and the ability to respond to incidents.

2. Context

2.1 The Azure Monitor logs ecosystem

100%

Scroll para zoom · Arraste para mover · 📱 Pinch para zoom no celular

2.2 The three main types of logs

Before configuring, it's essential to understand what each log type contains:

Activity Log: Records control plane operations: who created, modified, or deleted Azure resources. It's automatic, requires no configuration to exist. Covers the entire subscription. Example: "user X deleted VM Y at 14:32".

Resource Logs (formerly Diagnostic Logs): Record data plane operations: what happened inside the resource. Example for Storage Account: which blobs were read, written, which errors occurred. They are not collected automatically; they require configured Diagnostic Settings.

Azure Active Directory Logs: Include Sign-in logs (authentication attempts) and Audit logs (directory changes). Require Azure AD P1/P2 license for retention beyond 7 days.

3. Concept Building

3.1 Log Analytics Workspace: the central destination

The Log Analytics Workspace (LAW) is the central log repository in Azure Monitor. It's where logs from multiple resources become queryable data via KQL (Kusto Query Language).

A LAW has:

Interactive retention: 30 days (default, configurable from 4 to 730 days)
Archive retention: 7 years (data in reduced retention, queryable but slower)
Tables: Each log type goes into a different table (e.g., AzureActivity, StorageBlobLogs, SigninLogs)
Region: The LAW has a region and data stays in that region

3.2 Diagnostic Settings: the routing mechanism

Diagnostic Settings are the configuration that defines, for each resource, which log categories to send and to which destinations.

Each resource can have up to 5 simultaneous Diagnostic Settings, allowing logs to be sent to multiple destinations at the same time (e.g., Log Analytics for queries and Storage Account for archiving).

A Diagnostic Setting defines:

Log categories: Which types of events to capture (e.g., StorageRead, StorageWrite, StorageDelete)
Metrics: Optionally export metrics along with logs
Destinations: Log Analytics Workspace, Storage Account, Event Hub, and/or partner (e.g., Datadog)

3.3 Log Analytics tables and retention plans

With the introduction of Basic Logs and Analytic Logs, Log Analytics has two table plans with different costs and capabilities:

Plan	Ingestion cost	Query cost	Interactive retention	Recommended use
Analytics	High	Free	Up to 730 days	Critical logs queried frequently
Basic	Low (80% cheaper)	Per GB queried	Fixed 8 days	High-volume logs, rarely queried
Auxiliary	Very low	Per GB queried	No interactive retention	Compliance-only logs

Cost strategy: Critical audit logs (Activity Log, Sign-in logs) should stay in Analytics plan. High-volume diagnostic logs like NSG Flow Logs or Storage verbose logs can go to Basic if they're rarely queried.

3.4 Activity Log: configuration and retention

The Activity Log exists automatically for 90 days without any configuration. For retention beyond 90 days, you need to create a Diagnostic Setting at the subscription level that forwards to Log Analytics, Storage Account, or Event Hub.

The table in Log Analytics is AzureActivity. It contains fields like:

OperationName: What was done (e.g., Microsoft.Compute/virtualMachines/delete)
Caller: Who did it (user email or Service Principal)
Level: Informational, Warning, Error, Critical
ActivityStatus: Succeeded, Failed, Started
ResourceId: Which resource was affected

3.5 Data Collection Rules (DCRs): the new architecture

For VMs specifically, the modern way to collect logs is via Data Collection Rules (DCRs), which work together with the Azure Monitor Agent (AMA):

DCR defines: which logs to collect (e.g., Windows Event Log events, Linux syslog), filters and transformations
AMA is the agent installed on the VM that executes the DCR instructions
Collected data goes to the destination configured in the DCR (usually Log Analytics)

DCRs replace the Legacy Log Analytics Agent (MMA/OMS) and offer:

Data filtering at the source (sends only what's needed, reducing cost)
Data transformations before ingestion
Multiple destinations per DCR

4. Structural View

100%

Scroll para zoom · Arraste para mover · 📱 Pinch para zoom no celular

5. Practical Operation

5.1 Log categories by resource

Each resource type has its set of log categories. Examples:

Storage Account:

Category	What it logs
StorageBlobLogs	Blob API operations (read, write, delete)
StorageFileLogs	Files API operations
StorageQueueLogs	Queue API operations
StorageTableLogs	Table API operations

Azure SQL Database:

Category	What it logs
SQLInsights	Query Store diagnostic data
AutomaticTuning	Automatic tuning recommendations
QueryStoreRuntimeStatistics	Query runtime statistics
Errors	Database errors
DatabaseWaitStatistics	Wait statistics
Blocks	Blocking events between transactions
Deadlocks	Detected deadlocks

Key Vault:

Category	What it logs
AuditEvent	All access operations to secrets, keys, and certificates

5.2 Log ingestion latency

An important behavior: logs don't reach Log Analytics instantly. Typical latency is:

Activity Log: 2 to 15 minutes after the event
Resource Diagnostic Logs: 2 to 5 minutes
Guest OS logs via AMA: 1 to 5 minutes
Application Insights: Near real-time

When investigating a recent incident, wait a few minutes before concluding there are no logs.

5.3 Configuring retention and archiving in Log Analytics

100%

Scroll para zoom · Arraste para mover · 📱 Pinch para zoom no celular

6. Implementation Methods

6.1 Azure Portal

When to use: Initial setup, viewing all available categories, point adjustments.

Creating Diagnostic Setting for a resource:

Resource > Monitoring > Diagnostic settings > + Add diagnostic setting

The portal automatically lists all categories available for that resource type, allowing you to check each one individually or "allLogs" to capture everything.

Activity Log at subscription level:

Azure Monitor > Activity Log > Export Activity Logs

6.2 Azure CLI

Creating Diagnostic Setting for Storage Account:

# Capture all Blob logs and send to Log Analytics
az monitor diagnostic-settings create \
  --name "storage-diagnostics" \
  --resource <storage-account-resource-id> \
  --logs '[
    {"category": "StorageBlobLogs", "enabled": true, "retentionPolicy": {"days": 0, "enabled": false}},
    {"category": "StorageFileLogs", "enabled": true, "retentionPolicy": {"days": 0, "enabled": false}}
  ]' \
  --metrics '[
    {"category": "Transaction", "enabled": true}
  ]' \
  --workspace <log-analytics-workspace-id>

Creating Diagnostic Setting with multiple destinations:

az monitor diagnostic-settings create \
  --name "multi-destination" \
  --resource <resource-id> \
  --logs '[{"category": "allLogs", "enabled": true}]' \
  --workspace <law-id> \
  --storage-account <storage-account-id>

Configuring subscription Activity Log:

az monitor diagnostic-settings subscription create \
  --name "activity-log-to-law" \
  --location eastus \
  --logs '[{"category": "Administrative", "enabled": true}, {"category": "Security", "enabled": true}, {"category": "ServiceHealth", "enabled": true}, {"category": "Alert", "enabled": true}, {"category": "Policy", "enabled": true}]' \
  --workspace <log-analytics-workspace-id>

Listing available categories for a resource:

az monitor diagnostic-settings categories list \
  --resource <resource-id> \
  --output table

Checking active Diagnostic Settings:

az monitor diagnostic-settings list \
  --resource <resource-id> \
  --output table

6.3 Configuring Data Collection Rules for VMs

# Create DCR to collect Windows events
az monitor data-collection rule create \
  --resource-group myRG \
  --location eastus \
  --name dcr-windows-events \
  --data-flows '[
    {
      "streams": ["Microsoft-Event"],
      "destinations": ["myLAW"]
    }
  ]' \
  --data-sources '{
    "windowsEventLogs": [
      {
        "name": "eventLogsDataSource",
        "streams": ["Microsoft-Event"],
        "xPathQueries": [
          "Application!*[System[(Level=1 or Level=2 or Level=3)]]",
          "Security!*[System[(band(Keywords,13510798882111488))]]",
          "System!*[System[(Level=1 or Level=2 or Level=3)]]"
        ]
      }
    ]
  }' \
  --destinations '{
    "logAnalytics": [
      {
        "name": "myLAW",
        "workspaceResourceId": "<law-resource-id>"
      }
    ]
  }'

# Associate DCR with a VM
az monitor data-collection rule association create \
  --resource-group myRG \
  --resource <vm-resource-id> \
  --rule-id <dcr-resource-id> \
  --name dcra-vm01

6.4 Bicep: Diagnostic Settings as code

// Diagnostic Settings for Storage Account
resource diagnosticSettings 'Microsoft.Insights/diagnosticSettings@2021-05-01-preview' = {
  name: 'storage-diagnostics'
  scope: storageAccount
  properties: {
    workspaceId: logAnalyticsWorkspace.id
    storageAccountId: archiveStorage.id
    logs: [
      {
        category: 'StorageBlobLogs'
        enabled: true
        retentionPolicy: {
          days: 0
          enabled: false
        }
      }
      {
        category: 'StorageFileLogs'
        enabled: true
        retentionPolicy: {
          days: 0
          enabled: false
        }
      }
    ]
    metrics: [
      {
        category: 'Transaction'
        enabled: true
        retentionPolicy: {
          days: 0
          enabled: false
        }
      }
    ]
  }
}

6.5 Configuring Log Analytics Workspace retention

# Configure interactive retention for 90 days
az monitor log-analytics workspace update \
  --resource-group myRG \
  --workspace-name myLAW \
  --retention-time 90

# Configure total archive retention for 2 years (730 days interactive + rest in archive)
az monitor log-analytics workspace update \
  --resource-group myRG \
  --workspace-name myLAW \
  --retention-time 90 \
  --data-retention 730

7. Control and Security

7.1 Permissions to configure logs

Operation	Minimum role
Create/edit Diagnostic Settings on a resource	Monitoring Contributor on the resource
Create Activity Log Diagnostic Setting	Monitoring Contributor on the subscription
Create/edit DCRs	Monitoring Contributor
Query logs in Log Analytics	Log Analytics Reader
Configure LAW retention	Log Analytics Contributor

7.2 Log data security

Workspace isolation: Data in Log Analytics is isolated by workspace. Users with workspace access can query all data in it. For sensitive data from multiple teams, consider separate workspaces or use Table-level RBAC to restrict access by table.

Table-level RBAC:

# Restrict access to SecurityEvent table to security team only
az monitor log-analytics workspace table update \
  --resource-group myRG \
  --workspace-name myLAW \
  --name SecurityEvent \
  --retention-time 90

Data in transit: Logs are transmitted with HTTPS/TLS. There's no option to disable.

Log immutability: Logs sent to Log Analytics cannot be modified. For compliance requiring demonstrable immutability, also send to Storage Account with configured Immutability Policy.

7.3 Cost: the main configuration risk

The main risk when configuring logs is unexpectedly high cost. Log Analytics charges per GB ingested. High-volume categories without real necessity can generate significant costs.

Cost control strategies:

Enable only necessary categories. Don't use allLogs indiscriminately in high-volume production.
Use Basic Logs for high-volume categories rarely queried.
Configure transformations in DCRs to filter unnecessary fields before ingestion.
Monitor ingestion cost itself via Usage table in LAW:

Usage
| where TimeGenerated > ago(30d)
| summarize TotalGB = sum(Quantity) / 1000 by DataType
| order by TotalGB desc

8. Decision Making

8.1 Which destination to use for logs

Need	Destination	Reason
Investigation and troubleshooting	Log Analytics Workspace	Flexible and fast KQL queries
Compliance and long retention (> 2 years)	Storage Account	Lower cost for archiving
Integration with external SIEM (Splunk, QRadar)	Event Hub	Real-time streaming
Visualization in Grafana or Datadog	Event Hub or direct Partner	Native integration
Alerts based on log patterns	Log Analytics Workspace	Alert rules use KQL
Compliance audit with immutable evidence	Storage Account + Immutability	Unalterable data

8.2 Retention: interactive vs archive

Situation	Recommended configuration	Reason
Operational logs queried daily	30-90 days interactive	Balanced cost with need
Security logs audited monthly	90-180 days interactive	Reasonable investigation window
Compliance requiring 1 year retention	90 days interactive + archive up to 1 year	Optimized cost
Compliance requiring 7 years	30 days interactive + archive up to 7 years	Lowest possible cost

8.3 Activity Log: most important categories

Category	What it includes	Why enable
Administrative	Resource CRUD	Infrastructure change auditing
Security	Microsoft Defender alerts	Threat detection
ServiceHealth	Azure incidents and maintenance	Context for issues
Alert	Alert triggers	Alert history
Policy	Azure Policy evaluations	Compliance
Autoscale	Scale events	Capacity troubleshooting

9. Best Practices

Never rely only on the Activity Log's default 90 days. Configure subscription Diagnostic Settings to send Activity Log to Log Analytics from the beginning.
Create Diagnostic Settings as code (Bicep/Terraform) to ensure new resources inherit them automatically via Azure Policy.
Use Azure Policy with DeployIfNotExists effect to automatically apply Diagnostic Settings to new resources.
Separate workspaces by purpose in large environments: one for security (Security Events, Sign-in logs), one for operations (resource diagnostics), one for development. This reduces cost and improves access control.
Configure the SecurityEvent table with long retention (90-180 days) independent of workspace defaults.
Monitor ingestion cost monthly with queries on the Usage table. An ingestion spike can indicate misconfiguration or malicious activity generating many logs.
Use DCRs with XPath filters to collect only relevant events from Windows Event Log (e.g., avoid collecting Verbose or Debug level events in production).
Configure workspace with multiple destinations for critical logs: Log Analytics for querying and Storage Account for immutable archiving.

10. Common Errors

Error	Why it happens	How to avoid
Activity Log lost after 90 days	No Diagnostic Setting exporting to LAW	Configure export on day 1
Unexpectedly high Log Analytics cost	allLogs enabled on high-volume resources	Select only necessary categories
VM logs without memory data	Only platform agent installed, no AMA	Install Azure Monitor Agent and configure DCR
Logs arriving late during incident	Ingestion latency ignored	Wait 5-10 min; check if resource is sending via diagnostic portal
Query with no results for recent event	Logs still in transit (latency)	Wait 2-5 minutes and retry query
Diagnostic Settings lost after recreating resource	Not in IaC, was created manually	Include Diagnostic Settings in Bicep/ARM template
Archive retention not activated but charged	Incorrect policy configuration	Check via `az monitor log-analytics workspace show`
Data in Basic table not queryable in alert	Alert rules require Analytics table	Move alert tables to Analytics plan

11. Operation and Maintenance

11.1 Checking existing diagnostic configurations

# List all Diagnostic Settings for a resource
az monitor diagnostic-settings list \
  --resource <resource-id>

# View detailed configuration of a specific setting
az monitor diagnostic-settings show \
  --name "storage-diagnostics" \
  --resource <resource-id>

# List existing DCRs
az monitor data-collection rule list \
  --resource-group myRG \
  --output table

11.2 Auditing log coverage across subscription

KQL query to identify resources without Diagnostic Settings:

// Resources that have NOT sent any logs in the last 24 hours
Resources
| where type in~ (
    "microsoft.storage/storageaccounts",
    "microsoft.keyvault/vaults",
    "microsoft.sql/servers/databases"
)
| join kind=leftouter (
    AzureDiagnostics
    | where TimeGenerated > ago(24h)
    | summarize LastLog = max(TimeGenerated) by ResourceId
    | extend ResourceId = tolower(ResourceId)
) on $left.id == $right.ResourceId
| where isempty(LastLog)
| project name, type, resourceGroup

11.3 Monitoring log ingestion health

// Average ingestion latency in the last 6 hours
Heartbeat
| where TimeGenerated > ago(6h)
| summarize AvgLatency = avg(TimeGenerated - RawTime) by Computer
| order by AvgLatency desc

// Ingestion volume by table in the last 24 hours (in GB)
Usage
| where TimeGenerated > ago(1d)
| summarize TotalGB = round(sum(Quantity) / 1024, 2) by DataType
| order by TotalGB desc
| take 20

11.4 Important limits

Aspect	Limit
Diagnostic Settings per resource	5
Activity Log: default retention	90 days
Log Analytics maximum interactive retention	730 days
Log Analytics maximum archive retention	7 years (2,557 days)
Typical ingestion latency	2 to 5 minutes
Maximum ingestion throughput per LAW	No documented limit (automatic scale)
Workspaces per subscription	No practical limit
DCR associations per VM	10

12. Integration and Automation

12.1 Azure Policy to automatically apply Diagnostic Settings

# Built-in policy: enable Diagnostic Setting on Storage Accounts
az policy assignment create \
  --name "storage-diagnostic-settings" \
  --policy "59760408-1f12-4b5a-89a5-ce2d23f97e46" \
  --scope "/subscriptions/<sub-id>" \
  --params '{"logAnalytics": {"value": "<law-resource-id>"}}'

Policies with DeployIfNotExists effect ensure that any new resource in the subscription has Diagnostic Settings created automatically, even without manual intervention.

12.2 Exporting to Event Hub and integrating with Splunk

# Create Event Hub namespace and hub for logs
az eventhubs namespace create \
  --resource-group myRG \
  --name myLogStreamNamespace \
  --location eastus

az eventhubs eventhub create \
  --resource-group myRG \
  --namespace-name myLogStreamNamespace \
  --name azure-logs

# Create Diagnostic Setting sending to Event Hub
az monitor diagnostic-settings create \
  --name "stream-to-eventhub" \
  --resource <resource-id> \
  --logs '[{"category": "allLogs", "enabled": true}]' \
  --event-hub-name azure-logs \
  --event-hub-rule <eventhub-auth-rule-id>

12.3 Microsoft Sentinel: Azure's native SIEM

Microsoft Sentinel uses Log Analytics Workspace as a foundation and adds SIEM (Security Information and Event Management) capabilities. Integration is direct:

# Enable Sentinel on LAW
az sentinel onboarding-state create \
  --resource-group myRG \
  --workspace-name myLAW

Once enabled, Sentinel uses existing tables (AzureActivity, SigninLogs, SecurityEvent, etc.) for threat detection and investigation.

13. Final Summary

Essential concepts:

Activity Log exists automatically for 90 days and records control plane operations (resource CRUD). For long retention, configure Diagnostic Settings on the subscription.
Resource Logs record data plane operations (what happens inside the resource). Require Diagnostic Settings explicitly configured for each resource.
Log Analytics Workspace is the central destination for KQL queries. Has interactive retention (up to 730 days) and archive retention (up to 7 years).
Data Collection Rules (DCRs) with Azure Monitor Agent are the modern way to collect VM logs, replacing the Legacy Agent.

Critical differences:

Activity Log vs Resource Logs: Activity = control plane (who changed the resource). Resource = data plane (what happened inside the resource).
Log Analytics (Analytics) vs Basic Logs: Analytics = free querying, expensive ingestion, ideal for frequently queried logs. Basic = cheap ingestion, paid querying per GB, ideal for high-volume logs rarely queried.
Interactive vs Archive retention: Interactive = normal KQL querying (higher cost). Archive = data accessible via temporary Restore (lower cost).
Diagnostic Setting vs DCR: Diagnostic Settings route PaaS resource logs. DCR collects OS internal logs from VMs via Azure Monitor Agent.

What needs to be remembered:

Logs have 2 to 5 minutes latency after the event. Immediate queries may not find recent events.
allLogs on high-volume resources can generate significant costs. Select only necessary categories.
Diagnostic Settings are destroyed with the resource. Without IaC, they need to be recreated manually.
Default Activity Log is available for only 90 days. For long-term auditing, configure export immediately.
Up to 5 Diagnostic Settings per resource, allowing multiple simultaneous destinations.
Use Azure Policy with DeployIfNotExists to ensure log coverage across the entire subscription automatically.

1. Initial Intuition​

2. Context​

2.1 The Azure Monitor logs ecosystem​

2.2 The three main types of logs​

3. Concept Building​

3.1 Log Analytics Workspace: the central destination​

3.2 Diagnostic Settings: the routing mechanism​

3.3 Log Analytics tables and retention plans​

3.4 Activity Log: configuration and retention​

3.5 Data Collection Rules (DCRs): the new architecture​

4. Structural View​

5. Practical Operation​

5.1 Log categories by resource​

5.2 Log ingestion latency​

5.3 Configuring retention and archiving in Log Analytics​

6. Implementation Methods​

6.1 Azure Portal​

6.2 Azure CLI​

6.3 Configuring Data Collection Rules for VMs​

6.4 Bicep: Diagnostic Settings as code​

6.5 Configuring Log Analytics Workspace retention​

7. Control and Security​

7.1 Permissions to configure logs​

7.2 Log data security​

7.3 Cost: the main configuration risk​

8. Decision Making​

8.1 Which destination to use for logs​

8.2 Retention: interactive vs archive​

8.3 Activity Log: most important categories​

9. Best Practices​

10. Common Errors​

11. Operation and Maintenance​

11.1 Checking existing diagnostic configurations​

11.2 Auditing log coverage across subscription​

11.3 Monitoring log ingestion health​

11.4 Important limits​

12. Integration and Automation​

12.1 Azure Policy to automatically apply Diagnostic Settings​

12.2 Exporting to Event Hub and integrating with Splunk​

12.3 Microsoft Sentinel: Azure's native SIEM​

13. Final Summary​

1. Initial Intuition

2. Context

2.1 The Azure Monitor logs ecosystem

2.2 The three main types of logs

3. Concept Building

3.1 Log Analytics Workspace: the central destination

3.2 Diagnostic Settings: the routing mechanism

3.3 Log Analytics tables and retention plans

3.4 Activity Log: configuration and retention

3.5 Data Collection Rules (DCRs): the new architecture

4. Structural View

5. Practical Operation

5.1 Log categories by resource

5.2 Log ingestion latency

5.3 Configuring retention and archiving in Log Analytics

6. Implementation Methods

6.1 Azure Portal

6.2 Azure CLI

6.3 Configuring Data Collection Rules for VMs

6.4 Bicep: Diagnostic Settings as code

6.5 Configuring Log Analytics Workspace retention

7. Control and Security

7.1 Permissions to configure logs

7.2 Log data security

7.3 Cost: the main configuration risk

8. Decision Making

8.1 Which destination to use for logs

8.2 Retention: interactive vs archive

8.3 Activity Log: most important categories

9. Best Practices

10. Common Errors

11. Operation and Maintenance

11.1 Checking existing diagnostic configurations

11.2 Auditing log coverage across subscription

11.3 Monitoring log ingestion health

11.4 Important limits

12. Integration and Automation

12.1 Azure Policy to automatically apply Diagnostic Settings

12.2 Exporting to Event Hub and integrating with Splunk

12.3 Microsoft Sentinel: Azure's native SIEM

13. Final Summary