Ingesting Custom Logs to Azure Monitor with Java SDK

Application observability requires centralized log aggregation, and Azure Monitor provides powerful log analytics capabilities for querying and alerting on telemetry data. The Azure Monitor Ingestion SDK for Java enables AI agents to send custom logs directly to Azure Monitor using the Logs Ingestion API, supporting scenarios beyond standard application insights telemetry like custom security events, business metrics, or infrastructure health data.

What This Skill Does

The azure-monitor-ingestion-java skill provides Java client libraries for sending custom logs to Azure Monitor through Data Collection Endpoints and Data Collection Rules. It handles log batching for efficient transmission, concurrent upload for high-throughput scenarios, partial failure handling with error callbacks, schema validation against Data Collection Rules, and support for both synchronous and asynchronous upload patterns.

This skill enables agents to send custom log entries that don't fit standard telemetry categories, upload logs in batches to minimize network overhead and API calls, handle large log collections with concurrent uploads for improved performance, implement error handling for partial batch failures, and integrate with Data Collection Rules that define schema transformation and routing to Log Analytics tables.

The SDK works with both custom log tables (suffixed with _CL) and built-in tables like CommonSecurityLog, SecurityEvents, Syslog, and WindowsEvents, providing flexibility for different logging scenarios from application-specific metrics to security event forwarding.

Getting Started

Add the Azure Monitor Ingestion dependency to your Maven project:

<dependency>
    <groupId>com.azure</groupId>
    <artifactId>azure-monitor-ingestion</artifactId>
    <version>1.2.11</version>
</dependency>

Or use the Azure SDK BOM for coordinated version management across Azure SDKs:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>com.azure</groupId>
            <artifactId>azure-sdk-bom</artifactId>
            <version>{bom_version}</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

Configure environment variables for your Data Collection Endpoint and Rule:

export DATA_COLLECTION_ENDPOINT="https://my-dce.eastus.ingest.monitor.azure.com"
export DATA_COLLECTION_RULE_ID="dcr-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
export STREAM_NAME="Custom-MyAppLogs_CL"

Create a client with DefaultAzureCredential:

import com.azure.identity.DefaultAzureCredential;
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.azure.monitor.ingestion.LogsIngestionClient;
import com.azure.monitor.ingestion.LogsIngestionClientBuilder;

DefaultAzureCredential credential = new DefaultAzureCredentialBuilder().build();

LogsIngestionClient client = new LogsIngestionClientBuilder()
    .endpoint(System.getenv("DATA_COLLECTION_ENDPOINT"))
    .credential(credential)
    .buildClient();

Key Features

Batch Upload: The SDK accepts lists of log objects and efficiently batches them for transmission. Each upload operation can handle thousands of log entries, reducing API calls and network overhead compared to individual log submission.

Concurrent Uploads: For large log collections, configure concurrent uploads through LogsUploadOptions.setMaxConcurrency(). The SDK splits large batches and uploads them in parallel, dramatically improving throughput for high-volume logging scenarios.

Error Handling: Partial upload failures don't fail the entire batch. Configure an error consumer callback that receives failed log entries and exception details, enabling custom retry logic, alternative storage, or alerting for failed uploads.

Schema Flexibility: Log entry objects are serialized to JSON matching your Data Collection Rule schema. The DCR defines transformations that map incoming fields to table columns, enabling schema evolution without code changes.

Async Support: The SDK provides LogsIngestionAsyncClient for reactive programming patterns using Project Reactor. Async clients enable non-blocking uploads in applications with high concurrency requirements.

Authentication: Use DefaultAzureCredential for automatic credential discovery from Azure CLI, managed identity, environment variables, or Visual Studio. In production, managed identity eliminates credential management entirely.

Usage Examples

Uploading application logs with structured fields:

import java.util.List;
import java.util.ArrayList;

public class ApplicationLog {
    private final String timeGenerated;
    private final String level;
    private final String message;
    private final String component;
    
    public ApplicationLog(String timeGenerated, String level, String message, String component) {
        this.timeGenerated = timeGenerated;
        this.level = level;
        this.message = message;
        this.component = component;
    }
    
    // Getters required for JSON serialization
    public String getTimeGenerated() { return timeGenerated; }
    public String getLevel() { return level; }
    public String getMessage() { return message; }
    public String getComponent() { return component; }
}

List<Object> logs = new ArrayList<>();
logs.add(new ApplicationLog("2024-01-15T10:30:00Z", "INFO", "User login successful", "auth"));
logs.add(new ApplicationLog("2024-01-15T10:30:05Z", "ERROR", "Database connection failed", "database"));

String ruleId = System.getenv("DATA_COLLECTION_RULE_ID");
String streamName = System.getenv("STREAM_NAME");

client.upload(ruleId, streamName, logs);

Implementing high-throughput logging with concurrency:

import com.azure.monitor.ingestion.models.LogsUploadOptions;
import com.azure.core.util.Context;

List<Object> largeBatch = generateLargeBatch(); // 10,000+ logs

LogsUploadOptions options = new LogsUploadOptions()
    .setMaxConcurrency(5);

client.upload(ruleId, streamName, largeBatch, options, Context.NONE);

Handling partial upload failures with custom retry logic:

List<Object> retryQueue = new ArrayList<>();

LogsUploadOptions options = new LogsUploadOptions()
    .setLogsUploadErrorConsumer(uploadError -> {
        System.err.println("Upload error: " + uploadError.getResponseException().getMessage());
        System.err.println("Failed logs count: " + uploadError.getFailedLogs().size());
        
        // Queue failed logs for retry
        retryQueue.addAll(uploadError.getFailedLogs());
        
        // Don't throw - continue uploading remaining batches
    });

client.upload(ruleId, streamName, logs, options, Context.NONE);

// Later: Retry failed logs
if (!retryQueue.isEmpty()) {
    client.upload(ruleId, streamName, retryQueue);
}

Using async client for reactive applications:

import com.azure.monitor.ingestion.LogsIngestionAsyncClient;
import reactor.core.publisher.Mono;

LogsIngestionAsyncClient asyncClient = new LogsIngestionClientBuilder()
    .endpoint(endpoint)
    .credential(new DefaultAzureCredentialBuilder().build())
    .buildAsyncClient();

List<Object> logs = getLogs();

asyncClient.upload(ruleId, streamName, logs)
    .doOnSuccess(v -> System.out.println("Upload completed"))
    .doOnError(e -> System.err.println("Upload failed: " + e.getMessage()))
    .subscribe();

Best Practices

Batch log uploads rather than sending logs individually. Collect logs over a time window (e.g., 30 seconds) and upload in batches. This reduces API calls, improves throughput, and lowers costs compared to real-time log-by-log submission.

Use concurrent uploads for large log collections. Set maxConcurrency between 3-10 based on your throughput needs and network capacity. Higher concurrency improves upload speed but increases memory usage and network connections.

Implement error consumers for production deployments. Don't let partial failures silently lose logs. Log failed entries to a file, queue them for retry, or alert operations teams for investigation.

Match your log entry fields to the Data Collection Rule schema. DCR transformations map incoming JSON fields to table columns. Mismatched field names or types cause ingestion failures. Test schema compatibility before production deployment.

Include TimeGenerated fields in log entries. Most Azure Monitor tables require timestamp fields for time-series analysis. Use ISO 8601 format (e.g., "2024-01-15T10:30:00Z") for consistency.

Reuse client instances across multiple uploads. Client creation involves authentication and network setup overhead. Create once during application startup and reuse throughout the application lifecycle.

Use async clients in high-concurrency applications. If your application already uses reactive programming patterns (Project Reactor, RxJava), async clients provide better integration and resource utilization.

When to Use This Skill

Use this skill when sending custom application logs that don't fit standard Application Insights telemetry categories. Business metrics, workflow events, or domain-specific telemetry often require custom table structures and schemas.

It's ideal for aggregating logs from distributed systems into centralized Azure Monitor. Microservices, serverless functions, or batch jobs can all send logs to the same Log Analytics workspace for unified querying and alerting.

The skill is valuable for security event ingestion from custom sources. Security tools, network devices, or custom security scanners can forward events to Azure Monitor for SIEM correlation and alerting.

Use it for implementing custom metrics tracking beyond standard performance counters. Track business KPIs, SLA metrics, or operational health indicators in Log Analytics for dashboard visualization and alerting.

When Not to Use This Skill

Don't use this skill for standard application telemetry already covered by Application Insights. If you're tracking HTTP requests, dependencies, exceptions, or traces, use the Application Insights SDK designed for those scenarios.

If you're working with Azure services that support diagnostic settings, use those instead. Many Azure services can send logs directly to Log Analytics without custom code through diagnostic settings configuration.

Avoid it for real-time alerting with sub-second latency requirements. Log ingestion and indexing has latency (typically 1-5 minutes). For real-time alerting, use Azure Event Hubs or Stream Analytics instead.

Don't use it for high-frequency time-series metrics. Azure Monitor Metrics is optimized for numeric metric storage with millisecond timestamps. Logs Ingestion is designed for structured log events, not high-frequency metric samples.

Related Skills

• azure-monitor-ingestion-py - Ingest custom logs from Python applications
• azure-monitor-query-java - Query ingested logs using KQL
• azure-mgmt-applicationinsights-dotnet - Manage Application Insights resources

Source

This skill is provided by Microsoft as part of the Azure SDK for Java. Learn more at the Maven Central page, explore the GitHub source code, and review the Azure Monitor Logs Ingestion API documentation for comprehensive guidance.