Azure Data Lake Storage Gen2: Hierarchical File Systems for Big Data

Azure Blob Storage is great for object storage, but what if you need real directory structures, POSIX-like permissions, and hierarchical namespaces? That's where Azure Data Lake Storage Gen2 (ADLS Gen2) comes in. It combines the scalability and cost of blob storage with the organizational benefits of a filesystem—perfect for big data analytics and data lake architectures.

What This Skill Does

The azure-storage-file-datalake-py skill provides a Python SDK for Azure Data Lake Storage Gen2, a storage service optimized for analytics workloads. Unlike flat blob storage, ADLS Gen2 organizes data hierarchically with real directories and files, supports POSIX-style access control lists (ACLs), enables atomic directory operations like rename and delete, and is optimized for big data tools like Apache Spark, Hadoop, and Azure Databricks.

The SDK offers four client types: DataLakeServiceClient for account-level operations, FileSystemClient for managing file systems (containers), DataLakeDirectoryClient for directory operations, and DataLakeFileClient for individual file operations. This hierarchy mirrors how you think about filesystems: account contains file systems, file systems contain directories, directories contain files and subdirectories.

You can create nested directory structures, set ACLs at directory and file levels, upload large files with append operations, list contents recursively, and move or rename directories atomically. All with the same Azure authentication and scalability you expect from Azure Storage.

Getting Started

Install the Data Lake Storage SDK:

pip install azure-storage-file-datalake azure-identity

Set your storage account URL (note the dfs subdomain for Data Lake):

export AZURE_STORAGE_ACCOUNT_URL="https://mystorageaccount.dfs.core.windows.net"

Here's a quick example creating directories and uploading a file:

from azure.identity import DefaultAzureCredential
from azure.storage.filedatalake import DataLakeServiceClient

# Connect to Data Lake service
credential = DefaultAzureCredential()
account_url = "https://mystorageaccount.dfs.core.windows.net"
service_client = DataLakeServiceClient(account_url=account_url, credential=credential)

# Create file system (like a container)
file_system_client = service_client.create_file_system("analytics")

# Create directory structure
directory_client = file_system_client.create_directory("data/2026/sales")

# Upload file
file_client = file_system_client.get_file_client("data/2026/sales/report.csv")
with open("local-report.csv", "rb") as data:
    file_client.upload_data(data, overwrite=True)

print("Directory structure created and file uploaded!")

The key difference from Blob Storage: you create actual directories, not just blob names with slashes. This enables atomic directory operations and better performance for hierarchical workloads.

Key Features

True Hierarchical Namespace: Unlike blob storage where "directories" are virtual (just prefixes in blob names), ADLS Gen2 has real directories. You can rename, move, or delete entire directory trees atomically—critical for data lake operations.

POSIX-Style ACLs: Set access control lists on directories and files with owner, group, and permission bits (rwx). ACLs inherit from parent directories, making permission management straightforward for large data hierarchies.

Atomic Directory Operations: Rename or delete entire directories in a single operation. No need to iterate through thousands of files—the service handles it atomically and efficiently.

Append-Based Uploads: For large files, use append_data() + flush_data() to stream data in chunks. This is more efficient than uploading everything at once and enables resumable uploads.

Recursive Listing: List all files in a directory tree with get_paths(recursive=True). The service handles the traversal efficiently, returning both files and subdirectories.

Metadata and Properties: Attach custom metadata to files and directories, query properties like size and last modified time, and track access patterns for optimization.

Full Async Support: A complete async API enables high-throughput data processing pipelines to handle thousands of concurrent operations.

Usage Examples

Create Nested Directory Structure:

file_system_client = service_client.get_file_system_client("datalake")

# Create multiple nested directories
paths = [
    "raw/logs/2026/02",
    "processed/analytics/2026/02",
    "archive/backup/2026/02"
]

for path in paths:
    directory_client = file_system_client.create_directory(path)
    print(f"Created: {path}")

Upload Large File with Append Operations:

file_client = file_system_client.get_file_client("data/large-dataset.csv")

# Open file for append
file_client.create_file()

# Upload in chunks
chunk_size = 4 * 1024 * 1024  # 4MB chunks
offset = 0

with open("local-large-file.csv", "rb") as f:
    while chunk := f.read(chunk_size):
        file_client.append_data(data=chunk, offset=offset, length=len(chunk))
        offset += len(chunk)
        print(f"Uploaded chunk at offset {offset}")

# Commit all appended data
file_client.flush_data(offset)
print(f"Upload complete: {offset} bytes")

List Directory Contents Recursively:

file_system_client = service_client.get_file_system_client("analytics")

print("Directory tree:")
for path in file_system_client.get_paths(path="data", recursive=True):
    icon = "📁" if path.is_directory else "📄"
    size = f" ({path.content_length:,} bytes)" if not path.is_directory else ""
    print(f"{icon} {path.name}{size}")

Set Access Control Lists (ACLs):

directory_client = file_system_client.get_directory_client("sensitive/data")

# Get current ACL
acl = directory_client.get_access_control()
print(f"Current owner: {acl['owner']}")
print(f"Current permissions: {acl['permissions']}")

# Set ACL with specific permissions
directory_client.set_access_control(
    owner="user-id-123",
    permissions="rwxr-x---"  # Owner: rwx, Group: r-x, Others: none
)

# Update ACL recursively for all children
directory_client.update_access_control_recursive(
    acl="user:user-id-456:rwx"
)

print("ACLs updated")

Move/Rename Directories Atomically:

# Rename a directory (atomic operation)
old_directory = file_system_client.get_directory_client("temp/data")
new_path = "datalake/archive/data"

old_directory.rename_directory(new_name=new_path)
print(f"Directory moved atomically to {new_path}")

Download and Process Files:

file_client = file_system_client.get_file_client("data/sales.csv")

# Download to memory
download = file_client.download_file()
content = download.readall()

# Process content
lines = content.decode('utf-8').split('\n')
print(f"File has {len(lines)} lines")

# Download to local file
with open("downloaded-sales.csv", "wb") as f:
    download = file_client.download_file()
    download.readinto(f)

Async Operations for High Throughput:

from azure.storage.filedatalake.aio import DataLakeServiceClient
from azure.identity.aio import DefaultAzureCredential
import asyncio

async def process_files():
    credential = DefaultAzureCredential()
    
    async with DataLakeServiceClient(
        account_url="https://mystorageaccount.dfs.core.windows.net",
        credential=credential
    ) as service_client:
        file_system_client = service_client.get_file_system_client("analytics")
        
        # Upload multiple files concurrently
        tasks = []
        for i in range(10):
            file_client = file_system_client.get_file_client(f"data/file{i}.txt")
            content = f"File {i} content".encode('utf-8')
            tasks.append(file_client.upload_data(content, overwrite=True))
        
        await asyncio.gather(*tasks)
        print("Uploaded 10 files concurrently")

asyncio.run(process_files())

Best Practices

Enable Hierarchical Namespace: ADLS Gen2 requires the hierarchical namespace feature enabled on your storage account. This is set during account creation and cannot be changed later. Verify this before starting development.

Use Append Operations for Large Files: Instead of buffering entire files in memory, use create_file() + append_data() + flush_data() for uploads over 100MB. This reduces memory usage and enables resumable uploads.

Set ACLs at Directory Level: Define permissions on directories and let files inherit them. This is more maintainable than setting ACLs on thousands of individual files.

Design Directory Structures Early: Changing directory structures later requires moving millions of files. Plan your hierarchy for query patterns—e.g., partition by date (/data/2026/02/15/) for time-series data.

Use Async for Big Data Workloads: When processing thousands of files in a data pipeline, use the async client to maximize throughput. Sync operations are fine for simple scripts.

Integrate with Azure Databricks: ADLS Gen2 is optimized for Spark and Databricks. Mount file systems as Databricks File System (DBFS) paths for seamless access in notebooks.

Monitor with Azure Monitor: Track storage metrics, request counts, and latency. Set up alerts for unusual patterns that might indicate issues with data pipelines.

Implement Lifecycle Policies: Use Azure lifecycle management to automatically archive or delete old data. This keeps storage costs down and performance high.

When to Use This Skill

Perfect for:

• Data lake architectures for big data analytics


• ETL pipelines processing hierarchical datasets


• Applications requiring POSIX-style permissions


• Integration with Spark, Hadoop, or Databricks


• Directory-heavy workloads (millions of files in nested folders)


• Data science projects with organized datasets


• Compliance scenarios requiring granular access control

Use Blob Storage instead for:

• Simple object storage without directory operations


• Web applications serving static content


• Scenarios where flat namespace is sufficient


• Applications already using blob storage APIs


• Cost optimization (ADLS Gen2 has slightly higher transaction costs)

ADLS Gen2 adds complexity and cost compared to basic blob storage. Only use it when you actually need hierarchical namespaces, ACLs, or big data tool integration.

Related Skills

Explore the full Azure Data Lake Storage Gen2 skill: /ai-assistant/azure-storage-file-datalake-py

Source

This skill is provided by Microsoft as part of the Azure SDK for Python (package: azure-storage-file-datalake).

---

Building a data lake or working with hierarchical datasets? ADLS Gen2 brings filesystem semantics to cloud-scale storage.