Enterprise Backup Architecture: Block-Level Engineering Deep Dive
ShadowCradle operates where most backup products don't dare go - every 64 KB block is individually hashed, deduplicated, optionally compressed, encrypted, and verified. A kernel-mode CBT driver tracks writes in real time. CoW layers make iSCSI snapshots mount in milliseconds. The result: 1–2 minute incrementals, mathematically verifiable integrity, and bare-metal recovery over the network.
Every Component Stands Alone
The iSCSI server, backup engine, block store, and recovery layer are fully independent subsystems. Deploy what your environment needs - skip what it doesn't. No monolithic dependencies, no forced lock-in.
iSCSI Server
Expose any snapshot as an iSCSI target - on the same machine or on a remote host across the network. Runs without touching the backup pipeline. Mount on a different machine to recover a downed server without moving a single byte first.
Backup Engine
Smart, Compressed, Local Raw, or Appliance mode - each is a self-contained pipeline. Switch modes per-volume, per-schedule, or per-policy. The engine never assumes what the consumer will do with the data it produces.
Recovery Layer
Restore from local chain, appliance, S3, or a combination of all three simultaneously. Multi-source ingestion means the fastest available source serves each block - local cache first, then LAN appliance, then WAN. No single point of recovery failure.
Bitmap-indexed, 64 KB blocks. Fastest incrementals.
Smart + LZ4 or Zstd. 40–70% smaller at full speed.
Sector-level copy. No transformation, maximum auditability.
LZ4-compressed POST to appliance. Agent is stateless.
Combine Compressed + Appliance. A local copy keeps your most recent data on-machine for near-instant recovery. The appliance receives the same data simultaneously - two independent copies, one backup run.
When the local cache lives on an SSD USB drive or internal NVMe, recent snapshots restore at full drive speed - no network bottleneck. For blocks not in the local cache (older snapshots or evicted data), the appliance fills them over the network in parallel. The result: the vast majority of any restore is served locally at 1–7 GB/s, with only the missing delta fetched from the appliance.
The 7-Phase Backup Pipeline
Every backup - full or incremental - passes through a deterministic seven-phase pipeline with timing at each stage. No shortcuts, no silent failures.
SCAN
Enumerate all partitions, detect filesystem types, compute stable partition UIDs - GPT GUIDs for GPT tables, blake3 fingerprints for MBR disks. ~2–5 seconds on a typical 1 TB drive.
MAP
Build compact block change bitmaps - per-block blake3 hashes for every partition. Multi-threaded: 40–60% faster on quad-core systems. Produces the foundation for sub-minute incremental detection.
DETECT
Select the fastest available change-detection method: CBT kernel driver (real-time, ~1 min), VSS/USN Journal (near-real-time, ~2 min), or per-block hash comparison (universal fallback). Skip 90–99% of unchanged blocks.
COPY
Read only changed blocks. Check the content-addressable store - identical blocks are referenced, not re-written. At 400 MB/s sustained throughput, 10 GB of actual changes copies in ~25 seconds.
VERIFY
Sample-verify written blocks by re-reading and comparing blake3 hashes. CRC32 validates every block change bitmap. Configurable sample rate - defaults to 5% for low overhead with strong assurance.
COMMIT
Commit the new snapshot atomically. Write delta manifests, updated block change bitmaps, and rich snapshot metadata. The appliance manages all snapshot sequencing and block relationships server-side - no client-side chain management required.
CLEANUP
Release VSS shadow copies and volume locks. Remove the atomic crash-recovery state file. The backup is now permanent, verifiable, and available for instant restore or iSCSI mount.
CBT Kernel Driver - Real-Time Block Tracking
ShadowCradle ships a kernel-mode CBT (Changed Block Tracking) driver for Windows. It operates as a volume filter that intercepts every write to disk at the IRP_MJ_WRITE level - the lowest possible OS hook - and marks the affected 64 KB blocks in a real-time bitmap stored in non-paged kernel pool.
When backup starts, the driver hands back only the dirty block list - no scanning, no hashing, no guessing. A 1 TB volume with 2% daily change completes its incremental backup in 1–2 minutes instead of 10–40 minutes.
| Method | How It Works | Speed | Time / 1 TB |
|---|---|---|---|
| Windows - CBT Kernel Driver | Kernel intercepts every IRP_MJ_WRITE. Real-time dirty bitmap in non-paged pool | Fastest | 1–2 min / 1 TB |
| Windows - VSS Volume Bitmap | VSS FSCTL_GET_VOLUME_BITMAP query at snapshot time | Fast | 2–3 min / 1 TB |
| Windows - NTFS USN Journal | File change journal converts to block ID ranges, skips 99%+ of volume | Fast | 2–5 min / 1 TB |
| Linux - inotify + CoW Snapshot | inotify events + CoW snapshot compare for mounted filesystems | Standard | 3–8 min / 1 TB |
| All - Block Hash Comparison | Universal: compare per-block blake3 hashes against prior snapshot bitmap | Fallback | 10–20 min / 1 TB |
Without CBT, USN Journal converts NTFS file change events to block ranges - skipping 99% of the volume on a typical overnight incremental. With the CBT driver installed, that number reaches 99.9%+, pushing the same job under 2 minutes.
Fully Chain Free Based Backup Streaming
The agent streams each backup directly to the ShadowCradle appliance. We independently store each section of each snapshot and map it to deliver minimum space usage, maximum performance, and optimal reliability. No chains. No rebuilds. No waiting.
The agent streams each backup directly to the ShadowCradle appliance. No local storage, no local block store - the appliance independently stores each section of each snapshot with full deduplication, snapshot management, and metadata. Hash blocks, send new ones, restore done.
- Zero local storage required on the protected machine
- Restore in seconds - appliance serves snapshots and blocks directly
- Appliance deduplicates across all clients - shared block store
- Interrupted backups resume automatically from last block sent
- Ideal for desktops, laptops, and servers without large local storage
Beyond Backup Storage - The Appliance Computes
Cloud-native backup stops at storing data. The ShadowCradle appliance is a full compute node that actively verifies, scans, estimates, and guarantees your backups. It boots Windows VMs from every snapshot to confirm they'll actually start when you need them. It runs antivirus on mounted volumes before any restore touches your network. It measures real restore throughput and backs it with a calculated RTO - not a guess.
On-Appliance Antivirus Scanning
Every backup can be mounted read-only and scanned by ClamAV or integrated AV engines before any restore touches your production network. Stops re-infection at the gate - even if ransomware bypassed the sentinel check at backup time.
Windows Boot VM Verification
The appliance spins up each Windows snapshot as a Hyper-V / KVM guest over the CoW iSCSI layer. If it reaches the login screen, the backup is marked "boot-verified." Runs automatically after every full backup - no manual testing required.
RTO Estimation & Guarantees
The appliance measures actual local restore throughput (block size × link speed × dedup ratio) and computes a real RTO for each snapshot. MSPs can export RTO reports per client and SLA-back them - "this 500 GB workload restores in under 90 minutes, guaranteed."
1–10 Gbps Local Restore Speed
Restores run over LAN - not the internet. A local NVMe appliance delivers 1–10 Gbps to recovery targets vs. the 50–200 Mbps ceiling of most cloud providers. A 500 GB restore that takes 6 hours from cloud finishes in under 10 minutes locally.
Full Air-Gap Operation
The appliance needs zero internet connectivity to protect, restore, or verify machines. Ransomware often targets cloud credentials first - an air-gapped appliance on a dedicated VLAN has no credentials to steal and no cloud surface to attack.
Cross-Client Block Deduplication
The appliance deduplicates across every protected machine in a single shared block store. A Windows Server OS block stored from machine A is never re-uploaded from machine B. Cloud-native solutions typically deduplicate per-tenant - not per-block across the fleet.
| Capability | ShadowCradle Appliance | Cloud-Only |
|---|---|---|
| Antivirus scan before restore | ✓ On-device ClamAV / AV engine scan | - Not available |
| Boot verification (Windows VM) | ✓ Auto-tested after every full backup | - Requires separate hypervisor infra |
| RTO estimation | ✓ Calculated + SLA-reportable per client | ~ Estimated from cloud benchmarks |
| Restore bandwidth | ✓ 1–10 Gbps local NVMe | ✗ 50–200 Mbps internet throttle |
| Air-gap / offline operation | ✓ Zero internet required | ✗ Requires continuous connectivity |
| Cross-client deduplication | ✓ Shared block store across all clients | ~ Per-tenant only (no cross-org dedup) |
| Instant iSCSI expose + PXE BMR | ✓ Built-in CoW iSCSI target on agent | - Requires separate BMR workflow |
| Pre-restore ransomware isolation | ✓ Scan on air-gapped VLAN before restore | - Network path exposes risk |
CoW Layer - iSCSI Snapshots in < 20 ms
When ShadowCradle exposes a backup snapshot as an iSCSI target (for live-mount or BMR), a Copy-on-Write (CoW) layer sits between the consumer and the read-only backup blocks. Writes go to the CoW file; reads hit the CoW first, falling back to the backup store.
Traditional pre-allocation (ftruncate 1 TB) blocks mount for 30+ seconds. ShadowCradle's lazy CoW uses a compact bitmap (1 bit per 512-byte sector - just 128 KB for a 1 TB volume) so the mount completes in ~20 ms - a 1,500× improvement.
Compact Block Tracking - Tiny Overhead, Maximum Speed
Every partition is tracked with a compact bit array - one bit per 64 KB backup block. A 1 TB volume's entire change map fits in about 2 MB of memory. Checking whether any single block has changed is an O(1) bit read regardless of volume size.
When comparing the current state to the prior backup, a single bitwise XOR across the two bitmaps instantly produces the changed block list - no full-volume scanning, no re-hashing everything. Just a few milliseconds of bit operations over a compact data structure.
Compress as You Go - Zero Extra Passes
Every block is compressed inline during the COPY phase - immediately after reading and before writing to the block store. No second pass, no temporary uncompressed files on disk. The block is transformed once, in a single streaming pipeline:
On a 1 TB volume with 1% daily change, only ~160,000 of 16,000,000 blocks have bit=1 - the other 99%+ are skipped entirely.
The 256×256 directory tree gives 65,536 top-level buckets - no single directory grows unwieldy even at petabyte scale. Any block is located with a constant-time O(1) path lookup using its hash as the address - no index table, no scanning.
iSCSI Expose + PXE Boot BMR
ShadowCradle doesn't just protect machines - it can become a boot target. From the Windows agent, any backup snapshot can be exposed as a local iSCSI target. A replacement machine boots from it over PXE - no USB, no imaging, no downtime waiting for a restore to complete.
The replacement machine is running the original OS in minutes - the CoW mount is instant. Full local restore happens in the background while the user works.
Pure network-based BMR. No imaging USB drives, no shipping media. Any PXE-capable machine can become the recovery target.
The CoW layer absorbs all writes. The backup snapshot remains byte-for-byte intact - multiple simultaneous recoveries can boot from the same snapshot.
Expose any snapshot - from a local chain, S3, or the ShadowCradle appliance - as an iSCSI target. BMR source is wherever the backup lives.
The CoW iSCSI mount takes ~20 ms. Machine B is running the failed server's OS within minutes - while the full restore happens in the background. Users may never notice the outage.
Local cache serves the most recent hot data first. The appliance fills remaining blocks over LAN. If both are available, they run in parallel - every block is fetched from the fastest reachable source.
Machine B doesn't need to be on the same LAN. If the appliance is reachable over WAN, recovery works remotely - a data center machine can recover a branch office server without physical presence.
The CoW layer absorbs all writes. The backup snapshot on the appliance stays byte-for-byte intact - multiple machines can recover from the same snapshot simultaneously.
Content-Addressable Block Store
Every block is identified by its blake3 cryptographic hash. Identical blocks across snapshots, machines, and time are stored exactly once. Incremental backups store only delta manifests - tiny reference files pointing at blocks already in the store. A 0.1% incremental produces a ~288 KB manifest instead of re-writing gigabytes.
Direct-to-Cloud Streaming for SAN & VM Volumes
Back up volumes exposed via iSCSI protocol - SAN LUNs, VM disk images, remote NAS - without requiring a full mount. Blocks stream from the iSCSI target through a local SSD/NVMe cache, hashed, compressed, and uploaded to S3 or the appliance in parallel.
Six Output Modes - One Engine
The same backup pipeline drives four standalone modes. Add the optional Local Cache hybrid for near-instant recovery without network dependency.
- Blocks stored once by blake3 hash - zero redundancy
- Delta manifests reference blocks, never duplicate them
- Point-in-time recovery: Full + ordered deltas
- 20–50% storage savings on typical incremental chains
- LZ4 (fastest, ~1.5 GB/s encode) or Zstandard (best ratio)
- Configurable compression level 1–9
- Combined dedup + compression: 40–70% storage reduction
- Codec stored per-block - mixed archives fully supported
- Snapshots stored independently in S3-compatible storage
- Local SSD cache (1–500 GB) for hot-path restore speed
- Parallel upload streams + exponential-backoff retry
- Deduplication + compression on every upload - pay only for unique data
- Agent streams each backup directly - appliance manages snapshots
- Appliance owns the block store, snapshots, and all metadata
- Resume-capable: only missing blocks re-uploaded on interruption
- Restore starts in seconds - no chain assembly required on client
- Unprocessed sector copy: no compression, no deduplication, no chain
- Full-fidelity output for forensic imaging, compliance, or audits
- NTFS sparse file optimization - only sectors with data written
- Maximum simplicity and auditability - what you see is what you get
Defense in Depth - At Every Layer
AES-256-GCM Per-Block Encryption
Every 64 KB block encrypted individually. 12-byte nonce + 16-byte auth tag per block. Block offset as AAD prevents block-swap attacks. Just 28 bytes overhead per block (0.04%).
TLS 1.3 in Transit
All appliance and cloud communications use TLS 1.3. Certificate pinning supported. Credentials and keys never travel unencrypted.
Blake3 Integrity Hashing
Every block hashed with blake3 - ~2 GB/s on modern CPUs (4× faster than SHA-256), cryptographically secure, and natively parallelizable. Any hash mismatch triggers automatic rejection.
Ransomware Sentinel Detection
15 sentinel files placed in 3 strategic OS locations. Pre-backup integrity check detects ransomware modification across 5 file types in < 50 ms before any backup proceeds.
Scrypt Key Derivation
Password-derived encryption keys via Scrypt (N=2¹⁴). Secrets never stored in plaintext. Config files enforce OS-level read permissions.
Atomic Write + Crash Safety
All checkpoint files written atomically via temp-file + os.replace(). Power loss mid-backup = clean resume from last checkpoint - never a corrupt or partial snapshot.
Overhead: just 28 bytes per 64 KB block (0.04%). Keys are derived via Scrypt (N=2¹⁴) and never stored in plaintext. Block offset as AAD prevents block-swap attacks without any extra computation.
Stop Infected Backups Before They Happen
ShadowCradle places 15 sentinel files across three strategic OS locations. Before every backup, all 15 are re-hashed (< 50 ms total) and compared to their baseline. Ransomware that modifies any sentinel file triggers an immediate halt.
Performance That Scales
One Agent - Every Major OS
- CBT kernel driver (IRP_MJ_WRITE intercept, real-time bitmap)
- VSS snapshot creation + NTFS USN Journal fallback
- Sparse file optimization (NTFS hole-punching)
- iSCSI target exposure - expose any backup for PXE/BMR
- Physical drive + partition backup (GPT and MBR)
- EXT4 / btrfs filesystem support
- inotify change event monitoring
- CoW snapshot comparison for iSCSI volumes
- Direct-to-cloud streaming (S3-compatible)
- iSCSI target backup from SAN and NAS volumes
- APFS filesystem support
- FSEvents change tracking
- Full partition and drive imaging
- S3 and appliance streaming modes
Prometheus-Native Metrics Out of the Box
Every backup emits rich per-job telemetry - blocks written, compression ratios, deduplication savings, throughput, and duration. Metrics retained 30 days by default and exportable to any Prometheus-compatible stack.
Why This Engineering Translates to Real Business Value
Smaller Storage Bills
Content-addressable dedup + Zstd compression cut storage 40–70% vs. naive full-image backups. You pay for unique data, not duplication.
Faster Backup Windows
CBT driver + USN Journal push 1 TB incrementals under 2 minutes. Backup jobs complete while production workloads run unaffected.
Verifiable Integrity
Every block carries a blake3 hash. You can independently re-hash any backup and confirm against stored metadata. No vendor trust required.
Crash-Safe by Design
Atomic checkpoints mean power failure mid-backup = clean resume - not a corrupt snapshot that silently fails on restore day.
Pre-Backup Ransomware Check
15 sentinel files catch encryption activity before any backup commits. You never inadvertently seal an encrypted snapshot into your retention chain.
Remote BMR Without Physical Media
iSCSI expose + PXE boot turns any replacement machine into a network recovery target. Full system available to users in minutes, restore to disk in background.