Cyber Threat Intelligence & OSINT·5 min read·October 2026

Fast Cyber Threat Graphs on Serverless Cloud Run & DuckDB

Why We Replaced Managed Relational Clusters with Ephemeral Analytical Graphs for High-Volume OSINT & Brand Takedowns

Intended Audience:CISOs, Security Data Architects, Threat Intelligence Leads

Share & Discuss

⚡Bottom Line Up Front (BLUF)

Always-on relational and graph database clusters suffer from connection exhaustion and high idle TCO during bursty OSINT crawling. Embedding in-memory DuckDB analytical graphs inside ephemeral Cloud Run workers over Parquet snapshots delivers fast multi-hop DNS/ASN enrichment while scaling to zero between crawl waves.

Executive Highlights:

Phishing, typosquatting, and brand impersonation campaigns occur in violent bursts—requiring ingestion elasticity without paying for 24/7 provisioned cluster capacity.
Embedding DuckDB inside serverless Cloud Run containers over read-only Parquet threat graphs eliminates network I/O round-trips and database connection pool exhaustion.
Pairing deterministic Levenshtein/homoglyph graph scoring with multimodal LLM verification automates analyst takedown dossier preparation.

Executable Companion Models

Explore this brief's interactive React Flow topology & weighted trade-off matrix

📐 Cyber Threat OSINT→⚖️ Cyber & Streaming Matrix→

Architectural Drivers & Industry Dynamics

Burst Elasticity vs. Idle Compute Burn

CertStream and SERP crawlers generate sudden spikes of candidate domains. Serverless containers with embedded columnar memory scale horizontally and cost nothing when idle.

Zero-Network-Hop Graph Traversal

Correlating A/MX/TXT records, WHOIS registrars, TLS certificates, and Autonomous System Numbers (ASNs) in-process via DuckDB avoids the latency penalty of remote SQL/Cypher queries.

ISM-Aligned Out-of-Band Sandboxing

Ephemeral headless browser workers inspect evasive fast-flux DOMs in isolated sandboxes without exposing enterprise IP ranges or allowing malware persistence.

Side-by-Side Trade-off Matrix

Architectural Dimension	Serverless DuckDB + Cloud Run (SentrySurface)	Traditional RDS / Managed Graph Cluster
Multi-Hop DNS/ASN Lookup Latency	Low latency (in-process vectorized memory)	Higher latency (network I/O + join contention)
Burst Concurrency Handling	Linear horizontal scale (Isolated worker memory)	Connection pool exhaustion & lock contention
Idle Infrastructure TCO	Scales to zero (pay-per-use + object storage)	Always-on provisioned HA instances
Data Portability & Archival	Open Parquet / Iceberg over GCS / S3 / OneLake	Proprietary storage engines & snapshot lock-in
Evidentiary Audit Trail	Immutable Parquet partitions + signed JSON dossiers	Mutable row tables requiring separate audit logs

1. The Relational Bottleneck in High-Velocity OSINT

When monitoring global certificate transparency logs (CertStream), social marketplaces, and SERP deltas for brand impersonation, ingestion is highly spiky. Traditional architectures route every discovered URL into a central PostgreSQL or Neo4j instance to enrich DNS, WHOIS, and ASN relationships. Under a sharp burst, connection pools saturate, IOPS throttle, and enrichment latency spikes—delaying critical takedown actions.

2. The Ephemeral DuckDB + Parquet Graph Pattern

In SentrySurface.io, we inverted the database model. Instead of pushing compute to a monolithic database server, we pull compact, ZSTD-compressed Parquet threat-intelligence snapshots into memory on Cloud Run container warm-up: • **Decoupled Buffering:** Google Cloud Pub/Sub absorbs crawler spikes with dead-letter guarantees. • **In-Process Analytical Joins:** Each worker executes vectorized DuckDB SQL joins across passive DNS, ASN reputation, and Levenshtein homoglyph distance matrices without network round-trips. • **Multimodal Intent Triage:** Only domains crossing the structural risk threshold invoke Gemini multimodal classification to inspect visual brand logos and credential-harvesting DOM forms.

3. Automated MITRE ATT&CK Takedown Dossiers

Detection without rapid remediation leaves enterprises exposed. When the pipeline scores a domain below the trust threshold (<0.2), it automatically compiles a cryptographically signed registrar takedown dossier containing the full DNS chain, timestamped sandboxed DOM screenshot, MITRE ATT&CK technique mapping, and abuse contact routing—reducing manual analyst triage.

Strategic Recommendation

The Architectural Verdict

Stop over-provisioning monolithic relational or graph clusters for bursty security telemetry and OSINT pipelines. Adopt serverless container workers backed by embedded DuckDB and open Parquet lakehouse storage to achieve rapid graph enrichment at a lower cloud run-rate.

Recommended Action Items:

✓Profile your current SIEM or threat-intel enrichment queries to identify high-cost remote database joins suitable for columnar Parquet offloading.
✓Benchmark an embedded DuckDB worker on Cloud Run or Azure Container Apps against your existing relational lookup pipeline.
✓Standardize threat indicator archives in partitioned Parquet with strict row-level tenant isolation for 90-day longitudinal hunting.

Execution & Advisory Next Steps

Ready to implement or review this architecture?

Available for Principal Contract placements, milestone SOWs, and fixed-scope architecture audits in Sydney & Canberra.

Open Contact Gateway →