Data Analytics

Serverless Apache Spark on Google Cloud: Architecture Choices & AI Troubleshooting

Wed, 19 Aug 2026 16:00:00 +0000

In modern enterprise data engineering, Apache Spark remains a cornerstone framework for processing massive datasets at scale. However, managing infrastructure such as provisioning clusters, tuning YARN configurations, and avoiding costs for idle hardware often detracts from what matters most: building resilient data pipelines. Google Cloud addresses this operational overhead via its Managed Service for Apache Spark, offering flexible deployment modes of serverless and managed clusters tailored to specific operational needs.

This technical guide walks through the architectural decision matrix for deploying Spark on Google Cloud, details resource and cost optimization techniques, and demonstrates how to apply built-in Gemini Cloud Assist to rapidly troubleshoot and resolve serverless batch pipeline failures. While there is benefit to reading these three parts in a sequence, each one can be read independently and add value to how you approach Spark development on Google Cloud.

Part 1: Choosing your Apache Spark deployment model

When launching Spark workloads on Managed Service for Apache Spark, the first major decision point is evaluating whether to construct traditional managed clusters or transition to a zero-management, serverless infrastructure footprint.

Decision #1: Managed clusters vs. serverless

*Created using Nano Banana 2 in Gemini Enterprise Agent Platform

Choosing between traditional Managed Spark clusters and serverless depends on ecosystem requirements, infrastructure control needs, and financial utilization patterns:

Workload frequency, latency sensitive workloads & financial fit: For continuous, highly predictable, 24/7 streaming or batch processing pipelines where cluster nodes maintain constant high utilization baselines (80%+) or when the workflow’s accumulated startup time risk meeting SLA target, a permanently running, finely tuned traditional cluster, with custom YARN autoscaling rules, can sometimes be more cost-predictable. Conversely, for intermittent, bursty, ad-hoc, or orchestrator-triggered pipelines, Managed Spark serverless is highly optimal, eliminating operational management, requiring less planning time and ensuring you don’t pay for idle compute time.
Ecosystem & component requirements: Managed Spark serverless is strictly optimized for Apache Spark 3.x+ codebases. If your processing pipeline relies on other ecosystem components such as Apache Flink, Presto/Trino, Hive LLAP, or Apache HBase, or if you are locked into a legacy Spark 2.x codebase, you must use Managed Spark clusters.

Infrastructure customization needs: Managed Spark serverless abstracts away the underlying virtual machine (VM) layer. If your workload mandates deep OS-level hardware tuning, custom OS initialization actions, root SSH access to instances, specific local SSD configurations, or custom machine shapes, a traditional cluster is required. Note that serverless does support custom Docker container images for bundling specific application-level libraries.

Decision #2: Serverless interactive sessions vs. serverless batches

*Created using Nano Banana 2 in Gemini Enterprise Agent Platform

Once you select the serverless deployment mode, you must choose the appropriate execution model based on your development stage and operational requirements. Managed Service for Apache Spark provides two options for running serverless workloads:

Serverless interactive sessions

Interactive sessions are great for iterative and exploratory use cases. You write blocks of code, inspect intermediate DataFrames, modify variables, and generate visualizations with your dataset held warm in-memory.

Primary interface: Designed for human-in-the-loop interaction. Developers execute code cell-by-cell using their IDE of choice, such as Colab, Gemini Enterprise Agent Platform Workbench, Antigravity, Jupyter notebooks, etc.
Idle cost profile: Compute resources remain active to support immediate execution during developer thinking time, which can incur some idle compute charges if sessions are left inactive.

Serverless batches

Batches are useful when you know what you want to run, and need automated, non-interactive execution. The engine runs fully completed, packaged PySpark scripts (.py) or Java/Scala application files (.jar) from start to finish without manual human intervention.

Primary interface: Managed by automated orchestrators, such as Managed Service for Apache Airflow, Cloud Scheduler, or CI/CD pipelines.
Idle cost profile: Billed strictly for the duration of the run. Compute resources are provisioned on-demand, run the script, and immediately shut down upon completion to prevent idle costs.

The development-to-production lifecycle

These execution options are designed to work together as a natural pipeline lifecycle. During the initial development phase, you open a serverless interactive session within your notebook interface to explore datasets, clean schemas, and prototype transformations. Once your logic is validated and the transformations are finalized, you package the code into a Python script and schedule it as a serverless batch job orchestrated by Managed Service for Apache Airflow for production execution. This transition minimizes ongoing development costs while maintaining operational reliability.

Part 2: Advanced performance tuning and DCU cost optimization

While serverless Managed Spark eliminates the operational overhead of cluster maintenance, running production enterprise-grade pipelines on default settings can result in performance bottlenecks or budget waste. Resource allocation must be explicitly declared during submission using runtime configuration properties to maintain an efficient Data Compute Unit (DCU) burn rate.

Google recently introduced history-based autotuning. In the context of serverless, this capability automatically applies optimizations based on best practices and historical execution. It does this by grouping recurring batch workloads into what Google calls cohorts. The autotuner analyzes the telemetry and statistics from previous runs under that same cohort name to figure out where the bottlenecks are.

Customizing driver and executor shapes

By default, serverless batches allocate generic specifications (4 cores and 16,000MB RAM). This can cause critical efficiency issues depending on the nature of the application:

The Memory-Bound job: Pipelines processing highly uncompressed data volumes may hit Out-Of-Memory (OOM) errors and crash. To counter this, increase heap sizing independently using spark.driver.memory and spark.executor.memory.
The Compute-Bound Job: Processing-intensive jobs running mathematical modeling or heavy tokenization might saturate CPUs while leaving expensive RAM sitting idle. Fine-tune processing concurrency per instance by explicitly adjusting spark.driver.cores and spark.executor.cores.

Remember that by default increasing cores, automatically provisions a proportionate baseline of memory to match the vCPU-to-RAM ratio. This is why overriding the values for both cores and memory is critical

Controlling autoscaling boundaries

Managed Spark serverless dynamically scales up and down the number of active executors based on backlogged tasks. However, unconstrained scaling can lead to budget overruns if a rogue code loop or unoptimized cartesian join is introduced.

As a defensive guardrail, always declare an explicit upper limit using spark.dynamicAllocation.maxExecutors. This acts as your budget deadman-switch. By capping this at a reasonable ceiling, you guarantee that even if the code behaves sub-optimally, the job will never scale past a fixed infrastructure footprint.

High priority (SLA-driven): Set maxExecutors to a higher ceiling to allow resource bursting and minimize overall runtime duration.
Low priority (nightly batch): Set maxExecutors to a low, tight ceiling. The workload will run longer but will consume a predictable, flat, cost-efficient stream of DCUs.

Managing shuffle storage efficiency

When execution involves wide transformations like groupBy(), join(), or distinct(), data must be redistributed across the network, generating intermediate disk writes known as shuffle storage.

Spark defaults to a static setting of 200 partitions (spark.sql.shuffle.partitions). If you are processing a massive, multi-gigabyte dataset, 200 partitions means each individual chunk will be too large. When a partition's size exceeds available executor RAM (e.g., a 1GB partition trying to process inside 0.5GB of assigned heap space), data spills onto disk. This slows execution and incurs additional billing fees for premium or standard shuffle storage blocks. A helpful rule of thumb: Dynamically scale your partition parameters based on total data size so that each partition handles roughly 100MB to 200MB of data in memory. This may require a few iterations before the optimal results are achieved.

The above properties are the main tunable properties. Additional Serverless runtime configuration properties can be found in this link

Part 3: Operational diagnosis with Gemini Cloud Assist

When automated data pipelines fail in production, data engineers are traditionally forced to spend hours sifting through verbose, disjointed log files across drivers and executors. Managed Service for Apache Spark addresses this friction by natively integrating Gemini Cloud Assist into the Google Cloud console, allowing engineers to diagnose and resolve failures using natural language.

To illustrate this operational shift, we examine the typical troubleshooting lifecycle for a failed PySpark ETL pipeline that reads customer transaction data from a Google Cloud Storage (GCS) bucket, applies transformations, and encounters unexpected runtime errors.

Stage 1: Diagnosing missing execution parameters

During the initial execution attempt of a new pipeline, the batch job status switches from pending to running, and ultimately ends in a failed state with a generic exit message: Application failed with exit code 1.

Rather than manually querying Cloud Logging or navigating through multiple sections of the console, the engineer can locate the error log and select the ‘Investigate log’ option. This action opens a native conversation pane where Gemini Cloud Assist automatically analyzes the driver telemetry and system logs.

In this scenario, the assistant explains in plain English that the PySpark script failed because required runtime arguments (such as the source GCS bucket path) were omitted during submission. It instantly identifies the exact lines in the script expecting these arguments, eliminating the need to read through the stack trace.

Stage 2: Resolving schema and data type anomalies

Once the missing arguments are resolved and the job is re-submitted, the pipeline runs but encounters a secondary data anomaly. In high-volume ingest pipelines, upstream source files frequently contain corrupted records or formatting inconsistencies.

Upon the second failure, the engineer again prompts Gemini Cloud Assist to investigate the logs. The assistant identifies a TypeError and pinpoints the exact DataFrame transformation causing the crash: a division operation (df['amount'] / df['transaction_id']) that failed because the schema auto-inferred the columns as strings.

Additionally, the assistant scans the underlying GCS file data to identify the root cause: non-numeric anomalies (such as text strings within numerical cells) in the source dataset.

Stage 3: Generating and deploying verified code fixes

Rather than manually rewriting the PySpark logic to cast schema types and catch null values, the engineer can prompt Gemini Cloud Assist directly to generate a resilient solution:

User Prompt: "Suggest how to rewrite the code to divide the amount by quantity instead of transaction_id. In addition, add logic to skip invalid records without failing the process."

The assistant generates the corrected PySpark code block, using resilient casting and null-handling functions (such as coalesce and try_cast).

By implementing this corrected script, the orchestration pipeline can filter out bad source records smoothly without crashing the entire batch run. The subsequent execution completes successfully, preserving data freshness SLAs.

Unlock serverless Apache Spark: Benefits and next steps

Managing data processing pipelines should not require a deep specialization in infrastructure configuration. By pairing the hands-off scale of serverless batches with explicit resource tuning — such as dynamic allocation caps and calculated shuffle sizing — data teams can maintain strict control over performance and cost profiles. When failures do occur, integrating Gemini Cloud Assist directly into your logging workflows transforms complex troubleshooting from a manual log-sifting exercise into a rapid, automated cycle.

To start putting these architectures into practice, you can explore the Managed Service for Apache Spark documentation and execute a serverless batch directly in the Google Cloud console.

For a deep architectural analysis of these concepts, get instant access to A practitioner’s guide to Apache Spark® in the agentic era. This guide includes step-by-step workflows, Codelabs, and runnable PySpark and Terraform templates directly from our GitHub repository. If you are new to Google Cloud, you can test these blueprints on serverless and managed clusters at zero cost by signing up for a free trial with $300 in credits.

How to modernize Apache Hive using Google Cloud’s Lakehouse runtime catalog

Wed, 19 Aug 2026 16:00:00 +0000

For over a decade, the Apache Hive Metastore (HMS) has served as the de facto metadata authority for big data analytics. Whether it was deployed on Hadoop clusters, self-managed Compute Engine VMs backed by MySQL or PostgreSQL, HMS provided the central schema registry that let Apache Spark, Presto, and Hive query raw .parquet and .orc files.

However, as enterprise data architectures scale to petabytes and span multiple query engines (such as Google Cloud Managed Service for Apache Spark, BigQuery, and Trino), legacy Hive Metastores often become critical operational bottlenecks.

In this blog, we explore why legacy metastores struggle in modern cloud environments at agent scale, and show you how the serverless Google Cloud Lakehouse runtime catalog that we introduced last year can help: Built on the open Apache Iceberg REST catalog specification, it is a runnable, zero-data-copy migration solution to help you transition your production Hive tables in minutes.

The challenges of legacy Hive Metastores

When speaking with data engineers and infrastructure leads running production analytics at scale, three core pain points consistently emerge with standalone Hive Metastores:

Architectural and scaling bottlenecks
Standalone HMS deployments rely on relational database backends (such as MySQL or Postgres) to track table schemas, partitions, and storage locations. As data lakes grow to hundreds of thousands of partitioned tables, partition pruning and bulk listing operations lead to key performance bottlenecks on the relational database. A complex Spark job requesting partition metadata can spike metastore CPU to 100%, causing cluster-wide query delays or out-of-memory (OOM) failures.

Siloed identity and security governance
Legacy metastores were designed around perimeter-based Hadoop security models. Enforcing modern granular data governance — such as table-level access control lists (ACLs) — across both Apache Spark compute jobs and enterprise SQL engines like BigQuery requires maintaining fragmented, duplicated security policies across two distinct control planes.

Operational overhead and total cost of ownership (TCO)
Managing high-availability MySQL/Postgres instances, patching HMS daemons, tuning JDBC connection pools, and paying for idle instance-based metastore servers creates unnecessary operational toil for data platform teams, whose time is better spent building high-leverage data products for agents.

The solution: Lakehouse runtime catalog

To solve these architectural bottlenecks without forcing data engineers to rewrite petabytes of existing storage payloads, we built the Lakehouse runtime catalog with support for Iceberg Rest Catalog and Hive Catalog.

The Lakehouse runtime catalog is a fully serverless, highly available, and unified metadata registry designed from the ground up to support both legacy Hive/Parquet tables and modern open table formats like Apache Iceberg. By natively implementing the Apache Iceberg REST Catalog specification, the Lakehouse runtime catalog decouples metadata discovery from compute engines. This decoupling of the catalog and compute engines ensures multiple Iceberg compatible engines can access the same data in a zero copy fashion thereby reducing the need for customers to maintain multiple copies of the data and enables them to take their workloads to production sooner.

This approach offers a number of architectural benefits:

Multi-engine interoperability: Once registered, tables are immediately discoverable and queryable across Google Cloud Managed Spark, BigQuery, and open-source engines via standard REST interfaces.
Open APIs: Supports Iceberg Rest Catalog and Hive Catalog which enables different teams to use their preferred analytics tools on a single, unified dataset.
Zero-data copy: Table definitions point directly to your existing data in Google Cloud Storage. You do not move, rewrite, or duplicate your underlying data.
AI-powered governance, security and trusted context: The Lakehouse runtime catalog integrates directly with Knowledge Catalog and Cloud IAM, allowing you to define trusted context for your agents and table-level security that apply consistently across all compute engines. Further it supports key authorization mechanisms, such as credential vending. This means you can access your tables without needing direct access to the files in the underlying Cloud Storage bucket.
Enterprise-readiness, scale and reduced TCO: Backed by Google’s planet-scale infrastructure and Spanner, enabling your metadata to scale with your data. Support for Cloud Storage dual-region and multi-region buckets enables failover use cases. It also provides reduced TCO due to serverless and no-ops environments, and scalability for any workload size.

Zero-copy migration from legacy Hive Metastore in action

To demonstrate how smooth cutover is in practice, we have provided a capability that lets you modernize your legacy self-managed Hive Metastore to the Google Cloud Lakehouse. This capability connects directly to your legacy Hive Metastore, extracts external table definitions and partition maps, and registers them cleanly into the serverless Lakehouse catalog and then start using the data in Google Managed Spark, BigQuery and Conversational Analytics agents with Gemini.

Modernize to the Lakehouse and immediately tap your data in key agentic journeys

Ready to modernize your data architecture?

Modernizing from a legacy Hive Metastore to Google Cloud’s Lakehouse minimizes data silos across analytics engines and agents, unifies multi-engine governance, provides trusted context to your agents and slashes operational TCO. In other words, it helps prepare your modern cloud environments to operate at agent scale.

Get started and migrate your Apache Hive Metastore tables to Google Cloud today, and get ready for the agentic era. Learn more about Google Cloud Lakehouse here.

How Box is unlocking multimodal enterprise agents with Gemini Embeddings 2

Tue, 18 Aug 2026 16:00:00 +0000

Enterprise content management is experiencing its biggest architectural shift since the cloud migration era.

For years, enterprises have stored trillions of gigabytes of critical data in Box: financial models, clinical trial protocols, M&A due diligence rooms, engineering schematics, and legal compliance playbooks. Up to this point, text-based search and retrieval-augmented generation (RAG) have successfully unlocked the vast narrative knowledge within these repositories, establishing a powerful and highly effective baseline for enterprise AI intelligence.

Traditional RAG architectures have mastered text processing, but the agentic era demands more. The next logical evolution is to extend this framework to capture the inherently multimodal, deeply spatial, and highly structured elements that exist alongside text. While text embeddings excel at indexing prose, multimodal architectures unlock a major new capability: For example, they preserve the strict row-column semantics of financial tables, interpret visual evidence like clinical data, and map the logic of multi-page flowcharts without losing their spatial layout.

To deliver next-generation capabilities that can handle the vast universe of digital content, Google Cloud and Box are integrating advanced multimodal capabilities into Box's Agentic Platform, powered by Gemini Multimodal Embeddings 2 merging Box’s industry-leading Intelligent Content Management platform with Google Cloud’s advanced AI embeddings.

Benefits of improved embedding: Extending the dimensions of document content

Preserving visual and spatial geometry: Complex document elements like multi-column tables or financial matrices rely on their spatial layout to convey meaning. Converting these elements into a flat string of text can disassociate column headers from their corresponding data points. Multimodal embeddings allow systems to interpret the document exactly as a human does, maintaining the integrity of spatial relationships.
Illuminating the visual modality: Enterprise documents are filled with visual indicators: technical charts, process flowcharts, branding assets, and product photography. Multimodal capabilities ensure that these elements are no longer invisible to search systems, allowing users to query images and text simultaneously.
Connecting hybrid file formats: Real-world business workflows rarely live in a single document format. An agent may need to cross-reference a PDF policy, a spreadsheet tracking log, and a presentation deck. Extending RAG with multimodal embeddings creates a unified understanding across these varied formats.

The Architectural Solution: Gemini Multimodal Embeddings 2

Google Cloud’s Gemini Multimodal Embeddings 2 introduces a unified, multimodal vector space capable of embedding text, raster images, document pages, rendered spreadsheet tables, and visual charts into the same semantic representation space.

Key product capabilities unlocked by gemini-embeddings-2:

Crossmodal retrieval (text-to-visual / visual-to-text): Enables natural language queries to retrieve highly specific visual components, such as locating a target chart or diagram within a massive library of slides, without requiring manual tagging.
Layout-aware document embedding: Rather than breaking files into arbitrary text blocks, the system can embed document page renderings directly, preserving visual hierarchies, callout boxes, and structural context.
Heterogeneous format bridging: Native support for seamlessly bridging content across .docx, .xlsx, .pdf, .pptx, .png, and .csv without losing modality-specific structural information.

Three core patterns of multimodal enterprise agents

By leveraging multimodal embeddings within Box, we have identified three uniqueprimary design patterns that illustrate how organizations can extend traditional RAG to support complex, visual workflows.

Pattern 1: Complex financial & analytical reporting

The challenge

Corporate finance, research, and audit teams analyze highly structured documents where vital data resides in embedded tables, growth charts, and footnote annotations. Text-only indexing can separate these numbers from their context, making automated analysis challenging.

The multimodal advantage

Structural alignment: The embedding model captures the physical structure of tables and charts, allowing financial agents to understand that a column header applies to a specific row of metrics.
Visual trend analysis: Agents can cross-reference written summaries with visual trends in accompanying bar or line charts, identifying and pointing out discrepancies between written claims and source data.
Contextual sourcing: Users can query complex portfolios and instantly retrieve the exact page, table, or chart supporting a specific metric.

Pattern 2: Multimodal clinical decision support & assisted diagnosis

The challenge

In healthcare and clinical environments, critical patient data is fragmented across vastly different, unstructured visual and textual formats — ranging from external physical photos (visual evidence) and microscopic pathology slides (lab reports) to structured risk matrices (triage grids). Traditional text-based systems or isolated analysis tools cannot synthesize these cross-modal relationships simultaneously, which can delay critical diagnoses or risk missing immediate, life-threatening procedural complications.

The multimodal advantage

Cross-modal clinical synthesis: Evaluates physical symptoms alongside cellular-level laboratory evidence simultaneously by indexing clinical photos, histopathology imagery, and triage grids into a single space.
Granular anomaly identification: Connects niche visual patterns under a microscope (like parasitic cyst walls) with medical knowledge to rapidly isolate rare conditions.
Risk-aware decision support: Cross-references findings against triage frameworks to deliver instant warnings about immediate patient risks, such as life-threatening anaphylactic shock.

Pattern 3: Cross-document multimodal synthesis & data reconciliation

The challenge

Enterprise information is fragmented across disconnected files and formats (e.g., PDF minutes, Excel charts, PNG flyers, and email threads). Traditional tools analyze these files in isolation, failing to connect the dots when verifying details or resolving data contradictions across independent documents.

The multimodal advantage

Cross-file synthesis: Connects information across entirely different formats (PDFs, spreadsheets, images, emails) simultaneously to answer complex business queries.
Conflict resolution: Flags and resolves contradictions between assets, such as catching outdated pricing on an image by cross-checking it against the latest financial spreadsheets.
Visual-to-text auditing: Audits visual or scanned files against text-based records (e.g., verifying a signed PDF contract against a legal review email) to catch missing clauses or changes.

The future of agentic enterprise content management

The integration of gemini-embeddings-2 into Box’s Agentic Platform is an important new capability to improve the next era of content intelligence. Multimodal embeddings help Box to move beyond basic search to active, intelligent collaboration.Box's Intelligent Content Management platform represents a fundamental shift in enterprise AI infrastructure — moving beyond passive document storage to deliver a governed, semantically indexed reasoning layer where AI agents can interrogate, cross-reference, and act on content with full compliance and security controls already in place.

Powered by multimodal embeddings and a suite of native AI agents spanning search, metadata extraction, research, analysis, and composition, Box enables organizations to proactively surface insights such as flagging stale pricing data, expiring contract clauses, or cross-document contradictions before they become business risks. For high-complexity industries like financial services, life sciences, and legal operations, Box's ability to reason across text, tables, charts, and images makes multimodal understanding a competitive requirement.

Designed to interoperate with the broader enterprise AI ecosystem, Box serves as the single governed content foundation that ensures every AI-driven workflow is grounded in authorized, auditable enterprise data.

When you think about it, the enterprise data landscape was always multimodal. Now we have the technology to make the most of it. By integrating gemini-embeddings-2, Box helps its users unlock unprecedented value from unstructured enterprise content. Product leaders who embrace multimodal-first architectures, rigorous precision benchmarking, and audit-ready grounding will lead the next wave of enterprise productivity and innovation.

The team would like to thank Ken Ikeda, Afshaan Mazagonwalla, and Samip Thakkar for their work on this project.

Building cost-effective, high-throughput gen AI workflows in Google Dataflow

Tue, 18 Aug 2026 16:00:00 +0000

Real-time streaming pipelines are the operational backbone of modern enterprises, continuously processing everything from customer support interactions to transaction logs. Traditionally, streaming DAGs are static; once deployed, their processing logic and execution paths are fixed. However, by integrating generative AI agents, we can move beyond static logic to adaptive execution. This allows streaming workflows to dynamically construct plans, query databases, and trigger custom remediation paths at runtime depending on the content of the data.

For example, when a customer sends an angry message about a damaged order, a pipeline shouldn't just log the error or flag a dashboard. It should look up the order in the database that holds customer order and inventory records, decide on a remediation action (like shipping a replacement or issuing a refund), email the customer, and log the final resolution.

However, streaming systems face a fundamental engineering hurdle when executing gen AI workflows: scale, latency, and cost. Sending every raw event directly to a heavyweight model or multi-step agent equipped with external database and email tools is prohibitively expensive, introduces high latency, and quickly exhausts API rate limits.

This pattern addresses the scale and complexity challenge by combining Google Dataflow, Google Cloud's fully managed, serverless execution service for Apache Beam, and the Agent Development Kit (ADK) to build a hybrid streaming pipeline. By using a lightweight, CPU-bound machine learning model upstream to filter and qualify events, we keep the pipeline highly cost-effective, routing only the complex cases to the downstream agent. There, the agent dynamically decides what actions to take, introducing dynamic branching to the stream without hardcoding thousands of conditional steps into the pipeline's static DAG.

A universal blueprint for high-volume streams

While we use a customer support triage scenario below, this pre-filter + agentic action pattern is a universal paradigm. It applies to any stream where a high volume (>9X%) of events are routine, and only a small number require complex, contextual reasoning.

IT Operations & DevOps: Filtering millions of routine system logs on CPU, and triggering an agent to run diagnostics and open bug tickets only when a critical anomaly is flagged.
Financial Fraud Triaging: Passing millions of transactions through lightweight, local rules, and calling an agent to execute multi-database lookup tools only for highly suspicious patterns.
Industrial IoT: Monitoring normal telemetry on the edge, and routing erratic spikes to an agent to coordinate equipment shutdowns and email field engineers.

The architecture: Why pre-filter streaming events?

In a high-throughput stream, the vast majority of messages do not require complex reasoning or remediation. They might be positive feedback, neutral inquiries, or simple queries.

Routing every single event to a heavyweight LLM workflow creates three primary bottlenecks:

API cost: Frontier models charge per token. Under high throughput, cost scales linearly with stream volume.
Latency: Multi-step workflows (which involve database lookups and external API calls) take seconds, creating a bottleneck in streaming DAGs.
Quotas: External APIs have strict rate limits that streaming workers can easily exhaust.

To prevent this, we build a pre-filtered pipeline in Apache Beam/Dataflow:

Pipeline flow

Ingestion: Read raw customer messages from Google Pub/Sub.
Lightweight sentiment classifier (CPU): Run all messages through a lightweight, CPU-based Hugging Face model (distilbert-base-uncased-finetuned-sst-2-english) using Apache Beam’s RunInference transform. This executes locally on the Dataflow worker CPUs, avoiding external API costs.
Pre-qualification Gate: A simple DoFn filters the stream. Messages with POSITIVE or NEUTRAL sentiment are acknowledged and dropped.
Automated Remediation (ADK): If and only if a message is classified as NEGATIVE, we trigger the gen AI agent backed by gemini-3.5-flash using the ADKAgentModelHandler. The agent uses tools to look up the user in BigQuery, fetch orders, choose a remediation plan, and send a notification email via the Gmail API.

Adaptive execution: Making the Beam DAG dynamic

In traditional streaming architectures, the pipeline's Directed Acyclic Graph (DAG) is rigid. Once deployed to Dataflow, the sequence of transforms is set. If you need to handle new types of alerts or change how specific events are routed, you have to modify, test, and redeploy the entire pipeline.

By placing a gen AI agent downstream of our sentiment pre-filter, we introduce a dynamic, adaptive node inside the static DAG.

For the 95% of records that are positive or neutral, the pipeline runs along a fast, static path. But when the filter gates a negative record, the agent evaluates the payload and dynamically selects the correct sequence of API tools (e.g., database query, inventory check, or email notification) at runtime. This allows the pipeline to execute complex decision trees dynamically, eliminating the need to build and maintain thousands of hardcoded conditional branches in the static Apache Beam code.

Implementing the pipeline

Here is an example implementation in Apache Beam using the Google Agent Development Kit (ADK) and the RunInference framework.

1. Defining the lightweight sentiment model

We define the upstream CPU model using HuggingFacePipelineModelHandler. This model classifies sentiment into POSITIVE, NEUTRAL, or NEGATIVE on the worker instance.

code_block: <ListValue: [StructValue([('code', 'model_handler = HuggingFacePipelineModelHandler(\r\n task="sentiment-analysis",\r\n model="distilbert-base-uncased-finetuned-sst-2-english"\r\n)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda9a8f3410>)])]>

2. Building the heavyweight ADK agent

The ADK agent acts as our remediation assistant. We equip it with three tools:

lookup_user: Queries BigQuery for the customer's email.
lookup_orders: Queries BigQuery for the customer's orders and current product inventory.
send_email: Sends a remediation email to the customer using the Gmail API.

code_block: <ListValue: [StructValue([('code', 'def make_adk_tools(project: str, dataset: str = "sentiment_demo"):\r\n def lookup_user(user_id: int) -> dict:\r\n """Look up user information (email address) from BigQuery by user ID."""\r\n from google.cloud import bigquery\r\n\r\n client = bigquery.Client(project=project)\r\n query = (\r\n f"SELECT user_id, user_email "\r\n f"FROM `{project}.{dataset}.users` "\r\n f"WHERE user_id = @user_id"\r\n )\r\n job_config = bigquery.QueryJobConfig(\r\n query_parameters=[bigquery.ScalarQueryParameter("user_id", "INT64", user_id)]\r\n )\r\n try:\r\n results = list(client.query(query, job_config=job_config).result())\r\n if results:\r\n row = results[0]\r\n return {"user_id": row.user_id, "user_email": row.user_email}\r\n return {"error": f"No user found with user_id={user_id}"}\r\n except Exception as exc:\r\n return {"error": str(exc)}\r\n\r\n def lookup_orders(user_id: int) -> dict:\r\n """Look up a user\'s orders and current product inventory from BigQuery."""\r\n from google.cloud import bigquery\r\n\r\n client = bigquery.Client(project=project)\r\n query = (\r\n f"SELECT p.order_id, p.product_id, pr.remaining_inventory, pr.price "\r\n f"FROM `{project}.{dataset}.purchases` p "\r\n f"JOIN `{project}.{dataset}.products` pr ON p.product_id = pr.product_id "\r\n f"WHERE p.user_id = @user_id"\r\n )\r\n job_config = bigquery.QueryJobConfig(\r\n query_parameters=[bigquery.ScalarQueryParameter("user_id", "INT64", user_id)]\r\n )\r\n try:\r\n results = list(client.query(query, job_config=job_config).result())\r\n orders = [\r\n {\r\n "order_id": row.order_id,\r\n "product_id": row.product_id,\r\n "remaining_inventory": row.remaining_inventory,\r\n "price": float(row.price),\r\n }\r\n for row in results\r\n ]\r\n return {"orders": orders}\r\n except Exception as exc:\r\n return {"error": str(exc)}\r\n\r\n def send_email(to_address: str, subject: str, body: str) -> str:\r\n """Send a plain-text email to the customer via the Gmail API."""\r\n import google.auth\r\n import googleapiclient.discovery\r\n import email.mime.text\r\n import base64\r\n\r\n try:\r\n creds, _ = google.auth.default(\r\n scopes=["https://www.googleapis.com/auth/gmail.send"]\r\n )\r\n service = googleapiclient.discovery.build("gmail", "v1", credentials=creds)\r\n\r\n mime_msg = email.mime.text.MIMEText(body)\r\n mime_msg["to"] = to_address\r\n mime_msg["subject"] = subject\r\n raw = base64.urlsafe_b64encode(mime_msg.as_bytes()).decode("utf-8")\r\n service.users().messages().send(userId="me", body={"raw": raw}).execute()\r\n return "Email sent successfully"\r\n except Exception as exc:\r\n return f"Failed to send email: {exc}"\r\n\r\n return [lookup_user, lookup_orders, send_email]'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda9a38e1d0>)])]>

We configure the LlmAgent and package it in the ADKAgentModelHandler:

code_block: <ListValue: [StructValue([('code', 'adk_agent = LlmAgent(\r\n name="remediation_agent",\r\n model="gemini-3.5-flash",\r\n instruction=(\r\n "You are a customer service remediation assistant with access to "\r\n "BigQuery lookup tools and an email sending tool. "\r\n "When given a prompt describing a customer situation, follow the "\r\n "numbered steps exactly and use your tools to complete the task."\r\n ),\r\n tools=adk_tools,\r\n)\r\n\r\n# RunInference handler for the ADK agent\r\nadk_handler = ADKAgentModelHandler(agent=adk_agent)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda99b23190>)])]>

3. Assembling the Dataflow DAG

The entire pipeline is declared cleanly. The upstream sentiment inference feeds directly into the filtering step (FilterNegativeADK), which then conditionally executes the downstream ADKInference:

code_block: <ListValue: [StructValue([('code', 'with beam.Pipeline(options=pipeline_options) as p:\r\n # 1. Read from Pub/Sub and classify sentiment on CPU\r\n sentiment_results = (\r\n p\r\n | "ReadFromPubSub" >> beam.io.ReadFromPubSub(topic=known_args.input_topic)\r\n | "DecodeMessages" >> beam.Map(lambda x: x.decode(\'utf-8\'))\r\n | "SentimentInference" >> RunInference(model_handler)\r\n )\r\n\r\n # 2. Filter out non-negative sentiment and invoke the ADK Agent\r\n _ = (\r\n sentiment_results\r\n | "FilterNegativeADK" >> beam.ParDo(FilterNegativeAndPromptADK())\r\n | "ADKInference" >> RunInference(adk_handler)\r\n | "LogADKResults" >> beam.ParDo(LogADKResponse())\r\n )'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda99b23d50>)])]>

Cost and performance advantages

By introducing this filtering step, we gain major engineering and operational advantages:

1. Significant cost reductions

Instead of paying for Gemini input/output tokens on 100% of incoming events, we pay only for the fraction that represent negative customer sentiment (typically < 5% of messages). The other 95% are classified locally on CPU instances at zero incremental API cost.

2. High streaming throughput

Dataflow distributes the CPU classification workload across many instances. Since CPU inference takes milliseconds, the pipeline scales horizontally to handle high-throughput event streams. The heavyweight LLM agent, which can take seconds per request due to tool execution, is called sparingly, preventing backlog.

3. Native Apache Beam integration

Adding the agent into the DAG requires no complex orchestration logic or manual thread pools. Using ADKAgentModelHandler with Beam's native RunInference transform handles parallel worker threads, batching, and integration automatically, keeping the codebase maintainable and clean.

Key takeaways

Streaming data is fast and high-volume, while heavyweight generative AI reasoning is slow and costly.

By building a pre-filtered pipeline with Google Dataflow and the ADK, you get the best of both worlds: the cost and speed of local CPU-based models, and the deep, automated capabilities of Gemini-backed agents.

To see the complete codebase and deploy this yourself, check out the next-2026-demo GitHub repository.

_{Apache Beam is a trademark of the Apache Software Foundation}

Governance on autopilot, minus the turbulence

Tue, 18 Aug 2026 16:00:00 +0000

Every data team knows the moment. Someone opens a table, sees a column called cust_seg_flg, and has to go ask around to find out what it means, whether it's safe to use, and whether anyone has already answered that question in another dashboard three teams over. Multiply that by thousands of tables and views, and you get the real cost of governance debt: not a compliance failure, but a daily tax on every person trying to do honest work with your data.

Most governance tooling today is reactive. You scan for problems, you get a report, someone opens a ticket, and three weeks later a column gets a description. The Governance Agent project (built on Google Cloud Knowledge Catalog, BigQuery, and column-level lineage) takes a different starting point: if a table upstream is already documented, tagged, and trusted, why should every downstream view have to earn that trust from scratch, by hand, every time?

This post is about that shift, from governance as an audit you dread to governance that keeps itself current in the background.

The problem in plain terms

Data estates grow through pipelines. Raw tables get joined, filtered, and reshaped into views, and those views feed more views. Somewhere in that chain, the original context (what a column means, whether it's PII, what quality bar it's held to) tends to get lost. It just doesn't travel.

The result is a familiar pattern: a handful of gold tables are well governed because someone invested real time in them, and everything built downstream of them is progressively less documented, less tagged, and less trustworthy, even when the underlying data hasn't actually gotten worse. The governance quality of a table ends up depending on how long ago someone cared about it, not on how the data is actually being used today.

As data flows through a company, it gets combined, filtered, and reshaped for different teams to use. But somewhere along that journey, the important context—like what a piece of information means, whether it contains private details, or if it’s accurate—gets left behind. The metadata simply doesn't travel with the data through the progression of data assets within the ecosystem.

The result is a familiar pattern: a company will have a few perfectly documented "core" datasets because someone invested time in them, but everything built on top of them becomes a mystery. The data itself hasn't gone bad, but without the original context, people stop trusting it. Ultimately, data is only considered reliable if someone manually updated its metadata recently, rather than because of what it actually contains.

What the agent actually does

The core idea is straightforward: use column-level lineage to figure out where a column came from, and propagate the governance metadata that already exists upstream, rather than asking a human to re-derive it.

Concretely, it handles four things:

Descriptions. If transactions.customer_id has a clear description upstream, and a downstream view pulls that column through two or three hops of joins, the agent traces that lineage and proposes the same description downstream. When a column isn't a straight passthrough (it's a SUM(), a CASE WHEN, a COALESCE), the agent reads the actual SQL that generated it and writes a description that reflects the transformation, instead of copying an upstream description that no longer applies.

Business glossary terms. Technical column names rarely match the business language people actually use. The agent uses semantic similarity to map columns to a controlled glossary, and can also read unstructured documents (a PDF policy, a product spec, a markdown design doc) to find explicit definitions rather than guessing from column names alone.

Policy tags. This is the one that matters most for risk. If a column is tagged as PII upstream, the agent traces where that data flows and recommends the same tag downstream, along with a summary of who currently has read access and what masking rules apply. It also checks whether a transformation looks like a "straight pull" (the sensitive value passed through unchanged) versus something that's been aggregated or anonymized, so it isn't blindly stamping PII tags on data that no longer carries the risk.

Trust and data quality scores. Rather than treating every view as an unknown, the agent derives a trust score based on the Data Quality and Profiling results of its upstream sources, and gives credit when it detects that a transformation actually improved data quality (deduplication, null handling, and so on).

Every one of these runs through a confidence threshold before anything gets applied. The system is explicit about not inferring PII status or glossary mappings without solid grounding. If the evidence is weak, the propagation doesn't happen automatically. That's a deliberate design choice: the agent is meant to close obvious gaps quickly, not make judgment calls that a person should be making.

Why proactive is the right word, and not a stretch

Proactive governance doesn't mean predicting the future. It means the governance work happens as data moves, instead of waiting for a scheduled review or a compliance incident to trigger it. In practice, that shows up in three ways:

New views inherit context automatically, instead of starting undocumented and waiting for someone to notice.
Sensitive data is flagged as it flows, not discovered after it's already been queried by twelve people who didn't know they needed a masking policy.
Stewards spend their time on judgment calls, like ambiguous mappings or new glossary terms, instead of repetitive column-by-column tagging that a lineage graph could have told you.

None of this replaces a data steward. It changes what a data steward's day looks like: fewer hours spent typing descriptions into a UI, more hours spent deciding what should count as a business term or whether an edge case needs a policy exception.

When lineage runs out, bring your own context

Lineage is powerful, but it isn't complete. Plenty of tables have no clean upstream source to inherit from: a newly ingested dataset, a one-off import, a table that predates whatever lineage tracking you have in place. For those, the agent doesn't just shrug. It lets you point it at your own documents (a PDF policy, a product spec, a markdown design doc, even a spreadsheet or a screenshot of a data dictionary) and uses that as grounding instead.

There are three ways to feed it context, and which one you pick depends on the size of what you're handing over. For a short document, you can inject the full text directly into the prompt. For something long, like a fifty-page data classification policy, the agent chunks it, embeds it, and retrieves only the passage relevant to the specific column it's describing, so you're not paying to re-read the whole document for every field. And if your organization already has a proper document repository indexed in Vertex AI Search, the agent can query that directly instead of re-processing files every time.

The part worth calling out is how conservative the grounding is. This isn't "read the doc and take a guess." The instructions given to the model are explicit: if a column isn't clearly defined in the document you provided, it has to say so and stop, not fill in the gap with a plausible-sounding guess. That rule is strict for policy tags and glossary terms in particular. A column only gets marked as PII if the document says so in plain language, like an explicit "PII: Y" flag or a named sensitivity section. No inference from column names, no "this sounds like it might be sensitive."

That distinction matters more than it sounds like it should. A tool that infers PII status when it's uncertain is a tool that will eventually mask a column that didn't need it, or worse, wave through one that did. Making "I don't know" a valid answer is what makes the automation trustworthy enough to run without someone re-checking every single output.

Two signals, not one: lineage plus insights

Lineage is the primary signal, but it isn't the only one the agent listens to, and it's worth being precise about why. The Data Lineage API only knows what a job explicitly recorded. If a table was built through a well-instrumented pipeline, that's a clean, high-confidence trail. But plenty of real estates have gaps: a table that predates good lineage capture, a transformation that ran outside the tracked jobs, a relationship that technically exists but was never logged as such.

For those gaps, the agent has a second pass. It can trigger a Knowledge Catalog Data Documentation scan (an AI-driven analysis Gemini runs over a table or dataset) which infers relationships and column meaning even without a clean SQL trail behind them. Those inferred relationships get extracted and cached locally, then loaded into the same traversal engine that handles lineage, so both signals are checked together rather than living in separate systems a steward has to reconcile by hand.

The order matters. Standard lineage runs first, since it's grounded in an actual recorded job. The Insights pass runs second, filling in only what lineage didn't find, and anything it contributes is explicitly tagged as coming from that source rather than blended in silently. If you're auditing a propagated description or tag later, you can tell whether it came from a hard lineage link or an inferred one. There's a dedicated end-to-end flow for this path (trigger the scan, wait for it, extract the results, apply them) so it isn't a manual side-quest bolted onto the main workflow.

The practical effect: an incomplete or newly onboarded pipeline still gets useful propagation on day one, instead of waiting until lineage coverage catches up.

What it looks like day to day

The project ships with both a Gradio-based dashboard and a CLI, which matters more than it sounds like it should. A steward reviewing a handful of tables before a demo will want the dashboard: run a scan, see which tables have metadata gaps, preview a proposed description or tag, and approve it with a click. A platform team that wants this running as part of a nightly job or a CI/CD pipeline will use the CLI, scripting steward_cli scan, apply, and policy-propagate commands the same way they'd script any other pipeline step.

That dual interface reflects a real operational choice: governance tooling that only works from a UI never gets automated, and governance tooling that only works from a CLI never gets adopted by the people closest to the data.

Where this needs a human in the loop

Worth saying plainly: this is not a "set it and forget it" system, and it shouldn't be treated as one. Lineage confidence scoring can still get things wrong, especially across renamed columns or unusual joins. Semantic mismatch checks catch obvious errors (a date column shouldn't inherit a description from an id column) but they're heuristics, not guarantees. Every propagation is designed to be previewed before it's applied, and that preview step isn't a formality, it's the actual safety mechanism.

The honest pitch here isn't "governance without effort." It's "governance where the effort goes to the right five percent of decisions instead of the repetitive ninety-five percent."

Customer Testimonial

“As custodians of the VodafoneThree UK Datahub, one of our biggest challenges is that a significant proportion of our data estate remains undocumented or inconsistently labelled. This creates friction for data discovery, slows down delivery teams, and limits the value we can unlock from AI solutions built on top of our data. The Data Steward Agent changes that. By combining cataloguing, lineage, and automated metadata propagation, it enables us to focus governance effort where it adds the most value while automatically carrying trusted context downstream. Rather than manually reviewing thousands of tables, we can concentrate on governing source datasets and allow lineage to scale that knowledge across the platform. We estimate this approach can reduce cataloguing effort by up to 75%, while significantly improving data discoverability, trust, and AI readiness across the UK Datahub.” - Radina-Paola Ivanova, GenAI Engineer, VodafoneThree UK Datahub

The takeaway

Governance debt compounds the same way technical debt does: quietly, until someone downstream hits it at the worst possible time. The value of an approach like this isn't that it makes governance disappear as a concern. It's that it moves the work upstream, literally, so that context and controls travel with the data instead of being reconstructed from scratch every time someone builds a new view.

For teams sitting on years of undocumented BigQuery estates, that's not a nice-to-have. It's the difference between governance being something you catch up on twice a year, and governance being something that just keeps pace with how fast your data actually moves.

Using BigQuery Graphs with measures for trusted agentic workloads

Thu, 13 Aug 2026 17:00:00 +0000

When enterprises transition from using simple chat assistants to autonomous, agentic workloads, they quickly run into a hard truth: Agents are prone to inaccurate insights when working with directly raw tables.

BigQuery Graph helps organizations move beyond flat, static tables to represent enterprises exactly how they exist in the physical world: as interconnected business entities with real-world dependencies. With the support of measures in BigQuery Graph (preview), we are unifying governed metrics with relationship mapping. This allows your agents to reason across complex dependencies captured in graphs with precision of measures.

Why relationships matter

Traditional data structures are blind to multi-hop business context, causing AI agents to make incorrect operational decisions:

The concrete problem: If a retailer has an agent who is asked why winter jacket sales dropped 12% in Seattle, it can query flat tables to report the what (the 12% dip). But it fails at the why because it cannot trace the relational path: Seattle orders ➔ distribution centers ➔ suppliers delayed by regional storms.
The risk of disjointed systems: Lacking relationship context, the agent suggests an irrelevant 15% markdown campaign, needlessly eroding margins. Furthermore, maintaining separate systems - where one team maps supplier relationships in a separate graph database while another maintains SQL metrics - forces your agent to stitch these stacks together at runtime. This process is slow, expensive, and leads to inconsistent KPI calculations.

Measures in BigQuery Graph solves this by letting you map existing tables to a property graph in-place with zero ETL. This unified setup enables a logical evolution of inquiry:

Metadata grounding establishes what data you have.
Business metrics (measures) calculate how your business performed.
Relationship mapping (graph) uncovers why it happened.

Under the hood

Historically, standard SQL joins during graph traversals duplicate rows, leading to incorrect aggregation calculations. BigQuery Graph solves this natively.

Data modelers define a MEASURE (like SUM or AVG) directly within the Property Graph DDL. Using standard SQL via the GRAPH_EXPAND function and the AGG aggregator, the engine resolves the structural graph paths before evaluating metrics. This ensures your agent is smart enough to know when it needs a calculator (SQL) and when it needs a map (graph).

Because public projects like bigquery-public-data are strictly read-only, you must map the logical property graph inside your own project using a placeholder variable (YOUR_PROJECT_ID), while directly referencing the read-only public tables as nodes and edges.

code_block: <ListValue: [StructValue([('code', '-- 1. Map the graph inside YOUR project \r\n\r\n\r\nCREATE OR REPLACE PROPERTY GRAPH `YOUR_PROJECT_ID.YOUR_DATASET.thelook_ecommerce_graph`\r\nNODE TABLES(\r\n `bigquery-public-data.thelook_ecommerce.users` AS User\r\n KEY(id)\r\n LABEL User PROPERTIES(id, city, country),\r\n `bigquery-public-data.thelook_ecommerce.orders` AS Order\r\n KEY(order_id)\r\n LABEL Order PROPERTIES(\r\n order_id, \r\n MEASURE(AVG(num_of_item)) AS avg_items_per_order,\r\n MEASURE(SUM(num_of_item)) AS total_items\r\n )\r\n)\r\nEDGE TABLES(\r\n `bigquery-public-data.thelook_ecommerce.orders` AS OrderedBy\r\n SOURCE KEY(order_id) REFERENCES Order(order_id)\r\n DESTINATION KEY(user_id) REFERENCES User(id)\r\n LABEL ORDERED_BY\r\n);\r\n\r\n-- 2. Query your new graph with standard SQL—using standard {Label}_{Property} column outputs\r\nSELECT\r\n User_city AS city,\r\n ROUND(AGG(Order_avg_items_per_order), 2) AS agg_avg_items,\r\n ROUND(AGG(Order_total_items), 2) AS agg_total_items\r\nFROM GRAPH_EXPAND("YOUR_PROJECT_ID.YOUR_DATASET.thelook_ecommerce_graph")\r\nGROUP BY User_city\r\nORDER BY agg_total_items DESC\r\nLIMIT 10;'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda9bf6fc50>)])]>

Democratizing graph intelligence in BigQuery Studio

To make managing and deploying these relationship networks frictionless for both developers and business users, we have built native, intuitive operational tools directly into BigQuery Studio:

Visual graph modeler: A no-code, drag-and-drop interface inside BigQuery Studio that lets you visually build, edit, and map property graphs, nodes, and edges without writing complex DDL scripts manually.

Conversational Analytics (CA) integration: Users can interact with the graph naturally. Instead of guessing table joins, Conversational Analytics agents navigate the deterministic, relationship-aware map of the graph, converting natural language questions into precise, boundary-constrained GoogleSQL or ISO GQL queries. This prevents model hallucinations and enforces semantic consistency.

Unified semantics: Native Looker integration

To avoid maintaining fragmented logic stacks, business metrics must live at the data layer. By integrating Looker (LookML) natively with BigQuery Graphs as in-database analytic models, you define logic once at the core:

Database-managed models (sql_analytic_model_name): Point Looker directly to your database-defined BigQuery Graph using sql_analytic_model_name to map standard LookML dimensions and measures directly to your graph properties.
Looker-managed models (derived_analytic_model): Define your BigQuery Graph schema directly inside your LookML view using derived_analytic_model. Looker will dynamically generate and execute the SQL DDL statements to maintain the graph inside BigQuery.
Enterprise DevOps workflows: Manage your graph's entire lifecycle using the Looker IDE, Git-based version control, and Continuous Integration (CI). Core KPIs (like Churn Rate) remain completely identical, verified, and trusted.

Looker’s semantic layer governs Gemini Enterprise data for user trust

Tue, 11 Aug 2026 16:00:00 +0000

For organizations deploying AI agents at scale, there’s often a critical divide between structured and unstructured data. While large language models (LLMs) excel at parsing text documents, emails, and PDFs, they can struggle when presented with raw enterprise databases. Meanwhile, standard natural-language-to-SQL (NL2SQL) models often guess how database schemas fit together, which can lead to unpredictable queries, inconsistent metrics, and AI hallucinations that erode user trust.

Gemini Enterprise brings the best of Google AI to every employee through an intuitive chat interface that acts as a single front door for AI in the workplace. And now, Looker’s governed semantic layer serves as the trusted foundation for structured data within Gemini Enterprise, enabling trusted self-service business intelligence for all Gemini Enterprise users. With this integration, Looker analysts and admins can publish conversational agents natively into their Gemini Enterprise environments via the Agent-to-Agent (A2A) protocol. Now, organizations can provide their AI-accelerated taskforce with robust and trusted tools, powered by real-time analytics, that they can explore in natural language in addition to their daily workspace workflows. Making it easy to offer conversational agents in Gemini Enterprise expands discoverability and promotes a data-driven culture, while reducing friction to adoption.

Bringing a semantic foundation to structured and unstructured data

By combining Looker’s semantic layer with Gemini Enterprise, you can query both structured databases and unstructured documents in plain English, all in one place. Instead of jumping between dashboards and other tools to understand your numbers, teams can instantly connect hard metrics with real-world context to solve problems and make decisions faster.

Publishing Looker agents for consumption in Gemini Enterprise

Minimize AI hallucinations

If you ask the typical AI chatbot to calculate "revenue" or "churn rate" against an unstructured cloud database, it has to guess which tables to join, which filters to apply, and which timestamps to trust. This can result in different people asking the same question, only to get completely different answers.

Looker’s semantic layer eliminates this guesswork, serving critical context to Gemini Enterprise in the form of codified data, allowing the agent to give deterministic, predictable responses.

code_block: <ListValue: [StructValue([('code', '[ Gemini Enterprise Chat UI ] \r\n │\r\n (A2A Protocol / NLP)\r\n ▼\r\n [ Looker Governed Agent ] ──► Generates Deterministic SQL\r\n │\r\n [ Looker Semantic Layer ] ──► Business-Approved Definitions & Logic\r\n │\r\n ▼\r\n [ Enterprise Data Cloud ] ──► (BigQuery, AlloyDB, Spanner, etc.)'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda9a8ab1d0>)])]>

When a Gemini Enterprise user requests a business KPI in Gemini Enterprise, the request is routed directly to a Looker agent. The semantic layer generates deterministic, precise SQL based on version-controlled business logic. This helps ensure when an executive asks for "Revenue," they get the exact, governed enterprise metric — not a guess.

Robust governance and secure access management

Data governance and security are critical when introducing AI to enterprise data warehouses. Organizations can’t risk corporate information being loosely ingested, indexed, or exposed outside of strict permissions.

Looker’s integration with Gemini Enterprise is built on a zero-risk pass-through architecture, processing the data, but not writing to persistent storage. Gemini Enterprise does not ingest, replicate, or store your underlying database records. Instead, the integration operates safely and securely over the A2A protocol, following these core tenets:

OAuth authorization: In order to interact with a Looker agent within Gemini Enterprise, end users provide a secure, one-time OAuth consent. This binds their Gemini session to their specific Looker credentials.
Strong governance enforcement: Because the architecture relies on live pass-through queries, Looker’s existing row-level and column-level access controls are maintained.
Strict security isolation: If a user does not have permission to view, say, sensitive regional payroll or financial rows within the Looker platform, the Looker agent actively restricts that data in the Gemini environment. Should an agent be published to the Agent Gallery to simplify discovery, it still does not bypass the security controls that you established.

Technical capabilities and enterprise readiness

Deploying Looker agents natively into Gemini Enterprise via the A2A protocol doesn't just make it smarter — it makes it more interactive and interoperable, without sacrificing security. Here are some of the features you’ll find in this release.

Rich visual interactivity: support for charts

They say a picture is worth a thousand words. When users interact with Looker agents inside Gemini Enterprise, the platform goes beyond textual explanations and provides native, interactive data charts. If a user asks for a visual trend—such as monthly sales performance or regional distribution—the Looker agent maps the database response with rich, presentation-ready visualizations directly inside the universal chat box.

Note: If you published Looker agents in Gemini Enterprise prior to Looker release 26.12, we recommend updating or refreshing them to take advantage of these enhanced visualization capabilities.

Interoperability with first- and third-party agents

Looker agents published to Gemini Enterprise can understand context across different agents and data sources. Leveraging standard communication frameworks, these agents can securely share structured, governed insights with other first-party Google Cloud agents like the Deep Research Agent or external third-party agents to create structured workflows. This enables complex multi-agent orchestration, where an enterprise operational agent can pull data from a Looker agent to feed into a separate productivity or supply-chain workflow.

Looker-based user authentication

To preserve enterprise governance, this integration implements a robust, identity-centric authentication model. Users are required to provide a one-time OAuth consent, binding their active Gemini Enterprise session securely to their underlying Looker credentials. This helps ensure that every conversational query hitting your databases is authenticated at the user level, enforcing pre-existing Looker permission structures, row-level data access filters, and column-level masking rules — no exceptions.

Trusted data in Gemini Enterprise

The future of work is agentic. Gemini Enterprise provides a single, secure architecture to deploy a global digital task force,empowering your business with the best of Google AI for developers, employees, and customers.

The integration of Looker with Gemini Enterprise not only brings trusted data analytics to business users but also adds rich interactivity, visual charts, and data storytelling directly into their everyday workspace. As business users embrace this agentic new way of working, they aren't just getting text answers; they are getting presentation-ready visualizations that bring operational metrics to life and deliver complex insights.

To get started, learn how to publish your data agents in Gemini Enterprise to make your agent’s predefined context and analytics available to your entire organization.

How Malachyte solves retail’s cold-start problem with managed real-time AI

Mon, 10 Aug 2026 16:00:00 +0000

What’s the best way to recommend products to little-known users?

We’ve spent our careers trying to solve this problem for major companies like Spotify and Priceline, and it’s why Sidd founded Malachyte, an AI-powered ecommerce recommendation platform. These days, consumers have come to expect content that feels personalized and relevant, and online services competing for their attention have no choice but to do this exceptionally well.

Malachyte was inspired by some unique insights into how advanced AI models, and large language models in particular, could be applied in new ways to old challenges like personalization and recommendations.

As Malachyte set out to win potential customers’ business, we needed secure, scalable, reliable and, above all, leading-edge AI infrastructure to continue building the personalization algorithm we had always envisioned. By utilizing Google Cloud tools like Bigtable and Managed Service for Apache Kafka, Malachyte has been able to help some of its retailers double and sometimes even triple their sales.

This is the story of how we built it, and the ways any founder can use services like these to start deploying AI foundation models in new ways.

How Malachyte lifted sales for their users

For Malachyte, the aha moment was discovering that it could use neural networks with attention mechanisms — the same concept powering large language models — to personalize retail search and product pages. This approach is what enables LLMs to derive meaning from the relative order of items in a sequence, in their case the order of words and syllables in a sentence. When it comes to a retail website or app, what Malachyte wanted to capture was the sequence of customer interactions with the site.

What if we predicted the next thing a user wants on an ecommerce website just like LLMs predict the next word in a sentence?

A pre-GPT language model might have tried to look at a specific sequence of words or even fragments of words (what we now know of as tokens), but those earlier models wouldn’t examine what happens if the words were in the comparable order but weren’t contiguous or were re-arranged. The breakthrough came — in part through Google’s work on transformers — when LLMs gained the ability to understand complex and long-range dependencies within a sequence of items.

This more sophisticated method has delivered dramatic results — both for the proliferation of gen AI in general, and for Malachyte’s application of the technology.

To make this work in practice, Malachyte creates a vector of everything known about a visitor when they arrive on a site. Most users are visiting for the first time, so little is known about them. This is what’s known as the “cold start” problem. The trick is to use every interaction with a user to refine this vector. Each new addition to the vector, like a click or a query, does two things: it drives a prediction about the next thing the user wants, and it provides more information about the user.

Malachyte’s platform then updates the user vector and the prediction at the same time. This not only enhances the understanding of the individual user and their preferences, it also improves the overall model with the anonymized user data. With every inference, the context of both the average and the specific shopper grows.

The company further innovates by not just using attention-based neural networks but combining that with updating user profiles 100 milliseconds at time.

Malachyte’s recommendation and search agents populate the next page’s search results or recommendation carousels based on what users clicked on previous pages.

To be sure, this idea isn’t in itself new. Retailers have long used collaborative filtering recommendations systems to identify similar users and items that required massive sets of interaction history. These models typically required a lot of data, including third-party cookie-based profiles and demographics.

By focusing on the sequence of interactions in a session, retailers can achieve far more personalization — with less required data or spend — than by focusing only on a user’s profile. As a bonus, retailers can now offer their users more privacy by not relying on long-term cookie data.

This works because of the model structure and multimodal vectors that encode everything they know about a user, including browser data, click history and searches. The output, too, is multimodal: The same model can be applied to on-site search product pages, category pages, and add-to-cart carousels.

To make this work, each product in the catalog is embedded into the same space as the user vector, which gets updated and subsequently moves the vector closer to relevant products and further from those that aren’t. The neural network computing the embedding is being continuously trained across retailers who work with Malachyte, improving the quality for everyone. The system effectively becomes a data cooperative with each retailer's user helping make the model smarter for everyone.

A user session represented as a vector in a space of products.

To make this delivery for every user at every inference in 100 milliseconds, Malachite found real benefits in building onGoogle Cloud’s real-time AI stack.

With this system, every behavioral event streams into a Managed Service for Apache Kafka cluster. Rather than queuing for a future training job, each event immediately becomes an update to the user’s profile in Bigtable. The Kafka cluster allows the customer’s front-end to persist, so the user session signals quickly with little worry about how they fit into the user vector.

Bigtable allows Malachyte’s services to look up and update the right user vectors, and it and Kafka operate at the order of 10 milliseconds per step, which allows the entire recommendation loop to complete with no disruption to the user experience.

The three layer real-time AI architecture: a retailer’s website, Malachyte’s AI models and serving front ends, and context management infrastructure.

In addition to a fast core, a second layer of product catalog updates, inventory signals, and retailer-specific dimensional data keeps product data up to date. This flows through Cloud Pub/Sub, which offers globally accessible REST APIs that enable connections retailers can use without deep integration work. Malachyte agents run on Google Kubernetes Engine (GKE), with model inference on Google Compute Engine (GCE).

Continuous ingestion of external data, such as product catalog updates, operates through Pub/Sub’s global messaging system.

With its migration to Google Cloud’s AI architecture, Malachyte demonstrated that production AI inference and training are about more than GPUs and storage. They require real-time continuous learning infrastructure that includes a fast key-value store, a streaming layer, and a managed messaging system, all integrated with the foundation model architecture.

This approach also shows that even a small team like Malachyte’s can have a big impact in an industry. It just needs access to powerful infrastructure and core AI managed services.

Try it for yourself

Looking to shake up your industry or stay ahead of the competition like Malachyte? Try Managed Service for Apache Kafka, Cloud Pub/Sub, and Bigtable. New customers can receive $300 in Google Cloud credits.

Zero-code, low-cost data ingestion: New BigQuery DTS capabilities

Fri, 07 Aug 2026 17:00:00 +0000

In a fast-paced digital economy, data is your most critical engine. Yet, many enterprises find themselves trapped in a costly paradox, spending over 100 hours a week building and fixing fragile, in-house ETL pipelines or wrestling with unpredictable third-party tools.

Trusted by thousands of customers every single day, BigQuery Data Transfer Service (DTS) eliminates this engineering burden. As a fully managed, zero-code data movement solution, BigQuery DTS automates data ingestion into BigQuery allowing your teams to transition from pipeline maintenance to strategic data science in minutes.

Expanding the ecosystem: New connectors and capabilities

We are rapidly expanding our integration landscape to eliminate data silos across databases, ads and marketing platforms. Here are the latest additions and enhancements

Open Lakehouse ingestion

Direct ingestion into Apache Iceberg managed tables (Preview): You can now ingest data from common sources such as Google Cloud Storage, Amazon S3, and Azure Blob Storage directly into Iceberg managed tables. This enables you to maintain full multi-cloud storage cross-compatibility with other query engines while leveraging BigQuery's top-tier performance tuning.

Next-gen agentic architecture

Fully managed remote Model Context Protocol (MCP) Server (Preview): This allows your developers to connect their AI application and agents to DTS allowing them to programmatically discover data sources and configure/execute transfers on the user's behalf.

Enterprise and relational databases (supports both full or incremental transfers)

Microsoft SQL Server (Preview): Easily centralize transactional tables, schemas, and operational data directly into your analytical environment in BigQuery.
PostgreSQL(GA) and MySQL (GA): Automates data delivery and simplifies the replication of high-volume web and application workloads into your central data warehouse within minutes. Supports data replication from on-premise environments, CloudSQL, and other clouds.

E-commerce and growth marketing

Shopify(Preview): Automates the extraction of granular order histories, inventory logs, and customer profiles straight into your analytical schema.
Klaviyo(Preview): Extracts detailed email and SMS engagement logs (such as clicks, sends, and opens) to build precise multi-channel lifecycle attributes.
HubSpot(Preview) : Syncs pipeline, contact tracking, and inbound marketing metrics to keep your revenue operations teams aligned.
Mailchimp(Preview): Automatically moves campaign performances and audience list attributes directly into your data warehouse.

Migration connectors

Snowflake(GA): Migrate your data from Snowflake with features like incremental transfer, auto schema detection, private connectivity and support for migrating data residing on all three major clouds (Google Cloud, AWS, and Azure)

Enhancement to major connectors

ServiceNow, Salesforce, and Oracle: Enhanced with native incremental update support to speed up large-scale pipeline refreshes for enterprise CRM, ITSM, and financial workflows.

Why you should choose BigQuery Data Transfer Service

Moving data across an enterprise architecture shouldn't require complex compromises between cost, management overhead, and pipeline health. BigQuery DTS delivers unique advantages across three pillars:

1. Unbeatable cost efficiency

Zero Ingestion Costs for major sources: Data ingestion at no-charge for all first-party Google sources (except Google Play), including Google Ads, Google Analytics 4, Campaign Manager, YouTube and Google Cloud Storage. Ingestion cost are also free for Amazon S3, Azure Blob Storage, Amazon Redshift, and Teradata
Low consumption-based rates for third-party SaaS: Ingestion from 3rd-party sources are entirely on a flexible consumption model. Compute fees run less than 6 cents per slot-hour in major regions. Because pricing scales with compute footprint rather than row volume, depending on your source data compression format and available network bandwidth, you can efficiently transfer massive data volumes.

2. Frictionless security and native management

Eliminate middlemen servers and external security and API configurations. BigQuery DTS fully integrates with Cloud IAM. Data transfers instantly inherit your destination dataset’s Column-Level Security, Row-Level Security, and Customer-Managed Encryption Keys (CMEK) without extra configuration overhead. Your streams flow securely inside the native Google Cloud perimeter.

3. Industry-leading performance and resilience

When you manage data at enterprise scale, downtime means lost business. BigQuery DTS provides a highly resilient ingestion footprint backed by a strict Google Cloud Service Level Agreement (SLA). The system delivers a monthly uptime percentage of >=99.99%, guaranteeing your analytics, automated pipelines, and operational dashboards update reliably.

Ready to transform your data operations?

Stop letting manual ingestion scripts limit your growth. Join the thousands of companies relying on Google's native cloud lakehouse data movement architecture to build a modern, scalable data stack.

Try the platform out today: Navigate directly to the BigQuery Data Transfer Service console, pick your connector, and deploy your first automated transfer in just a few clicks!

What connectors should we build next?

We are constantly expanding our native integration library based on your business needs. What sources are you currently forced to extract manually? Are there specific relational databases, NoSQL engines, or regional SaaS platforms you need to replicate next? Let us know with a feature request.

Please file feature requests via the public issue tracker.

Unifying Structured and Unstructured Data Insights with BQ Search Innovations

Fri, 07 Aug 2026 16:00:00 +0000

Modern enterprises possess a vast amount of unstructured data, yet they frequently encounter significant challenges in managing and extracting value from it. Historically, unlocking the insights hidden within PDFs, audio files, images, and unstructured text required a fragmented architecture: moving data out of your warehouse, stitching together complex LLM pipelines, and managing disparate search indexes.

BigQuery has worked with many enterprises to make sense of their unstructured data sources. For example, consider an advanced healthcare company managing thousands of clinical trial documents in PDF form. BigQuery helps unlock insights from these documents through a simple, five-step lifecycle: Access, Process, Ground, Relate, and Activate.

In this post, we are highlighting three major milestones focused heavily on the "Ground" phase of this framework:

General Availability (GA) of Autonomous Embedding Generation
General Availability (GA) of AI.SEARCH with massive single-query performance gains
Public Preview of Hybrid Search

Let’s dive into how these features work together to simplify your AI architecture, using a real-world clinical trial research platform as an example.

Simplify Pipelines with Autonomous Embedding Generation (GA)

Building a retrieval-augmented generation (RAG) pipeline or search application usually requires managing complex, asynchronous embedding infrastructure. You have to handle retries, error logging, and pipeline orchestration every time a new record arrives.

With the General Availability of Autonomous Embedding Generation, BigQuery manages this entirely for you. By simply defining a column in your schema, BigQuery asynchronously and continuously generates embeddings as new data is ingested. You have the flexibility to choose external models (like Vertex AI text-embeddings) or natively utilize Gemma embedding models directly within BigQuery.

How it works in practice:

Imagine you are building a research platform analyzing clinical trial PDFs stored in Google Cloud Storage. After extracting the study titles and disease areas into a table, you can automatically embed those titles:

code_block: <ListValue: [StructValue([('code', "CREATE OR REPLACE TABLE mydataset.clinical_trials\r\n(\r\n trial_id STRING,\r\n study_title STRING,\r\n study_embedding STRUCT<result ARRAY<FLOAT64>, status STRING>\r\n GENERATED ALWAYS AS (AI.EMBED(\r\n study_title,\r\n connection_id => 'myconnection',\r\n endpoint => 'text-embedding-005'\r\n ))\r\n STORED\r\n OPTIONS( asynchronous = TRUE )\r\n);"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda99697ed0>)])]>

BigQuery eliminates the need for complex third-party vector databases by managing enterprise-scale processing with one configuration. This autonomous embedding generation keeps data synchronized automatically as source text changes, removing the need for manual machine learning pipelines. This integrated approach streamlines workflows for dynamic datasets and reduces the operational burden of maintaining custom data scripts.

Finally, with this GA launch, Autonomous Embedding Generation now also supports generating embeddings natively over images using ObjectRefs, unlocking true multimodal search and analytics.

Natural Language Search at Scale with AI.SEARCH (GA)

To truly enable conversational analytics agents and snappier user experiences, your underlying search infrastructure needs to be intuitive and performant.

Once your data is seamlessly embedded, you need an efficient way to query it. Today, we are announcing the General Availability of AI.SEARCH(). This function provides a streamlined, natural-language-focused search experience, allowing you to easily find semantically related records without generating embeddings in your search path. In pairing this with the Autonomous Embedding Generation, we leverage the same embedding model used in your dataset for easier use.

Furthermore, as part of efficiency investments in the last year, we have heavily optimized AI.SEARCH for single-query execution. For online applications and single-query searches (those most common in agentic searches), we have observed up to a 133x gain in slot efficiency..

This means you can serve highly concurrent, user-facing natural language searches directly out of BigQuery faster and more cost-effectively than ever before.

code_block: <ListValue: [StructValue([('code', 'SELECT base.trial_id, base.study_title, distance\r\nFROM\r\n AI.SEARCH(\r\n TABLE mydataset.clinical_trials,\r\n \'study_title\',\r\n "What treatments are available for advanced tumors?"\r\n );'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda9a01f690>)])]>

Unifying Keyword and Vector with Hybrid Search (Public Preview)

Semantic (vector) search is incredibly powerful; for example, the query above will successfully return conceptually related terms like "chemotherapy." However, semantic search isn't always enough. What if a researcher is searching for a specific, highly technical immunotherapy drug designation like "MK3475"? Because this alphanumeric string lacks broad semantic meaning, pure vector search might struggle to rank it correctly.

By merging lexical search with existing semantic capabilities, BigQuery's hybrid search allows for data retrieval based on both keyword similarity and underlying meaning. This approach unites the conceptual depth of semantic vector search with the pinpoint accuracy of lexical matching, utilizing algorithms such as Reciprocal Rank Fusion and BM25. The result is a significant boost in search precision and a reduction in LLM hallucination costs through the reranking of results based on keyword frequency and semantic relevance. Users can implement this via the AI.SEARCH and VECTOR_SEARCH functions by employing the hybrid mode or lexical_search_columns parameters. Furthermore, performance can be optimized by extending vector indexes to include keyword data, which accelerates the lexical search process.

You can now perform hybrid searches effortlessly using the AI.SEARCH() function by simply setting the mode to HYBRID:

code_block: <ListValue: [StructValue([('code', 'SELECT base.trial_id, base.study_title, distance\r\nFROM\r\n AI.SEARCH(\r\n TABLE mydataset.clinical_trials,\r\n \'study_title\',\r\n "Cancer treated by MK-3475",\r\n mode => \'HYBRID\'\r\n );'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda9a01d410>)])]>

To speed up these hybrid queries at scale, you can easily extend your CREATE VECTOR INDEX DDL to include the keyword columns you want to use for the lexical portion of the search, natively combining your indexes.

Building an End-to-End Unstructured Data Analytics Platform

The search and embedding features launching today are part of a much broader vision. We are building an end-to-end unstructured data analytics platform. One common type of unstructured data is documents, and BigQuery now provides the complete toolset to manage this workflow from end to end:

Access: Enable zero-ETL workflows by querying unstructured PDFs and documents directly where they live in Google Cloud Storage using Object Tables.
Process: Utilize embedded AI capabilities like AI.PARSE_DOCUMENT and AI.CHUNK_DOCUMENT (coming soon) for layout-aware chunking (perfect for RAG), AI.GENERATE for entity extraction/summarization, and AI.CLASSIFY to instantly categorize records using foundational models directly in your SQL pipelines.
Ground: Build highly accurate context using Autonomous Embeddings and Hybrid Search. Hybrid search combines the conceptual understanding of semantic vector search with the exact precision of lexical keyword matching By reranking results based on both semantic relevance and keyword frequency, you drastically increase search precision and drive down LLM hallucinations.
Relate: Uncover hidden multi-hop insights by mapping extracted entities (like Sponsors, Trials, and Drugs) into a BigQuery Graph—no specialized graph database required.
Activate: Bring it all together with BigQuery's Conversational Analytics agents. Using functions like AI.AGG, you can chat directly with your complex data, generate visualizations, and perform trend analysis at massive scale.

With unstructured data as a first-class citizen in BigQuery, you can finally bridge the gap between your raw documents and conversational AI.

Ready to get started?

Explore the Code: Check out the complete end-to-end clinical trials demonstration in our Document Analytics on BigQuery GitHub Repository.
Read the Docs: Dive into the official documentation for Autonomous Embeddings and Hybrid Search to start building your own unified data pipelines today.

Agentic Future Ready With BigQuery: Continually Improving Price-Performance, Zero Effort

Thu, 06 Aug 2026 16:00:00 +0000

In the modern data landscape, query performance tuning and managing system price-performance is challenging, especially as the number of agentic workloads increase. Even for experienced developers and DBAs, constantly analyzing query execution plans, tweaking schemas, and adding query hints with ever exploding volume, variety, and velocity of data is a never-ending cycle that drains business velocity.

While performance tuning is a common practice, a modern data platform should do more. As data platforms evolve from systems of intelligence to systems of action, and analytics workloads shift from humans running a few queries per day to countless agents running many thousands of queries per minute, the old way of manual query tuning doesn’t work. When queries are generated by agents and applications automatically based on user actions, manual optimization becomes practically impossible.

BigQuery has evolved from a data warehouse to the primary engine for the Agentic AI era. Building on a unique, truly disaggregated storage and compute architecture, serverless processing, and fine grained compute management, BigQuery continues to push the boundaries of autonomous query processing. Our North Star is an autonomous query processor powering both humans and agents for hands-free optimum price-performance regardless of query, schema, data, or workloads. Just in 2025, we delivered up to 35% better query performance and as much as 40% reduction in query processing costs (slot usage).

Figure 1. Summary of BigQuery price-performance improvements throughout 2025 based on industry standard benchmarks.

The following are some of the major innovations contributing to these improvements in performance and total cost of ownership (TCO), including the built-in guardrails against regression.

BigQuery’s Self-Learning Engine: History-Based Optimizations (HBO)

One of the foundational capabilities of BigQuery’s autonomous query processor is history-based optimizations.

Traditional query optimizers rely on static statistics, metadata and cardinality estimates, which can be wildly inaccurate when faced with highly complex, multi-table joins and rapidly growing/changing data. Managing these is part of the critical path for administrators and automation has to be tailored to individual workloads to be effective and efficient. Even when everything is up-to-date and correct, queries can still have vastly different behavior due to natural data skew or changes in available compute resources. History-based optimizations change the paradigm: in addition to BigQuery’s already adaptive query execution that can change plans and resource allocations while a query is running, it learns from past executions and automatically applies additional optimizations for future executions.

It tracks runtime statistics of past queries to "remember" which optimizations were beneficial and continue to apply them, and learn from prior mistakes to ensure they are not repeated. When the same or similar query runs again, BigQuery automatically applies any additional optimization technique that is known to be beneficial and avoids those that can cause regressions.

No User Action Required & Built-in Safety Guardrails
History-based optimizations require no application rewrites, SQL modifications, or schema changes. Users literally do nothing, and their recurring dashboards, ELT pipelines, agentic workloads, or line of business modules run faster.

Crucially, this is a self-maintaining, self-improving closed-loop system with built-in safety guardrails. History-based optimizations only apply an optimization when there is high confidence it will improve performance. What if an optimization makes the wrong decision? This capability is inherently self-correcting. When an optimization is applied, the system measures the result. If the expected improvement is observed, the optimization is accepted. If it does not significantly improve performance—or worse, regresses or leads to failure—the optimization is immediately rejected, revoked, and never tried again for that query. This includes detecting data skew so queries that have parameter sensitive plans do not run into major performance issues when just a single parameter (aka WHERE clause) is changed.

The result? A decrease in execution times and a reduction of slots. One enterprise customer reported P90 execution times drop by up to 50%, with slot usage falling by up to 15% resulting in substantial price-performance improvements and no regressions.

Figure 2. One example of History Based Optimization performance improvement reported by an enterprise customer.

Teaching the BigQuery execution engine newer tricks: Advanced runtime

BigQuery’s autonomous capabilities extend deep into the execution layer with BigQuery advanced runtime. This engine upgrade automatically determines the best physical execution path for a query without any manual knob-turning.

1. Enhanced vectorization
While vectorized execution is not a new concept, BigQuery enhanced its implementation by taking advantage of the newer processor efficiencies in Single Instruction Multiple Data (SIMD) instructions. The engine autonomously identifies opportunities to avoid duplicate computations, processing data in dictionary and run-length encodings natively. It couples with state-of-the-art parallel algorithms and is applied at eligible query stages to increase the opportunities for acceleration. The Impact: Fully automated, safely accelerating qualifying queries by up to 10x, yielding up to a 40% overall slot time reduction.

Figure 3. Advanced runtime - Enhanced vectorization

2. Short query optimizations
For high-concurrency and low latency BI dashboards or agentic applications generating thousands of queries that require sub-second latency, distributed processing overhead can be a bottleneck. BigQuery now autonomously accelerates eligible "short" queries without impacting other queries running concurrently. By reducing the number of stages hence reducing data shuffling, BigQuery improves overall query latency and resource efficiency. BigQuery has effectively implemented an efficient symmetric multi-processing (SMP) query path transparently within a scalable massively parallel processing (MPP), distributed databases. The Impact: up to 10x lower slot usage for short queries, with P99 sub-second query latencies. Because each query uses slots for shorter periods, we observed some customer workloads having up to 3x higher throughput—all completely transparent to the end-user.

Figure 4. Advanced runtime - Short query optimizations

Same Benefits Regardless of Data Formats

We believe you shouldn't have to sacrifice autonomous performance when adopting open lakehouse architectures using open table formats like Apache Iceberg. BigQuery performance improvements work the same regardless of the underlying table or data format. That means you still get the same benefits whether you’re using BigQuery’s native capacitor storage format or Iceberg tables where the underlying data is in parquet format.

Whether it’s automatically pushing down filters, employing Column Metadata Index (CMETA) pruning, and optimizing the I/O layer with page skipping and asynchronous read or employing the same enhanced vectorization, open formats benefit from the same hands-free acceleration as native tables.

Figure 5. BigQuery on Lakehouse Iceberg tables performance and costs tested internally against a popular open source, distributed SQL query engine using a benchmark derived from TPC-DS (10TB)

Fluid Scaling: The True Only-Pay-For-What-You-Use Autoscaler

Because BigQuery’s compute models are tied directly to slot-seconds consumed rather than a slice of nodes or clusters, autonomous performance gains can translate directly to cost savings. BigQuery autoscaler enhanced with fluid scaling enables you to run any mix of highly variable workloads with a premier autoscaling model that does not require a cost-and-performance trade-off. Fluid scaling in BigQuery enables true per-second billing for compute resources (slots) consumed and lowers costs by up to 34% on average for autoscaling workloads.

Figure 6. Fluid scaling enhanced autoscaler

On the price-performance benefits of fluid scaling, Chen Shalit, the CEO and Co-Founder at RISE, a leading AdTech company that processes 1 PB+ data per day and manages 3 trillion+ monthly bids across publishers and advertisers, summed it best. “In the high-stakes world of advertising, every cent counts. BigQuery’s Fluid Scaling has reduced our infrastructure cost by 25% allowing us to scale our slot consumption and accelerate our entire workflow. By delivering faster model results and increasing our hourly data processing throughput, we’re providing our customers with the 'freshest' data possible.”

The foundation for the agentic future

These autonomous capabilities are not just about making today's business intelligence dashboards and analytics queries run faster; they are foundational to the future of modern data platforms. As organizations rapidly deploy AI and machine learning, they are shifting toward agentic workloads.

AI agents do not interact with data the way humans do. Their query latency and concurrency (QPS) requirements are orders of magnitude more demanding than what human users generate—which is the limit of what most traditional analytics platforms were designed for. When autonomous agents are firing off thousands of distinct queries per second to make real-time decisions, human-in-the-loop query tuning is simply impossible.

BigQuery's differentiated abilities in a self-tuning and self-learning system include built-in safety guardrails against regression. With this standard of autonomy, you can spend your time and effort where it truly matters: delivering the best experiences for your users with advanced agentic applications, rather than wasting critical engineering cycles on manual infrastructure management, reactive query optimization, or trying to rein in spiraling compute costs.

BigQuery’s aggregate TCO impact: Faster and more cost-effective price-performance

When you combine a self-learning history-based optimizer with an auto-scaling advanced runtime engine supported by a highly scalable metadata platform (CMETA), you can see real impact on performance and total cost of ownership (TCO).

Conclusion

Our goal is for BigQuery to automatically and safely optimize queries without user action so our users can “just come in to work and BigQuery is faster than it was the day before."

You no longer need to worry about data skew or stale statistics causing performance regressions. No more trying to figure out which plan guide works best for which queries at what time of the day or month. No more hoping you won’t run out of budget because there aren’t enough low priced spot instances. Just focus on your organization’s goals, and your BigQuery data platform can deliver optimized price-performance for both agent and human workloads autonomously; it just works.

Learn more about BigQuery’s data and AI capabilities and get started with a free trial.

Solving the "Noisy Neighbor": How Sharded Architecture Protects Multi-Tenant Platforms

Wed, 05 Aug 2026 16:00:00 +0000

Whether you are a multi-tenant SaaS provider, a large enterprise managing internal data platforms, or a company handling mixed-workload data processing, managing a shared infrastructure environment means facing a common threat: the 'noisy neighbor'. A single tenant with a massive data burst or a failing database instance can bring down the entire neighborhood, manifesting as significant backlog accumulation and global Service Level Agreement (SLA) violations across critical data pipelines

Here is how to transition from a monolithic architecture to a sharded hub-and-spoke pattern to ensure platform resilience.

The problem: The monolithic bottleneck

A typical legacy architecture processes data for all tenants and business domains through a single, massive stream. Because the pipeline is unified, a performance issue with one specific database tenant instance creates back pressure that degrades performance for every other tenant on the platform.

Painful effects:

100% Blast radius: One database failure can stop all processing.
Inefficient scaling: Resources often have to be scaled for the "worst-case" tenant, leading to significant wasted spend.
SLA instability: Maintaining a global SLA is nearly impossible when one high-volume tenant can lag the entire system.

The solution: sharded hub-and-spoke architecture

To solve this, processing is decoupled into a "hub" for routing and "spokes" for isolated execution.

1. The hub: The router pipeline

The Hub is a lightweight Dataflow job that acts as a traffic controller. It reads from unified source topics, parses the Tenant ID or Business Domain, and fans the data out into isolated buffers. This keeps the entry point simple and robust.

2. The buffer: Durable isolation

Pub/Sub topics are introduced between the Hub and the Spokes. These act as a durable shock absorber, preventing a slow downstream sink from backing up the original source.

3. The spokes: Isolated execution

Instead of one giant pipeline, multiple, smaller Dataflow instances are deployed and categorized by workload:

Tier 1 (high-priority): Dedicated pipelines with high resource allocation for critical tenants.
Shared tiers: Grouped pipelines for smaller tenants to optimize costs.
Domain specific: Specialized pipelines for complex logic (e.g., separating distinct business domains) to isolate code complexity.

Comparative Benefits at a Glance

Feature	Monolithic (legacy)	Hub-and-spoke
Fault tolerance	One failing DB stops everything	Failures isolated to specific spoke
Blast radius	100%	< 5% (Isolated to one spoke)
Resource scaling	Scaled for "worst-case"	Independent scaling per tenant load
Maintenance	Global updates affect everyone	Update one domain without touching others

Pro-Tips for Implementation

Moving to this architecture is more than just shifting boxes on a diagram. Additional "Spoke" level optimizations are recommended for maximum stability:

Implement Dead Letter Queues (DLQ): Do not let a single SQL exception stall the pipeline. Route failed records to storage (like BigQuery or Google Cloud Storage) for later investigation.
Strict connection pooling: Databases have connection limits. Use a thread-safe singleton pattern and set a low MaximumPoolSize (e.g., 1-2) per worker to avoid exhausting the database during autoscaling.
Asynchronous I/O: Use the GroupIntoBatches transform to buffer writes, which reduces the connection overhead that often triggers database-induced latency.

Conclusion

By adopting a sharded approach, platforms can guarantee that a "noisy neighbor" is no longer a threat to the neighborhood. This architecture provides the isolation needed to maintain strict SLAs while allowing for independent scaling and safer deployments

To learn more about implementing a sharded hub-and-spoke architecture, explore the Dataflow documentation.

How Target is enhancing retail discovery and cutting database maintenance by 50% with Spanner Graph

Tue, 04 Aug 2026 16:00:00 +0000

In today’s retail environment, shoppers expect highly personalized product discovery experiences and conversational assistance that feels genuine, natural, and genuinely helpful. Today, successful product discovery is about understanding semantic meaning and the rich, connected relationships between products, categories, and guest intent. It is no longer just about keywords and basic browsing.

At Target, this work is handled by our Guest Product Confidence platform team. They are responsible for building the features that establish trust and guide purchasing decisions, such as ratings, reviews, and AI-driven digital shopping assistants. An exciting example of this is our Gift Finder chat agent, which we launched during the 2025 holiday season online and in the Target app to help shoppers discover the perfect items through friendly, conversational dialogue.

To deliver real-time personalization and context-rich semantic responses like these at global scale, we identified a critical architectural need to move away from a fragmented data ecosystem toward a unified data platform. We needed a solution capable of supporting high-throughput transactional workloads, highly connected graph relationships, vector similarity search, and full-text keyword search all at once.

In this post, we’ll explore how we achieved all four with Spanner.

Overcoming fragmented architecture

Previously, Target’s discovery data ecosystem relied on a combination of Elasticsearch clusters for search and inverted indexes, alongside separate NoSQL datastores for our transactional data. While functional, this fragmented architecture presented significant operational and technical challenges.

Disconnected context: Keeping separate search, vector, and transactional databases in perfect sync was a constant challenge. Siloed information led to missing context, disconnected attribute relationships, and inconsistent query results.
High operational overhead: Managing independent clusters, tuning search indexes, and handling complex, custom synchronization and aggregation logic required intensive manual intervention from our engineering teams.
Expansion bottlenecks: Expanding our retail data domains required adding new database collections, maintaining complex joins, and navigating weak transactional guarantees across our discovery and core transactional systems.
Siloed intelligence: We lacked the ability to query graph relationships, vector similarity, and keyword search indexes in a single transaction.

To build the next generation of AI-driven guest experiences, we needed to consolidate on one platform.

Building the enterprise ontology on Spanner Graph

We evaluated multiple specialized technologies, including standalone vector databases and niche graph databases. However, adding more single-purpose databases would have only worsened our operational complexity and data synchronization pipelines.

We ultimately chose Spanner Graph to build our enterprise ontology, which is a "graph-of-graphs" paradigm that allows us to construct a massive, generative AI-powered shopping graph.

By unifying our data, we bring semantic data, graph relationships, vector embeddings, and operational transactions under one roof. This establishes Spanner as our single authoritative source of truth for both transactional state and semantic intelligence.

Our high-level architecture now consists of three core pillars:

1. Enterprise augmentation
This layer captures our enterprise retail catalog, aggregates relevant metadata from multiple backend sources, and utilizes generative AI for agentic data enrichment to dramatically improve the quality and depth of the product data we ingest.

2. Unified graph, vector, and search store
Instead of shifting data across multiple databases, Spanner Graph stores our entity nodes, relationship edges, and vector embeddings in the same database engine. Spanner Graph natively supports multi-hop graph traversals, semantic vector similarity, and full-text keyword queries over our relational tables. Because this multi-model synergy is native, we get strict ACID transactions for absolute correctness across distributed workloads without the need for fragile external sync pipelines.

3. Orchestration and AI layer
This layer powers our conversational guest interfaces, utilizing rich, structured context fed directly from Spanner Graph to ground our LLMs. It extracts highly specific product relationships to power tools like the Gift Finder while governing responsible AI processes and evaluating generated outputs.

A smooth, zero-downtime incremental migration

Transitioning critical search and discovery infrastructure that millions of guests rely on required a cautious, zero-downtime approach. We executed this migration in four structured phases.

Schema and ontology mapping: We defined the specific retail entities, such as products, categories, brands, and guest preferences, and their corresponding relationships within the Spanner Graph schema.
Data integration and parallel replay: We built mutation-based data integrations in a parallel pipeline. This allowed us to continuously replay live transactional updates, apply schema transformations, generate embeddings, and write them directly into Spanner Graph in real-time.
Canary deployment: We gradually shifted live read traffic to the new Spanner Graph-backed platform, validating query performance, semantic accuracy, and database stability under real retail workloads.
Cutover and cleanup: Once performance was thoroughly verified, we fully transitioned all search and discovery traffic to Spanner and deprecated our legacy Elasticsearch stack, entirely removing the maintenance burden of those clusters.

Business impact

By building directly on Spanner Graph, we unlocked measurable technical and business outcomes:

The ultimate GraphRAG foundation: Traditional RAG relies on flat vector similarity, which often misses the structured associations between products, such as matching a toy with its compatible accessories or age-appropriateness. By combining deep graph traversals with semantic vector search in a unified GraphRAG architecture, we grounded our LLMs with highly precise context. This directly improved our recommendation relevancy, enhanced guest satisfaction, and boosted our Net Promoter Score.

Consolidated SQL + GQL interoperability: With Spanner Graph, our developers query structured relational catalog data and connected graph relationships in a single query using standard SQL and GQL (Graph Query Language). This eliminates the need for data duplication, latency, or complex ETL pipelines to bridge these paradigms.

Serverless scalability with zero growth ceiling: Spanner automatically handled massive, unpredictable traffic spikes during peak retail events like Black Friday and Cyber Monday. Spanner's built-in autoscaler dynamically adjusted computing capacity to handle burst traffic during high-intensity, limited-time promotional offers without sacrificing performance.

50% reduction in infrastructure maintenance: By consolidating our transactional NoSQL and search index databases into a single managed Google Cloud service, we eliminated the operational burden of maintaining separate database clusters. Our developers now spend 50% less time on database administration and infrastructure upkeep, allowing us to build and deploy new, customer-facing AI features much faster.

Migrating to Spanner Graph has accelerated our generative AI roadmap, serving as the ultimate proof of what is possible when you build on the right data foundation.

Want to supercharge your AI apps? It starts with databases with the right graph capabilities at virtually unlimited scale. Discover how Spanner Graph can turn data into action for your organization.

Cortex Framework v7 is GA: Build agentic workflows without disrupting SAP operations

Mon, 03 Aug 2026 16:00:00 +0000

Businesses want to quickly and safely deploy AI agents to drive revenue, mitigate risk, and optimize capital, all without disrupting mission-critical ERP systems. And to power AI agents, you need more than raw data: You need interoperable data products that act as a single, reusable source of truth, turning cryptic source system records into clear business terms.

Today, we are announcing the general availability of Google Cloud Cortex Framework version 7. This release modernizes your data architecture for agent readiness, helping you quickly deploy, customize, and extend robust data products while simplifying orchestration and reducing the infrastructure overhead of traditional approaches.

While ERP systems hold a wealth of foundational data, turning those transactional records into AI-ready data products is difficult, especially without slowing down core operations. Cortex Framework v7 solves this with purpose-built data product accelerators for SAP. Deployed directly in BigQuery and Knowledge Catalog, these data products feed Gemini Enterprise Agent Platform with accurate business context so your AI agents can execute with high fidelity. It also simplifies data orchestration using a modular, scalable deployment architecture powered by Dataform, an end-to-end experience that helps data teams build, version control, and orchestrate workflows in BigQuery.

Read on for more details about what you’ll find in Cortex Framework v7, how it integrates with the recently released SAP Business Data Cloud Connect for BigQuery, what our customers are saying, and how to get started.

What’s new in Cortex Framework v7

1. Deliver agent-ready data products

Traditional business intelligence (BI) dashboards tell you what happened last quarter, but modern AI models and agents help you act in real time. To power this shift, Cortex Framework v7 packages your enterprise data into semantically rich data products that contain AI-friendly metadata. This enables your agents to reason, orchestrate, and execute high-impact workflows.

By translating raw SAP tables into clear business terms, dynamically ingesting custom fields, and natively handling advanced logic (like SAP TCURX currency decimal shifts), the Framework maintains the high data fidelity required for large language model (LLM) interactions and enterprise analytics.

Image showing Cortex Framework v7 deployed data products for SAP ERP inside BigQuery with built-in description metadata for field names and descriptions.

These purpose-built data product accelerators for SAP ERP and SAP Business Data Cloud deploy directly in BigQuery. Once they’re deployed, you can trigger automatic registration in Knowledge Catalog to easily discover data and connect it with your AI agents.

Image showing a Cortex Framework v7 data product registered in Knowledge Catalog with built-in metadata for descriptions and labels.

To accelerate custom AI projects, this release also includes agent skills for an agentic data product builder that helps data and knowledge engineers use natural language to generate custom data products. Using the Framework's provided content, an AI agent can automatically handle the build process, including adapting to specific source data requirements and customizations.

GIF showing Cortex Framework v7 delivered agentic data product builder skills in action using Google Antigravity with Gemini to automate the build and deployment of new data products using natural language prompts.

With v7 you can also extend Cortex Framework to maintain a clean separation between Cortex Framework-delivered content and your custom models. This helps you apply the latest code updates without impacting your custom work.

2. Build faster with modular deployments

Dataform powers the modernized data pipelines in Cortex Framework v7, offering version-controlled SQL and native dependency management. When you select a packaged data product for deployment, Cortex Framework retrieves and processes only the required tables. Dependency resolution automatically handles order-of-operations and generates a dependency graph for efficient processing. You can also add custom fields or logic without breaking standard models.

Image showing Cortex Framework v7 data products in Dataform with automated dependency graph showing data asset and table dependencies as well as built-in metadata like natural language tags.

This modular approach extends natively to complex, multi-system SAP environments. Using built-in logic differentiation, Dataform compiles and deploys Cortex Framework data products for SAP ECC, SAP S/4HANA, and SAP BDC source systems in parallel, while dynamic schema discovery automatically ingests and processes custom SAP fields such as Z-fields. Together, these capabilities help you bring in data from multiple systems simultaneously, reducing the need to manually refactor pipelines.

3. Scale cost-effectively with flexible processing

Cortex Framework now defaults to BigQuery incremental loading and non-destructive schema updates, processing only new or modified data. This approach minimizes compute time and significantly lowers operational expenses. Because orchestration relies on Dataform’s serverless, BigQuery-native execution, data processing scales without additional infrastructure overhead.

You can also use a bring-your-own (BYO) change data capture (CDC) approach. You can use this approach to bypass built-in CDC data processing and connect your existing CDC-processed pipelines directly.

Interoperability with SAP Business Data Cloud

Following the recent general availability of SAP Business Data Cloud Connect for BigQuery, Cortex Framework v7 now natively supports SAP BDC standard and custom data products. By combining Cortex Framework-delivered data products with SAP BDC data products, you can uncover new business opportunities. To accelerate this, v7 includes solution samples across both SAP BDC and SAP ERP data to help you quickly answer a wide range of critical questions such as assessing the health of your sales pipeline or identifying procurement inefficiencies.

Image showing Cortex Framework v7 consumption data product solution samples deployed on SAP BDC sourced data products via SAP Business Data Cloud Connect for BigQuery integration.

Co-innovating with industry leaders

We’ve worked closely with our Professional Services Organization and industry leaders to test these new capabilities. By serving as our lighthouse customers, these organizations provided valuable feedback to help us shape a framework built for enterprise-scale agility and AI readiness.

“As an existing Cortex Framework customer, we are excited about the capabilities introduced with the latest v7 release. Bridging complex SAP ECC and S/4 environments with Google Cloud’s AI capabilities requires a trusted data foundation with accurate business context. Based on the capabilities announced for Cortex Framework v7, including the agentic data product builder, we see significant potential to accelerate and scale the creation of enterprise data products for analytics and agentic AI use cases. The direction of the platform is particularly promising for organizations looking to make SAP data more accessible, semantically meaningful, and ready to support future reporting and AI initiatives.”
- Dr Jwan Ibrahim, Director, Data and Analytics, SLB

Get started

Cortex Framework v7 changes how enterprises operationalize their most critical data for AI agents. By combining serverless execution, AI-ready semantics, and native SAP integration, Google Cloud helps you turn complex enterprise data into a strategic asset. To get started, check out the following resources:

Execute a deployment: Try a Cortex Framework demo deployment (with a predefined sample dataset) to explore your first v7 data products today.
Learn more: Read the comprehensive Cortex Framework v7 technical documentation for deep dives into production deployment steps, configuration, extensibility, source system integration, and packaged data products.
Talk to an expert: Contact your Google Cloud account team to learn how Cortex Framework v7 can accelerate your specific data-to-AI journey.

Unifying public and private data: Scale knowledge graphs with Data Commons on Spanner

Mon, 03 Aug 2026 16:00:00 +0000

To make informed decisions, businesses often need to connect their internal data with public reference data, to create a knowledge graph that connects real-world things and their relationships. However, bridging data from public and private worlds has traditionally been complex. Today, we are streamlining these connections with the general availability of Data Commons on Spanner Graph and the preview of the new Data Commons Platform to unify your private knowledge with knowledge graphs from public datasets.

The overarching Data Commons project supports Google’s mission to organize the world's information and make it universally accessible and useful. Data Commons unifies fragmented public datasets from over 100 authoritative providers, including the United Nations, World Bank, US Census Bureau, Eurostat, WHO, and NOAA, with over 400 billion data points structured using standardized Schema.org definitions. Data Commons provides data exploration tools, MCP tools, and cloud-based APIs to access and integrate the clean datasets.

Data Commons integrates public information across multiple domains, including agriculture, demographics, economy, environment, and health. This standardized approach unlocks powerful use cases, for instance, letting you analyze national GDP trends, map regional smoke pollution levels, track local health equity, or demographic distributions over time, all using data that has already been preprocessed and normalized for you.

Data Commons knowledge graph dimensions

Dimension	Size	Technical description
Statistical observations	400+ billion	Individual metric data points
Graph edges	2.6+ billion	Relationships
Knowledge graph nodes	1.7+ billion	Standardized entities
Data sources	100+ providers	Authoritative institutions

^{Data Commons makes meaningful quantities of public administrative data available to users on readily consumable cloud-based infrastructure.}

A modern infrastructure powered by Spanner Graph

When we first built Data Commons, our goal was to aggregate massive, disparate public datasets using the tools available at the time. The platform relied on Bigtable as a caching layer, which was an effective strategy for handling large-scale lookups in the absence of native graph database technology.

Today, we have transitioned our architecture to a native graph model with Spanner Graph, which brings the convenience of a SQL-like interface and graph expressiveness to Spanner, with its high availability, horizontal scale-out, multi-region transactional consistency, and native ISO/IEC 39075 Graph Query Language (GQL) support.

By adopting a multi-entity Spanner Graph schema, we represent entities as nodes and their domain links as dynamic graph edges, allowing us to move away from pre-computed cache structures and perform complex relationship queries directly within the database using GQL.

This architecture also simplifies our pipelines by removing the need for complex, pre-computed indices that require costly in-memory rebuilds and multiple snapshots. Spanner Graph enables incremental updates to specific datasets without refreshing the entire database, while stale reads maintain consistent data snapshots during ingestion.

Key benefits by moving to Spanner Graph

Unified storage and incremental updates: By utilizing Spanner Graph’s multi-entity schema, the platform replaces complex caches with a model that supports incremental data imports, allowing for targeted updates to specific datasets.
Dynamic graph traversals via GraphRAG: The system executes multi-hop queries such as navigating hierarchies like continent → country → state → county → city on the fly. This removes reliance on static caches and enables GraphRAG workflows, where the database maps natural language queries directly to structured path-matching traversals.
Consistent data snapshots: Leveraging Spanner TrueTime and stale reads, the platform provides you with a version-consistent snapshot of data, maintaining integrity across distributed nodes following batch ingestion cycles.
Operational analytics at scale: Spanner’s columnar engine efficiently scans massive time-series datasets by reading only the necessary fields, while BigQuery federation that leverages Spanner’s Data Boost technology performs complex aggregations via EXTERNAL_QUERY in an isolated environment, helping isolate production traffic.

Bridging systems with SDMX 3.0 interoperability

To facilitate the use of complex statistical data, Data Commons adopts a lean implementation of Statistical Data and Metadata eXchange (SDMX) technical standard. As an ISO specification, SDMX provides a consistent approach for describing and exchanging statistical data along with descriptive statistical meta-information.

In this Data Commons Platform update we added support for the SDMX technical standard version 3.0, providing out-of-the-box integration with third-party tools like Tableau, Flourish, and Observable for multi-dimensional datasets. This is made possible using the API standard SDMX-JSON and SDMX-CSV 2.0 formats across two high-value endpoints:

The availability API: A programmatic discovery mechanism to identify existing dimensions, variables, and date ranges without reading raw values.
The data API: Retrieves actual observations and metadata, using named parameters to help prevent code from breaking when dimensions are added.

Transforming private instances of Data Commons Platform

For organizations that want to build private instances of the Data Commons Platform, this new modern architecture resolves legacy scaling limits and simplifies data schematization. Developers can instantiate a private instance of the Data Commons Platform leveraging the same scalable technology that powers Google’s Data Commons instance. As a private instance, users retain full control of their own data and have the ability to limit access, while enabling natural language queries to blend results from their private data with Google’s public data that is hosted on the Google Data Commons instance. By federating across our public knowledge graph and a private knowledge graph containing your own data, you can light up exciting new use cases, while maintaining data isolation and ensuring no data duplication.

For instance, a retail enterprise can combine public data such as national GDP trends, regional demographic breakdowns, and employment statistics, with their own enterprise data, including sales histories, store performance metrics, and supply chain logistics. This allows analysts to contrast public macroeconomic indicators against their own company transactions to optimize merchandise distribution and identify untapped markets.

Example of a natural language query combining statistical data from the Directorate General of Commercial Intelligence and Statistics (DGCIS) stored in a Data Commons Platform private instance with World Development Indicators from the World Bank stored in the Google Data Commons public instance.

A user is querying a Data Agent for average annual temperature trends in the country. The agent retrieves information from Data Commons, explaining that while historical data is available, it provides projected temperature changes, climate drivers, and CMIP6 climate model scenarios (SSPs), with options to export the generated report.

A user asks the Data Agent to compare the Worker Population Ratio (WPR) of rural versus urban males in a country. Fetching data from Data Commons, the agent defines WPR—the percentage of workers relative to the total population—and outlines the available demographic variables to analyze and compare both groups.

Get started today

Explore Data Commons: Visit datacommons.org to query global statistical knowledge.
Explore Spanner Graph's use cases and setup guide for your knowledge graphs.
Deploy Data Commons Platform: contact support@datacommons.org to request preview access and to review the developer tools.

The borderless Lakehouse: Bring AWS, Databricks and Snowflake data to your AI agents

Wed, 29 Jul 2026 16:00:00 +0000

Today’s data lakehouse is no longer mere data repository, but increasingly a system of action, actively executing tasks via always-on, autonomous AI agents. Rather than waiting for static reports, these agents run continuous, real-time reasoning loops, monitoring supply chains, flagging anomalies, and executing business workflows. To scale this model, AI agents need to access your entire data estate and the right context to understand what data to use and when. However, traditional data architectures, with their high costs, fragmented security, and weak governance, don’t make it easy.

Today at Next Tokyo, we’re introducing enhancements to our borderless Lakehouse. Built on open Apache Iceberg, it connects your on-premises, cross-cloud operational systems, and SaaS application clouds, so you can activate and query your data wherever it lives, without moving it.

Federating multiple catalogs with Iceberg REST

The borderless Lakehouse enables Gemini Enterprise and conversational agents to analyze and act on data regardless of its physical location. Built on the Iceberg REST catalog, it lets you discover and query remote data instantly, eliminating the high costs and delays of building data pipelines. This connectivity is made possible with catalog federation (now in preview) for AWS Glue, Databricks Unity, and Snowflake Horizon, providing secure, bi-directional access via BigQuery, Managed Service for Apache Spark, and any Iceberg-compatible engine.

This open approach also extends to the application layer, with zero copy data integrations to major SaaS applications like SAP, Salesforce, and Workday. BigQuery can now directly query live application data in these platforms without complex ETL pipelines, while they can run BigQuery's AI engines on their own data in-place to easily unify finance, HR, and customer data. These capabilities unlock three significant benefits:

Zero-copy, cross-cloud analytics: Instantly discover and query enterprise data across platforms without duplicating files, allowing various data teams to analyze the exact same copy of Apache Iceberg data.
Bidirectional interoperability: Read and write across environments, querying external tables from BigQuery or Managed Spark, then sharing derived datasets enriched by Google AI back to partner systems for downstream action.
Unified governance and access control: Get out-of-the-box governance with trusted context and Gemini insights for your agents. Secure access control at the table level with support for credential vending, regardless of which platform initiates the query.

The borderless Lakehouse expands to Google Cloud’s database portfolio which lets you integrate transactional systems with data lakehouses: Spanner Omni lets you run the highly scalable database in any environment outside of Google Cloud, and Lakehouse Federation for AlloyDB allows transactional systems to directly query warehouses. By eliminating costly data movement, your teams can securely and efficiently analyze live operational and historical data together in real time. Now your lakehouse is truly borderless.

Bring Google AI directly to your AWS and Azure data

Historically, running advanced analytics or training ML models across clouds meant a cross-cloud tax: high egress fees, network latency, and fragile ETL pipelines. The borderless Lakehouse solves this with Cross-Cloud Interconnects. These private, dedicated links deliver consistent bandwidth and lower latency than the public internet, at a fraction of the cost of traditional cross-cloud connections.

The borderless Lakehouse supports zero variable egress costs when accessing your data from AWS¹. With Partner Cross-Cloud Interconnect pricing, you get predictable monthly costs with an SLA-backed connection. This managed, private connectivity simplifies provisioning from 1G to 100G using a flat-rate, subscription-based model.

In addition, intelligent cross-cloud caching in the borderless Lakehouse securely stores remote data fragments temporarily inside Google Cloud to eliminate repeated, costly transfers for subsequent ad-hoc or BI queries. These capabilities, combined with BigQuery's vectorized processing, BigQuery AI functions for multimodal analysis, Spark's Lightning Engine runtime, and scalable metadata storage, scale to petabytes of data without sacrificing performance and enable:

In-place AI and machine learning: Apply powerful AI models and Gemini directly to AWS data and Azure without migration. Ingesting metadata and context right where it lives helps guarantee high-accuracy grounding and a faster time-to-market.
Avoid the cost and delay of data copying: Query live data stored in other clouds directly and bring your data closer to your agents.
Unified analytics experience: Deliver consistent, hardware-optimized performance across BigQuery, Spark, or open-source engines by centralizing multi-cloud compute back to Google Cloud’s infrastructure.

Bridging the agent trust gap with universal context

To prevent hallucinations, AI agents need more than raw technical metadata; they need deep business context. The borderless Lakehouse relies on Knowledge Catalog, our always-on agentic context engine, to establish a unified view of your enterprise context across clouds, without moving physical files.

To support this, the borderless Lakehouse’s runtime catalog automatically synchronizes with AWS Glue, Databricks Unity Catalog, and Snowflake Horizon, all in preview. Knowledge Catalog then ingests these feeds to aggregate, extract, and index their metadata, translating raw schemas into clear business terminology and searchable column-level lineage. As schemas change, Knowledge Catalog instantly updates their business meaning, delivering:

Lower costs and overhead: Minimize expensive, time-consuming data migration pipelines while gaining a consolidated view of your entire multi-cloud estate.
Trustworthy AI decisions: Provide agents with a clear semantic layer and lineage tracking so they can quickly discover, trust, and accurately interpret data.
Automated governance: Embed security directly into the metadata layer, helping ensure AI agents strictly respect compliance guardrails and access permissions.

Build and scale agents with Gemini Enterprise

By pairing the open-source Google Cloud Data Agent Kit with the Conversational Analytics API, you can build and publish custom data agents that operate on the borderless Lakehouse directly into Gemini Enterprise. This allows business users to talk to their data using natural language. The Data Agent Kit is a full-stack collection of agent building capabilities — meeting developers inside their favorite IDEs (like VS Code) — to package pre-codified analytical skills and Model Context Protocol (MCP) tools. With the borderless Lakehouse, your agents operate across your entire data estate and help with:

Integrated data and agent ecosystem: Built-in MCP tools establish secure, direct connections to BigQuery, Managed Spark, and Cloud Storage, eliminating the need to write complex pipeline code or copy-paste massive table schemas into LLM prompts.
Self-service analytics: Business users bypass static dashboards, querying and visualizing multi-cloud datasets instantly in plain language within the Gemini Enterprise interface.
Grounded, high-accuracy agent results: Agents run on top of the Knowledge Catalog, ensuring that natural-language-to-SQL translations are anchored in curated business schemas, highly secure, and strictly governed.

The economics of the borderless Lakehouse and agents

The borderless Lakehouse redefines enterprise AI economics by delivering compounded savings across data transfer, compute, and token consumption. By utilizing Cross-Cloud Interconnects and zero-copy sharing, you bypass fragile data pipelines and unpredictable egress fees to query remote datasets at a flat, predictable rate. Knowledge Catalog filters and delivers the precise, minimal business context required for each prompt, preventing token bloat and eliminating unnecessary reasoning loops. BigQuery AI prevents runaway agent billing through built-in token controls that let you estimate token usage pre-query, enforce strict limits, and leverage an optimized mode that automatically uses smaller, distilled models. In fact, customers are seeing 230x reduction in token consumption using BigQuery’s cost-optimized, built-in AI functions.

Next steps

The future belongs to the system of action, and the borderless Lakehouse allows your AI agents to query, reason, and act on your data wherever it lives — safely, instantly, and cost-effectively. To start building, check out the official Google Cloud Lakehouse About Guide and explore our Building a borderless Lakehouse codelab.

_{1. Customers are required to pay an hourly fee for interconnection service.}

Bringing Conversational Analytics to your entire data ecosystem

Tue, 28 Jul 2026 17:30:00 +0000

Increasing the adoption of generative AI across the enterprise requires you to do more than deploy a generic chatbot with a custom wrapper. Interacting with business-critical databases demands absolute trust, strict governance, and deep grounding in enterprise semantics.

Over the last year, Conversational Analytics (CA) in Google Cloud has moved from isolated experiments to scaled, enterprise-wide deployments. BigQuery Conversational Analytics and the Conversational Analytics API are now generally available, adding to the general availability of Conversational Analytics in Looker last year. Building on this momentum, Conversational Analytics in Databases are also available in Preview. And so much more has happened — Google Cloud Conversational Analytics is available for more data, across more surfaces, with more enterprise controls, and greater capability than ever before.

Let’s take a deeper look at the state of Conversational Analytics in the Google Data Cloud — what you can do with it, the benefits that it brings, and how to get started with it today.

Query across multi-cloud and database workloads

Conversational Analytics is now generally available for BigQuery and Looker, and in preview for AlloyDB, Cloud SQL, and Spanner. You can also analyze data stored in Lakehouse Managed Service tables, Apache Iceberg REST catalogs, and federated AWS S3 Unity Catalogs. Whether your data resides exclusively in Google Cloud or across multiple cloud providers, your agents can query it natively.

For data practitioners, Conversational Analytics is integrated directly into BigQuery Studio, BigQuery Data Canvas, Database Studio and Data Agent Kit. For business teams, these conversational capabilities extend directly into Looker, Data Studio, and Gemini Enterprise. Data teams can publish Conversational Analytics agents created in BigQuery, Looker, AlloyDB, Spanner, and Cloud SQL directly into Gemini Enterprise, giving business leaders a centralized interface to query complex data safely.

Our APIs and MCP tools let you embed Conversational Analytics wherever your business users work, like custom applications and multi-agent systems, or as slack chatbot that can answer questions across data sources, as we showed at Google Cloud Next.

Enterprise security and governance controlsScaling generative AI to tens of thousands of users requires ironclad governance and transparent cost controls. Conversational Analytics includes Customer Managed Encryption Keys (CMEK), Private IP, and Virtual Private Cloud (VPC) controls.

We guarantee Data Residency (DRZ) at rest and machine learning processing inside multi-region endpoints within the European Union and the United States, along with HIPAA compliance. For data access, role-based controls, including parameterized secure views in AlloyDB for PostgreSQL, help ensure users chatting with an agent only see data they are authorized to view, enforced down to row- and column-level permissions.

Monitoring Conversational Analytics in BigQuery to track agent fleet health, active users, query volumes, and top knowledge sources.

As usage grows, administrators need tools to manage costs, observe system health, and improve accuracy. You can configure native cost controls to define limits on maximum query sizes in bytes, and track usage through BigQuery query labels and Looker system activity logs.

To maintain fleet visibility, agents can also export health, tool usage, latency, and token consumption metrics via OpenTelemetry (OTEL) standards. Integrated feedback loops allow administrators to review agent traces and user feedback, establishing a foundation for continuous evaluation and accuracy improvements over time.

Grounded context through agent and data co-designWrapping a generic LLM around an enterprise database can sometimes lead to hallucinated logic. To minimize this, we co-designed Conversational Analytics agents alongside the data platforms they query.

For instance, agents leverage Knowledge Catalog for data discovery, glossaries, and automated context enrichment like table joins and descriptions. BigQuery Graphs and Spanner Graphs allow agents to query structured and unstructured data across multi-hop relationships. Additionally, Looker’s semantic layer (LookML) grounds agent responses in centrally governed metric definitions, helping ensure answers remain deterministic rather than relying on guessed SQL joins.

Grounding Conversational Analytics across Knowledge Catalog, BigQuery Graph, and Looker’s semantic model helps ensure deterministic, enterprise-governed responses.

Conversational Analytics agents are also co-designed with the data they query. This means their tools are context-aware, to have the best understanding of the metadata. They also benefit from built-in capabilities like multimodal data querying using BigQuery object tables, operating over multimodal data with ai.search, ai.generate_embedding, ai.classify, ai.score and using ai.forecast and ai.detect_anomalies to use the TimesFM foundation for forecasting and anomaly detection.Additionally, ai.key_drivers performs automated contribution analysis to pinpoint exactly what is driving unexpected changes in your data. When integrated with Looker, these agents leverage the semantic layer to ground their responses in centrally governed, deterministic metrics. To avoid AI hallucinations, this API-first approach (using 'Golden Queries') ensures agents retrieve verified business logic rather than guessing at SQL joins. Looker additionally equips the agents to seamlessly navigate high-cardinality datasets with dynamic filtering, automatically enforce row-level security during the chat experience, and surface context-aware suggested questions.

Proactive insights with Agentic Workflows

Analytics is moving beyond reactive question-answering toward proactive intelligence. That is, instead of requiring users to ask the right question at the right time, Conversational Analytics agents can run multidimensional deep dives to analyze 10 to 20 contributing factors behind a change in a metric.

With Agentic Workflows, now in preview, you can schedule automated reporting routines delivered directly into your chat workflow. Agents continuously run anomaly detection across key metrics, sending daily or weekly summaries straight to your team. Streaming anomaly detection can also launch an agent automatically the moment a key metric deviates from baseline thresholds.

Running a multi-step deep dive in Conversational Analytics to automatically investigate complex data relationships across enterprise datasets.

Flexible integration with APIs, SDKs, and MCP

Conversational Analytics is available to developers and business users in their existing environments. The Conversational Analytics API includes native SDKs for Node.js, Java, Go, Python, PHP, Ruby, and .NET and keeps insights where the work happens. We are expanding how and where people use Conversational Analytics, starting with Looker Dashboards and Data Studio, as well as supporting publishing agents to Gemini Enterprise.

You can also add Conversational Analytics to other multi-agent systems. Using the Agent Development Kit (ADK) and Model Context Protocol (MCP), you can integrate Conversational Analytics into custom applications, Slack bots, or multi-agent orchestrators. For example, a supply chain orchestrator agent can query a financial data agent to calculate the margin impact of a shipping delay in real time.

Get started with Conversational Analytics

Google Cloud Conversational Analytics unifies your data estate, security control plane, and developer APIs to deliver proactive data insights wherever your team works. Explore our Conversational Analytics documentation, review our quickstart repositories, and sign up to try our new previews today.

Announcing general availability of SAP Business Data Cloud Connect for BigQuery

Mon, 27 Jul 2026 16:30:00 +0000

Traditional data replication techniques often struggle to deliver the data freshness that modern workflows require. To help organizations overcome this challenge, SAP and Google Cloud are announcing that SAP Business Data Cloud Connect for BigQuery is now generally available.

As the next phase of our long-standing partnership, SAP BDC Connect for BigQuery gives you zero-copy access to data products across SAP Business Data Cloud and BigQuery – reducing silos, preserving valuable business context, and accelerating analytics and AI. This makes current, semantically rich data available across both environments, helping AI agents execute complex business tasks while your teams remain in control.

Stop copying data for AI projects

Built to support a zero-copy architecture, SAP BDC Connect for BigQuery establishes a secure, bi-directional connection between SAP Business Data Cloud and Google’s data and AI ecosystem. Access SAP tables, metadata, and business semantics directly in BigQuery and Knowledge Catalog (formerly Dataplex) and vice versa. Ground your AI agents in operational reality instead of raw, isolated database fields.SAP BDC Connect for BigQuery enables organizations to:

Scale AI on trusted data: Build governed, trustworthy AI grounded on a semantically rich data foundation with live, zero-copy access to data products across SAP Business Data Cloud and BigQuery.
Deploy multi-agent workflows: Orchestrate workflows with intelligent agents across SAP and Google with Gemini Enterprise Agent Platform and SAP Joule.
Discover deeper patterns: Combine mission-critical SAP data with unique datasets, such as Google Trends and Google Maps geospatial data, to create better business context. You can then publish this enriched data directly back to SAP.
Improve business analytics: Empower employees to ask open-ended business questions in natural language using Gemini Enterprise or Joule and receive immediate, context-aware answers grounded in SAP and Google Cloud data.

Scale your data value without inflating costs

SAP BDC Connect for BigQuery’s zero-copy architecture helps lower your total cost of ownership (TCO) compared to alternative data platforms by minimizing the need to replicate, store, and maintain duplicate copies of your data. Because you can query data in place, you don’t incur data sharing fees. This creates a predictable billing model that insulates your organization from unexpected charges.

How customers translate shared data into actions

Organizations with preview access to SAP BDC Connect for BigQuery are already using its zero-copy architecture to solve more complex operational challenges. By speeding up data pipelines, early adopters are breaking down global data silos, predicting supply chain disruptions, and responding to fluctuating customer demands faster. Contrast that with traditional data practices where data connections take weeks to build, stalling enterprise AI initiatives. By interacting directly with SAP and Google data, teams can rapidly build functional prototypes and accelerate return on their AI investments.

ElringKlinger is a global automotive supplier managing complex, multi-country manufacturing and supply chain data, and is using zero-copy to break down data silos. By querying real-time ERP data without having to replicate it, the company aims to reduce the complexity of its data infrastructure while supporting rapid production analytics with localized inventory reporting.

"Operating a global automotive footprint means managing vast, complex supply-chain and production data. Today, the majority of that data sits in SAP BW/4HANA, our established reporting backbone. Testing SAP BDC Connect for BigQuery showed a powerful new path forward: BigQuery as an analytical layer on top of our existing BW/4HANA assets — no data copy, no migration — unlocking AI-based Conversational Analytics while fully preserving our prior investment. This also lays the groundwork for agentic use cases that automate supply chain processes, giving teams real-time visibility into manufacturing signals worldwide." - Cleve Bankston, Project Manager and Dirk Brümmer Head of IT Architecture, ElringKlinger

Unify data to build smart workflows with partners

SAP and Google Cloud’s global partner network is actively building use cases with SAP BDC Connect for BigQuery. These help you move from isolated AI experiments to amplifying your team's ability to handle highly variable tasks that are difficult to manage manually. To make this possible, our partners combine trusted SAP data with broader sources to help you deploy agentic AI and autonomous operations. With this unified approach, your team can connect disparate systems, maintain governance, and drive operations that deliver bottom-line impact.

“As enterprises accelerate their reinvention agenda, the ability to unify data across business systems is foundational to scaling agentic AI. And that’s where SAP Business Data Cloud Connect for BigQuery is quite powerful. It enables enterprises real-time access to trusted financial, operational, and sustainability data, eliminating traditional data silos. This, combined with Gemini Enterprise and Accenture's deep industry expertise, allows organizations to build intelligent agents, drive autonomous operations, and modernize their digital core faster while unlocking new levels of business value.” – Chetna Sehgal, Global Practice Lead, Accenture Google Business Group

“Organizations don’t want a data science project; they want concrete business outcomes like working capital improvement and reduced operational costs. By leveraging this zero-copy integration, we built an autonomous procurement agent for a client. Because the agent reads live SAP inventory levels and market data inside BigQuery simultaneously, it autonomously re-routed orders during a recent supplier shift — saving millions in potential downtime and demonstrating that agentic AI tied to real-time data can deliver bottom-line impact.” - Piyush Bhandari, Managing Director, Deloitte Consulting LLP

“Based on a recent assessment and prototyping experience with a KPMG member firm energy client, we believe there is strong potential for SAP BDC Connect for BigQuery to help organizations modernize enterprise planning, analytics, and data integration on a scalable foundation. KPMG firms’ evaluations indicate that this integration can support higher-volume analytical workloads at enterprise scale, helping to provide a critical data foundation to allow exploration of future AI and agentic use cases. As with any transformation, realizing the full business value can depend on the implementation approach, data quality, and organizational readiness.” - Mark Shank, Principal, Advisory, Banking, Tech & Data Engineering, Google in KPMG US

“Our clients aren't just looking for more information; they’re looking for better business decisions. At PwC, we’re helping organizations unify SAP business data with broader enterprise data sources inside Google Cloud. This creates a single, trusted foundation for AI that strengthens analytics, supports robust governance and compliance, and drives intelligent workflows across the enterprise. This unified approach is how organizations move from isolated AI experiments to true, enterprise-wide business transformation.” - Patrick Pugh, Global and US Alliances and Ecosystem Leader, PwC US

Power of agentic data: Perspectives from SAP and Google Cloud

"Every enterprise is striving to inject intelligence and automation into the core of their business operations. By uniting SAP Business Data Cloud with BigQuery and Gemini, we are breaking down one of the biggest walls in enterprise data. We are going far beyond raw data access by actively sharing rich semantic metadata and agentic context across platforms at zero data-sharing cost. This ensures our joint customers can deploy autonomous AI agents that operate with complete, real-time business awareness — executing complex supply chain and financial processes with absolute precision." - Andi Gutmans, VP/GM, Agentic Data Cloud, Google Cloud

"SAP Business Data Cloud Connect for Google BigQuery delivers real-time, bidirectional, zero‑copy data sharing that brings SAP’s mission‑critical data to Google BigQuery making it accessible to Google Gemini and Gemini Enterprise Agent Platform. With SAP BDC Connect, customers can unify SAP and non‑SAP data without complex extracts or custom APIs, reducing data complexity, strengthening governance, and accelerating trusted, AI powered business outcomes." - Irfan Khan, President & Chief Product Officer, SAP Data & Analytics

Regional availability

SAP Business Data Cloud: Available today in Google Cloud regions in Australia (Sydney), Brazil (São Paulo), India (Mumbai), Israel (Tel Aviv), Germany (Frankfurt), Japan (Osaka), Saudi Arabia (Dammam) and the United States (Iowa).
SAP BDC Connect for BigQuery: Generally available globally for all Google Cloud customers.
Cross-cloud support: SAP BDC Connect for BigQuery supports SAP Business Data Cloud instances hosted on Google Cloud and AWS, with support for SAP Business Data Cloud instances hosted on Microsoft Azure coming soon.

Get started

To learn more, visit our website or speak with your Google Cloud account representative. For details on getting started with SAP BDC connect for BigQuery, visit our documentation page to get started with your zero-copy connection today.

Open Knowledge format v0.2 tackles agentic trust

Fri, 24 Jul 2026 16:00:00 +0000

When we introduced the Open Knowledge Format (OKF) in June 2026, we asserted that the context that agents need (table schemas, metric definitions, runbooks) should live in a format, not in a proprietary service, and not scattered across unstructured text blobs. Accordingly, OKF v0.1 started simple: just markdown, YAML frontmatter, and a handful of conventions.

The strong response and contributions from the developer community informed our focus for the next version. Since launch, contributors opened extension proposals (typed relationship edges, agent-routing hint fields, an optional erasure conformance profile, an .okfignore convention, and more), sent new sample bundles, and begun cataloging OKF ecosystem tools built outside Google. Many of these contributions and contributor feedback reflected a larger concern about OKF: once agents are writing to the corpus, can it really be trusted?

The most valuable OKF bundles won't be written by hand once and then read forever. They're written continuously, by agents, and consumed by a different set of agents. A human-authored wiki page comes with an implicit guarantee: a person wrote it, and you can hold them accountable if it is wrong. When an agent generates ten thousand concepts overnight, that guarantee is gone. To provide accountability, a consumer (often another agent) has to judge each concept on explicit signals instead, and needs to answer five questions:

What was this created from? (provenance)
How much should I trust it? (trust)
Is it still true? (freshness)
Is it the current version? (lifecycle)
Was this number produced the way we said it must be? (attestation)

In OKF v0.2, it is now possible to answer all five of those questions from frontmatter, while the format remains as minimally opinionated as v0.1. It adds vocabulary, not rules: type is still the only always-required field, every new field is opt-in, custom keys are still preserved rather than rejected, and a bundle that adopts none of the additions is exactly as valid as it was under v0.1. Everything new below is optional, but its absence now carries meaning: an unverified concept is distinguishable from a verified one (although never rejected for the difference).

From describing to deciding

v0.1 already kept metadata in frontmatter: type, title, description, resource, tags. Those fields describe a concept: what it is and what it points at. v0.2 adds a second kind of frontmatter field, the kind you use to decide something about a concept before you read it: who produced it, whether it has been verified, whether it is still current, and how a value it reports should be computed.

The reason these fields belong in frontmatter is that most interactions with a concept never actually progress to accessing the information in the body of the file. A consumer, whether a person, deterministic code, or an agent scanning during search and discovery, first has to decide whether a concept is relevant at all. Everything in a concept should be concise, but frontmatter has the narrower job of elevating exactly the signals needed to make decisions about relevance and trustworthiness, so it can be made cheaply and often, without spending tokens on prose. The content that must be read in full stays in the body, accessed only once a concept is chosen. Trust becomes something you can filter on before you commit to reading.

To make the sections that follow concrete, every example below draws from a small OKF v0.2 bundle we've prepared as a companion to this post: acme_retail, a fictional US retail company's shared knowledge for AI-assisted analytics over BigQuery:

code_block: <ListValue: [StructValue([('code', 'acme_retail/\r\n├── index.md, log.md\r\n├── tables/ orders.md\r\n├── metrics/ revenue.md, gross-margin.md, gross-margin-legacy.md\r\n├── computations/ revenue-ytd.md, gross-margin-period.md\r\n├── skills/ run-on-bq.md\r\n├── attesters/ sql_equality.py\r\n└── policies/ revenue-recognition.md, margin-standard.md'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda996a8710>)])]>

Each section that follows shows the corresponding file. Here is tables/orders.md, a concept carrying the new signals:

code_block: <ListValue: [StructValue([('code', "---\r\ntype: BigQuery Table\r\ntitle: Customer Orders\r\ndescription: One row per completed customer order across web, mobile, and marketplace channels. The grain is the order, not the line item.\r\nresource: https://bigquery.googleapis.com/v2/projects/acme/datasets/sales/tables/orders\r\ntags: [sales, orders, revenue]\r\ngenerated: { by: reference_agent/gemini-2.5-pro, at: 2026-06-30T14:00:00Z }\r\nverified:\r\n - { by: human:kliu@acme, at: 2026-07-01T16:00:00Z }\r\nstatus: stable\r\nstale_after: 2026-12-31\r\nsources:\r\n - id: warehouse-schema\r\n resource: https://wiki.acme.internal/data/warehouse/schemas/sales\r\n title: Acme Retail warehouse schema — sales dataset\r\n author: team:data-platform\r\n usage_count: 1240\r\n last_modified: 2026-06-15\r\n - id: revenue-policy\r\n resource: policies/revenue-recognition.md\r\n title: Revenue Recognition Policy (FY2026)\r\n author: human:jsmith@acme\r\n last_modified: 2026-06-15\r\n---\r\n\r\n# Schema\r\n\r\n| Column | Type | Description |\r\n|----------------|---------------|----------------------------------------------------------------------------------------|\r\n| `order_id` | STRING | Globally unique order id. [^warehouse-schema] |\r\n| `order_ts` | TIMESTAMP | Order placement time in UTC; drives fiscal-year assignment. [^revenue-policy] |\r\n| `order_status` | STRING | Revenue is recognized only at `'delivered'` and after the 30-day return window. [^revenue-policy] |\r\n| `net_amount` | NUMERIC(18,4) | `gross_amount - discount_amount`. The recognized-revenue amount per policy. [^revenue-policy] |"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda996abf90>)])]>

Each of those families answers one of the five questions.

Provenance: sources, not a score

The new sources field records the materials a concept derives from: an external doc, a bundle-relative path, or even a scope descriptor like "all queries in project X." At the same time, an entry can carry objective credibility signals: author, usage_count, last_modified.

The deliberate choice here is what we didn't add. OKF records the signals, not a credibility score. A score is subjective, doesn't port across consumers, and goes stale the moment it's written. Instead, credibility is inferred from the signals by whoever is consuming (and can be dynamically scored by the consumer, if desired), the same way you'd trust a heavily used, recently updated, authoritatively authored source more than an anonymous one. And when the body cites a specific source, it does so with an ordinary markdown footnote keyed to the source id ([^export-schema]), so attribution is per-claim instead of a dangling list at the bottom.

Trust: `generated`, `verified`

Trust is established using two fields, kept deliberately distinct, because who wrote something need not be who confirmed it:

generated: { by, at }: how the current content was produced, and when it last meaningfully changed.
verified: [ { by, at } ]: a list of independent confirmations against the sources or the underlying resource; a human sign-off, a nightly finance process, or both.

From verified, a consumer derives a trust tier: no verified key is unverified; confirmation by machine actors only is machine-confirmed; confirmation by a human:<id> actor is human-reviewed. Tiers are advisory signals, not access control, but they let a consumer say "only surface human-reviewed metrics in the executive dashboard" as a frontmatter filter.

In acme_retail: metrics/revenue.md is authored by the reference agent and verified by the VP of Finance, which places it in the human-reviewed tier. An executive-dashboard consumer configured with a trust-tier filter surfaces it; a throwaway testing environment can accept lower-tier concepts:

code_block: <ListValue: [StructValue([('code', 'type: Metric\r\ntitle: Revenue\r\ngenerated: { by: reference_agent/gemini-2.5-pro, at: 2026-06-30T14:00:00Z }\r\nverified:\r\n - { by: human:jsmith@acme, at: 2026-07-01T09:00:00Z }'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda996a9290>)])]>

Freshness and lifecycle: stale_after, status

OKF v0.2 establishes freshness and lifecycle with the stale_after and status fields. status moves a concept through draft → stable → deprecated (absent means stable). stale_after is a single absolute date. We chose an absolute date over a relative TTL on purpose: staleness becomes a plain date comparison with no reference to when the concept happened to be read, which is exactly the kind of determinism a non-LLM consumer wants.

In acme_retail: metrics/revenue.md and metrics/gross-margin.md both carry stale_after: 2026-12-31 because Acme's finance team re-approves the underlying policies every January. On 2027-01-01 both concepts require re-verification against the FY2027 policy before serving.

And metrics/gross-margin-legacy.md is status: deprecated. Acme changed its cost allocation standard in Feb 2026 (the old formula excluded shipping and fulfillment from its cost of goods sold). The legacy definition is preserved for historical query reproducibility but not surfaced to new work:

code_block: <ListValue: [StructValue([('code', 'type: Metric\r\ntitle: Gross Margin (legacy, pre-FY2026)\r\nstatus: deprecated'), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda996aa750>)])]>

Attestation: Was this number computed the sanctioned way?

Provenance answers where a claim came from. Attestation answers a harder question that matters the moment an agent reports a dollar figure: was this number produced the way we said it must be, or did the agent improvise its own SQL?

OKF v0.2 introduces a new concept type, Attested Computation. It carries not just what a value means but a sanctioned way to compute it, and the means to check that the sanctioned thing actually ran. Here is acme_retail/computations/revenue-ytd.md:

code_block: <ListValue: [StructValue([('code', "---\r\ntype: Attested Computation\r\ntitle: Revenue for a fiscal year\r\nruntime: bigquery\r\nparameters:\r\n - { name: year, type: integer, required: true }\r\nexecutor:\r\n resource: skills/run-on-bq.md\r\n receipt: [job_id, executed_sql, result]\r\nattester:\r\n resource: attesters/sql_equality.py\r\ngenerated: { by: reference_agent/gemini-2.5-pro, at: 2026-06-30T14:00:00Z }\r\nverified:\r\n - { by: human:jsmith@acme, at: 2026-07-01T09:00:00Z }\r\nstatus: stable\r\nstale_after: 2026-12-31\r\nsources:\r\n - id: revenue-policy\r\n resource: policies/revenue-recognition.md\r\n title: Revenue Recognition Policy (FY2026)\r\n author: human:jsmith@acme\r\n last_modified: 2026-06-15\r\n---\r\n\r\n# Computation\r\n\r\nSELECT\r\n SUM(\r\n CASE\r\n WHEN o.currency = 'USD' THEN o.net_amount\r\n ELSE o.net_amount * fx.rate_to_usd\r\n END\r\n ) AS revenue_usd\r\nFROM `acme.sales.orders` AS o\r\nLEFT JOIN `acme.finance.fx_daily_rates` AS fx\r\n ON fx.currency = o.currency\r\n AND fx.rate_date = DATE(o.order_ts)\r\nWHERE o.order_status = 'delivered'\r\n AND DATE_DIFF(CURRENT_DATE(), DATE(o.order_ts), DAY) >= 30\r\n AND EXTRACT(YEAR FROM o.order_ts) = @year"), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fda996abf10>)])]>

The agent may only fill the declared parameters; it must never author or edit the computation. A consumer runs the computation through the executor, which returns a receipt (here: a BigQuery job_id, the SQL that was actually executed, and the result). Then a deterministic, no-LLM attester inspects that receipt and returns a verdict: did the query that ran equal the sanctioned computation bound with the claimed parameters, and does the displayed value match the receipt's authoritative source? Because the comparison is mechanical, a rewritten query, a swapped computation file, or a mutated dependency fails the check.

In acme_retail: the attester at attesters/sql_equality.py canonicalizes both SQLs (strips comments, collapses whitespace, uppercases known keywords) and refuses to return ok if the canonical forms differ. A swapped table name, an added filter, a dropped JOIN all fail attestation. The consumer refuses to display the value when the verdict comes back false.

While we used BigQuery (and SQL) in this example, the abstraction is deliberately flexible. Attested computations can correspond to invoking semantic models (in Looker, AtScale, or other systems), querying structured knowledge graphs, or even making arbitrary API calls.

Crucially, OKF records the computation and how to check it; it never executes anything itself. And attestation is distinct from verification: verified confirms the definition still matches policy (slow, doc-level, stored in the bundle); attestation confirms a single run produced the value correctly (per-call, runtime, never stored in the bundle). A stale definition can still attest cleanly; a freshly-verified definition still needs attestation on every run. That's why both exist.

What we're releasing with v0.2

As with v0.1, the reference implementations are deliberately proofs of concept; nothing about OKF requires them. Here’s what’s changing in the Github repo:

reference_agent now emits the provenance and trust families as it generates, so a freshly-minted bundle arrives with generated sources, and citations already in place.
The static visualizer surfaces trust tier, status, and staleness alongside the concept graph, so these signals are visible, not just parseable.
Updated sample bundles (GA4 e-commerce, Stack Overflow, Bitcoin, and the acme retail example used in this blog post) carry the v0.2 fields.
A Knowledge Catalog demo shows a bundle round-tripping through Google Cloud's Knowledge Catalog (formerly Dataplex): clean OKF on disk, trust and provenance signals preserved through the catalog and back.

Compatibility

v0.2 is a minor version bump that is additive, backward-compatible, with two deliberate renames: timestamp is superseded by generated.at, and the body # Citations list is superseded by sources; in both cases a v0.2 consumer can fall back to the v0.1 form. A v0.1 bundle drops in unchanged.

Where we go from here

Read the spec (it's still short). Write a producer that emits trust signals for your source system. Write a consumer that filters on them. Try an Attested Computation against your own finance definitions. File issues, send PRs, propose extensions. A lingua franca is only as good as the number of parties who speak it, and now they can also check each other's work.

OKF v0.2 spec, samples, and reference implementations: github.com/GoogleCloudPlatform/knowledge-catalog/tree/main/okf

Your AI agents are ready. Is your data?

Thu, 23 Jul 2026 16:00:00 +0000

What’s one of the biggest bottlenecks stopping organizations from scaling their AI initiatives? It isn’t the capabilities of today’s models — it’s their access to business context and semantic meaning.

In the agentic era, enterprises need to go beyond simply storing data to activating it with trusted context, moving from passive systems of record to proactive systems of action.

But AI agents operate with nonlinear speed; for example, a single prompt can trigger the agent to independently browse, query, and execute across multiple systems, placing stress on the underlying infrastructure. If the compute, networking, and storage layers aren't optimized for agentic AI, the data platform sitting on top of them will buckle.

It’s no wonder that, according to our State of infrastructure report, 83% of organizations believe they require infrastructure upgrades to support production-grade agentic AI systems.

To solve this problem, we introduced the Agentic Data Cloud at Google Cloud Next 2026; unifying your data, AI models, and operational databases into a single System of Action. To make an Agentic Data Cloud work, it must be AI-native from the chip to the model. The underlying infrastructure must be able to accommodate agentic load.

Google’s Agentic Data Cloud

Let’s explore how the right infrastructure foundation empowers an Agentic Data Cloud to solve the biggest data challenges organizations face today.

Overcoming a lack of context

To be effective, agentic systems require access to context that is often found in fragmented data systems and legacy architectures. This can make it hard for agents to get this context, leading to incomplete, inaccurate results. In fact, our report found that 43% of IT leaders cite “difficulty integrating with legacy APIs and data sources” as their biggest agentic AI infrastructure gap.

But organizations cannot simply move massive datasets and connect them to AI without increasing complexity and cost.

Our Agentic Data Cloud solves this by leveraging a borderless Lakehouse running on open, flexible infrastructure. By accessing powerful native engines like BigQuery and Spanner over open standards (Apache Spark, Apache Iceberg), agents can read, reason over, and activate data across environments as if it were local, bypassing the latency and costs of traditional setups.

Escaping unnecessary manual work

Scaling agents on a patchwork of disconnected systems can create significant bottlenecks. In our research, 81% of leaders called out operational complexity and engineering overhead as top unforeseen expenses when scaling AI, citing the time engineers spend doing manual work to patch together AI agents across disparate systems.

To move from thinking to doing, agents must be able to connect real-time data across both analytical and operational sources. This requires vertical integration. When an Agentic Data Cloud is built on an AI-native infrastructure where the models, data systems, and underlying accelerators are co-designed, there are fewer network hops and tooling is better integrated. This unified system allows an agent to reach an insight and trigger secure transactions without the typical engineering overhead.

Bringing trust and knowledge to the data

It’s not enough for agents to just discover and query data. To take safe, accurate actions, agents also need rich context and business logic. Yet, 36% of leaders cite a lack of specialized, high-throughput vector databases used for AI model grounding, as a key infrastructure gap, hindering their ability to give agents context.

In order to work to their full potential, agents need a foundation which is built to read and write data systems in real-time, including legacy ERPs and third-party CRMs. It also gives them the long-term memory to recall a user’s preference from, say, three weeks ago, while executing a complex task today. And without this real-time automation, agents have to re-process data for every single query.

To provide context for AI, organizations are using Knowledge Catalog to aggregate and enrich data in their data lakes, and enable agentic searches. By extracting meaning from unstructured data and automatically generating semantics, the catalog acts as an active reasoning layer. That catalog in turn, must be backed by high-throughput infrastructure, so that agents can retrieve the right context.

The path forward

To turn AI into a true competitive advantage, it’s time to build a connected, active data ecosystem. Giving your agents seamless access to all of your data is a must to move from pilots to production, and this must be supported by an infrastructure that can handle the demands of the agentic era. The winners in 2026 and beyond won’t necessarily be the ones with the smartest agents. They’ll be the ones who can feed those agents the right knowledge — securely, cost-effectively, and at scale. Is your data ready for the agentic era?

See how leaders are taking an AI-optimized approach to architecture in the State of infrastructure in the agentic AI era report.