Google Cloud observability for Rust

Formerly opentelemetry-gcloud-trace, see Migrating from 0.30.

The library sends what a Rust service emits through tracing and OpenTelemetry to Google Cloud:

• Cloud Logging: a tracing-subscriber layer that turns every event into a structured log entry, correlated with the trace and span that emitted it, so the Google Cloud console shows each log line under its trace;
• three log sinks: JSON lines on stdout for GKE, Cloud Run and hosts with the Cloud Logging agent, the Cloud Logging API, and OTLP;
• Error Reporting: errors logged with the code location, the source() chain and a backtrace, and panics, in the format Error Reporting groups;
• Cloud Trace: OpenTelemetry span exporters for the Telemetry API and for the Cloud Trace API v2;
• Authentication for any OTLP exporter, metrics included, with Application Default Credentials refreshed in the background.

Terms

OTLP is the OpenTelemetry protocol, the wire format OpenTelemetry SDKs send traces, metrics and logs in. The Telemetry API (telemetry.googleapis.com) is the Google Cloud endpoint that takes OTLP for all three signals; see Telemetry API overview. Application Default Credentials (ADC) is the way Google client libraries find credentials: an environment variable, a local file written by gcloud, or the metadata server of the host.

How the pieces fit

• A span is created by tracing macros or by the OpenTelemetry API. The tracing-opentelemetry layer turns tracing spans into OpenTelemetry spans, and a tracer provider hands finished spans to an exporter: otlp for the Telemetry API, or trace-api for the Cloud Trace API.
• A tracing event passes through the logging layer, which builds one log entry and gives it to the sink chosen when the layer was built.
• The logging layer reads the trace and span ids that the tracing-opentelemetry layer attached, so the entry and the span are linked in the console. The tracing-opentelemetry layer is therefore registered first, see The logging layer.
• The exporters, the Cloud Logging API sink and any OTLP exporter built elsewhere authenticate through the same ADC, see Authentication.

Neither exporter is enabled by default, see Feature flags. To start using the library, see Quick start.

Quick start

Cargo.toml:

[dependencies]
gcloud-observability = { version = "0.31", features = ["otlp"] }
opentelemetry_sdk = "0.33"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tracing-opentelemetry = "0.34"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }

Minimum supported Rust version is 1.91.

The logs feature, which provides the logging layer, is on by default. otlp adds the span exporter for the Telemetry API.

The setup below exports spans through OTLP and writes every event to stdout as a Cloud Logging JSON line that carries the trace and span it happened in:

    let otlp = GcpOtlpExporterBuilder::new(google_project_id.clone()).with_resource(
        opentelemetry_sdk::Resource::builder()
            .with_service_name("quickstart-example")
            .build(),
    );
    let tracer_provider = otlp.create_provider().await?;
    let tracer = otlp.install(&tracer_provider);

    let log_layer = GcpCloudLoggingLayerBuilder::new(google_project_id).build()?;

    // The OpenTelemetry layer goes before the logging layer: the logging layer reads the span
    // context the OpenTelemetry layer attaches, and sees only what is registered ahead of it.
    let subscriber = Registry::default()
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .with(log_layer);
    tracing::subscriber::set_global_default(subscriber)?;

The setup needs these imports:

use gcloud_observability::logs::GcpCloudLoggingLayerBuilder;
use gcloud_observability::otlp::GcpOtlpExporterBuilder;
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::{EnvFilter, Registry};

The JSON lines are what the Cloud Logging agent of GKE, Cloud Run and GCE collects from stdout. Run on a laptop they only print; the spans reach the Telemetry API either way.

Call tracer_provider.shutdown() before the process ends. It exports the spans still queued, and blocks the calling thread, so it needs the multi-threaded Tokio runtime that #[tokio::main] starts.

The credentials are found by ADC and need the role described in Authentication.

Full example available here.

Running the examples

All examples are in the examples directory, named after their goal. Each example names its features in its header comment and in required-features in Cargo.toml:

PROJECT_ID=<your-google-project-id> cargo run --example quickstart --features logs,otlp

Feature flags

The modules follow the features: logs, otlp and trace_api. ids holds LogId, which the logs-api and logs-otlp sinks take, and errors holds the error types. Nothing is re-exported at the crate root.

Choose the features by goal:

• logs to stdout for an agent: the defaults;
• logs through the Cloud Logging API: logs-api;
• logs and spans through the Telemetry API: logs-otlp;
• spans through the Telemetry API: otlp;
• spans through the Cloud Trace API: trace-api, see Choosing an exporter.

With default-features = false, enable a TLS roots feature and, for service account keys, auth-default-crypto:

[dependencies]
gcloud-observability = { version = "0.31", default-features = false, features = ["logs", "otlp", "tls-webpki-roots", "auth-default-crypto"] }

Authentication

Every exporter and sink that talks to Google authenticates with Application Default Credentials (ADC), found in this order:

• a JSON file whose path is in the GOOGLE_APPLICATION_CREDENTIALS environment variable;
• a JSON file in the location known to gcloud, written by gcloud auth application-default login;
• on Google Compute Engine, GKE and Cloud Run, the credentials of the metadata server.

The sinks that write to stdout need no credentials.

Local development

Don’t confuse gcloud auth login with gcloud auth application-default login. The first authorises only the gcloud tool; the second writes the user credentials that ADC reads.

A user credential sends its requests against a quota project. Set it with gcloud auth application-default set-quota-project <project> when requests are refused for lack of one. The credentials add the x-goog-user-project header themselves, and the library does not set it.

APIs to enable

Keep the Cloud Trace API enabled when exporting through the Telemetry API as well: with it disabled, Google Cloud Observability discards the trace data sent to the Telemetry API. See Telemetry API traces reference and the Cloud Trace release notes.

Roles

Grant the roles to the service account the application runs as.

The Telemetry API also needs roles/serviceusage.serviceUsageConsumer on the quota project.

Google’s migration guide for the Telemetry API lists narrower roles in place of roles/telemetry.writer: roles/telemetry.tracesWriter, roles/logging.logWriter and roles/monitoring.metricWriter. See Telemetry API overview and Migrate from the Cloud Trace API to OTLP endpoints.

Tokens and refresh

A tonic interceptor runs synchronously, while Google tokens are refreshed asynchronously. GcpAuthInterceptor bridges the two: it fetches the first headers when it is created, so bad credentials fail when the exporter is built rather than in a background task, and a background task polls the credentials every 5 seconds and publishes the headers when they change. A failed refresh keeps the headers published last. A warning is logged when refreshing starts failing and another when it recovers. The task runs on the Tokio runtime the interceptor was created on, and ends when the last clone is dropped.

All exporters built from one GcpOtlpExporterBuilder share one interceptor and one set of credentials. Pass other credentials with with_auth_headers, and build GcpAuthInterceptor yourself for an OTLP exporter built elsewhere, see Metrics over OTLP.

Crypto provider

Service account keys sign their tokens with a rustls crypto provider; see TLS and crypto features.

The logging layer

GcpCloudLoggingLayer is a tracing-subscriber layer. It turns every tracing event into one Cloud Logging entry, with these parts:

• severity, from the event level;
• message and target, and every other event field as the entry’s jsonPayload, Cloud Logging’s structured payload;
• logging.googleapis.com/sourceLocation, the file and line of the event;
• logging.googleapis.com/trace, spanId and trace_sampled, taken from the OpenTelemetry span the event happened in, which link the entry to its span in the console;
• httpRequest, labels and an operation, from conventional field names.

A JSON line from the default sink looks like this:

{
  "time": "2026-11-14T16:21:43.008982Z",
  "severity": "INFO",
  "message": "checkout request received",
  "target": "orders",
  "logging.googleapis.com/sourceLocation": { "file": "src/orders.rs", "line": "78" },
  "logging.googleapis.com/trace": "projects/my-project/traces/b3f6d198940b478f16da57f0c19e491b",
  "logging.googleapis.com/spanId": "cbdd640d48c893e0",
  "logging.googleapis.com/trace_sampled": true,
  "order_id": "order-42"
}

Building the layer

let log_layer = GcpCloudLoggingLayerBuilder::new(google_project_id).build()?;

build() returns an error when the project ID is empty or contains / or a control character: InvalidParametersError with the field google_project_id. GcpCloudLoggingLayerBuilder::for_default_project_id() takes the project of the environment the process runs in. with_source_location(false) leaves sourceLocation out.

A layer has exactly one sink, chosen when it is built.

Layer ordering

Register tracing_opentelemetry::layer() before the logging layer. A layer sees only the span context attached by layers registered ahead of it, so registering the logging layer first, or leaving the OpenTelemetry layer out, drops trace, spanId and traceSampled from every entry. The logging layer prints one warning to stderr the first time this happens.

    // `Option<Layer>` implements `Layer`, so both sinks are built from one `match` and composed
    // unconditionally below. The inactive one is a `None` that costs nothing.
    let (console_layer, json_layer) = if production {
        (
            None,
            Some(GcpCloudLoggingLayerBuilder::new(google_project_id).build()?),
        )
    } else {
        (Some(tracing_subscriber::fmt::layer()), None)
    };

    // The OpenTelemetry layer goes before the logging layer: the logging layer reads the span
    // context the OpenTelemetry layer attaches, and sees only what is registered ahead of it.
    let subscriber = Registry::default()
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .with(console_layer)
        .with(json_layer);
    tracing::subscriber::set_global_default(subscriber)?;

The layer reads the context through the public tracing_opentelemetry::get_otel_context API. Wrapping the layer in Option, Box<dyn Layer> or a per-layer filter needs no extra step. Under a scoped subscriber (tracing::subscriber::with_default) with the layer wrapped that way, the entry is correlated with the span entered on the thread, so an explicit parent: on an event is not honoured. A global subscriber has no such limit.

Full example available here.

Sinks

A layer writes its entries to one sink, chosen when the layer is built:

The JSON lines sink costs no queue, no background task and no network traffic of its own. Where an agent is present it is the cheaper and simpler choice.

Every sink builds its entry from the same event record, so the field conventions, Error Reporting and trace correlation behave the same on each.

The two network sinks, the Cloud Logging API and OTLP, skip events whose target is in h2, hyper, hyper_util, tonic, tower, rustls, gcloud_sdk, opentelemetry, opentelemetry_sdk or opentelemetry_otlp. At DEBUG or TRACE every export makes events in those crates, and an entry made of one is another export to make. The JSON lines sink writes them.

JSON lines

The default sink writes each entry as one JSON line to stdout. GKE, Cloud Run and a GCE instance with the Cloud Logging agent parse structured JSON from stdout, so the layer only has to serialise each event: the keys above map onto Cloud Logging’s LogEntry fields.

let log_layer = GcpCloudLoggingLayerBuilder::new(google_project_id).build()?;

with_json_writer takes any tracing_subscriber::fmt::MakeWriter in place of stdout. Each line is written with a single write_all, so concurrent events do not interleave partial lines on a shared writer. I/O errors are ignored: a full disk or a broken pipe does not turn into a panic.

On a host without the agent the lines only print. Use the Cloud Logging API or OTLP sink there.

Full example available here.

Cloud Logging API

The logs-api feature adds a sink that writes through the Cloud Logging API (logging.googleapis.com), for hosts with no logging agent to collect stdout. Entries are queued and written in batches by a background task on the Tokio runtime that build_async was called on.

    let (log_layer, log_handle) = GcpCloudLoggingLayerBuilder::new(google_project_id)
        .with_cloud_logging_api(GcpCloudLoggingApiConfig::new(LogId::new(
            "gcloud-observability-example",
        )?))
        .build_async()
        .await?;

    // The OpenTelemetry layer goes before the logging layer: the logging layer reads the span
    // context the OpenTelemetry layer attaches, and sees only what is registered ahead of it.
    let subscriber = Registry::default()
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .with(log_layer);
    tracing::subscriber::set_global_default(subscriber)?;

GcpCloudLoggingApiConfig::new takes a LogId, the log the entries go to: 1 to 511 letters, digits and / _ - .. LogId::from_static checks a constant at compile time. A / is URL-encoded as %2F in the logName, as Cloud Logging stores it.

A monitored resource is the resource an entry is attributed to, such as gce_instance or k8s_container. Set one with with_resource to have entries grouped with the rest of that resource’s logs.

A batch is written when batch_size entries are queued, or flush_interval after its first entry. When the queue is full, new entries are dropped instead of blocking the thread that logged them, and a message on stderr reports the drops.

Shutting down

build_async returns the layer and a GcpCloudLoggingHandle. Call handle.shutdown().await before the process ends: it closes the queue and waits, bounded by the shutdown timeout, for the final write. Dropping the handle without it leaves the queued entries unwritten. flush and shutdown are async and block no thread, so they work on a current-thread runtime.

Full example available here.

OTLP

The logs-otlp feature adds a sink that sends entries as OpenTelemetry log records to the Telemetry API. The same GcpOtlpExporterBuilder that exports the spans builds the logger provider, so one set of credentials and one endpoint serve both signals.

    let otlp = GcpOtlpExporterBuilder::new(google_project_id.clone()).with_resource(
        opentelemetry_sdk::Resource::builder()
            .with_service_name("logging-otlp-example")
            .build(),
    );
    let tracer_provider = otlp.create_provider().await?;
    let tracer = otlp.install(&tracer_provider);
    let logger_provider = otlp.create_logger_provider().await?;

    let log_layer = GcpCloudLoggingLayerBuilder::new(google_project_id)
        .with_otel_logger_provider(
            &logger_provider,
            LogId::new("gcloud-observability-example")?,
        )
        .build()?;

    // The OpenTelemetry layer goes before the logging layer: the logging layer reads the span
    // context the OpenTelemetry layer attaches, and sees only what is registered ahead of it.
    let subscriber = Registry::default()
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .with(log_layer);
    tracing::subscriber::set_global_default(subscriber)?;

with_otel_logger_provider takes the provider and the LogId of the log the entries are written to, which becomes the gcp.log.name attribute. Its build() is synchronous. The caller owns the provider and flushes it by shutting it down.

Shutting down

shutdown, force_flush and dropping the provider block the calling thread. The batch processor exports on a thread of its own, but the connection is driven by the Tokio runtime, which needs a thread free to do it. On a current-thread runtime they wait out their timeout and the queued records are not exported. Call them on a multi-threaded runtime, or from a thread outside the runtime.

The processor reports a failed export as an ERROR tracing event with the opentelemetry_sdk target. That target is on the list of transport crates the network sinks skip, so a failing export does not log itself into another export.

max_export_timeout bounds how long an export may take, see OTLP exporter.

Mapping

The Telemetry API maps an OTLP log record onto a LogEntry, see OTLP log record to LogEntry mapping. The sink follows it:

The mapping page names the line attribute code.function.number, a typo: Google reads code.line.number, and that is what the sink sets.

Attributes that map to no typed field become labels. The resource of the provider carries gcp.project_id, and service.name when set. See Comparison with the OpenTelemetry tracing bridge for entries side by side.

Entries sent over OTLP store trace as the bare trace ID rather than projects/PROJECT_ID/traces/TRACE_ID, so filter them with trace:"TRACE_ID", which matches both forms.

Full example available here.

Field conventions

Event fields map onto Cloud Logging’s structured LogEntry fields as follows. Anything else is written verbatim into the JSON payload.

The field names are not valid Rust identifiers, so they are written as string literals in the macros:

info!(
    "labels.tenant" = "acme",
    "http_request.request_method" = "GET",
    "http_request.request_url" = "https://example.test/orders",
    "http_request.status" = 200,
    "Serving a request."
);

Only the httpRequest.* keys are camel-cased, matching Google’s own field names; every other field name is written as given.

On the Cloud Logging API and OTLP sinks, a value that does not fit the typed field it is destined for, such as a malformed latency, stays in the payload instead of being dropped.

Errors

An event field passed as &dyn Error keeps its whole source() chain, one Caused by: line per source:

tracing::error!(error = &failure as &dyn std::error::Error, "checkout failed");

error = %err and error = ?err record only the text they format. See Error Reporting for how an error event is grouped.

One application, two modes

A service usually wants a console layer locally and the JSON layer once deployed, chosen by one match rather than duplicated main functions. Option<Layer> implements Layer, so exactly one sink is active per mode, without a boxed trait object or two separate subscriber types:

    // `Option<Layer>` implements `Layer`, so both sinks are built from one `match` and composed
    // unconditionally below. The inactive one is a `None` that costs nothing.
    let (console_layer, json_layer) = if production {
        (
            None,
            Some(GcpCloudLoggingLayerBuilder::new(google_project_id).build()?),
        )
    } else {
        (Some(tracing_subscriber::fmt::layer()), None)
    };

    // The OpenTelemetry layer goes before the logging layer: the logging layer reads the span
    // context the OpenTelemetry layer attaches, and sees only what is registered ahead of it.
    let subscriber = Registry::default()
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .with(console_layer)
        .with(json_layer);
    tracing::subscriber::set_global_default(subscriber)?;

The inactive layer is a None that costs nothing. Trace correlation keeps working with the logging layer wrapped in Option, with no extra step.

The example reads the mode from APP_MODE, default development:

PROJECT_ID=<your-google-project-id> APP_MODE=production cargo run --example logging-dev-and-prod --features logs,otlp

Run one sink in production. Writing every event to the terminal and to a network sink, as the development mode of the traces-and-logs-cloud-trace-api example does to show the correlation from a laptop, duplicates entries on a host that collects stdout.

Full example available here.

Comparison with the OpenTelemetry tracing bridge

opentelemetry-appender-tracing is the stock bridge from tracing events to OpenTelemetry log records. Sent to the Telemetry API, its records and those of the OTLP sink reach Cloud Logging differently, because the sink maps the conventional fields onto typed LogEntry fields before sending.

The same event, as the bridge sends it, trimmed:

{
  "logName": "projects/my-project/logs/event%20src%2Fbin%2Fbridge.rs%3A26",
  "severity": "INFO",
  "textPayload": "bridge probe checkout request received",
  "labels": {
    "http_request.request_method": "POST",
    "http_request.request_url": "https://example.test/orders/order-42/checkout",
    "http_request.status": "200",
    "labels.tenant": "acme"
  },
  "trace": "b5c11cf3e9cb0d1710251307eaaed739",
  "spanId": "49f19eb158811ff5",
  "traceSampled": true
}

and as the OTLP sink sends it, trimmed:

{
  "logName": "projects/my-project/logs/gcloud-observability-example",
  "severity": "INFO",
  "jsonPayload": {
    "message": "checkout request received",
    "target": "logging_otlp"
  },
  "httpRequest": {
    "requestMethod": "POST",
    "requestUrl": "https://example.test/orders/order-42/checkout",
    "status": 200,
    "latency": "0.125s"
  },
  "labels": {
    "gcp.log.name": "gcloud-observability-example",
    "tenant": "acme"
  },
  "sourceLocation": { "file": "examples/logging/otlp.rs", "line": "78" },
  "insertId": "1noy1uke8wh62",
  "trace": "b3f6d198940b478f16da57f0c19e491b",
  "spanId": "cbdd640d48c893e0",
  "traceSampled": true
}

The differences:

• the bridge writes a textPayload with the message only; the sink writes a jsonPayload with the message, the target and every other field with its type;
• the bridge sends every field as a string label, and the http_request.* fields stay labels; the sink fills the typed httpRequest and sourceLocation, and an operation and an insertId when the event has them;
• the bridge’s logName is made per call site; the sink writes every entry to the one log named by its LogId;
• both entries carry the trace and span of the enclosing span.

Error Reporting

Cloud Error Reporting groups the log entries it recognises as errors, counts them and shows them in one list. The logging layer writes that format on every sink when it is built with a service context:

    let (log_layer, log_handle) = GcpCloudLoggingLayerBuilder::new(google_project_id)
        .with_cloud_logging_api(GcpCloudLoggingApiConfig::new(LogId::new(
            "gcloud-observability-example",
        )?))
        .with_error_reporting(
            ServiceContext::new("gcloud-observability-error-reporting-example")
                .with_version(env!("CARGO_PKG_VERSION")),
        )
        .build_async()
        .await?;

    // The filter keeps the Cloud Logging client's own debug events out of the log it writes.
    let subscriber = Registry::default()
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
        .with(log_layer);

ServiceContext::new(service) names the service errors are grouped under, such as the Cloud Run service or the binary name, and with_version sets the deployed version Error Reporting shows beside the error.

Every event at ERROR severity or above becomes an error event, including an event with a severity field of critical, alert or emergency. Other events are untouched. The entry gets:

• @type, the ReportedErrorEvent type, which makes Error Reporting evaluate the entry;
• serviceContext, from the ServiceContext;
• context.reportLocation, from the file and line of the event, with the module path as the function name;
• context.httpRequest, mapped from the http_request.* fields when the event has them;
• a message made of the event message, the error, and a backtrace.

Without with_error_reporting an error event is written like any other.

The error in the message

Pass the error to the event as a &dyn Error, so that its source() chain is kept:

tracing::error!(error = &failure as &dyn std::error::Error, order_id, "checkout failed");

The first line of message is the event message followed by the Display of the error and of each source, outermost first: checkout failed: payment declined: card expired. error = %err and error = ?err record only the text they format, so the chain stops at the outer error.

The chain stays on that first line, and no Caused by: lines follow it: Error Reporting keys a group on the cause such a line names, so unrelated errors that share a root cause would become one group.

Grouping

Error Reporting groups errors with the same first line and error text, and it does so wherever they were logged: two call sites that log the same header and error are one group. A different text is a different group. Keep text that varies between occurrences, such as an order ID, in a field instead of the message.

context.reportLocation is always set, so grouping works without a backtrace. Error Reporting groups by the top frames of a stack trace when it has one.

OTLP sink

On the OTLP sink the body carries the same keys, so they reach jsonPayload, and the severity number is 17 or above. The sink also sets the exception.type and exception.message attributes, which Google maps to labels. A &dyn Error has no type name on stable Rust, so exception.type is set from an exception.type event field when the event has one.

Entries sent over OTLP are grouped by Error Reporting like the others.

Full example available here.

See Google’s Formatting error messages for the format.

Backtraces

An error event carries a backtrace when one is captured. The layer captures it with std::backtrace::Backtrace::capture() at the place the event is logged, so a capture costs nothing unless RUST_BACKTRACE or RUST_LIB_BACKTRACE enables it:

RUST_BACKTRACE=1 PROJECT_ID=<your-google-project-id> cargo run --example errors-error-reporting --features logs-api

Set one of the two variables in the environment of the deployed service to get backtraces in Error Reporting. The layer never forces a capture, so without them the entry has no backtrace.

The backtrace is rendered with Display, as std::backtrace frames, and follows the first line of message. Error Reporting has accepted Rust std::backtrace frames since 2026-08-13, see its release notes, and groups errors by their top frames. The trace is placed in message only, never in stack_trace, because Error Reporting reads stack_trace before message and the first present wins, which would hide the first line.

Error Reporting’s release note asks for debug symbols to be enabled. In a release build, debug = "line-tables-only" in [profile.release] keeps the file and line numbers of the frames.

Panic hook

install_panic_hook() emits one tracing error event for every panic, then calls the hook that was installed before it, so the default message on stderr stays. Call it once, early in main, after the subscriber is set:

    let (log_layer, log_handle) = GcpCloudLoggingLayerBuilder::new(google_project_id)
        .with_cloud_logging_api(GcpCloudLoggingApiConfig::new(LogId::new(
            "gcloud-observability-example",
        )?))
        .with_error_reporting(
            ServiceContext::new("gcloud-observability-panic-hook-example")
                .with_version(env!("CARGO_PKG_VERSION")),
        )
        .build_async()
        .await?;
    // The filter keeps the Cloud Logging client's own debug events out of the log it writes.
    let subscriber = Registry::default()
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
        .with(log_layer);
    tracing::subscriber::set_global_default(subscriber)?;

    install_panic_hook();

The event carries the panic message and panic.location, and context.reportLocation names that location rather than the hook. It reaches Error Reporting when the layer was built with with_error_reporting. The backtrace is captured inside the hook and needs RUST_BACKTRACE or RUST_LIB_BACKTRACE.

Flushing

The JSON lines sink writes the event before the hook returns. The Cloud Logging API and OTLP sinks only queue it and send it in the background, so it is lost when the process ends first: after panic = "abort", or when a panic unwinds out of main and nothing flushes on the way.

Shut down the GcpCloudLoggingHandle or the logger provider before the process ends. The example catches the panic with std::panic::catch_unwind and shuts the handle down afterwards. A guard that shuts the logger provider down when it is dropped covers a panic that unwinds out of main.

Full example available here.

Choosing an exporter

Two exporters send spans to Google Cloud Trace. Neither is enabled by default.

The Telemetry API takes traces, metrics and logs over OTLP. Choose otlp unless the case below applies. It shares credentials and an endpoint with the OTLP log sink and with metrics, and supports regional endpoints.

Choose trace-api when the project is in an Assured Workloads environment with data residency or Impact Level 4 (IL4, a US Department of Defense cloud security level) requirements: do not send spans to the Telemetry API there. See the section “Cloud Trace and data residency” of the Telemetry API traces reference.

Things to know before moving to OTLP

• Keep the Cloud Trace API enabled on the project, beside the Telemetry API: with it disabled, Google Cloud Observability discards the trace data sent to the Telemetry API. See the Telemetry API traces reference and the Cloud Trace release notes.
• The role to grant is roles/telemetry.writer, see Authentication.
• A VPC Service Controls restriction on the Telemetry API applies only to that service. The Cloud Trace API is not covered by it, so check the perimeter for the API the exporter sends to.
• Google notes that the Cloud Trace format can have lower limits for certain fields than OTLP, so some OTLP data may be lost or not map to Cloud Trace fields. See Migrate from the Cloud Trace API to OTLP endpoints.
• A regional endpoint, telemetry.REGION.rep.googleapis.com, is chosen with the exporter’s endpoint, see OTLP exporter.

Logging does not depend on the choice: the logging layer reads the span context from the tracing-opentelemetry layer, whichever exporter sits behind it.

OTLP exporter

The otlp feature adds GcpOtlpExporterBuilder, which builds OpenTelemetry exporters and providers that send spans, and with logs-otlp log records, to the Telemetry API through OTLP. It sets the endpoint, the TLS configuration, the gcp.project_id resource attribute and the authentication, see Authentication.

    let otlp = GcpOtlpExporterBuilder::new(google_project_id).with_resource(
        opentelemetry_sdk::Resource::builder()
            .with_service_name("traces-otlp-example")
            .build(),
    );
    let tracer_provider = otlp.create_provider().await?;
    let tracer = otlp.install(&tracer_provider);

create_provider builds a tracer provider with a batch span processor over the exporter, and install(&provider) returns a tracer for it. install is synchronous. Use create_provider_from_builder to start from your own SdkTracerProvider::builder(), with a sampler or an ID generator, for example.

Spans come from the OpenTelemetry API directly:

    tracer.in_span("my_parent_work", |cx| {
        let span = cx.span();
        span.set_attribute(KeyValue::new("http.client_ip", "42.42.42.42"));
        span.add_event(
            "test-event",
            vec![KeyValue::new("test_event_attr", "test-event-value")],
        );
        tracer.in_span("my_child_work", |cx| {
            cx.span().add_event(
                "test-child-event",
                vec![KeyValue::new("test_event_attr", "test-event-value")],
            );
        })
    });

or from tracing, through the tracing-opentelemetry layer, which is how most Rust code creates them:

    let subscriber = Registry::default().with(tracing_opentelemetry::layer().with_tracer(tracer));
    tracing::subscriber::with_default(subscriber, || {
        let root = span!(Level::INFO, "my_app", work_units = 2);
        let _enter = root.enter();

        span!(
            Level::INFO,
            "my_child",
            work_units = 2,
            "http.client_ip" = "42.42.42.42"
        )
        .in_scope(|| info!("Doing the child work."));

        error!("This event is added to the root span.");
    });

Resource

The provider’s resource always has gcp.project_id set to the project of the builder. A resource passed with with_resource is merged under it, key by key, so service.name is kept. create_provider starts from the OpenTelemetry SDK’s default resource. A builder passed to create_provider_from_builder without a resource of its own does not get that default: it ends up with gcp.project_id and the builder’s resource only, so set service.name on one of them.

The project ID is checked when an exporter is built: an empty ID, or one with / or a control character, fails with InvalidParametersError for the field google_project_id.

Span events

Span events are left out of the exported spans by default, so a tracing event is not shown twice in Trace Explorer, once as a span event and once as the log entry linked to the trace. with_span_events(true) keeps them. It applies to span_exporter and to every provider built from it. Links, attributes and status are exported either way, see Span events and annotations.

let otlp = GcpOtlpExporterBuilder::new(google_project_id)
    .with_span_events(true);

Regional endpoints

The default endpoint is https://telemetry.googleapis.com. A regional endpoint, telemetry.REGION.rep.googleapis.com, is set with with_endpoint, which takes a TelemetryEndpoint:

let otlp = GcpOtlpExporterBuilder::new(google_project_id)
    .with_endpoint("https://telemetry.europe-west1.rep.googleapis.com".parse()?);

TelemetryEndpoint accepts only https URIs with a host, because every request carries an access token. TLS is verified against the host of the endpoint.

Regional endpoints store log and trace data in the region. Metric data is stored there only when its location label matches the region or a zone of it. The endpoint does not decide where the observability bucket is created: that comes from the default settings, organisation policies and supported bucket locations. See Telemetry API overview.

Timeouts

max_export_timeout bounds how long a batch export may take before the batch is dropped. It is the timeout of connecting, the TLS handshake included, and of every export request, and for spans also the batch processor’s bound.

While it is unset, connecting, the TLS handshake and every request time out after 10 seconds, or after OTEL_EXPORTER_OTLP_TIMEOUT in milliseconds when that is set. The signal-specific OTEL_EXPORTER_OTLP_TRACES_TIMEOUT and OTEL_EXPORTER_OTLP_LOGS_TIMEOUT are not honoured. The span processor gives up after 30 seconds, or OTEL_BSP_EXPORT_TIMEOUT when that is set.

Shutting down

Call shutdown on the provider before the process ends. shutdown, force_flush and dropping the provider block the calling thread until the batch task, which runs on the Tokio runtime the provider was created on, is done, so on a current-thread runtime they never return. Call them on a multi-threaded runtime, or from a thread outside the runtime.

Shutdown exports every batch queued ahead of it, one after the other, and each may take the full timeout, so it can take the timeout once per 512 queued spans, rounded up. On a platform that kills the process shortly after SIGTERM, such as Cloud Run, keep that total below the time the platform allows:

let otlp = GcpOtlpExporterBuilder::new(google_project_id)
    .with_max_export_timeout(Duration::from_secs(3));

Full example available here.

Cloud Trace API exporter

The trace-api feature adds trace_api::GcpCloudTraceExporterBuilder, an exporter for the Cloud Trace API v2 (cloudtrace.googleapis.com). It authenticates with ADC, see Authentication, and needs the roles/cloudtrace.agent role.

    let exporter = GcpCloudTraceExporterBuilder::new(google_project_id)
        .with_resource(
            opentelemetry_sdk::Resource::builder()
                .with_service_name("traces-cloud-trace-api-example")
                .build(),
        )
        .with_span_events(true);
    let tracer_provider = exporter.create_provider().await?;
    let tracer = exporter.install(&tracer_provider);

create_provider and create_provider_from_builder build a tracer provider with a batch span processor, and install(&provider) returns a tracer. install is synchronous. with_resource sets the resource the exporter reports spans with.

Spans come from the OpenTelemetry API directly:

    tracer.in_span("my_parent_work", |cx| {
        let span = cx.span();
        span.set_attribute(KeyValue::new("http.client_ip", "42.42.42.42"));
        span.add_event(
            "test-event",
            vec![KeyValue::new("test_event_attr", "test-event-value")],
        );
        tracer.in_span("my_child_work", |cx| {
            cx.span().add_event(
                "test-child-event",
                vec![KeyValue::new("test_event_attr", "test-event-value")],
            );
        })
    });

or from tracing:

    let subscriber = Registry::default().with(tracing_opentelemetry::layer().with_tracer(tracer));
    tracing::subscriber::with_default(subscriber, || {
        let root = span!(Level::INFO, "my_app", work_units = 2);
        let _enter = root.enter();

        span!(
            Level::INFO,
            "my_child",
            work_units = 2,
            "http.client_ip" = "42.42.42.42"
        )
        .in_scope(|| info!("Doing the child work."));

        error!("This event is added to the root span.");
    });

Export timeout

with_max_export_timeout bounds how long a batch export may take before the batch is dropped. It defaults to the OpenTelemetry SDK default of 30 seconds, or OTEL_BSP_EXPORT_TIMEOUT when that is set. Shutdown exports every batch queued ahead of it, one after the other, so it can take the timeout once per 512 queued spans, rounded up. Keep that total below the time the platform allows after SIGTERM.

Shutdown blocks the calling thread until the batch task is done, so it needs a multi-threaded Tokio runtime, as for the OTLP exporter.

Full example available here.

Span events and annotations

tracing-opentelemetry records every tracing event inside a span as an event of that span. Neither exporter sends them by default. They would be reported twice, once with the span in Trace Explorer and once as a log entry by the logging layer.

• The OTLP exporter leaves them off the spans. with_span_events(true) on GcpOtlpExporterBuilder, before create_provider() or span_exporter(), sends them with the span.
• The Cloud Trace API exporter does not export them. with_span_events(true) on GcpCloudTraceExporterBuilder, before create_provider(), exports them as annotations (time_events).

Span status, links and attributes are exported either way.

Events added through the OpenTelemetry API with span.add_event(...) are span events too, and follow the same rule. Whether to log an event or attach it to a span is a choice of where it is read: a log entry is searched in Logging and linked to its trace, an annotation is shown on the span in Trace Explorer.

Traces and logs together

The traces-and-logs examples run a request handler with a multi-level span tree and a log line at every level, one of them on a spawned task. In the Google Cloud console, open a trace in Trace Explorer and its “Logs” panel lists each entry under the span that emitted it, or open an entry in Logs Explorer and follow its trace link back to the waterfall.

With OTLP

Spans go to the Telemetry API through OTLP, and so do the log records in development mode. APP_MODE (default development) picks the log sink:

• development: console lines, and the same entries over OTLP from the same builder, so that a run from a laptop shows the correlated entries in the console;
• production: JSON lines on stdout, which a logging agent on GKE, Cloud Run or GCE collects.

    let otlp = GcpOtlpExporterBuilder::new(google_project_id.clone()).with_resource(
        opentelemetry_sdk::Resource::builder()
            .with_service_name("traces-and-logs-otlp-example")
            .build(),
    );
    let tracer_provider = otlp.create_provider().await?;
    let tracer = otlp.install(&tracer_provider);

    // `Option<Layer>` implements `Layer`, so every sink is built from one `match` and composed
    // unconditionally below: an inactive sink is a `None` that costs nothing. Development mode
    // writes every event twice, to the terminal and over OTLP, purely so that a run from a
    // laptop shows the correlated entries in the console with nothing else to set up. A real
    // service picks one sink: the JSON layer where an agent collects stdout, or the OTLP layer
    // where none does, never both.
    let (console_layer, otlp_layer, logger_provider, json_layer) = match mode {
        AppMode::Development => {
            let logger_provider = otlp.create_logger_provider().await?;
            let otlp_layer = GcpCloudLoggingLayerBuilder::new(google_project_id.clone())
                .with_otel_logger_provider(
                    &logger_provider,
                    LogId::new("gcloud-observability-example")?,
                )
                .build()?;
            (
                Some(tracing_subscriber::fmt::layer()),
                Some(otlp_layer),
                Some(logger_provider),
                None,
            )
        }
        AppMode::Production => (
            None,
            None,
            None,
            Some(GcpCloudLoggingLayerBuilder::new(google_project_id.clone()).build()?),
        ),
    };

    // The OpenTelemetry layer goes before the logging layers: they read the span context it
    // attaches, and see only what is registered ahead of them.
    let subscriber = Registry::default()
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .with(console_layer)
        .with(otlp_layer)
        .with(json_layer);

    // A global subscriber, not a scoped `with_default`, because the workload below carries a
    // span across a `tokio::spawn` boundary onto a different thread.
    tracing::subscriber::set_global_default(subscriber)?;

PROJECT_ID=<your-google-project-id> cargo run --example traces-and-logs-otlp --features logs-otlp
PROJECT_ID=<your-google-project-id> APP_MODE=production cargo run --example traces-and-logs-otlp --features logs-otlp

The development mode writes every event to the terminal and over OTLP. A service picks one sink.

With the Cloud Trace API

Spans go to the Cloud Trace API and the entries to stdout, or to the Cloud Logging API:

    // `Option<Layer>` implements `Layer`, so every sink can be built from one
    // `match` and composed unconditionally below - an inactive sink is a
    // `None` that costs nothing on the subscriber. Development mode writes
    // every event twice, to the terminal and to the Cloud Logging API,
    // purely so that a run from a laptop shows the correlated entries in the
    // console with nothing else to set up. A real service picks one sink:
    // the JSON layer where an agent collects stdout, or the API layer where
    // none does - never both.
    let (console_layer, api_layer, api_handle, json_layer) = match mode {
        AppMode::Development => {
            let (api_layer, api_handle) =
                GcpCloudLoggingLayerBuilder::new(google_project_id.clone())
                    .with_cloud_logging_api(GcpCloudLoggingApiConfig::new(LogId::new(
                        "gcloud-observability-example",
                    )?))
                    .build_async()
                    .await?;
            (
                Some(tracing_subscriber::fmt::layer()),
                Some(api_layer),
                Some(api_handle),
                None,
            )
        }
        AppMode::Production => (
            None,
            None,
            None,
            Some(GcpCloudLoggingLayerBuilder::new(google_project_id.clone()).build()?),
        ),
    };

    // The OpenTelemetry layer must come first: the log layers read the span
    // context that layer attaches, and only see what is registered ahead of
    // them.
    let subscriber = Registry::default()
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .with(console_layer)
        .with(api_layer)
        .with(json_layer);

    // A global subscriber, not a scoped `with_default`, because the workload
    // below carries a span across a `tokio::spawn` boundary onto a different
    // thread.
    tracing::subscriber::set_global_default(subscriber)?;

PROJECT_ID=<your-google-project-id> cargo run --example traces-and-logs-cloud-trace-api --features trace-api,logs-api

The development mode of this example writes every event to the terminal and to the Cloud Logging API, so that a run from a laptop shows the correlated entries in the console. A service picks one sink.

Global subscriber

Both examples install a global subscriber, not a scoped with_default one, because the workload carries a span across a tokio::spawn boundary with Instrument, onto a thread a thread-local subscriber is not set on.

The last console line of each example prints the trace ID together with direct links to Trace Explorer and Logs Explorer.

Full example available here.

Metrics over OTLP

The library has no metrics API of its own. An opentelemetry-otlp MetricExporter is pointed at the Telemetry endpoint and authenticated with GcpAuthInterceptor, which attaches and refreshes the Google credentials of each request. The otlp feature enables the tonic transport and TLS of opentelemetry-otlp; enable its metrics feature in your own Cargo.toml:

[dependencies]
opentelemetry-otlp = { version = "0.33", default-features = false, features = ["metrics"] }

    let exporter = MetricExporter::builder()
        .with_tonic()
        .with_endpoint("https://telemetry.googleapis.com")
        .with_tls_config(ClientTlsConfig::new().with_enabled_roots())
        .with_interceptor(GcpAuthInterceptor::from_adc().await?)
        .build()?;

    let meter_provider = SdkMeterProvider::builder()
        .with_periodic_exporter(exporter)
        .with_resource(
            opentelemetry_sdk::Resource::builder()
                .with_service_name("metrics-otlp-example")
                .with_attribute(KeyValue::new(
                    "service.instance.id",
                    "metrics-otlp-example-1",
                ))
                .with_attribute(KeyValue::new("cloud.region", "europe-west1"))
                .with_attribute(KeyValue::new("gcp.project_id", google_project_id))
                .build(),
        )
        .build();

GcpAuthInterceptor::from_adc() uses the Application Default Credentials with the cloud-platform scope. GcpAuthInterceptor::new takes other GoogleAuthHeaders, which gcloud_observability::otlp re-exports. See Authentication for how the headers are kept current.

Resource attributes

The Telemetry API places each metric by the resource it comes from, and refuses a resource that misses one of these with InvalidArgument:

• gcp.project_id: the project;
• service.name and service.instance.id: the task;
• cloud.region, or cloud.availability_zone: where it ran. The value cannot be global.

Metric data is stored on a regional endpoint only when its location label matches the region or a zone of it, see OTLP exporter.

Call meter_provider.shutdown() before the process ends: it exports the metrics collected so far and stops the periodic reader.

Full example available here.

TLS and crypto features

Cargo provides support for different TLS features for dependencies:

• tls-roots: default feature to support native TLS roots;
• tls-webpki-roots: feature to switch to the roots of the webpki-roots crate;
• auth-default-crypto: default feature with the rustls crypto provider that google-cloud-auth signs service account keys with.

gcloud-sdk needs a TLS root store, so a build with default-features = false enables one of the two roots features. The OTLP exporter follows the same choice: tls-roots and tls-webpki-roots also enable the matching roots of opentelemetry-otlp.

Crypto provider

The default auth-default-crypto feature brings the aws-lc-rs rustls provider. The OTLP exporter’s TLS uses the same provider, so only one is compiled in. Be aware that aws-lc-rs is built from C sources.

With default-features = false and without auth-default-crypto, install a rustls CryptoProvider before creating an exporter or a logging layer that talks to Google, otherwise it fails with CryptoProviderMissing when the credentials are a service account key:

[dependencies]
rustls = "0.23"

rustls::crypto::ring::default_provider().install_default().expect("Failed to install rustls crypto provider");

An error like

no process-level CryptoProvider available -- call CryptoProvider::install_default() before this point

means more than one rustls provider is in the dependencies, and CryptoProvider::install_default() has to be called as above before creating the exporter.

Compatibility matrix

Versions up to 0.30 are published as opentelemetry-gcloud-trace.

Minimum supported Rust version is 1.91.

Limitations

• The exporters and the Cloud Logging API sink run on Tokio and need a Tokio runtime when they are created. Other runtimes are not supported.
• Shutting down a tracer provider or an OTLP logger provider blocks the calling thread, and on a current-thread runtime never returns or waits out its timeout. Shut them down on a multi-threaded runtime, or from a thread outside it. GcpCloudLoggingHandle::shutdown is async and has no such limit.
• The Cloud Logging API and OTLP sinks queue entries and send them in the background. Entries still queued when the process ends are lost, which includes a panic that unwinds out of main, see Panic hook.
• The Cloud Logging API sink drops entries when its queue is full.
• The Cloud Logging API and OTLP sinks skip events from the transport crates they export through, see Sinks.
• The OTLP exporters send only to https endpoints.
• The signal-specific OTEL_EXPORTER_OTLP_TRACES_TIMEOUT and OTEL_EXPORTER_OTLP_LOGS_TIMEOUT are not honoured, see OTLP exporter.
• A provider builder passed to create_provider_from_builder without a resource gets only gcp.project_id and the builder’s resource.
• There is no metrics API: metrics use opentelemetry-otlp with GcpAuthInterceptor.
• A &dyn Error event field keeps its source() chain; error = %err and error = ?err keep only the text.
• An error event has a backtrace only when RUST_BACKTRACE or RUST_LIB_BACKTRACE is set.
• The logging layer wrapped in Option, Box<dyn Layer> or a per-layer filter, under a scoped subscriber, is correlated with the span entered on the thread, not with an event’s explicit parent:.

Troubleshooting

Could not resolve host for telemetry.googleapis.com

DNS filters such as dnscrypt-proxy, Pi-hole and corporate blocklists often block every telemetry.* name. The example blocklist of dnscrypt-proxy, which OpenWrt enables, is one of them. The Telemetry API and its regional endpoints then fail with “could not resolve host”.

Allowlist telemetry.googleapis.com and telemetry.*.rep.googleapis.com. The upstream report is openwrt/packages#31291.

No trace or span on the log entries

The logging layer is registered before tracing_opentelemetry::layer(), or that layer is missing. Register the OpenTelemetry layer first, see The logging layer. The layer prints a warning to stderr the first time it finds a span without context.

CryptoProviderMissing

The credentials are a service account key and no rustls crypto provider is installed. Enable auth-default-crypto, or install a provider, see TLS and crypto features.

Entries or spans are missing at the end of a run

Queued entries and spans are exported in the background. Shut down the tracer provider, the logger provider or the GcpCloudLoggingHandle before the process ends. A provider shutdown on a current-thread Tokio runtime does not export, see Limitations.

InvalidParametersError

A value was refused before it was used. The field names it: google_project_id (empty, or containing / or a control character), log_id (1 to 511 letters, digits and / _ - .) or endpoint (an https URI with a host).

Metrics are refused with InvalidArgument

The resource lacks service.instance.id or a real cloud.region, see Metrics over OTLP.

Permission denied

Check the API is enabled and the role is granted for the service, see Authentication.

Migration guide

0.31

The crate is renamed from opentelemetry-gcloud-trace to gcloud-observability, and its Cloud Trace API exporter moves behind a feature.

The crate is no longer trace-only. With Google’s Telemetry API, which takes OTLP, a trace exporter alone adds little over opentelemetry-otlp. The value of the library is the Cloud Logging layer, Error Reporting and the authentication of OTLP exporters, so the name says what the crate is for.

• Version numbers continue from 0.30: the first release under the new name is 0.31.0.
• opentelemetry-gcloud-trace 0.31.0 is a forwarding crate. It re-exports gcloud-observability, with the same feature names, so a dependency on the old name keeps building. The breaking changes below apply to it as well.
• The GitHub repository is abdolence/gcloud-observability-rs. The old URL redirects.

Changes to make:

• the dependency is renamed, and trace-api or otlp is added to its features;
• the exporter types move to modules, and the error types are renamed;
• install is synchronous, and GcpCloudLoggingLayerBuilder::build() returns a Result;
• the log ID is a LogId;
• the example names changed;
• the instrumentation scope name changed.

The project ID is checked, and errors report their source differently, without a build failing.

Dependency and features

Rename the dependency and its use paths:

-opentelemetry-gcloud-trace = "0.30"
+gcloud-observability = { version = "0.31", features = ["trace-api"] }

-use opentelemetry_gcloud_trace::...;
+use gcloud_observability::...;

Without the rename, opentelemetry-gcloud-trace = { version = "0.31", features = ["trace-api"] } works, and the paths keep the old crate name: opentelemetry_gcloud_trace::trace_api::....

No exporter is enabled by default in 0.31. The exporter of 0.30 is the Cloud Trace API exporter, so features = ["trace-api"] keeps the behaviour. To export through the Telemetry API instead, use features = ["otlp"] and GcpOtlpExporterBuilder, see Choosing an exporter.

logs, logs-api, tls-roots, tls-webpki-roots and auth-default-crypto keep their names. logs-otlp and otlp are new.

Module paths

Nothing is re-exported at the crate root:

Error types

GcloudObservabilityError has a new variant, InvalidParametersError, so a match that lists every variant needs an arm for it.

Error display and source

The format of Display is unchanged. source() of GcloudObservabilityError returns the root cause of the inner error, or none, where 0.30 returned the inner error that Display had already printed. A program that prints the chain, such as anyhow or a tracing event with a &dyn Error field, shows each error once.

An error from gcloud-sdk has the message Google SDK error and the gcloud-sdk error as its source, where 0.30 put the source’s text in the message. Code that matches on the message text needs to read the source instead.

Install is synchronous

install takes &self, returns the tracer directly and has no await:

-let tracer = gcp_trace_exporter.install(&tracer_provider).await?;
+let tracer = gcp_trace_exporter.install(&tracer_provider);

Building the logging layer

GcpCloudLoggingLayerBuilder::build() returns a Result: it refuses an invalid project ID.

-let log_layer = GcpCloudLoggingLayerBuilder::new(project_id).build();
+let log_layer = GcpCloudLoggingLayerBuilder::new(project_id).build()?;

Log ID

GcpCloudLoggingApiConfig::new takes a LogId in place of a string. LogId is 1 to 511 letters, digits and / _ - ., and is checked when it is created:

-GcpCloudLoggingApiConfig::new("my-service")
+GcpCloudLoggingApiConfig::new(LogId::new("my-service")?)

LogId::from_static("my-service") checks a constant at compile time. LogId is in gcloud_observability::ids. A / in the log ID is URL-encoded as %2F in the logName, as Cloud Logging stores it.

Project ID

Building an exporter or a logging layer refuses a project ID that is empty or contains / or a control character, with InvalidParametersError for the field google_project_id.

Example names

The examples are named after their goal, and each has required-features in Cargo.toml:

quickstart, logging-otlp, errors-error-reporting, errors-panic-hook and metrics-otlp are new.

Instrumentation scope name

The tracer returned by install has the instrumentation scope name gcloud-observability, where 0.30 used opentelemetry-gcloud. Update a filter or a query that selects spans by the scope name.

Events written by the logging layer

An event field recorded as a &dyn Error keeps its whole source() chain, one Caused by: line per source. error = %err and error = ?err record the text only, as before.

The Cloud Logging API sink and the OTLP sink skip events from the crates they export through (h2, hyper, tonic, tower, rustls, gcloud_sdk, opentelemetry*), because at DEBUG every export would make events that are exported in turn. The JSON lines sink writes them.

New in 0.31

• The OTLP exporter for the Telemetry API, and the OTLP log sink. The OTLP log processor exports on a thread of its own, and reports a failed export as an ERROR tracing event with the opentelemetry_sdk target. The OTLP trace exporter does not send span events unless with_span_events(true) is set, as with the Cloud Trace API exporter.
• Error Reporting, with backtraces and a panic hook.
• GcpAuthInterceptor, to authenticate any OTLP exporter, see Metrics over OTLP.

Migrating from tracing-stackdriver

Swap the tracing-stackdriver layer for GcpCloudLoggingLayer at the same position in the subscriber stack:

- .with(tracing_stackdriver::layer().with_cloud_trace(
-     tracing_stackdriver::CloudTraceConfiguration { project_id },
- ))
+ .with(gcloud_observability::logs::GcpCloudLoggingLayerBuilder::new(project_id).build()?)

The layer reads the trace and span IDs through the public tracing-opentelemetry API, so register tracing_opentelemetry::layer() before it, see The logging layer.

Two things change in the emitted log lines:

• there is no span or spans object carrying the enclosing span’s name and fields: the span attributes live on the trace, and are not repeated into every log line;
• event field names are written as given, not camel-cased; only the httpRequest.* keys are camel-cased, matching Google’s own field names, see Field conventions.

A layer has one sink. JSON lines on stdout, the default, fit GKE, Cloud Run and hosts with the Cloud Logging agent.