Traces and logs together

The traces-and-logs examples run a request handler with a multi-level span tree and a log line at every level, one of them on a spawned task. In the Google Cloud console, open a trace in Trace Explorer and its “Logs” panel lists each entry under the span that emitted it, or open an entry in Logs Explorer and follow its trace link back to the waterfall.

With OTLP

Spans go to the Telemetry API through OTLP, and so do the log records in development mode. APP_MODE (default development) picks the log sink:

• development: console lines, and the same entries over OTLP from the same builder, so that a run from a laptop shows the correlated entries in the console;
• production: JSON lines on stdout, which a logging agent on GKE, Cloud Run or GCE collects.

    let otlp = GcpOtlpExporterBuilder::new(google_project_id.clone()).with_resource(
        opentelemetry_sdk::Resource::builder()
            .with_service_name("traces-and-logs-otlp-example")
            .build(),
    );
    let tracer_provider = otlp.create_provider().await?;
    let tracer = otlp.install(&tracer_provider);

    // `Option<Layer>` implements `Layer`, so every sink is built from one `match` and composed
    // unconditionally below: an inactive sink is a `None` that costs nothing. Development mode
    // writes every event twice, to the terminal and over OTLP, purely so that a run from a
    // laptop shows the correlated entries in the console with nothing else to set up. A real
    // service picks one sink: the JSON layer where an agent collects stdout, or the OTLP layer
    // where none does, never both.
    let (console_layer, otlp_layer, logger_provider, json_layer) = match mode {
        AppMode::Development => {
            let logger_provider = otlp.create_logger_provider().await?;
            let otlp_layer = GcpCloudLoggingLayerBuilder::new(google_project_id.clone())
                .with_otel_logger_provider(
                    &logger_provider,
                    LogId::new("gcloud-observability-example")?,
                )
                .build()?;
            (
                Some(tracing_subscriber::fmt::layer()),
                Some(otlp_layer),
                Some(logger_provider),
                None,
            )
        }
        AppMode::Production => (
            None,
            None,
            None,
            Some(GcpCloudLoggingLayerBuilder::new(google_project_id.clone()).build()?),
        ),
    };

    // The OpenTelemetry layer goes before the logging layers: they read the span context it
    // attaches, and see only what is registered ahead of them.
    let subscriber = Registry::default()
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .with(console_layer)
        .with(otlp_layer)
        .with(json_layer);

    // A global subscriber, not a scoped `with_default`, because the workload below carries a
    // span across a `tokio::spawn` boundary onto a different thread.
    tracing::subscriber::set_global_default(subscriber)?;

PROJECT_ID=<your-google-project-id> cargo run --example traces-and-logs-otlp --features logs-otlp
PROJECT_ID=<your-google-project-id> APP_MODE=production cargo run --example traces-and-logs-otlp --features logs-otlp

The development mode writes every event to the terminal and over OTLP. A service picks one sink.

With the Cloud Trace API

Spans go to the Cloud Trace API and the entries to stdout, or to the Cloud Logging API:

    // `Option<Layer>` implements `Layer`, so every sink can be built from one
    // `match` and composed unconditionally below - an inactive sink is a
    // `None` that costs nothing on the subscriber. Development mode writes
    // every event twice, to the terminal and to the Cloud Logging API,
    // purely so that a run from a laptop shows the correlated entries in the
    // console with nothing else to set up. A real service picks one sink:
    // the JSON layer where an agent collects stdout, or the API layer where
    // none does - never both.
    let (console_layer, api_layer, api_handle, json_layer) = match mode {
        AppMode::Development => {
            let (api_layer, api_handle) =
                GcpCloudLoggingLayerBuilder::new(google_project_id.clone())
                    .with_cloud_logging_api(GcpCloudLoggingApiConfig::new(LogId::new(
                        "gcloud-observability-example",
                    )?))
                    .build_async()
                    .await?;
            (
                Some(tracing_subscriber::fmt::layer()),
                Some(api_layer),
                Some(api_handle),
                None,
            )
        }
        AppMode::Production => (
            None,
            None,
            None,
            Some(GcpCloudLoggingLayerBuilder::new(google_project_id.clone()).build()?),
        ),
    };

    // The OpenTelemetry layer must come first: the log layers read the span
    // context that layer attaches, and only see what is registered ahead of
    // them.
    let subscriber = Registry::default()
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
        .with(tracing_opentelemetry::layer().with_tracer(tracer))
        .with(console_layer)
        .with(api_layer)
        .with(json_layer);

    // A global subscriber, not a scoped `with_default`, because the workload
    // below carries a span across a `tokio::spawn` boundary onto a different
    // thread.
    tracing::subscriber::set_global_default(subscriber)?;

PROJECT_ID=<your-google-project-id> cargo run --example traces-and-logs-cloud-trace-api --features trace-api,logs-api

The development mode of this example writes every event to the terminal and to the Cloud Logging API, so that a run from a laptop shows the correlated entries in the console. A service picks one sink.

Global subscriber

Both examples install a global subscriber, not a scoped with_default one, because the workload carries a span across a tokio::spawn boundary with Instrument, onto a thread a thread-local subscriber is not set on.

The last console line of each example prints the trace ID together with direct links to Trace Explorer and Logs Explorer.

Full example available here.