Terraform Enterprise Log Forwarding

Terraform Enterprise supports forwarding its logs to one or more external destinations, a process called log forwarding. Log forwarding provides increased observability, assistance complying with log retention requirements, and information during troubleshooting.

Requirements

Log forwarding requires:

Terraform Enterprise running on an instance using systemd-journald. Execute systemctl status systemd-journald to check if the systemd-journald service is started and enabled.
A version of Docker that supports the journald logging driver. Execute docker info --format '{{.Plugins.Log}}' to check if the journald plugin is listed.
Network connectivity between Terraform Enterprise and the external destination(s) where logs should be forwarded.

Enable Log Forwarding

Log forwarding is disabled by default. To enable log forwarding, set the log_forwarding_enabled Terraform Enterprise application setting to the value 1.

tfe-admin app-config -k log_forwarding_enabled -v 1

When log forwarding is enabled, the Terraform Enterprise application settings show the following for log_forwarding_enabled:

    "log_forwarding_enabled": {        "value": "1"    },

Configure External Destinations

The log_forwarding_config Terraform Enterprise application setting must contain valid Fluent Bit [OUTPUT] configuration specifying supported external destination(s) where Terraform Enterprise should forward logs. The default configuration does not forward any logs.

Since the Terraform Enterprise application settings are stored as JSON strings, we recommend first creating a fluent-bit.conf file with the valid Fluent Bit [OUTPUT] configuration and then using that file to configure the log_forwarding_config application setting. This method ensures that the configuration is stored in the application settings exactly how it appears in the fluent-bit.conf file.

For a Standalone installation of Terraform Enterprise:

replicatedctl app-config set log_forwarding_config --value "$(cat fluent-bit.conf)"

For an Active/Active installation of Terraform Enterprise:

tfe-admin app-config -k log_forwarding_config -v "$(cat fluent-bit.conf)"

Once configured, the Terraform Enterprise application settings show the log_forwarding_config setting in escaped JSON string format:

    "log_forwarding_config": {        "value": "# Match all logs and do not forward them anywhere.\n[OUTPUT]\n    Name null\n    Match *\n"    },

That escaped JSON string renders to the following:

# Match all logs and do not forward them anywhere.[OUTPUT]    Name null    Match *

To forward logs to multiple external destinations, use multiple [OUTPUT] directives.

# Forward all logs to Datadog.[OUTPUT]    Name datadog    Match *    ... # Forward all logs to Fluent Bit or Fluentd.[OUTPUT]    Name forward    Match *    ...

Note: Do not use an [OUTPUT] directive with the stdout Fluent Bit output plugin. Doing this creates a loop that continuously emits logs!

Restart Terraform Enterprise

Once log forwarding is enabled and configured, you need to restart Terraform Enterprise for the changes to take effect. Please refer to the HashiCorp Help Center for details on how to restart Terraform Enterprise.

Supported External Destinations

You can only forward logs to one of the supported external destinations below. Each supported external destination contains example configuration for convenience.

Amazon CloudWatch

Sending to Amazon CloudWatch is only supported when Terraform Enterprise is located within AWS due to how Fluent Bit reads AWS credentials.

This example configuration forwards all logs to Amazon CloudWatch. Refer to the cloudwatch_logs Fluent Bit output plugin documentation for more information.

[OUTPUT]    Name               cloudwatch_logs    Match              *    region             us-east-1    log_group_name     example-log-group    log_stream_name    example-log-stream    auto_create_group  On

Note: In Terraform Enterprise installations using AWS external services, Fluent Bit will have access to the same AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables that are used for object storage.

Amazon S3

Sending to Amazon S3 is only supported when Terraform Enterprise is located within AWS due to how Fluent Bit reads AWS credentials.

This example configuration forwards all logs to Amazon S3. Refer to the s3 Fluent Bit output plugin documentation for more information.

[OUTPUT]    Name                          s3    Match                         *    bucket                        example-bucket    region                        us-east-1    total_file_size               250M    s3_key_format                 /$TAG/%Y/%m/%d/%H/%M/%S/$UUID.gz    s3_key_format_tag_delimiters  .-

Azure Blob Storage

This example configuration forwards all logs to Azure Blob Storage. Refer to the azure_blob Fluent Bit output plugin documentation for more information.

[OUTPUT]    name                   azure_blob    match                  *    account_name           example-account-name    shared_key             example-access-key    path                   logs    container_name         example-container-name    auto_create_container  on    tls                    on

Azure Log Analytics

This example configuration forwards all logs to Azure Log Analytics. Refer to the azure Fluent Bit output plugin documentation for more information.

[OUTPUT]    name         azure    match        *    Customer_ID  example-log-analytics-workspace-id    Shared_Key   example-access-key

Datadog

This example configuration forwards all logs to Datadog. Refer to the datadog Fluent Bit output plugin documentation for more information.

[OUTPUT]    Name        datadog    Match       *    Host        http-intake.logs.datadoghq.com    TLS         on    compress    gzip    apikey      example-api-key    dd_service  terraform_enterprise    dd_source   docker    dd_tags     environment:development,owner:engineering

Forward

This example configuration forwards all logs to a listening Fluent Bit or Fluentd instance. Refer to the forward Fluent Bit output plugin documentation for more information.

[OUTPUT]    Name   forward    Match  *    Host   fluent.example.com    Port   24224

Google Cloud Platform Cloud Logging

Sending to Google Cloud Platform Cloud Logging is only supported when Terraform Enterprise is located within GCP due to how Fluent Bit reads GCP credentials.

This example configuration forwards all logs to Google Cloud Platform Cloud Logging (formerly known as Stackdriver). Refer to the stackdriver Fluent Bit output plugin documentation for more information.

[OUTPUT]    Name       stackdriver    Match      *    location   us-east1    namespace  terraform_enterprise    node_id    example-hostname    resource   generic_node

Note: In Terraform Enterprise installations using GCP external services, Fluent Bit will have access to the GOOGLE_SERVICE_CREDENTIALS environment variable that points to a file containing the same GCP Service Account JSON credentials that are used for object storage.

Splunk Enterprise HTTP Event Collector (HEC)

This example configuration forwards all logs to Splunk Enterprise via the HTTP Event Collector (HEC) interface. Refer to the splunk Fluent Bit output plugin documentation for more information.

[OUTPUT]    Name          splunk    Match         *    Host          example-splunk-hec-endpoint    Port          8088    Splunk_Token  example-splunk-token

Syslog

This example configuration forwards all logs to a Syslog-compatible endpoint. Refer to the syslog Fluent Bit output plugin documentation for more information.

[OUTPUT]    Name                 syslog    Match                *    host                 example-syslog-host    port                 514    mode                 tcp    syslog_message_key   message    syslog_severity_key  PRIORITY    syslog_hostname_key  _HOSTNAME    syslog_appname_key   SYSLOG_IDENTIFIER    syslog_procid_key    _PID

Audit Logs

Terraform Enterprise emits its audit logs along with its application logs. Currently, log forwarding can forward either all Terraform Enterprise logs or no logs at all. To distinguish audit logs from application logs, audit log entries contain the string [Audit Log].

Here's an example audit log entry formatted for readability:

2021-08-31 04:58:30 [INFO] [7a233ad1-c50c-4737-a925-3be901e55fcb] [Audit Log]{  "resource":"run",  "action":"create",  "resource_id":"run-nL77p69bsesoF3RK",  "organization":"example-org",  "organization_id":"org-pveSPvxocni226Fn",  "actor":"example-user",  "timestamp":"2021-08-31T04:58:30Z",  "actor_ip":"19.115.231.192"}

If you have a requirement to split audit logs from application logs, we recommend forwarding all Terraform Enterprise logs to a log aggregation system, filtering the audit logs based on the [Audit Log] string, and forwarding just the audit logs to the desired destination.

Log Rotation

Log forwarding uses the journald Docker logging driver to send Terraform Enterprise logs to systemd-journald. This can cause increased disk utilization for the /var/log/journal path.

To limit disk utilization, configure the SystemMaxFileSize and SystemMaxFiles settings within /etc/systemd/journald.conf.

The following configuration tells systemd-journald to use up to 7GB of disk space by limiting the size a log file to 1024MB and keeping up to 7 files at any given time:

[Journal]SystemMaxFileSize=1024MSystemMaxFiles=7

To apply these changes, restart systemd-journald:

sudo systemctl restart systemd-journald