SaltStack Config exposes several system metrics that can be used for monitoring and diagnostics. These metrics are available in graphical form on the SaltStack Config user interface dashboard and in machine-readable form using the /metrics http endpoint.

For more information about visualizing reports in the SaltStack Config user interface using the Dashboard, see Dashboard Reports.

Machine-readable metrics

SaltStack Config exports system metrics in OpenMetrics text-based format. This format is directly consumable by Prometheus and other monitoring and alerting tools.

SaltStack Config metrics configuration

Configuration for system metrics collection consists of the following settings in the /etc/raas/raas configuration file. Default values are shown.

# System metrics settings
metrics:
  enabled: true                         # If True, enable the collection of system metrics
  prometheus: false                     # If True, enable the Prometheus endpoint at /metrics
  prometheus_username:                  # Static username for retrieving /metrics
  prometheus_password:                  # Static password for retrieving /metrics
  snapshot_interval: 60                 # How often to record snapshot metrics, in seconds
  max_query_timedelta: 86400            # Maximum timedelta for a single call to get_system_metrics, in seconds
  keep: 30                              # How long to retain metrics data, in days

The following settings control the handling of machine-readable system metrics:

  • To disable metrics collection, set enabled:false. Note that this will also disable the SaltStack Config built-in dashboard.
  • To enable the export of machine-readable metrics from the /metrics http endpoint, set prometheus:true. This setting does not affect the SaltStack Config built-in dashboard.
  • Access to the /metrics http endpoint is controlled by http Basic Authentication with credentials configured in prometheus_username and prometheus_password. Non-empty values are required for these settings to enable the /metrics endpoint. These credentials are stored only in the /etc/raas/raas configuration file, are not associated with any SaltStack Config account, and cannot be used to authenticate to SaltStack Config other than for accessing the /metrics http endpoint.

The other settings shown above relate to the SaltStack Config built-in dashboard and do not affect the collection or reporting of machine-readable system metrics. In particular, the snapshot_interval setting determines how often, in seconds, metrics are recorded for display on the dashboard, and the keep setting determines how long, in days, metrics data will be kept in the database before being trimmed.

Although the /metrics http endpoint is the recommended way to gather machine-readable metrics data from SaltStack Config, you can use the API (RaaS) to retrieve the data presented on the built-in dashboard. The stats.get_system_metrics() API call lets you query metrics data by metric name, source, and date range. The configuration item max_query_timedelta limits how much data SaltStack Config will return from a single API call. To get metrics data from a longer time span, you can make multiple API calls with different start and end dates.

Salt Master metrics configuration

The availability of some metrics depends on the configuration of the Salt masters connected to SaltStack Config:

  • Metrics on Salt events and job returns will be accurate only if the sseapi returner is configured on the Salt masters. Metrics collection will not work properly with the sse_pgjsonb (direct-to-database) returner.
  • SaltStack Config will collect low-level function runtime information from Salt masters that have master_stats:true set in their configuration. This option is disabled by default. See master_stats in the Salt documentation for details.
  • Metrics on salt job states will be accurate only if the job completion engine is enabled in the Salt Master Plugin:
    engines:
   -jobcompletion:{}

    This engine is enabled in the Salt Master Plugin default configuration.

Configuring Prometheus to connect to SaltStack Config

You can enable a Prometheus server to scrape metrics from SaltStack Config by adding a scrape_configs job to the Prometheus configuration (typically prometheus.yml) for each API (RaaS) server instance you have:

scrape_configs:
  - job_name: 'sse'

    metrics_path: '/metrics'
    scheme: 'http'

    static_configs:
        - targets: ['localhost:8080']

    basic_auth:
      username: prometheus
      password: metrics

The credentials in the Prometheus configuration should match the prometheus_username and prometheus_password specified in the /etc/raas/raas configuration file, as noted above.

See the Prometheus project documentation for more information on setting up scrape targets and other Prometheus configuration topics.

Note: As part of VMware’s initiative to remove problematic terminology, the term Salt master will be replaced with a better term in SaltStack Config and related products and documentation. This terminology update may take a few release cycles before it is fully complete.

Metric descriptions

The machine-readable metrics that SaltStack Config exports fall into several categories:

Category Metric Name Metric Type Labels Description
Salt master low-level metrics salt_event_size_bytes Histogram master_id Salt event size, in bytes
salt_master_cmd_duration_seconds Histogram master_id, cmd Salt master command duration, in seconds. Reported only if master_stats is configured on the Salt master.
         
Salt Master Plugin metrics raas_master_commands_processed Counter master_id SSE commands processed
raas_master_master_grains_pushed Counter master_id Salt master grain updates pushed to SaltStack Config
raas_master_minion_keys_pushed Counter master_id Minion key states updates pushed to SaltStack Config
raas_master_minion_cached_pushed Counter master_id Minion cache updates pushed to SaltStack Config
raas_master_masterfs_pushed Counter master_id MasterFS updates pushed to SaltStack Config
raas_master_sseapi_engine_iteration_seconds Histogram master_id API (RaaS) engine iteration duration, in seconds
         
Server metrics redis_commands_executed Counter redis_instance Redis commands executed (system cache)
redis_memory_bytes Gauge redis_instance Redis memory usage (system cache)
celery_tasks_queued Counter raas_instance, task Celery tasks queued (background jobs)
celery_tasks_executed Counter raas_instance, task Celery tasks executed (background jobs)
celery_queue_length Gauge raas_instance Celery queue length (background jobs waiting)
raas_rpc_request_duration_seconds Histogram raas_instance SSE RPC API call duration, in seconds
         
PostgreSQL metrics postgres_connections Gauge postgres_instance Postgres connections
postgres_transactions Counter postgres_instance Postgres transactions committed
postgres_rows_read Counter postgres_instance Postgres rows read
postgres_rows_inserted Counter postgres_instance Postgres rows inserted
postgres_rows_updated Counter postgres_instance Postgres rows updated
postgres_rows_deleted Counter postgres_instance Postgres rows deleted
         
System metrics sse_jobs_in_progress Counter None SaltStack Config jobs in progress
sse_jobs_complete_all_successful Counter None SaltStack Config jobs complete with all successful returns
sse_jobs_complete_missing_returns Counter None SaltStack Config jobs complete with one or more missing returns
sse_jobs_complete_with_errors Counter None SaltStack Config jobs complete with one or more errors
sse_masters Gauge None Total Salt masters in SaltStack Config
sse_minions Gauge None Total minions in SaltStack Config
sse_minions_present Gauge master_id Minions present within the configured time limit raas_presence_expiration
sse_minions_lost Gauge master_id Minions not present within the time limit
sse_minions_unknown Gauge master_id Unknown minions (never present)
sse_users_authenticated Gauge None Users authenticated to SaltStack Config