Skip to main content
Version: 0.16.10

How to configure credentials

In progress

The Great Expectations CLI is no longer the preferred method for implementing and configuring Great Expectations. This topic will be updated soon to reflect this change. For more information, see A fond farewell to the CLI.

This guide will explain how to configure your great_expectations.yml project config to populate credentials from either a YAML file or a secret manager.

If your Great Expectations deployment is in an environment without a file system, refer to How to instantiate an Ephemeral Data Context.

Prerequisites​

Steps​

1. Save credentials and config​

Decide where you would like to save the desired credentials or config values - in a YAML file, environment variables, or a combination - then save the values.

In most cases, we suggest using a config variables YAML file. YAML files make variables more visible, easily editable, and allow for modularization (e.g. one file for dev, another for prod).

note
  • In the great_expectations.yml config file, environment variables take precedence over variables defined in a config variables YAML
  • Environment variable substitution is supported in both the great_expectations.yml and config variables config_variables.yml config file.

If using a YAML file, save desired credentials or config values to great_expectations/uncommitted/config_variables.yml or another YAML file of your choosing:

my_postgres_db_yaml_creds:
  drivername: postgresql
  host: localhost
  port: 5432
  username: postgres
  password: ${MY_DB_PW}
  database: postgres
note
  • If you wish to store values that include the dollar sign character $, please escape them using a backslash \ so substitution is not attempted. For example in the above example for Postgres credentials you could set password: pa\$sword if your password is pa$sword. Say that 5 times fast, and also please choose a more secure password!
  • When you save values via the CLICommand Line Interface, they are automatically escaped if they contain the $ character.
  • You can also have multiple substitutions for the same item, e.g. database_string: ${USER}:${PASSWORD}@${HOST}:${PORT}/${DATABASE}

If using environment variables, set values by entering export ENV_VAR_NAME=env_var_value in the terminal or adding the commands to your ~/.bashrc file:

export POSTGRES_DRIVERNAME=postgresql
export POSTGRES_HOST=localhost
export POSTGRES_PORT=5432
export POSTGRES_USERNAME=postgres
export POSTGRES_PW=
export POSTGRES_DB=postgres
export MY_DB_PW=password

2. Set config_variables_file_path​

If using a YAML file, set the config_variables_file_path key in your great_expectations.yml or leave the default.

config_variables_file_path: uncommitted/config_variables.yml

3. Replace credentials with placeholders​

Replace credentials or other values in your great_expectations.yml with ${}-wrapped variable names (i.e. ${ENVIRONMENT_VARIABLE} or ${YAML_KEY}).

datasources:
  my_postgres_db:
    class_name: Datasource
    module_name: great_expectations.datasource
    execution_engine:
      module_name: great_expectations.execution_engine
      class_name: SqlAlchemyExecutionEngine
      credentials: ${my_postgres_db_yaml_creds}
    data_connectors:
      default_inferred_data_connector_name:
        class_name: InferredAssetSqlDataConnector
  my_other_postgres_db:
    class_name: Datasource
    module_name: great_expectations.datasource
    execution_engine:
      module_name: great_expectations.execution_engine
      class_name: SqlAlchemyExecutionEngine
      credentials:
        drivername: ${POSTGRES_DRIVERNAME}
        host: ${POSTGRES_HOST}
        port: ${POSTGRES_PORT}
        username: ${POSTGRES_USERNAME}
        password: ${POSTGRES_PW}
        database: ${POSTGRES_DB}
    data_connectors:
      default_inferred_data_connector_name:
        class_name: InferredAssetSqlDataConnector

Additional Notes​

  • The default config_variables.yml file located at great_expectations/uncommitted/config_variables.yml applies to deployments created using great_expectations init.
  • To view the full script used in this page, see it on GitHub: how_to_configure_credentials.py