til.duyet.net
  • 🤪Today I Learned
  • Data Engineering
    • ☁️AWS
      • Make an NVMe volume available for use on AWS EC2
      • AWS EMR
        • S3 Dist CP
        • Manage services
        • EMR - Tez
        • Issues
    • ⚒️Tools
    • 💻Shell
      • MacOS's Touch ID on Terminal
      • Using `sed` to find and replace in file
      • Merging contents of multiple .csv files into single .csv file
    • env from ConfigMap or Secrets
    • ☸️Kubernetes, Helm, Kustomize
      • initContainer to download file to pod
      • Kustomize: omission of resources
      • K8S: Services and Labels
      • K8S: PVC stuck in status “Terminating”
      • K8S: Port Forward
      • K8S: Pull an Image from a Private Registry
      • Happy helm
      • Helm: render manifest locally
      • Helm: Public Helm chart repository with GitHub Pages
    • 🔄Apache Airflow
      • Extend official Docker image
      • Generate offline SQL upgrade script
      • Airflow in Docker
      • Mastering Airflow UI
      • Best Practices for Airflow and ETLs
      • Airflow in Docker Compose
      • Useful SQL queries for Apache Airflow
    • 🐳Docker
      • "Distroless" Docker Images
      • Docker cleanup
      • Optimize the Docker Image Size
      • The best Docker base image for Python
  • Database
    • Google BigQuery
      • BigQuery Cancel Running Query
      • BigQuery - Split string and get the first part
      • BigQuery - UNNEST in SELECT
      • Bigquery - Sample queries for audiences based
      • BigQuery cookbook for Google Analytics Exported Data
    • Apache Hive
      • Hive - SHOW schemas/tables/create
    • AWS Redshift / Postgres
      • Amazon Redshift Utilities
      • Postgres - Index Summary
      • Postgres - List tables
      • Redshift - GRANT
      • Redshift - tables and their owners
      • Redshift - Check the table size
    • Presto
      • Aliyun Data Lake Analytics (Presto) - Add partition on non-existing location
      • AWS Athena - Add Partition
  • Programming
    • 🍪Rust
      • cheats.rs
      • Imperative vs Declarative
      • Generate Struct from JSON
    • 🐍Python
      • YAML config file with environment variables
      • date_range_generator
      • get_all_s3_keys
      • Pipenv
    • 👻Golang
      • Functions and Methods in Go?
      • Convert JSON to Go struct
    • 💎Javascript / Typescript
      • Intl.ListFormat
    • ✨FE / React
      • Beautiful icons, images, ..
      • Flexbox
      • Create hook to inject JS script
  • Unix
    • Git - Pretty git branch graphs
    • Checking files in Docker build context
    • Bash get the directory of the current script
    • Vim
    • Find and replace
  • Miscellaneous
    • [Fig] Single machine and distributed system structure
    • Deploying Machine Learning Models at Scale
    • Bypass a Chrome certificate/HSTS error
    • Articles
Powered by GitBook
On this page

Was this helpful?

Edit on Git
  1. Programming
  2. Python

YAML config file with environment variables

import os
import re
import yaml


def parse_config(path=None, data=None, tag='!ENV'):
    """
    Load a yaml configuration file and resolve any environment variables
    The environment variables must have !ENV before them and be in this format
    to be parsed: ${VAR_NAME}.
    E.g.:
    database:
        host: !ENV ${HOST}
        port: !ENV ${PORT}
    app:
        log_path: !ENV '/var/${LOG_PATH}'
        something_else: !ENV '${AWESOME_ENV_VAR}/var/${A_SECOND_AWESOME_VAR}'
    :param str path: the path to the yaml file
    :param str data: the yaml data itself as a stream
    :param str tag: the tag to look for
    :return: the dict configuration
    :rtype: dict[str, T]
    """
    # pattern for global vars: look for ${word}
    pattern = re.compile('.*?\${(\w+)}.*?')
    loader = yaml.SafeLoader

    # the tag will be used to mark where to start searching for the pattern
    # e.g. somekey: !ENV somestring${MYENVVAR}blah blah blah
    loader.add_implicit_resolver(tag, pattern, None)

    def constructor_env_variables(loader, node):
        """
        Extracts the environment variable from the node's value
        :param yaml.Loader loader: the yaml loader
        :param node: the current node in the yaml
        :return: the parsed string that contains the value of the environment
        variable
        """
        value = loader.construct_scalar(node)
        match = pattern.findall(value)  # to find all env variables in line
        if match:
            full_value = value
            for g in match:
                full_value = full_value.replace(
                    f'${{{g}}}', os.environ.get(g, g)
                )
            return full_value
        return value

    loader.add_constructor(tag, constructor_env_variables)

    if path:
        with open(path) as conf_data:
            return yaml.load(conf_data, Loader=loader)
    elif data:
        return yaml.load(data, Loader=loader)
    else:
        raise ValueError('Either a path or data should be defined as input')

Example of a YAML configuration with environment variables:

config.yaml
database:
 name: database_name
 user: !ENV ${DB_USER}
 password: !ENV ${DB_PASS}
 host: !ENV ${DB_HOST}
 port: 5432ws:
 user: !ENV ${WS_USER}
 password: !ENV ${WS_PASS}
 host: !ENV ‘https://${CURR_ENV}.ws.com.local'

Usage

CONFIG = parse_config(path='config.yaml')
content = """
 database:
  name: database_name
  user: !ENV ${DB_USER}
  password: !ENV ${DB_PASS}
"""
CONFIG = parse_config(data=content)
PreviousPythonNextdate_range_generator

Last updated 4 years ago

Was this helpful?

🐍