Python: Use logging module with configparser or argparser

Question

What is the best way to use Python's logging module to log everything that your script is doing when also utilizing the configparser file to load a config file which contains the location of where you'd like your log to be saved.

Here is my example code:

import sys
import os
import logging
import configparser
import argparse

### Create Functions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
def get_logger(LOG_DIR, FULL_LOG_PATH):
    """Create logger."""
    # Create LOG_DIR if it doesn't exist already

    try:
        os.makedirs(f"{LOG_DIR}")

    except:
        pass

    try:
        # Create logger and set level
        logger = logging.getLogger(__name__)
        logger.setLevel(level=logging.INFO)

        # Configure file handler
        formatter = logging.Formatter(
            fmt = '%(asctime)s - %(name)s - %(levelname)s - %(message)s',
            datefmt = "%Y-%m-%d_%H-%M-%S")
        fh = logging.FileHandler(f"{FULL_LOG_PATH}")
        fh.setFormatter(formatter)
        fh.setLevel(level=logging.INFO)

        # Add handlers to logger
        logger.addHandler(fh)
        return  logger

    except:
        sys.exit(-1)

def parse_cl_args():
    """Set CLI Arguments."""

    try:
        # Initiate the parser
        parser = argparse.ArgumentParser(
            description="Script to scrape Twitter users account information."
        )
        # Add optional arguments
        parser.add_argument(
            "-c", "--config-file",
            metavar='Config-file',
            help="Full path to the global config file containing paths/file names for script.",
            required=True
        )

        # Read parsed arguments from the command line into "args"
        args = parser.parse_args()

        # Assign the file name to a variable and return it
        config_file_path = args.config_file
        return config_file_path

    except:

        sys.exit(-1)

def parse_config_file(config_file_path):

    try:
        config = configparser.ConfigParser()
        config.read(config_file_path)
        return config
    except:
        sys.exit(-1)

# A bunch of other functions

if __name__ == '__main__':
    # parse command line args
    config_file_path = parse_cl_args()

    # parse config file
    config = parse_config_file(config_file_path)

    # Set logging path
    LOG_DIR = os.path.join(config["PATHS"]["LOG_DIR"])

    # Set log file name
    FULL_LOG_PATH = os.path.join(config["PATHS"]["LOG_DIR"], "mylog.log")

    # Get logger
    logger = get_logger(
        LOG_DIR = LOG_DIR, 
        FULL_LOG_PATH= FULL_LOG_PATH
        )

Everything above the get_logger() line can't be recorded in the logger but the logger can't be created without first loading my commandline argument (config_file.ini) and then parsing that file(which contains the location of where I'd like my log to be saved). Is there a better way to do this?

question from:https://stackoverflow.com/questions/66054843/python-use-logging-module-with-configparser-or-argparser

Answer 1

If you want to record logs before you know the location of the log-file but want those logs in the file too you can use a MemoryHandler, which is a special type of BufferingHandler. So the flow of your program would be:

1. Set up a logger
2. add MemoryHandler to this logger
3. do stuff like reading config files while using the logger you have to create logs
4. Set up FileHandler with value from config
5. Call setTarget(file_handler) on the MemoryHandler passing it the FileHandler
6. Call flush() on the MemoryHandler -> logs from step 3 are written to file
7. Optionally you can now remove the MemoryHandler