pipeline.extern.logutils package

Submodules

pipeline.extern.logutils.adapter module

class pipeline.extern.logutils.adapter.LoggerAdapter(logger, extra)

Bases: object

An adapter for loggers which makes it easier to specify contextual information in logging output.

critical(msg, *args, **kwargs)

Delegate a critical call to the underlying logger.

debug(msg, *args, **kwargs)

Delegate a debug call to the underlying logger.

error(msg, *args, **kwargs)

Delegate an error call to the underlying logger.

exception(msg, *args, **kwargs)

Delegate an exception call to the underlying logger.

getEffectiveLevel()

Get the effective level for the underlying logger.

hasHandlers()

See if the underlying logger has any handlers.

info(msg, *args, **kwargs)

Delegate an info call to the underlying logger.

isEnabledFor(level)

Is this logger enabled for level ‘level’?

log(level, msg, *args, **kwargs)

Delegate a log call to the underlying logger, after adding contextual information from this adapter instance.

process(msg, kwargs)

Process the logging message and keyword arguments passed in to a logging call to insert contextual information. You can either manipulate the message itself, the keyword args or both. Return the message and kwargs modified (or not) to suit your needs.

Normally, you’ll only need to override this one method in a LoggerAdapter subclass for your specific needs.

setLevel(level)

Set the specified level on the underlying logger.

warn(msg, *args, **kwargs)

Delegate a warning call to the underlying logger.

warning(msg, *args, **kwargs)

Delegate a warning call to the underlying logger.

pipeline.extern.logutils.colorize module

class pipeline.extern.logutils.colorize.ColorizingStreamHandler(stream=None)

Bases: logging.StreamHandler

A stream handler which supports colorizing of console streams under Windows, Linux and Mac OS X.

Parameters

strm – The stream to colorize - typically sys.stdout or sys.stderr.

color_map = {'black': 0, 'blue': 4, 'cyan': 6, 'green': 2, 'magenta': 5, 'red': 1, 'white': 7, 'yellow': 3}

colorize(message, record)

Colorize a message for a logging event.

This implementation uses the level_map class attribute to map the LogRecord’s level to a colour/intensity setting, which is then applied to the whole message.

Parameters

• message – The message to colorize.

• record – The LogRecord for the message.

csi = '\x1b['

emit(record)

Emit a record.

If a formatter is specified, it is used to format the record. The record is then written to the stream with a trailing newline. If exception information is present, it is formatted using traceback.print_exception and appended to the stream. If the stream has an ‘encoding’ attribute, it is used to determine how to do the output to the stream.

format(record)

Formats a record for output.

This implementation colorizes the message line, but leaves any traceback uncolorized.

property is_tty

Returns true if the handler’s stream is a terminal.

level_map = {5: (None, 'blue', False), 10: (None, 'blue', False), 13: ('black', 'yellow', True), 20: (None, None, False), 30: (None, 'yellow', False), 40: (None, 'red', False), 50: ('red', 'white', True)}

Maps levels to colour/intensity settings.

output_colorized(message)

Output a colorized message.

On Linux and Mac OS X, this method just writes the already-colorized message to the stream, since on these platforms console streams accept ANSI escape sequences for colorization. On Windows, this handler implements a subset of ANSI escape sequence handling by parsing the message, extracting the sequences and making Win32 API calls to colorize the output.

Parameters

message – The message to colorize and output.

reset = '\x1b[0m'

pipeline.extern.logutils.dictconfig module

class pipeline.extern.logutils.dictconfig.BaseConfigurator(config)

Bases: object

The configurator base class which defines some useful defaults.

CONVERT_PATTERN = re.compile('^(?P<prefix>[a-z]+)://(?P<suffix>.*)$')

DIGIT_PATTERN = re.compile('^\\d+$')

DOT_PATTERN = re.compile('^\\.\\s*(\\w+)\\s*')

INDEX_PATTERN = re.compile('^\\[\\s*(\\w+)\\s*\\]\\s*')

WORD_PATTERN = re.compile('^\\s*(\\w+)\\s*')

as_tuple(value)

Utility function which converts lists to tuples.

cfg_convert(value)

Default converter for the cfg:// protocol.

configure_custom(config)

Configure an object with a user-supplied factory.

convert(value)

Convert values to an appropriate type. dicts, lists and tuples are replaced by their converting alternatives. Strings are checked to see if they have a conversion format and are converted if they do.

ext_convert(value)

Default converter for the ext:// protocol.

importer()

Allows the importer to be redefined.

resolve(s)

Resolve strings to objects using standard import and attribute syntax.

value_converters = {'cfg': 'cfg_convert', 'ext': 'ext_convert'}

class pipeline.extern.logutils.dictconfig.ConvertingDict

Bases: dict

A converting dictionary wrapper.

get(k[, d]) → D[k] if k in D, else d. d defaults to None.

pop(k[, d]) → v, remove specified key and return the corresponding value.

If key is not found, d is returned if given, otherwise KeyError is raised

class pipeline.extern.logutils.dictconfig.ConvertingList

Bases: list

A converting list wrapper.

pop([index]) → item – remove and return item at index (default last).

Raises IndexError if list is empty or index is out of range.

class pipeline.extern.logutils.dictconfig.ConvertingTuple

Bases: tuple

A converting tuple wrapper.

class pipeline.extern.logutils.dictconfig.DictConfigurator(config)

Bases: pipeline.extern.logutils.dictconfig.BaseConfigurator

Configure logging using a dictionary-like object to describe the configuration.

add_filters(filterer, filters)

Add filters to a filterer from a list of names.

add_handlers(logger, handlers)

Add handlers to a logger from a list of names.

common_logger_config(logger, config, incremental=False)

Perform configuration which is common to root and non-root loggers.

configure()

Do the configuration.

configure_filter(config)

Configure a filter from a dictionary.

configure_formatter(config)

Configure a formatter from a dictionary.

configure_handler(config)

Configure a handler from a dictionary.

configure_logger(name, config, incremental=False)

Configure a non-root logger from a dictionary.

configure_root(config, incremental=False)

Configure a root logger from a dictionary.

pipeline.extern.logutils.dictconfig.dictConfig(config)

Configure logging using a dictionary.

pipeline.extern.logutils.dictconfig.dictConfigClass

alias of pipeline.extern.logutils.dictconfig.DictConfigurator

pipeline.extern.logutils.dictconfig.named_handlers_supported()

pipeline.extern.logutils.dictconfig.valid_ident(s)

pipeline.extern.logutils.http module

class pipeline.extern.logutils.http.HTTPHandler(host, url, method='GET', secure=False, credentials=None)

Bases: logging.Handler

A class which sends records to a Web server, using either GET or POST semantics.

Parameters

• host – The Web server to connect to.

• url – The URL to use for the connection.

• method – The HTTP method to use. GET and POST are supported.

• secure – set to True if HTTPS is to be used.

• credentials – Set to a username/password tuple if desired. If set, a Basic authentication header is sent. WARNING: if using credentials, make sure secure is True to avoid sending usernames and passwords in cleartext over the wire.

emit(record)

Emit a record.

Send the record to the Web server as a percent-encoded dictionary

Parameters

record – The record to be emitted.

mapLogRecord(record)

Default implementation of mapping the log record into a dict that is sent as the CGI data. Overwrite in your class. Contributed by Franz Glasner.

Parameters

record – The record to be mapped.

pipeline.extern.logutils.queue module

This module contains classes which help you work with queues. A typical application is when you want to log from performance-critical threads, but where the handlers you want to use are slow (for example, SMTPHandler). In that case, you can create a queue, pass it to a QueueHandler instance and use that instance with your loggers. Elsewhere, you can instantiate a QueueListener with the same queue and some slow handlers, and call start() on it. This will start monitoring the queue on a separate thread and call all the configured handlers on that thread, so that your logging thread is not held up by the slow handlers.

Note that as well as in-process queues, you can use these classes with queues from the multiprocessing module.

N.B. This is part of the standard library since Python 3.2, so the version here is for use with earlier Python versions.

class pipeline.extern.logutils.queue.QueueHandler(queue)

Bases: logging.Handler

This handler sends events to a queue. Typically, it would be used together with a multiprocessing Queue to centralise logging to file in one process (in a multi-process application), so as to avoid file write contention between processes.

Parameters

queue – The queue to send LogRecords to.

emit(record)

Emit a record.

Writes the LogRecord to the queue, preparing it for pickling first.

Parameters

record – The record to emit.

enqueue(record)

Enqueue a record.

The base implementation uses put_nowait(). You may want to override this method if you want to use blocking, timeouts or custom queue implementations.

Parameters

record – The record to enqueue.

prepare(record)

Prepares a record for queuing. The object returned by this method is enqueued.

The base implementation formats the record to merge the message and arguments, and removes unpickleable items from the record in-place.

You might want to override this method if you want to convert the record to a dict or JSON string, or send a modified copy of the record while leaving the original intact.

Parameters

record – The record to prepare.

class pipeline.extern.logutils.queue.QueueListener(queue, *handlers)

Bases: object

This class implements an internal threaded listener which watches for LogRecords being added to a queue, removes them and passes them to a list of handlers for processing.

Parameters

• record – The queue to listen to.

• handlers – The handlers to invoke on everything received from the queue.

dequeue(block)

Dequeue a record and return it, optionally blocking.

The base implementation uses get(). You may want to override this method if you want to use timeouts or work with custom queue implementations.

Parameters

block – Whether to block if the queue is empty. If False and the queue is empty, an Empty exception will be thrown.

enqueue_sentinel()

Writes a sentinel to the queue to tell the listener to quit. This implementation uses put_nowait(). You may want to override this method if you want to use timeouts or work with custom queue implementations.

handle(record)

Handle a record.

This just loops through the handlers offering them the record to handle.

Parameters

record – The record to handle.

prepare(record)

Prepare a record for handling.

This method just returns the passed-in record. You may want to override this method if you need to do any custom marshalling or manipulation of the record before passing it to the handlers.

Parameters

record – The record to prepare.

start()

Start the listener.

This starts up a background thread to monitor the queue for LogRecords to process.

stop()

Stop the listener.

This asks the thread to terminate, and then waits for it to do so. Note that if you don’t call this before your application exits, there may be some records still left on the queue, which won’t be processed.

pipeline.extern.logutils.redis module

pipeline.extern.logutils.testing module

class pipeline.extern.logutils.testing.Matcher

Bases: object

This utility class matches a stored dictionary of logging.LogRecord attributes with keyword arguments passed to its matches() method.

match_value(k, dv, v)

Try to match a single stored value (dv) with a supplied value (v).

Return True if found, else False.

Parameters

• k – The key value (LogRecord attribute name).

• dv – The stored value to match against.

• v – The value to compare with the stored value.

matches(d, **kwargs)

Try to match a single dict with the supplied arguments.

Keys whose values are strings and which are in self._partial_matches will be checked for partial (i.e. substring) matches. You can extend this scheme to (for example) do regular expression matching, etc.

Return True if found, else False.

Parameters

kwargs – A set of keyword arguments whose names are LogRecord attributes and whose values are what you want to match in a stored LogRecord.

class pipeline.extern.logutils.testing.TestHandler(matcher)

Bases: logging.handlers.BufferingHandler

This handler collects records in a buffer for later inspection by your unit test code.

Parameters

matcher – The Matcher instance to use for matching.

property count

The number of records in the buffer.

emit(record)

Saves the __dict__ of the record in the buffer attribute, and the formatted records in the formatted attribute.

Parameters

record – The record to emit.

flush()

Clears out the buffer and formatted attributes.

matchall(kwarglist)

Accept a list of keyword argument values and ensure that the handler’s buffer of stored records matches the list one-for-one.

Return True if exactly matched, else False.

Parameters

kwarglist – A list of keyword-argument dictionaries, each of which will be passed to matches() with the corresponding record from the buffer.

matches(**kwargs)

Look for a saved dict whose keys/values match the supplied arguments.

Return True if found, else False.

Parameters

kwargs – A set of keyword arguments whose names are LogRecord attributes and whose values are what you want to match in a stored LogRecord.

shouldFlush()

Should the buffer be flushed?

This returns False - you’ll need to flush manually, usually after your unit test code checks the buffer contents against your expectations.

Module contents

The logutils package provides a set of handlers for the Python standard library’s logging package.

Some of these handlers are out-of-scope for the standard library, and so they are packaged here. Others are updated versions which have appeared in recent Python releases, but are usable with older versions of Python, and so are packaged here.

class pipeline.extern.logutils.BraceMessage(fmt, *args, **kwargs)

Bases: object

class pipeline.extern.logutils.DollarMessage(fmt, **kwargs)

Bases: object

class pipeline.extern.logutils.Formatter(fmt=None, datefmt=None, style='%')

Bases: logging.Formatter

Subclasses Formatter in Pythons earlier than 3.2 in order to give 3.2 Formatter behaviour with respect to allowing %-, {} or $- formatting.

format(record)

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

formatMessage(record)

usesTime()

Check if the format uses the creation time of the record.

class pipeline.extern.logutils.NullHandler(level=0)

Bases: logging.Handler

This handler does nothing. It’s intended to be used to avoid the “No handlers could be found for logger XXX” one-off warning. This is important for library code, which may contain code to log events. If a user of the library does not configure logging, the one-off warning might be produced; to avoid this, the library developer simply needs to instantiate a NullHandler and add it to the top-level logger of the library module or package.

createLock()

Since this handler does nothing, it has no underlying I/O to protect against multi-threaded access, so this method returns None.

emit(record)

Emit a record. This does nothing and shouldn’t be called during normal processing, unless you redefine handle().

handle(record)

Handle a record. Does nothing in this class, but in other handlers it typically filters and then emits the record in a thread-safe way.

class pipeline.extern.logutils.PercentStyle(fmt)

Bases: object

asctime_format = '%(asctime)s'

default_format = '%(message)s'

format(record)

usesTime()

class pipeline.extern.logutils.StrFormatStyle(fmt)

Bases: pipeline.extern.logutils.PercentStyle

asctime_format = '{asctime}'

default_format = '{message}'

format(record)

class pipeline.extern.logutils.StringTemplateStyle(fmt)

Bases: pipeline.extern.logutils.PercentStyle

asctime_format = '${asctime}'

default_format = '${message}'

format(record)

usesTime()

pipeline.extern.logutils.hasHandlers(logger)

See if a logger has any handlers.