Use Python Logging like a Professional
I can wager that nearly each Python developer generally makes use of “print” for debugging. There’s nothing fallacious with that for prototyping, however for manufacturing, there are far more efficient methods to deal with the logs. On this article, I’ll present 5 sensible explanation why Python “logging” is far more versatile and highly effective and why you completely ought to use it in case you have not began earlier than.
Let’s get into it.
Code
To make issues extra sensible, let’s take into account a toy instance. I created a small software that calculates a linear regression for 2 Python lists:
import numpy as np
from sklearn.linear_model import LinearRegression
from typing import Listing, Non-obligatorydef do_regression(arr_x: Listing, arr_y: Listing) -> Non-obligatory[List]:
""" LinearRegression for X and Y lists """
strive:
x_in = np.array(arr_x).reshape(-1, 1)
y_in = np.array(arr_y).reshape(-1, 1)
print(f"X: {x_in}")
print(f"y: {y_in}")
reg = LinearRegression().match(x_in, y_in)
out = reg.predict(x_in)
print(f"Out: {out}")
print(f"Rating: {reg.rating(x_in, arr_y)}")
print(f"Coef: {reg.coef_}")
return out.reshape(-1).tolist()
besides ValueError as err:
print(f"ValueError: {err}")
return None
if __name__ == "__main__":
print("App began")
ret = do_regression([1,2,3,4], [5,6,7,8])
print(f"LinearRegression consequence: {ret}")
This code works, however can we do it higher? We clearly can. Let’s see 5 benefits of utilizing “logging” as a substitute of “print” on this code.
1. Logging ranges
Let’s change our code a bit:
import loggingdef do_regression(arr_x: Listing, arr_y: Listing) -> Non-obligatory[List]:
"""LinearRegression for X and Y Python lists"""
strive:
x_in = np.array(arr_x).reshape(-1, 1)
y_in = np.array(arr_y).reshape(-1, 1)
logging.debug(f"X: {x_in}")
...
besides ValueError as err:
logging.error(f"ValueError: {err}")
return None
if __name__ == "__main__":
logging.basicConfig(stage=logging.DEBUG, format='%(message)s')
logging.information("App began")
ret = do_regression([1,2,3,4], [5,6,7,8])
logging.information(f"LinearRegression consequence: {ret}")
Right here I changed “print” calls with “logging” calls. We made a small change, nevertheless it makes the output far more versatile. Utilizing the “stage” parameter, we are able to now set completely different logging ranges. For instance, if we use “stage=logging.DEBUG”, then all output will likely be seen. Once we are certain that our code is prepared for manufacturing, we are able to change the extent to “logging.INFO”, and debugging messages is not going to be displayed anymore:
And what’s essential is that no code change is required besides the initialization of the logging itself!
By the best way, all out there constants might be discovered within the logging/__init__.py file:
ERROR = 40
WARNING = 30
INFO = 20
DEBUG = 10
NOTSET = 0
As we are able to see, the “ERROR” stage is the best; by enabling the “ERROR” log stage, we are able to suppress all different messages, and solely errors will likely be displayed.
2. Formatting
As we are able to see from the final screenshot, it’s simple to manage the logging output. However we are able to do far more to enhance that. We will additionally alter the output by offering the “format” string. For instance, I can specify formatting like this:
logging.basicConfig(stage=logging.DEBUG,
format='[%(asctime)s] %(filename)s:%(lineno)d: %(message)s')
With out another code modifications, I can see timestamps, file names, and even the road numbers within the output:
There are about 20 completely different parameters out there, which might be discovered within the “LogRecord attributes” paragraph of the guide.
3. Saving logs to a file
Python logging is a really versatile module, and its performance might be simply expanded. Let’s say we wish to save all our logs right into a file for future evaluation. To do that, we have to add solely two traces of code:
logging.basicConfig(stage=logging.DEBUG,
format='[%(asctime)s] %(message)s',
handlers=[logging.FileHandler("debug.log"),
logging.StreamHandler()])
As we are able to see, I added a brand new parameter “handlers”. A StreamHandler is displaying the log on the console, and the FileHandler, as we are able to guess from its title, saves the identical output to the file.
This method is actually versatile. Loads of completely different “handler” objects can be found in Python, and I encourage readers to check the manual on their very own. And as we already know, logging works virtually routinely; no additional code modifications are required.
4. Rotating log information
Saving logs right into a file is an effective possibility, however alas, the disk area will not be limitless. We will simply clear up this drawback through the use of the rotating log file:
from logging.handlers import TimedRotatingFileHandler...
if __name__ == "__main__":
file_handler = TimedRotatingFileHandler(
filename="debug.log",
when="midnight",
interval=1,
backupCount=3,
)
logging.basicConfig(stage=logging.DEBUG,
format='[%(asctime)s] %(message)s',
handlers=[file_handler, logging.StreamHandler()])
All parameters are self-explanatory. A TimedRotatingFileHandler object will create a log file, which will likely be modified each midnight, and solely the final three log information will likely be saved. The earlier information will likely be routinely renamed to one thing like “debug.log.2023.03.03”, and after a 3-day interval, they are going to be deleted.
5. Sending logs through socket
Python’s logging is surprisingly versatile. If we don’t wish to save logs into an area file, we are able to simply add a socket handler, which can ship logs to a different service utilizing a particular IP and port:
from logging.handlers import SocketHandlerlogging.basicConfig(stage=logging.DEBUG, format='[%(asctime)s] %(message)s',
handlers=[SocketHandler(host="127.0.0.1", port=15001),
logging.StreamHandler()])
That’s it; no extra code modifications are required!
We will additionally create one other software that may take heed to the identical port:
import socket
import logging
import pickle
import struct
from logging import LogRecordport = 15001
stream_handler = logging.StreamHandler()
def create_socket() -> socket.socket:
"""Create the socket"""
sock = socket.socket(socket.AF_INET)
sock.settimeout(30.0)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
return sock
def read_socket_data(conn_in: socket.socket):
"""Learn knowledge from socket"""
whereas True:
knowledge = conn_in.recv(4) # Information: 4 bytes size + physique
if len(knowledge) > 0:
body_len = struct.unpack(">L", knowledge)[0]
knowledge = conn_in.recv(body_len)
document: LogRecord = logging.makeLogRecord(pickle.masses(knowledge))
stream_handler.emit(document)
else:
logging.debug("Socket connection misplaced")
return
if __name__ == "__main__":
logging.basicConfig(stage=logging.DEBUG, format='[%(asctime)s] %(message)s',
handlers=[stream_handler])
sock = create_socket()
sock.bind(("127.0.0.1", port)) # Native connections solely
sock.hear(1) # One shopper might be related
logging.debug("Logs listening thread began")
whereas True:
strive:
conn, _ = sock.settle for()
logging.debug("Socket connection established")
read_socket_data(conn)
besides socket.timeout:
logging.debug("Socket listening: no knowledge")
The difficult half right here is to make use of the emit methodology, which provides all distant knowledge obtained by a socket to an lively StreamHandler.
6. Bonus: Log filters
Lastly, a small bonus for readers who had been attentive sufficient to learn till this half. It’s also simple so as to add customized filters to logs. Let’s say we wish to log solely X and Y values into the file for future evaluation. It’s simple to create a brand new Filter class, which can save to log solely strings containing “x:” or “y:” data:
from logging import LogRecord, Filterclass DataFilter(Filter):
"""Filter for logging messages"""
def filter(self, document: LogRecord) -> bool:
"""Save solely filtered knowledge"""
return "x:" in document.msg.decrease() or "y:" in document.msg.decrease()
Then we are able to simply add this filter to the file log. Our console output will keep intact, however the file can have solely “x:” and “y:” values.
file_handler = logging.FileHandler("debug.log")
file_handler.addFilter(DataFilter())logging.basicConfig(stage=logging.DEBUG,
format='[%(asctime)s] %(message)s',
handlers=[file_handler, logging.StreamHandler()])
Conclusion
On this quick article, we discovered a number of simple methods to include logs into the Python software. Logging in Python is a really versatile framework, and it’s positively price spending a while investigating the way it works.
Thanks for studying, and good luck with future experiments.
For those who loved this story, be happy to subscribe to Medium, and you’ll get notifications when my new articles will likely be revealed, in addition to full entry to 1000’s of tales from different authors.
