Implementation of the XGBoost algorithm in Python from scratch
Within the previous article we mentioned the XGBoost algorithm and confirmed its implementation in pseudocode. On this article we’re going to implement the algorithm in Python from scratch.
The supplied code is a concise and light-weight implementation of the XGBoost algorithm (with solely about 300 traces of code), supposed to exhibit its core performance. As such, it’s not optimized for velocity or reminiscence utilization, and doesn’t embody the complete spectrum of choices supplied by the XGBoost library (see https://xgboost.readthedocs.io/ for extra particulars on the options of the library). Extra particularly:
- The code is written in pure Python, whereas the core of the XGBoost library is written in C++ (its Python lessons are solely skinny wrappers over the C++ implementation).
- It doesn’t embody varied optimizations that permit XGBoost to take care of enormous quantities of information, resembling weighted quantile sketch, out-of-core tree studying, and parallel and distributed processing of the info. These optimizations can be mentioned in additional element within the subsequent article within the sequence.
- The present implementation helps solely regression and binary classification duties, whereas the XGBoost library additionally helps multi-class classification and rating issues.
- Our implementation helps solely a small subset of the hyperparameters that exist within the XGBoost library. Particularly, it helps the next hyperparameters:
- n_estimators (default = 100): the variety of regression timber within the ensemble (which can be the variety of boosting iterations).
- max_depth (default = 6): the utmost depth (variety of ranges) of every tree.
- learning_rate (default = 0.3): the step measurement shrinkage utilized to the timber.
- reg_lambda (default = 1): L2 regularization time period utilized to the weights of the leaves.
- gamma (default = 0): minimal loss discount required to separate a given node.
For consistency, I’ve stored the identical names and default values of those hyperparameters as they’re outlined within the XGBoost library.
