Gradient Boosting from Concept to Follow (Half 2) | by Dr. Roi Yehoshua | Jul, 2023

Within the first part of this text, we offered the gradient boosting algorithm and confirmed its implementation in pseudocode.

On this a part of the article, we are going to discover the courses in Scikit-Be taught that implement this algorithm, focus on their varied parameters, and exhibit learn how to use them to resolve a number of classification and regression issues.

Though the XGBoost library (which will likely be lined in a future article) supplies a extra optimized and extremely scalable implementation of gradient boosting, for small to medium-sized information units it’s usually simpler to make use of the gradient boosting courses in Scikit-Be taught, which have a less complicated interface and a considerably fewer variety of hyperparameters to tune.

Scikit-Be taught supplies the next courses that implement the gradient-boosted choice bushes (GBDT) mannequin:

1. GradientBoostingClassifier is used for classification issues.
2. GradientBoostingRegressor is used for regression issues.

Along with the usual parameters of decision trees, resembling criterion, max_depth (set by default to three) and min_samples_split, these courses present the next parameters:

1. loss — the loss perform to be optimized. In GradientBoostingClassifier, this perform could be ‘log_loss’ (the default) or ‘exponential’ (which can make gradient boosting behave like AdaBoost). In GradientBoostingRegressor, this perform could be ‘squared_loss’ (the default), ‘absolute_loss’, ‘huber’, or ‘quantile’ (see this article for the variations between these loss capabilities).
2. n_estimators — the variety of boosting iterations (defaults to 100).
3. learning_rate — an element that shrinks the contribution of every tree (defaults to 0.1).
4. subsample — the fraction of samples to make use of for coaching every tree (defaults to 1.0).
5. max_features — the variety of options to think about when looking for one of the best cut up in every…