Support vector machines: choose a kernel with validation

A kernel changes the relationships an SVM can represent. Choose it using a validation design that matches the task, with preprocessing fitted only on training data. A nonlinear kernel is not automatically better because it is more flexible.

Our original experiment uses 400 synthetic concentric-ring observations, generated with a fixed seed, ring factor 0.45 and noise 0.15. The included CSV fixes 240 training, 80 validation and 80 test rows. This is a controlled geometry example, not a real customer dataset.

Declare the candidate comparison

The two candidates use linear and radial-basis-function kernels, both with C=1. The RBF configuration uses gamma='scale'. Each candidate is wrapped in a pipeline that learns StandardScaler from training rows before fitting the SVM.

Validation balanced accuracy chooses the kernel; an exact tie would prefer the linear candidate. We do not search many hidden values and report only the most favorable one.

The SVC reference documents the kernel, C and gamma settings. The fixture and split seeds are recorded in the lab.

Report both candidates

RBF wins this declared comparison. Its test balanced accuracy is 0.9875 on the eighty held-out rows. The test result is higher than the validation result in this finite sample; neither number should be generalized to arbitrary datasets.

Support-vector counts describe the fitted solutions and can matter for prediction cost. A smaller count is not an independent proof of better generalization.

import numpy as np
from classical_cases import kernel_case

r = kernel_case()
assert r['split_counts']=={'train':240,'validation':80,'test':80}
scores={row['kernel']:row for row in r['candidates']}
assert scores['linear']['validation_balanced_accuracy']==.625
assert scores['rbf']['validation_balanced_accuracy']==.925
assert r['selected_kernel']=='rbf'
assert np.isclose(r['selected_test_balanced_accuracy'],.9875)
print(r)

Run from the supervised-model lab directory. The data generator and stratified split assignments are included, so reproduction does not depend on downloading a benchmark.

Explain why geometry matters here

A linear boundary cannot naturally separate an inner ring from an outer ring using only the two original coordinates. An RBF kernel can represent a nonlinear boundary that better matches this controlled structure.

That explanation comes from the known synthetic task. In another application, a linear model may be sufficient, more efficient or more stable. Feature engineering can also change which relationships are linear in the chosen representation.

C controls the penalty tradeoff, while gamma affects the RBF similarity scale. Expanding their search changes the selection procedure. Keep that search within development data and retain suitable final assessment evidence.

Do not call decision scores probabilities

The experiment uses class predictions and does not fit a probability-calibration procedure. An SVM decision-function value is not automatically a calibrated probability. If probabilities are required for a cost calculation, design and evaluate calibration separately.

Consider computational constraints as the dataset grows. Kernel methods can require substantial fitting and prediction resources; compare the actual task requirements and measured resource use rather than selecting by accuracy alone.

Exercise: declare a small C/gamma grid for the RBF candidate and include the linear baseline. Fit scaling within every training fold, record all validation results, and explain what additional evidence would be needed before replacing the current reference.

NeuraPath's Data Science course connects kernel mathematics with practical model selection. A strong result explains the geometry, selection rule and scope of the measured advantage.

Continue learning

This article is part of the Supervised learning methods sequence. Use the neighbouring tasks when you need the prerequisite or the next application.

• Review the prerequisite or neighbouring task in K-nearest neighbours: why scaling changes the prediction.
• Continue with Naive Bayes: test the independence assumption empirically.