Monitoring natural ecosystems and ecosystem transitions is crucial for a better understanding of land change processes. By providing synoptic views in space and time, remote sensing data have proven to be valuable sources for such purposes. With the forthcoming Environmental Mapping and Analysis Program (EnMAP), frequent and area-wide mapping of natural environments by means of high quality hyperspectral data becomes possible. However, the amplified spectral mixing due to the sensor’s ground sampling distance of 30 m on the one hand and the patterns of natural landscapes in the form of gradual transitions between different land cover types on the other require special attention. Based on simulated EnMAP data, this study focuses on mapping shrub vegetation along a landscape gradient of shrub encroachment in a semi-arid, natural environment in Portugal. We demonstrate how probability outputs from a support vector classification (SVC) model can be used to extend a hard classification by information on shrub cover fractions. This results in a more realistic representation of gradual transitions in shrub vegetation maps. We suggest a new, adapted approach for SVC parameter selection: During the grid search, parameter pairs are evaluated with regard to the prediction of synthetically mixed test data, representing shrub to non-shrub transitions, instead of the hard classification of original, discrete test data. Validation with an unbiased, equalized random sampling shows that the resulting shrub-class probabilities from adapted SVC more accurately represent shrub cover fractions (mean absolute error/root mean squared error of 16.3%/23.2%) compared to standard SVC (17.1%/29.5%). Simultaneously, the discrete classification output was considerably improved by incorporating synthetic mixtures into parameter selection (averaged F1 accuracies increased from 72.4% to 81.3%). Based on our findings, the integration of synthetic mixtures into SVC parameterization allows the use of SVC for sub-pixel cover fraction estimation and, this way, can be recommended for deriving improved qualitative and quantitative descriptions of gradual transitions in shrub vegetation. The approach is therefore of high relevance for mapping natural ecosystems from future EnMAP data.