Detrimental combustion instability is unwanted in gas turbines, aeroengines, rocket motors, and many other combustion systems. In this work, we design and implement a sliding mode controller (SMC) to mitigate self-sustained combustion oscillations in an open-ended thermoacoustic system. An acoustically compact heat source is confined and modeled by using a modified form of King's Law. Coupling the heat source model with a Galerkin series expansion of flow disturbances provides a platform to conduct pseudospectra analysis to gain insight on the system stability behaviors, and to evaluate the performance of the SMC. Two thermoacoustic systems with monopole-like actuators implemented are considered. One is associated with 1 mode and the other is with four modes. Both systems are shown to be controllable. Furthermore, it is found that self-sustained limit cycle oscillations can be successfully generated in both systems, when the actuators are not actuated. In order to gain insight on the thermoacoustic mode selection and triggering, the acoustical energy exchange between neighboring eigenmodes are studied and discussed. As the controller-driven actuators are actuated, the nonlinear limit cycle oscillations are quickly dampened. And both thermoacoustic systems are stabilized by reducing the sound pressure level by approximately 40 dB. Comparison is then made between the performance of the SMC and that of the classical LQR (linear-quadratic-regulator) one. The successful demonstration indicates that the SMC can be applied to stabilize unstable thermoacoustic systems, even with multiple unstable modes.