From Sonic Retro
Boost Mode refers, in its most general sense, to a high-speed dash state which various characters are able to perform, most notably Sonic the Hedgehog. The mode is generally accompanied by some sort of visual distortion to emphasise the eye-blistering velocities involved, and may also require the depletion of some sort of energy gague on the HUD.
Strictly speaking, only the Sonic Advance 2 iteration is actually called "Boost Mode"; in Sonic Rush, the manual calls it the Burst Dash; in Sonic '06 it is Mach Speed; and in Sonic Unleashed it is the Sonic Boost. See those pages for more details.
Sonic Advance 2
Available to all characters in Sonic Advance 2, Boost Mode is not an activated move as such; characters enter into this state simply by running really, really fast, while slowing down or being stopped by an obstacle cancels the effect. Boost Mode doesn't really have any specific effects beyond the fact that the character is, by the move's very nature, going at high speed; there are, however, cosmetic changes like a pursuing afterimage (a'la Hyper Sonic and Hyper Knuckles). Also, some of the character's moves change slightly, into high-speed forms which can be used without breaking stride.