From Sonic Retro
When Sonic destroys a badnik, and he is in the air, the engine checks to see if he should rebound or not.
If Sonic's Y position is greater than that of the badnik, or his Y speed is negative, he will not rebound. Instead, 1 times the sign of Y speed will be subtracted from Y speed (this is not limited to 0 - if Y speed were .25, it would not be set to 0, but to -.75). This slight attenuation of his speed helps give the impression that the badnik had some weight to it.
On the other hand, if Sonic's Y position is less than that of the badnik and his Y speed is positive, he will rebound. His Y speed will simply be multiplied by -1.
But can't you control the bounce by holding or not holding the jump button? Well yes, but that is actually part of the main jumping code.
If Sonic has jumped (and only if he has jumped) into the badnik, the variable jump height code is still in effect. As mentioned in the Jumping section of this guide - during a jump, at any point if the player isn't holding jump and his Y speed is less than -4, it will be set to -4. For this reason, if you are not holding jump when you collide with the enemy, and if your new Y speed is less than -4, it will be set to -4 because the jump code does that already. Effectively, allowing you to control the rebound height.
NOTE: X speed is totally unaffected by destroying a badnik, both in the air and when rolling on the ground.
Sonic 1 Method
When Sonic, spinning, collides with a monitor while in the air, the engine checks whether his Y speed is positive or not. If it is, he will rebound off of the monitor in the same way as he does badniks - only without bothering to compare his Y position to that of the monitor. Sonic will rebound upward every time, no matter what.
However, if his Y speed is negative upon collision with the monitor, it then compares Sonic's Y position with that of the monitor. If Sonic's Y position is less, then he is moved outside of the monitor to the side, as if it were solid, and it is not destroyed. If Sonic's Y position is greater, then Sonic bumps into the bottom of the monitor, bouncing it upward a little bit (if the monitor is in a tree, this knocks it down.)
Sonic & Knuckles Method
In Sonic & Knuckles, the monitor collision behaviour has been revised. Monitors are not considered solid when Sonic hits them while his Y speed is negative, and he can't bounce them up by hitting them from beneath. This means the Flame Shield Monitor in the tree in Angel Island Act 2 can't be knocked down when Sonic 3 is locked on to Sonic & Knuckles, even though it can be in Sonic 3 alone. Why this change was made is beyond me.
If his Y speed is positive when he hits one, he will rebound normally as he does in the other games. But if his Y speed is negative, his Y speed will be multiplied by -1. This means if he jumps into a monitor from underneath, he'll be bounced downward. Also, if he hits a monitor in its corner as he jumps over it, he'll be bounced back toward the ground.
NOTE: In all the games, X speed is totally unaffected by destroying a monitor, both in the air and when rolling on the ground.
When Sonic hits a boss while in the air, Y speed and X speed are both reversed (multiplied by -1). If he is rebounded upward by this, then the standard circumstances apply: releasing the Jump button while Y speed is less than -4 will set Y speed to -4, air drag is calculated, and so on.
When Sonic hits a boss while on the ground, his speeds are not affected at all. In your engine, you may wish to fix that and have X speed reverse.
Gravity is added to Y speed before the engine checks for and/or performs the rebounding routine.