Cut a piece of cardboard to fit into the hole, trace the pattern on to a pece of MDF (1/2" or 3/4"). Cut out with a jigsaw, then cut a circle in the panel for the speaker.
Shouldn't take more than an hour. Plus, you can fit 6 1/2" speakers if you want. Maybe 7".