Yeah your completely right, 4 or 5 look really bad. But at least the ROM speed will speed up dramatically. Unless your playing an action game, 4 or 5 are normally playable, of course 1 or 2 are the most preferable speeds, and if your computer is good enough 0.
Oh yeah, good idea, VisualBoyAdvanceCE is a great third party version of the emulator.