On the sound side, in addition to Canadian Hole (officially called Haulover), there's also a calm, sandy beach between Frisco and Hatteras Village.
On the ocean side, the waves are generally calmer south of Cape Point. But as Basskat pointed out, tide as well as wind speed and direction rule. Low tide has shallower water and less wave action than high tide. Persistent, strong east winds generate offshore swells that lead to big waves. Higher west winds exaggerate breakers by lifting them up.
The shape of the beach makes a big difference as well. Gently sloping beaches, such as those in Frisco, have shallower water and calmer waves than steeper beaches, such as those in Avon. Beach shape can also vary quite a bit within a short distance. If the waves are too big right where you are, climb up on the dunes and look for less waves. It could be that 200 yards from you there's a spot more to your liking.