AVR Reset Problem

I am using the AT90S2313 in a pulse generating project but I am having
trouble getting the AVR to come out of reset.

When I reset, or initially apply power, the AVR will not proceed to
execute until I ground Pin 4, which is one of the crystal pins.

I am using a 10Mhz crystal, with the standard 22 pF caps to ground.

I was having the same problem when I was using breadboard so
I made a small board for the AVR and soldered the crystal and caps 
directly to the avr.

Am I being naive and this is this normal operation?

Could it still be noise?

Sincerely,
James