The first defeat I can remember is a home loss to Torquay, I remember watching some of it from the elevated bit next to the chaddy whilst queuing for a pie, it was in 2005 and we lost 2-1 I think.
It can't be the first defeat I ever saw because I can remember an away game I went to in 2003 but I really can't remember any defeats earlier than that game in 2005.