Personally I don't collect coins just to collect them, I collect them because of the history. So I suggest that if you want a coin collection find an era that peaks a little interest and go from there. Read some of the postings here on Coin Talk about coin condition and cleaning which is not a good thing. I started collecting coins when I was 8 years old. I have one coin that is not worth too much but rich in history and wonderment. It has a hole through it, an 1861-O half dollar with a bisected date. Minted by the confederacy in New Orleans, a time in history when the Confederacy had control of New Orleans. Someone wore this around their neck, this peaks my interest even from a damaged coin. Coins for display I really have no interest in them.
A very specific coin that you can find slabbed and maintain some 'coolness' factor for around your budget would be something like the 1996 W Roosevelt dime or maybe a 2005 Bison Nickel with the detached leg (our modern day 3-legged buffalo, sort of). You could go for a nice toned bicentennial quarter in 40% silver. Same with the half and the Ike. I have purchased Indian head cents for around $30 in low mint state (MS-61 specifically). You just have to look around. Honestly though I would check out foreigns for cheap coins with lower mintages.