When is it to windy to ride?

For your abilities or just general comfort level, when do you think it is to windy to ride? Seeing as how this is my first full summer of riding, I find that almost the slightest breeze has me grabbing the grips a bit tighter on the road. I'm getting a bit more comfortable as I fine tune my balance, but when I see the flags snapping in the breeze out the window, I get a bit worried about my ride home.