I am an XB9SX owner (that is the City Cross). I think what people usually have a problem with on the Buell is that they lack the top end that the in line 4's have (as well as the jap twins).
As with any V-twin, they have gobs of low end torque, but not a lot of high end at all.
Add that to the history of HD being from the cruiser family, not the racing family, and there are many folks that don't like them.
From what I remember, buell used to have a problem with drive belts breaking. They have improved the technology greatly, and now the belts are MUCH better. They put up with quite a bit of abuse. Granted, I check my belt before I ride every time because I am still a little wary of it. There is an aftermarket conversion kit to switch to a chain, but I think it costs a good bit.
I for one, love my buell. I don't care much for going 140+. What I wanted in a bike was a short wheel base, and perfectly managed COG, the best setup for the twisties, which the Buell has in both areas. Also, I have always liked the "street fighter" look. Naked bike with a bit more upright posture. The XB was the right fit for me. I also didn't have a huge buget for a bike, so that ruled out my first choice which was a Ducati Monster.
The first mod that most folks do, is to add the HD race kit. That involves a new ECU, airbox, muffler. From what I have heard, that setup adds a LOT more power and usually runs about $1000 installed from the dealer.
The funny thing about owning a Buell, is that I get grief from both sides; The cruiser folks AND the jap sportbike folks. I have a thick skin, so it doesn't phase me. I have fun on my bike, and that is all that matters to me.
I am the type of person that likes all bikes, cruisers and rockets.
If you are interested in taking a ride on a Buell, I suggest going to a HD Dealer that carries them and take one for a test ride to see what you think.