Going back to those finer details, do find a list of stores with "good" XS sizes below (and if you forgot, everything is from the perspective of a 5'4" dude with a 37.5"-ish chest and 30"-ish waist):
JCrew
I love this store except they don't have any physical stores in my area. However, their paper catalogs are amazing and their online shop is great. Plus they stock XS sizes for the majority of their own items! The third-party collections they sell are usually the outliers. Prices are nowhere cheap averaging $50+ for a dress shirt and $35+ for t-shirts. Sales on regular items are also few and far between - you will find a lot of 20-30% off sales in their clearance section though. And if you're interested, they have a "factory" store, which I have never tried. Did I mention their clothes are usually labelless?
So, what about fit? Their XS regular shirts are generally a bit tight through the body and they run a bit long - this is perfect if you don't plan on wearing your shit untucked. By tight, I mean that when you do the "hug yourself" pose, you can feel the material in the back stretching and tight against your back. The waist is also a bit tight, but that never bothers me.
Depending on the exact line of shirts though, some are looser and feel great, while the length issue persists. You would probably be happy if you're pretty thin and don't have too much bulk in the upper and lower torso. Sweaters share the same issues. The only outerwear item I've purchased is the, ahem, $850 leather moto jacket (I don't know why...). It fits like a very tight glove and is ever so slightly long - I tell people it fits a bit like O.J.'s glove (I'm going to hell...).
Why don't I size up? I've tried. The jump to the S size is huge and you can fit two of me in a dress shirt. Maybe the slim S size would work, but the returns process is a cross-border pain in the ass.
Overall, JCrew's XS size is the slimmest of them all, and while a tad tight, they look great and still feel okay.
Gap / Banana Republic
Next up are the Gap and Banana Republic. If you didn't know, the Gap Corporation owns both brands, Old Navy, and a bunch of other stores. You probably figured that out when you went to their website.
Honestly, the majority of my clothes are from the Gap and Banana because they are comfortable and usually labelless. You'll get a lot of stuff referring to San Franciso for Gap attire though. If you aren't aware, the Gap is the "kiddy", "lower grade" line relative to Banana Republic; Old Navy is el cheapo and only stocks down to the S size. Price-wise, Gap is fairly cheap with shirts in the $50 range and t-shirts in the $25 zone. Banana Republic is more or less double that. However, both the Gap and Banana Republic have 25% and up sales practically every other week on regular items. Style-wise, the Gap is forward and more "kiddy" oriented, while Banana can be forward, but stylish and formal - you'll know it when you see it.
Okay, onto sizing. Their XS for the Gap is a bit all over the road, but generally looser than JCrew and slightly shorter. I have a lot of t-shirts and dress shirts that end at or near the waist, while giving a fitted look. However, I remember a phase a few years back when their XS size became bloated, and a lot of their shirts do run long usually. The good looking untailored, untuckable dress shirt has to be the holy grail for shorties, and I may have found one or two through the years at the Gap.
Banana Republic does not think Canadians are worthy of the XS size. If you're in the United States, then lucky you. All I've had to work with is the S size, but I must admit, it works. Dress shirts and outerwear actually fits pretty good. They will run a bit long though, so stick to stuff that is meant to be long like peacoats and trenches - no "short" waist length coats. Their suits and blazers are also quite good. Do stick to their tailored and Monogram lines though. The Monogram line fits me like a glove in a good way, while tailored fit is hit and miss, and they can run long.
H&M
It's been a while since I've shopped at H&M. They are really fashion forward, but have a lot of "traditional" items too. Quality is a bit cheaper than Gap in my experience. That's not to say that it's bad. They are European and everything is trimmer and more fitted. I would say that their S size is equivalent to an XS size at North American outlets. Since it's been so long, I'll keep it short. Their dress shirts are in the $50 range and everything else is all over the road, yet reasonably priced for the most part. The absolute gems are their suits. Suits with pants and jackets are around $300 and are generally slim, tailored fit.
Fjallraven
This is a Swedish store that specializes in apparel for the outdoors person. They have one store in New York City that I know of and sell online - just watch out for abhorent shipping and customs charges if you go crossborder (Canadians can check out AJ Brooks in Vancouver who ship nation-wide). You may also be able to find certain items from their line in American outlets. They are by no means cheap though, and shirts alone run north of $100. What I love about them are their jackets. Their XS size fits very much like JCrew items: a bit tight and long. However, stick to the "it is supposed to be long and past the waist" rule, and you'll be fine.
And there endeth the lesson. The last recommendation I can give as you set forth in your journey is to look for European outlets. If you're wondering why I didn't mention pants, it's because pants are easy. Buy your waist size and cut or fold the hems.
Good luck!