The raised brick garden beds were filled with sandy loam. Apparently it's a common soil for filling new garden beds. It looks nice and probably has good drainage, but it's actually fairly crap - too much sand, not enough organic matter or clay. Dries out pretty quickly. It's the top blotch in the photo.
Around the house is what I call "builder's dust." I'm not even sure what it's made from, it's not sandy, it's not clay, there's almost nothing organic in it. I think it might be sub-soil that got dug up when they built the house. The best description is "powdery", it dries out incredibly quickly into dust. You can see it in the bottom left.
All three of these soils got about the same amount of rainfall when I sampled them, but by then the builder's dust was bone-dry, the sandy loam was a bit moist and the dark loam was still beautifully moist. The reason the picture looks wet is because I wanted to demonstrate the water-holding qualities of the three soils. I dripped a few drops on each one. For the dark, rich loam the water soaked straight in so you can see the glisten but nowater around it. For the other two, the drops rolled straight off the soil and formed a puddle in between the two. This is called "hydrophobic soil" - not only does it not hold onto water well, when you water the soil it just puddles on top at first.
The only solution for builder's dust and sandy loam is more, and more, and more organic matter (and maybe some pellets of clay). It'll take a few years to get it as rich as the dark loam, but it'll be so worth it.
Edit: Update! The above photo is pretty awful so I recently decided to take a few better snaps. It rained only 2 days ago so you can really see the difference in how these soils hold the water.
Here's the horrible "builder's dust". Notice that it's already almost bone dry.