I also have all the albums by the same artist in one folder. It still plays all the albums in alphabetical order of the album name.
Do I create the playlist in iTunes or do I have to do it another way?
I don't think you've said what view you're in. I can view by album, artist, folder, genre and something else I think. Artist for me seemed redundant to folder view (my default) at first, then I realized that my tags were optimized for folder view, and there's a lot of loose stuff (like individual cuts that are tagged with a different artist tag from the rest of an album) that messes up artist view for me.
Even if I took the trouble to straighten the tags for artist, it would just be perfectly redundant to folder so I don't see the point.
Anyhow, if you're stuck in album view, of course it won't play anything but that (alphabetical by album name) so try artist or folder view like others have said.
Artist doesn't stick either, so you have to go back to it every time you want to change albums, it's just a way to one-time view your stuff as far as I can tell, but I'm not into all this like some are. I get to artist by slapping buttons until I see the musical note at the top, then touch that, then your source, and you have a library view available which for me includes the entries listed above.
My source is a SD card in SD1.