Very right! If you want it to be drapier, go for a lighter knit like a jersey or a rayon blend. If you want more structure, go for something heavier like a sweatshirt knit, ponte or a good interlock. This is a great example of what a thinner knit would look like: http://thegirlinspired.com/2014/03/just-add-jeans-womens-pattern-tour/

And something more structured like a ponte: http://mellysews.com/2014/03/city-girl-just-add-jeans.html

This post has some great ideas for fabrics for each top: http://mellysews.com/2014/03/favorite-knit-fabrics-just-add-jeans.html