To save money, buy everything you can from large grocery chains instead of health food stores. Some websites such as Amazon.com offer better prices if you buy in bulk, and usually have super saver shipping available on orders over $25. Just be aware that some flours and starches need to be kept in the freezer or refridgerator if not used right away.
The Bulk Barn has a good selection of gluten free flours and mixes to choose from and most are cheaper than the packaged flours you can find in health food stores.
I buy rice flour, lentil flour, corn flour, pea flour, garbonzo bean flour (channa) and tapioca starch from Superstore (Loblaws). They can be found in the ethnic foods section of some stores.
Sorghum and Millet Flour
Many recipes call for sorghum flour, but I often have trouble finding it (and when I do I have trouble affording it!). I have found that millet flour makes a great substitute for sorghum flour. Millet flour is a bit cheaper and contains more fibre than sorghum flour. I usually purchase my millet flour from a health food store (eg. Nutter's) and while it's one of the most expensive flours I use, it is an important part of my light flour blend that I use on a weekly basis.
I use the Suraj brand of lentil flour, which is called Urad flour. It adds fiber and protein to flour blends. It acts similar to bean flours. If you can find Bajra flour, it is a combination of rice flour and millet flour and can be used to replace millet flour or sorghum flour in recipes. Lately I haven't been able to find this flour at Superstore (Loblaws).
Oat flour can be combined with rice or other flours, but on it's own it can leave baking quite dense. I like to mix a little oat flour into my other blends for some extra fiber and iron. Don't buy oat flour unless it is cheaper than oats. You can make your own oat flour by grinding small amounts of GF oats in a coffee grinder until fine. Much cheaper! It's best to buy oats that are certified gluten free because oats and wheat are often planted in the same field (on alternate years) or side by side and sometimes a bit of wheat can grow up and contaminate the oat crop.
Corn Flour, Pea Flour, Garbonzo Bean (Channa) Flour
Look in the ethnic foods section of your supermarket. Corn flour is especially cheap. Bean flour is healthy, but has a fairly distinct taste. It tastes terrible raw (you might want to think twice before tasting the batter), but is much better cooked!
Brown Rice Flour
I haven't been able to find this for a good deal in a grocery store as of yet. My go to for brown rice flour is in bulk at the Bulk Barn. It works the same as white rice flour, but adds a bit of a gritty texture to baking. I often mix it half and half with white rice flour to keep some health benefits without too much grit. King Arthur Flour sells a fine brown rice flour that helps to omit the grittiness, while retaining the whole grain goodness.
White Rice Flour
This is probably one of the most common flours you will find at the grocery store. This is the brand I usually use. You can find it almost anywhere and it tends to be a pretty decent price. (This bag was around $2-3)
Xanthan Gum and Guar Gum
I have only found this at health food stores, and it is quite expensive, but you don't use much in a recipe. In order to save money I often use guar Gum instead, which is significantly cheaper, but remember to use 1 1/2 times the amount of the xanthan gum called for in the recipe.
This can also be found in the ethnic food section of many large supermarkets. This is the brand I like to use. It's usually fairly inexpensive (this bag is around $1 for 400g). Tapioca Starch and Tapioca Flour are the same thing. I also sometimes buy this flour in bulk from the Bulk Barn.
Check your local grocery stores for this one. It can often be found in the baking section in small packages. I often buy it in bulk from the Bulk Barn. This is one of the more expensive starches. Note: potato starch is much different than potato flour. Do not mix them up!
You can find this in most health food stores. I buy it at the Bulk Barn. Most recipes only call for a tablespoon or two of this and it helps to absorb liquid. You can usually omit this ingredient in a recipe and add a little extra of a different kind of flour to make up for it. Best stored in the freezer. This is not a substitute for Potato Starch.
I use buckwheat flour for buckwheat pancakes and sometimes mix it in with other flour blends. You can usually substitute buckwheat for millet flour with similar results. It has an interesting texture and it's very healthy. There are light and dark varieties. The dark has a stronger taste and includes more of the hulls, causing it to have a highter fiber content than the light buckwheat flour. I also buy this from the Bulk Barn.
You can find many different kinds of rice pasta in super markets. In the health food section you can usually find brown rice pasta that is certified gluten free. In the ethnic food sections there are various types of white rice noodles that are naturally gluten free but don't necessarily advertise as such.
(If the package says "gluten free", there is a good chance it costs 3 times more)
You can usually find round sheets of rice paper in the ethnic food sections of super markets. I have also seen them at the Bulk Barn and at health food stores. I like to use these to make home made perogies (wrap them around mashed potatoes and cheese and fry up with some onions) or fresh rolls.