Almost four years ago was the first time I ever used Mambo (the predecessor to Joomla!). As I played around with my newfound toy, I thought the template chooser was one of the coolest modules available. I could just imagine the possibilities. Users could customize the look of my site for how they wanted to view it. I could install 100 templates and let the visitor choose which one he or she liked best.
However, over time I came to realize that the template chooser module was a relatively useless feature that was very difficult to manage. Sure, I could install 100 templates and give users a choice, but those 100 templates would likely have very different layouts and include different sets of module positions. For example, one template might put the "left" module position in the left column while another template would place it at the bottom of the layout while yet another template might not include the left module position at all. You can imagine the headaches this type of situation might cause.
When developing your site, you should give careful consideration to where you want your modules positioned on the page and how you want your site to flow. It's difficult enough to accomplish this with one template, much less several templates. Also, your site's brand identity is dependent upon the design of the site. Giving your user multiple layout options only dilutes your brand and sends mixed messages to the end user.
You might have a valid reason to use the template chooser module, although at the moment I cannot think of an example of a good reason. If you do not have a specific reason to use the template chooser module (other than the "hey, that's neat" factor), my advice to you is not to use it.