There are many traps into testing

Testing trading ideas with programs is a matter that causes many discussions around the world, since Technical Analysis, although finally admitted as working, contains so many theoretical traps that one easily believes something to be proven when it is not, and the commercial hype around the programs make it hard to be sure what they really can and can not do. 

Most common error is over fitting

The error you are most likely to do is over fitting your trading idea until it works in exactly the environment you have tested it on, but nowhere else, specially if you have a program that only fulfills category one or two below.

There are programs out there that lets you:

1. Test and optimize your system on one item (stock/index/...) at a time. For example Superchart made by Omega Research is such a program.

2. Test but not easily optimize your system on many items (like all stocks in Sweden for example). For example Tradestation made by Omega Research is such a program.

3. Both optimize and test your system on many items. For example TAS (Technical analysis Scanner) made by Flexsoft is such a program.

4. Both optimize and test your system on many items, at the same time trade against a basket selection of all those items. For example Technifilter is such a program.

If you do an optimization that over fits your system, you might actually be close to optimizing your losses