if you want to get really advanced you could get a packet sniffer/ethernet sniffer and capture all the data being sent out and then analyze it.. but for the most part you have to some element of trust in the program that you plan to use. Look for suspicious behavior. If your program is accessing strange ports that it shouldn't be or if the port ranges are not within the specified range(consult the program's manual) then there may be a problem. Do your keystrokes get interrupted when you are typing or is there lag on your keys(keylogger). Does the program use a lot of CPU cycles or bandwidth when it is idle?
If you are creating an automated trading system, then make sure you download your program from a reputable source and that it is from a reputable dealer(Wealth-Lab, TradeStation, MT4, ect..) or you just simply make you own(that way you know 100% that it is safe).