Tests, Controls and Results
	Posted by Katie Beth DeSchepper at Mar 02, 2012 10:04 AM CST Categories: Fundraising, Nonprofit Trends, NPtech

I was chatting with a friend of mine the other day around an End of Year online campaign and we were discussing the results.  I was really intrigued at trying to understand whether or not a mobile text that went out before the campaign had increased the open and donation rates. I kept thinking, surely if I text someone ahead of an email (another Multi-Channel campaign case) I would see a significant increase in open rates and donation rates. However, I was so regretfully disappointed as they didn’t set up their test appropriately for us to really understand the impact. 

So, what do you have to do to set up a test RIGHT so that you can read results and have the significance needed to be sure that the results will be the same going forward? 

1. Establish the metrics that will measure success. Think about your objectives…in this case, it would be can I improve my open rate by setting up a pre-email text message. 
2. Establish your test and control groups. You must have a control.  If you don’t, you really will never know if the text message really made a difference, because you didn’t set up a group that stayed the same OR didn’t get a text message.
3. Make sure your groups (test and control) will have enough observations (opened email) that can be statistically compared. If you are focused on a response (open) rate, use this calculator provided by The DMA.

Why is it so important to have a control and statistically significant data to have a result? It is like building or buying a house, but not insuring it. Testing is expensive and has large opportunity cost both positive and negative. So, if you are going to test something, make sure it is worth it and you can after the test is complete, that you have a result. Otherwise, it is just interesting and the test was a fun exercise.