HubSpot's Inbound Internet Marketing Blog

Current Articles | RSS Feed RSS Feed

Lessons from a Marketing Experiment: Email Marketing Subject Lines

Digg digg it | Reddit reddit | del.icio.us del.icio.us | StumbleUpon StumbleUpon 

One of the things I love about marketing is finding out how people react differently to different versions of the same message, and testing the results in a scientific way so you can learn something. We recently sent an email to promote a webinar we were holding. We sent three different versions of the email to three randomly allocated segments of our list, changing only the subject line and first line of the email message. Based on the results, there were some interesting findings.


Subject Line

Headline
Within
Email

%
Open
Rate

%
Click
Rate

Grade
V.1 Free Internet Marketing Webinar Free Internet Marketing Webinar 18.7% 3.1% C +
V.2 Invitation to Internet Marketing Webinar Internet Marketing Webinar 21.8% 4.4% A -
V.3 Special Invitation to Expert Marketing Webinar Internet Marketing Webinar 21.4% 3.3% B


Email Marketing Subject Line Lessons Learned

1) Don't use the word "Free"
As you can see from the results above, when we used the word free in the subject line and the email text, we got a lower open rate and click through rate. A lot of Spam filters use the word free as a factor in deciding if they should block a message or not, so it is likely that a portion of these emails just got blocked, accounting for the lower open rate. In other tests I have dome about a year ago, I saw that the use of the word free increased click through rates enough to compensate for the lower open rate, but that was not the case here. Sounds like the conventional wisdom of avoiding the word "free" is a good idea.

2) Don't use the word "Special" or other superfluous language

The word special did not affect our open rate, so it looks like that word did not cause the messages to get caught in spam filters more than other words. But, our click through rate for these massages was less than if we used a simpler subject line. This seems to be because people saw the messages but then did not click or act upon them because they viewed them as "spammy" (even if they did not get caught in a spam filter).

Seems to me like you want to keep your subject lines simple and focus on the value of what you are providing - hence why "Invitation to Internet Marketing Webinar" performed the best.

Have you run any experiments on subject lines? What have you learned? Leave a comment below so we can discuss.

Posted by Mike Volpe on Wed, Oct 24, 2007 @ 11:59 AM

COMMENTS

Hi Mike
Interesting article.
I hope you guys keep cranking this kind of stuff out.
Best
Brad

posted on Wednesday, October 24, 2007 at 6:18 PM by Brad STOCK


Hi Mike- I had a similar experiment with similarly interesting results (and revelations). http://www.tonywright.com/2007/everything-is-linkbait/ It was for a beta invite... Next round (we're inviting 1000 people at a time), I'll probably do a bit of experimentation (A/B/C testing?).

posted on Wednesday, October 24, 2007 at 7:04 PM by Tony Wright


Did you do a chi-square analysis? How big was your sample size? Like most direct marketing A-B tests I've seen, the results you got just don't sound statistically significant or repeatable, but there are plenty of statistical techniques to find out if they are. For example, try taking different random samples of the result data set and check the probability that a random sample has the same results as the whole set. One reason I think so much misinformation gets out about direct marketing (like the story that you should use the word "free") is that very few people in the field are doing real statistical analysis; they're just counting % clickthru and calling that "science."

posted on Wednesday, October 24, 2007 at 7:13 PM by Joel Spolsky


Joel - Thanks for the comment. I typically try to keep too much technical / analytical detail off of the blog since most marketers don't care about that level of analysis. But rest assured that the sample size for each email group was "thousands" (I don't really want to talk about our exact list size).

Doing a Chi-squared test gives a chi-squared of "well over 20" (again, don't want to give away exact list size) so this data passes the generally accepted 95%+ probability threshold that the data is significant.

PS - For those of you who want to know more about this statistical test, I found this example. Unfortunately it is from a rival college of mine (I went to Bowdoin - Go Polar Bears!), but it still appears to be correct even though it is from Colby. :)

http://www.colby.edu/biology/BI17x/freq.html

posted on Thursday, October 25, 2007 at 9:37 AM by Mike Volpe


Cool! Well, we're in agreement that "most marketers don't care about that level of analysis" :-)

posted on Thursday, October 25, 2007 at 10:58 AM by Joel Spolsky


Thx Joel. And you are right that we (marketers) SHOULD care.

posted on Thursday, October 25, 2007 at 11:00 AM by Mike Volpe


Mike, I'm basically a solopreneur and haven't done sophisticated testing,but I'm thinking about trying two types of splits to get improved response: male/female with appropriate story heads local/national with local company featured vs national interest header. Have you ever tried something like this?

posted on Friday, October 26, 2007 at 2:56 PM by Newt Barrett


Newt - I have not done a test exactly like this, but I have done tons of other email tests. Give me a ring if you want to chat 800-482-0382 x2

posted on Friday, October 26, 2007 at 3:57 PM by Mike Volpe


I completed my senior school and will be moving to the tertiary and for that matter I want to go and do maker ting and hence I want to know more about before I go to school.Thaanks

posted on Monday, December 10, 2007 at 1:39 PM by Wisdom Owusu


Just thought I'd post a link to The Email Marketing Kit over at SitePoint for people who are new to Email marketing. I used it when I first started and it has some great tips and techniques for maximizing your email campaigns.
http://www.sitepoint.com/launch/c6ac756/3/84

posted on Friday, April 04, 2008 at 2:18 PM by Ryan


Post Comment
Name
 *
Email
 *
Website (optional)
Comment
 *

Allowed tags: <a> link, <b> bold, <i> italics

Receive email when someone replies.

Internet Marketing Blog

The HubSpot Inbound Internet Marketing blog covers all of inbound marketing - SEO, Blogging, Social Media, Landing Pages, Lead Generation and Analytics.

Subscribe to our RSS Feed
HubSpot RSS Feed

Connect with Us

HubSpot on Facebook

 

HubSpot on Twitter

 

HubSpot on Flickr

 

HubSpot on Upcoming

 

ProMarketers LinkedIn Group

 

ProMarketers Facebook Group

 

Want to share your Inbound Marketing advice with the community? Submit guest post ideas to rburnes[at]hubspot[dot]com. 

Marketing Resources

marketing survey
internet marketing group 

 

Website Grader Badge

Popular Posts

Browse by Tag