Reader Feedback about my Chapter List

For the past eight months, I’ve been working on my book, Refactoring English: Effective Writing for Software Developers. I’m publishing the book incrementally as I write, which means readers give me feedback about the book in real time.

I started writing the chapters I felt most confident readers would like, but now that I’m about 50% complete, I want to make sure the remaining chapters are things my readers actually want to learn.

I had two main questions:

1. Are there any chapters that nobody wants to read?
2. Do the most important chapters appear early in the book?

I realized I could answer this question by reaching out to my readers, and the results turned out to be interesting and useful.

The survey🔗

I created a survey that invited readers to rate each chapter on a scale from “Definitely won’t read” to “Definitely will read”:

I sent the survey to 1,644 people who had either purchased early access or subscribed to my mailing list for free updates and previews of the book. I received 133 responses, which means about one in 13 readers filled out the survey:

Full results🔗

Here are the full results among all readers who responded to the survey:

Comparing paying readers to free readers🔗

I was curious to see how the data would change if I split out readers who already purchased the book vs. readers who subscribed to the mailing list to receive updates and free sample chapters.

Paying readers only🔗

Free readers only🔗

Paying readers are more excited🔗

The biggest difference is that readers who purchased early access are more enthusiastic about reading the book. That’s what I expected, but it’s interesting to see the data illustrate it so clearly for every chapter.

The widest disparity between paying and free readers was on the chapter about writing compelling introductions. 66% of paying readers rated the chapter “Get to the Point” as “Definitely will read,” whereas that number was about half for free readers.

What do the different preferences reveal about paying vs. free readers?🔗

There were three chapters where I expected the answers between paying and free readers to differ most strongly:

• Write Blog Posts that Developers Read
• Good vs. Bad Content Marketing
• Write Compelling Software Release Announcements

My intuition was that readers who purchased early access were more likely to be bloggers or have their own products of some type, so these chapters would be more relevant.

The data didn’t support my intuition here. Paying users were more enthusiastic about those chapters, but only to the degree that they were extra enthusiastic about every chapter.

How do I interpret this data?🔗

There are a lot of ways to view the results, and I wasn’t sure which view would give me the best signal.

I decided to weigh the feedback as follows (from highest weight to lowest weight):

1. Paying readers, order by “Definitely will read.”
2. Paying readers, order by “Definitely will read + Will probably read.”
3. Free readers, order by “Definitely will read.”
4. Free readers, order by “Definitely will read + Will probably read.”

I want to write something that everyone will appreciate, but for practical reasons, I need readers to purchase the book. I think that readers who have already purchased early access are more representative of readers who will purchase in the future.

It’s possible that my strategy for interpreting the data is wrong. An alternative explanation would be that the non-paying readers are telling me why they haven’t purchased yet. Maybe if I catered to their feedback, I could earn many more pre-sales than I already have.

The reason I’m giving more weight to paying readers is that historically, feedback from paying users has led me to more paying users, whereas feedback from free users has led me to more free users.

What are the actionable insights from the data?🔗

I’m going in the right direction🔗

I don’t have a baseline to comare these numbers to, but they make me feel like the book is going in the right general direction.

The survey response rate itself makes me feel confident that people are interested in the high-level topic of improving writing for developers. I was expecting 2-3% to respond, but 8% responded overall, with 14% of early access customers responding.

I feel good about the per-chapter ratings, as well. Every chapter received ratings of “Definitely will read” or “Probably will read” from at least 50% of respondents.

Move up “Make Your Writing Sound Natural”🔗

Almost everyone said they wanted to read the chapter “Make Your Writing Sound Natural.” At only two pages long, it’s the shortest chapter.

In terms of giving the reader bang-for-their-buck, it’s a no-brainer to put this chapter early in the book where they can get the value immediately.

Rename “Good vs. Bad Content Marketing”🔗

Both paid and free readers seem unenthusiastic about the content marketing chapter. This surprised me because in one-on-one conversations, developers tell me they’d love to write blog posts that help them find customers, which is a form of content marketing.

Paying readers who gave a low rating to “Good vs. Bad Content Marketing” also gave a high rating to “Write Blog Posts that Developers Read.” If you’re interested in writing blog posts that appeal to developers, wouldn’t you also be interested in using those skills to find customers?

I wondered if the term “content marketing” was a turn-off, and readers would be more interested if I called it something like, “Find Customers through Blogging.”

I reached out to several readers who gave low ratings to the “Good vs. Bad Content Marketing” chapter to test my hypothesis. More than half responded to say that they indeed were reacting negatively to the term “content marketing.” They felt more interested in a chapter called, “Find Customers through Blogging,” so I’m going to rename the chapter.

Be more opinionated🔗

The negative reaction to “content marketing” also gives me confidence about being more opinionated in the book.

I dislike the term “content marketing” and generally avoid it. I used it in the chapter in hopes of appealing to people who see “content marketing” as valuable.

The results tell me that people who are interested in the book, especially people who purchased the book, are similar to me and will probably respond better if I just write in the way that feels natural to me.

Move up “Write Effective Design Documents”🔗

I’ll be honest: I was kind of hoping everyone would say they hate this chapter, so I could wriggle out of writing it.

I love design documents, but I haven’t written one in a few years. Worse, I don’t have access to any of my design documents because they’re silo’ed at my old employers.

Most developers I’ve worked with hate design docs and find them pointless, so I thought maybe I was off the hook. I’m glad to see further confirmation that readers for this book share my sensibilities and care about the things I care about.

Move up “Rules for Writing Software Tutorials”🔗

Both free and paying readers rated this chapter among their most interested. Paying readers rated it similarly to, “Write Blog Posts that Developers Read,” but free readers gave stronger ratings to the tutorials chapter.

I think more developers end up writing tutorials than blog posts, so more readers will likely find it useful. I’ll bump it slightly to be in front of the blogging chapters rather than behind them.

Leave the rest as-is🔗

The rest of the chapters, I’m going to leave in place.

Depending on how I slice the data, sometimes it tells me that the emails chapter should be ahead of the commit message chapter or something, but I don’t see any compelling reason to reorder anything else.

New chapter order🔗

Thanks🔗

Thanks to everyone who gave feedback in the survey and over email!

As always, reach out if you have any suggestions about the book or questions you’d like the book to answer.