Add a Skip and a Note to Every Question

Pain Point

It’s easy for surveys to get wrong data or frustrate users by not guessing the right multiple options to offer.

How many hours have you used X tool in the past month?

An unskippable input box with a number. You can’t answer “I don’t know”.

How often do you upgrade?

whenever there’s a new version

quarterly

every 6 months

every 12 months

No “other” option. When I come across a question like this, maybe my actual answer is:

“I upgrade when 1) if I don’t, things will break, or 2) I really want a specific feature that I’m fine paying the upgrade cost for”.

That answer gives the surveyor my actual intent, and doesn’t frustrate me as a respondent by not providing me with the right option/s. This answer also lets the surveyor realize that a good problem to solve may be advertising new features or showing how easy it is to upgrade versions. Follow up questions may be “What version are you currently using” and “do you recall the last time you upgraded? When and from which version?”. Those may give you more insight on upgrade patterns.

LLMs make it very easy to parse into buckets anyway, so you’ll still be able to cluster, analyze, show nice graphs and glean insights. But even without LLMs, understanding your real surveyed person’s intent over some non-representative choices is vital for your data quality.

The Rule

Add a skip and a free text field in EVERY answer. No exceptions. Even if they pick a multiple choice answer, they can expand on it, so that two people who both picked “every 6 months” may end up in different clusters in your final data.

Note

Bonus: Use a coding agent as the survey interface, so users can clarify what a question or concept means, or narrate an answer to help them pick.

Anecdotal Evidence: Applying This Lightly With Slack + Agents

Wanting a quick turnaround of answers in a short amount of time, I used an agent skill for both the surveyor and the surveyed (who were already using coding agents).

Note

An anonymized version of that is here skills/survey

The flow goes like this:

I (The surveyor) design the survey by talking to the skill. I paste my rough notes (half-formed questions or some with a 0-5 scale I had been using) and it shapes them into a survey. I keep it small on purpose.
The skill writes a plain Slack message: a short intro, the scale, the questions. It posts it to a channel and edits the “answer here” link back in, like so:

Survey: Team Tech Survey (Q2–Q3)
Results to: @me

To answer, paste this into the bot:
/survey Answer survey at <link>

The bot walks you through each question. You can answer, say skip if it
doesn't apply, or say "I don't know", which counts as a real answer here.
If you're not sure what something means, ask the bot to explain it and draft
your answer with you.

No access to the bot? Copy this message and DM me your answers directly.

Part 1: Score these tools, 0 to 5. Add a sentence to qualify it if you want,
like "4, I can write and run models fine but incremental builds still trip me up".
[scale 0-5] · [list of tools]

Part 2, open questions: what tools are you missing, most annoying thing this
quarter, best thing, a task that took far longer than expected...

It DMs the people I’m targeting. For anyone who hasn’t answered, I can send a nudge.
Each person answers by pasting the link into their own agent. It walks them through one question at a time. They can skip, say “I don’t know”, or ask what a question means and have the agent draft the answer with them.
Their answers come back to my DMs (or wherever I picked).
I gather and structure them (scores into a matrix, notes kept beside them) and go from there to the graphs.

flowchart TD
    subgraph author["Surveyor"]
        write("Write the survey as a plain message")
        post("Post to a channel")
        collect("Collect answers and read the notes")
    end

    subgraph channel["Shared message"]
        msg[/"Survey + link to answer"/]
    end

    subgraph respondent["Respondent, guided by the agent"]
        open("Open the link")
        read("Agent reads the survey")
        ask{"Next question?"}
        answer("Pick an option or type, plus an optional note")
        clarify("Ask the agent to explain, or align on what to pick")
        skip("Skip or I don't know")
        review("Review and edit the summary")

        confirm("Confirm before sending")
    end

    write --> post --> msg
    msg --> open --> read --> ask
    ask -- "has an answer" --> answer --> ask
    ask -- "no opinion" --> skip --> ask
    ask -- "unclear?" --> clarify --> ask
    ask -- "done" --> review --> confirm
    confirm -- "send to destination" --> collect

I’ve run this on a group in the low double digits and gotten 100% adherence in a very short amount of time, in contexts where before I wouldn’t have dreamed of asking complex questions. Since answering was cheap and felt natural for people already working in a chat interface, they used it and even gave positive feedback on the methodology during the survey results presentation. People did skip questions, and that was perfectly fine and a good data point.

Do

Attach an optional note to every question. Even when they select a specific choice.
Offer Skip and I don’t know as real answers, and record them as such.
Tell your users up front that they can skip most things if they want to. Those who do would otherwise have dropped out or picked any option in frustration. The amount of skipping is data too.

Do Not

Do not force a choice. A forced answer is a false answer, and later you won’t be able to tell it apart from a real one.
Do not invent answers on someone’s behalf. A skip is a skip.
Do not add a question to capture a nuance a note could hold. Long surveys get abandoned, so mind your users’ tolerance for length.

Be Conscious Of

The examples and choices you give will still heavily influence answers

Coding Agents

How you ask changes what you get. When the survey runs through an agent that already knows the person and their work, it can rephrase a confusing question and let them think out loud, so the notes come back richer than a form’s. That only works with an agent as the medium; a static form can’t reinterpret a question or draw someone out.

Warning

Since a coding agent has access to the person’s computer, it could answer some questions more effectively by actually checking. That’s risky: if the agent is wrong and the person can’t tell, they’re better off saying “I don’t know”.

Note

These principles as an Agent Skill are exemplified here: skills/survey.

This is not meant to be a runnable skill, but you could easily create one in your own secure environment, taking some ideas from it.