ChatGPT o1 is kind of a big deal
ChatGPT o1 is kind of a big deal
How I used it to design a synthetic focus group methodology with one prompt
Before we get into today’s substance, I’ve launched a new Substack, Business is Politics in which I share my takes and analysis at the intersection of business and politics. If you follow me on LinkedIn, or on my podcast, you’ll be familiar with the content. Short, snappy takes, 3x/week.
When OpenAI released ChatGPT4o earlier this year, I treated it as a non-event. It proved to be faster than ChatGPT-4, but the leap didn’t feel nearly as large when we went from 3 to 4. And because the difference between 4 and 4o felt minor, it had little impact on the way I used it.
This new release? o1? (I know, I know, its very lame name…) It feels like a big jump—one that could benefit knowledge workers like us in a big way. It was only two weeks ago that I wrote about how too few people are using AI as a thought partner, and those of us who learn how to do this well will boost our productivity and our careers.
The latest AI model, o1-preview, promised to “think” before acting. This advance means the AI can tackle complex problems that demand deep planning and multiple iterations—think advanced math and scientific challenges. The results? It’s outperforming human PhD experts in solving some of the toughest physics problems out there.
From the OpenAI blog:
We've developed a new series of AI models designed to spend more time thinking before they respond. They can reason through complex tasks and solve harder problems than previous models in science, coding, and math….
We trained these models to spend more time thinking through problems before they respond, much like a person would. Through training, they learn to refine their thinking process, try different strategies, and recognize their mistakes.
In our tests, the next model update performs similarly to PhD students on challenging benchmark tasks in physics, chemistry, and biology. We also found that it excels in math and coding. In a qualifying exam for the International Mathematics Olympiad (IMO), GPT-4o correctly solved only 13% of problems, while the reasoning model scored 83%. Their coding abilities were evaluated in contests and reached the 89th percentile in Codeforces competitions. You can read more about this in our technical research post.
On first blush, this model seems like a huge breakthrough for tasks that require planning.
It didn’t take long for AI experts and observers to put this thinking to the test.
Ben Thompson used the new model to solve the New York Times Daily mini crossword. Where 4o took 12 seconds to complete the crossword, it got it wrong, o1 took 124 seconds and got it right.
Researcher Shital Shah is captivated by the capabilities of the new model. “ChatGPT o1 is getting 80% on my privately held benchmark. The previous best was 30% by Sonnet 3.5 and 20% by GPT 4o.”
Mehran Jalali was able to get the new model to write a very difficult poem that no previous model “got even close to writing.”
The key difference with this model is that the LLM iterates repeatedly, creating and rejecting ideas. And you can see the thought process in great detail.
Here’s an example that is applicable to our work in public affairs:
Task: Develop a methodology for conducting synthetic focus groups
If this term is new to you, synthetic research involves using AI in place of humans. In other words, if I want to simulate a focus group, I get the AI to play the different people in the room. I wanted to compare how o1 performed the task of developing a methodology for this vs 4o.
Can you design a methodology for using ChatGPT for synthetic research? How can I simulate a focus group discussion with ChatGPT?
You’ll see that when the new model provides an answer, it takes time to answer the question. In this case, it took 14 seconds before spitting out a method. If you expand the toggle that appears next to the thinking time, you get a detailed view of the thought process the model went through:
When I talk about AI being a thought partner, this is what I mean. In seconds, I have access to a thought process that I can work with, validate, adapt, and learn from. And that’s without even spending any time with the output.
The output contained many of the elements I’m used to seeing from older models: a step-by-step guide, but the output from o1 is significantly better. Not only is the guide better, but it even thought through a specific and detailed prompt I could use to initiate the focus group:
Here’s how ChatGPT 4o answered the question:
ChatGPT 4o Response
To design a methodology for using ChatGPT for synthetic research, including simulating a focus group discussion, you can follow these steps to ensure a structured approach that mimics real-world research while leveraging ChatGPT’s strengths:
Methodology for Using ChatGPT in Synthetic Research
1. Define Research Objectives
Clearly state what you want to achieve with the synthetic research. Are you testing messaging, gauging sentiment, exploring new policy proposals, or understanding public concerns?
Example Objective:
“Understand public sentiment on private healthcare reform in British Columbia and explore potential objections or support from key demographics.”
2. Select Personas and Demographics
Create detailed personas representing the audience segments you want to analyze. These could include different age groups, political affiliations, socio-economic statuses, or professional backgrounds. This ensures that ChatGPT responds from varied perspectives.
Example Personas:
• Young professional, liberal, environmentally conscious
• Retired individual, conservative, concerned about healthcare costs
• Small business owner, pragmatic, fiscally focused
3. Develop a Discussion Guide
Just as in a real focus group, structure the flow of the discussion with a guide. This ensures that responses stay on track and align with the research objectives. Prepare questions, probes, and prompts designed to elicit responses that mimic real-world dynamics.
Example Questions:
• “How do you feel about the idea of allowing more private options within the healthcare system?”
• “What are your primary concerns with the current healthcare system in BC?”
• “Do you think allowing private healthcare would improve or worsen access to medical services?”
4. Create a Role-Playing Setup in ChatGPT
Design prompts that ask ChatGPT to simulate the various personas or stakeholder viewpoints. To do this, clearly describe the persona’s attributes and the context of the discussion, as well as the expected tone and bias they may hold.
Example Prompt:
• “You are a small business owner in BC, generally concerned about rising costs but open to private healthcare options if they help reduce waiting times. How do you feel about allowing more private healthcare providers into the system?”
5. Conduct Iterative Sessions
Run multiple synthetic discussions using the personas and adapt as necessary. You can iterate on responses to deepen the insight. For instance, you can ask follow-up questions, probe deeper into concerns, or ask for clarifications, just as you would with real participants.
Example Iteration:
• Follow-up: “You mentioned that private healthcare might help reduce wait times. Are there any potential downsides you see to this approach, specifically regarding accessibility for lower-income individuals?”
6. Analyze Synthesized Responses
After running these sessions, analyze the responses for patterns, key concerns, or common themes. Since you control the flow, you can ensure the focus remains on relevant issues and gain synthesized insights.
Analysis Approach:
• Group responses by persona to identify consistent viewpoints.
• Identify frequently mentioned concerns or key areas of support.
• Extract direct quotes to help craft messaging based on these synthesized opinions.
Simulating a Focus Group Discussion
Step-by-Step Approach:
1. Set the Scene:
Begin by describing the context of the focus group to ChatGPT. Mention the topic, the purpose of the discussion, and introduce the various participants.
Example Setup:
“We’re conducting a focus group discussion about healthcare reform in BC. Participants include a small business owner, a young professional, a retiree, and a public healthcare advocate. Each has different opinions on private healthcare options.”
2. Sequentially Engage Each Persona:
Mimic the natural flow of a focus group by having each persona respond in turn. Start with an open question that invites opinions and follow up with more targeted questions.
Prompt Example:
• “What are your initial thoughts on allowing private healthcare options in BC? Let’s start with the small business owner, then we’ll hear from the retiree.”
3. Generate Reactions and Cross-Talk:
Simulate group dynamics by asking participants to respond to or challenge each other’s statements. Use prompts that introduce friction or encourage debate.
Cross-Talk Example:
• “The retiree raised concerns about affordability—how do you, as the small business owner, respond to the idea that private healthcare could increase inequality?”
4. Moderate the Discussion:
Just like in a real focus group, you can moderate to keep the conversation balanced. Ask follow-up questions to clarify points or encourage quieter “participants” to engage.
Moderation Example:
• “We’ve heard from two participants. Let’s hear from the young professional—how do you see private healthcare fitting into your future needs?”
5. Probe for Deeper Insights:
Encourage participants to delve deeper into their reasoning. Use probing questions to explore motivations, values, and potential objections.
Probe Example:
• “Can you elaborate on why you think private healthcare might lower quality of care for public patients?”
6. Synthesize and Record Insights:
Conclude by summarizing key themes from the discussion, just as you would after a real focus group. You can also ask ChatGPT to generate a summary of viewpoints from the group.
Summarization Prompt:
• “Summarize the key concerns and supporting arguments for private healthcare expressed by the group.”
Key Considerations for Synthetic Research
• Diverse Representation: Ensure a wide range of personas to reflect the complexity of real-world stakeholders.
• Bias Awareness: Be mindful of inherent biases in the responses and cross-check against known data or human inputs.
• Iteration and Refinement: Adjust personas and follow-up questions based on early responses to fine-tune insights.
This methodology enables you to leverage ChatGPT’s capabilities for quick, flexible, and deep insights, offering a robust alternative to traditional research methods in public affairs planning.
ChatGPT 4o1-Preview Response
Designing a Methodology for Using ChatGPT in Synthetic Research
Introduction
Synthetic research involves using computational tools to simulate real-world phenomena or gather insights when traditional methods are impractical. ChatGPT can serve as a valuable asset in this domain by generating human-like text responses, allowing you to simulate interviews, focus groups, or explore complex scenarios. Below is a methodology to effectively utilize ChatGPT for synthetic research, followed by steps to simulate a focus group discussion.
Part 1: Methodology for Using ChatGPT in Synthetic Research
1. Define Clear Research Objectives
• Identify Goals: Determine what you aim to achieve—be it understanding public opinion, exploring potential reactions to a policy, or gathering ideas for communication strategies.
• Set Scope: Clearly outline the boundaries of your research to maintain focus.
2. Develop Detailed Personas
• Create Profiles: Construct detailed profiles representing different segments of your target audience. Include demographics, psychographics, beliefs, and behaviors.
• Ensure Diversity: Represent a broad spectrum to capture varied perspectives.
3. Craft Effective Prompts
• Contextual Information: Provide background relevant to the research topic.
• Specific Instructions: Clearly instruct ChatGPT on the desired role or perspective to adopt.
• Open-Ended Questions: Use questions that encourage detailed and thoughtful responses.
4. Simulate Interactions
• Interviews: Engage in one-on-one simulated interviews by interacting with ChatGPT as each persona.
• Focus Groups: Simulate group discussions by requesting responses from multiple personas in a single interaction.
5. Data Collection and Organization
• Record Keeping: Document all interactions meticulously.
• Organize Data: Categorize responses based on themes, sentiments, or any other relevant criteria.
6. Analyze the Data
• Qualitative Analysis: Identify patterns, recurring themes, and unique insights.
• Quantitative Metrics: If applicable, quantify the frequency of certain responses or sentiments.
7. Validate Findings
• Cross-Reference: Compare synthetic data with existing real-world data or studies.
• Expert Review: Have subject matter experts assess the plausibility and relevance of the findings.
8. Report with Transparency
• Methodology Disclosure: Clearly explain the synthetic nature of the research in any reports.
• Limitations: Acknowledge the limitations and potential biases inherent in AI-generated data.
Part 2: Simulating a Focus Group Discussion with ChatGPT
Step-by-Step Guide
1. Define the Purpose of the Focus Group
• Topic Selection: Choose a specific topic or question you want to explore.
• Objectives: Determine what insights you’re seeking from the discussion.
2. Create Participant Personas
• Number of Participants: Typically 5-8 personas for a manageable focus group size.
• Diversity: Ensure a mix of backgrounds, opinions, and demographics.
• Detailed Profiles: For each persona, define:
• Name
• Age
• Occupation
• Personality Traits
• Relevant Attitudes or Beliefs
3. Prepare the Initial Prompt
• Set the Scene: Describe the focus group setting.
• Instructions to ChatGPT: Include guidelines such as:
• Adopt the role of moderator and multiple participants.
• Provide responses for each participant.
• Encourage interaction among participants.
• Example Prompt Structure:
You are simulating a focus group discussion on [topic]. The participants are:
1. Alice, 30, teacher, environmentally conscious.
2. Bob, 45, engineer, skeptical about technology.
3. Carol, 25, student, tech-savvy early adopter.
4. Dan, 50, small business owner, budget-conscious.
5. Eve, 35, healthcare worker, prioritizes community well-being.
As the moderator, initiate the discussion by asking about [specific question]. Provide responses for each participant, simulating a natural conversation with back-and-forth dialogue.4. Start the Discussion
• Moderator’s Opening: Begin with a welcoming message and the initial question.
• Participant Responses: ChatGPT will generate responses for each persona.
5. Facilitate Interaction
• Follow-Up Questions: Pose additional questions to delve deeper into specific points.
• Encourage Debate: Ask participants to react to each other’s statements to simulate dynamic interaction.
6. Manage the Flow
• Balance Participation: Ensure all personas contribute without one dominating.
• Maintain Focus: Steer the conversation back to the main topic if it diverges.
7. Conclude the Session
• Summarize Key Points: Have the moderator recap the main insights.
• Closing Remarks: Thank the participants and end the session.
Tips for Effective Simulation
• Iterative Prompts: If the initial responses are not as desired, adjust your prompt for clarity or add more detail.
• Control the Tone: Specify the tone or style if necessary (e.g., formal, casual).
• Limit Responses: To prevent overly long outputs, you can instruct ChatGPT to keep responses concise.
Considerations and Limitations
• AI Biases: Be aware that the model’s training data may introduce biases.
• Authenticity: While ChatGPT can simulate human-like responses, it may not fully capture the depth of genuine human emotions or cultural nuances.
• Ethical Use: Use the synthetic data responsibly, acknowledging its origin in any reports or presentations.
Example of a Simulated Focus Group Interaction
Moderator: Welcome, everyone, and thank you for joining today’s discussion on renewable energy adoption. To start, could you share your thoughts on how renewable energy impacts our community?
Alice: I believe renewable energy is crucial for reducing our carbon footprint and ensuring a sustainable future for our children.
Bob: While I see the environmental benefits, I’m concerned about the reliability and cost-effectiveness of renewable energy sources.
Carol: As someone who follows tech trends, I think renewable energy technologies are improving rapidly and will soon outperform traditional energy sources.
Dan: My main worry is the initial investment required. Small businesses like mine might struggle with the costs of transitioning.
Eve: From a healthcare perspective, reducing pollution through renewable energy can lead to healthier communities.
Moderator: Those are great points. Bob and Dan, what factors would make you more comfortable with adopting renewable energy?
Conclusion
By following this methodology, you can leverage ChatGPT to conduct synthetic research and simulate focus group discussions effectively. This approach can provide valuable preliminary insights, help formulate hypotheses, and inform decision-making in public affairs planning. Always remember to consider the ethical implications and validate the AI-generated data against real-world information when possible.
What it means
I’m still scratching the surface with the new model, but one thing is clear: With older models, I felt like I was driving in manual transmission (which, to be clear, I still insist on doing and refuse to switch to automatic), in which I choose which gears to put the AI in.
With this new model, it feels much more like an automatic transmission. In the future, my prompts may note need to be exhaustive or methodic in their approach. Some of my earliest guides did a lot of hand-holding, making sure I guided the AI through my thought process. This may soon be a thing of the past.
My job is to figure out how to take advantage of this advanced reasoning, and use my own reasoning to get to a better end product.
It’s exhausting. And exciting.