Shogun: Exploring automated optimization with AI
Envisioning how AI could transform the way merchants optimize their stores.
Date: March – April 2024
My role: Staff Product Designer
Who I worked with: CEO, CTO, Director of Design
The problem & opportunity
A/B testing is one of the most effective ways to improve ecommerce performance, but running a successful optimization program requires significant expertise and ongoing effort.
While larger organizations may have dedicated CRO specialists to support this work, many small and mid-size organizations lack the time and resources to maintain a continuous optimization process. As we looked at the problem more closely, we realized that traditional testing tools place most of the responsibility on the merchant. The software provides the mechanics for running experiments, but merchants are still expected to determine where opportunities exist and how those opportunities should be pursued.
This led us to a broader question: If AI could identify opportunities and generate recommendations on behalf of merchants, could optimization become more accessible, more effective, and less dependent on specialized expertise?
Exploring solutions
Reflection
Designing for uncertainty
Unlike other product initiatives that I worked on while at Shogun, this project was never intended to produce features that could immediately move into development. Its purpose was to help leadership align around a long-term vision for optimization and create a shared understanding of where the product could evolve as AI capabilities matured.
By translating abstract ideas into tangible workflows and interfaces, we were able to move conversations beyond theoretical discussions and create a concrete picture of what the future might look like. The details of the flow and the screens were less important than the alignment they created.
This work reinforced an important lesson: Vision projects are not about predicting the future. They're about creating a shared destination. Once teams have a common understanding of where they want to go, shorter-term decisions become easier to evaluate and are more likely to move the product in a consistent direction.
Designing for uncertainty
One of the most interesting outcomes of the exploration was realizing that AI's greatest potential wasn't simply helping merchants run tests more efficiently. The more compelling opportunity was rethinking the role of optimization software altogether.
Instead of merchants operating testing tools, we explored a future where merchants would define goals and priorities while AI handled much of the discovery, experimentation, and iteration required to achieve them.
Whether or not the specific concepts from this project are ever implemented, the exercise helped us think critically about the future of optimization at Shogun and helped establish a strategic direction that could influence future product decisions.
Choosing where and how optimization happens: Not every page requires continuous optimization. We explored allowing merchants to enable or disable automated optimization on a page-by-page basis, giving them control over where AI should focus its efforts. Merchants could also adjust the percentage of the page’s traffic that saw tests, and could toggle on or off automatically publishing A/B tests, giving them the control necessary to make sure automated optimization aligned with their comfort level and business goals.
Monitoring optimization activity: For each page where optimization is enabled, merchants can monitor experiments in progress, track emerging winners, and access detailed analytics to understand exactly how each optimization is influencing performance over time.
Managing hypotheses: Rather than requiring merchants to generate ideas from scratch, AI would continuously suggest optimization hypotheses based on page performance and previous learnings. Merchants could review recommendations, reorder priorities, remove suggestions, or add their own hypotheses to guide the optimization process.
Reviewing and refining upcoming experiments: Before tests launch, merchants can review and approve the AI-generated variants, inspecting proposed design and content changes and then make adjustments where necessary by opening the page in the editor. This created a collaborative workflow where AI accelerated experimentation while merchants maintained control over the final customer experience.
Learning from past experiments: We explored surfacing the complete history of experiments for a page, helping merchants understand what had been tested, what had worked, and how those learnings informed future optimization decisions.
A simplified view of the optimization flow.
Overview
Working closely with Shogun's CEO and CTO, I led a vision exploration focused on the future of optimization. We saw significant potential in emerging AI technologies and wanted to better understand how they might reshape the way merchants improve their stores over time. Rather than simply helping merchants run A/B tests more efficiently, we explored a future where AI could identify optimization opportunities, generate hypotheses, create test variations, and continuously learn from results.
This project was not tied to a roadmap or implementation timeline. Its purpose was to create a tangible vision of what optimization might look like several years into the future and help align leadership around a shared direction for the product.
Shogun: Exploring automated optimization with AI
Envisioning how AI could transform the way merchants optimize their stores.
Date: March – April 2024
My role: Staff Product Designer
Who I worked with: CEO, CTO, Director of Design
The problem & opportunity
A/B testing is one of the most effective ways to improve ecommerce performance, but running a successful optimization program requires significant expertise and ongoing effort.
While larger organizations may have dedicated CRO specialists to support this work, many small and mid-size organizations lack the time and resources to maintain a continuous optimization process. As we looked at the problem more closely, we realized that traditional testing tools place most of the responsibility on the merchant. The software provides the mechanics for running experiments, but merchants are still expected to determine where opportunities exist and how those opportunities should be pursued.
This led us to a broader question: If AI could identify opportunities and generate recommendations on behalf of merchants, could optimization become more accessible, more effective, and less dependent on specialized expertise?
Exploring solutions
Reflection
Designing for uncertainty
Unlike other product initiatives that I worked on while at Shogun, this project was never intended to produce features that could immediately move into development. Its purpose was to help leadership align around a long-term vision for optimization and create a shared understanding of where the product could evolve as AI capabilities matured.
By translating abstract ideas into tangible workflows and interfaces, we were able to move conversations beyond theoretical discussions and create a concrete picture of what the future might look like. The details of the flow and the screens were less important than the alignment they created.
This work reinforced an important lesson: Vision projects are not about predicting the future. They're about creating a shared destination. Once teams have a common understanding of where they want to go, shorter-term decisions become easier to evaluate and are more likely to move the product in a consistent direction.
Designing for uncertainty
One of the most interesting outcomes of the exploration was realizing that AI's greatest potential wasn't simply helping merchants run tests more efficiently. The more compelling opportunity was rethinking the role of optimization software altogether.
Instead of merchants operating testing tools, we explored a future where merchants would define goals and priorities while AI handled much of the discovery, experimentation, and iteration required to achieve them.
Whether or not the specific concepts from this project are ever implemented, the exercise helped us think critically about the future of optimization at Shogun and helped establish a strategic direction that could influence future product decisions.
Choosing where and how optimization happens: Not every page requires continuous optimization. We explored allowing merchants to enable or disable automated optimization on a page-by-page basis, giving them control over where AI should focus its efforts. Merchants could also adjust the percentage of the page’s traffic that saw tests, and could toggle on or off automatically publishing A/B tests, giving them the control necessary to make sure automated optimization aligned with their comfort level and business goals.
Monitoring optimization activity: For each page where optimization is enabled, merchants can monitor experiments in progress, track emerging winners, and access detailed analytics to understand exactly how each optimization is influencing performance over time.
Managing hypotheses: Rather than requiring merchants to generate ideas from scratch, AI would continuously suggest optimization hypotheses based on page performance and previous learnings. Merchants could review recommendations, reorder priorities, remove suggestions, or add their own hypotheses to guide the optimization process.
Reviewing and refining upcoming experiments: Before tests launch, merchants can review and approve the AI-generated variants, inspecting proposed design and content changes and then make adjustments where necessary by opening the page in the editor. This created a collaborative workflow where AI accelerated experimentation while merchants maintained control over the final customer experience.
Learning from past experiments: We explored surfacing the complete history of experiments for a page, helping merchants understand what had been tested, what had worked, and how those learnings informed future optimization decisions.
A simplified view of the optimization flow.
Overview
Working closely with Shogun's CEO and CTO, I led a vision exploration focused on the future of optimization. We saw significant potential in emerging AI technologies and wanted to better understand how they might reshape the way merchants improve their stores over time. Rather than simply helping merchants run A/B tests more efficiently, we explored a future where AI could identify optimization opportunities, generate hypotheses, create test variations, and continuously learn from results.
This project was not tied to a roadmap or implementation timeline. Its purpose was to create a tangible vision of what optimization might look like several years into the future and help align leadership around a shared direction for the product.
Shogun: Exploring automated optimization with AI
Envisioning how AI could transform the way merchants optimize their stores.
Date:
March – April 2024
My role:
Staff Product Designer
Who I worked with:
CEO, CTO, Director of Design
The problem & opportunity
A/B testing is one of the most effective ways to improve ecommerce performance, but running a successful optimization program requires significant expertise and ongoing effort.
While larger organizations may have dedicated CRO specialists to support this work, many small and mid-size organizations lack the time and resources to maintain a continuous optimization process. As we looked at the problem more closely, we realized that traditional testing tools place most of the responsibility on the merchant. The software provides the mechanics for running experiments, but merchants are still expected to determine where opportunities exist and how those opportunities should be pursued.
This led us to a broader question: If AI could identify opportunities and generate recommendations on behalf of merchants, could optimization become more accessible, more effective, and less dependent on specialized expertise?
Exploring solutions
Reflection
Designing a destination, not a roadmap
Unlike other product initiatives that I worked on while at Shogun, this project was never intended to produce features that could immediately move into development. Its purpose was to help leadership align around a long-term vision for optimization and create a shared understanding of where the product could evolve as AI capabilities matured.
By translating abstract ideas into tangible workflows and interfaces, we were able to move conversations beyond theoretical discussions and create a concrete picture of what the future might look like. The details of the flow and the screens were less important than the alignment they created.
This work reinforced an important lesson: Vision projects are not about predicting the future. They're about creating a shared destination. Once teams have a common understanding of where they want to go, shorter-term decisions become easier to evaluate and are more likely to move the product in a consistent direction.
From testing tools to optimization systems
One of the most interesting outcomes of the exploration was realizing that AI's greatest potential wasn't simply helping merchants run tests more efficiently. The more compelling opportunity was rethinking the role of optimization software altogether.
Instead of merchants operating testing tools, we explored a future where merchants would define goals and priorities while AI handled much of the discovery, experimentation, and iteration required to achieve them.
Whether or not the specific concepts from this project are ever implemented, the exercise helped us think critically about the future of optimization at Shogun and helped establish a strategic direction that could influence future product decisions.
Choosing where and how optimization happens: Not every page requires continuous optimization. We explored allowing merchants to enable or disable automated optimization on a page-by-page basis, giving them control over where AI should focus its efforts. Merchants could also adjust the percentage of the page’s traffic that saw tests, and could toggle on or off automatically publishing A/B tests, giving them the control necessary to make sure automated optimization aligned with their comfort level and business goals.
Monitoring optimization activity: For each page where optimization is enabled, merchants can monitor experiments in progress, track emerging winners, and access detailed analytics to understand exactly how each optimization is influencing performance over time.
Managing hypotheses: Rather than requiring merchants to generate ideas from scratch, AI would continuously suggest optimization hypotheses based on page performance and previous learnings. Merchants could review recommendations, reorder priorities, remove suggestions, or add their own hypotheses to guide the optimization process.
Reviewing and refining upcoming experiments: Before tests launch, merchants can review and approve the AI-generated variants, inspecting proposed design and content changes and then make adjustments where necessary by opening the page in the editor. This created a collaborative workflow where AI accelerated experimentation while merchants maintained control over the final customer experience.
Learning from past experiments: We explored surfacing the complete history of experiments for a page, helping merchants understand what had been tested, what had worked, and how those learnings informed future optimization decisions.
A simplified view of the optimization flow.
Overview
Working closely with Shogun's CEO and CTO, I led a vision exploration focused on the future of optimization. We saw significant potential in emerging AI technologies and wanted to better understand how they might reshape the way merchants improve their stores over time. Rather than simply helping merchants run A/B tests more efficiently, we explored a future where AI could identify optimization opportunities, generate hypotheses, create test variations, and continuously learn from results.
This project was not tied to a roadmap or implementation timeline. Its purpose was to create a tangible vision of what optimization might look like several years into the future and help align leadership around a shared direction for the product.