The Challenge
Marketers have increasingly struggled with “email burnout” among portions of their customer base. Inversely, other portions were left undersaturated, leaving unlocked revenue potential. To meet the Marketing Cloud’s goal of being a leader in personalization, we needed to help marketers engage differently with these two types of segments.
Enter Einstein Engagement Frequency (EEF). Released in 2019, it provided likely the first-ever AI solution for saturation management.
I became responsible for the design shortly after its initial launch, standing on the shoulders of the designer before me. Now, my goal was to design the next version.
V1 (MVP)
Understanding: Product, Algorithm, & Design
For V1, the scenarios were narrowly scoped. Users would enable the feature, then, following some thoughtful FTUX, would tune the data model parameters based on their optimization goal. To use the tuned feature, they would then generate “Data Extensions”—Marketing Cloud’s early, primitive form of segmentation—which could then be used to market to.
When I first took hold of the project, my team knew that this was a rapid MVP and needed improvements. The experience centered around a massive chart, which was packed with beautiful data.
As often happens, the algorithm was complex. Boiling it down to its key inputs and outputs made it much more explainable. I didn’t need to understand what happened in the middle; that’s what Data Scientists are for. My goal in understanding was to be able to explain how it worked to a 5-year-old.
So how did the MVP land?
Feedback We Heard
As is often the reality in SaaS, the Product Manager was the conduit for most of our research; having routine calls with various customers, and capturing their feedback in unstructured ways.
Thankfully for EEF, the feedback was clear:
- Needed easier ways to use segmentation.
- Needed better reporting view; the chart was trying to do too much at once.
- Needed to identify at-risk segments, or people who were almost saturated.
- Needed to build trust in the data.
We knew what we needed to solve. Thus began the ideation.
V2
Segmentation Usage
To improve usage, we knew right away it needed to be integrated with our messaging orchestration tool, Journey Builder. But how?
We explored two key directions here (and, to be sure, I went bluesky with some ideas in my notebook):
- EEF Split – Create a new split activity
- Megasplit – Merge with existing “Einstein Split”
- Easy DEs – Allow EEF Data Extension creation from Journey Builder
In a crowded portfolio of enterprise features, feature discovery was always a challenge, so that reduced focus on the Megasplit. Maintainability was also a consideration, so that further reduced desireability of the Megasplit. And Easy DEs was only an option if our engineering resources were severely limited – which thankfully, they weren’t.
After research, we ultimately landed on creating a new activity for Journey Builder, the EEF Split.
New “Almost Saturated” Segment
This new EEF segment required a new lever: How “risk-off” do marketers want to be when defining almost-saturated? We boiled this down to two key questions:
- Where should this be configured, and at what level (org-level vs journey level)?
- What threshold values would marketers find helpful? (1 email away from saturation? 5? 10? 50?)
After much discussion, we realized that we had no clue what the answers were. So we tabled it to later validate in research.
Monitoring, Reporting, & Data Trust Improvements
While V1 was thick with data visualization, it tried to convey too much at once. I don’t care how smart a person is, any human looking at that chart would be overwhelmed—especially burnt-out marketers.
So we kept the original view mostly in-tact (an interesting decision, in retrospect), and planned to build a new Reporting dashboard. This new report would serve to answer key questions about their overall customer base, and how saturation impacted engagement.
Importantly, we recognized Monitoring as a different use-case than Reporting: We also built a view within Journey Builder, where marketers could see live insights into specific segments while they were in flight.
Finally, to build trust in the data, we leveraged our new Data Scorecards — an important output from our AI Ethics program.
Validating Our Improvements
Before the release began, we needed to be sure our solution was solving the known pain points, and not introducing new ones. We ran a small qualitative study with early adopters, validating our V2 solution.
We gained confidence in answering many of our questions, and from the feedback, identified new improvements for the future. Overall, participants rated the new designs with an average NPS of 6/7.
Reflection
Customer Feedback
Once this was in the wild, public feedback was over the moon:
- It boosted revenue from under-engaged segments, with early adopters reporting 10–20% engagement lifts in case studies.
- It boosted engagement, with a 2020 webinar by Eliot Harper predicting 3–15% open/click gains, calling it a “sweet spot” tool for saturation control.
- It curbed unsubscribes, with commentary consistently framing EEF as a “game-changer” for doing so.
What would I improve?
Tuning is an iterative process. But based on what?
Having a tighter feedback loop between viewing performance and making tweaks to the algorithm would take this to the next level. Providing recommended tuning tweaks would take this even further. We heard this in the research, and it was one of the biggest known opportunities when I rotated off of the project following V2.
Overall, the most important lesson was building trust: People need to be kept “in-the-loop” when it comes to AI-driven automation features, especially when it comes to how they interact with their customers.