Mobile App Personalization

Previously…

Before I worked on this personalization project, there was prior research done by a fellow researcher that indicated a “recent activity/updates” section on the home page is highly desired by users.

Taking this insight, the product team decided to add this new section as a place to provide personalized notifications for members based on their recent health activities and other known needs. As the next step, product came to the UX team to determine:

What events/needs do people want to receive push notifications about?
Out of all the notifications we can send, which notifications are the most helpful?

Starting out

To begin, I worked with the designer on the project, the product team, and the experience strategy team to create a large table of all the notifications that members could receive. This ended up being 129 different notifications. By referencing the member journey map created by the experience strategy team, we were able to group the notifications into 24 different “trigger events”. These trigger events are moments during a health insurance journey that would lead to certain notifications becoming relevant.

Master table of trigger events and notifications

For example:

One trigger event is “having a baby”. When this event occurs, the following notifications become relevant to the member:

“Add your new baby to your plan”
“ View your new baby’s ID card”
“View information on what to expect over the course of the year”
“Here are some options for when your pediatrician is not available”
“Learn about your new baby’s covered healthcare services”

The number of notifications attached to each trigger event varies from 2-9.

We now had a working table of trigger events and notifications to evaluate.

Study Optimization

Next, I started planning the study. My biggest concern, however, was the sheer quantity of things that I needed to evaluate. Combining both the events and notifications, there were 153 different pieces of information to test, which was definitely way too much for a participant to evaluate in one sitting. I thought through several different methods before setting on my final approach.

Option 2:

Run 1st study in which participants evaluate the 24 trigger events on a scale of 1 to 5.

Analyze results to see which events performed well (4+), and then run a 2nd study testing the notifications for these events.

Ruled out because:

Lengthy amount of time needed to run 2 studies with analysis in between.
Since the 2nd study is a separate group, there is also the chance that participants will be asked to evaluate notifications for events they don’t care about, which may skew the results.

Option 1:

Only evaluate the notifications. My thought was that by working backwards, we would be able to determine the events people wanted notifications about if the relevant notifications were well received.

Ruled out because:

This may not be true. It is possible that a highly desired event may have bad notifications, or vice versa.

Final approach:

Utilizing Userzoom’s logic and randomization capabilities, each participant will be asked to evaluate the usefulness of 8 out of the 24 total trigger events on a scale of 1 to 5. For events that they rated highly (4+), they will then be asked to evaluate the helpfulness of receiving notifications for those events.

In addition, by running 3 studies of 100 participants each, each trigger event will be evaluated around 100 times.

In order to combat fatigue since the study could still end up quite long depending on participant responses, I added red herring questions to every event that had 6 or more notifications associated with it. These red herrings were formatted to look similar to all the other questions as to not stand out, and allowed me to filter out participants that were not paying attention to the questions.

Example red herring (an answer of 4 or 5 would allow the participant to pass):

Notification: “Night” and “Day”

How opposite (1) or similar (5) are the two concepts listed above?

Option 3:

Split the table into smaller groups and ask participants to evaluate all events on a scale of 1 to 5. They would then be asked to evaluate the notifications for events that they rated highly (4+).

For example:

Study 1: Events A, B, C (and their related notifications)

Study 2: Events D, E, F (and their related notifications)

Ruled out because:

If the smaller groups are pre-determined, there may be biases created by grouping certain items together since participants may start to compare the events.

Analysis of large-scale quantitative studies done in Miro

Results were synthesized using Miro and Userzoom. I was able to identify high and low rated events and notifications, along with some interesting findings such as high rated events with low rated notifications (and vice versa), as well as events that had a large number of low rated notifications. These results helped the designers and product team to know what topics/events to focus on, as well as which areas need further development and improvement.

Follow-up

Next, I was asked to conduct follow-up research, which ultimately consisted of five studies.

Four were deeper dives into specific trigger events that the product team decided to focus on (premium payment due, claim processed/denied, deductible met, routine screening due). Within each of these studies, the main focus was user preferences around the timing and behaviour of notifications, as well as notification clarity.

The last study was around general expectations of behaviour for this new personalized notification area. This study included things like user expectations of what happens when you tap on a push notification and how notifications in the “Your Updates” area should behave when new ones are added or as time passes.

Results were synthesized using Miro and Userzoom.

Analysis of follow-up studies done in Miro

As of December 2024, this project is currently ongoing.