1. Top Image - Seattle Squeeze.png

Ease the Seattle Squeeze

USABILITY STUDY for/with the Seattle Department of Transportation

1. CHALLENGE

Improving the Seattle Department of Transportation website to make it seamless and easy to navigate while keeping Seattleites informed about potential changes in their commute and enabling change in their commuting behavior.

2. OUTCOME

A comprehensive usability test report and presentation for Seattle Department of Transportation (SDOT).

3. IMPACT

Immediate changes made by the Seattle department of transport to create an all encompassing, easy-to-use website for all things commute related while encouraging use of public transportation.

4. PROJECT SPECS

Area of Focus

Evaluative research, Usability study, Heuristic analysis.


Tools

Miro board, Zoom, InDesign, Illustrator, Google forms.


Methods

Usability test planning, Heuristic analysis, User interviews, Qualitative analysis, Quantitative analysis.


Timeline

January 2019 - March 2019


Team

University of Washington
Devanshi Chauhan, Jen Chiu, Spencer Wilkerson, Abhinav Yadav
Seattle Department of Transportation

Data Librarian & Information Infrastructure Lead

5. MY ROLE

I was tasked with being the Heuristic expert. Based on my findings, we were able to prepare our test kit. I moderated 2 test sessions and took notes for 4. We collectively analyzed the qualitative and quantitative data, and  prepared the report and presentation to the client.

6. PROCESS SUMMARY

We started by understanding the goals of the client at the Seattle department of Transportation. I performed heuristic analysis as well as mapped the site information architecture to set the stage for my team and I to draw out our usability test plan. We got our plan reviewed by our client to make sure we were aligned with their goals. While updating our test plan which included pre-task questions, 5 tasks, and post-task questions, we simultaneously sent out a survey to recruit participants. We conducted a pilot study with a friend to further refine our interview script and task list. We recruited 10 participants out of which 5 were part of our study. We then analyzed the data using thematic and quantitative analysis and synthesized the results and recommendations in a report and presentation form for our clients.

For a detailed case study, use the desktop

 

TIMELINE

Intro | 1 day

Testing with 5 users | 1 week

RQs + Task list | 2 days

3

Recruitment survey

Task List + Interview guide

Research Questions

10 mins Pre-task 

40 mins 5 tasks

10 mins Post-task 

5

Mapping IA

Heuristic Evaluation

2

Preliminary Eval | 2 days

1

Stakeholder Goals, Timeline, Logistics, Methods, KPIs

Pilot tests with peers

Participant selection

4

6

Synthesized Quantitative & Qualitative results

Prioritized findings and design recommendations

Deliverables | 2 days

Usability Report and presentation for SDOT.

7

Analysis | 1 week

Recruitment + Pilot test | 2 Weeks

 

BUT WHY?

Seattle is undergoing massive redevelopment which will lead to worsening traffic conditions. The website seattletraffic.org website aims to inform Seattleites about these changes as well as encourage use of public transport.

My team and I were introduced to the SDOT information infrastructure lead and the data librarian. We understood their need for a usability test. This helped us narrow down our scope as well as focus our attention on elements that were important to them.

 

Seattle is going through a period of massive redevelopment with several projects planned from 2019 to 2025, the consequent worsening traffic condition is known as the ‘Seattle squeeze.’ As a way to tackle this period of rapid change the Seattle Department of Transportation (SDOT) launched their website www.seattletraffic.org to —

Icon for Inform

    Inform    

Inform Seattleites about the various changes and projects the city has undertaken.

Icon for Change Behavior

Change Behavior

Enable change in commuting behavior to encourage use of public transportation.

Video – Landing page for Seattletraffic.org

 

HEURISTIC EVALUATION

I implemented Jakob Nielsen’s 10 heuristics to evaluate the deeper underlying issues with the website. The website seemed confused about its own purpose. This evaluation set the stage for our research questions and consequent tasks for the usability study.

I implemented Jakob Nielsen’s 10 heuristics to evaluate the website. I chose to focus on the deeper underlying issues rather than just aesthetic UI problems. Hence in terms of severity things like "Buttons have different colors" are less severe than "A wrong sign up message."

This exercise left me confused as to what the website is trying to do. There is overwhelming amount of useful information but very little is relevant for one person.

 

The analysis helped us understand the problem area and potential interventions and eventually led us to our research questions.

Visibility of System Status

⦿⦿ What are the blue dots on the 'Current Traffic' Page? 


⦿⦿⦿ Dates are missing on blog posts. Leading the user to question "Is this info still relevant?"

Aesthetic and minimalist design

⦿⦿ There are a lot of aesthetically pleasing elements like the pictures and videos on top of the pages that have very little functionality and increase fold count delaying the the user from getting to the desired/relevant information.


⦿ Alignment of text is inconsistent


⦿ Buttons are different colors

Consistency and standards

⦿ Buttons have inconsistent names


⦿⦿⦿ There are buttons and links on the same page with different names that lead to the same place.

Match between system and the real world

⦿⦿ The website uses certain jargons that are not user friendly. Eg: Commercial Core.

⦿⦿⦿ Names of certain elements do not align with user expectations. Eg: The partners page has a very valuable & relevant twitter feed but the name 'partners' is misleading.

Flexibility and efficiency of use

⦿⦿⦿ The home page and the tools page are too long/ unorganized, this discourages users from reaching the end of the page where there is a lot of relevant information like the twitter feed and the blog posts.

⦿ Action Buttons are below the folds, leading to delay in achieving user goals. 

 Error prevention

⦿⦿⦿ No input field validation; Confirmation message appears even without checking any of the neighborhood boxes.

 User control and freedom

⦿⦿ No indication when a new tab gets opened, leaving the user feeling lost.

⦿⦿⦿      Major Issue

⦿⦿         Moderate Issue

⦿            Minor issue

 

RESEARCH QUESTIONS

Test— the usability of the mobile website, finding transit planning info, usefulness of the tools, user expectations, and ability of the website to enable change in commuting behaviors.

Based on the heuristic analysis and our conversation with our clients, testing the usability of the entire website was still a daunting task. We got in touch with SDOT again to validate our focus and these were our research questions:​

Mobile Website

Is the website easily navigable on mobile platforms?

Expectations

What are the user expectations for a website that purports to contain this type of information?

Transit planning

Are downtown commuters able to easily locate the transit planning information?

Tools

Are the tools provided useful to users?

Change Behavior

Does the website provide enough impetus to change commuting behavior from car to public transportation?

 

PARTICIPANTS

We sent out a google survey to select 10 folx, age 18 years and over who own a smart phone and travel through downtown Seattle at least once a week.

We created a Google survey to recruit our participants via social media and our own personal network.

We had three requirements:​

  • Must be over the age of 18.

  • Must commute to or through downtown Seattle at least once a week — client requirement.

  • Must own a smartphone

We were primarily testing the mobile website since that is most used medium.

Of the 121 responses, we selected 10 participants such that they either had different modes of transport or had a unique commute. Of these 10, 2 were selected for our pilot study, 3 were kept as backup, and 5 were part of the actual study. 

Participants map
Participant Table

Participants map. Hover to see details

 

THE USABILITY TEST

FACILITATION

In person interview with 5 participants, with one moderator and note taker in the room and other teammates using zoom to join from the next room.

We conducted in-person interviews with tasks and questions carefully curated from our prior research. The study took place at the University of Washington the participants were compensated with a $25 gift card, thanks to our client.

 

We had one interviewer and one notetaker in the same room which included a camera to gauge the users expressions and gestures while we used the Zoom app to record the user audio and their phone screen.

TASK LIST

Task

Success Criteria

Rationale

User completes the survey

Pre-test Questionnaire

Task 1

Explore the Website

User is able to navigate the website without frustration for 10-15 mins

Task 2

Locate resources for your commute

Rate the difficulty of this task (1: very easy - 5: very difficult)

User is able to find 3 tools necessary to find their current route within 5 minutes.

Task 3

Locate resources to plan an alternate commute

Rate the difficulty of this task (1: very easy - 5: very difficult)

User is able to plan a hypothetical commute within 10 minutes.

Task 4

Assess the usefulness of the Traveler’s Map and the Projects & Construction Map

Rate the usefulness of the map (1: Not useful at all - 5: Very useful)

User is able to locate and understand the Maps within 10 minutes.

Task 5

Locate SDOT social media (twitter and facebook) and subscribe to email alerts

Rate the difficulty of this task (1: very easy - 5: very difficult)

User is able to find the email sign up in 5 minutes.


User locates the twitter feed in 5 minutes.

Post test Questionnaire

User completes the survey

Understand user expectations vs reality and overall experience of the user.

Here, users were tasked with signing up for email alerts through the website, with additional success measured through finding the aggregated Twitter feeds. This addressed whether or not this key capability was easily accessed via mobile and whether users would adopt this tool.

Traveller's map and the Projects & Construction Maps are specifically under the aegis of SDOT, and they requested that we have users evaluate these two features. Our focus when moderating this task was the ease of navigation on a mobile platform and users’ willingness to adopt these maps as part of their personal information streams.

A huge part of Seattle Squeeze is changing commuting behavior to improve the transportation for everyone, this means less cars on the road. A big part of changing behavior is the ability to do so easily. We were trying to gauge if the website provides an easy way of planning ahead to change long-term commuting behavior.

The website is a comprehensive one-stop-shop for Seattle transportation, but that also means that relevant information is buried under heaps of information. Through this, we were able to assess research questions about the ease of locating transit planning information and whether or not users were willing to switch to new tools that might improve their commuting experience.

We wanted the participant to get familiar with the website and validate our heuristic analysis.

Understand current commuting behavior & User expectations from the website.

ANALYSIS

We analyzed the data through quantitative analysis, including statistics such as the average number of times the participants required assistance during tasks. We examined the qualitative data through affinity analysis in order to identify any common themes (complied here) among the participants.

 

FINDINGS AND RECOMMENDATIONS

Overwhelming amount of information. Enables car drivers, less public transport. Too many options for tools. Travelers map is confusing. Email is not a preferred method to receive notifications. SEO needs to be optimized.

More visual, less text — Users really wanted a timeline to understand the big picture. Visual elements could be added to provide information in a succinct manner to reduce the cognitive load on users.

Customizable — make relevant information for each user’s commute easier to locate (e.g., filters for type of commute, customizable text alerts)

Focus information on commuter impact — users don’t need all the details about delays, just the most updated transit times and alternative routes.

RECOMMENDATIONS

FINDING

⦿⦿⦿ Overwhelming amount information

6 of 7 participants

“I have to say I’m astonished by the amount of the text that they put in here. It’s just, it’s just bad.” -P6

Remove any car oriented resources (e.g., Current Traffic page) OR make them background information — information on here is only for cars yet the goal of this website is to reduce cars on the road.

⦿⦿ Targeted towards car drivers

4 of 7 participants

“Is this all driving? It doesn't say anything about public transit and there are pictures of cars.” -P3

Combine valuable information — reduce the mental load of users by connecting relevant tools (e.g., traffic cameras to twitter alerts, blog posts to a project timeline)

Categorize — based on the type of mode of transport the tool is for. Use the horizontal carousel vs the current 45 fold vertical scroll.

⦿⦿⦿ Tools Page — Too many options as well as unclear description for what they do.

4 of 7 participants

“You need to tell me why I need this thing.” -P2

Redesign the map — based on user expectations with a clear legend.

⦿⦿⦿ Traveller's Map — Not easy to understand

7 of 7 participants

“I REALLY don't like this one” -P1

Make alerts text based and customizable based on the neighborhood, frequency, and mode of travel.

⦿⦿ Email Alerts — Email is not the preferred method of receiving alerts

7 of 7 participants

“I’m a big fan of text alerts. I feel a lot of pressure to keep my inbox clean.” -P1

SEO Optimization

⦿⦿ Website Discoverability — Its the 7th option on the second page of Google

7 of 7 participants

 

REPORT AND PRESENTATION

We presented our findings (presentation & report) to the Seattle Department of Transportation in a meeting attended by the Data Librarian, the Information Infrastructure Lead, a Communications Department representative, and a Parking Department representative.

 

This led to several valuable high priority changes made to the website and smaller changes in the works.

 

REFLECTION

LIMITATIONS

TAKEAWAYS

FUTURE DIRECTION

IF I GOT A REDO

Comprehensive accessibility test — Even though I did a quick and dirty accessibility test, It wasn’t a comprehensive one. I would like to test it in accordance with WCAG.

Blog content — Due to lack of time we couldn't  investigate the content-heavy areas of the site, such as the blog.

Long-term value evaluation — We were unable to study site usage over a long period of time.

ALWAYS ask for pronouns — in fact offer yours in your introduction to put participants at ease. Respect your people and their identity.

Pilot the study – We were able to work out the issues with our script and realized our initial equipment setup wasn’t really conducive to the study. ALSO very important to test the equipment before each interview.​

How are businesses/ ride share drivers affected by the website.


More behavioral design investigation, using persuasive design for benevolent change.


Deep dive into text heavy features as well as tools.

Clearly delineate the success/ fail criteria of a task.

Create an efficient note-taking/ data collection system.

Set a timeline but leave space for unforeseeable circumstances.