Case Studies

Beta testing as a product launch campaign for a SaaS platform

beta tests

In this case study you will find answers to the following questions:

  • What is product beta testing?
  • Why is it worth conducting it?
  • How to prepare for a beta product launch?
Client:
Polish start-up developing a SaaS platform
Goal:
test SaaS platform reception among users, collect feedback, confront the product with the market
Collaboration period:
2021
Number of sprints:
5
Number of people in the project:
3

Project context:

In 2020, we conducted a product discovery process for a client developing a SaaS platform. At that time, we identified the problems and needs of users working with tools from the client’s domain. As a result, we developed guidelines for a platform that could solve these problems. To put it simply: we determined what kind of product that could be created using the client’s technology was desired by potential users. In the following months, the client’s team worked on a minimal version of the product that would deliver the intended value (MVP). After conducting all necessary tests, the tool entered the beta phase.  

What is product beta testing?

Product beta testing means releasing an early version of a tool to a group of users for a certain period. The users called early adopters are aware that what they get is not the final version of a product. In fact, they see this as a value – they get to be the first to test the tool. The purpose of beta testing is to gather feedback from users and make quick modifications to the product. Such modifications may include fixing bugs and introducing improvements, as well as changing the business model. This is another stage of product idea validation – this time directly on the market.

Beta tests can be open or closed. Open tests are accessible to everyone – of course, after registration. Closed tests mean that only a selected few can be accepted into the group of testers – for example, based on their scores or waiting time.

The tests in question had an open formula.

What were the goals of the beta test?


The beta tests conducted by the Project: People team had two main goals:

  • product verification – to determine if the product meets the assumptions and project objectives;
  • business verification – establishing whether the value proposition developed in the product discovery phase is sufficient to achieve Product-Market Fit;


The client wanted the beta test to be a form of an initial product launch. Therefore, we added to the list of goals some aspects related to marketing, for example:

 

  • acquiring and engaging users who could ultimately become customers of the platform;
  • building brand awareness.

 

The remaining objectives were:

  • collecting feedback on the tool from users,
  • finding as many bugs in the software as possible,
  • verification of marketing communication assumptions.

Preparation for beta tests

We started designing the process a few weeks before the official beta tests.

Beta tester journey map


We designed the beta tester path based on our benchmarks and knowledge concerning the target group. We did this to streamline the acquisition of early adopters willing to test innovations and to develop a plan for marketing communication and onboarding in the beta testing process. The beta tester journey map allowed us to pinpoint the stages “where something could go wrong” and a user could get discouraged from using our client’s SaaS platform. It helped us prevent foreseeable problems.

beta tester jo
Ścieżka beta testera
The beta tester’s path

 

The elements of the tester’s path we took into account:

  • stages in the process and actions taken by a tester,
  • touch points with the aforementioned SaaS tool,
  • factors that promote positive, negative and neutral reception,
  • stages at which certain groups of users are likely to resign from participation in beta testing,
  • recommended involvement of the tool creators at each stage (a kind of a brief service blueprint, in simple terms: information about what is happening on the part of the beta test organizers when a tester is taking a particular step, what actions are taken “behind the scenes”).

 

The value proposition of beta testing

What did we do to encourage early adopters to participate in beta testing? As the name suggests, early access means that the tool is not yet polished. How do we convince users if we can’t present the full list of impressive features right away?

We had to demonstrate the benefits of joining the tests and sharing feedback. We pointed out the following:

  • participation in the process of building the tool, direct contact with its creators,
  • joining an exclusive community of people focused on innovation and looking for new trends,
  • networking and gaining new knowledge (e.g. about beta tests, product development process, or the early stage of product creation),
  • discount on the paid version of the tool / free access to the tool for a specified period after the beta testing phase when other users will no longer be able to use the tool for free.

 

Automated communication with testers

Once we had designed the overall user path and value proposition, we could proceed with detailed plans. One of them concerned the flow of transactional emails the users should receive.

beta testy onboarding
Schemat onboardingu w beta testach i automatycznej komunikacji mailowej z uczestnikami
Flowchart of onboarding for beta testing and automated email communication with the testers

 

The plans and scenarios allowed us to break down all the tasks into detailed steps in the subsequent phase of actual testing. 

The final tasks that awaited us during the preparation phase for beta testing were:

    • designing a landing page for signing up for early access to the product, 
    • defining the details of the feedback solicitation (this process is described in more detail below),
  • testing of… beta tests, i. e. alpha tests.

 

Alpha tests

We can consider beta testing as a type of research, so it was necessary to test and validate it as a research tool. For this purpose, we organized alpha tests on a small scale. We invited 10 users to register for beta testing on a prepared landing page. This allowed us to make sure that everything was understandable and that there were no technical problems. Skipping this stage and not improving the onboarding experience could have made even those who were very curious about the new product resign from testing it.


The actual beta testing phase

The launch of an early version of the tool took place a few weeks after the alpha phase and the development of the test plan. During this time, the product team developed the product and conducted load tests. These tests determined the maximum number of users that could use the tool at the same time. Once the team was satisfied with the results, we could start beta testing.

Sprint 1: kick-off and tactical plan for the campaign phase


We begin each project with a kick-off workshop to:

  • discuss the goals of our actions and mutual expectations;
  • analyze the competencies and responsibilities of each team member;
  • establish a method of communication that suits everyone involved in the project (in this case, current communication and daily status reported on Slack and a weekly one-hour meeting);
  • make preliminary decisions related to the order of upcoming activities.

This beta testing was no exception. Although we had worked with this client before, the change in the scope of cooperation and new people in our project team were sufficient reasons to conduct the workshop again. 


Team Canvas

During the kick-off workshop, we used Miro to remotely collaborate on our Team Canvas.

 


Team Canvas


This meeting gave us a solid foundation for further work. In this case, it was a stage of detailed planning and division of tasks based on the previously elaborated beta tester’s journey. 

 

Diversification of channels

An important assumption that we made was the greatest possible diversification of channels and ways of recruiting beta testers. Why? We were at the beginning of the process and could not predict the response to the invitation to beta testing. On the one hand, we wanted as much traffic as possible, and on the other hand, we wanted each acquired beta tester to “infect” others with curiosity about the tool. Hence, in communication with the testers, we used the contacts established during the product discovery phase, the client’s channels, partners’ channels, social media, newsletters, and PR activities. The details of these activities are described in the following sections.

 

Tactical Plan

The tool that helped us manage all the tasks was the tactical plan. This is a document containing: 

  • individual tasks,
  • categories of tasks,
  • stage of beta testing to which the tasks apply,
  • deadlines,
  • people responsible for completing particular tasks,
  • status – whether the task has been completed or not.

 

 

Tactical plan

 

The beta testing tactical plan included tasks such as:

  • creation of necessary content (posts for SM profiles, content for transactional emails, content for ads, images for ads, etc.),
  • research work (e.g. creating a database of magazines potentially interested in publishing press releases or longer articles),
  • technical work (configuration of marketing automation system, preparation of advertising campaigns based on selected systems e.g. Facebook Ads or Google Ads),
  • analytical and automation work (related to compiling particular metrics, speeds up and facilitates ongoing campaign optimization),
  • other organizational work (e.g. working out how to aggregate and process feedback).

 

Prioritization of tasks

Having laid out the plan in such detail, there was nothing left to do but prioritize the activities. We divided the tasks into the following categories:

  • one-off tasks that should be performed before the campaign launch, necessary for achieving the project goals,
  • one-off tasks that can be performed during the campaign and that will bring additional value to the client,
  • ongoing tasks necessary to complete during the campaign.

 

When we had to choose between tasks, we used the effort vs. expected result matrix

It is worth noting that thanks to ongoing communication and exchange of information on progress, we did not waste time on unimportant tasks and the work was constantly moving forward.


Feedback management

At this stage, we also updated the assumptions about feedback management that were developed during the alpha testing phase.

We decided to collect feedback from beta testers in the following ways:

  • directly in the tool using the Usersnap plug-in, which transferred testers’ comments and screenshots to Asana where we categorized the feedback and centralized all information,
  • through an application form in Google Forms – access was sent to the testers in an email during the onboarding process,
  • in a closed Facebook group, where we actively animated early adopters, interacted with them, and build a community,
  • through private channels, if it was testers’ preferred way to share feedback.

 

It was also important to categorize and process feedback paths. In the end, we wanted the product team to receive filtered and validated information on what changes should be made to the tool. Hence the categorization of feedback into:

  • errors/bugs,
  • feedback concerning UX/UI,
  • suggestions related to product development and additional functionalities.

This categorization was the responsibility of Project: People while prioritization of feedback according to the product road map was the Product Owner’s task.

Sprint 2: Campaign launch

Hard hit and use of the freshness effect

You can only make a first impression once. It’s no different when you introduce your product to the users. Therefore, from the very beginning, our actions were intensive and we assumed reaching a large group of favorable recipients as soon as possible.

To achieve this, we started with mailing to people who participated in the process of product discovery (workshops and alpha tests). We also contacted users who had signed up for a beta testing notification on a landing page that was part of an experiment in the earlier product discovery phase.

We leveraged the client’s reach and took advantage of the power of recommendations from their partners. We used personal branding channels of people involved in creating the product. Thanks to active PR we also gained publicity in large specialist media.

However, what was most important at this stage was the involvement of beta testers who shared their impressions of using the tool on their social media channels and thus attracted other early adopters. 

Additionally, we acquired testers through paid Google Ads and Facebook Ads campaigns. 

 

Tests of advertising creations

We treated the paid ads as an additional activity. We wanted to gain first-hand information on the effectiveness of particular creations and the behavior of particular target groups.

In the case of Facebook, we conducted 4 parallel experiments related to target groups such as:

  • people interested in new technologies in IT/HR/consulting industries – the largest group;
  • early adopters, i.e. people identified by Facebook as willing to use technological innovations to whom Facebook displays its own new solutions in the form of A/B tests. We narrowed this group down to people interested in startup topics and CEOs;
  • lookalikes (a group of similar users defined based on shared interests and behavior patterns) of people who filled in the form on the landing page;
  • remarketing targeted at people who visited the beta testing sign-up landing page, but didn’t proceed to the tool from there.

It also allowed us to determine which of these groups were most likely to become beta testers and modify communication depending on the outcome.

 

Sprints 3-5: ongoing campaign management

 

We spent the following weeks on regular beta testing. Our activities included:

  • constant communication with testers – via social media and emailing,
  • regular feedback management,
  • creating content for publications in various channels,
  • experimenting with new forms of tester engagement.

Formally, this time could be equated to a marketing subscription service, which is one of the elements of our service portfolio. 


Engaging the community

We animated the members of the closed beta testers group in several ways. First of all, we regularly encouraged them to share their opinions by asking about their experiences with specific features in the tool. We never left a comment without a reply, and we engaged in long conversations with early adopters. We tried to show them how much we appreciate their contribution to the development of the tool.

We initiated discussions about their experiences related to the client’s domain – remote work. As beta testing was underway during the next wave of the Covid-19 pandemic, there was no shortage of conclusions. 

And since the community also forms around a common sense of humor, we shared gifs and memes that matched the brand’s communication style.

Regular contact with the development team

However, one of the most crucial communication elements during beta testing was only possible thanks to regular contact with the development team. They provided the information about subsequent releases of new features and improvements, and we passed them on to the beta testers. This way, we could show them their real impact on the tool. If we had omitted this element, our beta tests could have become worthless to them.

 

What were the conclusions and next steps after beta testing?

During the beta testing, we obtained feedback indicating that:

  • the value proposition promised by the tool is attractive to users, but they do not find it in the early version of the product,
  • users are willing to use the tool to test it, but do not return to it afterward,
  • not all elements of the tool are intuitive for users. 

However, we lacked an answer to the question: why is it so? Therefore, we decided to conduct task-based and moderated usability tests. The tests allowed us to get elaborate answers to this question directly from users. We will tell you all about it in the next case study.

Project in numbers

200

beta testers

4

feedback channels

500

feedback tickets received

Tools used in the project

  • Beta tester journey map
  • Team Canvas
  • Mailchimp
  • Usersnap
  • Asana
  • Miro

Case study isn't enough?

Would you like to learn about the whole process and how we could carry it out in your organization?

Let's talk

Project team, i.e. who was responsible for what

Author of the Case Study

Katarzyna Harmata Lean Marketing Manager & Strategist
In business for five years, during which she managed a marketing team of over a dozen people, created cycles of business events, co-created and managed the company. Currently, she specializes in Content Marketing and Personal Branding.

Katarzyna has been successfully combining marketing and business for years. Currently, she specializes in project management and content marketing. He has already managed a marketing team of over a dozen people, created a series of business events, co-created and managed the company.

She helps companies find the sense of their own self in business and communicate it to the outside world (marketing & business strategy).

She is involved in numerous social activities. She is a certified Business Coach and conducts workshops on strategy and marketing. Author of texts for industry-related portals, including Nowy Marketing, Marketing w Praktyce, E-commerce & Digital Marketing.

Other members of the team

Sylwia Paszyna Lean Marketing Consultant & Strategist
Language holds no secrets from her. Sylwia works with texts at different angles. She designs communication, edits, and proofreads the content. She cooperated, among others, with the Czarny publishing house. She graduated in editing at the Jagiellonian University and UX & Business Design at Tischner's European University.

She develops knowledge in the field of UX research to understand the needs expressed in language at their source. She designs marketing strategies, conducts research and analysis. He uses all her experience to create better communication.
Kamil Cupial Lean Marketing Consultant & Strategist
Kamil gained experience in NGOs, start-ups, and creative agencies, so he can easily understand seemingly contradictory perspectives. At Project: People he helps clients find their place on the market by creating and implementing marketing, business, and product strategies. He loves out-of-the-box solutions, experiments, and relentless testing.

In his professional life, he coordinated the work of both creative and technical teams. He sees the greatest value in a holistic view of the situation, organization, and individuals.

He spends his free time outdoors with a book in one hand and a cup of steaming yerba mate in the other. Moderate dissenter who claims the status quo does not exist. He knows that by learning new things, it is always possible to find different, possibly better solutions. He believes the world is not black and white and perceives many shades of gray.