The Full-stack Testing Mindset

“I love contributing to and maintaining our end-to-end test suite” - Nobody, Ever

End-to-end tests have a poor reputation for being brittle, slow, and frustrating for both QA and engineering teams to manage.

TL;DR

If you’re pushed for time, here are the main takeaways:‍

• Adopt a Full-Stack Testing Mindset Unless you’re an API-only company, backend systems don’t operate in isolation. Pay special attention to the integration points between different backend services, especially when async operations and task queues are involved.

• The Importance of Simulating Real-World Conditions
  Include tests that simulate unpredictable real-world conditions, such as variable network latency, high traffic loads, and interactions with external services. These conditions can expose backend issues that are not apparent in a controlled test environment.

• Developers: Collaborate with and Educate QA Teams
  Work closely with QA automation engineers to understand the limitations and deficiencies in the current end-to-end testing codebase and guide them in upskilling and improving their ability to write comprehensive and robust tests.

• QA: Proactively Learn from Developers
  QA should take every opportunity to learn how data flows from the front-end to the backend and other architectural and infrastructure concerns so you can start to troubleshoot and test with the mindset of a developer.

• Consider Playwright for Boosting Speed and Productivity
  If you haven’t already, look into switching to Playwright, as that will likely offer the biggest bang for your buck for simplifying the test code, shrinking test times in CI, and reducing the flakiness of tests overall.

‍

My goal with this article is to surface the often overlooked role end-to-end testing can play in uncovering types of backend bugs that are difficult, if not impossible, to find with integration or unit tests alone and why the solution lies in both software engineers and QA adopting a full-stack testing mindset.

And don’t worry - this isn’t a thinly veiled attempt to position our product as the answer to all your end-to-end testing woes. I simply want to inspire you to write better code that provides a more robust customer experience by embracing the value E2E testing can offer.

Why developers should care about e2e tests

Most developers consider their job done once unit (and hopefully, integration tests) have been written. End-to-end tests are usually an afterthought for developers, if they’re thought of at all.

Customer success is not whether your backend tests pass and the APIs under test fulfil their contract - it’s whether an entire customer flow works as expected in a production setting. ‍

There are many real world examples to illustrate the importance of end-to-end tests to identify backend issues. Just recently, we would have shipped a broken settings feature without E2E tests.

The backend test suite passed, but the Auth API call was incorrectly restricted to admin users only, and because every developer has admin permissions, the issue was invisible to us.

Luckily, we had an end-to-end test fail as it used a real customer account with standard permissions, revealing the underlying issue. Without this test, the user flow would have been broken on release for our users to discover.

The organizational divide between engineering and QA

This is typically how organizations think of end-to-end testing:

Dividing test design and responsibility solely according to an org chart makes logical sense but doesn’t work well in practice.

While a poor-quality or brittle end-to-end test suite with constant failures frustrates everyone, it’s ultimately developers who pay the price as they will most likely be responsible for investigating each failure and determining the solution. It’s therefore in their best interests to ensure E2E tests provide maximum value with minimal false positives.‍

While I’m not suggesting developers save the day and take ownership of end-to-end testing, they need to play a leadership role to drive improvements in test quality by:‍

• Understanding the limitations of unit and integration tests for simulating real-world scenarios, mainly where external services are involved.

• Working closely with QA automation engineers to educate them about flows where async task processing or data synchronization operations may create a more nuanced happy path.

• Reviewing test cases to ensure best practices are being followed, ensuring test code is easily maintainable, readable and debuggable.‍

• Encouraging and helping QA learn more about the system architecture, infrastructure, and application code so they may learn how to think like a developer and write more sophisticated and comprehensive test cases.

‍

The goal is for QA and development teams to act as collaborative partners and take greater ownership of the quality and reliability of their end-to-end tests together. This helps foster a better team-oriented environment and puts the customer experience and code quality as the shared objective.

Ultimately, the best developers are the ones that don’t just focus on addressing the strict requirements of their user story or task, but care deeply about the quality and delivered customer value and don’t let organizational boundaries limit the scope of their testing.

The e2e testing landscape is transforming

There’s never been a better time to be optimistic about end-to-end testing improving, thanks to Microsoft’s open-source testing framework, Playwright, and automated testing fixes powered by AI.

Since Playwright’s release in 2020, it’s racked up more than 60k stars on GitHub and over 4.5 million installs per week from npm — and with good reason, as it’s the most modern and forward-thinking test framework. We’re betting our entire company on it.

It’s blazingly fast, tests multiple browsers in parallel, runs headed and headless, has flexible layout-based selectors such as near, below, and above, flakiness prevention features such as auto-wait, integrated VS Code debugging, and much more.

This means writing high quality and reliable E2E tests doesn’t require years of experience and weird timing hacks. And the fact it’s used and owned by Microsoft means that you can be sure it’s here to stay and will continue to be invested in for the long term.

Where AI fills the gaps

But Playwright alone doesn’t take all the pain away. You still have to identify, write, run, and maintain your Playwright E2E tests. ‍

At Octomind, we think that AI has the capacity to make that part much easier. Our LLM based agent is trained to have a semantic understanding of code at the UI layer, enabling them to infer relationships between related UI components and make intelligent suggested changes to fix failures caused by issues such as broken Playwright locators.

We want to remove the needless interruptions caused by failing UI tests that aren’t genuine regressions by automatically fixing them, e.g. when link text is updated or a button position changes. It’s a task AI is well suited to.

Summary

End-to-end testing isn’t just for testing the front-end and catching simple UI regressions and the testing pyramid is changing to reflect the value they can offer.

End-to-end tests are unique in that they can simulate real-world conditions such as network latency, high traffic loads, and interactions with external services. These conditions can expose backend issues unlikely to surface in controlled environments such as those where integration and unit tests are run.

By adopting a collaborative and comprehensive approach to end-to-end testing where developers support QA in developing a deeper understanding of the entire application stack, software teams can significantly improve the quality of their code and provide a more robust and reliable customer experience.

With Playwright and AI revolutionizing end-to-end testing, there’s never been a better time to adopt a ‘Full-Stack Testing Mindset’ to get the most out of your E2E test suite.

Daniel Draper
Lead Engineer at Octomind