Skip to main content

Quality Engineering

Dedication to quality​

As with any development project, the concern of releasing unexpected defects is always a possibility, and this concern grows as the project gets larger. With the introduction of a dedicated Quality Engineering process in 2023, Abyss has focused resources to improving quality in its development process. The goal is to eliminate and/or greatly minimize defects that Abyss users may encounter.

Test plan​

Using the Abyss documentation as a guide, test plans have been organized for each relevant category. The main goals of the test plan are:
  • High level metric data to calculate test coverage.
  • Aid in automation coverage creation
  • View automated percentage of the test plan
  • Checklist for manual regression testing, as needed.
To see the current test plan with the most up to date metrics, please see the following:

Automation testing​

Automation testing of Abyss components is a top priority. Currently, our automation tests consist of the following:

Web​

  • Visual Testing using Chromatic and Storybook
    • Uses pre-existing snapshots to compare latest snapshots, testing for matches/mismatches.
    chromatic dashboard showing test results
  • Functional Testing using Playwright
    • More interaction with the components than visual testing.
    • Perform functionality tests for various components such as click, drag, type, text compare, etc.
    • Framework Details.
      • Part of the Abyss monorepo.
      • Set up to run in the Abyss CI (Continuous Integration) pipeline.
      • Parallel test runs, which enhances test run speeds.
      • Multiple Browser test runs.
        • Chromium (Chrome, Edge engine).
        • Webkit (Safari engine).
        • Chrome
        • Utilizing Component Object Model architecture for clean coding, long term scalability, DRY methodology, and maintainability.
    • A11y Evinced SDK Integration.
      • This is a tool that has been integrated with the existing Abyss Playwright framework to test for and flag potential accessibility issues.
    • Report.
      • The report supplies relevant information for viewing both high and low-level details of the test run.
testing on github actions

Mobile​

  • Functional Testing using Appium
    • Mobile test automation framework for Abyss component native app.

To see current and up-to-date test automation coverage, please view the test plans.

Unit testing​

Manual testing​

Manual testing is an important process step, to ensure quality components are released. While automation testing can provide greater coverage of existing components already released, manual testing is primarily executed during the development cycle of new tasks/updates before a release goes to production. This helps reduce defects and instability before release. To better understand the current testing process, please visit: QE Review Task Process Flowchart

FAQ​

What is the Abyss QE process and how can I better understand it?
  • For manual testing: please see the QE Review Task Process Flowchart link under the Manual Testing section.
  • For automation testing, please review the details under the Automation Testing section.
Who decides what browsers/mobile devices are covered for QE testing?Does this mean I don’t have to do QE testing on my app?
  • No, this QE process does not replace a dedicated QE team/process on a project. The goal of the Abyss QE process is to rigorously test Abyss components and minimize defects for releases. While we test common use cases and some edge cases, our QE process may not consider all the ways that the Abyss components end up being used by teams, so the need for dedicated QE is still particularly important.
I have found an issue/bug/defect when using Abyss components. What should I do?
  • If you have found an issue/bug/defect that appears to be from an Abyss component, please follow these steps:
    1. Go to the Contact Us page.
    2. Scroll to the form near the bottom of the page.
    3. Select the Abyss Product you are reporting the issue for.
    4. Under Request Type, select the “Bug” option that best fits the issue.
    5. Fill out the rest of the form fields as accurately as possible.
    6. In the description, include as many details as possible, such as clear and concise steps to reproduce the issue.
    7. Issues are sometimes tied to specific versions, browser types, mobile devices, operating systems, etc. Including as much detailed information in your description as possible will help the Abyss team address the issue more effectively.
    8. Add any helpful attachments, such as screenshots, videos, etc.
    9. Before submitting the form, read through what you wrote and make sure all information is included. If anything else should be added, add it now.
    10. Click the Submit button and wait for the success message.
    11. Once you have successfully submitted the form, the Abyss Team will review the issue and plan next steps.
  • Overall, the current QE process has helped minimize defects getting released to production, which can be seen in the Abyss Rally defect ticket lists. As of September 2024, more than 500 Rally defect tickets have been entered, as well as hundreds of PR (Pull Request) comments as a result of QE reviews.
  • Since the current QE process started in 2023 and Abyss has been releasing components well before that, there may be existing defects that did not go through the current QE process. When any such defects are found during testing, they are added to the Abyss Rally backlog and prioritized for resolution.
What are the benefits of implementing automation testing for Abyss?
  • Broader test coverage.
  • Better quality and more rigorously tested components.
  • Faster regression testing time.
  • Ensuring any new updates don’t cause issues to existing code.
  • Defects found and reported faster.
  • Better integration with the Abyss CI pipeline.
What are some challenges with automation testing for Abyss?
  • Not necessarily ideal for work in progress tasks, or larger updates that have many changing parts and are not yet stable.
  • As automation coverage grows, maintenance concerns grow as well.
  • Ensuring it is stable, relevant and testing what is expected. For example, ensuring the tests have proper assertions that will alert the Abyss team of a relevant failure.
Table of Contents