QA sign-in

OverviewCopy Anchor Link

QA verifies your acceptance criteria on the deployed preview, in a real browser. In most apps, most pages are behind a login, and QA cannot guess how your login works. You give QA the sign-in steps in ship.yml.

If ship.yml has no sign-in, QA still checks every page that it can open while signed out. QA marks a criterion that needs a signed-in page as unverifiable, and gives the reason: it had no way to sign in. QA never tries a sign-in method of its own.

ConfigurationCopy Anchor Link

Add a qa.signIn block to the ship.yml at the root of your repository:

version: 1
qa:
  signIn:
    path: /login
    steps: >-
      Enter operator@riverbank.studio in the Email field and press
      "Send magic link". The demo signs you straight in.

QA opens the page and follows your steps. Before QA captures proof, it makes sure that it is signed in. QA does not try a different way.

What it's forCopy Anchor Link

Use it for a sign-in that needs no secret, for example:

• a demo login
• a magic-link stub that signs you in immediately
• a seeded test account with a fixture password

A seeded test accountCopy Anchor Link

Your preview can create its own database for each pull request (see Deployments). When that database is seeded from your fixtures, it can hold a test account with a fixture password. That password opens only the preview database, so it is safe to commit. Name the datastore in account.seededIn, and SHIP refuses the file if no datastore has that name:

version: 1
qa:
  signIn:
    path: /auth/signin
    account:
      seededIn: db
    steps: |
      Fill the email field with qa@preview.example.invalid.
      Fill the password field with the fixture password.
      Press the Sign in button.

Use a fixture password only for an account that exists in the seeded database and nowhere else.

When the sign-in does not workCopy Anchor Link

A new preview can need some time before its pages load. QA first waits for the sign-in page, up to readyTimeoutSec. Then QA uses the last answer that it got:

1. If the page loads, QA follows your steps.
2. If the preview does not answer, QA tests on a local copy of your change. The qaSurface block in ship.yml sets how SHIP builds and serves that copy.
3. If the page returns a server error from your app, QA reports it as a defect in the change. The criteria behind the login fail.

If the page loads but your steps do not sign QA in, QA marks the criteria behind the login as unverifiable. When ship.yml has a qaSurface block, QA then checks those criteria again on a local copy of your change. A pass there counts as a pass. The QA report and the pull request comment say that QA did not sign in to the deployed preview. To keep QA on the preview, set qaSurface.onPreviewSignInBlocked to unverifiable.

When QA cannot check a criterionCopy Anchor Link

Sometimes QA fails only because it could not check some criteria. A code change cannot fix that, so SHIP does not send the change back to the Builder. The mission stops at the QA checkpoint and waits for you. Check the criteria another way, then give one of these answers:

• Accept: QA counts as passed, and the mission continues.
• Reject: the mission ends as failed.
• Re-run QA: QA checks the same commit again, for example after you fix the preview.

You can answer in the console, with ship mission qa-checkpoint, or through the API. The mission waits until you answer.

Where it's read fromCopy Anchor Link

SHIP reads qa.signIn from your base branch, not from the pull request that QA tests. Because of this, a pull request cannot change how QA signs in to test it. A change to the block applies to missions after the change merges.