@aichemy/testgold-cypress-plugin

This is the [TestGold](https://testgold.dev) plugin for [Cypress](https://www.cypress.io/). It enables auto-healing of failed selectors.

Usage no npm install needed!

<script type="module">
  import aichemyTestgoldCypressPlugin from 'https://cdn.skypack.dev/@aichemy/testgold-cypress-plugin';
</script>

README

This is the TestGold plugin for Cypress. It enables auto-healing of failed selectors.

Compatibility

  • Tested with Cypress 6.2.0 to 7.6.0 and Node 12+
  • Cypress 4.8.0 is the oldest Cypress version supported
  • ES6 and Node 14+ is highly recommended

How to use the TestGold Cypress plugin

  1. Install the TestGold Cypress plugin package with NPM or yarn: @aichemy/testgold-cypress-plugin.

  2. Add the required config object to your cypress.json file:

    {...,
  "testGold": {
    "serverTimeout": 180.0,
    "handleFailure": "exception",
    "filterEnabled": true,
    "filterDisplayed": true,
    "neighborMaxDist": 300.0,
    "fastHeal": false,
    "takeScreenshots": false,
    "imageComparison": false,
    "allSuggestEnabled": false
  }...
}

    See Plugin settings for what these settings do.

  3. Update the cypress/support/index.ts or cypress/support/index.js file in your tests' repo to include an import of the @aichemy/testgold-cypress-plugin, using the ES6 module import syntax:

    import '@aichemy/testgold-cypress-plugin';

    or CommonJS import syntax:

    require('@aichemy/testgold-cypress-plugin');

  4. Export the required environment variables before starting Cypress:

    export CYPRESS_TG_ENABLED=true
export CYPRESS_TG_TOKEN='your TestGold AIO token'

    Or place the corresponding items: TG_ENABLED and TG_TOKEN in your cypress.env.json file.

  5. Review how to ignore selectors below if required for your use case.

  6. Start Cypress from your repo and run your tests:

    ./node_modules/.bin/cypress open

Plugin settings

  • "serverTimeout" (default: 180.0) sets the maximum time in seconds to wait for a response from the TestGold API. It's also a good idea to set the value of "defaultCommandTimeout" in your cypress.json file to a similar value, so your Cypress tests don't time out while waiting for a selector to be healed by the TestGold service. ( NOTE: "defaultCommandTimeout" is specified in milliseconds).
  • "handleFailure" (default: exception) tells the plugin what to do when healing a selector fails because there is no previous training data. If this is set to the default exception, the plugin will not attempt to fix a broken selector that has no training data and fail with an exception so the test run fails normally. If set to suggest-xpaths, the plugin will request a best-effort heal from the TestGold API for a broken selector without any prior training data. If a healed selector is found in this case, the plugin will retry the element query and attempt to continue with your test run.
  • "filterEnabled" (default: true) tells the plugin how to treat disabled elements. If set to true, elements on the page that are disabled will be ignored when collecting training data for a selector. In most cases, this will speed up the test run. If set to false, disabled elements will still be considered when collecting training data.
  • "filterDisplayed" (default: true) tells the plugin how to treat invisible elements. If set to true, elements on the page that are hidden will be ignored when collecting training data for a selector. If you're sure your selectors will never try to find an invisible target element on the page, set this to true; this will speed up the test run. If set to false, invisible elements will still be considered when collecting training data.
  • "neighborMaxDist" (default: 300.0) tells the plugin the radius in pixels around each selector target element it should consider when collecting training data. Only elements within this radius will be processed for training if this option is set to a number greater than 0.0. Use this option when you're sure your target elements won't move too far from their current location when the page is revised (and their selectors need to be updated). Adjusting this parameter can greatly speed up the test run when our plugin is being used.
  • "fastHeal" (default: false) tells the plugin to check if the TestGold API has cached versions of training and healing data for a selector before processing it. If set to true, this check will be done for each selector. If your page has not changed between test runs, and cached versions of these data exist, the plugin will not need to collect training or healing data when processing the selector, thus speeding up your test run. Set this to false when the page being tested is volatile, and the TestGold API should always be kept up to date on the current version.
  • "takeScreenshots" (default: false) tells the plugin whether to take screenshots for every selector encountered during the Cypress test run. Setting this to true allows us to point out the candidate selected element locations on the page screenshot when we heal selectors. Disabling this option by setting it to false can greatly speed up your Cypress test run, since each Cypress screenshot invocation takes about 2 seconds.
  • "allSuggestEnabled" (default: false) tells the plugin whether it should collect information on all elements on the page whenever the URL changes during the test run. The plugin asks the TestGold API to generate CSS selectors for all of these elements. This can help augment your test cases by presenting robust selectors for all elements encountered during the test run. This adds about 2 seconds per page encounter to the test run time. Note that generating selectors for all elements encountered on each page encountered is a CPU intensive task, and is performed asynchronously by the TestGold API. In the worst case, all generated selectors may not appear in the TestGold dashboard UI for a few minutes after your test run finishes.
  • "imageComparison" (default: false) is left in for compatibility with our Selenium selector auto-healing solution; it doesn't do much at this stage, so leave it as specified in the instructions above.

Ignoring selectors

Selectors that you don't want healed (e.g. for negative test scenarios) may be placed in a .tgignore file. In this file, specify selectors to be ignored during processing, listing them one per line, for example:

# comment lines starting with '#' are ignored
# list selectors to ignore, one per line
*[id="office_phone1234"]
.mobilePhoneClass1234

To make the TestGold API aware of the ignored selectors in this file, you must run a pre-processing step before your test run. After the TestGold Cypress plugin is installed, it places a script entry for this purpose in your package.json that you can run:

npm run tgignore [add|remove|check] [path to .tgignore file]

If the first script argument is add, all selectors in the provided .tgignore file will be added to the Cypress ignored selectors list. The TestGold Cypress plugin will ignore these selectors when they are encountered in any test from now on.

If the first argument is remove, all selectors in the provided .tgignore file will be removed from the Cypress ignored selectors list. The TestGold Cypress plugin will process these selectors when they are encountered in any test from now on.

If the first argument is check, the selectors in the provided .tgignore file will be sent to the TestGold API to check if they are on the Cypress ignored selectors list. The script will print out each selector and its ignore status.

If no second argument is given, the script will look for a .tgignore file in the current working directory.

Examples

Run npm run tgignore add with a .tgignore file in the Cypress project directory:

{
  message: "Operation: 'add' for tgIgnore successful.",
  response: {
    userId: '96b755d1-8418-4aa2-b38f-049dda7bcd8e',
    ignoredSelectors: [ "a.btn.btn-primary.btn-lg[href='/login']" ],
    createdAt: '2021-09-20T19:37:11.193435+00:00',
    updatedAt: '2021-09-20T20:30:55.613033+00:00',
    isActive: true
  },
  status: 'success'
}

Run npm run tgignore remove with a .tgignore file in the Cypress project directory to remove all the selectors in the file from the ignored-selectors list:

{
  message: "Operation: 'remove' for tgIgnore successful.",
  response: {
    userId: '96b755d1-8418-4aa2-b38f-049dda7bcd8e',
    ignoredSelectors: [],
    createdAt: '2021-09-20T19:37:11.193435+00:00',
    updatedAt: '2021-09-20T20:31:49.128542+00:00',
    isActive: true
  },
  status: 'success'
}

Run npm run tgignore check with an updated .tgignore file in the Cypress project directory to see if all selectors in the file are registered as ignored selectors:

{
  message: "Operation: 'check' for tgIgnore successful.",
  response: {
    userId: '96b755d1-8418-4aa2-b38f-049dda7bcd8e',
    ignoredSelectors: [ "a.btn.btn-primary.btn-lg[href='/login']" ],
    createdAt: '2021-09-20T19:37:11.193435+00:00',
    updatedAt: '2021-09-20T20:34:26.122700+00:00',
    isActive: true,
    missingIgnoredSelectors: [ 'input[name="password"]' ]
  },
  status: 'success'
}

In this case, the missingIgnoredSelectors item contains selectors that are in the .tgignore file but aren't registered as ignored by the TestGold API. To add these missing selectors, run npm run tgignore add again:

{
  message: "Operation: 'add' for tgIgnore successful.",
  response: {
    userId: '96b755d1-8418-4aa2-b38f-049dda7bcd8e',
    ignoredSelectors: [ "a.btn.btn-primary.btn-lg[href='/login']", 'input[name="password"]' ],
    createdAt: '2021-09-20T19:37:11.193435+00:00',
    updatedAt: '2021-09-20T20:35:19.811131+00:00',
    isActive: true
  },
  status: 'success'
}

Changelog

1.7.5 - 2021-10-01

Changes

  • The plugin now uses fire-and-forget requests to the TestGold API in the case where a selector does not require healing. This should moderately improve performance in these cases.

1.7.4 - 2021-09-30

Changes

  • Removed an unneeded wait that was previously required for the DOM to settle. This should improve performance again in most cases.

1.7.3 - 2021-09-30

Changes

  • Optimized waiting strategy for resolving selector queries. This should improve performance in the case where healing selectors is required.

1.7.2 - 2021-09-28

Changes

  • More performance improvements for the case where selector healing is not required.

1.7.1 - 2021-09-28

Changes

  • Some more performance improvements if handleFailure is set to exception (default). See the updated notes above for this option.

1.7.0 - 2021-09-27

This version of the plugin adds 'all-selectors-suggest' mode. This collects all elements on a page whenever the URL changes during a test run and sends this info to the TestGold API. The API then generates suggested CSS selectors for all elements on each page encountered during the test run to help augment your test cases. Enable this mode by setting "allSuggestEnabled": true inside the "testGold" object in your cypress.json file.

New stuff

  • Added 'all-selectors-suggest' mode.

1.6.2 - 2021-09-21

Changes

  • Added back .tgignore file processing. See the updated instructions for ignoring selectors.
  • Removed redundant instructions for updating the cypress/plugins/index.js file.

1.6.1 - 2021-09-17

Fixes

  • Failing tests will now correctly throw errors, stop the Cypress Runner, and show their failed state as expected.

1.6.0 - 2021-09-16

This release features a stripped-down plugin, which removes .tgignore file processing and per-selector-encounter screenshots. These changes were required for Cypress aliasing, and higher-level commands like contains() to work properly. The selector suggestion and healing functionality will still work as expected.

NOTE: If you require .tgignore file processing and per-selector-encounter screenshot generation for your use-case, please continue with the releases in the 1.5.x series. You'll have to forgo the use of aliases, and higher-order Cypress commands like 'contains()'. We'll revisit this issue later to see if these features can be re-enabled safely.

Changes

  • Removed per-selector encounter screenshots
  • Removed .tgignore file processing

1.5.2 - 2021-09-15

Fixes

  • Optimized retries and wait strategy for a bit more speed.

1.5.1 - 2021-05-10

Fixes

  • Clean up for package contents.

1.5.0 - 2021-05-10

This version adds the ability to ignore selectors for heal processing by listing them in a .tgignore file. See Ignoring selectors for details.

Added

  • New .tgignore handling to exclude selectors from TG heal processing.

Changes

  • Internal API call changes for the next generation of the TG API.

Fixes

  • Bugfixes for the case of fast-heal screenshots not working as expected.

1.4.0 - 2021-04-12

This version changes the default behavior of the plugin in the interest of making the selector training and auto-healing processes more performant out of the box. See the changes below.

Changes

  • The options "filterEnabled" and "filterDisplayed" are now set to true by default if not explicitly specified in the "testGold" options object in cypress.json. This means the plugin will ignore any disabled or invisible elements while collecting training data.
  • The option"neighborMaxDist" is now set to 300.0 by default if not explicitly specified in the "testGold" options object in cypress.json. This means the plugin will now only search within a radius of 300 pixels around a target element while collecting training data.

1.3.3 - 2021-04-06

Changes

  • Added a required request param for the payload sent by startTestRequest() to start a new TG API session.

1.3.2 - 2021-03-31

Added

  • New "takeScreenshots" Cypress config item to control whether screenshots should be taken for every selector encountered during the test run. This is set to true by default.

1.3.1 - 2021-03-29

Changes

  • Removed export function, export class bits, changed to CommonJS exports for better ES5 compatibility.
  • The minimum version of Cypress where this plugin is expected to work is 4.8.0.

1.3.0 - 2021-03-29

This version of the TestGold Cypress plugin has fast-heal mode enabled by default.

Added

  • Added compatibility notes to NPM package README.
  • Added description of plugin options to NPM package README.

Changes

  • Swapped out the @aws-crypto/sha256-browser dependency to use jsSHA instead.
  • If the fastHeal option in the cypress.json testGold configuration object is set to true,the plugin will first check with the TG API if the selector has any previous training or healing results.
    • If the selector works as-is, has been seen by the TG API before, and the page under test hasn't changed, re-training the selector is now unnecessary.
    • If the selector being processed doesn't work, but has been seen by the TG API before, and the page under test hasn't changed, cached processing results from the TG API will be used to heal the selector immediately with no additional processing required.
    • Setting this option to true should significantly speed up Cypress tests in most cases when this plugin is enabled.

Fixes

  • Up to date screenshots are now available for fast-healed selectors.

1.2.2 - 2021-03-10

Fixes

  • Trigger a TG API test-run "finished" event on Cypress "failed" and "uncaught:exception" events to catch more test-run-ended scenarios.

1.2.1 - 2021-02-26

Fixes

  • Correct parsing of the values of TG_TOKEN and TG_ENABLED from the Cypress env object

1.2.0 - 2021-02-25

Changes

  • Screenshots are now taken in the "runner" mode as opposed to "viewport" in 1.1.x.

1.1.19 - 2021-02-24

Initial complete NPM release.

1.1.18 - 2021-02-24

Testing NPM release.