[Security] Update yarn packages to fix 19 vulnerabilities by updating nightwatch

We should update all vulnerable packages:

19 vulnerabilities found - Packages audited: 9037
Severity: 3 Low | 7 Moderate | 8 High | 1 Critical

Ok, the globals.js error from above can be fixed, now I get a different error:

Running:  Page objects test page

   Error: No selector property for selector object. Instead found properties: onlyOnError
       at Object.Page objects test page (/var/www/html/core/tests/Drupal/Nightwatch/Tests/exampleTest.js:25:8)

FAILED: 1 errors (13ms)
   Error: No selector property for selector object. Instead found properties: onlyOnError
       at Object.Page objects test page (/var/www/html/core/tests/Drupal/Nightwatch/Tests/exampleTest.js:25:8)

[Login Test] Test Suite
=======================
Running:  Test login


In Database.php line 361:
                                                             
  The specified database connection is not defined: default  
                                                             

user-login [--site-path SITE-PATH] [--] <uid>

   Error: Error while running "getCookies" command: Command failed: php ./scripts/test-site.php user-login 1 --site-path undefined
  
  In Database.php line 361:
                                                               
    The specified database connection is not defined: default  
                                                               
  
  user-login [--site-path SITE-PATH] [--] <uid>
  
  
   
   In Database.php line 361:
                                                                
     The specified database connection is not defined: default  
                                                                
   
   user-login [--site-path SITE-PATH] [--] <uid>
   
   
       at checkExecSyncError (child_process.js:607:13)
       at execSync (child_process.js:647:13)
       at Object.drupalUserIsLoggedIn.sessionExists (/var/www/html/core/tests/Drupal/Nightwatch/Commands/drupalLoginAsAdmin.js:19:22)
       at NightwatchAPI.getCookies.cookies (/var/www/html/core/tests/Drupal/Nightwatch/Commands/drupalUserIsLoggedIn.js:16:16)
       at <anonymous>
       at process._tickCallback (internal/process/next_tick.js:189:7)

FAILED: 1 errors (352ms)

So 2 problems: Page objects test page is broken and the database connection does not work for some reason.

Here is patch without the long yarn.lock changes, but should update it if you run yarn.install.

OK, so I was able to "fix" those 2.

The first problem is because of https://github.com/nightwatchjs/nightwatch/issues/1969 (I think), so I simplified the command parameter to just a bool and then it works.

The second problem is that whatever object we are storing some settings on is resetting our drupalDbPrefix and drupalSitePath variables. I could not find Nightwatch documentation how you pass around dynamic settings during test runs, so I just decided to slap them on as global variables. I guess that is not good, would love some ideas how this should be done properly.

With this patch we are down to yarn audit: 11 vulnerabilities found - Packages audited: 9023
Severity: 2 Low | 6 Moderate | 3 High

At least Nightwatch, chromedriver and dependencies are updated with this. The remaining problems are stylelint, eslint that we need to upgrade.

Also adding a patch without the lock file changes for easier review.

Yay, nightwatch tests are passing!

Here is a yarn update for the other vulnerable packages (eslint and dependencies), no other changes. Let's see if this runs eslint correctly and does not break.

So we still have the global JS settings nightwatch problem in this patch, otherwise we should be ready.

You can still review the nolock.txt patch from before, this is only yarn.lock changes.

"This is the only concerning change for me" thats also the same thing I notice here. It looks as if its changing a bit how those parameters are coming in. My guess is now this is never going to be false (onlyOnError) since its being called like this :

```.drupalLogAndEnd(false);``` but then its destructuring from "false" which is not an object so will always set it to the default value.

See my attached screenshot or just paste this into google chrome console :

let drupalLogAndEnd = ({onlyOnError = true}) => {
  console.log('The value is --- ', onlyOnError);
}
drupalLogAndEnd(false);

EDIT :

Sry I overlooked that in line 12 of drupalLogAndEnd.js the function signature did change. so yes this looks good now :+1

The reason I changed the function signature is https://github.com/nightwatchjs/nightwatch/issues/1969 , could not get nightwatch running otherwise.

More concerning to me is the use of global.drupalDbPrefix as global variable because I could not find out how to pass around configuration in Nightwatch. Any ideas on that?

Patch updated slightly (and with some added documentation from having to set this up again from scratch).

I'm going to look into the function signature change some more, as it'd be nice to avoid changing our API
some more info here: https://github.com/nightwatchjs/nightwatch/issues/2018

Oh nice, you made my global variable hack better, thanks a lot! Nightwatch tests passing on the testbot as well, yay!

@justafish
seems issue was fixed in https://github.com/nightwatchjs/nightwatch/commit/a17ff29488a154b3d77b80...

BUT we still have error. when i debug this place:

selector: /test-page
isPossibleElementSelector: true
selector: @body
isPossibleElementSelector: true
selector: @body
isPossibleElementSelector: true
selector: { onlyOnError: false }
isPossibleElementSelector: true
   Error: No selector property for selector object. Instead found properties: onlyOnError

So we need to check this https://github.com/nightwatchjs/nightwatch/commit/a17ff29488a154b3d77b80...

we need to have Utils.isObject({ onlyOnError: false }) return true.

gonna digg a bit more.

Sent PR in nightwatch.
https://github.com/nightwatchjs/nightwatch/pull/2156

So yeah, for now we can use strings or arrays.

Starting with Nightwatch 1.0 Selenium Standalone Server is no longer required.
Here is a patch for removing Selenium settings in nightwatch.conf.js and replace by webdriver.

https://github.com/nightwatchjs/nightwatch/pull/2156#issuecomment-515748578
> Will be merged in the next few days.

on nightwatch side everything merged. so this patch should be updated now. our api should be same as before update.

Nigtwatch patch merged but we still needs to wait until npm version will be updated https://www.npmjs.com/package/nightwatch
To test patch locally you may use nightwatch from github master branch: `yarn && yarn add https://github.com/nightwatchjs/nightwatch`

1) combined config updates from previous patches.
2) didn't add any api changes.
3) updated some packages to minor versions.
4) updated chromedriver to 75.1.0

Current result:

 yarn run test:nightwatch -> OK. 11  total assertions passed. (47.024s)
 yarn audit v1.17.3 -> 0 vulnerabilities found - Packages audited: 10089

fix in npm already.
https://github.com/nightwatchjs/nightwatch/releases/tag/v1.2.1
patch should be updated with latest nightwatch version.

Fixed. All fine now with `yarn audit` and all tests.

Nice, thank you!

• PHP tests are green on the testbot
• Coding standard errors on the testbot are only about ignored files, so should be fine
• Nightwatch test are failing on the testbot, see https://dispatcher.drupalci.org/job/drupal_patches/5049/consoleFull . Looks like you broke the chromedriver connection somehow, can you check?

Not sure how to run CI tests locally. i didn't have that error using local installation. So doing blind patch based on similar issue:
https://www.drupal.org/project/drupal/issues/3073342
https://github.com/nightwatchjs/nightwatch#chromedriver

Same error. seems i need to configure testbot locally

@mogtofu33i gonna remove changes added in #18 since they aren't exactly related to this issue. and make CI fallen, i think should be added follow-up issue about it.

Updated patch without chromedriver changes.
Let's see what drupal ci says. i found it totally unconfigurable locally :)

The drupalci_testbot chromedriver:production image is with Google Chrome 74.0.3729.157 and chromedriver 2.38 for now. Using Selenium as webdriver.

There is a context problem for the tests, If I am drupalci_testbot:

I have a docker image with chromedriver and chrome to be used for running JS tests, this is ok until perhaps nightwatch replace functional-js tests.
But then we rely on chromedriver/chrome version from this image and we cannot set server_path option in nightwatch to use chromedriver from package.json because chromedriver will fail from missing dependency libnss3.so which is related to the fact that chrome is not installed in the php image.

If I am a dev:

I just want to run locally my nightwatch tests following instructions in core/tests/REDAME.md.
But with the current config, I need to add a server_path option in nightwatch.conf.

My point is that the chromedriver version in package.json is not used by drupalci_testbot, so it can be "^75.1.0" and we can adpat nightwatch.conf for testbot and local tests.
In the same manner, the .env is not really used by drupalci_testbot because there is some forced values set in NightwatchJS.php.

To continue on a patch for this issue, as per #3073342: JavaScript tests don't work with Chromedriver 75 and higher we should still pass { "chromeOptions": { "w3c": false } } to prevent failure when the testbot update to Chrome 75+.

As Nightwatch doc recommend to go to webdriver I think we should switch the nightwatch.conf to use webdriver and local chromedriver, this is then easier for local tests.

Here is a patch that should pass on ci and work locally if running with --env local as noted in the patched core/tests/README.md

Super!

• Code Changes look good and the local NightwatchJS instructions make sense!
• PHP tests are green on the testbot
• Coding standard errors on the testbot are only about ignored files, so should be fine
• Nightwatch test are green on the testbot
• yarn audit finds 0 vulnerabilities, yay!

Given all that I think this is RTBC.

@klausi i found one wrong part here. we need to remove dependencies part from core/package.json seems was added by yarn.

@finnsky, here is a patch with dependencies removed and chromedriver 75+ for local tests.

@mogtofu33 i feel some changes should be in yarn.lock aswell

argh:) chromedriver

Ah sorry, the dependencies entry was a good point. Let's do this game again:

• Code Changes look good. Not sure why we updated stylelint-no-browser-hacks now, but should not matter.
• PHP tests are green on the testbot
• Coding standard errors on the testbot are only about ignored files, so should be fine
• Nightwatch test are green on the testbot
• yarn audit finds 0 vulnerabilities, yay!

Back to RTBC.

👍👍👍

This is failing prettier linting that we run as part of core commit hooks - fixable?

Checking changed files...

/Volumes/files/code/git/core/drupal/core/tests/Drupal/Nightwatch/Commands/drupalInstall.js
  28:72  error  Replace `⏎··········process.env.DRUPAL_TEST_BASE_URL⏎········` with `process.env.DRUPAL_TEST_BASE_URL`  prettier/prettier

✖ 1 problem (1 error, 0 warnings)
  1 error, 0 warnings potentially fixable with the `--fix` option.


/Volumes/files/code/git/core/drupal/core/tests/Drupal/Nightwatch/Commands/drupalLoginAsAdmin.js
  21:65  error  Replace `⏎··········this.globals.drupalSitePath⏎········` with `this.globals.drupalSitePath`  prettier/prettier

✖ 1 problem (1 error, 0 warnings)
  1 error, 0 warnings potentially fixable with the `--fix` option.


/Volumes/files/code/git/core/drupal/core/tests/Drupal/Nightwatch/globals.js
  41:69  error  Replace `⏎··········browser.currentTest.module⏎········` with `browser.currentTest.module`  prettier/prettier

✖ 1 problem (1 error, 0 warnings)
  1 error, 0 warnings potentially fixable with the `--fix` option.

Updated with prettier fix. Since it is rtbc`ed already will mark it back.

Committed 4936a17 and pushed to 8.8.x. Thanks!

This doesn't apply to 8.7.x

Added valid patch for 8.7.x

* Nightwatch tests are green.
* Compared the patches, code changes have been ported correctly.
* yarn audit is clean, no security issues remaining
* yarn.lock has very minor differences for uuid-3.3.2 vs. uuid.3.3.3 and table-5.4.6 vs table-5.4.5, but I think we don't really care.

Thank you!

Committed 5e87ef7 and pushed to 8.7.x. Thanks!

These sorts of dependency updates should go in the release notes. Thanks for writing the snippet!

Follow-up #3118741: [Security] Update yarn dependencies to fix security issues