Feature: Cypress Frontend-testing (#712)

* adding cypress

* fix stagename

* sanitize before_script

* removing cypress examples

* test with background

* adding cypress stuff to .gitignore

* more cypress tests

* including bitcoind in tests

* bring the tests in order

* get tests to work

* naive try with docker

* docker executable no longer necessary

* install test_requirements

* fix ip-address

* uncomment not running tests

* fix broken ci

* fix ip-address

* reshuffle cypress tests, some docs

* ci-specific IP-address in test, we need to fix this

* fix dependency issue rimraf

* adding json-fie-creation in bitcoind and usage in cypress

* fix ci, forgot create-conn-json

* documentation

* echo Namespaces to fix tests not executed

* added testscript and moved to baseUrl

* Making the tests work and more robust

* improving tests and avoiding normal specter-folder

* reworking testscript, snapshots, decouple runs

* get ci back to work

* getting CI back to work

* get ci back to work

* get ci back to work

* get ci back to work

* get ci back to work

* get ci back to work

* get ci back to work

* tidy up

* fix uid-issue

* Update src/cryptoadvance/specter/templates/wallet/history/components/total_wallet_balances.jinja

Co-authored-by: benk10 <ben.kaufman10@gmail.com>

* Update src/cryptoadvance/specter/templates/wallet/wallets_overview.jinja

Co-authored-by: benk10 <ben.kaufman10@gmail.com>

* do tests again on cryptoadvance-project

* Fixed orphane btcd-process kill issue

* Update docs/cypress-testing.md

Co-authored-by: benk10 <ben.kaufman10@gmail.com>

* Update cypress/integration/spec_node_configured.js

Co-authored-by: benk10 <ben.kaufman10@gmail.com>

* improvements and bugfixes on the script

* script improvements and more proper process management

* consistency checks for running services and outdated snapshots

* fix build

* more logging

* kick

* debug on

* no longer reusing container

* test-cypredd: waiting on the container to come up

* pstree command not found (on gitlab)

* hopefully fix MacOS

* support MacOS for cypress

* use name instead of number for signal

* fix MacOS and a bit more docs

Co-authored-by: benk10 <ben.kaufman10@gmail.com>

.gitignore

@@ -22,3 +22,7 @@ pyinstaller/electron/node_modules
 pyinstaller/electron/dist
 .DS_Store
 .coverage
+cypress/videos
+cypress/screenshots
+node_modules
+btcd-conn.json

.gitlab-ci.yml

@@ -18,7 +18,9 @@ stages:
   - releasing
 
 before_script:
-  - docker info # Print out docker version for debugging
+  - docker info || echo "no docker-command found" # Print out docker version for debugging
+  - echo CI_PROJECT_NAMESPACE = $CI_PROJECT_NAMESPACE
+  - echo CI_PROJECT_ROOT_NAMESPACE = $CI_PROJECT_ROOT_NAMESPACE
   - python -V  # Print out python version for debugging
   - apt update
   - apt install -y libusb-1.0-0-dev libudev-dev # usb-support in hidapi
@@ -32,8 +34,9 @@ test:
  # relying on PRs and people who are working on gitlab-forks are working
  # on CI which probably want fast feedback on the releasing-jobs 
  # and therefore skip the test-job
- only:
-    - $CI_PROJECT_NAMESPACE =~ "cryptoadvance" 
+ # tem deactivated as it did not work as expected
+ #only:
+ #   - $CI_PROJECT_ROOT_NAMESPACE =~ "cryptoadvance" 
  script:
    - pip3 install -r requirements.txt
    - pip3 install -e .
@@ -42,6 +45,23 @@ test:
    # - python3 tests/conftest.py 
    - py.test --cov-report term --cov cryptoadvance --docker
 
+test-cypress:
+  image: registry.gitlab.com/k9ert/specter-desktop/cypress-python:20201111231032
+  stage: testing
+  script:
+    # start the server in the background
+    - pip3 install -e .
+    - pip3 install -r test_requirements.txt
+    - npm i
+    - ./utils/test-cypress.sh --docker --debug run
+    - docker ps || echo "probably no docker available anyway"
+  artifacts:
+    when: always
+    paths:
+      - cypress/videos/**/*.mp4
+      - cypress/screenshots/**/*.png
+    expire_in: 1 day
+
 release_pip:
   stage: releasing
   only: 

DEVELOPMENT.md

@@ -1,4 +1,4 @@
-## How to run
+## How to run the Application
 
 Install dependencies:
 
@@ -21,7 +21,7 @@ cd specter-desktop
 python3 -m cryptoadvance.specter server
 ```
 
-# Run the tests
+# Howto run the tests
 Run the tests (still very limited):
 
 ```sh
@@ -43,6 +43,8 @@ pytest tests/test_specter
 pytest tests/test_specter -k Manager 
 ```
 
+Check the cypress-section on how to run cypress-frontend-tests.
+
 # Code-Style
 
 Before your create a PR, make sure to [blackify](https://github.com/psf/black) all your changes. In order to automate that,
@@ -52,6 +54,8 @@ pre-commit install
 ```
 
 # Developing on tests
+We use pytest and for frontend-testing the amazing [cypress.io](https://www.cypress.io/).
+
 ## bitcoin-specific stuff
 
 There are some things worth taking a note here, especially if you rely on a specific state on the blockchain for your tests. Bitcoind is started only once for all the tests. If you run 
@@ -62,11 +66,28 @@ it each time it's starting with the genesis-block. This has some implications:
 * This also means that it makes a huge difference whether you run a test standalone or together with all other tests
 * Depending on whether you do one or the other, you cannot rely on transactionIDs. So if you run a test standalone twice, you can assert txids but you can't any longer when you run all the tests
 
+## Cypress UI-testing
+Cypress is just awesome. It's quite easy to create Frontend-tests and it's even recording all tests and you can immediately see how it went. So each test-run, the tests are kept for one day (see the ["artifacts-section"](https://github.com/k9ert/specter-desktop/blob/cypress/.gitlab-ci.yml#L53-L58)) and you can watch them by browsing the artifacts on any gitlab-job-page (right-hand-side marked with "Job artifacts").
+
+Executing the tests is done via `./utils/test-cypress.sh`:
+```
+# make sure you have npm on the path
+
+# run the tests
+./utils/test-cypress.sh run
+# open the cypress application (to develop/debug/run tests interactively)
+./utils/test-cypress.sh open
+```
+The test_specifications which get executed are specified in cypress.json which looks something like this:
+
+More details on cypress-testing can be found in [cypress-testing.md](docs/cypress-testing.md).
+
+
 # Flask specific stuff
 
 Other than Django, Flask is not opionoated at all. You can do all sorts of things and it's quite difficult to judge whether you're doing it right.
 
-One strange thing which we're doing to get the tests working is forcing the reload of the controller-code (if necessary) [here](https://github.com/cryptoadvance/specter-desktop/blob/master/src/cryptoadvance/specter/server.py#L83-L90).
+One strange thing which we're doing to get the tests working is forcing the reload of the controller-code (if necessary) [here](https://github.com/cryptoadvance/specter-desktop/blob/master/src/cryptoadvance/specter/server.py#L88-L93).
 
 The if-clause might be quite brittle which would result in very strange 404 in test_controller.
 Check the [archblog](./docs/archblog.md) for a better explanation.

cypress.json

@@ -0,0 +1,8 @@
+{
+    "testFiles": [
+      "spec_empty_specter_home.js",
+      "spec_node_configured.js",
+      "spec_existing_history.js"
+    ],
+    "baseUrl": "http://localhost:25444"
+}

cypress/fixtures/.gitignore

@@ -0,0 +1 @@
+*.tar.gz

cypress/integration/spec_empty_specter_home.js

@@ -0,0 +1,66 @@
+
+describe('Completely empty specter-home', () => {
+  beforeEach(() => {
+    cy.task("clear:specter-home")
+  })
+  it('Visits specter and clicks around', () => {
+    cy.viewport(1200,660)
+    cy.visit('/')
+    cy.contains('Welcome to Specter Desktop')
+    cy.get('[href="/settings/"] > img').click()
+    cy.contains('Bitcoin Core settings - Specter Desktop custom')
+    cy.get('[href="/settings/general"]').click()
+    cy.contains('General settings - Specter Desktop custom')
+    cy.get('[href="/settings/auth"]').click()
+    cy.contains('Authentication settings - Specter Desktop custom')
+    cy.get('.right').click()
+    cy.contains('HWI Bridge settings - Specter Desktop custom')
+  })
+
+  it('Creates a device in Specter', () => {
+    cy.viewport(1200,660)
+    cy.visit('/')
+    cy.addDevice("Some Device")
+  })
+
+  it('Configures the node in Specter', () => {
+    cy.viewport(1200,660)
+    cy.visit('/')
+    cy.get('[href="/settings/"] > img').click()
+    cy.get('#datadir-container').then(($datadir) => {
+      cy.log($datadir)
+      if (!Cypress.dom.isVisible($datadir)) {
+        cy.get('.slider').click()
+      }
+    })
+    cy.get('.slider').click()
+    cy.get('#username').clear()
+    cy.get('#username').type("bitcoin")
+    cy.get('#password').clear()
+    cy.get('#password').type("wrongPassword") // wrong Password
+    cy.get('#host').clear()
+    // This is hopefully correct for some longer time. If the connection fails, check the 
+    // output of python3 -m cryptoadvance.specter bitcoind (in the CI-output !!) for a better ip-address.
+    // AUtomating that is probably simply not worth it.
+    cy.readFile('btcd-conn.json').then((conn) => {
+      cy.get('#host').type("http://"+conn["host"])
+    })
+    cy.get('#port').clear()
+    cy.get('#port').type("18443")
+    cy.get('[value="test"]').click()
+    cy.get(':nth-child(2) > button > div').should('have.css', 'color', 'rgb(0, 128, 0)') // connectable: green
+    cy.get(':nth-child(5) > button > div').should('have.css', 'color', 'rgb(255, 0, 0)') // Credentials: red
+    cy.get('#password').clear()
+    cy.get('#password').type("secret")
+    cy.get('[value="test"]').click()
+    cy.get(':nth-child(2) > button > div').should('have.css', 'color', 'rgb(0, 128, 0)') // connectable: green
+    cy.get(':nth-child(5) > button > div').should('have.css', 'color', 'rgb(0, 128, 0)') // Credentials: green
+    cy.get(':nth-child(8) > button > div').should('have.css', 'color', 'rgb(0, 128, 0)') // Version green
+    cy.get(':nth-child(11) > button > div').should('have.css', 'color', 'rgb(0, 128, 0)') // Walletsenabled green
+    cy.get('[value="save"]').click()
+
+  })
+
+})
+
+

cypress/integration/spec_existing_history.js

@@ -0,0 +1,8 @@
+describe('Rescanning existing wallets', () => {
+    it('Creates a wallet on specter', () => {
+        cy.viewport(1200,660)
+        cy.visit('/')
+        // empty so far
+    })
+
+})

cypress/integration/spec_node_configured.js

@@ -0,0 +1,33 @@
+describe('Node Configured', () => {
+    it('Creates a wallet on specter', () => {
+        cy.viewport(1200,660)
+        cy.visit('/')
+        cy.addDevice("Testdevice Ghost")
+        cy.get('body').then(($body) => {
+            if ($body.text().includes('Testwallet Ghost')) {
+                cy.get('#wallets_list > .item > svg').click()
+                cy.get(':nth-child(6) > .right').click()
+                cy.get('#advanced_settings_tab_btn').click()
+                cy.get('.card > :nth-child(9) > .btn').click()
+            }
+        })
+        cy.get('#btn_new_wallet').click()
+        cy.get('[href="./simple/"]').click()
+        cy.get('#testdevice_ghost').click()
+        cy.get('#keysform > :nth-child(2) > .inline').type("Testwallet Ghost")
+        cy.get('#keysform > .centered').click()
+        cy.get('body').contains("New wallet was created successfully!")
+        // Download PDF
+        // unfortunately this results in weird effects in cypress run
+        //cy.get('#pdf-wallet-download > img').click()
+        cy.task("node:mine") 
+        cy.get('#btn_continue').click()
+        cy.get('#btn_transactions').click()
+        cy.get('#fullbalance_amount')
+        .should(($div) => {
+            const n = parseFloat($div.text())
+            expect(n).to.be.gt(0).and.be.lte(50)
+        })
+
+    })
+})

cypress/plugins/index.js

@@ -0,0 +1,49 @@
+/// <reference types="cypress" />
+// ***********************************************************
+// This example plugins/index.js can be used to load plugins
+//
+// You can change the location of this file or turn off loading
+// the plugins file with the 'pluginsFile' configuration option.
+//
+// You can read more here:
+// https://on.cypress.io/plugins-guide
+// ***********************************************************
+
+// This function is called when a project is opened or re-opened (e.g. due to
+// the project's config changing)
+
+/**
+ * @type {Cypress.PluginConfig}
+ */
+const fs = require('fs');
+
+module.exports = (on, config) => {
+  // `config` is the resolved Cypress config
+  const conn_file = fs.readFileSync('btcd-conn.json');
+  const conn = JSON.parse(conn_file);
+  on('task', {
+    'clear:specter-home': () => {
+      console.log('Removing and recreating Specter-data-folder %s', conn["specter_data_folder"])
+      const specter_home=conn["specter_data_folder"];
+      var rimraf = require("rimraf");
+      rimraf.sync(specter_home);
+      fs.mkdirSync(specter_home);
+      fs.mkdirSync(specter_home+"/devices");
+      fs.mkdirSync(specter_home+"/wallets");
+      return null
+    }
+  })
+
+  on('task', {
+    'node:mine': () => {
+      // sending the bitcoind-process a signal SIGUSR1 (10) will cause mining towards all specter-wallets
+      // See the signal-handler in bitcoind
+      console.log('Sending SIGUSR1 to '+conn["pid"])
+      process.kill(parseInt(conn["pid"], 10), 'SIGUSR1');
+      return null
+    }
+  })
+
+
+
+}

cypress/support/commands.js

@@ -0,0 +1,51 @@
+// ***********************************************
+// This example commands.js shows you how to
+// create various custom commands and overwrite
+// existing commands.
+//
+// For more comprehensive examples of custom
+// commands please read more here:
+// https://on.cypress.io/custom-commands
+// ***********************************************
+//
+//
+// -- This is a parent command --
+// Cypress.Commands.add("login", (email, password) => { ... })
+//
+//
+// -- This is a child command --
+// Cypress.Commands.add("drag", { prevSubject: 'element'}, (subject, options) => { ... })
+//
+//
+// -- This is a dual command --
+// Cypress.Commands.add("dismiss", { prevSubject: 'optional'}, (subject, options) => { ... })
+//
+//
+// -- This will overwrite an existing command --
+// Cypress.Commands.overwrite("visit", (originalFn, url, options) => { ... })
+
+Cypress.Commands.add("addDevice", (name) => { 
+    cy.get('body').then(($body) => {
+        if ($body.text().includes(name)) {
+          cy.get('#devices_list > .item > div').click()
+          cy.get('#forget_device').click()
+        } 
+        cy.get('#side-content').click()
+        cy.get('#btn_new_device').click()
+        // Creating a Device
+        cy.contains('Select Your Device Type')
+        cy.get('#trezor_device_card')
+        cy.get('#step1 > [type="text"]').type("specter")
+        cy.contains('Select Your Device Type')
+        cy.get('#trezor_device_card').should('not.have.class', 'disabled')
+        cy.get('#specter_device_card').click()
+        cy.get('h2 > input').type(name)
+        cy.get('#wizard-previous').click()
+        cy.get('#step1 > .note').click()
+        cy.get(':nth-child(1) > input').type(name)
+        cy.get('#device_type').select("Specter-DIY")
+        cy.get('#txt').type("[8c24a510/84h/1h/0h]vpub5Y24kG7ZrCFRkRnHia2sdnt5N7MmsrNry1jMrP8XptMEcZZqkjQA6bc1f52RGiEoJmdy1Vk9Qck9tAL1ohKvuq3oFXe3ADVse6UiTHzuyKx")
+        cy.get('#cold_device > [type="submit"]').click()
+        cy.get('#devices_list > .item > div').contains(name)
+      })    
+})

cypress/support/index.js

@@ -0,0 +1,20 @@
+// ***********************************************************
+// This example support/index.js is processed and
+// loaded automatically before your test files.
+//
+// This is a great place to put global configuration and
+// behavior that modifies Cypress.
+//
+// You can change the location of this file or turn off
+// automatically serving support files with the
+// 'supportFile' configuration option.
+//
+// You can read more here:
+// https://on.cypress.io/configuration
+// ***********************************************************
+
+// Import commands.js using ES2015 syntax:
+import './commands'
+
+// Alternatively you can use CommonJS syntax:
+// require('./commands')

docs/continuous-integration.md

@@ -1,5 +1,6 @@
 # introduction
-specter-desktop is using gitlab and Travis-CI for continuous integration purposes. Both have advantages and disadvantages so ... let's use both!
+Specter-Desktop is using gitlab, Travis-CI and Github-Actions for continuous integration purposes but Github-actions only for Blackify so far. It might be more effort using more than one CI-approach but it makes us also more resilient. 
+Gitlab and Travis-CI have both advantages and disadvantages so ... let's use both!
 Gitlab:
 * is completely open Source for server- and clients
 * the gitlab-runner can run docker and is itself running on docker
@@ -38,10 +39,10 @@ Travis-CI setup is very straightforward. As we're using the build-cache, the bit
 
 # Releasing
 
-## What gets releases
+## What gets released
 
-We're mostly releasing automatically. Currently the following artifacts are releases:
-* specterd (aemon) is a binary for kicking off the specter-desktop service on the commandline. We have binaries for windows, linux and macos
+We're mostly releasing automatically. Currently the following artifacts are releasesd
+* specterd (daemon) is a binary for kicking off the specter-desktop service on the commandline. We have binaries for windows, linux and macos
 * We have an Electron-App which we're also releasing for windows, linux and macos. Unfortunately the macOS build is not yet automated
 * We release a pip-package
 * We release docker-images, these are also not yet automated

docs/cypress-testing.md

@@ -0,0 +1,71 @@
+# Cypress Tests
+
+The UI is tested via [cypress](https://www.cypress.io/) which is built with node.js. The tests are specified in `cypress.json`. One of the challenges here is management of state (specter-folder and state of regtest). cypress is [discouraging](https://docs.cypress.io/guides/references/best-practices.html#Web-Servers) to start/stop the webserver or its prerequisites as part of executing the tests. Therefore we need to manage that ourself. This has been tested in Linux and MacOS. Windows is not supported.
+
+ So the tests in general are designed to run in a strict sequence specified in cypress.json. So later tests might need the state created in former tests.
+
+This is very different than on unit-tests and, with an increasing amount of tests, this might be cumbersome in order to only execute a specific test or develop on a specific one.
+Therefore there is the possibility to snapshot and restore the state of a specific test-run. Also, in parallel to running the pytests, it would be beneficial to not use the default-folders (`~/.specter` and ) for the state but completely separate that. This way, you can develop live on the application and running the tests in parallel without interference. Doing that is possible via the test-cypress.sh:
+```
+$ ./utils/test-cypress.sh --help
+Usage: ./utils/test-cypress.sh [generic-options] <subcommand> [options]"
+Doing stuff with cypress-tests according to <subcommand>"
+
+Subcommands:"
+    open [spec-file]    will open the cypress app."
+    run  [spec-file]    will run the tests."
+                        open and run take a spec-file optionally. If you add a spec-file, "
+                        then automatically the corresponding snapshot is untarred before and," 
+                        in the case of run, only the spec-file and all subsequent spec_files "
+                        are executed."
+    snapshot <spec_file>  will create a snapshot of the spec-file. It will create a tarball"
+                        of the btc-dir and the specter-dir and store that file in the "
+                        ./cypress/fixtures directory"
+
+generic-options:
+    --debug             Run as much stuff in debug as we can
+    --docker            Run bitcoind in docker instead of directly
+$
+```
+Apart from dealing with snapshots, it'll also take care to use different folders (`~/.specter-cypress` and `/tmp/specter_cypress_btc_regtest_plain_datadir` and a different post for specter (`25444` instead of `25441`). However, bitcoind-port is still the same. So currently you can't run bitcoind-regtest for developemnt and do (reliable) tests with cypress. You will get failing tests in that case. Will get fixed in the future.
+
+Let's look at some typical use-cases in order to understand how to use this script. In any case, you'll need nodejs installed, so do something like:
+```
+wget https://nodejs.org/dist/v15.3.0/node-v15.3.0-linux-x64.tar.xz
+sudo tar -xJf node-v15.3.0-linux-x64.tar.xz -C /opt
+sudo ln -s /opt/node-v15.3.0-linux-x64 /opt/node
+export PATH=$PATH:/opt/node/bin # maybe make that permanent
+npm ci
+```
+
+## Run tests
+`./utils/test-cypress.sh run` will simply run all the tests. If there are any issues, you'll find screenshots of failed states and mp4-videos in `cypress/screenshots` and `cypress/videos`. In the case of running in gitlab, thanks to the ["artifacts-section"](https://github.com/k9ert/specter-desktop/blob/cypress/.gitlab-ci.yml#L53-L58), you can watch them by browsing the artifacts on any gitlab-job-page (right-hand-side marked with "Job artifacts"). This will hopefully also be available on travis.
+
+## Interactively run tests
+`./utils/test-cypress.sh open` will open the cypress application (after spinning up bitcoind-regtest and specter). Here, you can choose the test-suite you want to execute. As the tests rely on the state of former-tests and we have a clean state, now, you have to run them in sequence no matter which test-file you're interested in. The tests are (and should be) written in a way that is resilient to this but that's especially difficult on the btc-regtest side of the story.
+
+You can run the tests more than once but the regtest state will simply continue with its history. Especially if you want to focus on higher level tests (which are running later in the sequence), it's anoying to run all the tests before that.
+
+## Create a snapshot
+Let's say you want to focus on the currently last test-file `spec_existing_history`. You can create a snapshot of the expected state at the start of the test via:
+```
+$ ./utils/test-cypress.sh snapshot # will output possible arguments
+we need one of these arguments:
+spec_empty_specter_home.js
+spec_node_configured.js
+spec_existing_history.js
+$ ./utils/test-cypress.sh snapshot spec_existing_history.js
+```
+
+Now you can run (or open) specifically this test-file via:
+```
+$ ./utils/test-cypress.sh run spec_existing_history.js
+```
+This will restore the snapshot created above and run this test (and all subsequent ones). Opening the cypress app works the same although there you're responsible to be aware that the state is fitting to the spec-file you want to execute (just like in the case of empty state above)
+
+## Develop on tests
+The spec-files mainly select some element on a webpage and then act on them. Mainly `.click()` and `.type("some Text")`. So it's quite helpfull to have unique IDs for all the elements which we want to use in the tests. The cypress-app has a very convenient way of selecting. Make sure you don't miss that.
+
+For specific things there are `cy.tasks` which can be implemented. Two of the already existing tasks:
+* purging the specter-folder is possible via `cy.task("clear:specter-home")`
+* Mining some coins to each of the wallets defined in the specter-folder is possible via `cy.task("node:mine")`. Depending on the height of the blockchain, you might get very different results. The coins are immediately spendable.

package.json

@@ -0,0 +1,29 @@
+{
+  "name": "specter-desktop-cypress-testing",
+  "version": "0.0.1",
+  "description": "node-module to test specter-desktop with cypress",
+  "main": "index.js",
+  "directories": {
+    "doc": "docs",
+    "test": "tests"
+  },
+  "dependencies": {
+    "cypress": "^5.6.0",
+    "rimraf": "^3.0.2",
+    "wait-on": "^5.2.0"
+  },
+  "devDependencies": {},
+  "scripts": {
+    "test": "echo \"Error: no test specified\" && exit 1"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/k9ert/specter-desktop.git"
+  },
+  "author": "",
+  "license": "ISC",
+  "bugs": {
+    "url": "https://github.com/k9ert/specter-desktop/issues"
+  },
+  "homepage": "https://github.com/k9ert/specter-desktop#readme"
+}

src/cryptoadvance/specter/bitcoind.py

@@ -4,6 +4,8 @@
 import atexit
 import logging
 import os
+import signal
+import psutil
 import shutil
 import subprocess
 import tempfile
@@ -215,7 +217,7 @@ class BitcoindPlainController(BitcoindController):
 
     def _start_bitcoind(self, cleanup_at_exit=True, cleanup_hard=False, datadir=None):
         if datadir == None:
-            datadir = tempfile.mkdtemp(prefix="bitcoind_plain_datadir_")
+            datadir = tempfile.mkdtemp(prefix="specter_btc_regtest_plain_datadir_")
         bitcoind_cmd = self.construct_bitcoind_cmd(
             self.rpcconn,
             run_docker=False,
@@ -229,20 +231,35 @@ class BitcoindPlainController(BitcoindController):
             "Running bitcoind-process with pid {}".format(self.bitcoind_proc.pid)
         )
 
-        def cleanup_bitcoind():
+        def cleanup_bitcoind(*args):
+            timeout = 50  # in secs
             if cleanup_hard:
                 self.bitcoind_proc.kill()  # might be usefull for e.g. testing. We can't wait for so long
-                logger.info("Killed bitcoind with pid {self.bitcoind_proc.pid}")
+                logger.info(
+                    f"Killed bitcoind with pid {self.bitcoind_proc.pid}, Removing {datadir}"
+                )
+                shutil.rmtree(datadir)
             else:
                 self.bitcoind_proc.terminate()  # might take a bit longer than kill but it'll preserve block-height
                 logger.info(
-                    f"Terminated bitcoind with pid {self.bitcoind_proc.pid}, waiting for termination ..."
+                    f"Terminated bitcoind with pid {self.bitcoind_proc.pid}, waiting for termination (timeout {timeout} secs)..."
                 )
-                self.bitcoind_proc.wait()
+                # self.bitcoind_proc.wait() # doesn't have a timeout
+                procs = psutil.Process().children()
+                for p in procs:
+                    p.terminate()
+                _, alive = psutil.wait_procs(procs, timeout=timeout)
+                for p in alive:
+                    logger.info("bitcoind did not terminated in time, killing!")
+                    p.kill()
 
         if cleanup_at_exit:
-            logger.debug("REGISTERING EXIT FUNCTIONS")
-            atexit.register(cleanup_bitcoind)
+            logger.debug("Register function cleanup_bitcoind for SIGINT and SIGTERM")
+            # atexit.register(cleanup_bitcoind)
+            # This is for CTRL-C --> SIGINT
+            signal.signal(signal.SIGINT, cleanup_bitcoind)
+            # This is for kill $pid --> SIGTERM
+            signal.signal(signal.SIGTERM, cleanup_bitcoind)
 
     def stop_bitcoind(self):
         # not necessary as the cleanup_bitcoind() will do it automatically!
@@ -265,13 +282,13 @@ class BitcoindDockerController(BitcoindController):
     def __init__(self, rpcport=18443, docker_tag="latest"):
         self.btcd_container = None
         super().__init__(rpcport=rpcport)
-        self.docker_exec = which("docker")
         self.docker_tag = docker_tag
-        if self.docker_exec == None:
-            raise ("Docker not existing!")
+
         if self.detect_bitcoind_container(rpcport) != None:
-            rpcconn, self.btcd_container = self.detect_bitcoind_container(rpcport)
-            self.rpcconn = rpcconn
+            rpcconn, btcd_container = self.detect_bitcoind_container(rpcport)
+            logger.debug("Detected old container ... deleting it")
+            btcd_container.stop()
+            btcd_container.remove()
 
     def _start_bitcoind(self, cleanup_at_exit, cleanup_hard=False, datadir=None):
         if datadir != None:
@@ -297,13 +314,20 @@ class BitcoindDockerController(BitcoindController):
             detach=True,
         )
 
-        def cleanup_docker_bitcoind():
+        def cleanup_docker_bitcoind(*args):
             logger.info("Cleaning up bitcoind-docker-container")
             self.btcd_container.stop()
             self.btcd_container.remove()
 
         if cleanup_at_exit:
-            atexit.register(cleanup_docker_bitcoind)
+            logger.debug(
+                "Register function cleanup_docker_bitcoind for SIGINT and SIGTERM"
+            )
+            # This is for CTRL-C --> SIGINT
+            signal.signal(signal.SIGINT, cleanup_docker_bitcoind)
+            # This is for kill $pid --> SIGTERM
+            signal.signal(signal.SIGTERM, cleanup_docker_bitcoind)
+
         logger.debug(
             "Waiting for container {} to come up".format(self.btcd_container.id)
         )
@@ -325,6 +349,7 @@ class BitcoindDockerController(BitcoindController):
                 if container == self.btcd_container:
                     self.btcd_container.stop()
                     logger.info("Stopped btcd_container {}".format(self.btcd_container))
+                    self.btcd_container.remove()
                     return
         raise Exception("Ambigious Container running")
 

src/cryptoadvance/specter/cli/__init__.py

@@ -12,7 +12,7 @@ logger = logging.getLogger(__name__)
 def entry_point(config_home, debug=False):
     # ctx.obj = Repo(config_home, debug)
     if debug:
-        logging.basicConfig(level=logging.INFO)
+        logging.basicConfig(level=logging.DEBUG)
         logging.getLogger("cryptoadvance").setLevel(logging.DEBUG)
     logger
 

src/cryptoadvance/specter/cli/cli_bitcoind.py

@@ -41,7 +41,6 @@ class Echo:
 )
 @click.option(
     "--data-dir",
-    default="/tmp/specter_btcd_regtest_plain_datadir",
     help="specify a (maybe not yet existing) datadir. Works only in --nodocker (Default:/tmp/bitcoind_plain_datadir) ",
 )
 @click.option(
@@ -115,6 +114,9 @@ def bitcoind(
         echo("$ python3 -m cryptoadvance-specter --debug bitcoind")
         exit(1)
 
+    if data_dir:
+        config_obj["BTCD_REGTEST_DATA_DIR"] = data_dir
+
     if reset:
         if not nodocker:
             echo("ERROR: --reset only works in conjunction with --nodocker currently")
@@ -132,32 +134,39 @@ def bitcoind(
                     os.kill(pid, signal.SIGTERM)
             except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):
                 echo(f"Pid {pid} not owned by us. Might be a docker-process? {proc}")
-        if Path(data_dir).exists():
-            echo(f"Purging Datadirectory {data_dir} ...")
+        if Path(config_obj["BTCD_REGTEST_DATA_DIR"]).exists():
+            echo(f"Purging Datadirectory {config_obj['BTCD_REGTEST_DATA_DIR']} ...")
             did_something = True
-            shutil.rmtree(data_dir)
+            shutil.rmtree(config_obj["BTCD_REGTEST_DATA_DIR"])
         if not did_something:
             echo("Nothing to do!")
         return
     mining_every_x_seconds = float(mining_period)
     if nodocker:
         echo("starting plain bitcoind")
-        my_bitcoind = BitcoindPlainController()
+        if os.path.isfile("tests/bitcoin/src/bitcoind"):
+            my_bitcoind = BitcoindPlainController(
+                bitcoind_path="tests/bitcoin/src/bitcoind"
+            )  # always prefer the self-compiled bitcoind if existing
+        else:
+            my_bitcoind = (
+                BitcoindPlainController()
+            )  # Alternatively take the one on the path for now
         # Make sure datadir does exist if specified:
-        Path(data_dir).mkdir(parents=True, exist_ok=True)
+        Path(config_obj["BTCD_REGTEST_DATA_DIR"]).mkdir(parents=True, exist_ok=True)
     else:
-        echo("starting or detecting container")
+        echo("starting container")
         my_bitcoind = BitcoindDockerController(docker_tag=docker_tag)
     try:
         my_bitcoind.start_bitcoind(
-            cleanup_at_exit=True, cleanup_hard=cleanuphard, datadir=data_dir
+            cleanup_at_exit=True,
+            cleanup_hard=cleanuphard,
+            datadir=config_obj["BTCD_REGTEST_DATA_DIR"],
         )
     except docker.errors.ImageNotFound:
         echo(f"Image with tag {docker_tag} does not exist!")
         echo(
-            f"Try to download first with docker pull \
-                     registry.gitlab.com/cryptoadvance/specter-desktop\
-                     /python-bitcoind:{docker_tag}"
+            f"Try to download first with docker pull registry.gitlab.com/cryptoadvance/specter-desktop/python-bitcoind:{docker_tag}"
         )
         sys.exit(1)
     if not nodocker:
@@ -185,7 +194,10 @@ def bitcoind(
 
     if create_conn_json:
         conn = my_bitcoind.rpcconn.as_data()
-        conn["pid"] = os.getpid()  # usefull to sen signals
+        conn["pid"] = os.getpid()  # usefull to send signals
+        conn["specter_data_folder"] = config_obj[
+            "SPECTER_DATA_FOLDER"
+        ]  # e.g. cypress might want to know where we're mining to
         with open("btcd-conn.json", "w") as file:
             file.write(json.dumps(conn))
 
@@ -224,7 +236,15 @@ def miner_loop(my_bitcoind, data_folder, mining_every_x_seconds, echo):
     )
     i = 0
     while True:
-        my_bitcoind.mine()
+        try:
+            my_bitcoind.mine()
+            current_height = my_bitcoind.rpcconn.get_rpc().getblockchaininfo()["blocks"]
+        except Exception as e:
+            logger.debug(
+                "Caught {e}, Couldn't mine, assume SIGTERM occured => exiting!"
+            )
+            echo(f"THE_END(@height:{current_height})")
+            break
         echo("%i" % (i % 10), prefix=False, nl=False)
         if i % 10 == 9:
             echo(" ", prefix=False, nl=False)
@@ -233,7 +253,7 @@ def miner_loop(my_bitcoind, data_folder, mining_every_x_seconds, echo):
             i = 0
             echo("", prefix=False)
             echo(
-                f"height: {my_bitcoind.rpcconn.get_rpc().getblockchaininfo()['blocks']} | ",
+                f"height: {current_height} | ",
                 nl=False,
             )
         time.sleep(mining_every_x_seconds)

src/cryptoadvance/specter/config.py

@@ -40,6 +40,13 @@ class BaseConfig(object):
     # CERT and KEY is for running self-signed-ssl-certs. Check cli_server for details
     CERT = os.getenv("CERT", None)
     KEY = os.getenv("KEY", None)
+    # This will get passed to initialize the specter-object
+    DEFAULT_SPECTER_CONFIG = {}
+
+    # only used by cli_bitcoind.py, we want to have that static for the same reason
+    BTCD_REGTEST_DATA_DIR = os.getenv(
+        "BTCD_REGTEST_DATA_DIR", "/tmp/specter_btc_regtest_plain_datadir"
+    )
 
 
 class DevelopmentConfig(BaseConfig):
@@ -57,6 +64,13 @@ class CypressTestConfig(TestConfig):
     )
     PORT = os.getenv("PORT", 25444)
 
+    # need to be static in order to (un-)tar bitcoind-dirs reliable
+    DEFAULT_SPECTER_CONFIG = {"uid": "123456"}
+
+    BTCD_REGTEST_DATA_DIR = os.getenv(
+        "BTCD_REGTEST_DATA_DIR", "/tmp/specter_cypress_btc_regtest_plain_datadir"
+    )
+
 
 class ProductionConfig(BaseConfig):
     pass

src/cryptoadvance/specter/server.py

@@ -52,7 +52,10 @@ def init_app(app, hwibridge=False, specter=None):
     if specter is None:
         # the default. If not None, then it got injected for testing
         app.logger.info("Initializing Specter")
-        specter = Specter(data_folder=app.config["SPECTER_DATA_FOLDER"])
+        specter = Specter(
+            data_folder=app.config["SPECTER_DATA_FOLDER"],
+            config=app.config["DEFAULT_SPECTER_CONFIG"],
+        )
 
     # version checker
     # checks for new versions once per hour

src/cryptoadvance/specter/specter.py

@@ -24,13 +24,12 @@ import threading
 logger = logging.getLogger(__name__)
 
 
-def get_rpc(conf, old_rpc=None):
+def get_rpc(conf, old_rpc=None, return_broken_instead_none=False):
     """
-    Checks if config have changed,
-    compares with old rpc
+    Checks if config have changed, compares with old rpc
     and returns new one if necessary
-    If there is no working rpc-connection,
-    it has to return None
+    If there is no working rpc-connection, it has to return None
+    If return_broken_instead_none is True, it'll return even a broken connection.
     """
     if "autodetect" not in conf:
         conf["autodetect"] = True
@@ -52,6 +51,8 @@ def get_rpc(conf, old_rpc=None):
         if not conf.get("port", None):
             conf["port"] = 8332
         rpc = BitcoinRPC(**conf)
+    if return_broken_instead_none:
+        return rpc
     # check if we have something to compare with
     if old_rpc is None:
         return rpc if rpc and rpc.test_connection() else None
@@ -328,7 +329,7 @@ class Specter:
     def test_rpc(self, **kwargs):
         conf = copy.deepcopy(self.config["rpc"])
         conf.update(kwargs)
-        rpc = get_rpc(conf)
+        rpc = get_rpc(conf, return_broken_instead_none=True)
         if rpc is None:
             return {"out": "", "err": "autodetect failed", "code": -1}
         r = {}
@@ -350,13 +351,13 @@ class Specter:
             r["err"] = ""
             r["code"] = 0
         except ConnectionError as e:
-            logger.error("Caught an ConnectionError while test_rpc: ", e)
+            logger.error("Caught an ConnectionError while test_rpc: %s", e)
 
             r["tests"]["connectable"] = False
             r["err"] = "Failed to connect!"
             r["code"] = -1
         except RpcError as rpce:
-            logger.error("Caught an RpcError while test_rpc: " + str(rpce))
+            logger.error("Caught an RpcError while test_rpc: %s", rpce)
             logger.error(rpce.status_code)
             r["tests"]["connectable"] = True
             if rpce.status_code == 401:

src/cryptoadvance/specter/static/helpers.js

@@ -1,13 +1,18 @@
 window.addEventListener('load', (event) => {
 	let main = document.getElementsByTagName("main")[0];
 	main.addEventListener('click', (event) => {
-		document.getElementById("side-content").classList.remove("active");
+		side_content = document.getElementById("side-content")
+		if (side_content != null) {
+			side_content.classList.remove("active");
+		}
 	});
 	let menubtn = document.getElementById("menubtn");
-	menubtn.addEventListener('click', (event) => {
-		document.getElementById("side-content").classList.add("active");
-		event.stopPropagation();
-	});
+	if (menubtn != null) {
+		menubtn.addEventListener('click', (event) => {
+			document.getElementById("side-content").classList.add("active");
+			event.stopPropagation();
+		});
+	}
 });
 
 document.addEventListener("errormsg", (e)=>{

src/cryptoadvance/specter/templates/device/device.jinja

@@ -80,7 +80,7 @@
 			<button type="button" class="btn centered" onclick="togglePassphrase('{{ device.device_type }}')">Toggle device passphrase</button>
 		{% endif %}
 		<form action="./" method="POST">
-			<button type="submit" name="action" value="forget" class="btn danger centered">Forget the device</button>
+			<button type="submit" name="action" value="forget" class="btn danger centered" id="forget_device">Forget the device</button>
 		</form>
 	</div>
 	<br>

src/cryptoadvance/specter/templates/includes/sidebar/components/sidebar_btn.jinja

@@ -3,10 +3,11 @@
     Parameters:
     - url_path: URL path the button should lead to (ie. 'new_wallet', 'new_device').
     - text: The text of the button.
+    - id: an optional id for the "a" element
  #}
-{% macro sidebar_btn(url_path, text, icon=true) -%}
+{% macro sidebar_btn(url_path, text, id="", icon=true) -%}
     <div>
-        <a href="{{url_path}}/" class="btn" style="max-width: 90%;margin: auto;">
+        <a href="{{url_path}}/" class="btn" {% if id %}id="{{id}}"{% endif %} style="max-width: 90%;margin: auto;">
             {% if icon %}<svg width="20" height="20" viewBox="0 0 24 24"><path d="M19 13h-6v6h-2v-6H5v-2h6V5h2v6h6v2z"/></svg>{% endif %}{{ text }}
         </a>
     </div>

src/cryptoadvance/specter/templates/includes/sidebar/sidebar.jinja

@@ -114,7 +114,7 @@
             </div>
         {% endif %}
         {% if specter.chain %}
-            {{ sidebar_btn(url_for('wallets_endpoint.new_wallet_type'), 'Add new wallet') }}
+            {{ sidebar_btn(url_for('wallets_endpoint.new_wallet_type'), 'Add new wallet', 'btn_new_wallet') }}
         {% else %}
             <p class="warning">
             &#9432;<br>Wallets are unavailable if Specter is not connected to Bitcoin Core!<br>
@@ -135,7 +135,7 @@
                     {{ sidebar_device_list_item(specter.device_manager.devices[device_name], device_alias) }}
             {% endfor %}
         </div>
-        {{ sidebar_btn(url_for('devices_endpoint.new_device'), 'Add new device') }}
+        {{ sidebar_btn(url_for('devices_endpoint.new_device'), 'Add new device','btn_new_device') }}
         <br>
         <div class="footer">
 	        <span>Specter Version: <strong>{{ specter.version.current }}</strong></span>

src/cryptoadvance/specter/templates/login.jinja

@@ -8,9 +8,9 @@
         <h1>Login to Specter</h1>
         <form action="{{ url_for('auth_endpoint.login') }}" method="POST" role="form">
             {% if specter.config['auth'] == 'usernamepassword' %}
-                <input class="form-control" placeholder="Username" name="username" type="text" value=""><br><br>
+                <input id="username" class="form-control" placeholder="Username" name="username" type="text" value=""><br><br>
             {% endif %}
-            <input class="form-control" placeholder="Password" name="password" type="password" value="">
+            <input id="password" class="form-control" placeholder="Password" name="password" type="password" value="">
             <input type="hidden" name="next" value="{{ data.next }}">
             <br>
             <br>

src/cryptoadvance/specter/templates/settings/bitcoin_core_settings.jinja

@@ -63,7 +63,7 @@
 						{% if my_boolean %}
 							<div style="color: green; font-size: 1.5em;">&#x2714;</div> 
 						{% else %} 
-							&#x274C; 
+							<div style="color: red; font-size: 1.5em;">&#x274C;</div>  
 						{% endif %}
 					</button>
 					{% endmacro %}

src/cryptoadvance/specter/templates/wallet/components/wallet_menu.jinja

@@ -8,10 +8,10 @@
  #}
 {% macro wallet_menu(active_menuitem, wallet_alias) -%}
 	<nav class="row collapse-on-mobile">
-		{{ wallet_menu_item('tx', 'Transactions', wallet_alias, active_menuitem, isLeft=true) }}
-		{{ wallet_menu_item('receive', 'Receive', wallet_alias, active_menuitem) }}
-		{{ wallet_menu_item('send', 'Send', wallet_alias, active_menuitem) }}
-		{{ wallet_menu_item('settings', 'Settings', wallet_alias, active_menuitem, isRight=true) }}
+		{{ wallet_menu_item('tx', 'Transactions', wallet_alias, active_menuitem, isLeft=true, id='btn_transactions') }}
+		{{ wallet_menu_item('receive', 'Receive', wallet_alias, active_menuitem, id='btn_receive') }}
+		{{ wallet_menu_item('send', 'Send', wallet_alias, active_menuitem, id='btn_send') }}
+		{{ wallet_menu_item('settings', 'Settings', wallet_alias, active_menuitem, isRight=true, id='btn_settings') }}
 		<a href="javascript:void(0);" class="mobile-nav-icon" onclick="toggleMobileNav(this, `{{ url_for('static', filename='img/expand-more.svg') }}`, `{{ url_for('static', filename='img/expand-less.svg') }}`)">
 			<img style="width: 36px;" src="{{ url_for('static', filename='img/expand-more.svg') }}"/>
 		</a>

src/cryptoadvance/specter/templates/wallet/components/wallet_menu_item.jinja

@@ -8,10 +8,11 @@
     - isLeft: Is the item at the left edge (first item) of the navigation bar?
     - isRight: Is the item at the right edge (last item) of the navigation bar?
  #}
-{% macro wallet_menu_item(tab, title, wallet_alias, active_menuitem, isLeft=false, isRight=false) -%}
+{% macro wallet_menu_item(tab, title, wallet_alias, active_menuitem, isLeft=false, isRight=false, id="") -%}
 	<a 
         href="{{ url_for('wallets_endpoint.' ~ tab, wallet_alias=wallet_alias)}}"
-        class="btn radio {% if isLeft %}left{% endif %} {% if isRight %}right{% endif %} {% if active_menuitem == tab %}checked{% endif %}">
+        class="btn radio {% if isLeft %}left{% endif %} {% if isRight %}right{% endif %} {% if active_menuitem == tab %}checked{% endif %}"
+        {% if id!=""%} id="{{id}}"{% endif %} >
         {{ title }}
     </a>
 {%- endmacro %}

src/cryptoadvance/specter/templates/wallet/history/components/total_wallet_balances.jinja

@@ -9,7 +9,7 @@
     <h1>
         <small style="line-height:30px">Total balance:</small><br>
         <span style="color: #fff">
-            {{ wallet.fullbalance | btcunitamount }}
+            <span id="fullbalance_amount">{{ wallet.fullbalance | btcunitamount }}</span>
             {% if specter.unit == 'sat' %}
 				sats
 			{% else %}

src/cryptoadvance/specter/templates/wallet/new_wallet/new_wallet.jinja

@@ -12,7 +12,7 @@
 				{% set device = specter.device_manager.devices[device_name] %}
 				<label>
 					<input type="{{ 'checkbox' if wallet_type == 'multisig' else 'radio' }}" {% if wallet_type == 'simple' %} onchange="document.getElementById('submit-device').click()" {% endif %} name="devices" value="{{ device.alias }}" class="hidden" chain="{{specter.chain}}" {% if not device.has_key_types(wallet_type, specter.chain) %}disabled{% endif %}>
-					<div class="small-card radio">
+					<div class="small-card radio" id="{{device.alias}}">
 						<img src="{{ url_for('static', filename='img/devices/' ~ device.icon) }}" width="18px">
 						{{ device_name }}
 					</div>

src/cryptoadvance/specter/templates/wallet/receive/wallet_receive.jinja

@@ -192,7 +192,7 @@
 		{% if wallet.is_multisig and supports_export_to_device != [] %}
 			<button type="button" class="btn centered padded" onclick="hidePageOverlay();showPageOverlay('new_wallet_devices_popup')">Continue</button>
 		{% else %}
-			<button type="button" class="btn centered padded" onclick="hidePageOverlay()">Continue</button>
+			<button type="button" class="btn centered padded" onclick="hidePageOverlay()" id="btn_continue">Continue</button>
 		{% endif %}
 	</div>
 	<script>

src/cryptoadvance/specter/templates/wallet/wallets_overview.jinja

@@ -25,7 +25,7 @@
         <small style="line-height:30px">Total balance:</small><br>
         <span style="color: #fff">
 			{% set fullbalance = specter.wallet_manager.wallets.values() | sum(attribute='fullbalance') %}
-            {{ fullbalance | btcunitamount }}
+            <span id="fullbalance_amount">{{ fullbalance | btcunitamount }}</span>
             {% if specter.unit == 'sat' %}
 				sats
 			{% else %}

utils/calc_cypress_test_spec.py

@@ -0,0 +1,46 @@
+#!/usr/bin/env python3
+" Some tooling for calculating dependencies for cypress_spec_files e.g. ready to pass to cypress --spec"
+
+import json
+import sys
+import click
+import logging
+
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+
+@click.command()
+@click.option("--debug/--no-debug")
+@click.option(
+    "--run/--no-run",
+    default=False,
+    help="--run for creating a run list, otherwise create-list",
+)
+@click.option("--delimiter", default=",", help="the delimiter")
+@click.argument("spec_file")
+def execute(debug, run, delimiter, spec_file):
+    with open("cypress.json") as json_file:
+        data = json.load(json_file)
+
+    spec_create_list = []
+    spec_run_list = []
+    hit = False
+    for my_file in data["testFiles"]:
+        if my_file == spec_file:
+            hit = True
+        if hit:
+            logger.debug(f"iterating {my_file} adding to run_list")
+            spec_run_list.append("./cypress/integration/" + my_file)
+        else:
+            logger.debug(f"iterating {my_file} adding to create_list")
+            spec_create_list.append("./cypress/integration/" + my_file)
+
+    if run:
+        print(delimiter.join(spec_run_list))
+    else:
+        print(delimiter.join(spec_create_list))
+
+
+if __name__ == "__main__":
+    execute()

utils/test-cypress.sh

@@ -0,0 +1,255 @@
+#!/bin/bash
+set -e
+
+PORT=25444
+
+# This needs to be the same than in config.py CypressTestConfig BTCD_REGTEST_DATA_DIR
+# As we don't want to speculate here, we're injecting it via Env-var
+export BTCD_REGTEST_DATA_DIR=/tmp/specter_cypress_btc_regtest_plain_datadir
+# same with SPECTER_DATA_FOLDER
+export SPECTER_DATA_FOLDER=~/.specter-cypress
+
+. ./.env/bin/activate
+function check_consistency {
+    if ps | grep python | grep -v grep ; then # the second grep might be necessary because MacOs has a non POSIX ps
+        echo "there is still a python-process running which is suspicious. Maybe wait a few more seconds"
+        sleep 5
+        ps | grep python && (echo "please investigate or kill " && exit 1)
+    fi
+}
+check_consistency
+
+function sub_default {
+    cat << EOF
+Usage: ./utils/test-cypress.sh [generic-options] <subcommand> [options]"
+Doing stuff with cypress-tests according to <subcommand>"
+
+Subcommands:"
+    open [spec-file]    will open the cypress app."
+    run  [spec-file]    will run the tests."
+                        open and run take a spec-file optionally. If you add a spec-file, "
+                        then automatically the corresponding snapshot is untarred before and," 
+                        in the case of run, only the spec-file and all subsequent spec_files "
+                        are executed."
+    snapshot <spec_file>  will create a snapshot of the spec-file. It will create a tarball"
+                        of the btc-dir and the specter-dir and store that file in the "
+                        ./cypress/fixtures directory"
+
+generic-options:
+    --debug             Run as much stuff in debug as we can
+    --docker            Run bitcoind in docker instead of directly
+EOF
+}
+
+function send_signal() {
+  # use like send_signal <SIGNAL> <PID>
+  # whereas SIGNAL is either SIGTERM or SIGKILL
+  if [[ "$OSTYPE" == "linux-gnu"* ]]; then
+    # Linux wants no SIG-prefix'
+    signal_name=$(echo $1 | sed -e 's/SIG//')
+    kill -${1} $2
+  elif [[ "$OSTYPE" == "darwin"* ]]; then
+    # MacOS needs SIG-prefix
+    kill -s $1 $2
+  fi
+}
+
+function start_bitcoind {
+
+  while [[ $# -gt 0 ]]; do
+    arg="$1"
+    case $arg in
+      --reset)
+        echo "--> Purging $BTCD_REGTEST_DATA_DIR"
+        rm -rf $BTCD_REGTEST_DATA_DIR
+        shift
+        ;;
+      --cleanuphard)
+        addopts="--cleanuphard"
+        shift
+        ;;
+      *)
+        echo "unrecognized argument for start_bitcoind: $1 "
+        exit 1
+        shift
+        ;;
+    esac
+  done
+
+  if [ "$1" = "--reset" ]; then
+      echo "--> Purging $BTCD_REGTEST_DATA_DIR"
+      rm -rf $BTCD_REGTEST_DATA_DIR
+  fi
+  if [ "$1" = "--cleanuphard" ]; then
+      addopts="--cleanuphard"
+  fi
+  if [ "$DOCKER" != "true" ]; then
+      addopts="$addopts --nodocker"
+  fi
+  echo "--> Starting bitcoind with $addopts..."
+  python3 -m cryptoadvance.specter $DEBUG bitcoind $addopts --create-conn-json --config CypressTestConfig &
+  bitcoind_pid=$!
+  while ! [ -f ./btcd-conn.json ] ; do
+      sleep 0.5
+  done
+
+}
+
+function stop_bitcoind {
+  if [ ! -z ${bitcoind_pid+x} ]; then
+    echo "--> Killing/Terminating bitcoindwrapper with PID $bitcoind_pid ..."
+    send_signal SIGTERM $bitcoind_pid
+    wait $bitcoind_pid
+    unset bitcoind_pid
+  fi
+}
+
+function start_specter {
+  if [ "$1" = "--reset" ]; then
+    echo "--> Purging $SPECTER_DATA_FOLDER"
+    rm -rf $SPECTER_DATA_FOLDER
+  fi
+  echo "--> Starting specter ..."
+  python3 -m cryptoadvance.specter $DEBUG server --config CypressTestConfig --debug > /dev/null &
+  specter_pid=$!
+  $(npm bin)/wait-on http://localhost:${PORT}
+}
+
+function stop_specter {
+  if [ ! -z ${specter_pid+x} ]; then
+    echo "--> Killing specter with PID $specter_pid ..."
+    send_signal SIGTERM $specter_pid # kill -9 would orphane strange processes
+    # We don't need to wait as that wastes time.
+    unset specter_pid
+  fi
+}
+
+function cleanup()
+{
+    stop_specter
+    stop_bitcoind
+}
+
+
+trap cleanup EXIT
+
+
+function restore_snapshot {
+  spec_file=$1
+  # Checking whether spec-files exists
+  [ -f ./cypress/integration/${spec_file} ] || (echo "Spec-file $spec_file does not exist, these are the options:"; cat cypress.json | jq ".testFiles[]"; exit 1)
+  snapshot_file=./cypress/fixtures/${spec_file}_btcdir.tar.gz
+  [ -f ${snapshot_file} ] || (echo "Snapshot for Spec-file $spec_file does not exist, these are the options:"; ls -l ./cypress/fixtures; exit 1)
+  ts_snapshot=$(stat --print="%X" ${snapshot_file})
+  for file in $(./utils/calc_cypress_test_spec.py --delimiter " " $spec_file) 
+  do 
+    ts_spec_file=$(stat --print="%X" $file)
+    if [ "$ts_spec_file" -gt "$ts_snapshot" ]; then
+      echo "$file is newer ($ts_spec_file)than the snapshot for $spec_file ($ts_snapshot)"
+      echo "please consider:"
+      echo "./utils/test-cypress.sh snapshot $spec_file"
+      exit 1
+    fi
+  done
+  rm -rf /tmp/${BTCD_REGTEST_DATA_DIR}
+  rm -rf $SPECTER_DATA_FOLDER
+  echo "--> Unpacking ./cypress/fixtures/${spec_file}_btcdir.tar.gz ... "
+  tar -xzf ./cypress/fixtures/${spec_file}_btcdir.tar.gz -C /tmp
+  echo "--> Unpacking ./cypress/fixtures/${spec_file}_specterdir.tar.gz ... "
+  tar -xzf ./cypress/fixtures/${spec_file}_specterdir.tar.gz -C ~
+}
+
+function sub_open {
+  spec_file=$1
+  if [ -n "${spec_file}" ]; then
+    restore_snapshot ${spec_file}
+    start_bitcoind --cleanuphard --reset
+    start_specter
+  else
+    start_bitcoind --reset
+    start_specter --reset
+  fi
+  start_specter
+  $(npm bin)/cypress open
+}
+
+function sub_run {
+  spec_file=$1
+  if [ -f ./cypress/integration/${spec_file} ]; then
+    restore_snapshot ${spec_file}
+    start_bitcoind --cleanuphard --reset
+    start_specter
+    # Run $spec_file and all of the others coming later which come later!
+    $(npm bin)/cypress run --spec $(./utils/calc_cypress_test_spec.py --run $spec_file)
+  else 
+    start_bitcoind --reset
+    start_specter --reset
+    $(npm bin)/cypress run
+  fi
+}
+
+function sub_snapshot {
+  spec_file=$1
+  # We'll create a snapshot BEFORE this spec-file has been tested:
+  if [ ! -f ./cypress/integration/$spec_file ]; then
+    echo "ERROR: Use one of these arguments:"
+    cat cypress.json | jq -r ".testFiles[]"
+    exit 2
+  fi
+  start_bitcoind --reset
+  start_specter --reset
+  $(npm bin)/cypress run --spec $(./utils/calc_cypress_test_spec.py $spec_file)
+  echo "--> stopping specter"
+  stop_specter
+  echo "--> stopping bitcoind gracefully ... won't take long ..."
+  stop_bitcoind
+  echo "--> Creating snapshot  $BTCD_REGTEST_DATA_DIR)"
+  rm ./cypress/fixtures/${spec_file}_btcdir.tar.gz
+  tar -czf ./cypress/fixtures/${spec_file}_btcdir.tar.gz -C /tmp $(basename $BTCD_REGTEST_DATA_DIR)
+  echo "--> Creating snapshot of $SPECTER_DATA_FOLDER"
+  rm ./cypress/fixtures/${spec_file}_specterdir.tar.gz
+  tar -czf ./cypress/fixtures/${spec_file}_specterdir.tar.gz -C ~ $(basename $SPECTER_DATA_FOLDER)
+}
+
+
+function parse_and_execute() {
+  if [[ $# = 0 ]]; then
+    sub_default
+    exit 0
+  fi 
+
+  while [[ $# -gt 0 ]]
+  do
+  arg="$1"
+  case $arg in
+    "" | "-h" | "--help")
+      sub_default
+      shift
+      ;;
+    --debug)
+      set -x
+      DEBUG=--debug
+      shift
+      ;;
+    --docker)
+      DOCKER=true
+      shift
+      ;;
+    *)
+      shift
+      sub_${arg} $@
+      ret_value=$?
+      if [ $ret_value = 127 ]; then
+        echo "Error: '$arg' is not a known subcommand." >&2
+        echo "       Run '$progname --help' for a list of known subcommands." >&2
+        exit 1
+      elif [ $ret_value = 0 ]; then
+        exit 0
+      else
+        exit $ret_value
+      fi
+      ;;
+  esac
+  done
+}
+parse_and_execute $@