GlcoinCore/specter-desktop
1
0
Fork 0
mirror of https://github.com/cryptoadvance/specter-desktop.git synced 2026-08-13 12:33:29 +02:00
Code Issues Projects Releases Packages Wiki Activity Actions 8

Documentation: Create document for testing-setups and cleaning up swan (#1526)

Browse source 
* testing env setup markdown

* make service-list more resilient

* tiny bit more logging in case of issues.

* Swan api firewall fix

* Restoring lost services-related code in wallets_api.py; bugfix on service_data mismatch

* Bugfix on updated completed autowithdrawal addr labels

* publish markdown on doc-page

Co-authored-by: benk10 <ben.kaufman10@gmail.com>
Co-authored-by: kdmukai <kdmukai@gmail.com>
Co-authored-by: moneymanolis <moneymanolis@protonmail.com>
This commit is contained in:
Kim Neunert 2022-01-17 18:33:33 +01:00 • committed by GitHub
parent 8e8c4e5f9e
commit ea111c94c5
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
10 changed files with 214 additions and 22 deletions
Download patch file Download diff file Expand all files Collapse all files

BIN
docs/images/test-new-release/fork-branch.png Normal file
View file

Binary file not shown.
Side by side

After

Width:  |  Height:  |  Size: 45 KiB

148
docs/test-new-releases.md Normal file
View file

@ -0,0 +1,148 @@
# Introduction
You have a production-system and you want to keep it running. Upgrades to specter are usually not that tricky. You simply download the new app and you can usually rollback to the older version in most cases.
However, sometimes you want to test a new feature from a PR or a release-candidate or a new specter-service which is in beta and you don't want to interfere with your production system. So how can you do that?
We'll discuss here some hints on separating environments and maybe even running installations in parallel. At the end, i'll show how to make a copy of a prod system and let it run completely in parallel with the prod-system. But let's go one after another and look at the building-blocks.
# On specterd, branches and PRs
We'll assume here, that you're familiar with the commandline. If you haven't worked with it, don't worry too much but also don't shy away. If a change has been made but not yet releases, there are several ways how this could be made available. Here are the possibilities:
The easiest one is a release-candidate of the specter-project. This would be available in the [Release-section](https://github.com/cryptoadvance/specter-desktop/releases) on github. We'll assume here that we start the service on the commandline, so please download the corresponding `specterd` for your platform.
The same is true for release-candidates from forks of the project. In that case, you'd simply download the `specterd` from the release-page of that fork. In both cases, at least in linux, you have to make the `specterd` executable. You can do that via the UI or preferreable via the commandline with a command like `chmod u+x ./specterd`.
Let's assume you'd try out a feature from a PR. In that case you would need a [developmenet-environment](https://docs.specter.solutions/desktop/development/#set-up-virtualenv). However after cloning the repo (first command in the link), you need to change to the fork/branch mentioned in the PR. You can see the fork branch if you navigate to the PR. Here is an example:
![](./images/test-new-release/fork-branch.png)
So the branch, in this case, is called `service_swan`. The branch is located in a fork by the user `k9ert`. So in order to update the code you just cloned from github, do this:
```
git checkout -b k9ert-service_swan master
git pull https://github.com/k9ert/specter-desktop.git service_swan
```
Adjust the branch/user accordingly. After that, continue to setup your environment as described in the link above.
# specterd binary or source cli
No matter whether you've downloaded a specterd-binary or you've cloned/checked out a specific environment, you're ready to start specter via the cli. A specterd would be started like
```
$ ./specterd
Initializing HWI...
Usage: specterd [OPTIONS] COMMAND [ARGS]...
Options:
--debug Show debug information on errors.
--tracerpc / --no-tracerpc Will trace all calls to BitcoinCore or
ElementsCore if in --debug
--help Show this message and exit.
Commands:
bitcoind This will start a bitcoind regtest and mines a block every...
elementsd This will start a bitcoind regtest and mines a block every...
server
```
The very same output you would get if you do a `$ python3 -m cryptoadvance.specter`.
Effectively you get hints on how to use the cli. We're interested in spinning up the server and we can get additional information on the server-command by:
```
$ ./specterd server --help
Initializing HWI...
Usage: specterd server [OPTIONS]
Options:
--port TEXT TCP port to bind specter to
--host TEXT if you specify --host 0.0.0.0 then specter will
be available in your local LAN.
--cert TEXT --cert and --key are for specifying and using a
self-signed certificate for SSL encryption.
--key TEXT --cert and --key are for specifying and using a
self-signed certificate for SSL encryption.
--ssl / --no-ssl By default SSL encryption will not be used. Use
-ssl to create a self-signed certificate for SSL
encryption. You can also specify encryption via
--cert and --key.
--debug / --no-debug
--filelog / --no-filelog
--tor
--hwibridge Start the hwi-bridge to use your HWWs with a
remote specter.
--specter-data-folder TEXT Use a custom specter data-folder. By default it
is ~/.specter.
--config TEXT A class from the config.py which sets reasonable
default values.
--help Show this message and exit.
```
The relevant interesting switches are `port`, `specter-data-folder` and config. Let's go with one after the other. But before that, let's clarify what you get if you start specter like this on the commandline.
```
python3 -m cryptoadvance.specter server
Initializing HWI...
[2022-01-13 10:29:51,415] INFO in server: Configuration: cryptoadvance.specter.config.ProductionConfig
[lotsOfOutput]
[2022-01-13 10:30:07,560] INFO in _internal: * Running on http://127.0.0.1:25441/ (Press CTRL+C to quit)
[aBitMoreOutput]
```
As the output stated, specter is now running on your local machine and you can go visit it by following the link shown: [http://127.0.0.1:25441/](http://127.0.0.1:25441/) and you can stop specter again, by pressing `CTRL+C` in the terminal.
# The specter-data-folder and port
As almost all reasonable programs on a computer. specter needs a place to store its data somewhere. Some programs use Databases, specter uses files on the harddisk, only. These files are all located in a single directory, usually in your homefolder called `.specter`. The dot in the name will effectively make that directory hidden, so you might switch to `show-hidden-files` in your filebrowser or use the a-switch in ls (`ls -a`).
So now you know where your production data is stored. Maybe you want to make a backup. If you use a internal Bitcoin-Core-Node, then this directory might take up 4GB or more. If not, it's only a few kb big. You can zip that directory in linux like:
```
zip -r specter-prod.zip .specter
```
Obviously, that might not make so much sense, if there is an internal node stored in that directory. In such a case you can still zip that folder by:
```
zip -r specter-prod.zip .specter -x ".specter/nodes/*"
```
But you can also start specter with a different specter-folder. For example like:
```
$ ./specterd server --specter-data-folder ~/.specter_manual_test
```
It doesn't matter whether the directory exists. If not, it will be created automatically. So now specter is running in a different folder, but it's running on the same port. However, you can also choose a different port via:
```
$ ./specterd server --specter-data-folder ~/.specter_manual_test --port 26441
```
You will then find this instance running at [http://127.0.0.1:26441/](http://127.0.0.1:26441/). The nice thing here is, that you can run many specters in parallel if they run on different port. For sure, you should also choose a different `specter-data-folder` (otherwise you'd get an error-message).
# Configurations
Running a development-environment has different requirements than a production-env, e.g.:
* You want debugging-messages
* You don't want to relogin just because the server restarted
* You don't want to mess with the `~/.specter` data-folder
* You want all the potential service-integration to be available, even if they are alpha
That set of configurations can be passed over by referencing the `DevelopmentConfig` from `config.py` like this:
```
$ ./specterd server --config DevelopmentConfig --debug
```
# Walkthrough
So let's assume you want to run a test-system almost completely separate from your production-system. The only thing which is shared is the external Bitcoin Core node. So here are the steps:
1. Make a zip from the prod-folder
```
zip -r specter-prod.zip .specter -x ".specter/nodes/*"
```
2. Unzip in new folder
```
unzip specter-prod.zip -d tmp
mv tmp/.specter .specter_copy
```
3. Start with new folder (and port)
```
$ ./specterd server --config DevelopmentConfig --debug --specter-data-folder ~/.specter_copy --port 27441
```

1
mkdocs.yml
View file

@ -22,6 +22,7 @@ nav:
- Development:
- 'Contributing': contributing.md
- 'Developing': development.md
- 'Testing': test-new-releases.md
- build-instructions.md
- 'Continuous Integration': continuous-integration.md
- cypress-testing.md

3
src/cryptoadvance/specter/persistence.py
View file

@ -33,6 +33,9 @@ def read_json_file(path):
# if failed - try reading from the backup
except Exception as e:
logger.error(
f"Exception {e} while reading file {path}. Reading from backup"
)
# if no backup exists - raise
if not os.path.isfile(bkp):
raise e

35
src/cryptoadvance/specter/server_endpoints/wallets_api.py
View file

@ -181,7 +181,7 @@ def broadcast_blockexplorer(wallet_alias):
@app.csrf.exempt
def decoderawtx(wallet_alias):
try:
wallet = app.specter.wallet_manager.get_by_alias(wallet_alias)
wallet: Wallet = app.specter.wallet_manager.get_by_alias(wallet_alias)
txid = request.form.get("txid", "")
if txid:
tx = wallet.rpc.gettransaction(txid)
@ -212,6 +212,18 @@ def decoderawtx(wallet_alias):
try:
rawtx = decoderawtransaction(tx["hex"], app.specter.chain)
# Enrich utxo data with address_info if the addr is from our Wallet
for utxo in rawtx.get("vout", []):
address = utxo.get("address")
addr_obj = wallet.get_address_info(address)
if addr_obj:
utxo.update(addr_obj)
if not utxo.get("label"):
utxo["label"] = addr_obj.label
# TODO: Fetch the relevant Input utxo details so the JS doesn't have to
# make a separate call.
except:
rawtx = wallet.rpc.decoderawtransaction(tx["hex"])
# add assets
@ -226,8 +238,18 @@ def decoderawtx(wallet_alias):
rawtx=rawtx,
walletName=wallet.name,
)
except RpcError as e:
if "Invalid or non-wallet transaction id" in str(e):
# Expected failure when looking up a txid that didn't originate from the
# user's Wallet.
pass
else:
app.logger.warning(
"Failed to fetch transaction data. Exception: {}".format(e)
)
except Exception as e:
handle_exception(e)
app.logger.warning("Failed to fetch transaction data. Exception: {}".format(e))
return jsonify(success=False)
@ -289,11 +311,13 @@ def txlist(wallet_alias):
search = request.form.get("search", None)
sortby = request.form.get("sortby", None)
sortdir = request.form.get("sortdir", "asc")
service_id = request.form.get("service_id", None)
fetch_transactions = request.form.get("fetch_transactions", False)
txlist = wallet.txlist(
fetch_transactions=fetch_transactions,
validate_merkle_proofs=app.specter.config.get("validate_merkle_proofs", False),
current_blockheight=app.specter.info["blocks"],
service_id=service_id,
)
return process_txlist(
txlist, idx=idx, limit=limit, search=search, sortby=sortby, sortdir=sortdir
@ -329,11 +353,14 @@ def wallets_overview_txlist():
sortby = request.form.get("sortby", None)
sortdir = request.form.get("sortdir", "asc")
fetch_transactions = request.form.get("fetch_transactions", False)
service_id = request.form.get("service_id")
txlist = app.specter.wallet_manager.full_txlist(
fetch_transactions=fetch_transactions,
validate_merkle_proofs=app.specter.config.get("validate_merkle_proofs", False),
current_blockheight=app.specter.info["blocks"],
service_id=service_id,
)
return process_txlist(
txlist, idx=idx, limit=limit, search=search, sortby=sortby, sortdir=sortdir
)
@ -361,7 +388,7 @@ def wallets_overview_utxo_list():
def addresses_list(wallet_alias):
"""Return a JSON with keys:
addressesList: list of addresses with the properties
(index, address, label, used, utxo, amount)
(index, address, label, used, utxo, amount, service_id)
pageCount: total number of pages
POST parameters:
idx: pagination index (current page)
@ -370,7 +397,7 @@ def addresses_list(wallet_alias):
(index, address, label, used, utxo, amount)
sortdir: 'asc' (ascending) or 'desc' (descending) order
addressType: the current tab address type ('receive' or 'change')"""
wallet = app.specter.wallet_manager.get_by_alias(wallet_alias)
wallet: Wallet = app.specter.wallet_manager.get_by_alias(wallet_alias)
idx = int(request.form.get("idx", 0))
limit = int(request.form.get("limit", 100))

2
src/cryptoadvance/specter/services/service_manager.py
View file

@ -110,7 +110,7 @@ class ServiceManager:
@property
def services(self) -> Dict[str, Service]:
return self._services
return self._services or {}
@property
def services_sorted(self):

10
src/cryptoadvance/specter/services/swan/client.py
View file

@ -77,6 +77,9 @@ def get_access_token(code: str = None, code_verifier: str = None):
If we don't have the refresh_token, raise SwanApiRefreshTokenException.
"""
# Must explicitly set User-Agent; Swan firewall blocks all requests with "python".
auth_header = {"User-Agent": "Specter Desktop"}
if code:
# Requesting initial refresh_token and access_token
payload = {
@ -86,7 +89,6 @@ def get_access_token(code: str = None, code_verifier: str = None):
"grant_type": "authorization_code",
"code": code,
}
auth_header = None
else:
service_data = SwanService.get_current_user_service_data()
if SwanService.is_access_token_valid():
@ -106,9 +108,7 @@ def get_access_token(code: str = None, code_verifier: str = None):
}
auth_hash = base64.b64encode(f"{client_id}:{client_secret}".encode()).decode()
auth_header = {
"Authorization": f"Basic {auth_hash}",
}
auth_header["Authorization"] = f"Basic {auth_hash}"
logger.debug(f"auth_hash: {auth_hash}")
logger.debug("Using the refresh_token to request an access_token")
logger.debug(f"payload: {json.dumps(payload, indent=4)}")
@ -173,7 +173,9 @@ def authenticated_request(
access_token = get_access_token()
# Must explicitly set User-Agent; Swan firewall blocks all requests with "python".
auth_header = {
"User-Agent": "Specter Desktop",
"Authorization": f"Bearer {access_token}",
}
try:

22
src/cryptoadvance/specter/services/swan/service.py
View file

@ -6,6 +6,7 @@ import pytz
from flask import current_app as app
from flask_babel import lazy_gettext as _
from typing import List
from cryptoadvance.specter.specter_error import SpecterError
from cryptoadvance.specter.user import User
@ -61,9 +62,15 @@ class SwanService(Service):
if not service_data or cls.SPECTER_WALLET_ALIAS not in service_data:
# Service is not initialized; nothing to do
return
return app.specter.wallet_manager.get_by_alias(
service_data[cls.SPECTER_WALLET_ALIAS]
)
try:
return app.specter.wallet_manager.get_by_alias(
service_data[cls.SPECTER_WALLET_ALIAS]
)
except SpecterError as e:
logger.debug(e)
# Referenced an unknown wallet
# TODO: keep ignoring or remove the unknown wallet from service_data?
return
@classmethod
def set_associated_wallet(cls, wallet: Wallet):
@ -371,23 +378,18 @@ class SwanService(Service):
service_id=cls.id, unused_only=False
)
for addr_obj in reserved_addresses:
if addr_obj["used"] and addr_obj["label"] == cls.default_address_label:
if addr_obj.used and addr_obj.label == cls.default_address_label():
# This addr has received an autowithdrawal since we last checked
logger.debug(
f"Updating address label for {json.dumps(addr_obj, indent=4)}"
)
addr_obj.set_label(str(_("Swan autowithdrawal")))
wallet.setlabel(addr_obj.address, str(_("Swan autowithdrawal")))
num_pending_autowithdrawal_addrs = len(
[addr_obj for addr_obj in reserved_addresses if not addr_obj["used"]]
)
if num_pending_autowithdrawal_addrs < cls.MIN_PENDING_AUTOWITHDRAWAL_ADDRS:
from . import (
client as swan_client,
) # Import here to avoid circular dependency
logger.debug("Need to send more addrs to Swan")
cls.reserve_addresses(
wallet=wallet, num_addresses=cls.MIN_PENDING_AUTOWITHDRAWAL_ADDRS
)

7
src/cryptoadvance/specter/templates/includes/tx-table.html
View file

@ -220,6 +220,7 @@
* - Showing amounts in either BTC or sats (set attribute `btc-unit` to either "btc" or "sat")
* - Showing prices next to the amounts (set attribute `price` to the BTC price and symbol to the symbol of the currency you're pricing at)
* - Showing validated blockhash column (set attribute `blockhash` to either "true" or "false")
* - Limiting txs/utxos to just those associated with the specified `data-service-id`.
* - Hide columns by hide-columns="category time"
*/
class TxTableElement extends HTMLElement {
@ -442,7 +443,7 @@
}
static get observedAttributes() {
return ['blockhash', 'btc-unit', 'price', 'symbol', 'type', 'hide-sensitive-info', 'wallet', 'selected-coins'];
return ['blockhash', 'btc-unit', 'price', 'symbol', 'type', 'hide-sensitive-info', 'wallet', 'service-id', 'selected-coins'];
}
connectedCallback() {
@ -522,7 +523,8 @@
this.symbol != this.getAttribute('symbol') ||
this.listType != this.getAttribute('type') ||
this.hideSensitiveInfo != this.getAttribute('hide-sensitive-info') ||
this.wallet != this.getAttribute('wallet')
this.wallet != this.getAttribute('wallet') ||
this.serviceId != this.getAttribute('service-id')
) {
this.blockhash = this.getAttribute('blockhash');
this.btcUnit = this.getAttribute('btc-unit');
@ -531,6 +533,7 @@
this.listType = this.getAttribute('type');
this.hideSensitiveInfo = this.getAttribute('hide-sensitive-info') == 'true';
this.wallet = this.getAttribute('wallet');
this.serviceId = this.getAttribute('service-id');
if (!this.listType) {
return
}

8
src/cryptoadvance/specter/user.py
View file

@ -86,7 +86,7 @@ class User(UserMixin):
self.wallet_manager = None
self.device_manager = None
self.manager = None
self.services = services
self._services = services
# Iterations will need to be increased over time to keep ahead of CPU advances.
self.encryption_iterations = 390000
@ -123,6 +123,12 @@ class User(UserMixin):
return ""
return f"_{self.id}"
@property
def services(self):
if self._services:
return self._services
return []
@property
def is_user_secret_decrypted(self):
return self.plaintext_user_secret is not None