Bugfix: Very last fixes for Service swan (#1537)

* Feature: Voltoro trading first commit

* fix create_order issue

* fix url and better error-handling

* basic balances tab

* Add deposit vaultoro page

* activating services

* Service management

* Improve settings page and default to it if token unset

* Fix history tab

* Fix trade error handling

* Add withdraw

* calling specter-cloud for creating vaultoro orders

* Improve trade screen and fixes

* sidebar fix

* black

* refactor Service integration

* refactor to have Service Classes like manifests

* maturity

* dynamic initialisation of service-classes and blueprints

* fix sidebar_services

* migrated templates and static into vaultoro folder

* refactor config to manifest

* some minor things

* swan initial

* rename and fix test

* adding ServiceApiKeyStorageUserAware

* fix

* store the access token

* directory indirection to shield templates from each others blueprint

* first attempts with automatic withdrawals

* proper tab highlighting

* Interim commit

* Update service_apikey_storage.py

* Update oauth2_success.jinja

* Awaiting refresh_token support

* Service logo display on Addresses

* Associate addr with a Service

* address-data component reorg

Separates the presentation html from the data as much as possible.

* Services data/icon added to tx History

* Now hitting the updated Swan endpoint to save deposit addrs

* Simplified injecting Services data into JS

* Reducing js calls back to server in tx-data; templatizing utxo in/outs

* renaming "reserving" to "associating" an Address with a Service.
* rename `manifest.py` files to `service.py`.
* rename "api_data" to "service_data" to make the storage a bit more generalized. `ServiceApiKeyStorage` is now `ServiceEncryptedStorage` to match.
* Beginning of factoring out Swan api to its own `api.py` file; need to rectify with `swan_client.py`.

* interim commit

* Removed tx-table/row/data changes and address-table/row/data

Kept only the bare minimum changes required to display the Services icon, plus optimizations.

* interim commit

* Interim commit

* Cleanup commit

* Update controller.py

* Update services.md

* Update services.md

* deleted no longer used CustomElement

* Adding services docs to mkdocs

* Configuration for Services

* more clever configuration

* fix test

* deleted swan_client

* Changing the address abbreviation format to 7...7 and little big fix for not vertically aligned addresses in Firefox.

* Better state management if Auth method changes

* Cleanup, better user messaging; pulling Service methods out of controller and User

* Redirect to services endpoint after setting up authentication.

* PR cleanup, bug fixes, test suite updates

* Fixed test case problem

* First fix for delete API key button.

* Service hooks; Option to fully remove Swan Integration; Logout clears plaintext_user_secret

* Restoring bugfix from @moneymanolis

* black

* cleanup and black

* Make service-decovery in AppImage work

* import hashlib, maybe fix cypress

* Update service_encrypted_storage.py

* further bugfix on update

* TODO: remove debugging in client.py before first release

* testing env setup markdown

* make service-list more resilient

* tiny bit more logging in case of issues.

* Swan api firewall fix

* Restoring lost services-related code in wallets_api.py; bugfix on service_data mismatch

* Bugfix on updated completed autowithdrawal addr labels

* publish markdown on doc-page

* Still awaiting final Swan prod tests

* move ServiceManager outside Specter.__init__()

* Update config.py

* Applying Kim's prop patch

* Disabling extension loading from cwd in prod

* build-script adjustments for clarity and right order

* More comprehensive input validation for the rate limit in the auth settings.

* monkey patch rthooks for a successfull MacOS-build

* fix build-ci.sh

* Refactor service_manager and templates where they belong to

* Swan - different fronend-links links for dev/prod

* include templates and services in sdist

* implement dynamic loading but from list in config

* No dynamic cwd services in appimages

* tidy up

Co-authored-by: benk10 <ben.kaufman10@gmail.com>
Co-authored-by: kdmukai <kdmukai@gmail.com>
Co-authored-by: moneymanolis <moneymanolis@protonmail.com>
This commit is contained in:
Kim Neunert 2022-01-26 22:11:26 +01:00 committed by GitHub
parent 352355e8d7
commit e2ad1eaea1
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
18 changed files with 202 additions and 47 deletions

View file

@ -1,5 +1,9 @@
recursive-include src/cryptoadvance/specter/templates *
recursive-include src/cryptoadvance/specter/static *
recursive-include src/cryptoadvance/specter/services/templates *
recursive-include src/cryptoadvance/specter/services/static *
recursive-include src/cryptoadvance/specter/services/*/templates *
recursive-include src/cryptoadvance/specter/services/*/static *
recursive-include src/cryptoadvance/specter/translations/*/LC_MESSAGES *.mo
recursive-include src/cryptoadvance/specter/translations/*/LC_MESSAGES *.po
include requirements.txt

View file

@ -1,4 +1,4 @@
from cryptoadvance.specter.services.service_manager import ServiceManager
from cryptoadvance.specter.managers.service_manager import ServiceManager
# Collecting template and static files from the different services in src/cryptoadvance/specter/services

View file

@ -143,6 +143,12 @@ class BaseConfig(object):
# THIS MIGHT BE A SECURITY_CRITICAL SETTING. DON'T SWITH TO TRUE IN PROD
SERVICES_LOAD_FROM_CWD = False
# List of extensions (services) to potentially load
EXTENSION_LIST = [
"cryptoadvance.specter.services.swan.service",
"cryptoadvance.specter.services.bitcoinreserve.service",
]
# This is just a placeholder in order to be aware that you cannot set this
# It'll be filled up with the fully qualified Classname the Config is derived from
SPECTER_CONFIGURATION_CLASS_FULLNAME = None

View file

@ -1,21 +1,28 @@
import json
import logging
import os
from flask import current_app as app, url_for
from flask.blueprints import Blueprint
from importlib import import_module
from inspect import isclass
from pathlib import Path
from pkgutil import iter_modules
from typing import Dict, List
from cryptoadvance.specter.user import User
from cryptoadvance.specter.config import ProductionConfig
from cryptoadvance.specter.managers.singleton import ConfigurableSingletonException
from ..util.reflection import get_subclasses_for_class, _get_module_from_class
from cryptoadvance.specter.user import User
from flask import current_app as app
from flask import url_for
from flask.blueprints import Blueprint
from .service import Service
from .service_encrypted_storage import ServiceEncryptedStorageManager
from ..services.service import Service
from ..services.service_encrypted_storage import ServiceEncryptedStorageManager
from ..util.reflection import (
_get_module_from_class,
get_classlist_of_type_clazz_from_modulelist,
get_package_dir_for_subclasses_of,
get_subclasses_for_clazz,
get_subclasses_for_clazz_in_cwd,
)
logger = logging.getLogger(__name__)
@ -30,9 +37,19 @@ class ServiceManager:
# Each Service class is stored here, keyed on its Service.id str
self._services: Dict[str, Service] = {}
logger.info("----> starting service discovery <----")
for clazz in get_subclasses_for_class(
Service, app.config.get("SERVICES_LOAD_FROM_CWD", False)
):
# How do we discover services? Two configs are relevant:
# * SERVICES_LOAD_FROM_CWD (boolean, CWD is current working directory)
# * EXTENSION_LIST (array of Fully Qualified module strings like ["cryptoadvance.specter.services.swan.service"])
# Ensuring security (especially for the CWD) is NOT done here but
# in the corresponding (Production)Config
logger.debug(f"EXTENSION_LIST = {app.config.get('EXTENSION_LIST')}")
class_list = get_classlist_of_type_clazz_from_modulelist(
Service, app.config.get("EXTENSION_LIST", [])
)
if app.config.get("SERVICES_LOAD_FROM_CWD", False):
class_list.extend(get_subclasses_for_clazz_in_cwd(Service))
class_list = set(class_list) # remove duplicates (shouldn't happen but ...)
for clazz in class_list:
compare_map = {"alpha": 1, "beta": 2, "prod": 3}
if compare_map[self.devstatus_threshold] <= compare_map[clazz.devstatus]:
# First configure the service
@ -84,9 +101,6 @@ class ServiceManager:
cls.import_config(clazz)
return
logger.warning(
f"Could not find a configuration for Service {module} ... trying parent-classes of main-config"
)
config_module = import_module(".".join(main_config_clazz_name.split(".")[0:-1]))
config_clazz = getattr(config_module, main_config_clazz_slug)
@ -97,10 +111,13 @@ class ServiceManager:
cls.import_config(clazz)
return
config_candidate_class = config_candidate_class.__bases__[0]
logger.warning(
f"Could not find a configuration for Service {module}. Skipping configuration."
)
@classmethod
def import_config(cls, clazz):
logger.info(f"Loading Service-specific configuration from {clazz}")
logger.info(f" Loading Service-specific configuration from {clazz}")
for key in dir(clazz):
if key.isupper():
if app.config.get(key):
@ -166,19 +183,26 @@ class ServiceManager:
@classmethod
def get_service_x_dirs(cls, x):
"""returns a list of package-directories which represents a specific service.
This is primarily used by the pyinstaller packaging specter
This is used by the pyinstaller packaging specter
"""
arr = [
Path(Path(_get_module_from_class(clazz).__file__).parent, x)
for clazz in get_subclasses_for_class(Service)
for clazz in get_subclasses_for_clazz(Service)
]
arr = [path for path in arr if path.is_dir()]
return [Path("..", *path.parts[-6:]) for path in arr]
@classmethod
def get_service_packages(cls):
"""returns a list of strings containing the service-classes. This is used for hiddenimports in pyinstaller"""
arr = get_subclasses_for_class(Service)
"""returns a list of strings containing the service-classes (+ controller/config-classes)
This is used for hiddenimports in pyinstaller
"""
arr = get_subclasses_for_clazz(Service)
arr.extend(
get_classlist_of_type_clazz_from_modulelist(
Service, ProductionConfig.EXTENSION_LIST
)
)
arr = [clazz.__module__ for clazz in arr]
# Controller-Packagages from the services are not imported via the service but via the baseclass
# Therefore hiddenimport don't find them. We have to do it here.
@ -201,4 +225,11 @@ class ServiceManager:
# RuntimeError: Working outside of application context.
# shows that the package is existing
arr.append(controller_package)
config_arr = [".".join(package.split(".")[:-1]) + ".config" for package in arr]
for config_package in config_arr:
try:
import_module(config_package)
arr.append(config_package)
except ModuleNotFoundError as e:
pass
return arr

View file

@ -12,7 +12,7 @@ from flask_wtf.csrf import CSRFProtect
from cryptoadvance.specter.liquid.rpc import LiquidRPC
from cryptoadvance.specter.rpc import BitcoinRPC
from cryptoadvance.specter.services.service_manager import ServiceManager
from cryptoadvance.specter.managers.service_manager import ServiceManager
from cryptoadvance.specter.util.reflection import get_template_static_folder
from .helpers import hwi_get_config

View file

@ -67,6 +67,7 @@ class Service:
controller_module = (
f"cryptoadvance.specter.services.{self.id}.controller"
)
logger.info(f" Loading Controller {controller_module}")
import_module(controller_module)
app.register_blueprint(
self.__class__.blueprint, url_prefix=f"/svc/{self.id}"

View file

@ -10,6 +10,7 @@ class BaseConfig:
"BcetcVcmueWf5P3UPJnHhCBMQ49p38fhzYwM7t3DJGzsXSjm89dDR5URE46SY69j"
)
SWAN_API_URL = "https://dev-api.swanbitcoin.com"
SWAN_FRONTEND_URL = "https://dev-app.swanbitcoin.com/signup"
class ProductionConfig(BaseConfig):
@ -18,3 +19,4 @@ class ProductionConfig(BaseConfig):
"UcqMZw3D70#E*Zo1hnC8f8P^Ils^6wligXMB*vL1fX@DYm6zloDI#p9Eemk8!y9#"
)
SWAN_API_URL = "https://api.swanbitcoin.com"
SWAN_FRONTEND_URL = "https://www.swanbitcoin.com/Specter/"

View file

@ -54,7 +54,7 @@ def index():
# User has already completed Swan integration; skip ahead
return redirect(url_for(f"{SwanService.get_blueprint_name()}.withdrawals"))
return render_template(
"swan/index.jinja",
"swan/index.jinja", swan_frontend_url=app.config["SWAN_FRONTEND_URL"]
)

View file

@ -77,7 +77,7 @@
<img src="{{ url_for(service.id +'_endpoint' + '.static', filename=service.logo) }}" width="300"/>
<div class="tagline">&ldquo;Swan is the best way to accumulate Bitcoin with automatic recurring buys and instant buys.&rdquo;</div>
<div class="card center" style="width: auto; min-width: 90%; margin: 40px;">
<a class="button_wide" href="https://www.swanbitcoin.com/Specter/" target="_swan">
<a class="button_wide" href="{{ swan_frontend_url }}" target="_swan">
<span class="big_option">
<div class="big_option_text">
Join Swan!

View file

@ -37,7 +37,7 @@ from .rpc import (
RpcError,
get_default_datadir,
)
from .services.service_manager import ServiceManager
from .managers.service_manager import ServiceManager
from .services.service import devstatus_alpha, devstatus_beta, devstatus_prod
from .specter_error import ExtProcTimeoutException, SpecterError
from .tor_daemon import TorDaemonController

View file

@ -1,7 +1,19 @@
import re
def str2bool(my_str):
"""returns a reasonable boolean from a string so that "False" will result in False"""
if my_str is None:
return False
elif isinstance(my_str, str) and my_str.lower() == "false":
return False
elif isinstance(my_str, str) and my_str.lower() == "off":
return False
return bool(my_str)
def camelcase2snake_case(name):
"""If you pass DeviceManager it returns device_manager"""
pattern = re.compile(r"(?<!^)(?=[A-Z])")
name = pattern.sub("_", name).lower()
return name

View file

@ -6,6 +6,8 @@ from pathlib import Path
import pkgutil
from pkgutil import iter_modules
import sys
from .common import camelcase2snake_case
from ..specter_error import SpecterError
logger = logging.getLogger(__name__)
@ -28,11 +30,12 @@ def get_template_static_folder(foldername):
def get_package_dir_for_subclasses_of(clazz):
"""There are two occasions where this is used: migrations and service-classes. Depending on the clazz
"""There are two occasions where this makes sense: migrations and service-classes. Depending on the clazz
this function is returning package_directories where subclasses from clazz are supposed to be located
* subclasses of SpecterMigration are located in a subpackage called migrations
* subclasses of Servuce are located in subpackages of package cryptoadvance.specter.services
I have to admit that this is not a pure util class as it's containing business-logic
I have to admit that this is not a pure util class as it's containing business-logic.
It's no longer used for Service.
"""
if clazz.__name__ == "SpecterMigration":
return str(
@ -47,21 +50,62 @@ def get_package_dir_for_subclasses_of(clazz):
import_module("cryptoadvance.specter.services").__file__
).parent.resolve()
)
# This is mainly for testing purposes for now
elif clazz.__name__ == "Device":
return str(
Path(
import_module("cryptoadvance.specter.devices").__file__
).parent.resolve()
)
raise SpecterError("Unknown Class: {clazz}")
def get_subclasses_for_class(clazz, services_load_from_cwd=False):
"""Returns all subclasses of class clazz located in the specific package for that class"""
def get_classlist_of_type_clazz_from_modulelist(clazz, modulelist):
"""A helper method converting a List of modules as described in config.py
into a List of classes. In order to make that more util-like, you
have to pass the the class you're searching for in the modules
"""
class_list = []
loopdir = Path(__file__).resolve()
package_dir = get_package_dir_for_subclasses_of(clazz)
logger.info(f"Collecting subclasses of {clazz.__name__} in {package_dir}...")
package_dirs = [package_dir]
if services_load_from_cwd:
if not Path("./src/cryptoadvance").is_dir():
package_dirs.append(".")
logger.debug(
"Running in non-specter-src-folder. Added CWD to Service-Discovery"
)
for fq_module_name in modulelist:
module = import_module(fq_module_name)
logger.debug(f"Imported {fq_module_name}")
for attribute_name in dir(module):
attribute = getattr(module, attribute_name)
if isclass(attribute):
if (
issubclass(attribute, clazz)
and not attribute.__name__ == clazz.__name__
):
class_list.append(attribute)
logger.info(f" Found class {attribute.__name__}")
return class_list
def get_subclasses_for_clazz_in_cwd(clazz):
"""Returns all subclasses of class clazz located in the CWD if the cwd
is not a specter-desktop dev-env-kind-of-dir
"""
package_dirs = []
if Path("./src/cryptoadvance").is_dir() or getattr(sys, "frozen", False):
# No discovery in specter-desktop-dev-env (doesn't make sense)
# or appimage-mode (technically difficult on Linux and security-risk for
# appimage-users even on --config DevelopmentConfig)
return []
else:
package_dirs.append(".")
logger.info("Running in non-specter-src-folder. Added CWD to Service-Discovery")
return get_subclasses_for_clazz(clazz, package_dirs)
def get_subclasses_for_clazz(clazz, package_dirs=None):
"""Returns all subclasses of class clazz located in the CWD
potentially add additional_packagedirs which is usefull for
calculating pyinstaller hiddenimports
"""
if package_dirs == None:
package_dirs = [get_package_dir_for_subclasses_of(clazz)]
class_list = []
logger.info(f"Collecting subclasses of {clazz.__name__} in {package_dirs}...")
for (_, module_name, _) in iter_modules(
package_dirs
): # import the module and iterate through its attributes
@ -91,6 +135,19 @@ def get_subclasses_for_class(clazz, services_load_from_cwd=False):
module = import_module(
f"cryptoadvance.specter.util.migrations.{module_name}"
)
else:
try:
module = import_module(
f"{module_name}.{camelcase2snake_case(clazz.__name__)}"
)
logger.debug(
f"Imported {module_name}.{camelcase2snake_case(clazz.__name__)}"
)
except ModuleNotFoundError as e:
logger.debug(
f"No Service Impl found in {module_name}.service. Skipping!"
)
continue
for attribute_name in dir(module):
attribute = getattr(module, attribute_name)
if isclass(attribute):

View file

@ -1,12 +1,18 @@
import logging
from unittest.mock import MagicMock
from flask import Flask
from cryptoadvance.specter.services.service_manager import ServiceManager
from cryptoadvance.specter.managers.service_manager import ServiceManager
def test_ServiceManager():
def test_ServiceManager(caplog):
caplog.set_level(logging.DEBUG)
specter_mock = MagicMock()
specter_mock.config = {"services": {}}
flaskapp_mock = Flask(__name__)
flaskapp_mock.config["EXTENSION_LIST"] = [
"cryptoadvance.specter.services.swan.service",
"cryptoadvance.specter.services.bitcoinreserve.service",
]
ctx = flaskapp_mock.app_context()
ctx.push()
# The ServiceManager is a flask-aware component. It will load all the services

View file

@ -16,7 +16,7 @@ from cryptoadvance.specter.services.service_encrypted_storage import (
ServiceEncryptedStorageError,
ServiceEncryptedStorageManager,
)
from cryptoadvance.specter.services.service_manager import ServiceManager
from cryptoadvance.specter.managers.service_manager import ServiceManager
from cryptoadvance.specter.user import User, hash_password

22
tests/test_util_common.py Normal file
View file

@ -0,0 +1,22 @@
from cryptoadvance.specter.util.common import camelcase2snake_case, str2bool
def test_str2bool():
assert not str2bool(None)
assert str2bool("true")
assert str2bool("True")
assert str2bool("tRuE")
assert not str2bool("false")
assert not str2bool("False")
assert not str2bool("fAlsE")
assert str2bool("On")
assert str2bool("oN")
assert str2bool("ON")
assert not str2bool("Off")
assert not str2bool("oFF")
assert not str2bool("OFF")
def test_camelcase2snake_case():
assert camelcase2snake_case("Service") == "service"
assert camelcase2snake_case("DeviceType") == "device_type"

View file

@ -1,13 +1,14 @@
import logging
from pathlib import Path
from cryptoadvance.specter.util.reflection import (
get_subclasses_for_class,
get_subclasses_for_clazz,
get_classlist_of_type_clazz_from_modulelist,
_get_module_from_class,
get_package_dir_for_subclasses_of,
)
from cryptoadvance.specter.util.specter_migrator import SpecterMigration
from cryptoadvance.specter.util.migrations.migration_0000 import SpecterMigration_0000
from cryptoadvance.specter.services.service_manager import Service
from cryptoadvance.specter.managers.service_manager import Service
from cryptoadvance.specter.services.swan.service import SwanService
from cryptoadvance.specter.services.bitcoinreserve.service import BitcoinReserveService
@ -32,10 +33,23 @@ def test_get_package_dir_for_subclasses_of():
)
def test_get_subclasses_in_packagedir(caplog):
caplog.set_level(logging.INFO)
classlist = get_subclasses_for_class(SpecterMigration)
assert SpecterMigration_0000 in classlist
classlist = get_subclasses_for_class(Service)
def test_get_classlist_from_importlist(caplog):
caplog.set_level(logging.DEBUG)
modulelist = [
"cryptoadvance.specter.services.swan.service",
"cryptoadvance.specter.services.bitcoinreserve.service",
]
classlist = get_classlist_of_type_clazz_from_modulelist(Service, modulelist)
assert len(classlist) == 2 # Happy to remove that at some point
assert SwanService in classlist
assert BitcoinReserveService in classlist
def test_get_subclasses_for_class(caplog):
caplog.set_level(logging.INFO)
classlist = get_subclasses_for_clazz(SpecterMigration)
assert SpecterMigration_0000 in classlist
classlist = get_subclasses_for_clazz(Service)
assert len(classlist) == 2 # Happy to remove that at some point
assert SwanService in classlist
assert BitcoinReserveService in classlist