Feature: new extensionpoint specter_persistence_callback (#1723)

* new extpoint specter_persistence_callback

* more docs and rendering of cli_ext

* unrelated documentation

* docs: update TOC

* adjusting error-handling

Co-authored-by: k9ert <k9ert@users.noreply.github.com>

docs/faq.md

@@ -35,7 +35,6 @@
   - [*Which hardware wallets are supported?*](#which-hardware-wallets-are-supported)
   - [*Can this also work with external nodes like Casa, MyNode, and Raspilitz?*](#can-this-also-work-with-external-nodes-like-casa-mynode-and-raspilitz)
   - [*Can I use Tor?*](#can-i-use-tor)
-  - [*How to set the URL for the block explorer?*](#how-to-set-the-url-for-the-block-explorer)
   - [I forgot my password, how can I reset it?](#i-forgot-my-password-how-can-i-reset-it)
 - [BACKING UP FUNDS](#backing-up-funds)
   - [*If something happens to the `~/.specter` folder, is it still possible to **restore** access to multisigs created there (assuming there is no backup of the `~/.specter` folder)?*](#if-something-happens-to-the-specter-folder-is-it-still-possible-to-restore-access-to-multisigs-created-there-assuming-there-is-no-backup-of-the-specter-folder)
@@ -195,7 +194,11 @@ There is a great tutorial [here](https://bitcoiner.guide/verifysoftware/) explai
 
 ## *Where do i find the logs?*
 
-There is a log-file called specter.log in the SPECTER_DATA_FOLDER in your user-directory. If you're having trouble finding that file in your harddrive, have a look at the tooltip in the settings/general/Loglevel item.
+If you use a binary-installation (a platform specific download-package) there is a log-file called `specterApp.log` in the SPECTER_DATA_FOLDER in your user-directory. So for different platforms, default places are: 
+* Windows: `C:\Users\YourUser\.specter\specterApp.log`
+* Linux: `/home/yourUser/.specter/specterApp.log`
+For pip installations, you need to look into the file `specter.log` in the same directory.
+If you still have trouble finding that file on your harddrive, have a look at the tooltip in the settings/general/Loglevel item.
 
 ## *What types of ways can I run specter-desktop?*
 
@@ -204,11 +207,11 @@ There are many ways how to run Specter:
 - Specter on a remote node, web interface in local network or over Tor (but hardware wallets need to be connected to the node where Specter is running)
 - Specter on a remote node, another Specter on your computer in "hwibridge" mode that gives access to your hardware wallets from the remote node (configurable whitelist)
 
-It depends on your setup, and can be customized accordingly.
+Also you can run it via a platform-specific binary, pip-installation or via a Docker-container. It depends on your needs, and can be customized accordingly.
 
 Specter-desktop makes many requests to Bitcoin Core RPC, so it works better from the same machine where Core is running, but remote is also possible. With that being said, by default Bitcoin Core RPC is connecting over HTTP, so everything including your RPC login and password are flying around as plaintext. You can use HTTPS and a [self-signed certificate](https://github.com/cryptoadvance/specter-desktop/blob/master/docs/self-signed-certificates.md) to fix that.
 
-If you use hardware wallets and they are usb-connected to specter-desktop then you should take these precautions, but if they are air-gapped (ColdCard, specter-diy, cobo) - then you can use remote web interface.
+If you use hardware wallets and they are usb-connected to specter-desktop then you need to use the hwibridge in the remote cases. However if your HWW is air-gapped (ColdCard, specter-diy, cobo) - then you can use remote web interface.
 
 ## Devices? Wallets? What is the difference?
 
@@ -222,13 +225,12 @@ XPUBs are needed (from HWW's, laptop with Electrum desktop wallet, Specter-DIY,
 
 ## *Is my understanding correct that specter-desktop does not hold any keys and you need to create a multisig wallet in order to sign transactions and send funds?*
 
-As of late, you can also use a hot wallet as a signer with specter-desktop, but since it is so new it is not recommended. You can however use devices like Electrum wallet or FullyNoded for example (Electrum or Bitcoin Core can be air-gapped). This [video](https://youtu.be/4YXklLh2srA) is quite useful for using Electrum, and this [guide](https://github.com/Fonta1n3/FullyNoded/blob/master/Docs/Connect-node.md#importing-a-wallet-from-specter) is useful for connecting with FullyNoded.
+As of late, you can also use a hot wallet as a signer with specter-desktop, but this is not recommended as a default setup. You can however use devices like Electrum wallet or FullyNoded for example (Electrum or Bitcoin Core can be air-gapped). This [video](https://youtu.be/4YXklLh2srA) is quite useful for using Electrum, and this [guide](https://github.com/Fonta1n3/FullyNoded/blob/master/Docs/Connect-node.md#importing-a-wallet-from-specter) is useful for connecting with FullyNoded.
 
 ## *How would one sign with Electrum? Do I need to create multisig wallet in Electrum first or can I create it with specter-desktop?*
 
 You need to create it in both wallets. When you start creating multisig wallet in Electrum it will give you the bech32 extended public key (ZPUB) where you can then add it to specter-desktop as well as other ZPUBS from other devices, and then add them to Electrum. After that you can start using Electrum as a signer.
-Full Electrum support is not out yet, but it should already work with files and copy paste of the transaction.
-See also this [video](https://www.youtube.com/watch?v=4YXklLh2srA) for more details.
+Electrum support got much better lately. We now have a dedicated electrum device with a specific explanation. If you're running into trouble, this [video](https://www.youtube.com/watch?v=4YXklLh2srA) might still help a lot and gives more details.
 
 ## *Can I use Ledger and ColdCard multisig while CC remains air-gapped?*
 
@@ -240,30 +242,27 @@ Yes you can use BlueWallet in watch-only mode and sign with Specter DIY. See it
 
 ## *Which hardware wallets are supported?*
 
-Any HWW with HWI, including USB HWW's (ColdCard, Trezor, Ledger, KeepKey, BitBox(2), CoboVault etc.)
+All major once! Checkout the list when you add a new device.
 
 ## *Can this also work with external nodes like Casa, MyNode, and Raspilitz?*
 
 Absolutely, as well as any other DIY bitcoin full, or pruned, node!
 
-Currently Raspiblitz (https://github.com/rootzoll/raspiblitz), has explicit support and you can automatically install it as bonus-software. Also [umbrel](https://getumbrel.com/) has it in the app-store. Mynode has it on Mynode [premium](https://mynodebtc.com/products/premium). There are differences mainly on update-policy and update-freuency.
+Currently Raspiblitz (https://github.com/rootzoll/raspiblitz), has explicit support and you can automatically install it as bonus-software. Also [umbrel](https://getumbrel.com/) has it in the app-store. Mynode has it on Mynode [premium](https://mynodebtc.com/products/premium). Start9 is currently preparing support. There are differences mainly on update-policy and update-freuency.
 
 ## *Can I use Tor?*
 
 Yes there is a way to access specter-desktop over Tor from outside, here is the [doc](https://github.com/cryptoadvance/specter-desktop/blob/master/docs/tor.md).
 
-In the new version v1.3.0, there will also be the possibility to activate a tor-installation from within Specter-Desktop.
+Since version v1.3.0, there will also be the possibility to activate a tor-installation from within Specter-Desktop.
 
 With that being said, beware that it's not practical yet to sign transactions via Tor:
 
  - Specter-DIY needs the camera which is not available in the Tor-browser (yet)
  - You could use HWI-wallets, but you would need to plug the wallet into the machine where specter-desktop is running on, but this is usually not the use-case you're looking for when using Tor.
 
-## *How to set the URL for the block explorer?*
+ Progress on Tor Support for QR-code scanning is tracked in this [issue](https://github.com/cryptoadvance/specter-desktop/issues/536)
 
-This feature is optional and not needed for the wallet to function. It's only used for convenience in order to generate URLs for addresses.
-Technically, you can use any block explorer but that's not what you want to do, unless you want to try out the feature.
-Simply fill in https://blockstream.info/ to use that block explorer, but you will leak privacy doing that.
 
 ## I forgot my password, how can I reset it?
 
@@ -365,6 +364,8 @@ This is a known issue. See [here](https://github.com/cryptoadvance/specter-deskt
 * Also, make sure to upgrade to the latest firmware, ledger but also others are known to not work with specific older versions. 
 * On Linux, there is also something called [udev-rules](../udev/README.md) which have to be installed.
 * Then, there might be confusion about the computer to plug it in. Do you run specter locally or on some remote-computer? Without the hwi-bridge, you need to plug your hardware wallet in the USB-port of the computer you're running specter on. If you want to use your computer and not the remote one, checkout the [HWIBridge](./hwibridge.md)
+* Also unplug other maybe exotic USB hardware like game-controllers, printers and basically everything, just for testing purposes.
+* If you're using the hwi-bridge, skip it for testing purposes and use specter locally to let it connect to your local 
 
 ## *How to upgrade Specter-desktop?*
 

docs/windows.md

@@ -1,6 +1,8 @@
 # Running Specter-Desktop on Windows
 
-1. Install Python 3.7 from Microsoft Store (python 3.8 is not supported yet). You may need to install [Visual C++ Redistributable](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads) for it.
+This description is for a pip-installation on windows. Regular Windows users please simply install the windows-specific binary. However having a pip-installation on windows might make sense for specific users:
+
+1. Install Python 3.9 from Microsoft Store. You may need to install [Visual C++ Redistributable](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads) for it.
 2. Install libusb:
 	- Download latest binary from https://libusb.info/ and extract
 	- For 32-bit Windows:
@@ -24,5 +26,3 @@ If you are running into problems with pip, you can install it via:
 - in Command Prompt navigate to the folder where this file is located
 - run `python3 get-pip.py`
 - now you can install specter-desktop from Command Prompt and run as usual
-
-*Note:* `--daemon` flag is not available on Windows yet

src/cryptoadvance/specter/cli/__init__.py

@@ -45,6 +45,8 @@ def entry_point(config_home, debug=False, tracerpc=False, tracerequests=False):
         # No need for timestamps while developing
         formatter = logging.Formatter("[%(levelname)7s] in %(module)15s: %(message)s")
         logging.getLogger("cryptoadvance").setLevel(logging.DEBUG)
+        # but not that chatty connectionpool
+        logging.getLogger("urllib3.connectionpool").setLevel(logging.INFO)
     else:
         formatter = logging.Formatter(
             # Too early to format that via the flask-config, so let's copy it from there:

src/cryptoadvance/specter/managers/service_manager.py

@@ -228,7 +228,9 @@ class ServiceManager:
             raise Exception(f"Non existing callback_id: {callback_id}")
         logger.debug(f"Executing callback {callback_id}")
         for ext in self.services.values():
-            if hasattr(ext, "callback"):
+            if hasattr(ext, f"callback_{callback_id}"):
+                getattr(ext, f"callback_{callback_id}")(*args, **kwargs)
+            elif hasattr(ext, "callback"):
                 ext.callback(callback_id, *args, **kwargs)
 
     @property

src/cryptoadvance/specter/persistence.py

@@ -14,6 +14,7 @@ import threading
 from flask import current_app as app
 
 from .specter_error import SpecterError
+from .services.callbacks import specter_persistence_callback
 from .util.shell import run_shell
 
 logger = logging.getLogger(__name__)
@@ -200,6 +201,27 @@ def storage_callback(mode="write", path=None):
     # Might be usefull to figure out why the callback has been triggered:
     # traceback.print_stack()
     # logger.debug(f"Storage Callback called mode {mode} with path {path}")
+
+    # First, call extensions which want to get informed.
+    # This is working synchronously!
+    try:
+        app.specter.service_manager.execute_ext_callbacks(
+            specter_persistence_callback, path=path, mode=mode
+        )
+    except AttributeError as e:
+        # chicken-egg poroblem:
+        if str(e).endswith("object has no attribute 'specter'"):
+            pass
+        else:
+            raise e
+    except RuntimeError as e:
+        if str(e).startswith("Working outside of application context"):
+            # not yet (?!) supported
+            pass
+        else:
+            raise e
+
+    # Now maybe we have async scripts?
     if os.getenv("SPECTER_PERSISTENCE_CALLBACK_ASYNC"):
         cmd_list = os.getenv("SPECTER_PERSISTENCE_CALLBACK_ASYNC").split(" ")
         cmd_list.append(mode)
@@ -209,6 +231,7 @@ def storage_callback(mode="write", path=None):
             args=(cmd_list,),
         )
         t.start()
+    # Or maybe even synchronous scripts?
     elif os.getenv("SPECTER_PERSISTENCE_CALLBACK"):
         cmd_list = os.getenv("SPECTER_PERSISTENCE_CALLBACK").split(" ")
         cmd_list.append(mode)

src/cryptoadvance/specter/services/callbacks.py

@@ -11,5 +11,21 @@
     
 """
 
+"""
+    I don't know why we have this one. Doesn't seem to be used anywhere.    
+"""
 afterServiceManagerInit = "afterServiceManagerInit"
+
+"""
+    This one is called, after the init_app method has finished. The "run" method has not 
+    been executed yet and so urls can't be called yet.
+    So this is the best place for almost all extensions to do their initializing work.
+"""
 after_serverpy_init_app = "after_serverpy_init_app"
+
+"""
+    This one is called, whenever a file is persisted. To call external scripts in another
+    process, you better use the SPECTER_PERSISTENCE_CALLBACK Env Var or it's asynchronous cousin
+    SPECTER_PERSISTENCE_CALLBACK_ASYNC.
+"""
+specter_persistence_callback = "specter_persistence_callback"

src/cryptoadvance/specter/services/extension_gen.py

@@ -110,10 +110,12 @@ class ExtGen:
             self.render(f"{package_path}/templates/dummy/components/dummy_menu.jinja")
             self.render(f"{package_path}/templates/dummy/components/dummy_tab.jinja")
 
+        self.render(f"pytest.ini", env=self.sd_env)
         self.render(f"tests/conftest.py", env=self.sd_env)
         self.render(f"tests/fix_ghost_machine.py", env=self.sd_env)
         self.render(f"tests/fix_devices_and_wallets.py", env=self.sd_env)
         self.render(f"tests/fix_testnet.py", env=self.sd_env)
+        self.render(f"tests/fix_keys_and_seeds.py", env=self.sd_env)
 
     def create_binary_file(self, sourcepath):
         """textfiles can all be rendered. Binaries must be wgettet or copied"""