Releasing procedure with pip (#69)

* automate pip-packages-gen + upload for gittags
release documentation
proper package including and  static + templates

* Change import- and start  logic, removed cli.py

* project-renaming to cryptoadvance.specter
propoer packaging
include proper dependencies

* Adding requirements.txt

* Specify maintainers and thanks to contributors

* Removing test-url for pip-uload. This is now live!

.gitlab-ci.yml

@@ -13,6 +13,10 @@ cache:
     - .cache/pip
     - .env/
 
+stages:
+  - testing
+  - releasing
+
 before_script:
   - docker info # Print out docker version for debugging
   - python -V  # Print out python version for debugging
@@ -23,6 +27,7 @@ before_script:
   - source .env/bin/activate
 
 test:
+  stage: testing
   script:
     - pip3 install -r requirements.txt
     - pip3 install -e .
@@ -30,3 +35,21 @@ test:
     # - python3 tests/conftest.py 
     - pytest --docker
 
+release:
+  stage: releasing
+  only: 
+    - tags
+  script:
+    - pip3 install setuptools wheel twine
+    # verifying the version number follows vx.y.z (e.g. "v1.2.3")
+    - if ! [[ $CI_COMMIT_TAG =~ ^v[0-9]{1,2}\.[0-9]{1,2}\.[0-9]{1,2}$ ]]; then exit 1; fi
+    # set version number in setup.py
+    - echo Releasing $CI_COMMIT_TAG
+    - sed -i "s/version=\".*/version=\"$CI_COMMIT_TAG\",/" setup.py
+    - cat setup.py
+    - python3 setup.py sdist bdist_wheel
+    # twine ready the password from the env-var TWINE_PASSWORD
+    # Add --repository-url https://test.pypi.org/legacy/ for testing the release procedure
+    - ls -l dist && python3 -m twine upload --verbose --user __token__ dist/*
+
+

CONTRIBUTING.md

@@ -0,0 +1,12 @@
+Specter-desktop is an Open Source Project under the MIT-License and everyone is invited to contribute to it.
+
+We haven't created many explicit processes and rely on the best practices of Open Source projects. If you want to contribute, fork the project and create a PR. 
+
+If it's necessary to add processes, we'll probably looking into Pieter Hintjens' [Social Architecture](https://hintjens.gitbooks.io/social-architecture/content/) and specifically the C4 process. Pieter is excplicitely mentioning two roles: Contributors and maintainers. 
+
+Thank you very much to all our [Contributors](https://github.com/cryptoadvance/specter-desktop/graphs/contributors). 
+We're planning to mention individual contributors on the release-notes of each new release.
+
+The maintainers are the once who are able to merge PRs and create tags/releases. They are listed as "authors" in setup.py.
+
+For practical considerations of a dev-setup, please have a look in the [DEVELOPMENT.md](./DEVELOPMENT.md). If you need support, join our  [Telegram group](https://t.me/spectersupport). 

DEVELOPMENT.md

@@ -25,7 +25,7 @@ Developing against a bitcoind-API makes most sense with the [Regtest Mode](https
 In order to make the "docker-way" even easier, there is a python-script which detects a running-docker-bitcoind and/or is booting one up. Use it like this:
 
 ```
-python3 src/specter/cli.py bitcoind
+python3 -m cryptoadvance.specter bitcoind
 ```
 
 This will also:

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2019 cryptoadvance
+Copyright (c) 2020 cryptoadvance
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

MANIFEST.in

@@ -0,0 +1,3 @@
+recursive-include src/cryptoadvance/specter/templates *
+recursive-include src/cryptoadvance/specter/static *
+include requirements.txt

README.md

@@ -30,13 +30,14 @@ cd specter-desktop
 virtualenv --python=python3 .env
 source .env/bin/activate
 pip3 install -r requirements.txt
+pip3 install -e .
 ```
 
 Run the server:
 
 ```
 cd specter-desktop
-python3 src/specter/server.py
+python3 -m cryptoadvance.specter server
 ```
 
 If your Bitcoin Core is using a default data folder the app should detect it automatically. If not, consider setting `rpcuser` and `rpcpassword` in the `bitcoin.conf` file and in the app settings.

docs/continuous-integration.md

@@ -38,7 +38,19 @@ Travis-CI setup is very straightforward. As we're using the build-cache, the bit
 
 # Releasing
 
-We're not yet ready to release (semi-) automatically. The current release-artifact is based on pyinstaller. To create the pyinstaller-artifact:
+## pip-based release to pypi
+
+We're about to release (semi-) automatically. The relevant release-artifact is a pip-package which will get released to pypi.org. A manual description of how to create this kind of releases can be found [here](https://packaging.python.org/tutorials/packaging-projects/). 
+
+The automation of that kicks in if someone creates a tag which is named like "vX.Y.Z". This is specified in the gitlab-ci.yml. The release-job will only be triggered in cases of tags. One step will also check that the tag follows the convention above.
+The package upload will need a token. How to obtain the token is described in the packaging-tutorial. It's injected via gitlab-variables. ToDo: put the token on a trusted build-node.
+
+The alternative would have been to use travis-ci for releasing. In that case we would encrypt the token with a private-key from travis and commit to the repo. This looks more safe to me then the above scenario but less safe then the todo, where we're storing the token on the build-node.
+
+
+## Old pyinstaller based releases
+The old pyinstaller based artifact will be kept here for the reference:
+(attention, hirarchy changed, so below won't work ootb)
 ```
 $ pyinstaller --onefile  --clean --paths .env/lib/python3.7/site-packages:src/specter  --add-data 'src/specter/templates:templates' --add-binary '.env/bin/hwi:.'  --add-data 'src/specter/static:static' src/specter/server.py
 ```

setup.py

@@ -1,9 +1,36 @@
-from setuptools import setup, find_packages
 from glob import glob
 
+from setuptools import find_packages, setup
+
+with open('requirements.txt') as f:
+    install_reqs = f.read().strip().split('\n')
+
+
+reqs = [str(ir) for ir in install_reqs if not ir.startswith("#") ]
+
+
+with open("README.md", "r") as fh:
+    long_description = fh.read()
 
 setup(
-    name="specter-desktop", 
+    name="cryptoadvance.specter",
-    packages=find_packages('src/specter'),
+    version="v0.0.11",
-    package_dir={'': 'src/specter'}
+    author="Stepan Snigirev, Kim Neunert",
-)
+    author_email="snigirev.stepan@gmail.com, kim.neunert@gmail.com",
+    description="A GUI for Bitcoin Core optimised to work with airgapped hardware wallets",
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url="https://github.com/cryptoadvance/specter-desktop",
+    packages=find_packages('src'),
+    package_dir={'': 'src'},
+    # take METADATA.in into account, include that stuff as well (static/templates)
+    include_package_data=True,
+    install_requires=reqs,
+    classifiers=[
+        "Programming Language :: Python :: 3",
+        "License :: OSI Approved :: MIT License",
+        "Operating System :: OS Independent",
+        "Framework :: Flask",
+    ],
+    python_requires='>=3.6',
+)

src/cryptoadvance/specter/__main__.py

@@ -1,26 +1,48 @@
-''' Stuff to control a bitcoind-instance. Either directly by access to a bitcoind-executable or
-    via docker.
-'''
-import os
-import sys
 import atexit
 import logging
-import shutil
+import os
-import subprocess
+import sys
-import tempfile
 import time
 
 import click
-import docker
-from bitcoind import BitcoindDockerController
-from server import DATA_FOLDER
-from helpers import which, load_jsons
 
+import docker
+
+from .bitcoind import (BitcoindDockerController,
+                       fetch_wallet_addresses_for_mining)
+from .helpers import load_jsons, which
+from .server import DATA_FOLDER, create_app
+
+DEBUG = True
 
 @click.group()
 def cli():
     pass
 
+
+@cli.command()
+def server():
+    app = create_app()
+    # watch templates folder to reload when something changes
+    extra_dirs = ['templates']
+    extra_files = extra_dirs[:]
+    for extra_dir in extra_dirs:
+        for dirname, dirs, files in os.walk(extra_dir):
+            for filename in files:
+                filename = os.path.join(dirname, filename)
+                if os.path.isfile(filename):
+                    extra_files.append(filename)
+    
+    # Note: dotenv doesn't convert bools!
+    if os.getenv('CONNECT_TOR', 'False') == 'True' and os.getenv('TOR_PASSWORD') is not None:
+        import tor_util
+        tor_util.run_on_hidden_service(
+            app, port=os.getenv('PORT'), 
+            debug=DEBUG, extra_files=extra_files
+        )
+    else:
+        app.run(port=os.getenv('PORT'), debug=DEBUG, extra_files=extra_files)
+
 @cli.command()
 @click.option('--debug/--no-debug', default=False)
 @click.option('--mining/--no-mining', default=True)
@@ -73,19 +95,6 @@ def bitcoind(debug,mining, docker_tag):
                 click.echo("    --> ",nl=False)
             time.sleep(mining_every_x_seconds)
 
-def fetch_wallet_addresses_for_mining(data_folder=None):
-    ''' parses all the wallet-jsons in the folder (default ~/.specter/wallets/regtest)
-        and returns an array with the addresses 
-    '''
-    if data_folder == None:
-        data_folder = os.path.expanduser(DATA_FOLDER)
-    wallets = load_jsons(data_folder+"/wallets/regtest")
-    address_array = [ value['address'] for key, value in wallets.items()]
-    # remove duplicates
-    address_array = list( dict.fromkeys(address_array) )
-    return address_array
-
-
 
 
 if __name__ == "__main__":

src/cryptoadvance/specter/bitcoind.py

@@ -3,14 +3,18 @@
 '''
 import atexit
 import logging
+import os
 import shutil
 import subprocess
 import tempfile
 import time
 
 import docker
-from rpc import BitcoinCLI, RpcError
+
-from helpers import which
+from .helpers import which
+from .server import DATA_FOLDER
+from .rpc import BitcoinCLI, RpcError
+from .helpers import load_jsons
 
 
 class Btcd_conn:
@@ -294,3 +298,16 @@ class BitcoindDockerController(BitcoindController):
             i = i + 1
             if i > 20:
                 raise Exception("Timeout while starting bitcoind-docker-container!")
+
+
+def fetch_wallet_addresses_for_mining(data_folder=None):
+    ''' parses all the wallet-jsons in the folder (default ~/.specter/wallets/regtest)
+        and returns an array with the addresses 
+    '''
+    if data_folder == None:
+        data_folder = os.path.expanduser(DATA_FOLDER)
+    wallets = load_jsons(data_folder+"/wallets/regtest")
+    address_array = [ value['address'] for key, value in wallets.items()]
+    # remove duplicates
+    address_array = list( dict.fromkeys(address_array) )
+    return address_array

src/cryptoadvance/specter/controller.py

@@ -7,11 +7,11 @@ from threading import Thread
 from flask import Flask, Blueprint, render_template, request, redirect, jsonify
 from flask_qrcode import QRcode
 
-from helpers import normalize_xpubs, run_shell
+from .helpers import normalize_xpubs, run_shell
-from descriptor import AddChecksum
+from .descriptor import AddChecksum
-from rpc import BitcoinCLI, RPC_PORTS
+from .rpc import BitcoinCLI, RPC_PORTS
 
-from logic import Specter, purposes, addrtypes
+from .logic import Specter, purposes, addrtypes
 from datetime import datetime
 import urllib
 

src/cryptoadvance/specter/logic.py

@@ -1,12 +1,15 @@
-from rpc import BitcoinCLI, RPC_PORTS, autodetect_cli
-import os, json, copy
-from helpers import deep_update, load_jsons
-from collections import OrderedDict
-from descriptor import AddChecksum
 import base64
-from serializations import PSBT
+import copy
-import helpers
+import json
+import os
 import random
+from collections import OrderedDict
+
+from . import helpers
+from .descriptor import AddChecksum
+from .helpers import deep_update, load_jsons
+from .rpc import RPC_PORTS, BitcoinCLI, autodetect_cli
+from .serializations import PSBT
 
 WALLET_CHUNK = 5
 

src/cryptoadvance/specter/server.py

@@ -7,14 +7,13 @@ from dotenv import load_dotenv
 from flask import Flask
 from flask_qrcode import QRcode
 
-from descriptor import AddChecksum
+from .descriptor import AddChecksum
-from logic import Specter
+from .logic import Specter
-from views.hwi import hwi_views
+from .views.hwi import hwi_views
 
 env_path = Path('.') / '.flaskenv'
 load_dotenv(env_path)
 
-DEBUG = True
 
 def create_app():
     if getattr(sys, 'frozen', False):
@@ -34,7 +33,7 @@ def create_app():
     app.specter = specter
     app.register_blueprint(hwi_views, url_prefix='/hwi')
     with app.app_context():
-        import controller
+        from . import controller
     return app
 
 
@@ -49,31 +48,4 @@ SINGLE_TYPES = {
     "legacy": "P2PKH",
     "p2sh-segwit": "P2SH_P2WPKH",
     "bech32": "P2WPKH"
-}
+}
-
-
-
-
-############### startup ##################
-
-if __name__ == '__main__':
-    app = create_app()
-    # watch templates folder to reload when something changes
-    extra_dirs = ['templates']
-    extra_files = extra_dirs[:]
-    for extra_dir in extra_dirs:
-        for dirname, dirs, files in os.walk(extra_dir):
-            for filename in files:
-                filename = os.path.join(dirname, filename)
-                if os.path.isfile(filename):
-                    extra_files.append(filename)
-
-    # Note: dotenv doesn't convert bools!
-    if os.getenv('CONNECT_TOR', 'False') == 'True' and os.getenv('TOR_PASSWORD') is not None:
-        import tor_util
-        tor_util.run_on_hidden_service(
-            app, port=os.getenv('PORT'), 
-            debug=DEBUG, extra_files=extra_files
-        )
-    else:
-        app.run(port=os.getenv('PORT'), debug=DEBUG, extra_files=extra_files)

src/cryptoadvance/specter/views/hwi.py

@@ -5,7 +5,7 @@ import subprocess
 import sys
 
 from flask import Flask, Blueprint, render_template, request, redirect, jsonify, current_app
-from helpers import normalize_xpubs, convert_xpub_prefix, which
+from ..helpers import normalize_xpubs, convert_xpub_prefix, which
 from hwilib import commands as hwilib_commands
 from hwilib import base58
 

tests/conftest.py

@@ -9,9 +9,8 @@ import time
 
 import pytest
 import docker
-from bitcoind import BitcoindDockerController, BitcoindPlainController
+from cryptoadvance.specter.bitcoind import BitcoindDockerController, BitcoindPlainController
-
+from cryptoadvance.specter.logic import Specter, DeviceManager
-from logic import Specter, DeviceManager
 
 
 def pytest_addoption(parser):

tests/test_bitcoind.py

@@ -1,9 +1,8 @@
 import logging
 
-
 def test_bitcoinddocker_running(caplog):
     caplog.set_level(logging.DEBUG)
-    from bitcoind import BitcoindDockerController
+    from cryptoadvance.specter.bitcoind import BitcoindDockerController
     my_bitcoind = BitcoindDockerController(rpcport=18999) # completly different port to not interfere
     #assert my_bitcoind.detect_bitcoind_container() == True
     rpcconn = my_bitcoind.start_bitcoind(cleanup_at_exit=True)

tests/test_cli.py

@@ -1,4 +1,4 @@
-from cli import fetch_wallet_addresses_for_mining
+from cryptoadvance.specter.bitcoind import fetch_wallet_addresses_for_mining
 
 def test_fetch_wallet_addresses_for_mining(wallets_filled_data_folder):
     # Todo: instantiate a specter-testwallet

tests/test_helpers.py

@@ -1,5 +1,5 @@
 def test_load_jsons():
-    import helpers
+    import cryptoadvance.specter.helpers as helpers
     mydict = helpers.load_jsons("./tests/helpers_testdata")
     assert mydict["some_jsonfile"]["blub"] == "bla"
     assert mydict["some_other_jsonfile"]["bla"] == "blub"
@@ -18,7 +18,7 @@ def test_load_jsons():
     # os.remove(mydict["ID123"]['fullpath'])
 
 def test_which():
-    import helpers
+    import cryptoadvance.specter.helpers as helpers
     try:
         helpers.which("some_non_existing_binary")
         assert False, "Whould raise an Exception"

tests/test_rpc.py

@@ -1,6 +1,6 @@
 import pytest
 
-from rpc import BitcoinCLI, RpcError
+from cryptoadvance.specter.rpc import BitcoinCLI, RpcError
 
 def test_BitcoinCli(bitcoin_regtest):
     brt = bitcoin_regtest # stupid long name

tests/test_specter.py

@@ -4,8 +4,8 @@ import shutil
 
 import pytest
 
-from rpc import RpcError
+from cryptoadvance.specter.rpc import RpcError
-from logic import (Device, DeviceManager, Specter, Wallet, WalletManager,
+from cryptoadvance.specter.logic import (Device, DeviceManager, Specter, Wallet, WalletManager,
                      alias)