Deploying an Electrum Buildpack to Heroku
Long story short, I had an idea for a project but no real time to make it happen. Rather than let my initial research go to waste, here are the steps I took to get the Electrum Bitcoin client running on Heroku.
Warning##
Storing your Bitcoin private keys anywhere that is accessible to the Internet is dangerous! Please take caution when implementing the steps read here. I am not responsible for coins lost as a result of running electrum in the cloud.
I will talk about proper architecture on here in the future but always have as few coins as operationally possible in online 'hot' wallets.
I should also point out that I am just a hobbyist programmer. I managed to get this working but that doesn't mean I did it 'the right way'. Herokuists may cringe while reading through this.
What is a buildpack?###
A buildpack is used in the Heroku world to setup your worker agent's local environment (also known as a 'slug'). If no buildpack is installed during deployment, you are basically just left with a barebones OS - incapable of doing all that much.
Most of the time, the buildpack your application needs is automatically detected and ran during deployment. For example, if you pushed a Ruby application, the deployment process would detect the Ruby code and install heroku-buildpack-ruby, giving the slug the ability to run ruby code.
Buildpacks are not solely for adding language support though. This same system can be leveraged for installing other services such as database servers or messaging queue.
In short, a buildpack is a set of shell scripts that bootstrap the slug so that the deployed application actually has all of the prerequisites available when it goes to execute.
Why Electrum and not Bitcoind?###
Electrum has a number of advanced features, which are awesome to have, although in this case they were irrelevant to my motives. Electrum was chosen because it doesn't need to store the entire blockchain locally, it can query the details it needs from external services.
The filesystem on a slug is completely volatile and anything stored within a worker may be lost at any time due to redeployments, restarts and scaling changes. To have bitcoind deploy as a worker, would mean 14GB of blockchain needs to somehow show up on the slug every deployment. Alternatively, storing the blockchain external of the worker agent would mean using a service such as Amazon S3 ($$$).
Writing the buildpack###
If you don't care about how the buildpack is put together, you can skip this step all together and go straight to including this buildback in your project.
To start my buildpack, I setup a directory with the files needed by Heroku.
> mkdir buildpack-electrum && cd buildpack-electrum
> mkdir bin
> touch bin/compile
> touch bin/release
> touch bin/detect
Simple enough! Now we need to write these files.
buildpack-electrum/bin/compile
#!/usr/bin/env bash
BUILD_DIR=$1
export PATH=$BUILD_DIR/.heroku/python/bin:$PATH
# Without this, pip will not be found
ln -s $BUILD_DIR/.heroku /app/
ls -al /app/.heroku
echo "-----> Installing electrum"
$BUILD_DIR/.heroku/python/bin/pip install https://download.electrum.org/Electrum-1.9.7.tar.gz#md5=5764f38d6e4bc287a577c8d16e797882
echo "-----> Electrum has been installed!"
touch $1/.heroku/electrumed
echo "-----> Wallet Not Configured..."
This script is the heart of the buildpack and, even so, is fairly straightforward. As you can hopefully see, the main action is the pip call, which actually installs the electrum client.
There are a couple tricks within the above though. Particularly regarding the fact that pip needs to be installed prior to this buildpack or it will completely fail during deployment. Finding where pip was stored was also a bit of a chore but that fun is done now.
Also note the last line, there is no configuration of wallets that take place during the deployment. Your worker will need to handle this part.
buildpack-electrum/bin/release
#!/usr/bin/env bash
# bin/release <build-dir>
# I didn't really feel the need to add anything here.
# This script can be used to start services but in our
# case electrum is just an application that gets called
# later by our worker.
buildpack-electrum/bin/detect
#!/bin/sh
# bin/detect <build-dir>
# Similarly, not much to detect here prior to the buildpacks install
# Without the following line, the buildpack deployment will fail
exit 0
That's pretty much it for the buildpack. Now we just need to push it somewhere that Heroku can access so that it can clone.
> git init
> git add .
> git commit -am "The first commmit of my electrum buildpack"
> git remote add origin https://github.com/<your username>/<proj name>.git
> git push -u origin master
Creating your application###
Now we have our electrum buildpack, either through creating your own or by using the one I have already deployed to git://github.com/Abstrct/buildpack-electrum.git. We can now build our own application that will use the electrum client.
Just like we did above, let's setup a new project workspace.
> mkdir hot-electrum && cd hot-electrum
> git init
> heroku create hot-electrum
> heroku config:add BUILDPACK_URL=https://github.com/ddollar/heroku-buildpack-multi.git
That last command is one of the most important so far. Since Electrum is actually a python application, we need to install python to the slug before we are able to get Electrum on there. Unfortunately, there can only be one buildpack specified in the BUILDPACK_URL config var.
To resolve this issue, we tell Heroku that our new application needs to the the heroku-buildpack-multi buildpack. This will allow us to define multiple buildpacks in the next step.
Alternatively, we could have forked the heroku-buildpack-python codebase and then added in the elctrum details along with it, but that seems short sighted. Any time changes are made to the python buildpack, we would need to rewrite the electrum buildpack to include the new code.
Since we are using the heroku-buildpack-multi package, we need to create a file listing all the buildpacks we need installed.
hot-electrum/.buildpacks
https://github.com/heroku/heroku-buildpack-python.git
git://github.com/Abstrct/buildpack-electrum.git
Now we should quickly write up our Procfile so that Heroku knows how to run our application after deployment.
hot-electrum/Procfile
worker: python worker.py
And finally, let's quickly write that worker.py file.
hot-electrum/worker.py
import os
quit()
Realistically, we would be writing our python code to setup the wallet, handle lookup requests, craft transactions, etc, etc. But this is where I ran out of time. So let's cut to the important part, confirming that we have electrum running inside Heroku!
> git add .
> git commit -am "Heelllllllo Electrum!"
> git push heroku master
Fetching repository, done.
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 285 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
-----> Fetching custom git buildpack... done
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-python.git
=====> Detected Framework: Python
-----> No runtime.txt provided; assuming python-2.7.4.
-----> Using Python runtime (python-2.7.4)
-----> Installing dependencies using Pip (1.3.1)
You must give at least one requirement to install (see "pip help install")
=====> Downloading Buildpack: git://github.com/Abstrct/buildpack-electrum.git
=====> Detected Framework:
lrwxrwxrwx 1 u43165 43165 55 Feb 15 22:23 /app/.heroku -> /tmp/build_d6e9cae8-5be8-4bd6-b11f-16eec492bc35/.heroku
-----> Installing electrum
Downloading/unpacking https://download.electrum.org/Electrum-1.9.7.tar.gz
Running setup.py egg_info for package from https://download.electrum.org/Electrum-1.9.7.tar.gz
Including all files
Downloading/unpacking slowaes (from Electrum==1.9.7)
Downloading slowaes-0.1a1.tar.gz
Running setup.py egg_info for package slowaes
Requirement already satisfied (use --upgrade to upgrade): ecdsa>=0.9 in /tmp/build_d6e9cae8-5be8-4bd6-b11f-16eec492bc35/.heroku/python/lib/python2.7/site-packages (from Electrum==1.9.7)
Installing collected packages: slowaes, Electrum
Running setup.py install for slowaes
Running setup.py install for Electrum
Including all files
changing mode of build/scripts-2.7/electrum from 600 to 755
changing mode of /app/.heroku/python/bin/electrum to 755
Successfully installed slowaes Electrum
Cleaning up...
-----> Wallet Not Configured!
Using release configuration from last framework :
-----> Discovering process types
Procfile declares types -> worker
-----> Compressing... done, 27.5MB
-----> Launching... done, v44
http://hot-electrum.herokuapp.com deployed to Heroku
To git@heroku.com:hot-electrum.git
850cc9a..28b59d1 master -> master
Deployment seems to have gone well. Now we can login to a slug
> heroku run bash
Running `bash` attached to terminal... up, run.1980
~ $ electrum --help
Usage: %prog [options] command [options]
Options:
-h, --help show this help text
-g GUI, --gui=GUI User interface: qt, lite, gtk, text or stdio
-w WALLET_PATH, --wallet=WALLET_PATH
wallet path (default: electrum.dat)
-o, --offline remain offline
-C, --concealed don't echo seed to console when restoring
-a, --all show all addresses
-l, --labels show the labels of listed addresses
-f TX_FEE, --fee=TX_FEE
set tx fee
-F FROM_ADDR, --fromaddr=FROM_ADDR
set source address for payto/mktx. if it isn't in the wallet, it will ask for the private
key unless supplied in the format public_key:private_key. It's not saved in the wallet.
-c CHANGE_ADDR, --changeaddr=CHANGE_ADDR
set the change address for payto/mktx. default is a spare address, or the source address if
it's not in the wallet
-s SERVER, --server=SERVER
set server host:port:protocol, where protocol is either t (tcp), h (http), s (tcp+ssl), or g
(https)
-p PROXY, --proxy=PROXY
set proxy [type:]host[:port], where type is socks4,socks5 or http
-v, --verbose show debugging information
-P, --portable portable wallet
-L LANGUAGE, --lang=LANGUAGE
defaut language used in GUI
-u, --usb Turn on support for hardware wallets (EXPERIMENTAL)
-G GAP_LIMIT, --gap=GAP_LIMIT
gap limit
-W PASSWORD, --password=PASSWORD
set password for usage with commands (currently only implemented for create command, do not
use it for longrunning gui session since the password is visible in /proc)
-1, --oneserver connect to one server only
Type 'electrum help <command>' to see the help for a specific command
Type 'electrum --help' to see the list of options
List of commands: contacts, create, createmultisig, createrawtransaction, decoderawtransaction, deseed, dumpprivkey, dumpprivkeys, freeze, getaddressbalance, getaddresshistory, getbalance, getconfig, getmpk, getpubkeys, getrawtransaction, getseed, getservers, getversion, help, history, importprivkey, listaddresses, listunspent, mksendmanytx, mktx, password, payto, paytomany, restore, sendrawtransaction, setconfig, setlabel, signmessage, signrawtransaction, unfreeze, validateaddress, verifymessage
Success!
Now, I really feel the need to say this again. If you create a wallet on this instance, and you do not back it up, any bitcoin being stored by this wallet should be considered gone.
Maybe I should say this again. If you run heroku run bash and you create a new wallet. This wallet will be destroyed when you exit the terminal. If Bitcoin are stored in this wallet, and there isn't a backup, then the Bitcoin is as good as gone.
As a final thought, I just want to make this a bit more useful in the real world. Let's make a couple quick changes to the worker.py script, so that it creates a wallet locally, based off of heroku environment variables.
First, set the variables. These variables are generally considered to be pretty secure. I can't speak for Heroku really, but the idea is that you should use these so that static passwords/values are not stored directly in your codebase - which sounds like a good plan to me.
I had already created an electrum wallet, using electrum create and this is the seed that it generated. This is a string of words that you should NEVER tell anyone - my display of them is for training purposes only.
> heroku config:set WALLET_SEED="fur sky unable grandma shove help cross scent minute dumb suicide power"
Now let's edit worker.py to actually generate this wallet during deployment.
hot-electrum/worker.py
import os
from multiprocessing import Queue
from subprocess import PIPE, Popen
try:
p = Popen(['electrum', 'restore'], stdin=PIPE, shell=False)
# Skip password protecting the wallet
p.stdin.write('\n')
# Set the default fee for transactions
p.stdin.write('0.0002' + '\n')
# The number of addresses to generate and add to the wallet
p.stdin.write('5' + '\n')
# The seed used to generate the address/key pairs
p.stdin.write(os.environ['WALLET_SEED'] + '\n')
p.stdin.close()
except:
quit()
quit()
This really doesn't do all that much still, just sets up the environment. From here, you would likely want to start working with a queue service so that your web app can send commands back to this worker such as transactions, balance lookups, new address requests, etc.