Meteor CI

Getting Started with Meteor Continuous Deployment

These explanatory scripts, videos and slideshows, will start your Meteor development infrastructure immediately with: package development, testing, documenting, logging, code style linting all on a platform of continuous integration, delivery and deployment.

Jump to Part #1 Watch Videos View on GitHub

Seriously?

Learn Meteor? Ok!
But ...
DevOps at the same time?!?


Learning new tools is exciting.
Meeting deadlines with them is hard.
Having to integrate, at the same time, the tools to automate the development workflow is a nightmare, yet ... retrofitting those tools to a developed project isn't much better.

If this is your situation, then these tutorials are for you.

Starting your development from the tools in this tutorial, you will immediately comply with the first four criteria of the Joel Test.

You'll be closer to meeting #5 & #10 as well. But it goes beyond that, to stuff that ought to be on the "Test" but isn't, like, um . . . generating documentation from code annotations?

The Joel Test

  1. Do you use source control?
  2. Can you make a build in one step?
  3. Do you make daily builds?
  4. Do you have a bug database?
  5. Do you fix bugs before writing new code?
  6. Do you have an up-to-date schedule?
  7. Do you have a spec?
  8. Do programmers have quiet working conditions?
  9. Do you use the best tools money can buy?
  10. Do you have testers?
  11. Do new candidates write code during their interview?
  12. Do you do hallway usability testing?

Components

Can you get all these to work together?

Quality is free!

But getting there can be daunting.

The goal of this tutorial series is to help Meteor developers put together a complete continuous deployment work environment from which to build out :

  • Package-only Apps
  • Logging
  • Version Control
  • Generated Documentation
  • Unit Tests
  • End-to-End Tests
  • Code Style Verification
  • Deployment to Web
  • Deployment to Android


The focus is on 'wiring', with very little discussion of what more one can do with each of the various elements once they're wired together -- there are plenty of tutorials about all of that, and I link to some good ones. 'Wiring' refers to the challenge of connecting a variety of different tools together into an integrated whole -- even if the whole doesn't actually do much to begin with.


Here's the complete "wiring diagram"...


CI work flow
Continuous Delivery Work Flow

Another goal is to help developers begin building Meteor applications as though they were working in multiple teams on multiple projects! In particular, this means going beyond "normal" configuration of ssh and git, to handle multiple user aliases and GitHub deploy keys. The end result will be fractionally more difficult in your daily routine, but much more flexible and "future proof".

No Fluff Here

You want to learn the most in the least time, yes?

I try to be very succinct, with every word dedicated to helping you get the most out of the tutorial in the least time.

In fact, to save you even more time, I generate code fragments, "on-the-fly", employing a naming scheme you choose, so you're getting more than a tutorial -- it's also a toolkit of boilerplate snippets. The end result is a complete work environment designed to elevate you quickly into a high-velocity developer with very low "maintenance drag".

Skim it to see if it's what you need, but to avoid problems later -- please do -- carefully follow all the instructions.

I hope to raise the bar on developer tutorials.

Please let me know if I have succeeded . . . or failed!

Typical Usage Pattern

Getting Directly to What You Need

I've found few tutorials that make it easy to jump directly to what I need, so I try to avoid that mistake here.

Most likely you'll want to flip through the slides to get an overview, then watch the videos, then spin up a fresh virtual machine and execute the scripts. Since the scripts are semi-automatic, you'll have very little typing to do, and so the whole thing shouldn't cost you more than a few hours.

In fact, grouped as it is, into spoon-sized doses, you can step through it a little bit at a time, when you have time.

You will NOT be able to complete this tutorial without these things :

  • A GitHub account with SSH keys set up: If you have not signed up, create a GitHub account (Free!) . If you do have a GitHub account you must also provide GitHub with your SSH key. Refer to their guide to generating SSH keys or troubleshoot common SSH Problems.
  • A Meteor account: If you aren't yet signed up you will have to create a Meteor account (Also free!) .
  • A CircleCI account: You are automatically signed up, for free!, to CircleCI when you have a GitHub account. When you go to CircleCI you just click the "Log in" button, and confirm to GitHub that you authorize CircleCI to access your repositories.
  • Running on a virtual machine: The scripts that make up the tutorial install a lot of utilities and programs. I have included a trap that tries to prohibit execution on a bare hardware machine. NEVER use these scripts on a machine you fear to lose! As an added incentive for you to NOT use the tutorial on real hardware, I created a (Yup! Free!) Docker image of Xubuntu and X2Go, and a video about how to start it up and use it for the tutorial. You just need to get Docker installed and running, which is easy (and, well ... free!)

Other stuff

  • Please note that the slides change with the keyboard or scroll wheel, but not mouse clicks. This is because there are "Hover Notes" and hyperlinks to deeper details that are problematic if the slides are also enabled for control by touchscreen taps or mouse click.
  • Except where indicated, the bash scripts make every attempt to be 'idempotent' -- you can run them repeatedly without adverse consequences. Nevertheless, you may want to be liberal with virtual machine snapshots of your work.
  • I have not been consistent with manual vs. automated. If the icon in the right margin says "No manual input required here." then it is automatic, otherwise there may be a combination of manual and automatic. Start and stop the video to see when you really do need to edit something and when a file is pulled in to do the work for you. In general, you're going to need to type when it's just a line or two, or when small edits are spread over several slides. If in doubt, look at the source code for the slide by clicking the link at the bottom.
  • The tutorial is designed for a small (10GB) Xubuntu 14.04 virtual machine (Docker and Qemu/KVM have been used, others should be fine, but haven't been tried). It hasn't been tested in any other environment since the idea is to provide clear proven instructions that can be adapted easily to other circumstances. Once you see how it works in one environment, adapting to a different one should be trivial.
  • LEGAL: THIS IS A FREE NON-COMMERCIAL TUTORIAL WITH NO GUARANTEE OF ANY KIND. Use at your own risk! Run the tutorial on a disposable, preferably virtual, machine. Do NOT run the tutorial on a machine with any sensitive or irreplaceable information.

Contents

Here are the Twelve Parts

Spin up a virtual machine [Video]

This is not strictly part of the tutorial, but I created a Docker image that should make this step very easy for you.

Prepare the Machine [Video] [Slide Show]

We start off with a fresh Ubuntu 14.04 LTS virtual machine and install all the platform dependencies. These are global installs requiring 'sudo'. Subsequent sections should not require 'sudo'. The script shown can run to completion without intervention, or stop at intermediate steps to view explanations.

Version Control In The Cloud [Video] [Slide Show]

Towards the goal of "packages-only-apps", we are mainly interested in version control of each distinct package, but we still need a 'do-nothing' app in which to instantiate the packages.

So, we'll prepare a Meteor project and store it in GitHub.

Create and Unit Test a Package [Video] [Slide Show]

This section shows the steps needed to begin testing a Meteor package.

We'll prepare a package, secure it in GitHub and run preliminary tests with TinyTest in the browser and on the command line.

Code Quality - Coding Style and Linting [Video] [Slide Show]

If you want reusable components, especially if you are working with one or several teams, it's very important to ensure consistent styles and documentation standards from all contributors to all your projects. In this and the next section we'll set up an environment where standards conformance is easy.

The first element is linting with esLint, run in the editor for instant feedback as well as on the command line so conformance can be published and monitored as part of continuous deployment.

Code Quality - Automatic Documentation in the Cloud [Video] [Slide Show]

The second part about code quality exploits two powerful tools: jsDoc and "GitHub Pages".

We'll put together a mechanism for easily documenting code files and then publishing them as a browsable manual, automatically, and hosted, free of charge, on GitHub.

Cloud Continuous Integration [Video] [Slide Show]

This section expands on your workstation toolbox to include working with continuous deployment in the cloud. It covers everything involved in setting up linting, automated documentation, unit testing, version control, end-to-end testing and continuous delivery in-the-cloud. So, while a working system is integrated and delivered continuously, it is not yet continuously deployed.

Code Quality - Production Logging [Video] [Slide Show]

A vital piece of application architecture is scaleable logging: run-time logs that track critical server-side activity at appropriate levels of detail. We "require" Bunyan, a toolkit for NodeJS that focuses on very flexible logging with JSON output, that can be readily archived and subsequently "mined", for example, for usage patterns.

A Real World Package [Video] [Slide Show]

Here we create a complete working, tested, documented Meteor package from a pre-exising NodeJS module. This step also introduces the "impedance mis-match" between the synchronous layer that Meteor builds on top of the asynchronous substrate of NodeJS.

Package Self Test and Publication [Video] [Slide Show]

To close the loop on all the preceding development stage parts we now bring the online documentation up-to-date and automate end-to-end testing of our package in-the-cloud; such that interested users can quickly get started using it.

Automated Deployment [Video] [Slide Show]

We build our project as an Android app, place it into our project as a download link, then deploy it to Meteor's servers. We use our understanding of that to complete our CI script so we can do it all automatically in CircleCI, every time we push a commit to GitHub.

Avoid the horrible Android emulator [Video]

You cannot run the Android emulator in the Docker machine we created earlier. It's very slow anyway. In this video I show how to install Android KitKat x86 in a Qemu/KVM virtual machine and then connect to it from Meteor in order to "run android-device". You get most of the advantages of the emulator and most of the advantages of connected physical devices, with the downside of neither.

First steps

Here's How to Get Started

To begin, you will need to obtain the tutorial.

Open up a terminal window, then copy'n paste these commands into it:

export BRANCH="master" # modularize
export URI="http://github.com/martinhbramwell"
export TUT="Meteor-CI-Tutorial"
export FILE="archive/${BRANCH}.zip"

echo "Downloading : \n${URI}/${TUT}/${FILE}"
echo "         as : ${TUT}.zip"

cd ${HOME}
wget -O ${TUT}.zip ${URI}/${TUT}/${FILE}
unzip ${TUT}.zip
mv ${TUT}-${BRANCH} ${TUT}
cd ${TUT}

# Ready to run the first script :
#             ./Tutorial01_PrepareTheMachine.sh

The result should look like this.

./Tutorial01_PrepareTheMachine.sh

After some preparatory steps, you will see a plain text version of the first slide of the "executable slideshow" in your terminal.

My idea, however, is it that you should follow the terminal action while viewing the slides in a browser in any one of the following ways:

  • Open your browser to the GitHub copy of the slideshow.
  • If you already know Meteor, you can just stuff a copy of this directory in the 'public' directory of a Meteor app and view the table of contents at http:// your host:3000/toc.html.
  • If you think you have Python in your machine, you can open a second terminal window and, into it, copy'n paste the following;

echo Your Python version is : $(python -c "import sys;t='{v[0]}.{v[1]}'.format(v=list(sys.version_info[:2]));sys.stdout.write(t)";).
export TUT="Meteor-CI-Tutorial"
cd ${HOME}/${TUT}
python -m SimpleHTTPServer 8080 &
echo -e "\n\n Please open your browser to http://localhost:8080/toc.html?part=A#1 \n\n";
fg

Source Links