Automated Deployment or Automated Build tools are all complicated to give a true evaluation. Before you even try to evaluate the tools, working through the complete process manually is normally the best place to start. You need to understand how the deployment process works before you can figure out what tool will meet your needs. Things to take note of is the tests you will need to run, the operating systems involved, and how you plan to do the deployments to production. You are trying to achieve consistency across all your environments so deploying a tool designed for web applications may not have the features you need to deploy smart phone apps. This isn't to say that most of the tools can't do both just that you need to probe deeply to make sure all your requirements are met.
Below is my top list of things to look at when choosing a tool of this class. As with all the other articles on tools I have written this list probably isn't complete. You will need to assure that you are meeting all of the "it depends" requirements for your environment. Taking a disciplined approach in choosing the tool can save you months of time trying to configure it for your companies needs.
- *Do you need to host the tool yourself or can you use one of the many PaaS options available?
Tools like TravisCI are wonderful tools which will let you scale your costs of the tool and infrastructure as your company grows. However, if you are an established company with a lot of existing applications it may be cost prohibitive. So be sure to look at the cost of the solution before committing to the cloud.
- How are you going to test your applications or infrastructure as part of this process?
Does the tool support all of your testing tools? Or do they support it? Some testing tools are better than others at inter-operating with this class of tools than others. In some cases, PaaS options most notably, may not be an option because you can't connect to one or more pieces of infrastructure in your test environment.
- * Does the tool have hooks into your version control system that will adequately support your needs?
This is less and less of a problem with most systems but it should be validated before a final decision is made. If you use SVN and the tool only supports GIT it's going to knock it out completely. Also where does the code have to be stored? If it only supports GitHub and Bitbucket but you store your code in a local gitlab server it is not going to work for you.
- * How much and what kinds of work flow do you need?
Some of these tools are only designed to do the most basic of work flows like push the code to the next environment. Others can do what would be considered full orchestration like building complete systems from scratch. From a DevOps perspective not being able to do full orchestration is an issue. We want the system to never need human interaction. So only doing things half way isn't helping us to reach the goal. Luckily most of the systems while they may not directly support full orchestration can call on other tools like Chef via Build/Deploy scripts to accomplish the task. At the same time if you are only going to be developing desktop and smart phone apps a manual process may be your only option. Not all distribution mechanisms have API's and ways to automatically deploy them. While these are becoming more rare every day it's something to keep in mind.
- What kinds of reporting do you need it to do?
This is normally a rather broad set of requirements. Generally they will create a web report of the progress of the current builds, last successful build, and last failed builds. What is contained in those reports can vary far more than you might expect. So be sure to pay attention to what the sample reports show you. Be sure to discuss this with everyone to make sure that all of the teams needs are being met.
- How much does the tool do out of the box versus needs to be developed to handle?
Most of the tools rely on at least some development to make the system meet all of your needs. This is generally in the form of a wrapper shell script of some type. Some tools however may require you write in things like Java, Python or Ruby. So make sure you understand what is included and supported versus capable but needs development.
- Once you have narrowed your choices do a pilot with your top one or two choices.
Piloting your top choices with an app or two will help you find it's rough spots. What sounds great on a website could be more marketing hype than reality. While I don't think any of the vendors or projects are trying to deceive us they just may not understand that having to learn Java, Python or Ruby may be a deal breaker for you. The setup on these tools can be time consuming. So allow for that time in your estimates. Your pilot will help you understand most of the full complexities the setup and configuration.
- Do you need Role based access control?
In some companies only certain people are allowed to push buttons to move to new environments. This could be for any number of reasons. One example is that production deployments may only be allowed after all the sign-offs have been given and the change manager blesses the build. Not all of the tools support this feature.
To get you started here are three examples I have used in the past:
- TravisCI - A cloud based CI suite which gives it's services away for free to FLOSS projects.
- Jenkins - An open source solution with tons of plugins and possibilities if your not afraid of a little scripting.
- UrbanCode's Anthill Pro - A closed source and not free tool recently acquired by IBM. This has a lot of out of the box integrations and a fully featured work flow engine that includes Role Based Access Control.