The other question I see with CI tools (disclaimer: I work for a vendor) is how far do you want to go with CI? A major purpose of CI tooling is testing your builds after a code commit and letting everyone know how OK things are.
Are you just compiling? That's a great first test. Running unit tests? Even better. Deploying to a test server and running some automated functional tests? Better yet. Continuous Deployment to production IMVU style? Moderately crazy, but very impressive.
How far you want to push the limit will inform what kinds of capabilities you need. Will a single machine work or do you need a tool with server-agent capabilities? Is this just developer feedback or are we managing the hand-off to test or operations teams?
Haha, I like the Bonus question. I'm not even sure if i'm after a CI or BA tool.
What I want it something that will automatically automatically kick off a get latest from my SVN Repository, Build the project, Run the unit tests, and Deploy the site to staging.
But not sure what I'm after other than i hope it's freee cos its for personal use.
Am I using someone else's checklist or do I make a decision based on my own list of criteria?
Disclaimer: I work for Atlassian, a vendor of CI tools.
A few more things to consider include that really help determine how easily a tool can be adopted by the entire team
* Does the tool let me use my existing automated tests (Unit and functional/UI) without having to modify them? Does it display the results in a sensible and useful way?
* Does the tool notify me of important events (failed builds, hung builds, etc.) using the notification methods I prefer (IM, email, RSS, desktop/ IDE pop-up)?
* Does the tool integrate in a useful way with my bug tracker, IDE and other parts of the existing tool stack?
* How will I scale the CI system if the team adopts CI and needs additional computing resources?
Disclaimer: I work for Electric Cloud, a vendor of BA and CI tools.
This is a great list of questions so far! When you want to scale up past an individual team there are a few extra questions to ask:
* How easy is maintenance on the system? Can I upgrade the system without interrupting long running builds?
* Can I reuse existing corporate infrastructure? (e.g. Oracle, Active Directory)
* What is the strategy for supporting multiple groups in the company? Can they share resources?
* What support is there for data security or access control?