Finding the right partner to be your CTO is no easy task. I have had negative experiences in the past where my CTO was not able to deliver our MVP after 1 year of development (my company already has a functional product for several years, but I need to upgrade). I believe the best way to measure success in the future is to observe past success. However, how do you recommend choosing the CTO? Thank you.
That is a really good question. How will you know if your accountant is up to the task? Do you give him a test? Not likely, right? I have written an article about that as I get asked to be a CTO of a start-up all the time and gauged for my technical skills for freelance projects as well. It's more geared for a development role, however it seems that this is what you're looking for: http://thecodebug.com/vetting-a-technical-person/
It seems very worrisome to me that your first CTO could not complete the MVP in one year. Was he part time and on equity only? That would definitely also have a bearing on it. It is really hard to motivate someone without compensation - especially really good technical people that know how much their skills are worth. The idea that possible future compensation is equal to actual current compensation just doesn't have legs. Now, I only say that since you have not mentioned whether he/she was paid or strictly on equity.
In the article I chat about vetting someone more on integrity in a very directed way. You can also gauge them on actual skills, but only as it pertains to your particular project. Tech skills are broad. If they just take an online test, how do you know what the test is actually gauging?
Have a read, let me know if you'd like me to expand on anything and good luck to you!
If you want your future CTO to do tasks X, Y, and Z, you should make sure the candidates you interview have done these tasks in their prior jobs as well as they have deep interest in doing these tasks in future in your company.
Besides the ability to do these tasks, the context in which they have done these tasks is also important. Meaning if you are a startup you should ideally get someone who has done these tasks in a prior startup environment and not from a large company.
Either get a bunch of geek friends to evaluate them or have the candidates evaluate each other. Your bullshit detector should be able to extract what you need.
If you know exactly what you are looking for in terms of technological skills, than that part of the interview shouldn't be too difficult. However, it is harder to judge how someone will fit into on the team, whether they have managed a team before and their success rate in delivering projects. I would definitely check references, and reach out to the previous employer for feedback.