Will Your Project Ship On Time?
I am working on a simple test that software development teams could use to determine whether or not their projects will ship on-time. I ran this test by Eric Sink who helped refine a couple of the questions. I think his input made the test stronger and I’d like to get some feedback from you.
It would be great to see what your results are and if you agree with the results based on this test:
| Question | Response | Calculate Points |
| 1. How many total team members do you have (count everybody)? | _____ | x -2 = _____ |
| 2. How many non-engineers (non-developers) in your team (testers/mgrs/etc.)? | _____ | x -2 = _____ |
| 3. Break down your team to # of Members with: - Negative or Neutral Productivity - Normal Positive Productivity - Superstar Productivity |
_____ _____ _____ |
x -5 = _____ x 3 = _____ x 10 = _____ |
| 4. Expected Project Duration – Use This Scale: < 3 Months = 0 3-6 Months = -2 6-10 Months = -6 11-18 Months = -15 18+ Months = -30 |
x 1 = _____ | |
| 5. Organizational strictness on methodologies or processes: - Hard Core about following a heavy methodology (6-sigma or similar) = -6 - We strictly follow an agile process (scrum, XP, etc.) = 0 - We follow best practices or loosely follow agile process = 2 - We don’t do any process or best practices = -6 |
x 1 = _____ | |
| 6. Is your project broken down into tasks & features with estimates for each? - Yes = 2 - No = -6 |
x 1 = _____ | |
| 7. Does each team member have a task-level checklist of when the project is done? - Yes = 5 - No = -15 |
x 1 = _____ | |
| Total: | _______ |
To determine whether your team will ship the project on-time, use your Total Score and compare it to the following scale:
| Negative Total: | Forget about it – Your project is at risk of cancellation. |
| 1-9: | Project will finish late. |
| 10-24: | Project has a good chance of finishing close to schedule (within 10-15% of estimate) |
| 24-50: | Your team has a reputation of delivering on-time! |
| 50+: | It’s unlikely that you have that many superstars. Go back and take the test again. |
The goal here is to make questions easy to answer and still be able to get accurate results. I am going to make this test into a web-based calculator that any member of the development team could use to help determine their team’s chances of shipping software on-time, so I need as much feedback as you can give.
There are a lot of other factors that came to mind, but were omitted from the test to keep things simple. For example, you might be thinking “what about using past success rates” or “quality of estimates” in determining whether the team will ship on-time. However, I’m trying to leave out questions that are difficult to answer (too subjective) and allow for first-time teams to still be able to use the calculator to estimate their success rate (no reliance on historical data).
You might also ask “isn’t it subjective to categorize your team into 3 levels of productivity?” The answer is yes it’s a little subjective, but I bet most people would agree with your categorizations (with possible exception of the individuals who are being categorized in the negative or neutral category). Most of us know who the super-stars and the bozos on the team are. Everybody else is a normal, productive team-member.
So the questions are easy to answer, but the result should be extremely accurate. I’m curious to know if you agree.
[...] one of his recent posts, Hamid has started a new thread by creating a short test that lets software development teams, their members and their leaders to [...]
Will Your Project Ship On Time? : Keyvan Nayyeri
June 17, 2008 at 9:06 am
I think the -15 for a 12 month project is too large a negative number. I also feel that the rating for productivity is subjective, we employ good staff but they would all have to be superstars for us to get anywhere near a positive result let alone a “will ship on time” rating. We regularly undertake 12 month development contracts with 3-5 programmers per project and normally bring them in on time and within budget (always within the customer price). We use best of breed tools where applicable and everyone knows everything about the project (via PWA). I would say that the biggest driver has to be accuracy of estimating.
Graham Wade
June 18, 2008 at 3:23 am
Graham, I just did some math taking a staff of 3, all developers, 1 superstar (you need one for success) for a 12 month project, best practices, with a task list and estimates (yeses to Q 6 & 7) and I get a score of -2+0+16-15+2+2+5=8.
In this case, the score of 8 is heavily weighed down (as you pointed out) because of the 12-month project. I agree the -15 on that is too high for the 12 month range, but adequate when you get closer to 18 months. I’ll have to play with that question. I think the score should have been somewhere around 15 using the scenario I provided above, which would estimate the project would finish within 10-15% of scheduled time.
Hamid Shojaee
June 18, 2008 at 9:53 pm
if only this could ever work…..
Great book by the way (grin)
Damon Carr
Damon Wilder Carr
June 19, 2008 at 1:28 pm
Hamid
According to your chart it’s (-2) for each developer which gives -6+0+16-15+2+2+5 = 4 => project will be late. Or have I misread something here.
Graham
Graham Wade
June 25, 2008 at 1:25 am
What about breaking up question 5 (Organizational strictness on methodologies or processes) into more detail? E.g. I would ask whether a team uses source control and continuous integration, does code reviews, etc.
Florian Potschka
June 25, 2008 at 1:40 am
Graham and Hamid
It appears that this test only works if there are 0 non-developers on the team. That would contradict most other project assessment tools.
Question 1 states count everyone on the team. If your team only has developers on it, then you are saying there is no management, project management, testers or stakeholders on the team. That’s not a good thing for the project.
Including 1 project manager (10 pts), 2 testers (6 pts) and 2 stakeholders (6 pts) to questions 1 and 3 scores out this way – -16-10+38-15+2+2+5 = 6.
You’re also missing user training and training documentation (unless, of course, the software is so user friendly that it doesn’t need user training and/or training documentation). Adding 2 user document writers (6 pts) and 1 software trainer (6 pts) to questions 1 and 3 scores out this way – -22-16+47-15+2+2+5 = 1.
You need more than just developers to be on time and successful. I think the chart needs to score favorably for having a project manager, testers and training, and even higher for a good and excellent PMs and trainers.
Mike Nasca
June 25, 2008 at 6:55 am
Actually the final score should be -22-16+44-15+2+2+5 = -2. Only 3 pts for the software trainer under question 3, not 6 pts.
Mike Nasca
June 25, 2008 at 2:23 pm