On Tuesday, build "RC1" was released to Revolution Enterprise users. The wider beta test signup community received their invitations on Wednesday morning. Recently, Revolution staff held a conference to review the beta program and candidly answer some questions about the upcoming release. Here are some excerpts from the conversion.

Bill Marriott (Director of Marketing): So Mark, just how close are we to release of the software?

Mark Waddingham (Director of Technology): Well, it looks like we're days away at this point. My goal is for shipment to occur before the end of the month. We've been working very hard to close open bugs. There's only a half dozen or so we know of that need to be addressed before ship. There are a couple more chapters yet to come in the documentation. Otherwise, things are looking very good.

Bill: How do you decide when to ship a product?

Mark: Well, a product like Revolution is never truly "finished" in the sense that you can always fix one more bug or add one more feature. But since the release of [Revolution] 2.8.1, we've been establishing the scope of what would be in this release, not just that we would do Linux, but that there were areas we could focus on, and things that weren't going to be possible. Indeed, that scope changed a little over time. Some things we were going to work on but we decided ultimately would take more time or would be too risky. And some we found we were able to do that we hadn't expected - such as media playback support in Linux. Once that scope is established, though, we do the work and monitor bugs reported against it. When the bugs go down and we see stability across the builds, we know we're close.

Kevin Miller (CEO): There are also business drivers. We want to be sure people are getting value out of their update packs, for one. We've been promising this version for a long time,. It's important that we deliver on that. Releasing this version means we can move on to the next one, with more features for more customers. By the way, work on the next release is well advanced and it will follow 2.9 quicker in comparison. And of course, whenever we launch a new version that generates new sales and upgrades. And renewed interest from the community and press. All of which are very important for the platform. But especially for this version, because it's the manifestation of our promise to deliver improved quality and an updated Linux version.

Bill: And let's not forget the beta user surveys and their role in helping us make that decision. There's a question toward the end for people who actually tested the product, "Would you ship this? Why?" And, the answers to that question have been one of the most important metrics we've looked at.

Heather Nagey (Manager of Customer Support): Yes, Bill. How have we done on that answer?

Bill: Quite well. Prior to the release candidate the overall percentage was just fractionally lower than it was for the 2.8.1 release. But when we drilled down and looked at it carefully, we found a lot of respondents were using older beta versions. When we looked deeper at who used the software daily, who kept up with all the versions, the answers to "why," we found those people -- that the "Yes" votes were on par with the ones for 2.8.1, which we know was a very solid and well-regarded release.

Heather: Mmmhm. And the expectations for this release are rather high. I have seen a lot of positive email from our power users however. A lot of them seem very happy with 2.9 and the number and quality of bugs we've fixed.

Mark: Yes, we've had more time to work on it, and even though we've substantially rewritten the Linux platform, and areas of functionality like printing and the clipboard and drag-drop, people expect it to be even more robust than 2.8.1. Actually, we've fixed more than 500 bugs that were present in that and earlier versions. So, objectively it's quite a bit better in that regard than any previous version. Obviously, given that overall number of changes and the diverse way our customers use each feature, it is possible that a handful of regressions may yet emerge in the weeks after we ship. That's par for the course with a release of this scope and we can do a 2.9.1 if it looks like that would be beneficial.

Bill: Right, and that's why, I think, we're going to do one more survey focusing just on the RC versions before we release. We've polished a lot of the remaining rough edges in the RC and so we want to make sure we've past where we were for 2.8.1 on the final build.

Kevin: Mark, maybe you could explain a little bit about why DP4 [Beta 11] had more issues than DP3?

Mark: Well essentially, we had just completed comprehensive work on some deep parts of the software: things at the engine level, or affecting multiple areas of the Revolution-based IDE. And these parts of course were affecting multiple platforms. Whenever we do that there is a high risk of issues appearing in unexpected areas. We can test these things internally, extensively, but at some point we have to do a beta release.

Heather: Yes. And I remember people went a very long time without a new beta and some were getting a bit, how shall I say, antsy about it! By the way, Bill, what happened with some of the notifications? Can you tell people a little bit about how the beta tester applications work?

Bill: Sure. A bit of "mea culpa" on that. We had one major beta notice that hit a snag early in the bulk email process, and I simply didn't notice it. So a majority of people had a couple days delay in receiving notices about the beta and during that time the old beta expired! Just so that people know, we have that signup form and you get an auto-mail from that to watch for an acceptance letter, a beta kit, in about a week. The process is that all those applications get kept in a database, then I periodically import them to my mailing software. I review each application to read the comments and eliminate spam submissions or duplicates. There have been some very interesting and helpful comments provided there, by the way. Anyway, when there is a new beta released, it goes first to the Enterprise users, via the Improve list. Then if we don't hear anything calamitous from that, we send it out to the "open beta" community 24 to 48 hours afterward. Or for example, to Improve on a Friday and Beta testers on a Monday.

Kevin: How many people ended up participating in the open beta?

Bill: We have more than 800 beta testers and counting. As we get closer to release, the interest just keeps growing.

Kevin: Yes, we're very grateful for all the feedback people have sent in.

Mark: Indeed. Though the open beta program has added some challenges too. We have a lot more reports entered in the Quality Control Center that are harder to pin down because they are duplicates, not reproducible, or don't have enough information for us to act on right away. The system also limits us a bit in that we don't have an instant status button that shows that we've reviewed something but haven't yet scheduled it for being addressed. I think it's important for people to know that a developer does review every single new report in the system, but we have fallen a bit behind at moments in the recording of that.

Heather: From a customer perspective its been a big success. A lot of people have come to it through support, and have been really happy that they could be involved in the creation of 2.9. It's been important for many Linux users too - they have been able to see that we're really delivering on our promise to bring their platform up to date, and we've even added new multimedia capabilities to it that were never there before. It's going to be great not to have to generate 2.6.1 codes for our Linux users anymore, they'll be getting the same great Revolution experience as everyone else.

Bill: Right. Going back to what Mark said, let's not forget that the RQCC [Revolution Quality Control Center] is totally open. Anyone can go in there and see the status of things.

Kevin: Right, and it's happened in the past that people have searched the RQCC, which is very easy to do, and say "oh there are so many open bugs, so many unconfirmed reports, and so on."

Heather: Are you thinking of changing that?

Kevin: We're not planning to make it private at the moment. I think that distinguishes us from a lot of other companies who are content to have a one-way feedback form. But I think it helps people to understand a little bit more about how the tool works and what it's like for us to use it. Not all reports are created equal. Some are really minor, but we're not going to work on those just to inflate our bug fix tally. Some are really great feature suggestions. Some are not-so-critical or seldom-used issues that would be very difficult to fix. And so on. So you can't just look at the gross number of reports closed, but also weigh the value of the ones that were fixed versus the ones that remain open. You also have to look at us in comparison to a typical product - we're actually doing pretty well. Of course we run a risk that some people feel we're not fixing enough issues fast enough, but the feedback we've seen is that people are impressed with the progress we have made against the issues that really matter to them.

Heather: Yes, I do get emails from time to time that ask why bug number so-and-so which was reported a long time ago is still open.

Bill: Right, well we knew that to close or fix every single bug in the system will have us busy for a long time.

Mark: We wouldn't be able to make any of the larger feature changes that our customers need if we tried to do that. For example, some of the bugs can't be handled until we can replace the feature with a completely new one rewritten from scratch. And realize that some of those features, like the field object, touch hundreds of other areas of the software. A new field object would close literally dozens of bugs and enhancement requests just in and of itself. But it's not trivial and it wouldn't be in the interests of most of our customers to have to wait until it was completed to get the hundreds of valuable improvements that are in 2.9 and finished. Even for changes that don't involve that kind of rewriting, there is a limit to the number of people we have and thus the amount that can be done for any one version.

Bill: Right. And so to better address this problem of having so many entries we tried to reduce the universe to the bugs that mattered most to people. We tried to educate people about the voting system and make it more obvious how to vote, etc. And then we also did something where we mailed people and asked them to tell us, in a fairly unstructured way -- a simple email message -- what their personal "top five issues" were. And even though that took a lot of time to compile and analyze, we were able to see some consensus emerging from those responses and moved them to the top of our list.

Mark: Yes, that was very helpful. One of the items I remember from that was the ability to work well under Remote Desktop in Windows. We took a look at that because of the list, and it turned out not so difficult to correct. It was one of those things that might have gotten lost, but for the open beta test program.

Heather: Going back to the surveys, Bill, what are people telling us about the software?

Bill: Well they love the software, and they love the opportunity to be a part of the process. People are consistently telling us it's great, it's better than ever, and the progress has been phenomenal. One thing that keeps coming up consistently is documentation, actually.

Kevin: Ah yes. Well. We've taken that to heart and made a significant effort on the users guide. It's going to be roughly twice the size of the previous version. Lots more code samples. Explanations of how to extend the Database Query Tool through scripting. New sections on Unicode, Arrays, encoding and decoding data, XML. Whole chapters added on printing, standalones, debugging. A big new section on using the Internet....

Heather: Why Kevin, you seem to know a lot about the new manual! Could it be that you wrote it?

Kevin: (Laughs) I played a big part in it.

Bill: You know, that is huge, actually. It's the number one thing that has come up in the surveys. People are asking for more beginners help and sample stacks, etc. They're thirsty for knowledge!

Heather: Yes indeed. We can never have too much documentation, and the more formats it comes in the better. Different people learn in different ways, some love the video tutorials, some are happy browsing the dictionary, and many want a searchable, downloadable, comprehensive user guide. I'm really happy with the new user guide and I know our customers will be too.

Mark: Quite simply, it's not so much that we don't have this information available; it's just that it's hard to find in the current interface. We have literally hundreds of code snippets and samples that ship with the product. But people have to look in all kinds of places to find them. In a later version, we're going to reorganize that into a proper library and polish all the resources we provide in that area. They're going to be much easier to discover and navigate. And we'll also be adding even more information for beginners. In the mean time, we've done a lot of work on the online learning sections of the web site, which will be launched together with 2.9.

Heather: Great! Information is only useful if you can actually find it! People don't know about a lot of the stuff that's in there, like the ability to search the Use list directly from within Revolution.

Bill: Suffice to say that it's well worth getting the RC builds just to marvel at the new users guide.

Kevin: Bill, what would you say we've learned from the beta program?

Bill: Well, I think people for the most part really enjoyed it. Compared to where we started, we have implemented an overhaul of the bug reporting tool that was a solid success. We've gotten a lot more people engaged in the development process. We've learned a lot more about our customers and how they use Revolution. We've also got some really good ideas for the future. People want to have better lists of what's new, they'd like more guidance on what to test, and they'd like more frequent builds and notifications. Actually the one thing that comes through in the comments, not making this up, is people would like more hours in the day to be able to test Revolution!