Runtime Revolution
 
Articles Other News

Development Postmortem: BvG Docu

by Bjoernke von Gierke

1. First there's the idea

This idea goes back to when Runtime Revolution changed the UI of their documentation stack, and somewhat later the data of it to XML. That must have been three or four years ago. I really didn't like the UI decisions they made, but with Revolution it's quite simple to change the IDE. Almost all features are accessible by third parties like me, especially if it's a mostly self contained part like the documentation. At that time however, I did not muster the initiative to do anything about it.

Fast forward to March 2007 when I suddenly had lots of energy. So much, that I couldn't concentrate on only one idea and constantly began new projects. I still didn't like the Revolution Documentation UI, so I decided to revisit the idea of making my own GUI for the Revolution IDE's documentation.

The Three Stooges

2. The Anger

I really do not like XML. Maybe that also explains why I waited 3 years to touch the Revolution documentation. But now that I was determined to make my own BvG Docu, it really dawned upon me, that to use the XML files, I'd need to learn how to handle XML. To save others the pain of dealing with XML also, I started by making a stack that would let me extrude the XML into a suitable format for my use, which I then could interface with. This would allow me to not touch XML again. Basically I decided to export the existing data into something else.

My first task was to find out how to get at the dictionary entries. Since version 2.7, Revolution stores the XML entries as custom properties in stacks, and gets at them via two index files (which are plain text lists). Surprisingly, I was able to learn how to use the XML commands in less than a day, and extruded the documentation into a format I deemed usable within the same day. Then I wrote a handler that exported these custom properties into files. But it was still all XML.

The Raw XML

Thus, using my newly acquired XML command prowess, I decided to write my next handler: a way to get at the contained data and to store it somehow more to my liking. I thought long and hard about ways to store the data. As custom properties, in fields, files or on the web. Finally I decided on text files, as having all of the ca. 1500 entries in memory would be a bit heavy (though extremely fast to access them). Of course, if I wanted to store the data as files, I needed a way to order the different stuff within the single entries. Simply put, a hierarchically ordering, without to resort to binary data.

3. The revelation

After some seconds of thinking about it, I had to admit that that's a pretty good definition of XML. So I decided to just use the existing files, and make a handler to store only the last accessed entry into memory, as custom properties. That way a user wouldn't have to parse any XML, and instead could work with custom properties. However, to develop it, I myself had to use the XML commands quite a lot. Which made me write on my (newly created) blog: "So to not having to use XML, I had to use it. A lot. (If you are not laughing and pointing at me now, then you’re not human.)"

One especially hard learned lesson was that of the DTD. I tried to read the XML data in Revolution's files, but it would fail to parse because of not being XML. That of course surprised me, seeing how there clearly were XML-looking tags in these files. The XML library Revolution provides has a feature to make a lax parse, basically continuing even if there are errors in the XML, but using that would not output all the data. Revolution's documentation stack on the other hand was able to use the data, so I took the plunge and looked at the scripts from Revolution.

The adventures and surprises I encountered in there would be a whole story by itself (of which I thankfully have forgotten most by now). However I found out that the handler used by Revolution to parse the XML added a custom property at the front of the data. By now I had learned that XML allows one to include arbitrary data, if one explains this in a so called "Document Type Definition". When I too added the mentioned custom property before parsing the XML, all would work.

All in all, the extraction of a raw form of the relevant data proved a not too difficult task. So within not more than two additional days, I proclaimed my "DocsLib by BvG" done (boy, was I wrong), and began to write some documentation on how to use it.

4. Didn't you want to make a GUI?

Now I began to write a second stack which would present the data in the custom property of the library. A first version was cobbled together quite quickly. But in GUI design the first version is most often just a stone to step on.

For example, I tried to use a combo box, which basically is a button in combination with a field for the search/filter field. The drop down would contain the history of previously entered entries, while you could enter search terms into the field of the combo box. Of course that's not exactly how one is supposed to use a combo box, and it would not work the way I wanted. Because of that, I ditched it in favour of a separate drop down and an entry field. I also made the "related topics" a drop down at first, something that proved too click-heavy to be useful. Then I made the terms list field and the related terms list field resizable, added forward and back buttons to the history and made up a prototype settings substack (which didn't do anything yet).

5. Setbacks and difficulites

There's an index list file that contains every entry in the Revolution documentation data folder. But 2 chars are html encoded, greater than and lesser than ("<" and ">"). These are also encoded in the "name" part of the XML entries, but all other "name" tags aren't. Some of the data was html text, but some wasn't. Image files would be referenced, but didn't exist. Some entries even would contain the 404 page from the runrev.com homepage. All this stuff I had to work around in my library stack, and, of course these problems got resolved in newer versions of Revolution.

Also I had problems with escaped chars, because of my history menu button. And various small problems to overcome related to how I wanted my data displayed, as opposed to how it was stored in the XML. Then, after a week or so, I had workarounds or fixes for most of the problems, so now it was time to give my new project some peer review.

6. The crazy

To give something away, even if it's only a pre-release alpha and proof-of-concept version, one must always include help files and documentation. Fortunately the library stack already had extensive help attached, which I only needed to update a bit. The GUI parts however didn't.

After having fumbled my way through some parsing problems the whole day, I went to bed but couldn't sleep. So instead of throwing myself from left to right, I made a help button. I decided to use a transparent green (I like green) overlay, to put help fields on, and that is how the crazy began. Pretty soon the night began to develop into a contest of how many translucent objects I could pile onto each other.

The crazy, it burns in the eyes!

Now, if you look at the help screen, and deem it too trippy, I can only claim sleep deprivation and temporary loss of brain functions as defences. In the end I left it in, mainly because I didn't want to do it again, and most functions of the stack are quite self explanatory anyway.

7. Give your work to others early and often

The first stack I gave out for testing lacked a few key features the current stack has. There was no way to adjust anything in the layout at all. The setting that allowed you to only filter for functions or command (or both, or any of the other types) didn't work yet. In addition, the "hijack script clicked" button didn't work correctly. But most importantly there was no GUI to make the needed data files, so the ChatRev guys (and gal), to which I gave my stack, needed to run some scripts from the message box before using it.

The feedback I got from them was really encouraging and they found quite a few bugs and problems, mostly related to problems with showing the data. There were also problems with the import, so most things I needed to fix were in the library (Remember how I deemed it finished after two days?).

Around now I also made the observation that I was reluctant to open the original Runtime Revolution documentation stack. Sometimes I needed to, because my stack wouldn't show an entry, or show it incorrectly. Still, I always tried to avoid the built-in one, doing very long fixes in my stack, before giving up, and opening the Revolution documentation. This, before most other things, motivated me to go on: My own stack was much more usable to me than what Revolution offered.

8. The dangers of depending on the kindness of strangers

If you make a stack that takes data from someone else and uses it, that can be quite a lesson in accepting differences between humans by itself. If you however try to tie into an existing product, there's simply much more to go wrong.

A bit later I had already made the (almost) final version of the stack, including the import wizard that runs at the beginning. But then one of the testers told me my stacks file creation wizard would run, but not generate any files in the process.

I panicked. The wizard was made to check for everything I could imagine to go wrong, including no write permission, non-existent files, and so on. In any such case, it would prompt about the error and abort. So why wouldn't it work for him, yet proclaim it was successful?

After some questioning (and lots of my own testing), he mentioned that he had run it on the recently released beta version 2.8.1dp1 of Revolution, which I hadn't installed yet. More panic. Did Runtime Revolution change the file format for 2.8.1 making my library useless?

Luckily for me, it turned out to be an error by Revolution, not by me. Basically they failed to include one of the files which contain the documentation XML entries. Meanwhile, my stack assumed that if there's one file, all other files existed too. So I also changed my handler, making it proceed, even if it couldn't find an entry, and then report all missing entries.

I also had huge problems when I wanted to run my stack as a productivity stack within Revolution. Basically Revolution doesn't really allow for that. For example it would always pop up a dialogue for saving when I closed the stack, and there's just nothing I could do about it, unless running it as a plugin. There was also a problem with setting my stack to the mode I needed, especially when switching between "editable" and "modeless" a lot.

9. Pre-release slumping

I wrote to Runtime Revolution about the legal situation of taking their content and presenting it in my own stack. Of course this communication took a bit longer then anticipated, and although I deemed the project finished, I added polish and small changes while waiting. Finally I received a satisfactory answer and proceeded to make the last small changes in the stack.

This was a hard time. I added more and more bells and whistles, knowing that people would want them, but not needing them myself. On the other hand, it was a nice way of saying goodbye to working on this stack. But then, almost a month after starting, I could write a mail with "[ANN]" in it's subject line to the Use-Revolution list.

Finished!

10. Learning is living

One of my favourite features of BvG Docu is how one can change size, order and style of the parts that make up the documentation entries. Even though no one else really applauded it, to me it's the best feature in the stack. Incidentally, it's also the hardest to use and understand, probably because it is the only part I really made up from scratch myself, instead of thinking "how can I improve upon this feature in the Revolution documentation".

The Revolution IDE really is unique among IDE's, because you can replace almost anything with your own stuff. On the other hand, it's also quite a complicated product, with many dependencies interconnecting the different parts. But I was lucky, as this is not so much the case with the documentation. After all it is focused on data and less on function, so I could just make my own stack, without the need to replace or remove the existing one.

Maybe I should have asked about the legal situation before almost being finished with my work, but fortunately this turned out not to be an issue. In hindsight I also could have tried to make a smaller project first, instead of jumping right in, but on the other hand, where's the fun in that...

I learned much more then I ever wanted to know about XML and about interfacing with other peoples' data and projects. I also found out that it's simple to make something better, but much harder to do something new. The separation of tasks and GUI suited my style of work really well, and I should do this with future projects too.

I am glad to report that some people are choosing to use my solution instead of the Revolution documentation. What is most important to me however is, that despite all the aggravating and cursing that went on, I had lots of fun working

Download the documentation stack from my page.

 
©2005 Runtime Revolution Ltd, 15-19 York Place, Edinburgh, Scotland, UK, EH1 3EB.
Questions? Email info@runrev.com for answers.