log in | register | forums
Show:
Go:
Forums
Username:

Password:

User accounts
Register new account
Forgot password
Forum stats
List of members
Search the forums

Advanced search
Recent discussions
- WROCC Newsletter Volume 41:11 reviewed (News:)
- WROCC March 2024 meeting o... Hughes and Peter Richmond (News:1)
- Rougol March 2024 meeting on monday with Bernard Boase (News:)
- Drag'n'Drop 13i2 edition reviewed (News:)
- South-West Show 2024 talks (News:4)
- February 2024 News Summary (News:1)
- Next developer fireside chat (News:)
- DDE31d released (News:)
- South-West Show 2024 Report (News:)
- South-West Show 2024 in pictures (News:)
Related articles
- Testing on RISC OS
- The state of PackMan in 2018
- A fresh look at the Desktop Development Environment Manuals
- Getting FAT32FS working on a RiscPC with a Castle USB Card
- The Icon Bar needs YOU
- What is the point of RISCOS Ltd?
- Bob and Trev: Resurrection: Just in time
- Monster AI
- Combat
- Visibility and pathfinding
Latest postings RSS Feeds
RSS 2.0 | 1.0 | 0.9
Atom 0.3
Misc RDF | CDF
 
View on Mastodon
@www.iconbar.com@rss-parrot.net
Site Search
 
Article archives
The Icon Bar: News and features: RISC OS Documentation
 

RISC OS Documentation

Posted by Charles Justin Ferguson on 08:00, 9/8/2021 |
 

What have we got?

RISC OS is documented in many different places these days.
 
Originally it was documented through the Programmers Reference Manuals ('PRMs'), a physical set of manuals in multiple volumes for each of the major areas. Over the years these have been augmented by updates in the form of volume 5/5a, application notes, specifications for some of the 'new' components that were never integrated into centralised documentation, and information that was noted by people and submitted to the StrongHelp manuals.
 
The PRMs were released as part of the 'Techie CD', which had a hypertext application which could be browsed on RISC OS. The content was the same as the physical books, with the same page numbers. Later, there were PDFs and HTML versions 1 produced by RISCOS Ltd, which were more accessible, but had their own problems 2.
 
The application notes and various specifications came out of Acorn which included some detailed documentation which covered usage notes and APIs for 'newer' features. These 'newer' features include such things as the URI/URL fetcher stack, nested window manager, redraw manager, and miscellaneous updates that followed on.
 
The StrongHelp manuals for SWIs and the like were originally written by Guttorm Vik, and maintained by myself for a few years, before being passed on to other people - they were never official. However, they included many of the fixes and notes about oddities within the PRM and the different implementations within various RISC OS versions. The StrongHelp manuals were the primary place for developers to refer to APIs whilst writing code, with reference to the PRMs when more detail was required. The main 'SWIs' StrongHelp manual contains links to the pages of the original form of the PRMs to aid in this process of escalating gathering information.
 
There is also a Wiki on the RISC OS Open site, which is very similar to the StrongHelp documentation in its detail and degree of authority.
 
So we have...

  • Original PRMs, in printed, Techie CD, PDF and HTML forms.
  • Random specifications produced officially and unofficially.
  • Random user-sourced cheat-sheet documentation like StrongHelp (and later the RISC OS Open Wiki).
That's the current state of the documentation - it hasn't really changed in the last 15 years, as far as I can see.

What did I want?

Back when I was doing RISC OS work in the dim past, after the RISCOS Ltd days, I felt that this situation was not ideal and needed to be addressed. So I tried to work out what the goal was, and what was problematic.
 
The goal...

  • Produce documentation that's viewable on RISC OS and other systems.
  • Manage the source as a living document that can be editable by developers, on RISC OS (or other) systems.
  • A documentation system that was easy to work with, so that people without experience could produce documentation.

The documentation as delivered by Pace was in FrameMaker format, as that's how Acorn's publications department had managed it. FrameMaker is an Adobe product which was only available for Windows. As supplied, the documentation was in the binary file format, which meant that at any time only one person could work on a chapter 20. We only had a single license for the product, so in any case only one person could work on any part of the documentation at a time.
 
FrameMaker did have textual formats, but these were not easy to work with, and were not easy to bring into an automation system which we were hoping to use. FrameMaker could output to PDF, HTML and PostScript, which was useful to us for physical manuals if we had wanted to produce them in the future. It's a good system if you happen to be producing manuals for large projects through a central publications group (as Acorn had).
 
However, we wanted to be able to work on the documentation ourselves, and to be able to farm out documentation to others outside the company if they had an interest (or as contract work). We hoped that this would encourage others who had had complaints about the documentation to be able to contribute.
 
The problems with this...
  • FrameMaker had a steep learning curve for those who hadn't used it before.
  • It was important that what was being produced was able to be worked on within RISC OS.
  • Not being able to be worked on by multiple people was quite limiting.
  • The binary file format meant that changes couldn't easily be managed in the source control system in the same way as the rest of the sources.
  • Expecting third parties to purchase the expensive product 18, just to update documentation, was not reasonable.
  • The HTML that FrameMaker produced was not good in RISC OS browsers - hence the transformation work to make it usable.
  • RISC OS native products like Ovation or Impression would show a preference and be prejudicial to others.
What does that mean? Well, it implied that the output format would need to be HTML or PDF. I had already decided that FrameMaker was not appropriate, and so working directly in the HTML was an option. Much earlier, I'd used 'HSC' 3 as a tool for generating some simple documentation in HTML format 4. This worked well in producing the right sort of documentation but it was a little specialised. Because it was a niche system, I did not feel that it would work well in a wider community. Whatever I used really needed to be able to be used by people without too much knowledge of the system, and for which tooling was widely available and standardised.
 
This is where I began to look at an XML format with transformations through XSLT.
 
DocBook become a thing that people were using, so I had a look at it and ... it's quite generic, a little weird in places 6, and had a steep learning curve. But because it was generic, it didn't have any specialised layout for the style that I wanted for the PRM - that would have to be built on top. You'd essentially be writing XML elements that were for DocBook styled to look a particular way.
 
There is documentation for DocBook and you'd be able to produce something useful, but you're still not capturing the essence of the RISC OS documentation in a useful way - I'd become convinced that the benefit in using XML was not so much in the fact that I could use it to produce readable documentation, but that it was capturing structured information about how the APIs worked. That is, not only could you write the XML and get out a page of readable documentation at the end, but that the XML that you wrote could be cross referenced, indexed and even converted to other formats easily.
 
I'd been tinkering with other things at the same time. Thoughts about how modules might become self-documenting were going around my head - if a module could report information about its SWI interfaces in a structured manner, then in-system lookup of APIs, and debug information for tools became possible.
 
Toolbox already had its own format for describing the toolbox SWI interfaces, which were then built into C/ObjAsm files that were used in toolboxlib. If those definitions were built into the module, we could always extract them from a running system. And things like the toolbox test applications (!ResTest) could use toolbox event information from the module to present messages about what was going on.
 
The BTS system 7 recorded every piece of information about modules when an abort happened. You could later show that information on another system. If the modules that were running reported their API's parameters, then it would be possible to report the meaning of those recorded registers within a diagnostic dump.

Creating PRM-in-XML

All of these things were pointing to having structured information about interfaces and data structures 8.
 
This produced the idea of using an XML representation to hold the structured information in the PRM. Having separate tags to describe SWIs, vectors and events, etc, with information about the register usage being structured within that. This seemed like a workable solution to me, and I set about trying to represent the content of a few pages of the PRMs in an XML format.
 
Initially it was a bit HTML-like, but some elements were culled - most of the elements which were related to formatting were replaced by semantic elements 21. The only formatting elements which remain are for stress within the prose, line breaks, and for super-script and sub-script, as these are often needed to explain more maths-heavy sections.
 
As I was working on it, I created the XSLT - the transformation that turns it into HTML - so that it could render the elements using tables. That meant that it worked in the browsers of the time. The transformation is important to make it look presentable, but the point of the XML format is that if you want it to look different, or you want to extract just certain information from the files, you can do so with whatever XML tools you have. And there are a lot of tools for doing that.
 
There are actually a few transformations that were working, to varying degrees:

  • There's the HTML transformation I've mentioned, which used tables and inline formatting.
  • There's a simple extraction of *Command usage, so you can get simple information that might go into *Help.
  • There's a C header file generation that creates constants from the APIs in the file.
  • There's an Impression DDF generator, that really didn't work very well.
  • There's a rudimentary StrongHelp format generator, which worked reasonably but wouldn't be suitable for the quick-reference use that you would have in StrongHelp.
  • There's a specialised HTML transformation that uses frames to link content in a different way. It didn't work very well.
And I needed a way to pull all this together, so I created a set of index/contents pages, which extracted the cross-reference information from the files to provide direct links toeach of the SWIs, events, vectors, etc, just like the equivalent sections in the PRMs did.
 
Although these are the transformations that exist, more can be created with a little effort.

What does it look like?

Here's the definition of a SWI for AMPlayer in the PRM-in-XML format:
 

<swi-definition name="AMPlayer_Pause"
number="52E02"
description="Pauses playback"
irqs="undefined"
fiqs="enabled"
processor-mode="SVC"
re-entrant="no">
 
<entry>
<register-use number="0">flags :
<bitfield-table>
<bit number="0" name="Resume">
Resumes playback.</bit>
<bit number="1-30">Reserved, must be 0.</bit>
<bit number="31">R8 contains the instance handle to which this call
should be directed.</bit>
</bitfield-table>
</register-use>
<register-use number="8">if bit 31 of R0 set:<br />
instance handle to direct at, or 0 for
the base</register-use>
</entry>
 
<use>
<p>This SWI is used to pause or resume playback. When paused, decoding to
the output buffer continues, but at a much reduced rate. There is no sound
output.</p>
 
<p>Pause mode may also be cancelled by stopping. If
<reference type="swi" name="AMPlayer_Stop" /> is used to
cut to the next file, or if a different file is started, pause mode will
continue to be in effect, freezing the new file at the start of the file.
This can be used to ensure that playback starts at the instant of calling
AMPlayer_Pause (as opposed to calling
<reference type="swi" name="AMPlayer_Play" />, which can have a delay while
opening the file etc).</p>
 
</use>
 
<related>
<reference type="command" name="AMPause" />
<reference type="swi" name="AMPlayer_Play" />
<reference type="swi" name="AMPlayer_Stop" />
</related>
 
</swi-definition>
  • The definition declares, as attributes of the swi-definition element, what the SWI is and some of the common fields that are described in the PRMs. Because these are attributes with defined meaning, they can be extracted - the contents generation uses this information to create references to the SWIs.
  • Then there is a description of the registers on entry to the SWI call (entry). Each of these is described through a register-use, which describes one or more registers. The flags in R0 are described in a bitfield-table which explains what each bit means.
  • There could be a description of the registers on exit from the SWI (exit), but this SWI does not have any, so it is omitted.
  • There's a longer section of prose which describes what the SWI does and how it is used (use). This includes cross-references to other SWIs or sections as necessary.
  • Finally, there's a section for related interfaces, which includes a collection of references to other sections within the documentation.
It's a specialised format and there's a little bit of a learning curve to find the right way to express things, but because you're focusing on what the interface is, rather than how it's presented it is much easier to write. Mostly you know that the produced documentation will be in a consistent format and you cannot accidentally deviate from the style of the PRMs. That can be a bad thing for some use cases but, for consistent and easy to follow documentation, it is important.

What did converting the PRMs involve?

So having decided on a format that can represent the information contained within the PRM, the next step is to turn the chapters of the PRMs into that format. However, it's useful to prove that it will work first by starting to use it for real things. Using it for real APIs and structures meant that it got 'battle hardened' and the error messages improved, and redundant parts stripped or made more useful. Then real conversions began.
 
And that's a lot of work. It's not a process I completed, but it's one that's largely mechanical. Take the HTML PRMs, pass it through a Perl script which turns the patterns of usage into skeletons of structured elements, and then chip away at the many bits that it got wrong until you have a document in the right format. It's tedious and it takes time to do, but the results are XML files that describe the same content as the PRMs. It's important at this stage to not start introducing differences. Ideally you can visually skim two copies of the documents - old and new - and see that, apart from styling, the content is the same.
 
This was a long running process that I embarked on, and got some help from Andrew Hill in creating some of the documents. Take a chapter, convert it, review in comparison to the original and fix. Repeat. Then go and do something else because it's a bit mind numbing 10.
 
There are bound to be mistakes, but the point is to get it into a format that's easier to manage, and has greater flexibility. Each chapter of the PRMs can be handled in (mostly) isolation. There will always be cross references, but these can be given general targets and we can be reasonably sure that a link checker will be able to verify these later. Because new documentation is all being written in this format, you're not falling behind and having to convert things repeatedly.
 
When the original PRMs were written, my understanding is that the original developers of the OS had written notes about the components to explain the interfaces and interactions. I believe that there were also some interviews done by the documentation group to understand the intent of the parts of the system 9. So there was some point of divergence at the point that the documentation and transfer of understanding of the components were made - from the developers to the documentation department.
 
This transfer was necessitated by the large, and quite skilled, job of writing documentation being different to the day to day work of developers. Generally, developers don't write good documentation, and they certainly don't have access to the tools and the knowledge of how to drive those tools well to create a full manual. Or at least they didn't at that time.
 
However, I was never going to get professionals skilled in documenting and publishing tools, so the PRM-in-XML project, as I named it, was going to have to be 'good enough' with developer written documentation. That also meant that the tools needed to be pretty simple to use. So I wrote a RISC OS application that could take a given XML file and turn it into HTML - which is about as simple as it gets. It's just invoking xsltproc on the file in a TaskWindow and showing you the output.

Moving the goal posts

The goals have changed slightly, as the needs become more obvious. Instead of purely wanting to have something that was easy to work with, it should be something that was easy to work into the software development life cycle. That is, as you update a component like a module or tool, the documentation gets updated with it. And the easiest way to do that is to have the documentation sit alongside the component in its source directory.
 
If the multi-stage build process that builds an application (or the whole OS) includes a set of targets for exporting documentation, much as you export the resources for ResourceFS, then you can keep the documentation in the same system. That's where it is most useful and it can be used to construct a set of manuals from the application or system that you're building. Or even just a set of manuals for a section, if you only process the parts you need.
 
By keeping the source with the documentation with the component, you remove that need to transfer knowledge from one place to another at marked points. You can update the documentation as you change the code. Yes that means that adding a new feature is more than just writing code, but if you believed it was just writing code in the first place you were woefully mistaken.
 
And it's not that hard to do. Before the PRM-in-XML began to be used in builds of components (and that was really late on), most components just had plain text files which described the functioning of the components. Usually these were structured as an introduction and description of the component and its interfaces, then a description of the SWIs and services that it provided. They were intentionally structured that way because that eased the transition to the PRM-in-XML format, or to be released directly in that format because it would still be a familiar and useful manner of introducing new features.

Retrospective

So, with an extra 15 years of experience, what do I think about it?
 
I dusted off the PRM-in-XML project last year and started to make it work. The xsltproc tool had become a bit more strict so there were some operations that were broken that I'd had to fix. I cleaned up and exported the style sheets and tools as part of the release of resources with my Build service presentation 22 - because it's used within Pyromaniac's build, and in a number of my other builds that I do.
 
It's even more useful to me now because systems are so much faster - processing a chapter is a sub-second task. And it's still a structured format so the pages that were converted years ago are still useful, and can be converted to other formats if I wanted.
 
Other technologies exist for managing documentation, so is it still the right way to do things? The goals these days, I think, have changed a little...

  • Being viewable on other systems is more important now - most users will be using a second device (a tablet or phone, or Windows/Linux/macOS device), so being able to be viewed on such systems is vital.
  • Editing in the single format by developers is more important - as the community is more open, it is important to allow anyone to be able to submit documentation changes, and doing so in a similar way to the regular flow of development means that having the documentation alongside the main source is vital.
  • Having a community of unskilled developers is a useful commodity for documentation. They can provide better insight into what is useful, and be able to offer better suggestions. And if the format is simple enough that it's relatively obvious what they need to do to work with it, then that is all the better.
  • A textual format that is diff-able is even more important for the review process in source control systems such as git, as exposed by GitLab, GitHub, etc. This was important in my original goals but essentially implicit. With the expectation that contributors will want to be able to review changes that happened and submit their own, this is important to the review process.
  • The documentation can be worked on by multiple people at once - because it's text, and therefore able to be processed by the standard tools, two people working on a documentation file is very easy to resolve. And because chapters are split between components, the likelihood of conflicting will be low-to-non-existent.
  • It's all free - there's no cost in using a the XML processing tools, and therefore you don't have to buy Impression, Ovation, or FrameMaker to work on the documentation.
  • Anyone can do it - the tools and docs available to everyone, and anyone can process the content into HTML, or into other formats with some processing.
This last point offers the most flexibility - in that if someone wants to do something fun like produce a documentation browser 11, or in-editor context help 12, that would be possible, and would grow in usefulness as more sections of the system were documented.
 
What about PDF production? Change the HTML stylesheet to use classes 13 and then use CSS media queries to allow the pages to be formatted for printing. Or change the output format to directly output a presentation format like DocBook which is then able to be laid out for printing. This isn't so much a challenge as just another step.
 
Honestly, though, HTML is more flexible than a PDF most of the time, and PDF as a format to transition to a physical manual is a huge waste of time and money 14. I looked into it for a print version of my Rambles - it would have been about £80 a copy if I remember correctly 15. If the new PRMs were going to be sold in printed format they'd be much more expensive, for content that you could download yourself and browse on your machine, or online. So, what's the point? Some people might prefer that, but not enough to make a printed version of the manual viable. At least that's my opinion - nothing about the conversions precludes you doing it if that's what you want.
 
EPub is probably more useful, as it would allow for layout on tablets and other mobile devices. And EPub is just HTML 16 with some metadata around it, so that's easy.

Conclusion

PRM-in-XML offers...

  • Structured information for the documentation and APIs.
  • The ability to transform into other formats easily - HTML, StrongHelp, PDF, etc.
  • Requires only a text editor to work with.
  • Easy to read and edit format.
  • Easy to process format.
  • Easy to get community involvement for creation and review.
  • Easy to ensure it remains up to date (if it remains with the source).
  • Simple to integrate with build systems.
  • Interacts well with modern version control systems like git.
  • Zero start up cost.
So I believe that it meets the goals that I set myself, and should be a good solution for the documentation of RISC OS and its interfaces. With an active community it should be relatively easy for people to get involved, whether it be for just simple typos, or more wide scale changes. Similarly, developers can keep their documentation right beside the code that they're working on, ensuring that it stays up to date.
 
A few of the things that I have released include some documentation in the PRM-in-XML format, because for me it's a good solution.
 
Any new things that people write could always be updated to use the PRM-in-XML format if they wished. One way of doing that is to just start out with some basic documentation and as people provide feedback on problems, or where things aren't right, it's easy to update.
 
Of course, updating all the RISC OS documentation to this format takes time and effort. But it doesn't have to be done all at once. Taking on small parts of the effort of documenting components breaks it up into nice chunks, that are relatively easy to do (if a little dull). That's where having an active community is pretty handy - anyone that's got an interest in improving things, can do so.
 
I'll write more in the future about how the PRM-in-XML can be used, but for now I think this covers what I think should be done to create consistent and community managed documentation - whether it be to replace the PRMs, or for your own software.

Resources

I'm working on releasing these resources out on GitHub, but the release from the original Build system talk is available right now and usable. The release contains documentation on how to use the tools and what the format looks like.



  1. The HTML manuals released by RISCOS Ltd were intended to make the manuals more accessible to people, and update them to a format that could be handled by the browsers of the time. This meant no special symbols, removing the class-based styling and fixing much of the poor output from FrameMaker, so that the manuals were appropriate for use by most RISC OS users. 
  2. At RISCOS Ltd, David Thomas originally did the HTML conversion from FrameMaker. Due to mistakes 17, these were released in an incomplete form with incorrectly formatted content. Recently he's been producing some patches to fix these issues so that they're in a state that he feels more comfortable with. You can find the fixes at https://github.com/dpt/PatchedPRMs
  3. 'HTML Sucks Completely' - a tool originating on the Amiga that had been ported to RISC OS well by Sergei Monesi. It was originally a tool from the Amiga and the author had a dim view if the direction that HTML was going at the time. It allowed new macro elements to be defined which could perform conditional transformations of the content which was very powerful and made building sites very easy. This easy came with a learning curve and having to use a system which was 'novel' to say the least. 
  4. For many years HSC managed the creation of my personal website. Actually it's used in the building of the riscos.online website content - it works just as well on macOS as on Linux and RISC OS 5
  5. This is the reason why I wanted to be using standardised, non-platform specific tools - they keep working years later and don't have a running cost in upgrades, or an initial cost when you start using them (well, other than the usual learning curve). 
  6. Little weird is fine, sometimes. 
  7. The BTS system is what reports the full backtrace from the first Kernel call all the way to the thing that aborted, through the DiagnosticDump module. 
  8. In the very dim past of (I think) 1998, I had been playing with updating Zap to understand module workspace blocks through structured data definitions. That would allow you to look at a given module's workspace and have symbols and more readable information about the module's operation. This didn't come to anything, but the idea of having this information available to you to aid in debug wasn't new. 
  9. Actually, that's just a feeling I get from the manner in which some parts are documented, some of the drafts of the PRM that I've seen, and how I think it worked back then - maybe it was all from specifications and documents in the drawing office, but I doubt it. 
  10. Mind numbing but it gave a really good feeling when documents were finished, or at least partially finished, enough to slot them into the index. You could see a new, structured, and maintainable, set of manuals coming out. 
  11. StrongHelp is a great viewer, but with structured information about APIs, rather than formatted API information for presentation, you could produce a much more featureful browser. 
  12. All modern editors have means for integrating intelligence into what you are editing in the form of suggestions for completions at its most basic, but also including documentation hints for APIs. 
  13. This should be done anyhow; I've just not got around to it. It's not a small job, but it's also not that hard - it just takes time to get the XSLT conversion right and produce the right stylesheets. 
  14. And trees. 
  15. It was a long time ago so I can't remember exactly. I think I was looking at maybe 10% of the pages being colour, plus the size of content meant that it probably would have to be in multiple volumes just because of the binding needs once you get above a certain number of pages - that is only worse in the case of the PRMs because despite not having colour (although the style guide does have colour) there are a lot more pages. 
  16. Styled HTML, strictly. Using tables in EPub is problematic because the device's wrapping of tables horizontally and vertically within a variable sized display is going to produces a very compromised output. Tables can't be avoided, but styling them suitably means that the device is able to give a better approximation. 
  17. The original HTML conversion came from David Thomas, I packaged it up with other things that were going on the CD and then sent it off... I'm pretty sure that's where the mistakes crept in - I think I sent off the wrong collection of documents and not the final version. Mea culpa. Oh well. 
  18. Modern versions of FrameMaker are available for subscription at about £30/month for an individual user 19. Back in the 2000s we bought it as a non-subscription application. I don't remember how much it cost but I'm certain it was not cheap. 
  19. Although it's a monthly cost, you have to commit to 12 months, so a cost of £360 for a year. 
  20. If multiple people wanted to work on it, they would have their own copies and one them would have to manually copy the changes from the other versions into a master document. This would be tedious and error prone. 
  21. The idea behind having semantic elements is that you describe what you're trying to express, rather than how you want it to look. Modern HTML has elements like <section> which indicate that a block is intended to group a section of information, where earlier versions would have merely been able to say that it was a divided block with div. Semantic markup is easier to read raw, and easier to extract information from automatically. If the content you produce uses semantic markup then changing the style that it's presented in is a lot easier. 
  22. https://pyromaniac.riscos.online/ is the main site, and the released files are in the Technologies section

 
  RISC OS Documentation
  nytrex (17:52 10/8/2021)
  CJE (11:10 11/8/2021)
  arawnsley (12:26 11/8/2021)
    nytrex (10:59 12/8/2021)
  gerph (22:31 13/8/2021)
    wpb (13:40 15/8/2021)
      gerph (18:00 15/8/2021)
        flibble (19:35 15/8/2021)
          gerph (20:19 15/8/2021)
            flibble (21:40 15/8/2021)
              gerph (23:22 15/8/2021)
                flibble (00:34 16/8/2021)
                  nytrex (12:08 16/8/2021)
                    gerph (13:23 16/8/2021)
                      nytrex (13:36 16/8/2021)
                        gerph (16:31 16/8/2021)
                          gerph (18:50 16/8/2021)
                            gerph (20:58 17/8/2021)
 
Alan Robertson Message #125171, posted by nytrex at 17:52, 10/8/2021
Member
Posts: 101
An interesting article on documenting the Programmer's Reference Manuals PRMs. Thank you.

I think everyone would agree that the documentation on RISC OS is scattered around too many locations and covering different versions of the OS. With ownership changing to RISC OS Developments Ltd (ROD) semi-recently, perhaps now is the time to try and finally get everyone in the community working on one definitive version.


RISC OS Open Ltd (ROOL) currently have a 1 of 2 PRM Documentation bounty with the first step to combine PRM 5a addendum into the rest of the manuals, along with some other deliverables. It is proposing to keep using Framemaker for this task.

I think your work on this topic should be investigated, and would be beneficial to get some people talking from ROD, ROOL, other main developers and yourself to determine if your tool could be used or not. Perhaps it needs a few extra features before it can be used, or perhaps its fit-for-purpose as-it-is. But certainly, the potential of the tool should be investigated by the people making the decisions and the developers who currently update documentation.

I helped convert a lot of the StrongHelp files to the RISC OS Open Wiki but it seems that this is a dead end going forward as the Textile format is not ideal going forward - Much like as you have already said in your article. But there is a lot of new information contained in the wiki that needs adding to the proposed consolidated PRM's.

What are you thoughts? Has anyone approached you regarding your tool and if it could be used for the official set of PRM's?

[Edited by nytrex at 18:56, 10/8/2021]
  ^[ Log in to reply ]
 
Chris Evans Message #125172, posted by CJE at 11:10, 11/8/2021, in reply to message #125171
CJE Micros chap
Posts: 228
Thanks very much Charles for your efforts.

+1 to everything Alan has said.

It's great to see there is life in the old dog yet.
  ^[ Log in to reply ]
 
Andrew Rawnsley Message #125173, posted by arawnsley at 12:26, 11/8/2021, in reply to message #125171
R-Comp chap
Posts: 593
Thanks for a thoughtful post, Alan.

Here at ROD, we've been in positive discussions with Charles for a while on this project, as it seems to us to make far more sense to move to PRM-in-XML as a base for documentation rather than anything else, *since it exists*.

I think the drive behind the bounty keeping it in Framemaker was one of "path of least resistance", but the reality is that basing the documentation in a "for paper" system like Framemaker, which has a financial cost associated, make maintenance of the PRMs nigh on impossible for those working on the OS itself.

From ROD's perspective, the key advantage of PRM-in-XML is that documentation can exist alongside the modules it pertains to, *in the Git repositories*. With documentation alongside code, it is far easier for a developer to update docs as changes are made, since they're *right there* in an open format.

From XML, it is possible to generate other formats of documentation, such as PDF and HTML. This means that it is perfectly practical to build paper-based PRMs if that's what people want to do (ROD wouldn't, but ROOL might). The key is to start with a maintainable base, rather than a proprietory base.

Bearing in mind that you were involved in this before, Alan, would you be interested in dropping me a line to discuss being involved in the future? It is certainly a project ROD are keen to move on, and indeed we've previously agreed that this is the road we want to take.

[Edited by arawnsley at 13:26, 11/8/2021]
  ^[ Log in to reply ]
 
Alan Robertson Message #125174, posted by nytrex at 10:59, 12/8/2021, in reply to message #125173
Member
Posts: 101
It's reassuring to hear that conversations regarding documentation are happening. It's such an important area as you know.

For ROOL and ROD to come to an agreement on unifying RISC OS documentation, I think it would need:

1) most/all existing developers to be happy with the any new proposed solution
2) a plan detailing how to handle the transition from all the different documentation sources (Framemaker, ROOL wiki pages, git documentation, Acorn specification documents) towards the new true source. We don not want to just add yet another source
3) a plan detailing the order in which the PRM's are unified
4) list of all known sources of documentation that should be in the PRM's (all need to be cross-referenced to determine the last/corrected inforamation)
5) still need to be able to build a paper-based PRM from the sources (assuming ROOL wanted such a thing)
6) to be a cross-platform solution as much as possible

and plenty of other things I haven't even thought of.



Andrew, expect an email from me later this afternoon. I would be delighted to help where and when I can.
  ^[ Log in to reply ]
 
Charles Justin Ferguson Message #125175, posted by gerph at 22:31, 13/8/2021, in reply to message #125171
Member
Posts: 35
I think everyone would agree that the documentation on RISC OS is scattered around too many locations and covering different versions of the OS. With ownership changing to RISC OS Developments Ltd (ROD) semi-recently, perhaps now is the time to try and finally get everyone in the community working on one definitive version.
It was the intention when I created PRM-in-XML in around 2002 that it be available for all so that any documentation that was created could easily be brought together in a relatively simple editable and processable format. There was very little interest from anyone at that time. It's been around for a very long time, and isn't at all new. So, I disagree that now is the time for it - the time for doing it was over a decade ago smile

RISC OS Open Ltd (ROOL) currently have a 1 of 2 PRM Documentation bounty with the first step to combine PRM 5a addendum into the rest of the manuals, along with some other deliverables. It is proposing to keep using Framemaker for this task.
I hope that my view on the suitability of FrameMaker for the task is clear from the article.

I think your work on this topic should be investigated, and would be beneficial to get some people talking from ROD, ROOL, other main developers and yourself to determine if your tool could be used or not. Perhaps it needs a few extra features before it can be used, or perhaps its fit-for-purpose as-it-is. But certainly, the potential of the tool should be investigated by the people making the decisions and the developers who currently update documentation.
It was created specifically for the purpose of handling the PRM's structure and format. There are a few things that could be added, but those would be 'nice to haves' rather than exclusionary requirements. Back when I was making the decisions on what would be used for documentation, that was the decision I'd made. The article is largely to say that I believe my decision to be correct and the best way to take things forward. That's not to say that there can't be better ways to do it, but FrameMaker is not one of them, and nobody has (to the best of my knowledge) even tried.

I design and build the tools that people can use. If people choose not to use them for whatever reasons, that's their choice. Although I do often design and implement things speculatively, PRM-in-XML was designed for a specific purpose which it met very well.

I helped convert a lot of the StrongHelp files to the RISC OS Open Wiki but it seems that this is a dead end going forward as the Textile format is not ideal going forward - Much like as you have already said in your article. But there is a lot of new information contained in the wiki that needs adding to the proposed consolidated PRM's.
There's always going to be new information. The intent of the PRM-in-XML format was that it was easy to add new information, without the barrier to entry that FrameMaker imposed, the stylistic and structural problems that would be caused by editing HTML, the editorial restrictions that would be imposed by using Impression or Ovation, or the limitations involved in plain text documents.

Incorporating information into the documents should be easy. But so is creating new documents which explain, in the right format and style, the changes from the earlier documents. That is important because then you can prepare documentation in new chapters (for addendums) in that way, in a similar manner to the 5 and 5a volume's chapters), and eventually integrate them to the main chapters with the minimal of difficulty. Indeed, if you want, you can just use the include feature to pull the content directly from the addendum into the main chapter.


The sequence of updating the PRMs should be very obvious really.

For each PRM chapter on a per chapter basis:

1. Convert chapter to PRM-in-XML *without changes* (ie just transfer the content).
2. Add to a suitable component in git (either a general one, or a component specific one).
3. Review and check that it's correct.

For each PRM chapter that needs updates from PRM 5a.

1. Take the converted addendum chapter and incorporate into the existing chapter.
2. Remove addendum chapter (or leave there if you're using the include mechanism).
3. Review and check that it's correct.

After you've done this:

1. Collect together the content that needs to be added, from other sources. (eg app notes, tech notes, wiki sources, etc)
2. Validate them.
3. For each of the content...
- Identify definition information (because most of the corrections will be in the definitions, not the 'technical details' sections) and incorporate into the definitions.
- Update the prose to describe the changes, because in most cases (as I said) they'll be purely changes to the definitions and the PRMs should describe the context properly.

In parallel:

1. Decide whether chapters can be split up (WindowManager was one that I split up because it really doesn't make sense to have such a monolithic chapter covering so much).
2. Decide whether chapters need expanding (Sound chapter, I'm looking at you).

For each document which is standalone:

1. Convert bare to PRM-in-XML.
2. Make sure that the style follows that of the PRMs, eg an introduction, technical details to describe what's the component does, details about the services and errors that it uses, etc
3. Validate.

For new compoents or any changes that people make:

1. Decline any documentation which is in any other format.
2. New documents created in PRM-in-XML from the start.
3. If an existing document has not been converted yet, or the content isn't appropriate, start a new one. You can always integrate it into others later, and it doesn't block you from documenting your work because there's nowhere quite right just yet.
4. When necessary, amalgamate documents to make them more suitable for consumption.


Because every single one of these chapters, or sections can be handled independant of the others, it is easy for the task to be done simultaneously either by individuals or by many people. But the important thing first is to start out with a raw conversion. Don't mix the conversion and fixes, because that way lies madness. Methodically, build yourself a baseline for the things that you're trying to do.



What are you thoughts? Has anyone approached you regarding your tool and if it could be used for the official set of PRM's?
The article you've just read is the justification for using PRM-in-XML. I believe it's the only viable solution for managing content, short of designing something different.

I'd spoken to Andrew about this a while back. Do remember that the PRM-in-XML documentation project started in 2002 and has been available since then. It's not like it's anything new, and anyone could have used it at any time since then.

As for the question of 'if it could be used for the official PRMs', that was its entire reason for being, and was what I was using it for.
  ^[ Log in to reply ]
 
W. P. B. Message #125176, posted by wpb at 13:40, 15/8/2021, in reply to message #125175
Member
Posts: 22
I found this article very interesting, and had no knowledge of PRM-in-XML before this. It certainly seems like a very sane and sustainable way to centralise and maintain the relevant documentation.

One thing that occurs to me is that the ROOL wiki, which in my opinion is currently the easiest way for part-time OS users and developers to contribute to documentation additions and corrections, would need to be tied into the system somehow.

Perhaps the wiki engine would need to be able to serve its pages directly from the XML files, which would therefore probably need to be hosted by ROOL. But a wiki engine crucially keeps update histories as well, so that would need some thought, too. I have a feeling the overall system would require quite a bit of thought on top of the excellent work done by Charles.
  ^[ Log in to reply ]
 
Charles Justin Ferguson Message #125177, posted by gerph at 18:00, 15/8/2021, in reply to message #125176
Member
Posts: 35
I found this article very interesting, and had no knowledge of PRM-in-XML before this. It certainly seems like a very sane and sustainable way to centralise and maintain the relevant documentation.
Thank you.

One thing that occurs to me is that the ROOL wiki, which in my opinion is currently the easiest way for part-time OS users and developers to contribute to documentation additions and corrections, would need to be tied into the system somehow.
The wiki is a low quality, tertiary source of information. What I mean by this is that it's not authoritative - it has no provenence or review to it - and it is derived from primary (such as the PRMs) and secondary sources such as OSLib, StrongHelp, etc). As such, it's something that may be fed from other documentation, but wouldn't be trusted by anyone explicitly needing the information. That's true of all Wikis, pretty much.

The PRMs are the authoritive, primary source of information for the operating system.

The source code is, of course, a primary source of an implementation of a version of the operating system.

There are sources such as StrongHelp (and thus the wiki), which were curated from those sources are pretty much tertiary sources. I can't really say they're secondary because they don't really provide any value over the primary other than accessibility. The information in StrongHelp (from which it seems the wiki has been derived) was found by experimentation, some internal information, various releases, etc, but isn't a reference manual. It's essentially a cheatsheet for how things could work. When they don't work that way, you go deeper to one of the other sources of information (from a tertiary source to the secondary or primary sources), until you find the answer.

Perhaps the wiki engine would need to be able to serve its pages directly from the XML files, which would therefore probably need to be hosted by ROOL. But a wiki engine crucially keeps update histories as well, so that would need some thought, too. I have a feeling the overall system would require quite a bit of thought on top of the excellent work done by Charles.
Update histories are not special to the wiki system. Management of history /is/ important, though, and it's not something that I discussed in the article, so here's a few quick words.

The source control system, git, allows all changes to be tracked together with an explanation of why they are made. The review system (which, obviously is part of GitLab) provides inline review of content before acceptance. And for public consumption, changes to the documents themselves can be annotated within the history of the document (obviously the document's history is a different level of detail to the commit history, review history, etc). Management of history is significantly better under a system where it is tracked and controlled by source management, and reviewed for accuracy.

With a correctly managed system using PRM-in-XML as the primary documentation source, the wiki serves little purpose. Information can be submitted by anyone for incorporation into the documentation. The documentation is kept in a state which is maintained as a primary source, and from which other sources may be derived. This reduces the number of places where you might find information (which, as I've mentioned, are legion).

That is one of the purposes of PRM-in-XML - it makes it easier to get information from primary source to being accessible to people. As I explained in the article, when the documentation was first received in FrameMaker format, it just wasn't going to be maintainable for the long term there. If we had made the decision at that time to keep updating the FrameMaker documentation we would be restricted ourselves in ways that would have meant that we would have been unable to keep up with the changes - because everything had to go through a bottle neck which is unsustainable.

These days, where you want community input to documentation, that's even more obvious. If you use something like FrameMaker for your documentation, you tie yourselves to a complex delivery chain that invariably involves very few people who are gatekeepers to the information. PRM-in-XML removes the gatekeepers, and the blockers that make management of the documentation hard. It allows you to move the primary source to being edited by individuals who actually know the code, or who have found problems with it. They can put the code into the source tree, with the documentation change, and have them automatically published. A user who finds a mistake in the documentation can update the sources to reflect the change, have them checked by people who know that stuff, and on approval it's rebuilt into the latest version of the documentation.

In this way, the wiki becomes redundant, because the primary source information is accessible to all, and editable by all. If there's a need for secondary sources, which are derived from the primary source, eg StrongHelp lookups, they can be derived from the PRM-in-XML sources. Another goal of the PRM-in-XML system is to provide semantic meaning to the content - allowing it to be produced in a different manner for just this reason. [1]

Maybe in the early days you would retain a wiki whilst content was being accumulated in authoritative form, but as content is transferred, and the accessible primary source is made available, the wiki pages can be deprecated and pointers to the authoritative PRMs made.

Obviously that's just my view, but high quality, accessible, authoritative information wins over a wiki pretty much any day of the week for me. And PRM-in-XML makes that a possibility in a way that the current PRMs and the Wiki do not.

[1] To be a true second source, of course, there must be additional content and analysis added, so you wouldn't purely derive the content, but consider a 'annotated PRM' which describes the development behind particular features as well... or a discussion of what the interfaces mean in good application design (which doesn't belong in a manual, and is a nice value-add to have as a separate document). Those sort of things would be akin to annotated study guides.
  ^[ Log in to reply ]
 
Peter Howkins Message #125178, posted by flibble at 19:35, 15/8/2021, in reply to message #125177
flibble

Posts: 891
I've a few questions here

1) Who owns the copyright to the existing PRM documentation (of any vintage)? So that you don't have to start from scratch.

2) What licence is any of the PRM documention under? So that it's possible for a non-NDAed community member to actually work on it.

3) What license is PRM-in-XML released under? Is it possible for community members to contribute to that?

[Edited by flibble at 20:41, 15/8/2021]
  ^[ Log in to reply ]
 
Charles Justin Ferguson Message #125179, posted by gerph at 20:19, 15/8/2021, in reply to message #125178
Member
Posts: 35
1) Who owns the copyright to the existing PRM documentation (of any vintage)? So that you don't have to start from scratch.

2) What licence is any of the PRM documention under? So that it's possible for a non-NDAed community member to actually work on it.
I don't think I'm authoritive enough to answer questions 1 or 2. However, my intention had always been that editors would grant ROL the rights to use it, and that ROL would then distribute the documentation in suitable ways. Initially through either documentation on the site, and potentially printed versions (although that was, as I said, barely a viable thing, IMO), and later as open documents that can be derived from so long as they're not misrepresented, etc. ie you couldn't make money off of it, so that 3rd party contributors would not feel ripped off (obviously - if you've got people putting their time in, you don't want people to be able to sell their work).

As I implied in the article, the way this becomes really useful is if the documentation is kept with the source code, so viewable and editable by all. A major goal is to remove that barrier between code being written and documentation being available. As anyone in professional software knows, until you've documented something it might as well not exist, and you're not done [1] [2].

3) What license is PRM-in-XML released under? Is it possible for community members to contribute to that?
As it was when it was released in 2002, and as it states in the release in November, 'distribution is unlimited', because there tools are only useful when you are unencumbered. I'm going to put it under BSD-2 clause when I publish the git repo. Git repo will come when I get time. I suspect that there's few people who'd want to contribute to the stylesheets initially, but as it'll be on github it'll be easy for people to submit pull requests.

If you've worked with XSLT (or looked at the code that I've published) you'll know that it's... amusing and it's very easy to make a big mess of things if you're not changing stuff carefully, so guidance almost certainly necessary.

Anyone wanting access to the repo in advance, please contact me. Anyone wanting to just use it, the scripts are entirely useable right now - version 1.02 is pretty stable. There's a couple of minor fixes for things that I've spotted over the last few months, but it's quite usable. I'm currently (well, not at this /very/ second, but in the current few days) updating the stylesheets for 1.03. No big changes to the format, but improved and more consistent stylesheet, 'cos I know a lot more about how to write this stuff better now.

And of course, if you want to see the output from it you'll find that in the API documentation for Pyromaniac, or the example stylesheets.

[1] In the same way that until you've got tests for it, it's not done and it's not going in.
[2] And of course, this should be mandated by the policy for components as well.
  ^[ Log in to reply ]
 
Peter Howkins Message #125180, posted by flibble at 21:40, 15/8/2021, in reply to message #125179
flibble

Posts: 891
3) What license is PRM-in-XML released under? Is it possible for community members to contribute to that?
As it was when it was released in 2002, and as it states in the release in November, 'distribution is unlimited', because there tools are only useful when you are unencumbered. I'm going to put it under BSD-2 clause when I publish the git repo.
This is fine.

As it appears not stated, it's probably worth pointing out that ROOL have already released updated versions of several manuals using their own workflow. They have

1) Believed that a commercial release is viable and desirable.
2) Targeted them at paper-based print, with PDF as a secondary release.
3) Continued to use Framemaker successfully
4) Kept them "Closed source", only specific ROOL people being capable of editing them. But taking in suggestions and changes from the community (on the User Guide at least)
6) Not released the manuals as re-distributable even when released as a free download.
7) Fiercely held the copyright on the updates done (even docs that have had community input).

I think the biggest hurdle here isn't changing what tool is used, but convincing an open source project that it's docs could be open too.

I won't go into a long rant here, but simply put on the issue of docs, ROOL are doing it wrong.

[Edited by flibble at 22:40, 15/8/2021]
  ^[ Log in to reply ]
 
Charles Justin Ferguson Message #125181, posted by gerph at 23:22, 15/8/2021, in reply to message #125180
Member
Posts: 35
3) Continued to use Framemaker successfully
Please don't get me wrong; FrameMaker is a good tool, but I feel that it's unsuited to being used in a way that works well with source control, with multiple people editing things, being easy to add new content, and doesn't allow any reuse of the content. In particular it effectively forces those editors of the documents to be gatekeepers, which is particularly undesirable as the authors of the component are best to provide the review of documentation, not just someone who happens to have a license to use a given application.

I suspect that the fact that it has been used successfully is largely because none of my goals are part of the intended use cases.

That said, I have tried to be very careful to focus on what the tool does, and why it is appropriate for the task. I feel it is technically the best solution to the problem. I've made my case for it, and if people don't want to use it then... I'm used to disappointment big smile
  ^[ Log in to reply ]
 
Peter Howkins Message #125182, posted by flibble at 00:34, 16/8/2021, in reply to message #125181
flibble

Posts: 891
3) Continued to use Framemaker successfully
Please don't get me wrong;
Ditto. I'm certain that this is a better tool. However it doesn't match with ROOL's desire to keep the docs closed and to sell them.

So it's a rather moot point :-(

[Edited by flibble at 01:35, 16/8/2021]
  ^[ Log in to reply ]
 
Alan Robertson Message #125183, posted by nytrex at 12:08, 16/8/2021, in reply to message #125182
Member
Posts: 101
I had a good read through the prm-in-xml documentation at the weekend and came away rather impressed by the capabilities. It does indeed seem to be a really good solution.

I also so happened to come across a recent article posted by the Raspberry Pi Foundation where they discuss their project to migrate their documentation to the AsciiDoc language from their previous native git markup language. I then started to investigate AsciiDoc, and came away with the belief that's not suitable for our use - it just seems to be a more advanced textile format that the ROOL wiki currently uses.

So, this morning I tried converting Chapter 3 of the PRM's to HTML, and came away with an awful headache due to the awful export output. I would hope exporting to HTML from Framemaker produces a better output.

I'll try different converters and see if any will produce a format that's suitable and if not, then I will use the prm-in-xml tool to create a skeleton file and start a manual conversion of a SWI and learn the markup language, and test the html output it generates.

I will keep you posted. I'm looking forward to it.
  ^[ Log in to reply ]
 
Charles Justin Ferguson Message #125185, posted by gerph at 13:23, 16/8/2021, in reply to message #125183
Member
Posts: 35
I had a good read through the prm-in-xml documentation at the weekend and came away rather impressed by the capabilities. It does indeed seem to be a really good solution.

I also so happened to come across a recent article posted by the Raspberry Pi Foundation where they discuss their project to migrate their documentation to the AsciiDoc language from their previous native git markup language. I then started to investigate AsciiDoc, and came away with the belief that's not suitable for our use - it just seems to be a more advanced textile format that the ROOL wiki currently uses.

So, this morning I tried converting Chapter 3 of the PRM's to HTML, and came away with an awful headache due to the awful export output. I would hope exporting to HTML from Framemaker produces a better output.
Now you have an idea why David Thomas hated that process.

I'll try different converters and see if any will produce a format that's suitable and if not, then I will use the prm-in-xml tool to create a skeleton file and start a manual conversion of a SWI and learn the markup language, and test the html output it generates.

I will keep you posted. I'm looking forward to it.
I have perl scripts that will convert the HTML that we generated into PRM-in-XML-like output. It still needs tidying but it's pretty good.

I have already converted a good chunk of the PRMs to PRM-in-XML so please feel free to use it as a learning exercise but be aware that there has been a lot of work done.

I have started writing a guide for how to do it as part of a git repo; if you want to work through that then contact me and I'll make it available to you. It's part of an ongoing process to make these tools available.

Drop me an email if you want to discuss this process more.
  ^[ Log in to reply ]
 
Alan Robertson Message #125186, posted by nytrex at 13:36, 16/8/2021, in reply to message #125185
Member
Posts: 101
Drop me an email if you want to discuss this process more.
Look out for an email soon. And thank your for your offer of assistance and help.
  ^[ Log in to reply ]
 
Charles Justin Ferguson Message #125187, posted by gerph at 16:31, 16/8/2021, in reply to message #125186
Member
Posts: 35
Drop me an email if you want to discuss this process more.
Look out for an email soon. And thank your for your offer of assistance and help.
Thanks; I've passed on (hopefully) access to the site, and some more information.

If anyone else is interested, please do drop me a line. It's one thing reading about something, and another in seeing proper examples, and yet more important to play with things yourself. Which doesn't always mean that you'll be more convinced... but at least you have more confidence that you have the right information big smile
  ^[ Log in to reply ]
 
Charles Justin Ferguson Message #125188, posted by gerph at 18:50, 16/8/2021, in reply to message #125187
Member
Posts: 35
Alan has pointed out that it wasn't obvious where the RISC OS tools were located on the GitHub site. I forget the obvious things to me 'cos I use it day to day...

GitHub is most comonly used to distribute sources, but the built binaries can usually be found under 'Releases' on the right side of the page. Within the releases, you'll find downloadable source code for a given release, and - in the case of my RISC OS CI-built components - RISC OS zip archives of the built versions.

https://github.com/gerph/libxml2/releases/tag/v1.39 (DO NOT USE - use the ones in the next post)
and
https://github.com/gerph/libxslt/releases/tag/v1.39 (DO NOT USE - use the ones in the next post)

are the shortcuts for the github components.
It's still a little fiddly to use the PRM-in-XML build on RISC OS because it's not a platform I use regularly (!)

I'll do a more complete release later (more like the 2002 version!), but I wanted to make sure everything was available first, even if it wasn't perfect.

[Edited by gerph at 21:59, 17/8/2021]
  ^[ Log in to reply ]
 
Charles Justin Ferguson Message #125189, posted by gerph at 20:58, 17/8/2021, in reply to message #125188
Member
Posts: 35
Turns out those versions didn't work on RISC OS 5. They work find on RISC OS 4 but there's something awry that causes a crash on RISC OS 5.

I've released updates for the tools:

https://github.com/gerph/libxml2/releases/tag/v1.40
and
https://github.com/gerph/libxslt/releases/tag/v1.40
  ^[ Log in to reply ]
 

The Icon Bar: News and features: RISC OS Documentation