SiSU - SiSU information Structuring Universe - Structured information, Serialized Units,
Ralph Amissah

Structured information, Serialized Units

SiSU - from less markup than the most elementary equivalent html, you can have more

1. Description

1.1 Outline

SiSU is a flexible document preparation, generation publishing and search system.  ¹ 

SiSU ("SiSU information Structuring Universe" or "Structured information, Serialized Units"),  ²  is a Unix command line oriented framework for document structuring, publishing and search. Featuring minimalistic markup, multiple standard outputs, a common citation system, and granular search.

Using markup applied to a document, SiSU can produce plain text, HTML, XHTML, XML, OpenDocument, EPUB, LaTeX or PDF files, and populate an SQL database with objects  ³  (equating generally to paragraph-sized chunks) so searches may be performed and matches returned with that degree of granularity (e.g. your search criteria is met by these documents and at these locations within each document). Document output formats share a common object numbering system for locating content. This is particularly suitable for "published" works (finalized texts as opposed to works that are frequently changed or updated) for which it provides a fixed means of reference of content.

SiSU is the data/information structuring and transforming tool, that has resulted from work on one of the oldest law web projects. It makes possible the one time, simple human readable markup of documents, that SiSU can then publish in various forms, suitable for paper  ⁴  , web  ⁵  and relational database  ⁶  presentations, retaining common data-structure and meta-information across the output/presentation formats. Several requirements of legal and scholarly publication on the web have been addressed, including the age old need to be able to reliably cite/pinpoint text within a document, to easily make footnotes/endnotes, to allow for semantic document meta-tagging, and to keep required markup to a minimum. These and other features of interest are listed and described below. A few points are worth making early (and will be repeated a number of times):

(i) The SiSU document generator was the first to place material on the web with a system that makes possible citation across different document types, with paragraph, or rather object citation numbering  ⁷  a text positioning system, available for the pinpointing of text, 1997, a simple idea from which much benefit, and SiSU remains today, to the best of my knowledge, the only multiple format e-book/ electronic-document system on the web that gives you this possibility (including for relational databases).

(ii) Markup is done once for the multiple formats produced.

(iii) Markup is simple, and human readable (with a little practice), in almost all cases there is less and simpler markup required than basic html. In any event the markup required is very much simpler than the html, EPUB, LaTeX, [lout], structured XML, ODF (OpenDocument), PostgreSQL or SQLite feed etc. that you can have SiSU generate for you.

(iv) SiSU is a batch processor, dealing with as many files as you need to generate at a time.

(v) Scalability is dependent on your file system (in my case Reiserfs), the database (currently Postgresql and/or SQLite) and your hardware.

SiSU Sabaki  ⁸  (or just SiSU) is the provisional name given to the software described here that helps structure documents for web and other publication. The name SiSU is a loose anagram for something along the lines of "SiSU is structuring unit", or "SiSU, information structuring unit" or the more descriptive "Structured information, Serialized Units" or "simple - information structuring unit" or the more descriptive "Structured information, Serialized Units" or what it may be directed towards "*semantic* and information structuring universe",  ⁹  tongue in cheek, only just. Guess I'll get away with "Simple - information Structuring Universe". SiSU is also a Finnish word roughly meaning guts, inner strength and perseverance.  ¹⁰ 

SiSU was born of the need to find a way, with minimal effort, and for as wide a range of document types as possible, to produce high quality publishing output in a variety of document formats. As such it was necessary to find a simple document representation that would work across a large number of document types, and the most convenient way(s) to produce acceptable output formats. The project leading to this program was started in 1993 (together with the trade law project now known as Lex Mercatoria) as an investigation of how to effectively/efficiently place documents on the web. The unified document handling, together with features such as paragraph numbering, endnote handling and tables... appeared in 1996/97. SiSU was originally written in Perl,  ¹¹  and converted to Ruby,  ¹²  in 2000, one of the most impressive programming languages in existence! In its current form it has been written to run on the Gnu/Linux platform, and in particular on Debian,  ¹³  taking advantage of many of the wonderful projects that are available there.

SiSU markup is based on requiring the minimum markup needed to determine the structure of a document. (This can be as little as saying in a header to look for the word Book at a specified level and the word Chapter at another level). SiSU then breaks a document into its smallest parts (at a heading, and paragraph level) while retaining all structural information. This break up of the document and information on its structure is taken advantage of in the transformations made in generating the very different output types that can be created, and in providing as much as can be for what each output type is best at doing, e.g. LaTeX (professional document typesetting, easy conversion to pdf or Postscript), EPUB, XML (in this case, structural representation), ODF (OpenDocument [experimental]), SQL (e.g. document search; representing constituent parts of documents based on their structure, headings, chapters, paragraphs as required; user control).  ¹⁴ 

From markup that is simpler and more sparse than html you get:

For more see the short summary of features provided below.

SiSU processes files with minimal tagging to produce various document outputs including html, EPUB, ODF, LaTeX (which is converted to pdf) and if required loads the structured information into an SQL database (PostgreSQL and SQLite have been used for this). SiSU produces an intermediate processing format.  ¹⁵ 

SiSU was originally used in constructing Lex Mercatoria ‹http://lexmercatoria.org/› or ‹http://www.jus.uio.no/lm/› (one of the oldest law web sites), and considerable thought went into producing output that would be suitable for legal and academic writings (that do not have formulae) given the limitations of html, and publication in a wide variety of "formats", in particular in relation to the convenient and accurate citation of text. However, the construction of Lex Mercatoria uses only a fraction of the features available from SiSU today, /vis/ generation of flat file structures, rather than in addition the building of ("granular") SQL database content, (at an object level with relevant relational tables, and other outputs also available).

1.2 Short summary of features

(i) markup syntax: (a) simpler than html, (b) mnemonic, influenced by mail/messaging/wiki markup practices, (c) human readable, and easily writable,

(ii) (a) minimal markup requirement, (b) single file marked up for multiple outputs,

notes:

* documents are prepared in a single UTF-8 file using a minimalistic mnemonic syntax. Typical literature, documents like "War and Peace" require almost no markup, and most of the headers are optional.

* markup is easily readable/parsed by the human eye, (basic markup is simpler and more sparse than the most basic html), [this may also be converted to XML representations of the same input/source document].

* markup defines document structure (this may be done once in a header pattern-match description, or for heading levels individually); basic text attributes (bold, italics, underscore, strike-through etc.) as required; and semantic information related to the document (header information, extended beyond the Dublin core and easily further extended as required); the headers may also contain processing instructions.

(iii) (a) multiple outputs primarily industry established and institutionally accepted open standard formats, include amongst others: plaintext (UTF-8); html; EPUB; (structured) XML; ODF (Open Document text)l; LaTeX; PDF (via LaTeX); SQL type databases (currently PostgreSQL and SQLite). Also produces: concordance files; document content certificates (md5 or sha256 digests of headings, paragraphs, images etc.) and html manifests (and sitemaps of content). (b) takes advantage of the strengths implicit in these very different output types, (e.g. PDFs produced using typesetting of LaTeX, databases populated with documents at an individual object/paragraph level, making possible granular search (and related possibilities))

(iv) outputs share a common numbering system (dubbed "object citation numbering" (ocn)) that is meaningful (to man and machine) across various digital outputs whether paper, screen, or database oriented, (PDF, html, EPUB, XML, Opendocument, sqlite, postgresql), this numbering system can be used to reference content.

(v) SQL databases are populated at an object level (roughly headings, paragraphs, verse, tables) and become searchable with that degree of granularity, the output information provides the object/paragraph numbers which are relevant across all generated outputs; it is also possible to look at just the matching paragraphs of the documents in the database; [output indexing also work well with search indexing tools like hyperesteier].

(vi) use of semantic meta-tags in headers permit the addition of semantic information on documents, (the available fields are easily extended)

(vii) creates organised directory/file structure for (file-system) output, easily mapped with its clearly defined structure, with all text objects numbered, you know in advance where in each document output type, a bit of text will be found (e.g. from an SQL search, you know where to go to find the prepared html output or PDF etc.)... there is more; easy directory management and document associations, the document preparation (sub-)directory may be used to determine output (sub-)directory, the skin used, and the SQL database used,

(viii) "Concordance file" wordmap, consisting of all the words in a document and their (text/ object) locations within the text, (and the possibility of adding vocabularies),

(ix) document content certification and comparison considerations: the document and each object within it stamped with an md5 hash making it possible to easily check or guarantee that the substantive content of a document is unchanged.

(x) SiSU's minimalist markup makes for meaningful "diffing" of the substantive content of markup-files,

(xi) easily skinnable, document appearance on a project/site wide, directory wide, or document instance level easily controlled/changed,

(xii) in many cases a regular expression may be used (once in the document header) to define all or part of a documents structure obviating or reducing the need to provide structural markup within the document,

(xiii) prepared files may be batch process, documents produced are static files so this needs to be done only once but may be repeated for various reasons as desired (updated content, addition of new output formats, updated technology document presentations/representations)

(xiv) possible to pre-process, which permits: the easy creation of standard form documents, and templates/term-sheets, or; building of composite documents (master documents) from other sisu marked up documents, or marked up parts, i.e. import documents or parts of text into a main document should this be desired

there is a considerable degree of future-proofing, output representations are "upgradeable", and new document formats may be added.

(xv) there is a considerable degree of future-proofing, output representations are "upgradeable", and new document formats may be added: (a) modular, (thanks in no small part to Ruby) another output format required, write another module.... (b) easy to update output formats (eg html, XHTML, EPUB, LaTeX/PDF produced can be updated in program and run against whole document set), (c) easy to add, modify, or have alternative syntax rules for input, should you need to,

(xvi) scalability, dependent on your file-system (ext3, Reiserfs, XFS, whatever) and on the relational database used (currently Postgresql and SQLite), and your hardware,

(xvii) only marked up files need be backed up, to secure the larger document set produced,

(xviii) document management,

(xix) Syntax highlighting for SiSU markup is available for a number of text editors.

(xx) remote operations: (a) run SiSU on a remote server, (having prepared sisu markup documents locally or on that server, i.e. this solution where sisu is installed on the remote server, would work whatever type of machine you chose to prepare your markup documents on), (b) generated document outputs may be posted by sisu to remote sites (using rsync/scp) (c)document source (plaintext utf-8) if shared on the net may be identified by its url and processed locally to produce the different document outputs.

(xxi) document source may be bundled together (automatically) with associated documents (multiple language versions or master document with inclusions) and images and sent as a zip file called a sisupod, if shared on the net these too may be processed locally to produce the desired document outputs, these may be downloaded, shared as email attachments, or processed by running sisu against them, either using a url or the filename.

(xxii) for basic document generation, the only software dependency is Ruby, and a few standard Unix tools (this covers plaintext, html, EPUB, XML, ODF, LaTeX). To use a database you of course need that, and to convert the LaTeX generated to PDF, a LaTeX processor like tetex or texlive.

as a developers tool it is flexible and extensible

SiSU was developed in relation to legal documents, and is strong across a wide variety of texts (law, literature...). SiSU handles images but is not suitable for formulae/ statistics, or for technical writing at this time.

SiSU has been developed and has been in use for several years. Requirements to cover a wide range of documents within its use domain have been explored.

Some modules are more mature than others, the most mature being html and LaTeX / pdf. PostgreSQL and search functions are useable and together with /ocn/ unique (to the best of my knowledge). The XML output document set is "well formed" but largely proof of concept.

1.3 How it works

SiSU markup is fairly minimalistic, it consists of: a (largely optional) document header, made up of information about the document (such as when it was published, who authored it, and granting what rights) and any processing instructions; and markup within text which is related to document structure and typeface. SiSU must be able to discern the structure of a document, (text headings and their levels in relation to each other), either from information provided in the instruction header or from markup within the text (or from a combination of both). Processing is done against an abstraction of the document comprising of information on the document's structure and its objects,  ¹⁶  which the program serializes (providing the object numbers) and which are assigned hash sum values based on their content. This abstraction of information about document structure, objects, (and hash sums), provides considerable flexibility in representing documents different ways and for different purposes (e.g. search, document layout, publishing, content certification, concordance etc.), and makes it possible to take advantage of some of the strengths of established ways of representing documents, (or indeed to create new ones).

1.4 Simple markup

SiSU markup is based on requiring the minimum markup needed to determine the structure of a document. (This can be as little as saying in a header to look for the word Book at a specified level and the word Chapter at another level). SiSU then breaks a document into its smallest parts (at a heading, and paragraph level) while retaining all structural information. This break up of the document and information on its structure is taken advantage of in the transformations made in generating the very different output types that can be created, and in providing as much as can be for what each output type is best at doing, e.g. LaTeX (professional document typesetting, easy conversion to pdf or Postscript), EPUB, XML (in this case, structural representation), ODF (OpenDocument), SQL (e.g. document search; representing constituent parts of documents based on their structure, headings, chapters, paragraphs as required; user control).  ¹⁷ 

1.4.1 Sparse markup requirement, try to get the most out of markup

One of its strengths is that very small amounts of initial tagging is required for the program to generate its output.

This is a basic markup example:

Emphasis has been on simplicity and minimalism in markup requirements. Design philosophy is to try keep the amount of markup required low, for whatever has been determined to be acceptable output.  ¹⁹ 

SiSU's markup is more minimalistic and simpler than (the equivalent) html and for it, you get considerably more than just html, as this preparation gives you all available output formats, upon request.

1.4.2 Single markup file provides multiple output formats

For each document, there is only one (input, minimalistically marked up) file from which all the available output types are generated.  ²⁰ 

Eg. the markup example:

Produces the following output:

(and in addition to these: PostgreSQL, SQLite, texinfo and YAML   ³³  versions if desired)

1.4.3 Syntax relatively easy to read and remember

Syntax is kept simple and mnemonic.  ³⁴ 

1.4.4 Kept simple by having a limited publishing feature set, and features identified as most important, are available across several document types

To keep SiSU markup sparse and simple SiSU deliberately provides a limited publishing feature set, including: indent levels; bold; italics; superscript; subscript; simple tables; images; tables of contents and; endnotes. Which in most cases are available across the different output formats.

The publishing feature set may be expanded as required.

1.5 Designed with usability in mind

Output is designed to be uniform, easy to read, navigate and cite.

1.6 Code separate from content

Code  ³⁵  is separated from content. This means that when changes are desired in the output presentation, the code that produces them, and not the marked up text data set (which could be thousands of documents) is modified. Separating code from content makes large scale changes to output appearance trivial, and permits the easy addition of new output modules.

1.7 Object citation numbering, a text or object positioning / citation system - "paragraph" (or text object) numbering, that remains same and usable across all output formats by people and machine

Object citation numbering is a simple object (text) positioning and cition system that is human relevant and machine useable, used by SiSU for all manner of presentations, and that is available for use in all text mappings. It is based on the automated sequential numbering of objects (roughly paragraphs, (headings, tables, verse) or other blocks of text or images etc.). The text positioning system (in which I claim copyright) is invaluable for publishing requiring the citing text across multiple output formats, and for the general mapping of text within a document:

I claim copyright on the system I use which is the most basic of all, numbering all text in headings and paragraphs sequentially (with tables and images being treated as a single paragraph) and only footnotes/endnotes not following this numbering, as their position in text is not strictly determined, (a change from footnotes to endnotes would change their numbering), footnotes instead "belong" to the paragraph from which they are referenced, and have sequential numbers of their own.

SiSU has a paragraph numbering system, that remains the same regardless of the output format. This provides an effective means of citation, pinpointing text accurately in all output formats, using the same reference. This is particularly useful where text has to be located across different output formats - for example once html is printed the number of pages and pages on which given text is found will vary depending on the browser, its settings the font size setting etc. Similarly SiSU produces pdf in different forms, eg. on the example site Lex Mercatoria as portrait and landscape documents - here too page numbering varies, but paragraph numbering is the same, vis a vis all versions of the text (portrait and landscape pdf and the html versions of the text, and as stored (with "paragraphs" as records) to the PostgreSQL or SQLite database).

These numbers are placed in the text margins and are intended to be independent of and not to interfere with authors tagging. [The citation system (object citation numbering system, automated "paragraph numbering") which is automatically generated and is common and identical across all document formats] The paragraph numbering system is more accurately described as an (text) object numbering system, as headings are also numbered... all headings and paragraphs are numbered sequentially. Endnotes are automatically numbered independently and rather "belong" to the paragraph from which they are referenced, as an endnote does not (necessarily) form a part of a documents sequence, (they may be produced as either endnotes or footnotes (or both depending on what output you choose to look at - if you take the segmented html version document provided as an example, you will find that the endnotes are placed both at the end of each section, and in a separate section of their own called endnotes, and these are hyper-linked)). An attractive feature of providing citation numbering in this way is that it is independent of the document structure... it remains the same regardless of what is done about the document structure.

The rules have been kept very simple, unique incremental object citation numbers are assigned to headings, paragraphs, verse, tables and images. It is possible to manually override this feature on a per heading or comment basis though this should be used exceptionally, it may be of use where there a substantive text, and the addition of a minor comment by the publisher that should not be mapped as part of the text.

The object citation number markers contain additional numbering information with regard to the document structure, that can be used for alternative presentations, including such detail as the type of object (heading, paragraph, table, image, etc.), numbered sequentially.

An advantage is that the numbering remains the same regardless of document structure.

Text object ("paragraph") numbering is the same for all output versions of the same document, vis html, epub, pdf, pgsql, etc.

In the relational database, as individual text objects of a document stored (and indexed) together with object numbers, and all versions of the document have the same numbering, the results of searches may be tailored just to provide the location of the search result in all available document formats.

Note: there is a bug in the released behaviour of object citation numbering, (not certain when it was introduced) tables should be numbered, ie each table gets an ocn, required amongst other things for relational database. This will be corrected in a future release. Citation numbering of existing documents that contain tables will changed.

1.8 Handling of Dublin Core meta-tags making use of the Resource Description Framework

SiSU is able to use meta tags based on the Dublin Core  ³⁷  and Resource Description Framework  ³⁸ 

This provides the means of providing semantic information about a document, both as computer processable meta-tags, and as human readable information that may be of value for classification purposes.

This information is provided both in html metatags, and (where available) under the section titled "Document Information - Metadata", near the end of a document, for example in the segmented html version of this text at: ‹http://www.jus.uio.no/sisu/SiSU/metadata.html›

1.9 Easy directory management

1. Directory file association, skins and special image management, made simpler.  ³⁹ 

The last part of the name of the work directory in which markup is being done, or rather from where SiSU is run in order to generate document output, is used in determining the sub-directory name for output files, that is created in the document output directory. This provides a rather easy way to associate documents e.g. of a given subject, or by owner.

  /www/docs
      /intellectual_property
      /arbitration
      /contract_law

  /www/docs
      /ralph
      /sisu

all are placed in their own directories within the directory structure created. Similar rules are used in the creation of sql type databases (though they can be overridden).

There are a couple of further associations with these directories.

Directory wide skins.

Directory specific images.

2. If there is a "directory skin", that is a skin of the same name as the directory, it is used in the generation of the documents within it, rather than the default skin, unless the document has a specific skin associated with it.

a. default skin (always available)

b. directory skin (precedence over default if exists)

c. document skin (takes precedence wherever document requests a specific skin)

Skins are defined in the document skin directory and if a directory association is desired a softlink made to the relevant skin. Skins (directory association auto load) auto load skin if a directory skin exists of same name as directory stub, (and there is no specific doc skin)

3. If the working directory has within it a sub-directory called image_local, the images within that directory are used for references to images, that are not part of the default site build.

1.10 Document Version Control Information

The possibility of citing an exact document version.

Permits the inclusion of document version control information to the document body and metatags.  ⁴⁰  This provides a much more certain method of referring to the exact version of a particular document, (assuming that the document is from a trusted source, that will retain earlier versions of a document).  ⁴¹ 

This information (where available) is provided under the section of the document titled "Document Information - MetaData", near the end of a document, for example in the segmented html version of this text at: ‹http://www.jus.uio.no/sisu/SiSU/metadata.html›

1.11 Table of contents

SiSU produces a rudimentary a table of contents based on document headings.

1.12 Auto-numbering of headings

Headings can be automatically numbered, (and automatically named for hyper-linking)

1.13 Numbering and cross-hyperlinking of endnotes

SiSU can automatically number footnotes/endnotes. This is the default operation where no number is provided.

Footnotes/endnotes may also be manually numbered. Where a number, or numbers are provided for a footnote/endnote, this does not increment the automatic footnote/endnote number counter.

In the html output footnotes/endnotes are cross-hyper-linked (to their reference point and vice versa). In th pdf output footnotes are linked from their reference point only.

1.14 "Skinnable"

SiSU is skinnable, on a site-wide, directory-wide and per document basis, so different looking versions of things may be produced with little difficulty. There is a default skin which may be modified, as the background site skin, and each working directory may have a skin associated with it, as may each individual document. The hierarchy of application is document, directory, then site... ie if a document skin exists it gets precedence.

Whilst it is skinnable, the default output styles are selected to work across the widest possible range of document types.

1.15 Multiple Outputs

From markup that is simpler and more sparse than html you get:

As many output formats/presentations as one cares to write modules for - several types of html (e.g. structure based on css, or structure based on tables); LaTeX/pdf and Lout/pdf; pgsql other databases easily added; yaml...

1.15.1 html - several presentations: full length & segmented; css & table based

Most documents are produced in single and segmented html versions, described below:

The Scroll (full length text presentations)

The full length of the text in a single scrollable document.  ⁴³  As a rule the files they are saved in are named: /doc/ or more precisely doc.html

For various reasons texts may only be provided in this form (such as this one which is short), though most are also provided as segmented texts.

"Scroll" is a reference to the historical scroll, a single long document/ parchment, and also no doubt to what you will have to do to get to the bottom of the text.  ⁴⁴ 

The Segmented Text

The text divided into segments (such as articles or chapters depending on the text)  ⁴⁵  As a rule the files they are saved in are named: /toc/ and /index/ or more precisely toc.html and index.html

If you know exactly what you are looking for, loading a segment of text is faster (the segments being smaller). Occasionally longer documents such as the WTA 1994 ‹http://www.jus.uio.no/lm/wta.1994/toc› are only provided in segmented form.

Cascading Style Sheet, and Table based html

SiSU outputs html, two current standard forms available are:

css based

and

table based [largely discontinued]  ⁴⁶ 

The html is tested across several browsers

I like to remind you that there are other excellent browsers out there, many of which have long supported practical features like tabbing.

The html is tested across several browsers, including:

Also lighter weight graphical browsers:

And for console/text browsing:

The html tables output is rendered more accurately across a wider variety set and older versions of browsers (than the html css output).

1.15.2 EPUB

SiSU generates EPUB documents.

1.15.3 XML

SiSU generates well formed XML, and multiple versions. An XML SAX version with a flat/shallow structure, and XML DOM version with a deeper (embedded) structure. There is also a released working xhtml module. Examples of SAX and DOM versions are provided within this document.

1.15.4 ODT:ODF, Open Document Format - ISO/IEC 26300:2006

SiSU generates Open Document Output format.

1.15.5 PDF - portrait and landscape, (through the generation of LaTeX output which is then transformed to pdf)

SiSU outputs LaTeX if required which is easily transformed to PDF.  ⁶⁰  PDF documents are generated on the site from the same source files and Ruby program that produce html. Landscape oriented pdf introduced, providing easier screen viewing, they are also (paper saving, being currently) formatted to have fewer pages than their portrait equivalents.

1.15.6 Search - loading/populating of relational database while retaining document structure information, object citation numbering and other features (currently PostgreSQL and/or SQLite)

SiSU (from the same markup input file) automatically feeds into PostgreSQL  ⁶⁴  and/or SQLite  ⁶⁵  database (could be any other of the better relational databases)  ⁶⁶  - together with all additional information related to document structure, and the alternative ways in which it is generated on the site retained. As regards scaling of the database, it is as scalable as the database (here Postgresql or SQLite) and hardware allow. I will prune the images later.

This is one of the more interesting output forms, as all the structural data for the documents are retained (though can be ignored by the user of the database should they so choose). All site texts/documents are (currently) streamed to four pgsql database tables:

There is of course the possibility to add further structures.

At this level SiSU loads a relational database with documents broken in to their smallest logical structurally constituent parts, as text objects, with their object citation number and all other structural information needed to construct the structured document. Text is stored (at this text object level) with and without elementary markup tagging, the stripped version being so as to facilitate ease of searching.

Because the document structure of sites created is clearly defined, and the text object citation system is available for all forms of output, it is possible to search the sql database, and either read results from that database, or just as simply map the results to the html output, which has richer text markup.

The combination of the SiSU citation system with a relational database is pretty powerful, giving rise to several possibilities. As individual text objects of a document stored (and indexed) together with object numbers, and all versions of the document have the same numbering, complex searches can be tailored to return just the locations of the search results relevant for all available output formats, with live links to the precise locations in the database or in html/xml documents; or, the structural information provided makes it possible to search the full contents of the database and have headings in which search content appears, or to search only headings etc. (as the Dublin Core is incorporated it is easy to make use of that as well).

This is a larger scale project, (with little development on the front end largely ignored), though the "infrastructure" has been in place since 2002.

1.15.7 Search - database frontend sample, utilising database and SiSU features, including object citation numbering (backend currently PostgreSQL)

Sample search frontend   ⁶⁷  A small database and sample query front-end (search from) that makes use of the citation system, object citation numbering to demonstrates functionality.  ⁶⁸ 

SiSU can provide information on which documents are matched and at what locations within each document the matches are found. These results are relevant across all outputs using object citation numbering, which includes html, EPUB, XML, LaTeX, PDF and indeed the SQL database. You can then refer to one of the other outputs or in the SQL database expand the text within the matched objects (paragraphs) in the documents matched.

(further work needs to be done on the sample search form, which is rudimentary and only passes simple booleans correctly at present to the SQL engine)

A few canned searches, showing object numbers. Search for:

English documents matching Linux OR Debian

GPL OR Stallman

invention OR innovation

copyright in English language documents

Note that the searches done in this form are case sensitive.

Expand those same searches, showing the matching text in each document:

English documents matching Linux OR Debian

GPL OR Stallman

invention OR innovation

copyright

Note you may set results either for documents matched and object number locations within each matched document meeting the search criteria; or display the names of the documents matched along with the objects (paragraphs) that meet the search criteria.  ⁶⁹ 

OCN index mode, (object citation number) the numbers displayed are relevant (and may be used to reference the match) in any sisu generated rendition of the text  ⁷⁰  the links provided are to the locations of matches within the html generated by SiSU.

Paragraph mode, you may alternatively display the text of each paragraph in which the match was made, again the object/paragraph numbers are relevant to any SiSU generated/published text.

Several options for output - select database to search, show results in index view (links to locations within text), show results with text, echo search in form, show what was searched, create and show a "canned url" for search, show available search fields. Also shows counters number of documents in which found and number of locations within documents where found. [could consider sorting by document with most occurrences of the search result].

Simple search, results with files in which search found, and text object (paragraph or endnote) where found within files.

1.15.8 Other forms

There are other forms as well, YAML file, Ruby Marshal dumps, document pre-processing (processing of documents prior to the steps described here, to produce input suitable for the program) snap in a new module as required/desired, well formed XML, no problem.

1.16 Concordance / Word Map or rudimentary index

Concordance /WordMaps:  ⁷¹  SiSU produces a rudimentary index based on the words within the text, making use of paragraph numbers to identify text locations. This is generated in html and hyper-linked but identifies these words locations in the other document formats. Though it is possible to search using a search engine, this is a means for browsing an alphabetical list of words which may suggest other useful content.

1.17 Managed (document) directory, database, or site structure

SiSU builds the web site (or more generically provides a suitable directory structure) - placing various output texts in the hierarchy of the web-site (or db), which (for directories) is a sub-directory with the name of the text file.

1.18 Batch processing

SiSU is a batch processing tool, handling and transforming multiple (or individual) documents (in many ways) with a single instruction.

1.19 Integration to superior Gnu/Linux and Unix tools

As should have been noted by the above description of SiSU, it makes use of existing programs found on Gnu/Linux and Unix, amongst those already mentioned include the LaTeX to pdf converters and the database PostgreSQL or SQLite.

1.19.1 Backup and version control

Unix provides many tools for version control. For documents Subversion, CVS and even the old RCS are useful for the per-document histories they provide.

For writing code superior (more recent) version control system exist. These can also be used for documents though they tend to take stamps of changes across the repository as a whole, rather than for each individual file that is tracked, (as CVS and RCS do). My personal preference is for distributed systems such as Git, Mercurial or Darcs, of which I use Git for both code and documents.

Several backup tools exist. At the base level I tend to use rdiff.

1.19.2 Editor support

SiSU documents are prepared / marked up in utf-8 text you are free to use the text editor of your choice.

Syntax highlighting for a number of editors are provided. Amongst them Vim, Kwrite, Kate, Gedit and diakonos. These may be found with configuration instructions at ‹http://www.sisudoc.org/sisu/sisu_syntax_highlighting/doc.html› Vim   ⁷²  as of version 7 has built in sytax highlighting for SiSU.

1.20 Modular design, need something new add a module

Need a new output format that does not already exist, write a new module.

Prefer a new input syntax, you could write a new syntax matching the existing design, though my personal preference is some uniformity in entry appearance. If necessary has been fairly easy to extend the design parameters. It is intended to incorporate some additional basic semantic tagging, (book, article, author etc.) However, keeping the requirements for input minimal, and relatively simple has been a design goal.

2. Markup and Output Examples

2.1 Markup examples

Current markup examples and document output samples are provided at ‹http://www.jus.uio.no/sisu/SiSU/examples.html›

For some documents hardly any markup at all is required at all, other than a header, and an indication that the levels to be taken into account by the program in generating its output are.

2.2 A few book (and other) examples

Aukio, by Leena Krohn

2.2.1 "Viral Spiral", David Bollier

"Viral Spiral", David Bollier
      document manifest   ⁷⁴ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"The Wealth of Networks", Yochai Benkler

"The Wealth of Networks", Yochai Benkler
      document manifest   ⁷⁵ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Two Bits", Christopher Kelty

"Two Bits", Christopher Kelty
      document manifest   ⁷⁶ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Free Culture", Lawrence Lessig

"Free Culture", Lawrence Lessig
      document manifest   ⁷⁷ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"CONTENT", Cory Doctorow

"CONTENT", Cory Doctorow
      document manifest   ⁷⁸ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Democratizing Innovation", by Eric von Hippel

"Democratizing Innovation", by Eric von Hippel
      document manifest   ⁷⁹ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Free as in Freedom: Richard Stallman's Crusade for Free Software", by Sam Williams

"Free as in Freedom: Richard Stallman's Crusade for Free Software", by Sam Williams
      document manifest   ⁸⁰ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Free For All: How Linux and the Free Software Movement Undercut the High Tech Titans", by Peter Wayner

"Free For All: How Linux and the Free Software Movement Undercut the High Tech Titans", by Peter Wayner
      document manifest   ⁸¹ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"The Cathedral and the Bazaar", by Eric S. Raymond

"The Cathedral and the Bazaar", by Eric S. Raymond
      document manifest   ⁸² 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Down and out in the Magic Kingdom", Cory Doctorow

"Down and out in the Magic Kingdom", Cory Doctorow
      document manifest   ⁸³ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Little Brother", Cory Doctorow

"Little Brother", Cory Doctorow
      document manifest   ⁸⁴ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"For the Win", Cory Doctorow

"For the Win", Cory Doctorow
      document manifest   ⁸⁵ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Accelerando", Charles Stross

"Accelerando", Charles Stross
      document manifest   ⁸⁶ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Tainaron", Leena Krohn

"Tainaron", Leena Krohn
      document manifest   ⁸⁷ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Sphinx or Robot", Leena Krohn

Sphinx or Robot by Leena Krohn

"Sphinx or Robot", Leena Krohn
      document manifest   ⁸⁸ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"War and Peace", Leo Tolstoy, PG Etext 2600

"War and Peace", Leo Tolstoy   ⁸⁹ 
      document manifest   ⁹⁰ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Don Quixote", Miguel de Cervantes [Saavedra], translated by John Ormsby, PG Etext 996

"Don Quixote", Miguel de Cervantes [Saavedra]
      document manifest   ⁹¹ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Gulliver's Travels", Jonathan Swift, transcribed from the 1892 George Bell and Sons edition by David Price, PG Etext 829

"Gulliver's Travels", Jonathan Swift
      document manifest   ⁹² 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Alice's Adventures in Wonderland", Lewis Carroll, PG Etext 11

"Alice's Adventures in Wonderland", Lewis Carroll
      document manifest   ⁹³ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Through The Looking-Glass", Lewis Carroll, PG Etext 12

"Through The Looking-Glass", Lewis Carroll
      document manifest   ⁹⁴ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Alice's Adventures in Wonderland" and "Through The Looking-Glass", Lewis Carroll, PG Etexts 11 and 12

"Alice's Adventures in Wonderland" and "Through The Looking-Glass", Lewis Carroll
      document manifest   ⁹⁵ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Gnu Public License 2", (GPL 2) Free Software Foundation

"Gnu Public License 2", (GPL 2) Free Software Foundation
      document manifest   ⁹⁶ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Gnu Public License v3 - Third discussion draft", (GPLv3) Free Software Foundation

"Gnu Public License 3 - Third discussion draft", (GPL v3 draft3) Free Software Foundation
      document manifest   ⁹⁷ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Debian Social Contract"

"Debian Social Contract"
      document manifest   ⁹⁸ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Debian Constitution v1.3", (simple/default markup)

"Debian Constitution v1.3"
      document manifest   ⁹⁹ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Debian Constitution v1.3", (markup adjusted for output to more closely match the original)

"Debian Constitution v1.3", (markup adjusted for output to more closely match the original)
      document manifest   ¹⁰⁰ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Debian Constitution v1.2", (simple/default markup)

"Debian Constitution v1.2 (more translations)"
      document manifest   ¹⁰¹ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"Debian Constitution v1.2", (markup adjusted for output to more closely match the original)

"Debian Constitution (more translations)", (markup adjusted for output to more closely match the original)
      document manifest   ¹⁰² 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"A Uniform Sales Terminology", Vikki Rogers and Albert Kritzer

"A Uniform Sales Terminology", Vikki Rogers and Albert Kritzer
      document manifest   ¹⁰³ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"The Autonomous Contract" 1997 - markup sample

"The Autonomous Contract" 1997 - markup sample
      document manifest   ¹⁰⁴ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"The Autonomous Contract Revisited" - markup sample

"The Autonomous Contract Revisited" - markup sample   ¹⁰⁵ 
      document manifest   ¹⁰⁶ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

"United Nations Convention on Contracts for the International Sale of Goods"

"United Nations Convention on Contracts for the International Sale of Goods"   ¹⁰⁷ 
      document manifest   ¹⁰⁸ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

/PECL/ the "Principles of European Contract Law"

"Principles of European Contract Law"
      document manifest   ¹⁰⁹ 
        html, segmented text
        html, scroll, document in one
        epub
        pdf, landscape
        pdf, portrait
        odf:odt, open document text
        xhtml scroll
        xml, sax
        xml, dom
        plain text utf-8
        concordance
        dcc, document content certificate (digests)
      markup source text
      markup source (zipped) pod

2.3 SQL - PostgreSQL, SQLite

A Sample search form is available at ‹http://search.sisudoc.org›

A few canned searches, showing object numbers. Search for:

English documents matching Linux OR Debian

GPL OR Stallman

invention OR innovation

copyright in English language documents

Note that the searches done in this form are case sensitive.

Expand those same searches, showing the matching text in each document:

Linux OR Debian

GPL OR Stallman

invention OR innovation in English language

copyright in English language documents

Note you may set results either for documents matched and object number locations within each matched document meeting the search criteria; or display the names of the documents matched along with the objects (paragraphs) that meet the search criteria.  ¹¹⁰ 

2.4 Lex Mercatoria as an example

There is quite a bit to peruse if you explore the site Lex Mercatoria:

‹http://www.lexmercatoria.org/›   ¹¹¹ 

or perhaps:

‹http://lexmercatoria.org/treaties.and.organisations/lm.chronological›   ¹¹² 

2.5 For good measure the markup for a document with lots of (simple) tables

SiSU is not optimised for table making, but does handle simple tables.

2.6 And a link to the output of a reported case

‹http://www.jus.uio.no/lm/england.fothergill.v.monarch.airlines.hl.1980/toc.html›

3. A Checklist of Output Features

This table gives an indication of the features that are available for various forms of output of SiSU.

sisu-2.0.0 on 2010-03-06

sisu-1.0.0 on 2009-10-28

sisu-0.36.6 on 2006-01-23

  Done
  * yes/done
  . partial
  - not available/appropriate
  Not Done
  T task todo
  t lesser task/todo
    not done

4. Introduction to SiSU Markup  ¹¹⁴ 

4.1 Summary

SiSU source documents are plaintext (UTF-8)  ¹¹⁵  files

All paragraphs are separated by an empty line.

Markup is comprised of:

Some interactive help on markup is available, by typing sisu and selecting markup or sisu --help markup

To check the markup in a file:

sisu --identify [filename].sst

For brief descriptive summary of markup history

sisu --query-history

or if for a particular version:

sisu --query-0.38

4.2 Markup Examples

4.2.1 Online

Online markup examples are available together with the respective outputs produced from ‹http://www.jus.uio.no/sisu/SiSU/examples.html› or from ‹http://www.jus.uio.no/sisu/sisu_examples/›

There is of course this document, which provides a cursory overview of sisu markup and the respective output produced: ‹http://www.jus.uio.no/sisu/sisu_markup/›

an alternative presentation of markup syntax: /usr/share/doc/sisu/on_markup.txt.gz

4.2.2 Installed

With SiSU installed sample skins may be found in: /usr/share/doc/sisu/markup-samples (or equivalent directory) and if sisu-markup-samples is installed also under: /usr/share/doc/sisu/markup-samples-non-free

5. Markup of Headers

Headers contain either: semantic meta-data about a document, which can be used by any output module of the program, or; processing instructions.

Note: the first line of a document may include information on the markup version used in the form of a comment. Comments are a percentage mark at the start of a paragraph (and as the first character in a line of text) followed by a space and the comment:

  % this would be a comment

5.1 Sample Header

This current document is loaded by a master document that has a header similar to this one:

  % SiSU master 2.0

  @title: SiSU
   :subtitle: Manual

  @creator: :author: Amissah, Ralph

  @rights: Copyright (C) Ralph Amissah 2007, part of SiSU documentation, License GPL 3

  @classify:
   :type: information
   :topic_register: SiSU:manual;electronic documents:SiSU:manual
   :subject: ebook, epublishing, electronic book, electronic publishing,
      electronic document, electronic citation, data structure,
       citation systems, search

  % used_by: manual

  @date:
   :published: 2008-05-22
   :created: 2002-08-28
   :issued: 2002-08-28
   :available: 2002-08-28
   :modified: 2010-03-03

  @make:
   :num_top: 1
   :breaks: new=C; break=1
   :skin: skin_sisu_manual
   :bold: /Gnu|Debian|Ruby|SiSU/
   :manpage: name=sisu - documents: markup, structuring, publishing in multiple standard formats, and search;
       synopsis=sisu [-abcDdeFhIiMmNnopqRrSsTtUuVvwXxYyZz0-9] [filename/wildcard ]
       . sisu [-Ddcv] [instruction]
       . sisu [-CcFLSVvW]
       . sisu --v2 [operations]
       . sisu --v3 [operations]

  @links:
   { SiSU Homepage }http://www.sisudoc.org/
   { SiSU Manual }http://www.sisudoc.org/sisu/sisu_manual/
   { Book Samples & Markup Examples }http://www.jus.uio.no/sisu/SiSU/examples.html
   { SiSU Download }http://www.jus.uio.no/sisu/SiSU/download.html
   { SiSU Changelog }http://www.jus.uio.no/sisu/SiSU/changelog.html
   { SiSU Git repo }http://git.sisudoc.org/?p=code/sisu.git;a=summary
   { SiSU List Archives }http://lists.sisudoc.org/pipermail/sisu/
   { SiSU @ Debian }http://packages.qa.debian.org/s/sisu.html
   { SiSU Project @ Debian }http://qa.debian.org/developer.php?login=sisu@lists.sisudoc.org
   { SiSU @ Wikipedia }http://en.wikipedia.org/wiki/SiSU

5.2 Available Headers

Header tags appear at the beginning of a document and provide meta information on the document (such as the Dublin Core), or information as to how the document as a whole is to be processed. All header instructions take the form @headername: or on the next line and indented by once space :subheadername: All Dublin Core meta tags are available

@indentifier: information or instructions

where the "identifier" is a tag recognised by the program, and the "information" or "instructions" belong to the tag/indentifier specified

Note: a header where used should only be used once; all headers apart from @title: are optional; the @structure: header is used to describe document structure, and can be useful to know.

This is a sample header

  % SiSU 2.0 [declared file-type identifier with markup version]

  @title: [title text] [this header is the only one that is mandatory]
    :subtitle: [subtitle if any]
    :language: English

  @creator:
   :author: [Lastname, First names]
   :illustrator: [Lastname, First names]
   :translator: [Lastname, First names]
   :prepared_by: [Lastname, First names]

  @date:
   :published: [year or yyyy-mm-dd]
   :created: [year or yyyy-mm-dd]
   :issued: [year or yyyy-mm-dd]
   :available: [year or yyyy-mm-dd]
   :modified: [year or yyyy-mm-dd]
   :valid: [year or yyyy-mm-dd]
   :added_to_site: [year or yyyy-mm-dd]
   :translated: [year or yyyy-mm-dd]

  @rights:
   :copyright: Copyright (C) [Year and Holder]
   :license: [Use License granted]
   :text: [Year and Holder]
   :translation: [Name, Year]
   :illustrations: [Name, Year]

  @classify:
   :topic_register: SiSU:markup sample:book;book:novel:fantasy
   :type:
   :subject:
   :description:
   :keywords:
   :abstract:
   :isbn: [ISBN]
   :loc: [Library of Congress classification]
   :dewey: [Dewey classification
   :pg: [Project Gutenberg text number]

  @links: { SiSU }http://www.sisudoc.org
    { FSF }http://www.fsf.org

  @make:
   :skin: skin_name [skins change default settings related to the appearance of documents generated]
   :num_top: 1
   :headings: [text to match for each level
      (e.g. PART; Chapter; Section; Article; or another: none; BOOK|FIRST|SECOND; none; CHAPTER;)
   :breaks: new=:C; break=1
   :promo: sisu, ruby, sisu_search_libre, open_society
   :bold: [regular expression of words/phrases to be made bold]
   :italics: [regular expression of words/phrases to italicise]

  @original:
   :language: [language]

  @notes:
   :comment:
   :prefix: [prefix is placed just after table of contents]

6. Markup of Substantive Text

6.1 Heading Levels

Heading levels are :A~ ,:B~ ,:C~ ,1~ ,2~ ,3~ ... :A - :C being part / section headings, followed by other heading levels, and 1 -6 being headings followed by substantive text or sub-headings. :A~ usually the title :A~? conditional level 1 heading (used where a stand-alone document may be imported into another)

:A~ [heading text] Top level heading [this usually has similar content to the title @title: ] NOTE: the heading levels described here are in 0.38 notation, see heading

:B~ [heading text] Second level heading [this is a heading level divider]

:C~ [heading text] Third level heading [this is a heading level divider]

1~ [heading text] Top level heading preceding substantive text of document or sub-heading 2, the heading level that would normally be marked 1. or 2. or 3. etc. in a document, and the level on which sisu by default would break html output into named segments, names are provided automatically if none are given (a number), otherwise takes the form 1~my_filename_for_this_segment

2~ [heading text] Second level heading preceding substantive text of document or sub-heading 3 , the heading level that would normally be marked 1.1 or 1.2 or 1.3 or 2.1 etc. in a document.

3~ [heading text] Third level heading preceding substantive text of document, that would normally be marked 1.1.1 or 1.1.2 or 1.2.1 or 2.1.1 etc. in a document

  1~filename level 1 heading,

  % the primary division such as Chapter that is followed by substantive text, and may be further subdivided (this is the level on which by default html segments are made)

6.2 Font Attributes

markup example:

  normal text,  *{emphasis}*, !{bold text}!, /{italics}/, _{underscore}_, "{citation}",
  ^{superscript}^, ,{subscript},, +{inserted text}+, -{strikethrough}-, #{monospace}#

  normal text

  *{emphasis}* [note: can be configured to be represented by bold, italics or underscore]

  !{bold text}!

  /{italics}/

  _{underscore}_

  "{citation}"

  ^{superscript}^

  ,{subscript},

  +{inserted text}+

  -{strikethrough}-

  #{monospace}#

resulting output:

normal text, emphasis, bold text, italics, underscore, citation, ^superscript, _subscript, inserted text, strikethrough, monospace

normal text

emphasis [note: can be configured to be represented by bold, italics or underscore]

bold text

italics

underscore

citation

^superscript

_subscript

inserted text

strikethrough

monospace

6.3 Indentation and bullets

markup example:

  ordinary paragraph

  _1 indent paragraph one step

  _2 indent paragraph two steps

  _9 indent paragraph nine steps

resulting output:

ordinary paragraph

indent paragraph one step

indent paragraph two steps

indent paragraph nine steps

markup example:

  _* bullet text

  _1* bullet text, first indent

  _2* bullet text, two step indent

resulting output:

Numbered List (not to be confused with headings/titles, (document structure))

markup example:

  # numbered list                numbered list 1., 2., 3, etc.

  _# numbered list numbered list indented a., b., c., d., etc.

6.4 Footnotes / Endnotes

Footnotes and endnotes are marked up at the location where they would be indicated within a text. They are automatically numbered. The output type determines whether footnotes or endnotes will be produced

markup example:

  ~{ a footnote or endnote }~

resulting output:

  ¹¹⁶ 

markup example:

  normal text~{ self contained endnote marker & endnote in one }~ continues

resulting output:

normal text  ¹¹⁷  continues

markup example:

  normal text ~{* unnumbered asterisk footnote/endnote, insert multiple asterisks if required }~ continues

  normal text ~{** another unnumbered asterisk footnote/endnote }~ continues

resulting output:

normal text   ^*  continues

normal text   ^**  continues

markup example:

  normal text ~[* editors notes, numbered asterisk footnote/endnote series ]~ continues

  normal text ~[+ editors notes, numbered asterisk footnote/endnote series ]~ continues

resulting output:

normal text   ^*1  continues

normal text   ⁺¹  continues

Alternative endnote pair notation for footnotes/endnotes:

  % note the endnote marker "~^"

  normal text~^ continues

  ^~ endnote text following the paragraph in which the marker occurs

the standard and pair notation cannot be mixed in the same document

6.5 Links

6.5.1 Naked URLs within text, dealing with urls

urls found within text are marked up automatically. A url within text is automatically hyperlinked to itself and by default decorated with angled braces, unless they are contained within a code block (in which case they are passed as normal text), or escaped by a preceding underscore (in which case the decoration is omitted).

markup example:

  normal text http://www.sisudoc.org/ continues

resulting output:

normal text ‹http://www.sisudoc.org/› continues

An escaped url without decoration

markup example:

  normal text _http://www.sisudoc.org/ continues

  deb http://www.jus.uio.no/sisu/archive unstable main non-free

resulting output:

normal text http://www.sisudoc.org/ continues

deb http://www.jus.uio.no/sisu/archive unstable main non-free

where a code block is used there is neither decoration nor hyperlinking, code blocks are discussed later in this document

resulting output:

  deb http://www.jus.uio.no/sisu/archive unstable main non-free
  deb-src http://www.jus.uio.no/sisu/archive unstable main non-free

6.5.2 Linking Text

To link text or an image to a url the markup is as follows

markup example:

  about { SiSU }http://url.org markup

resulting output:

about SiSU markup

A shortcut notation is available so the url link may also be provided automatically as a footnote

markup example:

  about {~^ SiSU }http://url.org markup

resulting output:

about SiSU   ¹¹⁸  markup

Internal document links to a tagged location, including an ocn

markup example:

  about { text links }#link_text

resulting output:

about text links

Shared document collection link

markup example:

  about { SiSU book markup examples }:SiSU/examples.html

resulting output:

about SiSU book markup examples

6.5.3 Linking Images

markup example:

  { tux.png 64x80 }image

  % various url linked images

  {tux.png 64x80 "a better way" }http://www.sisudoc.org/

  {GnuDebianLinuxRubyBetterWay.png 100x101 "Way Better - with Gnu/Linux, Debian and Ruby" }http://www.sisudoc.org/

  {~^ ruby_logo.png "Ruby" }http://www.ruby-lang.org/en/

resulting output:

Gnu/Linux - a better way

Way Better - with Gnu/Linux, Debian and Ruby

Ruby

linked url footnote shortcut

  {~^ [text to link] }http://url.org

  % maps to: { [text to link] }http://url.org ~{ http://url.org }~

  % which produces hyper-linked text within a document/paragraph, with an endnote providing the url for the text location used in the hyperlink

  text marker

note at a heading level the same is automatically achieved by providing names to headings 1, 2 and 3 i.e. 2~[name] and 3~[name] or in the case of auto-heading numbering, without further intervention.

6.6 Grouped Text

6.6.1 Tables

Tables may be prepared in two either of two forms

markup example:

  table{ c3; 40; 30; 30;

  This is a table
  this would become column two of row one
  column three of row one is here

  And here begins another row
  column two of row two
  column three of row two, and so on

  }table

resulting output:

a second form may be easier to work with in cases where there is not much information in each column

markup example:  ¹²⁰ 

  !_ Table 3.1: Contributors to Wikipedia, January 2001 - June 2005

  {table~h 24; 12; 12; 12; 12; 12; 12;}
                                  |Jan. 2001|Jan. 2002|Jan. 2003|Jan. 2004|July 2004|June 2006
  Contributors*                   |       10|      472|    2,188|    9,653|   25,011|   48,721
  Active contributors**           |        9|      212|      846|    3,228|    8,442|   16,945
  Very active contributors***     |        0|       31|      190|      692|    1,639|    3,016
  No. of English language articles|       25|   16,000|  101,000|  190,000|  320,000|  630,000
  No. of articles, all languages  |       25|   19,000|  138,000|  490,000|  862,000|1,600,000

  \* Contributed at least ten times; \** at least 5 times in last month; \*\** more than 100 times in last month.

resulting output:

Table 3.1: Contributors to Wikipedia, January 2001 - June 2005

* Contributed at least ten times; ** at least 5 times in last month; *** more than 100 times in last month.

6.6.2 Poem

basic markup:

  poem{

    Your poem here

  }poem

  Each verse in a poem is given an object number.

markup example:

  poem{

                      `Fury said to a
                     mouse, That he
                   met in the
                 house,
              "Let us
                both go to
                  law:  I will
                    prosecute
                      YOU.  --Come,
                         I'll take no
                          denial; We
                       must have a
                   trial:  For
                really this
             morning I've
            nothing
           to do."
             Said the
               mouse to the
                 cur, "Such
                   a trial,
                     dear Sir,
                           With
                       no jury
                    or judge,
                  would be
                wasting
               our
                breath."
                 "I'll be
                   judge, I'll
                     be jury,"
                           Said
                      cunning
                        old Fury:
                       "I'll
                        try the
                           whole
                            cause,
                               and
                          condemn
                         you
                        to
                         death."'

  }poem

resulting output:

                    `Fury said to a
                   mouse, That he
                 met in the
               house,
            "Let us
              both go to
                law:  I will
                  prosecute
                    YOU.  --Come,
                       I'll take no
                        denial; We
                     must have a
                 trial:  For
              really this
           morning I've
          nothing
         to do."
           Said the
             mouse to the
               cur, "Such
                 a trial,
                   dear Sir,
                         With
                     no jury
                  or judge,
                would be
              wasting
             our
              breath."
               "I'll be
                 judge, I'll
                   be jury,"
                         Said
                    cunning
                      old Fury:
                     "I'll
                      try the
                         whole
                          cause,
                             and
                        condemn
                       you
                      to
                       death."'

6.6.3 Group

basic markup:

  group{

    Your grouped text here

  }group

  A group is treated as an object and given a single object number.

markup example:

  group{

                      `Fury said to a
                     mouse, That he
                   met in the
                 house,
              "Let us
                both go to
                  law:  I will
                    prosecute
                      YOU.  --Come,
                         I'll take no
                          denial; We
                       must have a
                   trial:  For
                really this
             morning I've
            nothing
           to do."
             Said the
               mouse to the
                 cur, "Such
                   a trial,
                     dear Sir,
                           With
                       no jury
                    or judge,
                  would be
                wasting
               our
                breath."
                 "I'll be
                   judge, I'll
                     be jury,"
                           Said
                      cunning
                        old Fury:
                       "I'll
                        try the
                           whole
                            cause,
                               and
                          condemn
                         you
                        to
                         death."'

  }group

resulting output:

                    `Fury said to a
                   mouse, That he
                 met in the
               house,
            "Let us
              both go to
                law:  I will
                  prosecute
                    YOU.  --Come,
                       I'll take no
                        denial; We
                     must have a
                 trial:  For
              really this
           morning I've
          nothing
         to do."
           Said the
             mouse to the
               cur, "Such
                 a trial,
                   dear Sir,
                         With
                     no jury
                  or judge,
                would be
              wasting
             our
              breath."
               "I'll be
                 judge, I'll
                   be jury,"
                         Said
                    cunning
                      old Fury:
                     "I'll
                      try the
                         whole
                          cause,
                             and
                        condemn
                       you
                      to
                       death."'

6.6.4 Code

Code tags code{ ... }code (used as with other group tags described above) are used to escape regular sisu markup, and have been used extensively within this document to provide examples of SiSU markup. You cannot however use code tags to escape code tags. They are however used in the same way as group or poem tags.

A code-block is treated as an object and given a single object number. [an option to number each line of code may be considered at some later time]

use of code tags instead of poem compared, resulting output:

                      `Fury said to a
                     mouse, That he
                   met in the
                 house,
              "Let us
                both go to
                  law:  I will
                    prosecute
                      YOU.  --Come,
                         I'll take no
                          denial; We
                       must have a
                   trial:  For
                really this
             morning I've
            nothing
           to do."
             Said the
               mouse to the
                 cur, "Such
                   a trial,
                     dear Sir,
                           With
                       no jury
                    or judge,
                  would be
                wasting
               our
                breath."
                 "I'll be
                   judge, I'll
                     be jury,"
                           Said
                      cunning
                        old Fury:
                       "I'll
                        try the
                           whole
                            cause,
                               and
                          condemn
                         you
                        to
                         death."'

From SiSU 2.7.7 on you can number codeblocks by placing a hash after the opening code tag code{# as demonstrated here:

1  ┆                      `Fury said to a
2  ┆                     mouse, That he
3  ┆                   met in the
4  ┆                 house,
5  ┆              "Let us
6  ┆                both go to
7  ┆                  law:  I will
8  ┆                    prosecute
9  ┆                      YOU.  --Come,
10 ┆                         I'll take no
11 ┆                          denial; We
12 ┆                       must have a
13 ┆                   trial:  For
14 ┆                really this
15 ┆             morning I've
16 ┆            nothing
17 ┆           to do."
18 ┆             Said the
19 ┆               mouse to the
20 ┆                 cur, "Such
21 ┆                   a trial,
22 ┆                     dear Sir,
23 ┆                           With
24 ┆                       no jury
25 ┆                    or judge,
26 ┆                  would be
27 ┆                wasting
28 ┆               our
29 ┆                breath."
30 ┆                 "I'll be
31 ┆                   judge, I'll
32 ┆                     be jury,"
33 ┆                           Said
34 ┆                      cunning
35 ┆                        old Fury:
36 ┆                       "I'll
37 ┆                        try the
38 ┆                           whole
39 ┆                            cause,
40 ┆                               and
41 ┆                          condemn
42 ┆                         you
43 ┆                        to
44 ┆                         death."'

6.7 Book index

To make an index append to paragraph the book index term relates to it, using an equal sign and curly braces.

Currently two levels are provided, a main term and if needed a sub-term. Sub-terms are separated from the main term by a colon.

    Paragraph containing main term and sub-term.
    ={Main term:sub-term}

The index syntax starts on a new line, but there should not be an empty line between paragraph and index markup.

The structure of the resulting index would be:

    Main term, 1
      sub-term, 1

Several terms may relate to a paragraph, they are separated by a semicolon. If the term refers to more than one paragraph, indicate the number of paragraphs.

    Paragraph containing main term, second term and sub-term.
    ={first term; second term: sub-term}

The structure of the resulting index would be:

    First term, 1,
    Second term, 1,
      sub-term, 1

If multiple sub-terms appear under one paragraph, they are separated under the main term heading from each other by a pipe symbol.

    Paragraph containing main term, second term and sub-term.
    ={Main term:sub-term+1|second sub-term

    A paragraph that continues discussion of the first sub-term

The plus one in the example provided indicates the first sub-term spans one additional paragraph. The logical structure of the resulting index would be:

    Main term, 1,
      sub-term, 1-3,
      second sub-term, 1,

7. Composite documents markup

It is possible to build a document by creating a master document that requires other documents. The documents required may be complete documents that could be generated independently, or they could be markup snippets, prepared so as to be easily available to be placed within another text. If the calling document is a master document (built from other documents), it should be named with the suffix .ssm Within this document you would provide information on the other documents that should be included within the text. These may be other documents that would be processed in a regular way, or markup bits prepared only for inclusion within a master document .sst regular markup file, or .ssi (insert/information) A secondary file of the composite document is built prior to processing with the same prefix and the suffix ._sst

basic markup for importing a document into a master document

  << filename1.sst

  << filename2.ssi

The form described above should be relied on. Within the Vim editor it results in the text thus linked becoming hyperlinked to the document it is calling in which is convenient for editing. Alternative markup for importation of documents under consideration, and occasionally supported have been.

  << filename.ssi

  <<{filename.ssi}

  % using textlink alternatives

  << |filename.ssi|@|^|

Markup Syntax History

8. Notes related to Files-types and Markup Syntax

2.0 introduced new headers and is therefore incompatible with 1.0 though otherwise the same with the addition of a couple of tags (i.e. a superset)

0.38 is substantially current for version 1.0

depreciated 0.16 supported, though file names were changed at 0.37

provides a short history of changes to SiSU markup

SiSU 2.0 (2010-03-06:09/6) same as 1.0, apart from the changing of headers and the addition of a monospace tag related headers now grouped, e.g.

  @title:
   :subtitle:

  @creator:
   :author:
   :translator:
   :illustrator:

  @rights:
   :text:
   :illustrations:

see document markup samples, and sisu --help headers

the monospace tag takes the form of a hash '#'

  #{ this enclosed text would be monospaced }#

1.0 (2009-12-19:50/6) same as 0.69

0.69 (2008-09-16:37/2) (same as 1.0) and as previous (0.57) with the addition of book index tags

  /^={.+?}$/

e.g. appended to a paragraph, on a new-line (without a blank line in between) logical structure produced assuming this is the first text "object"

   ={GNU/Linux community distribution:Debian+2|Fedora|Gentoo;Free Software Foundation+5}

  Free Software Foundation, 1-6
  GNU/Linux community distribution, 1
      Debian, 1-3
      Fedora, 1
      Gentoo,

0.66 (2008-02-24:07/7) same as previous, adds semantic tags, [experimental and not-used]

  /[:;]{.+?}[:;][a-z+]/

0.57 (2007w34/4) SiSU 0.57 is the same as 0.42 with the introduction of some a shortcut to use the headers @title and @creator in the first heading [expanded using the contents of the headers @title: and @author:]

  :A~ @title by @author

0.52 (2007w14/6) declared document type identifier at start of text/document:

SiSU 0.52

or, backward compatible using the comment marker:

% SiSU 0.38

variations include 'SiSU (text|master|insert) [version]' and 'sisu-[version]'

0.51 (2007w13/6) skins changed (simplified), markup unchanged

0.42 (2006w27/4) * (asterisk) type endnotes, used e.g. in relation to author

SiSU 0.42 is the same as 0.38 with the introduction of some additional endnote types,

Introduces some variations on endnotes, in particular the use of the asterisk

  ~{* for example for describing an author }~ and ~{** for describing a second author }~

* for example for describing an author

** for describing a second author

and

  ~[* my note ]~ or ~[+ another note ]~

which numerically increments an asterisk and plus respectively

*1 my note +1 another note

0.38 (2006w15/7) introduced new/alternative notation for headers, e.g. @title: (instead of 0~title), and accompanying document structure markup, :A,:B,:C,1,2,3 (maps to previous 1,2,3,4,5,6)

SiSU 0.38 introduced alternative experimental header and heading/structure markers,

  @headername: and headers :A~ :B~ :C~ 1~ 2~ 3~

as the equivalent of:

  0~headername and headers 1~ 2~ 3~ 4~ 5~ 6~

The internal document markup of SiSU 0.16 remains valid and standard Though note that SiSU 0.37 introduced a new file naming convention

SiSU has in effect two sets of levels to be considered, using 0.38 notation A-C headings/levels, pre-ordinary paragraphs /pre-substantive text, and 1-3 headings/levels, levels which are followed by ordinary text. This may be conceptualised as levels A,B,C, 1,2,3, and using such letter number notation, in effect: A must exist, optional B and C may follow in sequence (not strict) 1 must exist, optional 2 and 3 may follow in sequence i.e. there are two independent heading level sequences A,B,C and 1,2,3 (using the 0.16 standard notation 1,2,3 and 4,5,6) on the positive side: the 0.38 A,B,C,1,2,3 alternative makes explicit an aspect of structuring documents in SiSU that is not otherwise obvious to the newcomer (though it appears more complicated, is more in your face and likely to be understood fairly quickly); the substantive text follows levels 1,2,3 and it is 'nice' to do most work in those levels

0.37 (2006w09/7) introduced new file naming convention, .sst (text), .ssm (master), .ssi (insert), markup syntax unchanged

SiSU 0.37 introduced new file naming convention, using the file extensions .sst .ssm and .ssi to replace .s1 .s2 .s3 .r1 .r2 .r3 and .si

this is captured by the following file 'rename' instruction:

  rename 's/\.s[123]$/\.sst/' *.s{1,2,3}
  rename 's/\.r[123]$/\.ssm/' *.r{1,2,3}
  rename 's/\.si$/\.ssi/' *.si

The internal document markup remains unchanged, from SiSU 0.16

0.35 (2005w52/3) sisupod, zipped content file introduced

0.23 (2005w36/2) utf-8 for markup file

0.22 (2005w35/3) image dimensions may be omitted if rmagick is available to be relied upon

0.20.4 (2005w33/4) header 0~links

0.16 (2005w25/2) substantial changes introduced to make markup cleaner, header 0~title type, and headings [1-6]~ introduced, also percentage sign (%) at start of a text line as comment marker

SiSU 0.16 (0.15 development branch) introduced the use of

the header 0~ and headings/structure 1~ 2~ 3~ 4~ 5~ 6~

in place of the 0.1 header, heading/structure notation

SiSU 0.1 headers and headings structure represented by header 0{~ and headings/structure 1{ 2{ 3{ 4{~ 5{ 6{

9. Commands Summary

9.1 Description

SiSU SiSU is a document publishing system, that from a simple single marked-up document, produces multiple of output formats including: plaintext, html, xhtml, XML, epub, odt (odf text), LaTeX, pdf, info, and SQL (PostgreSQL and SQLite), which share numbered text objects ("object citation numbering") and the same document structure information. For more see: ‹http://www.jus.uio.no/sisu›

9.2 Document Processing Command Flags

-a [filename/wildcard]
produces plaintext with Unix linefeeds and without markup, (object numbers are omitted), has footnotes at end of each paragraph that contains them [ -A for equivalent dos (linefeed) output file] [see -e for endnotes]. (Options include: --endnotes for endnotes --footnotes for footnotes at the end of each paragraph --unix for unix linefeed (default) --msdos for msdos linefeed)

-b [filename/wildcard]
see --xhtml

--color-toggle [filename/wildcard]
screen toggle ansi screen colour on or off depending on default set (unless -c flag is used: if sisurc colour default is set to 'true', output to screen will be with colour, if sisurc colour default is set to 'false' or is undefined screen output will be without colour). Alias -c

--concordance [filename/wildcard]
produces concordance (wordmap) a rudimentary index of all the words in a document. (Concordance files are not generated for documents of over 260,000 words unless this limit is increased in the file sisurc.yml). Alias -w

-C [--init-site]
configure/initialise shared output directory files initialize shared output directory (config files such as css and dtd files are not updated if they already exist unless modifier is used). -C --init-site configure/initialise site more extensive than -C on its own, shared output directory files/force update, existing shared output config files such as css and dtd files are updated if this modifier is used.

-CC
configure/initialise shared output directory files initialize shared output directory (config files such as css and dtd files are not updated if they already exist unless modifier is used). The equivalent of: -C --init-site configure/initialise site, more extensive than -C on its own, shared output directory files/force update, existing shared output config files such as css and dtd files are updated if -CC is used.

-c [filename/wildcard]
see --color-toggle

--dal [filename/wildcard/url]
assumed for most other flags, creates new intermediate files for processing (document abstraction) that is used in all subsequent processing of other output. This step is assumed for most processing flags. To skip it see -n. Alias -m

--delete [filename/wildcard]
see --zap

-D [instruction] [filename]
see --pg

-d [--db-[database type (sqlite|pg)]] --[instruction] [filename]
see --sqlite

--epub [filename/wildcard]
produces an epub document, [sisu version 2 only] (filename.epub). Alias -e

-e [filename/wildcard]
see --epub

-F [--webserv=webrick]
see --sample-search-form

--git [filename/wildcard]
produces or updates markup source file structure in a git repo (experimental and subject to change). Alias -g

-g [filename/wildcard]
see --git

--harvest *.ss[tm]
makes two lists of sisu output based on the sisu markup documents in a directory: list of author and authors works (year and titles), and; list by topic with titles and author. Makes use of header metadata fields (author, title, date, topic_register). Can be used with maintenance (-M) and remote placement (-R) flags.

--help [topic]
provides help on the selected topic, where topics (keywords) include: list, (com)mands, short(cuts), (mod)ifiers, (env)ironment, markup, syntax, headers, headings, endnotes, tables, example, customise, skin, (dir)ectories, path, (lang)uage, db, install, setup, (conf)igure, convert, termsheet, search, sql, features, license

--html [filename/wildcard]
produces html output, segmented text with table of contents (toc.html and index.html) and the document in a single file (scroll.html). Alias -h

-h [filename/wildcard]
see --html

-I [filename/wildcard]
see --texinfo

-i [filename/wildcard]
see --manpage

-L
prints license information.

--machine [filename/wildcard/url]
see --dal (document abstraction level/layer)

--maintenance [filename/wildcard/url]
maintenance mode files created for processing preserved and their locations indicated. (also see -V). Alias -M

--manpage [filename/wildcard]
produces man page of file, not suitable for all outputs. Alias -i

-M [filename/wildcard/url]
see --maintenance

-m [filename/wildcard/url]
see --dal (document abstraction level/layer)

--no-ocn
[with --html --pdf or --epub] switches off object citation numbering. Produce output without identifying numbers in margins of html or LaTeX/pdf output.

-N [filename/wildcard/url]
document digest or document content certificate ( DCC ) as md5 digest tree of the document: the digest for the document, and digests for each object contained within the document (together with information on software versions that produced it) (digest.txt). -NV for verbose digest output to screen.

-n [filename/wildcard/url]
skip the creation of intermediate processing files (document abstraction) if they already exist, this skips the equivalent of -m which is otherwise assumed by most processing flags.

--odf [filename/wildcard/url]
see --odt

--odt [filename/wildcard/url]
output basic document in opendocument file format (opendocument.odt). Alias -o

-o [filename/wildcard/url]
see --odt

--pdf [filename/wildcard]
produces LaTeX pdf (portrait.pdf & landscape.pdf). Default paper size is set in config file, or document header, or provided with additional command line parameter, e.g. --papersize-a4 preset sizes include: 'A4', U.S. 'letter' and 'legal' and book sizes 'A5' and 'B5' (system defaults to A4). Alias -p

--pg [instruction] [filename]
database postgresql ( --pgsql may be used instead) possible instructions, include: --createdb; --create; --dropall; --import [filename]; --update [filename]; --remove [filename]; see database section below. Alias -D

--po [language_directory/filename language_directory]
see --po4a

--po4a [language_directory/filename language_directory]
produces .pot and po files for the file in the languages specified by the language directory. SiSU markup is placed in subdirectories named with the language code, e.g. en/ fr/ es/. The sisu config file must set the output directory structure to multilingual. v3, experimental

-P [language_directory/filename language_directory]
see --po4a

-p [filename/wildcard]
see --pdf

--quiet [filename/wildcard]
quiet less output to screen.

-q [filename/wildcard]
see --quiet

--rsync [filename/wildcard]
copies sisu output files to remote host using rsync. This requires that sisurc.yml has been provided with information on hostname and username, and that you have your "keys" and ssh agent in place. Note the behavior of rsync different if -R is used with other flags from if used alone. Alone the rsync --delete parameter is sent, useful for cleaning the remote directory (when -R is used together with other flags, it is not). Also see --scp. Alias -R

-R [filename/wildcard]
see --rsync

-r [filename/wildcard]
see --scp

--sample-search-form [--webserv=webrick]
generate examples of (naive) cgi search form for sqlite and pgsql depends on your already having used sisu to populate an sqlite and/or pgsql database, (the sqlite version scans the output directories for existing sisu_sqlite databases, so it is first necessary to create them, before generating the search form) see -d -D and the database section below. If the optional parameter --webserv=webrick is passed, the cgi examples created will be set up to use the default port set for use by the webrick server, (otherwise the port is left blank and the system setting used, usually 80). The samples are dumped in the present work directory which must be writable, (with screen instructions given that they be copied to the cgi-bin directory). -Fv (in addition to the above) provides some information on setting up hyperestraier for sisu. Alias -F

--scp [filename/wildcard]
copies sisu output files to remote host using scp. This requires that sisurc.yml has been provided with information on hostname and username, and that you have your "keys" and ssh agent in place. Also see --rsync. Alias -r

--sqlite --[instruction] [filename]
database type default set to sqlite, (for which --sqlite may be used instead) or to specify another database --db-[pgsql, sqlite] (however see -D) possible instructions include: --createdb; --create; --dropall; --import [filename]; --update [filename]; --remove [filename]; see database section below. Alias -d

--sisupod
produces a sisupod a zipped sisu directory of markup files including sisu markup source files and the directories local configuration file, images and skins. Note: this only includes the configuration files or skins contained in ./_sisu not those in ~/.sisu -S [filename/wildcard] option. Note: (this option is tested only with zsh). Alias -S

--sisupod [filename/wildcard]
produces a zipped file of the prepared document specified along with associated images, by default named sisupod.zip they may alternatively be named with the filename extension .ssp This provides a quick way of gathering the relevant parts of a sisu document which can then for example be emailed. A sisupod includes sisu markup source file, (along with associated documents if a master file, or available in multilingual versions), together with related images and skin. SiSU commands can be run directly against a sisupod contained in a local directory, or provided as a url on a remote site. As there is a security issue with skins provided by other users, they are not applied unless the flag --trust or --trusted is added to the command instruction, it is recommended that file that are not your own are treated as untrusted. The directory structure of the unzipped file is understood by sisu, and sisu commands can be run within it. Note: if you wish to send multiple files, it quickly becomes more space efficient to zip the sisu markup directory, rather than the individual files for sending). See the -S option without [filename/wildcard]. Alias -S

--source [filename/wildcard]
copies sisu markup file to output directory. Alias -s

-S
see --sisupod

-S [filename/wildcard]
see --sisupod

-s [filename/wildcard]
see --source

--texinfo [filename/wildcard]
produces texinfo and info file, (view with pinfo). Alias -I

--txt [filename/wildcard]
produces plaintext with Unix linefeeds and without markup, (object numbers are omitted), has footnotes at end of each paragraph that contains them [ -A for equivalent dos (linefeed) output file] [see -e for endnotes]. (Options include: --endnotes for endnotes --footnotes for footnotes at the end of each paragraph --unix for unix linefeed (default) --msdos for msdos linefeed). Alias -t

-T [filename/wildcard (*.termsheet.rb)]
standard form document builder, preprocessing feature

-t [filename/wildcard]
see --txt

--urls [filename/wildcard]
prints url output list/map for the available processing flags options and resulting files that could be requested, (can be used to get a list of processing options in relation to a file, together with information on the output that would be produced), -u provides url output mapping for those flags requested for processing. The default assumes sisu_webrick is running and provides webrick url mappings where appropriate, but these can be switched to file system paths in sisurc.yml. Alias -U

-U [filename/wildcard]
see --urls

-u [filename/wildcard]
provides url mapping of output files for the flags requested for processing, also see -U

--v2 [filename/wildcard]
invokes the sisu v2 document parser/generator. This is the default and is normally omitted.

--v3 [filename/wildcard]
invokes the sisu v3 document parser/generator. Currently under development and incomplete, v3 requires >= ruby1.9.2p180. You may run sisu3 instead.

--verbose [filename/wildcard]
provides verbose output of what is being generated, where output is placed (and error messages if any), as with -u flag provides a url mapping of files created for each of the processing flag requests. Alias -v

-V
on its own, provides SiSU version and environment information (sisu --help env)

-V [filename/wildcard]
even more verbose than the -v flag.

-v
on its own, provides SiSU version information

-v [filename/wildcard]
see --verbose

--webrick
starts ruby's webrick webserver points at sisu output directories, the default port is set to 8081 and can be changed in the resource configuration files. [tip: the webrick server requires link suffixes, so html output should be created using the -h option rather than -H ; also, note -F webrick ]. Alias -W

-W
see --webrick

--wordmap [filename/wildcard]
see --concordance

-w [filename/wildcard]
see --concordance

--xhtml [filename/wildcard]
produces xhtml/XML output for browser viewing (sax parsing). Alias -b

--xml-dom [filename/wildcard]
produces XML output with deep document structure, in the nature of dom. Alias -X

--xml-sax [filename/wildcard]
produces XML output shallow structure (sax parsing). Alias -x

-X [filename/wildcard]
see --xml-dom

-x [filename/wildcard]
see --xml-sax

-Y [filename/wildcard]
produces a short sitemap entry for the document, based on html output and the sisu_manifest. --sitemaps generates/updates the sitemap index of existing sitemaps. (Experimental, [g,y,m announcement this week])

-y [filename/wildcard]
produces an html summary of output generated (hyperlinked to content) and document specific metadata (sisu_manifest.html). This step is assumed for most processing flags.

--zap [filename/wildcard]
Zap, if used with other processing flags deletes output files of the type about to be processed, prior to processing. If -Z is used as the lone processing related flag (or in conjunction with a combination of -[mMvVq]), will remove the related document output directory. Alias -Z

-Z [filename/wildcard]
see --zap

10. command line modifiers

--no-ocn
[with --html --pdf or --epub] switches off object citation numbering. Produce output without identifying numbers in margins of html or LaTeX/pdf output.

--no-annotate
strips output text of editor endnotes  ^*2  denoted by asterisk or dagger/plus sign

--no-asterisk
strips output text of editor endnotes  ^*3  denoted by asterisk sign

--no-dagger
strips output text of editor endnotes  ⁺²  denoted by dagger/plus sign

11. database commands

dbi - database interface

-D or --pgsql set for postgresql -d or --sqlite default set for sqlite -d is modifiable with --db=[database type (pgsql or sqlite)]

--pg -v --createall
initial step, creates required relations (tables, indexes) in existing postgresql database (a database should be created manually and given the same name as working directory, as requested) (rb.dbi) [ -dv --createall sqlite equivalent] it may be necessary to run sisu -Dv --createdb initially NOTE: at the present time for postgresql it may be necessary to manually create the database. The command would be 'createdb [database name]' where database name would be SiSU_[present working directory name (without path)]. Please use only alphanumerics and underscores.

--pg -v --import
[filename/wildcard] imports data specified to postgresql db (rb.dbi) [ -dv --import sqlite equivalent]

--pg -v --update
[filename/wildcard] updates/imports specified data to postgresql db (rb.dbi) [ -dv --update sqlite equivalent]

--pg --remove
[filename/wildcard] removes specified data to postgresql db (rb.dbi) [ -d --remove sqlite equivalent]

--pg --dropall
kills data" and drops (postgresql or sqlite) db, tables & indexes [ -d --dropall sqlite equivalent]

The -v is for verbose output.

12. Shortcuts, Shorthand for multiple flags

--update [filename/wildcard]
Checks existing file output and runs the flags required to update this output. This means that if only html and pdf output was requested on previous runs, only the -hp files will be applied, and only these will be generated this time, together with the summary. This can be very convenient, if you offer different outputs of different files, and just want to do the same again.

-0 to -5 [filename or wildcard]
Default shorthand mappings (note that the defaults can be changed/configured in the sisurc.yml file):

-0
-mNhwpAobxXyYv [this is the default action run when no options are give, i.e. on 'sisu [filename]']

-1
-mhewpy

-2
-mhewpaoy

-3
-mhewpAobxXyY

-4
-mhewpAobxXDyY --import

-5
-mhewpAobxXDyY --update

add -v for verbose mode and -c for color, e.g. sisu -2vc [filename or wildcard]

consider -u for appended url info or -v for verbose output

12.1 Command Line with Flags - Batch Processing

In the data directory run sisu -mh filename or wildcard eg. "sisu -h cisg.sst" or "sisu -h *.{sst,ssm}" to produce html version of all documents.

Running sisu (alone without any flags, filenames or wildcards) brings up the interactive help, as does any sisu command that is not recognised. Enter to escape.

Technical Information

13. Technical notes

This section has been much reduced in content since the release of SiSU which it predated. It provides links to some relevant information.

13.1 See abandoned U.S. Provisional Patent Application

The description provided in the abandoned U.S. Provisional Patent Application may be of interest as it provides greater detail and by an large supersedes the description given here ‹http://www.jus.uio.no/sisu/sisu_provisional_patent_application_200408› and accompanying diagrams ‹http://www.jus.uio.no/sisu/diagram/sisu_provisional_patent_application_diagram_200408.pdf› and reasons for abandoning ‹http://www.jus.uio.no/sisu/SiSU/2005.html#ppa›

Of particular interest is the ease of streaming documents to a relational database, at an object (roughly paragraph) level and the potential for increased precision in the presentation of matches that results thereby. The ability to serialise html, latex, xml, sql, (whatever) is also inherent in / incidental to the design.

14. Diagram / Chart

This is the short form of an old summary based on design decisions of 2002. It predates the release of SiSU by a number of years, and should possibly be removed.

14.1 The Chart

A rough chart of the SiSU program structure can be found here:

‹http://www.jus.uio.no/sisu/diagram/sisu.chart.pdf›

What follows is a brief description of the chart's components, based on the numbers and letters used in the chart.

14.2 I/O

A Input text ascii with minimalistic human markup requirements

B Machine intermediate processing output, used by all other modules - there for the time being is a selection: human readable; a Ruby marshal dump of the same, and; a YAML file   ¹²¹  Once the intermediate stage is created, if no changes to input (i.e. A) are made, it is possible to start with B as input for program (i.e. to skip stage A and processing required to get to stage B). This might be of interest if document appearance is modified but not content. Abstract document structure is "created" here, with the pre-processing of the likes of tables, numbering (headings, paragraphs etc.) and endnotes, to ensure that all subsequent processing is based on the same integral document structures.

C Various final publication outputs that all share a common citation numbering system
html - there are possibilities for output based on tables or output based on css
pdf - landscape and portrait currently set to A4 paper size
XML with a flat structure,sax, and with a deeper (embedded) structure, dom
sql - data in sql database retaining document structure, this is in some ways similar to B output, as is likely to be further processed for presentation.

14.3 The Program

1 data feed controller for other program components

2 Creation of intermediate stage B, which contains information related to document structure used by all subsequent data output modules.

3 Parameter extraction. Program takes data related to the document being processed.

4 Relates primarily to appearance/ design, how the site or document should look:

4a Initialised variables used for "typesetting". eg. Margin widths etc. called by program. This can be done in 3 stages, there are i. the default program-wide settings, ii. possibility of setting alternative site-wide settings, iii. possibility of providing settings for an individual document

4b Template, includes formatting classes eg. for appearance of html (whether table based or css) or for pdf output. (For examples of templates at work, see examples provided earlier of html output in css and tables versions, and of pdf landscape and portrait outputs that result from templates that provide different the LaTeX output for the resulting pdfs.

5 Here we have the logic engines that call process B the intermediate machine generated data and call upon the relevant templates to produce the different presentations of the document.

5a html module - to construct html documents

5b LaTeX module - to construct LaTeX, which is then fed to pdflatex to produce pdf files

5c SQL module - to import data into PostgreSQL database retaining document structure detail and other detail common to the other output formats. This keeps all information regarding document structure in four relational database tables, one containing semantic and other headers, a second substantive texts, a third endnotes, a fourth pre-formatted texts. (the flexibility exists to carry this further)

14.4 Software utilised

14.4.1 SiSU

SiSU is written in Ruby and assumes Linux OS (development has been on Debian/Gnu/Linux)

14.4.2 SiSU Modules

SiSU generates

html output

LaTeX output, then uses LaTeX (and /pdflatex/ LaTeX to pdf) for pdf output

lout output lout, then uses Lout to produce postscript (and postscript to pdf conversion), [not currently maintained]

sql output (database feed) eg PostgreSQL, making use of Ruby dbi or pgsql modules to be used by PostgerSQL, or sqlite, making use of Ruby dbi or sqlite modules to be used by sqlite

Not required but taken advantage of if available:

tidy (XML, xhtml well formed check)

trang (relaxng, rnc to dtd conversion)

there are other modules ... see this document.

15. SiSU development environment and technologies of interest, including data formats

SiSU started as a way to make html manageable, together with the core concept of making text citable through the use of object character numbering. LaTeX/pdf provided a way of making near print quality output, and demonstrating how conveniently the concept worked across different output formats. Relational database storage using the same concept underscored this and the concept makes database search results relevant, to locating results quickly in all output formats that use object character numbers.

There are a number of data formats and technologies that are of particular interest to SiSU, and to keep an eye on more generally. These links are kept here for convenience. Note that whilst all the technologies mentioned are of interest in the context of SiSU, not all of them are supported by SiSU.

15.1 Development environment, Debian