44. FAQ - Frequently Asked/Answered Questions

44.1 Why are urls produced with the -v (and -u) flag that point to a web server on port 8081 ?

Try the following rune:

This should start the ruby webserver. It should be done after having produced some output as it scans the output directory for what to serve.

44.2 I cannot find my output, where is it?

The following should provide help on output paths:

44.3 I do not get any pdf output, why?

SiSU produces LaTeX and pdflatex is run against that to generate pdf files.

If you use Debian the following will install the required dependencies

the following packages are required: tetex-bin, tetex-extra, latex-ucs

44.4 Where is the latex (or some other interim) output?

Try adding -M (for maintenance) to your command flags, e.g.:

this should result in the interim processing output being retained, and information being provided on where to find it.

should also provide some relevant information as to where it is placed.

44.5 Why isn't SiSU markup XML

I worked with text and (though I find XML immensely valuable) disliked noise ... better to sidestep the question and say:

SiSU currently "understands" three XML input representations - or more accurately, converts from three forms of XML to native SiSU markup for processing. The three types correspond to SAX (structure described), DOM (structure embedded, whole document must be read before structure is correctly discernable) and node based (a tree) forms of XML document structure representation. Problem is I use them very seldom and check that all is as it should be with them seldom, so I would not be surprised if something breaks there, but as far as I know they are working. I will check and add an XML markup help page before the next release. There already is a bit of information in the man page under the title SiSU VERSION CONVERSION

sisu --to-sax [filename/wildcard]

sisu --to-dom [filename/wildcard]

sisu --to-node [filename/wildcard]

The XML should be well formed... must check, but lacks sensible headers. Suggestions welcome as to what to make of them. [For the present time I am satisfied that I can convert (both ways) between 3 forms of XML representation and SiSU markup].

sisu --from-xml2sst [filename/wildcard]

44.6 LaTeX claims to be a document preparation system for high-quality typesetting. Can the same be said about SiSU?

SiSU is not really about type-setting.

LaTeX is the ultimate computer instruction type-setting language for paper based publication.

LaTeX is able to control just about everything that happens on page and pixel, position letters kerning, space variation between characters, words, paragraphs etc. formula.

SiSU is not really about type-setting at all. It is about a lightweight markup instruction that provides enough information for an abstraction of the documents structure and objects, from which different forms of representation of the document can be generated.

SiSU with very little markup instruction is able to produce relatively high quality pdf by virtue of being able to generate usable default LaTeX; it produces "quality" html by generating the html directly; likewise it populates an SQL database in a useful way with the document in object sized chunks and its meta-data. But SiSU works on an abstraction of the document's structure and content and custom builds suitable uniform output. The html for browser viewing and pdf for paper viewing/publishing are rather different things with different needs for layout - as indeed is what is needed to store information in a database in searchable objects.

The pdfs or html produced for example by open office based on open document format and other office/word processor suits usually attempt to have similar looking outputs - your document rendered in html looks much the same, or in pdf... sisu is less this way, it seeks to have a starting point with as little information about appearance as possible, and to come up with the best possible appearance for each output that can be derived based on this minimal information.

Where there are large document sets, it provides consistency in appearance in each output format for the documents.

The excuse for going this way is, it is a waste of time to think much about appearance when working on substantive content, it is the substantive content that is relevant, not the way it looks beyond the basic informational tags - and yet you want to be able to take advantage of as many useful different ways of representing documents as are available, and for various types of output to to be/look as good as it can for each medium/format in which it is presented, (with different mediums having different focuses) and SiSU tries to achieve this from minimal markup.

44.7 Can the SiSU markup be used to prepare for a LaTex automatic building of an index to the work?

Has not been, is of interest though the question on introducing such possibilities is how to keep them as unobtrusive as possible, and as generically relevant as possible to other output formats (which is why the focus on object numbers). Unobtrusive refers both to the markup (where there is no big problem with introducing optional extras); and, more challengingly how to minimise impact on competing ideas/interests, such allowing the addition of semantic tags which could be tied to objects, mapped against the objects that contain them, (permitting mapping and mining of content in various ways that would be largely agnostic of output format - object numbering being an attempt to move beyond output format based content locators (such as page numbers). The desire being to (be a meta markup and) maintain agnosticism as to what is being generated and in development to favor solutions of that nature. Keep bridging LaTeX, XML, SQL ... make use of objects and serialisation for mapping whether against content or meta-content (such as semantic [or additional structural] markers).

44.8 Can the conversion from SiSU to LaTeX be modified if we have special needs for the LaTeX, or do we need to modify the LaTeX manually?

Should be possible to modify code, it is GPLv3, should be possible either to modify existing modules or write an independent module for generating bespoke latex. Generic improvements are welcome for inclusion/incorporation in the existing code base.

If there are tools to generate mathematical/scientific formula from latex to images (jpg, png), the latex parser could conceivably be used to make these available to other output formats.

44.9 How do I create GIN or GiST index in Postgresql for use in SiSU

This at present needs to be done "manually" and it is probably necessary to alter the sample search form. The following is a helpful response from one of the contributors of GiN to Postgresql Oleg Bartunov 2006-12-06:

"I have tsearch2 slides which introduces tsearch2 ‹http://www.sai.msu.su/~megera/wiki/tsearch2slides›

FTS in PostgreSQL is provided by tsearch2, which should works without any indices (GiST or GIN) ! Indices provide performance, not functionality.

In your example I'd do ( simple way, just for demo):

0. compile, install tsearch2 and load tsearch2 into your database

cd contrib/tsearch2; make&&make&&install&&make installcheck; psql DB < tsearch2.sql

1. Add column fts, which holds tsvector

alter table documents add column fts tsvector;

2. Fill fts column

update document set fts = to_tsvector(clean);

3. create index - just for performance !

create index fts_gin_idx on document using gin(fts);

4. Run vacuum

vacuum analyze document;

That's all.

Now you can search:

select lid, metadata_tid, rank_cd(fts, q,2)as rank from document, plainto_tsquery('markup syntax') q where q @@ fts order by rank desc limit 10;

44.10 Are there some examples of using Ferret Search with a SiSU repository?

Heard good things about Ferret, but have not used it. The output directory structure and content produced by SiSU is very uniform. Have looked at a couple of other engines (hyperestraier, lucene). There it was enough to identify the files that needed to be indexed and pass them to the search indexing tool. Some Unix rune doing the job, such as:

  find /home/ralph/sisu_www -type f | \
  egrep '/sisu_www/(sisu|document_archive)/.+?.html$' | \
  egrep -v '(doc|concordance).html$' | \
  estcmd gather -sd casket -

you would have to experiment with what gives the desired result, the file doc.html is the complete text in html (there are additional smaller html segments), and plain.txt the document as a text file. It may be possible to index the text file and return the html document.

Have you had any reports of building SiSU from tar on Mac OS 10.4?

None. In the early days of its release a Mac friend built and run the ruby code part that did not rely on system calls to bits like the latex engine. That is already some years back. He was not into writing or document markup, and did it as a favour at the time. I have not followed up that thread of development.

It should however be possible, much of the output relies on plain ruby, and the system commands to latex etc. could be made appropriate for the underlying OS.

44.12 Where is version 1?

Version 1 was finally released on December 21, 2009, and largely to make a version 1 branch available as version 2 was in the pipeline with an imminent release. Most of SiSU was mature and stable long before the release of version 1.

44.13 What is the difference between version 1 and 2?

Input and output of version 1 and 2 are largely the same with the following significant exceptions. On the input side, document headers, that is metadata and processing instructions have changed in version 2. On the output side, version 2 was introduced with EPUB documents as a possible output, and over time is likely to be developed further.

Version 2 introduces a new processing layer, which relies more on the programming language Ruby' s objects (and regular expressions), than version 1 does which relies on regular expressions. The thinking behind version 1 use of regular expressions was that it made it more straightforward to switch languages for processing, as many languages support regular expressions; the thinking behind version 2 was version 1 was more complicated than it need be, and since Ruby was the language used, why not make programming more straightforward, as it would be easier to develop further.

Version 1 was removed with the introduction of the version 3 development branch