When I have to convert data between formats, I reach for Perl. While many
people think Perl's built in regular expressions make data munging easy, my
experience is that Perl's multi-paradigm nature and dynamic programming
flexibility are more important.
The Problem
I help run ClubCompy, a retro-inspired,
zero-installation, browser-based programming environment designed to help
children learn about computing. One of the reasons they recruited me is to
design the educational components, including documentation. (I also know a few
things about compilers and business.)
While ClubCompy has a surprising amount of power in its underlying virtual
machine, that power is currently exposed in a programming language called
Tasty—a
mixture of 8-bit BASIC and Logo.
As with most systems which evolve from a simple idea into something else,
following the law of opportunism, the project's structure and organization and
tooling has accreted organically instead of following a rigid design. (Startup
hackers: your job is to prune things when necessary until you discover the core
of your business.) In particular, the documentation for the Tasty language
exists in a series of OpenOffice files,
one per language keyword.
The good news is that documentation exists. It's mostly complete, too: every
keyword has documentation, and most of it is comprehensive. (Maybe 15 or 20%
needs expansion, but we'll get there.)
The bad news is that the documentation exists in .odt files.
They're not binary blobs, but they don't fit with our publishing system:
they're too difficult to convert to clean PDF or very clean HTML for use
throughout the system. They're also a mess when checked into source
control.
Monday I decided to convert them to POD. (ClubCompy uses the Onyx Neon publishing toolchain designed for
things like Modern
Perl: the book. Everything not yet available on the CPAN is available from
my Github account.)
Inside ODT Files
An OpenOffice .odt file is a zipped archive of several other files.
Fortunately, there's only one file I care about and very fortunately, it's a
reasonably self-contained XML file. Getting the contents of
content.xml is easy with a little bit of Archive::Zip code:
use Archive::Zip;
sub get_xml_contents
{
my $file = shift;
my $zip = Archive::Zip->new( $file );
my $content = $zip->memberNamed( 'content.xml' );
return $content->contents;
}
All of the Tasty keywords follow a standard template for documentation. This
is both good and bad. It's good that discovering out how OpenOffice represents
each unique element in XML is relatively easy: figure it out once and that
representation should apply to all files. It's bad that the documentation
template didn't use custom semantic styles, like "Top-level Header" and
"Program Code".
That means all of the styles are ad hoc:
<office:automatic-styles>
<style:style style:name="P1" style:family="paragraph" style:parent-style-name="Standard">
<style:paragraph-properties fo:background-color="#666699">
<style:background-image />
</style:paragraph-properties>
<style:text-properties fo:color="#ffffff" style:font-name="Calibri" fo:font-size="14pt" fo:font-weight="bold" style:font-size-asian="14pt" style:font-weight-asian="bold" style:font-size-complex="14pt" style:font-weight-complex="bold" />
</style:style>
<style:style style:name="P2" style:family="paragraph" style:parent-style-name="Standard" style:master-page-name="">
<style:paragraph-properties fo:margin-left="0.2602in" fo:margin-right="0in" fo:text-indent="0in" style:auto-text-indent="false" style:page-number="auto" fo:background-color="#9999cc">
<style:background-image />
</style:paragraph-properties>
<style:text-properties fo:color="#ffffff" style:font-name="Calibri" fo:font-size="12pt" fo:font-weight="bold" style:font-size-asian="12pt" style:font-weight-asian="bold" style:font-size-complex="12pt" style:font-weight-complex="bold" />
</style:style>
...
</office:automatic-styles>
I'll explain that more later.
The actual text of each file resembles:
<office:body>
<office:text>
<text:sequence-decls>
<text:sequence-decl text:display-outline-level="0" text:name="Illustration" />
<text:sequence-decl text:display-outline-level="0" text:name="Table" />
<text:sequence-decl text:display-outline-level="0" text:name="Text" />
<text:sequence-decl text:display-outline-level="0" text:name="Drawing" />
</text:sequence-decls>
<text:p text:style-name="P1">Keyword</text:p>
<text:p text:style-name="P9">WHILE<text:span text:style-name="T1">-DO</text:span>
</text:p>
<text:p text:style-name="P8">
<text:span text:style-name="T2">END</text:span>
</text:p>
...
</office:text>
</office:body>
All of the text of the documentation is available under
<text:p> tags.
Extracting Text
Extracting this text is a job for XPath. While I could get more specific
with the XPath expression (find all direct children of
<office:text>), I went for the simple solution at first:
use XML::XPath;
use XML::XPath::XMLParser;
sub rewrite_xml
{
my $contents = shift;
my $xpath = XML::XPath->new( xml => $contents );
set_methods_for_styles( get_xml_style_methods( $xpath ) );
my $pod = xml_to_pod( $xpath );
clear_methods_for_styles();
return $pod;
}
sub xml_to_pod
{
my $xpath = shift;
my $nodeset = $xpath->find( '//text:p' );
my $pod;
for my $node ($nodeset->get_nodelist)
{
my $style = $node->getAttribute( 'text:style-name' );
$style = 'Empty' if @{ $node->getChildNodes } == 0;
my $method = get_method_for_style( $style );
$pod .= $node->$method;
}
return $pod;
}
Ignore the get_method_for_style() calls for now. The important
part of xml_to_pod is that it finds these tags in the XML and
performs an action on each of them.
What's that action? Transforming it to POD, of course.
Look in the sample XML again. Each of the paragraphs has an associated style
tag. That style refers to one of the styles declared earlier in that file.
Given the name of a style, the body of the loop finds the name of a
method and calls that method to transliterate the contents of that tag to
POD.
Transliterating to POD
Here's where the power of Perl really shines. Every node in that nodeset is
an instance of XML::XPath::Node::Element.
That class knows nothing about POD. At least, it knows nothing about POD until
I declared some methods in it:
package XML::XPath::Node::Element;
sub kidsToPod { join '', map { $_->toPod } shift->getChildNodes }
sub toPod
{
my $self = shift;
my ($name) = $self->getName =~ /text:(\w+)/;
my $method = 'toPodFor' . ucfirst $name;
return $self->$method;
}
sub toPodForEmpty { '' }
sub toPodForS { ' ' }
sub toPodForTab { ' ' }
sub toPodForSpan
{
my $self = shift;
my $style = $self->getAttribute( 'text:style-name' ) // '';
$style = 'Empty' if @{ $self->getChildNodes } == 0;
my $method = main::get_method_for_style( $style );
return $self->$method;
}
sub toPodForBold { 'B<' . shift->kidsToPod . '>' }
sub toPodForCode { 'C<' . shift->kidsToPod . '>' }
sub toPodForCodePara { ' ' . shift->kidsToPod . "\n" }
sub toPodForItalic { 'I<' . shift->kidsToPod . '>' }
sub toPodForPlain { shift->wrapKids( '', '' ) }
sub toPodForPlainPara { shift->wrapKids( '', '' ) . "\n\n" }
sub toPodForBoldCode { 'C<B<' . shift->kidsToPod . '>>' }
sub toPodForBoldCodePara { 'C<B<' . shift->kidsToPod . ">>\n" }
sub toPodForHead0 { shift->wrapKids( '=head0 ', "\n\n" ) }
sub toPodForHead1 { shift->wrapKids( '=head1 ', "\n\n" ) }
sub toPodForHead2 { shift->wrapKids( '=head2 ', "\n\n" ) }
sub wrapKids
{
my ($self, $pre, $post) = @_;
my $kid_text = $self->kidsToPod;
return '' unless $kid_text;
return $pre . $kid_text . $post;
}
Because Perl has open classes, you can add methods to classes (or redefine
methods) any time you want. Because Perl has dynamic method dispatch, you can
use a string as the name of a method to call.
You can see that this code gets a little bit messy here. That's part and
parcel of the
tree transformation technique central to compilers; the real world is
messy, and that mess has to go somewhere.
The wrapKids() method handles the case where one of these nodes has no textual content but does have a specific style. Given a snippet of documentation like:
Example 1:
10 x = 0
20 WHILE x LT 26 DO
30 PRINT TOCHAR x + 65
40 x = x + 1
50 END
RUN
(prints ABCDEFGHIJKLMNOPQRSTUVWXYZ)
... the blank line between RUN and the output is a unique
paragraph with the monospace font applied. A naïve output from one of
these methods might produce the POD C<> for that line.
wrapKids() prevents that.
This open class approach works very well. It scales well too in
terms of complexity. Even if this code eventually migrates to build a POD
document model (see Pod::PseudoPod::DOM),
giving individual nodes the responsibility of emitting a tree or text moves the
custom behavior to where it most belongs.
(The benefit of a DOM is that basic tree transformation rules can take care
of pruning out unnecessary elements, such as the blank code line.)
The Little Details
The XML::XPath::Node::Elements may nest, but you can see how
that nesting works just fine through the toPod() method. Those
::Element classes may themselves also contain XML::XPath::Node::Text
instances as children. These objects represent plain text.
So far, I've only found one situation where this plain text needs any
manipulation. Adding one method fixes this:
package XML::XPath::Node::Text;
sub toPod
{
my $raw_text = HTML::Entities::decode_entities( shift->toString );
return main::encode_pod( $raw_text );
}
The encode_pod() function (it's in main so as not
to make it available as a method inadvertently) is:
use Regexp::Assemble;
my %escapes =
(
'<' => 'E<lt>',
'>' => 'E<gt>',
);
sub encode_pod
{
state $replace = make_regexp( \%escapes );
my $text = shift;
$text =~ s/($replace)/$escapes{$1}/g;
return $text;
}
sub make_regexp
{
my $escapes = shift;
my $ra = Regexp::Assemble->new;
$ra->add( $_ ) for keys %$escapes;
return $ra->re;
}
More robust solutions exist, but so far this is all I've needed.
I do admit that the implementation is a little messy in places. That's one
of the problems with this compiler technique: sometimes you have data that
needs to be available everywhere but you don't want to pass it as arguments
everywhere and you don't want to wrap up everything in intermediary objects
because you're already using perfectly good objects from elsewhere.
I haven't shown the code which identifies styles and makes the hash of style
name to output method yet; that's for the next post. I'm sure you can start to
figure out how it works already.
