IT challenges in Wycliffe Bible Translators - part 2

Last month I wrote about IT challenges in Wycliffe Bible Translators with regards to software. This month I'll tell you about the challenges we have with our site. The Wycliffe Centre is set in a beautiful location. However it's quite spread out.

View Larger Map

This means that wiring up the site has been quite a task for the past few years. We have a mixture of copper and fibre for the longer stretches between buildings, or even along some of the longer buildings. This means the fun of pulling cables as well as switches and converters along the way. We use wireless coverage for some locations where it is appropriate. We've managed to get network connections to pretty much every location. However maintenance and upgrading is an ongoing task.

Charity bag overload

This is a picture of all the bags that have been put through our door by charities over the last couple of years. They are all asking you to put unused stuff in there and leave the bag on your doorstep on a certain day. Even if we'd used them (which we haven't) by the time we'd used the first couple or so we wouldn't have much more to give away every few weeks as the next bags arrive.

It's like fishing in an empty pond. Come on charities, be more smart.

Lotus Notes to mediawiki

I'm in the middle of moving a Lotus Notes document database to a mediawiki wiki. (Or is that a media wiki?)

I did it by means of a Perl script using the HTML::WikiConverter module and a Python script.

The starting point is to save each Notes document as a web page (using Firefox, and saving as "complete" so you get all the images). The Perl script (reproduced at the end) then converts each web page to a text file. I did a bit of custom processing to remove some tags: font, div, center, and some attributes: width, border, valign, bgcolor.

(In order to get the WikiConverter module to work I had to fix a bug in either the module, or the media wiki specific module, sorry I can't remember which. However a workaround was listed in the bug report and involved a script to rebuild the grammer in the CSS::Parse module.)

The Python script then takes the text files, now in mediawiki format, and converts to XML file(s). For testing I use an xml file per text file. For the real thing I put them all in one xml file. The Python puts the appropriate XML around the text so that the pages have titles. I haven't included the Python, as it's quite specific to what I want. However here's a clue for you - the title is two lines after a line with the word "Subject:" in it.

As well as parsing the text to find the title at the top, I also converted a document history table at the bottom of each file (part of our documents, not part of the Notes template) into a series of mediawiki "revisions", so that the information on what, who and when each document was changed wasn't lost. This is useful even though I don't have the actual revisions.

Each page does need a bit of attention, because this three stage conversion isn't perfect. For example, successive bullet points have blank lines between them, which is fine until you have indented bullets, when they don't render properly in mediawiki.

One thing I wish I'd done with hindsight, is put a category onto each page, which I could remove once I'd tidied it up, to see what remains to be done. As it is I've used a category once I've tidied it, but eventually every page will have that category, which will be meaningless. To remove it would mean editing every page, unless there's some global change plugin I'm not aware of.

# Convert saved pages from Notes Documents to media wiki format

use HTML::WikiConverter;

sub DropTag($) {
 my ($page, $tag) = @_;
 my @tags = $page->look_down("_tag",$tag); # Font tags
 foreach my $element (@tags) {
  $element->replace_with_content();
 }
}

sub DropAttr($$) {
 my ($page, $attr) = @_;
 my @attrs = $page->look_down($attr,qr/.*/); # Tags with appropriate attribute set to anything
 foreach my $element (@attrs) {
  $element->attr($attr, undef);
 }
}

sub ExtraProcessing ($) {
# Does various extra things that we need:
my($page) = @_;

DropTag($page, "font");
DropTag($page, "div");
DropTag($page, "center");

DropAttr($page, "width");
DropAttr($page, "border");
DropAttr($page, "valign");
DropAttr($page, "bgcolor");
}


my $wc = new HTML::WikiConverter( dialect => 'MediaWiki' );
opendir(DIR, "saved html files");
@FILES= readdir(DIR);
foreach my $path (@FILES) {
 if ($path =~ m/\.htm/) {
  print $path."\n";
  open FILE, ">output text files".$path or die("Could not open file for output\n");
  print FILE $wc->html2wiki( file => "saved html files".$path, strip_tags => [ '~comment', 'head', 'script', 'style' ], preprocess => \&ExtraProcessing);
  close FILE;
 }
}

IT challenges in Wycliffe Bible Translators

Here at the UK HQ of Wycliffe Bible Translators we have a number of challenges when it comes to IT. Some of our activities fit in with normal business or charitable activities and so you can get off-the-shelf software that mostly does what we want. Other things we do aren't a good match.

We use The Raiser's Edge for Donor Relationship Management (a bit like CRM except that the first word begins with D). We sent out mailings and receive donations, so that's a fairly good fit.

We have Sage for accounts. It's not to great for doing charitable type accounts, but then most charities find that and get around it with spreadsheets or third-party solutions. We use spreadsheets.

On the personnel side most of our staff are volunteers, and most don't work in this country. So personnel software isn't much use, as it thinks about employees, and payroll, and contracts and leave whereas we think about membership agreements, and seconding people overseas and strange concepts like "furlough" (the old fashioned term) aka Home Leave, and "topping up support". Personnel software doesn't care about people's children, and what medical and schooling needs they have, whereas we care about the whole family that we send abroad. So all the personnel stuff has been done in-house.

Where the people and the money come together we use the services of Wycliffe International who save us having to send money to 50 different countries, by acting as a clearinghouse. You can't get off-the-shelf software for interfacing UK accounts software with Wycliffe International, so we've done that ourselves.

We run courses, so we're like a small university with the need to book people in on courses, give them accommodation and food. We also have spare accommodation so we host conferences. We have software such as a hotel might use, but that's orientated towards individual guests, not conferences or courses. That's where my current challenges lie.

Essential reading for IT Directors

When I learnt I was going to become an IT Director I search out a page I'd come across a while before - Desert Island Standards on the excellent ICT Knowledgebase from LASA (London Advice Services Alliance). It contains 8 IT-related standards. IT is more than complying to standards documents, but it's a good place to start.

Update: 25/9/2021 - that article has gone so I've linked to the archived version instead. LASA seem to have changed direction so I've unlinked them.

My first month as IT Director

I've just written my first monthly report as IT Director for Wycliffe Bible Translators in the UK. Don't worry, I'm not reproducing it here. Whilst a lot of what I did was part of my previous job, there were some Director-type things, like looking into Data Protection.

One of the most useful things I found whilst doing that was that the scope of the Act wasn't as broad as I first thought. When I attended a seminar on it many years ago the person leading it said that unless you chucked all your paperwork on people into a large pile in a room, than any filing system would fall under the scope of the Act. However on the Information Commissioner's Office website it has a document which describes the "temp test". So if your data about people is arranged in such a way that a temp couldn't find information about specific people, then it's not a "relevant filing system" under the act. This means that every scrap of personal data, even, say, squirreled away on archive tapes in a safe, doesn't have to be erased when we no longer have need to store that person's data.