Managing a Floss Manuals website

Introduction

What's a Floss Manual website?

FLOSS Manuals consists of a central organisation website - www.flossmanuals.org - and several language sites hosted on *.flossmanuals.net. The three most active language sites are:

en.flossmanuals.net
fr.flossmanuals.net
fi.flossmanuals.net

Each of the language communities manage their own web services. The set up is more or less the same. Each language has a booktype (booki) installation for book production and then there is a 'frontpage' which is actually a separate code base known as 'bookpublisher'.

For example, the book production environment for english manuals hosted at 'booki.flossmanuals.net' is an entirely separate codebase from where the books are accessed by readers - en.flossmanuals.net

Booktype is written in python and now maintained by Sourcefabric. and Bookipublisher is maintained by FLOSS Manuals. It is the responsibility of each language community to maintain their own installations of Booktype and Bookipublisher.

Installation

Installing Booki

Booki is the place where the manuals are created and managed, and thus can be considered as the core of a Floss Manuals website. It relies on the Django (Python) framework, the Redis data structure server, the Apache 2 web server, the database of your choice (MySQL, PostgreSQL or SQLite), and many other tools and services. The following explanations are based on a Debian 6 server installation. Your preferred distribution may differ a bit, mainly due to package versions, but you've got the idea. If you have a sufficiently recent distribution, you can just use the distro versions. In that case, skip the easy_install steps and do whatever is appropriate with your package manager.

Prerequisites

Before installing Booki, you must install the following packages:

Python, minimum 2.5 (http://www.python.org/) with headers,
PostgreSQL,
Psycopg (PostgreSQL adapter),
Git (you'll use it to download the Booki and Booki Publisher sources),
Python setup tools,
Redis (manages all the communications between the browser and the Booki server; at least version 2.0),
lxml and its Python adapter,
Apache2 (web server),
the WSGI module for Apache2,
Python Imaging Library (PIL),
Python feed parser,
Gettext (needed for translation stuff),
SimpleJSON,
South,
Django-1.2.5 (1.3 reports deprecation warnings).

First, open a shell with the root account and install all the packaged applications with apt-get:

$ apt-get install python-dev postgresql python-psycopg2 git-core python-setuptools libxml2-dev libxslt-dev python-lxml apache2 libapache2-mod-wsgi python-imaging python-feedparser gettext

Then use easy-install for the following ones:
```
$ easy_install simplejson South
```

Download and manually install Django 1.2.5 (the Debian repositories have Django 1.3):

$ wget http://www.djangoproject.com/download/1.2.5/tarball/ -O Django-1.2.5.tar.gz
$ tar -xvf Django-1.2.5.tar.gz
$ cd Django-1.2.5
$ python ./setup.py install

The Redis server shipped with Debian 6 is a bit too old (1.2.6) and causes issues when used with Booki. Therefore it's necessary to install the Backports package. Add the backports repository in the apt sources list, and install Redis:
```
$ vi /etc/apt/sources.list
...
deb http://backports.debian.org/debian-backports squeeze-backports main
$ apt-get update
$apt-get -t squeeze-backports install redis-server
```

Installing the Booki files

Now let's download the booki files to server.

With the root account, add a new user (here: fm) with normal user rights that will be used to host the files:
```
$ adduser fm
```
Create a new folder on the web server root, and give the user fm full access to it:
```
$ mkdir /var/www/mybooki
$ chown -R fm /var/www/mybooki
```
Add write access rights to the web server on the directories that need to be written (data contains the books and profiles images, log the Booki warnings and errors):
```
$ cd mybooki
$ mkdir data
$ mkdir log
$ chmod g+w data/ log/
```

Open a shell as fm, and download the files on its home:

$ su - fm
$ git clone https://github.com/esetera/Booki.git

Change the Booki folder group ownership to give read and execute access (the web server should never write here!) to www-data:
```
$ chgrp -R www-data Booki
```

Create the Booki structure:

$ cd ~fm/Booki/scripts
$ ./createbooki --database postgresql_psycopg2 /var/www/mybooki/

In the install folder, edit the django settings for the booki site (here for the French site):

$ cd /var/www/mybooki
$ vi setting.py
...
# DJANGO ADMIN - set your administrators name and email address here
ADMINS = ( ('myname', 'myname@mydomain'), )
...
BOOKI_NAME = 'Floss Manuals francophone'
# "ecrire" stands for "write"
THIS_BOOKI_SERVER = 'fr.flossmanuals.net/ecrire'
...
BOOKI_ROOT = '/var/www/mybooki'
BOOKI_URL = 'http://%s' % THIS_BOOKI_SERVER
...
# Some providers may not accept the default "root@localhost" and fail with mail delivery errors.
SERVER_EMAIL = 'booki@flossmanuals.net'
...
# login URL for redirections
LOGIN_URL = '%s/accounts/signin/' %BOOKI_URL
...
# DATABASE STUFF - leave USER, PASSWORD, HOST and PORT empty
DATABASE_ENGINE ='django.db.backends.postgresql_psycopg2'
DATABASE_NAME = 'booki'
...
TIME_ZONE = 'Europe/Paris'
LANGUAGE_CODE = 'fr'
LANGUAGES = (
  ('fr', gettext('French')),
  ('en-us', gettext('English')),
)

Installing the database

Connect as postgres user, and then create the fm database user and the booki database:
```
$ su
$ su postgres
$ createuser -DRS fm
$ createdb -O fm booki
$ exit
$ exit
```
Load the booki environment and synchronize the database (don't create the superuser here, it would break the installation process):
```
$ su - fm
$ cd /var/www/mybooki
$ . ./booki.env
$ django-admin syncdb
```
Migrate booki data:
```
$ django-admin migrate
```
Now create the superuser manually (name: fm, address: fm@flossmanuals.net -or whatever you want):
```
$ django-admin.py createsuperuser
```
If you encounter a problem while installing the database, just drop booki and restart from step 1, createdb command.

Configuring Redis and Apache2

Connect as root, edit the sysctl.conf file (needed to prevent Redis from crashing at startup), and restart the server:
```
$ vi /etc/sysctl.conf
...
vm.overcommit_memory=1
$ reboot
```

Connect as fm, and edit the wsgi configuration:

$ cd /var/www/mybooki
$ vi wsgi.apache
...
# CHANGE THIS
ServerName fr.flossmanuals.net
SetEnv HTTP_HOST "fr.flossmanuals.net"
SetEnv LC_TIME "fr_FR.UTF-8"
SetEnv LANG "fr_FR.UTF-8"

Connect as root, copy the content of the wsgi file to the apache server (httpd.conf should be empty ; if it's not the case, add the content of the wsgi file to the existing httpd.conf):
```
$ cp /var/www/mybooki/wsgi.apache /etc/apache2/httpd.conf
```
Restart the apache server to take the new configuration into account:
```
$ /etc/init.d/apache2 restart 
```

Start booki

Connect as fm and start the application:

$ cd /var/www/mybooki/mybooki
$ . ./booki.env
$ django-admin.py runserver

Robot access restrictions

Some URLs needs to be hidden to indexing robots (such as Google Bot), because they are not useful to access them directly, or because they lead to admin front pages.

Booki uses a robot.txt file that contains the URLs to be disallowed. The default content is as follows, but you may need to edit it to disallow more content (don't forget to restart your web server after any change).

$ vi ~/Booki/lib/booki/portal/templates/robot.txt
# robots.txt for Booki
User-agent: *
Disallow: /admin/
Disallow: /debug/
Disallow: /_utils/
Disallow: /export/
Disallow: /_sputnik/

If you have Publisher installed, and thus Booki installed in a subdirectory (such as /write), you will need to put the robot.txt file directly at the root of the Apache server, and use it to protect all the sub-sites (booki, piwik, blog). See "Installing Booki Publisher" for details.

Production mode

With the Django's default parameters, the site runs in debug mode. It means that critical information could show in the browser if the application encounter a technical issue. Once you've correctly set and debugged Booki, it's far safer to switch to production mode, by setting the DEBUG parameter to False in the /var/www/mybooki/settings.py file.

Installing Booki Publisher

Booki Publisher is the public front section, where we show our work to the world. Users can read, download, and even buy (at Lulu.com) the books made with Booki.

Prerequisites

Before installing Booki Publisher, you must install the following packages:

PHP 5,
Apache 2 PHP 5 module,
SQLite connectors for PHP 5,
PHP 5 GD module (for the freecap widget),
tidy.
Gearman
Sqlite

Everything can be done in one go with the following command:

$ apt-get install php5 libapache2-mod-php5 php5-sqlite tidy php5-gd

$ apt-get install gearman-job-server gearman-server

$ apt-get install sqlite3

Installing the files

The Booki Publisher files are hosted in a git repository.

Connect to the server with the fm account, move to the web server's file system root, and download the files. Note that the fm account must have write privileges to the /var/www directory for git to be able to create the project.

$ cd /var/www
$ git clone https://github.com/esetera/BookiPublisher.git

Change the Booki Publisher configuration:

$ vi /var/www/BookiPublisher/config.inc.php
...
define('BOOKI_SERVER_TARGET','<your locale>.flossmanuals.net/ecrire');
...
define('DB_DSN', 'sqlite:' . INSTALLED_DIR . '/db/booki.sqlite');
...
define('BOOKI_SALT', 'wwetr74vejhg73v47tyg7thf74g'); // Don't keep the default value! Password MD5 hashes use it and keeping it would lead to security issues.

Installing the database

Booki Publisher uses a local dedicated SQLite database. You can change it and use PostgreSQL or MySQL, but since Booki Publisher is very lightweight and has no real database constraints, it's safer and more optimal to keep it in a separate database.

Connect as fm, and create the database:

$ cd /var/www/BookiPublisher/db
$ sqlite3 booki.sqlite
  sqlite> .read sqlite_db_scheme.sql
  sqlite> .exit

Modify the initialization script by replacing the user name and password with the ones you want for your database admin user:
```
$ vi ./db_init.php
...
$tableUsers->username = '<dbuser>';
$tableUsers->password = sha1('<password>' . BOOKI_SALT);
...
```
Launch the database initialization:
```
$ ./db_init.sh
```

Configuring Apache

Connect as root, and activate the mod_rewrite module:
```
$ a2enmod rewrite
```

Edit the httpd.conf created for Booki, and add the Booki Publisher's configuration:

$ vi /etc/apache2/httpd.conf

<VirtualHost *:80>
    ServerName <your locale>.flossmanuals.net
    SetEnv HTTP_HOST <your locale>.flossmanuals.net
    ...
    DocumentRoot "/var/www/BookiPublisher/"
    <Directory /var/www/BookiPublisher>
      Options Indexes FollowSymLinks MultiViews
      AllowOverride All
      Allow from all
    </Directory>

    <Directory "/usr/lib/cgi-bin">
      AllowOverride None
      Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
      Order allow,deny
      Allow from all
    </Directory>
    ...

Rename the htaccess file and edit it to reflect the server's configuration (you can remove the commented lines):

$ cd /var/www/BookiPublisher
$ mv htaccess .htaccess
$ vi .htaccess
...
AddType application/epub+zip .epub
Options FollowSymLinks
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d

RewriteRule ^robots\.txt$ - [L]

RewriteCond %{REQUEST_URI} !(\.[a-zA-Z0-9]{1,5}|/)$
RewriteRule ^/(.*)$ http://<your locale>.flossmanuals.net/$1 [R=301,L]
RewriteRule  http://<your locale>.flossmanuals.net(/)$  /index.php [NC,L]

RewriteRule  (.*)index.php(.*) index.php$2 [NC,L]
RewriteRule  (.*)admin.php(.*) - [NC,L]
RewriteRule  (.*)/_booki/(.*) _booki/$2 [NC,L]
RewriteRule  (.*)/widgets/(.*) widgets/$2  [NC,L]
RewriteRule  (.*)/_templates/(.*) _templates/$2 [NC,L]
RewriteRule  ^rss(/)$ /index.php?plugin=blog&action=rss [NC,L]

RewriteRule  ^([A-Za-z0-9-_\.]*)/([A-Za-z0-9_-]*)(/)?$/index.php?book=$1&chapter=$2 [NC,L]
RewriteRule  ^([A-Za-z0-9-_\.]*)/([A-Za-z0-9_-]*)/([A-Za-z0-9_-]*)(/)?$ /index.php?book=$1&dir=$2&chapter=$3 [NC,L]

Restart the apache server:
```
$ /etc/init.d/apache2 restart
```

Robot access restrictions

Some URLs needs to be hidden to indexing robots (such as Google Bot), because they are not useful to access them directly, or because they lead to admin front pages.

Booki uses a robot.txt file that contains the URLs to be disallowed, but this file can't be used if you have installed Booki in a subdirectory (robot.txt needs to be located at the root of the apache server). Thus as a FM website, you must put the file in your Booki Publisher directory:

$ vi /var/www/BookiPublisher/robot.txt # robots.txt for FM User-agent: *

# Booki site Disallow: /booki-path/admin/ Disallow: /booki-path/debug/ Disallow: /booki-path/_utils/ Disallow: /booki-path/export/ Disallow: /booki-path/_sputnik/

# Publisher admin page Disallow: /admin.php

# Piwik stats Disallow: /piwik

Also add any other sub-site you want to hide on your server (piwik, custom blog admin page...).

Running Bookpublisher

To run Bookpublisher the geqarman daemon must be operating:

 gearmand -p=4730 -d

Booktype

Performance tuning

Now that the server is up and running, there are some specific parts that we may want to tune in order to improve the site performance.

Set character encoding

Browsers spend some time trying to guess the character set of your HTML page if it's not specified correctly. It is recommended to always set the Content-Type meta tag in your pages as below:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

but you should also set the character encoding in the HTTP header, in your Apache site configuration or in a .htaccess file:

AddDefaultCharset UTF-8

Cache control

At least Booki Publisher (which mainly serves static content) would benefit the activation of cache control on some content type. To do so, activate the Apache Expires module (as root):

$ a2enmod expires

And then add the following lines in the Publisher's .htaccess file (values are in seconds):

FileETag INode MTime Size
ExpiresActive On
ExpiresByType image/gif A2592000
ExpiresByType image/png A2592000
ExpiresByType image/jpeg A2592000
ExpiresByType image/x-icon A2592000
ExpiresByType application/pdf A2592000
ExpiresByType application/x-javascript A2592000
ExpiresByType application/javascript A2592000
ExpiresByType text/plain A2592000
ExpiresByType text/css A604800

PNG optimization

PNG images created with Inkscape or other image tools can often be reduced with no loss of quality, using the optipng package. Of course, it's a bit difficult to reduce every image added by users with Booki, but at least, you should reduce the static ones, such as the header, site icons, etc. To do so, install the package, and run it with your image as argument:

$ apt-get install optipng
$ optipng myimage.png

Translation

Using a Floss Manuals website as offered on the git repository is great for any language. But using it with a fully translated user interface is far better!

Translating Booki

The largest part of Booki is internationalized, and you just need to create and translate a simple PO file to make it talk your preferred language. But some parts are a bit more hidden, and will require some file system speleology.

Generating a PO file

Here's the easiest part of the translation process. Thanks to Django and the Booki devs, the templates content can be quickly converted to a translation file.

Connect as fm, and create a new locale folder:
```
$ mkdir ~fm/Booki/locale
```

Modify the django settings file to tell the application where the translations are located:

$ vi /var/www/mybooki/settings.py
...
LOCALE_PATHS = (
    '/home/fm/Booki/locale/',
  )
...

Create the translation database:

$ cd ~fm/Booki
$ django-admin.py makemessages -e html -e py -e js -l <your locale>

Edit ~fm/Booki/locale/<your locale>/LC_MESSAGES/django.po with your preferred text editor or with a dedicated PO editor (such as POedit).
Compile the translation file, and enjoy your translation:
```
$ django-admin.py compilemessages
```

Configuring Xinha

Xinha is the nice text editor you use when editing a chapter in Booki. Since Xinha is already translated (at least its core modules and supported plugins), we don't need to include it in the global PO file. Instead, we just have to switch the language code in its configuration file:

$ cd /home/fm/Booki/lib/booki/editor/templates/editor/
$ vi edit_init.html
...
_editor_lang = "<your locale>";

Translating Publisher

Known issues (and workarounds)

Redis is too slow

After some weeks of intensive use, it may happen that Redis starts to eat too much RAM and respond too slowly to user request. That's something that was very common with older versions of Redis (1.2, default with Debian 6), but seems to be rare with Redis 2+.

To solve that issue, log in as root and use the following commands:

$ /etc/init.d/redis-server stop
$ rm /var/lib/redis/dump.rdb
$ /etc/init.d/redis-server start

Renaming a book breaks the attachment links

When renaming a book in Booki with:

$ django-admin.py bookrename ....

the database links to the attachments are broken, and the new url is added to the old one instead of replacing it. When renaming a book named foo with bar, the new attachment URL becomes /var/www/mybooki/data/books/bar/foo/1.0/ for all your attachments.

To fix the links, log in as your Booki database user, and run the following commands:

$ psql booki
booki=> update editor_attachment set attachment=replace(attachment, '<bad_path>', '<good_path>')

with the example above, you'll have to use bar/foo as your bad_path, and bar as the good one.

Dev Needed

Bookipublisher

Managing Booki Publisher

All the procedures below require to be connected to the admin page (http://<your locale>.flossmanuals.net/admin.php), with the database admin account.

Importing a book from Booki

Add the book to the user's access list

When importing a book for the first time, its name doesn't show in the books list of the admin user.

Click on the Userinfo menu.
In the User edit zone (at the bottom of the page), add the books you want to manage in the Book access list, and validate with the Change button.

Import the book

Once the book is in the user's access list, you can import (or update) it directly.

Click on the get Booki menu.
Select the book to import, the file formats, and click on the Get Book button.
Wait (can be long...).
Click on the bookinfo menu.
Modify the book's parameters (see the Modifying the book's parameters chapter below).

Modifying the book's parameters

Known Issues and Solutions

A book with no image doesn't import in Booki Publisher

When a book as no image at all, the HTML exportation process in Objavi fails.

Workaround: add an image (even an invisible one) somewhere in the book and retry.

Development Needed

System administration

Apache server optimization

Even on a powerful server memory can be consumed quickly when it encounters intense activity. To reduce this consumption, it is possible to tweak the configuration of the Apache server, in the /etc/apache2/apache2.conf file.

Limit the Apache 2 processes time to live

By default, they have an unlimited lifespan, and can then grow without limit. But it is possible to force them to restart (and release the consumed memory) after a certain number of requests served. To make this adjustment, change the MaxRequestsPerChild parameter (with 1000, for example) in the prefork MPM zone.

Limit the maximum number of simultaneous clients

At startup, Apache creates 10 processes. Five are ready to respond to requests (they consume about 17 MiB each), five are preloaded (6.3 MiB each). When necessary, Apache creates additional processes, which are killed when no longer useful. To avoid saturating the memory, it is possible to limit the number of additional processes. If you want to limit the memory consumption to 2 GiB, considering the average load of a process is 20 Mib, set the limit of the MaxClients parameter to 100 processes (default 150) in the prefork MPM zone.