In 2014, I migrated the OpenPetra forum from phpBB to Discourse. The reason was that Discourse had a fresh look on the way how forums can be done, it had a nice design, and easy and modern user interface. Here is the post about the details: https://www.pokorra.de/2014/09/migrate-phpbb-forum-to-discourse-using-the-ubuntu-packages-from-packager-io/

Unfortunately, the requirements of Discourse for RAM is quite high, and you cannot just install it inside a shared hosting environment, because it is recommended to be run inside a docker container. It uses technology that I just don’t know very much, with Ruby on Rails and Redis etc.

Then I discovered Flarum, which is written in PHP, but looks as nice and feels as modern as Discourse. It does not have exotic requirements, it just works. It is helpful to have an SSH account, to install the dependencies with composer, but that is available at Hostsharing for my shared hosting anyway.
Flarum is still work in progress, and officially in Beta testing. But you can run your forums in production with the latest Beta already, if you know what you are doing.

So now the issue is, how to migrate your forum from Discourse to Flarum.

At TBits.net, we worked on a migration tool, based on the scripts https://github.com/robrotheram/phpbb_to_flarumhttps://github.com/viruxe/phpbb_to_flarum, and https://github.com/Reflic/phpbb_to_flarum.

We did not cover all aspects, but only those issues we needed for our OpenPetra forum.

Things we covered were:

  • Format text properly, code, lists, http links, etc.
  • convert categories to tags
  • keep the users email addresses

We did not cover:

  • migration of uploaded content
  • we did not convert the user passwords (each user must use the resend password functionality of Flarum)
  • and probably more…

So this blog post has the goal to make our script public, and to encourage you to use it and improve it! Pull Requests are welcome, or just fork it and use it as you like!

We started to host the script first at https://github.com/Crusader99/discourse_to_flarum, then at https://github.com/The-CJ/discourse-flarum, and now the official home for the script is:

https://github.com/TBits/discourse_to_flarum

Of course, in your real life scenario, you will have a database already. But for contributing to this script, we might need a test database, for reproducing specific situations. See my other post for details how to create a test database with dummy data for Discourse!

Setting up Flarum is quite easy if you can access your webspace with SSH and you can run Composer. See the german instructions how to install Flarum inside your Hostsharing account.

If you are working with a virtual machine and CentOS7, this Ansible script might be useful:
https://github.com/TBits/discourse_to_flarum/blob/master/ansible/flarum.yml

You can run the playbook like this (assuming your test server or container has the IP 192.168.122.52, and you can access it as root via SSH and via HTTP):

ansible-playbook flarum.yml -u root -e working_host=192.168.122.52

For running the script discourse_to_flarum.php, you also need a PostgreSQL database inside that container. This Ansible playbook will setup PostgreSQL, and will load the Discourse Backup into the PostgreSQL database. It will also install pgadmin and phpMyAdmin, which helps with debugging of the conversion routines.
You can run the playbook like this:

ansible-playbook migration.yml -u root -e working_host=192.168.122.52

After this, you can access phpMyAdmin on the url http://192.168.122.52/phpMyAdmin/, the user is flarum and the password is flarum. You can access phpPgAdmin on the url http://192.168.122.52/phpPgAdmin/, the user is discourse, and the password is discourse.
And on http://192.168.122.52/, your Flarum forum is running.

Now you can actually run the conversion script:

cd /root/flarummigration
composer update
cp migrate-example.yaml migrate.yaml
# perhaps you need to modify the database credentials in migrate.yaml if you have your own setup
# if your prefix is not fl_ for the flarum tables
sed -i "s/fl_//g" migrate.yaml
php discourse_to_flarum.php

to reset the flarum database for a rerun:

zcat ../flarum.sql.gz | mysql -u flarum flarum -p
# vi discourse_flarum.php, or vi migrate.yaml
php discourse_to_flarum.php

Here are two screenshots of the sample posts in Discourse and in Flarum after the migration (mind you, some of the BBCodes also don’t work in Discourse…)

       

             

 

So there is probably still quite some work to do. Pull requests are welcome!

Tags: , , , , ,
Posted in Hosting, Software Development | Comments Closed

Seit Juli 2017 unterstützt Hostsharing die automatische Verwaltung von Let’s Encrypt Zertifikaten für einzelne Domains und auch Subdomains.

In diesem Post soll es um die Wildcard Zertifikate gehen, die Let’s Encrypt seit März 2018 ausstellt. In meinem Fall brauche ich ein Wildcard Zertifikat für das Massenhosting von OpenPetra. Die Domain wird bei Hostsharing verwaltet, aber die Server laufen bei TBits.net, also muss ich die Zertifikate nicht bei Hostsharing einrichten. Wildcard Zertifikate werden nicht über HTTP authentifiziert, sondern über DNS. Daher muss ich also bei Hostsharing die Einträge im DNS vornehmen.

Ich habe nach einem leichtgewichtigen ACME Client gesucht, der Wildcard Domains unterstützt. Dabei bin ich auf acme.sh gestoßen, das als Shell Script (bash, dash, sh kompatibel) geschrieben ist, und auf Github gehostet ist.

Installiert wird es mit diesem Befehl:

curl https://get.acme.sh | sh

Es wird dadurch ein Verzeichnis ~/.acme.sh angelegt und mit entsprechenden Dateien gefüllt, weniger als ein MB.

Hier ist eine Anleitung für manuelle Konfiguration über DNS: https://github.com/Neilpang/acme.sh/wiki/dns-manual-mode

Der erste Aufruf sieht so aus, für die Domain openpetra.com:

~/.acme.sh/acme.sh  --issue  -d '*.openpetra.com'  --dns --yes-I-know-dns-manual-mode-enough-go-ahead-please
Die Ausgabe sieht dann in etwa so aus:
[Tue Sep 25 23:08:42 CEST 2018] Creating domain key
[Tue Sep 25 23:08:42 CEST 2018] The domain key is here: /home/pacs/tim00/users/.../.acme.sh/*.openpetra.com/*.openpetra.com.key
[Tue Sep 25 23:08:42 CEST 2018] Single domain='*.openpetra.com'
[Tue Sep 25 23:08:42 CEST 2018] Getting domain auth token for each domain
[Tue Sep 25 23:08:43 CEST 2018] Getting webroot for domain='*.openpetra.com'
[Tue Sep 25 23:08:43 CEST 2018] Add the following TXT record:
[Tue Sep 25 23:08:43 CEST 2018] Domain: '_acme-challenge.openpetra.com'
[Tue Sep 25 23:08:43 CEST 2018] TXT value: 'nrsGiN7Yb4R8eYxx7RNRAP8x6MA2zTrgbCLemD1t7aB'
[Tue Sep 25 23:08:43 CEST 2018] Please be aware that you prepend _acme-challenge. before your domain
[Tue Sep 25 23:08:43 CEST 2018] so the resulting subdomain will be: _acme-challenge.openpetra.com
[Tue Sep 25 23:08:43 CEST 2018] Please add the TXT records to the domains, and re-run with --renew.
[Tue Sep 25 23:08:43 CEST 2018] Please add '--debug' or '--log' to check more details.
[Tue Sep 25 23:08:43 CEST 2018] See: https://github.com/Neilpang/acme.sh/wiki/How-to-debug-acme.sh

Nun muss ich diesen TXT record für die Domain openpetra.com einrichten.

Ich muss zuerst eine Datei im Verzeichnis ~/doms/openpetra.com/etc einrichten, mit dem Namen pri.openpetra.com. Der Inhalt sieht in etwa so aus:

{DEFAULT_ZONEFILE}
$TTL 60M
 
_acme-challenge.openpetra.com.  3600  IN  TXT  "nrsGiN7Yb4R8eYxx7RNRAP8x6MA2zTrgbCLemD1t7aB" 
 
www       IN  A {DOM_IPNUMBER}
op012345  IN  A 178.123.12.123

Nach etwa 3 Minuten sieht man in der Datei /var/log/named/named.log, dass die eigene Domain aktualisiert wurde.

Dann kann der zweite Schritt mit acme.sh ausgeführt werden:

~/.acme.sh/acme.sh  --issue  -d '*.openpetra.com'  --dns --yes-I-know-dns-manual-mode-enough-go-ahead-please --renew

Die Ausgabe sieht wie folgt aus:

[Tue Sep 25 23:09:54 CEST 2018] Renew: '*.openpetra.com'
[Tue Sep 25 23:09:55 CEST 2018] Single domain='*.openpetra.com'
[Tue Sep 25 23:09:55 CEST 2018] Getting domain auth token for each domain
[Tue Sep 25 23:09:55 CEST 2018] Verifying:*.openpetra.com
[Tue Sep 25 23:09:58 CEST 2018] Success
[Tue Sep 25 23:09:58 CEST 2018] Verify finished, start to sign.
[Tue Sep 25 23:10:00 CEST 2018] Cert success.
-----BEGIN CERTIFICATE-----
...........entfernt.................
-----END CERTIFICATE-----
[Tue Sep 25 23:10:00 CEST 2018] Your cert is in  /home/pacs/tim00/users/..../.acme.sh/*.openpetra.com/*.openpetra.com.cer 
[Tue Sep 25 23:10:00 CEST 2018] Your cert key is in  /home/pacs/tim00/users/..../.acme.sh/*.openpetra.com/*.openpetra.com.key 
[Tue Sep 25 23:10:00 CEST 2018] The intermediate CA cert is in  /home/pacs/tim00/users/..../.acme.sh/*.openpetra.com/ca.cer 
[Tue Sep 25 23:10:00 CEST 2018] And the full chain certs is there:  /home/pacs/tim00/users/..../.acme.sh/*.openpetra.com/fullchain.cer 
[Tue Sep 25 23:10:01 CEST 2018] It seems that you are using dns manual mode. please take care: The dns manual mode can not renew automatically, you must issue it again manually. You'd better use the other modes instead.
[Tue Sep 25 23:10:01 CEST 2018] Call hook error.

Ich kopiere nun die Dateien *.openpetra.com.key nach wildcard.openpetra.com.key und fullchain.cer nach wildcard.openpetra.com.crt auf den Produktionsserver, und binde sie im Nginx ein:

    ssl_certificate /var/lib/certs/wildcard.openpetra.com.crt;  
    ssl_certificate_key /var/lib/certs/wildcard.openpetra.com.key;

Da das Zertifikat nur für 3 Monate gültig ist, habe ich eine automatische Erinnerung
in meinem Kalender eingerichtet, die mich alle 2 Monate daran erinnert, das Wildcard Zertifikat zu erneuern.

Tags: , , , ,
Posted in Hosting | Comments Closed

Migrating from KnowledgeTree to SeedDMS   September 24th, 2018

For many years, I had an installation of KnowledgeTree for our small charity “Nazareth Kinderheim Kenia e.V.”.

Unfortunately, it has not been maintained anymore for many years, and the replacement of PHP5 with PHP7 forced me to do something.

I finally settled for SeedDMS as a replacement, and I really like it!

See instructions how to install SeedDMS in a shared hosting environment, at Hostsharing: https://wiki.hostsharing.net/index.php?title=SeedDMS_installieren (in German).

To export the files from KnowledgeTree, I made the decision, that I would ignore deleted files, and would not need the history of files, so only the latest version of each file. That made the migration much easier.

I found a Python script by Ben E. C. Boyter at https://boyter.org/2015/09/exporting-documents-knowledgetree-3-7-0-2/ which was a good start. I found only two issues with it: it would restore even deleted or archived files, and it would restore a random version of a file, not the latest version. So I fixed the Python script, and you can see it here as a gist: https://gist.github.com/tpokorra/71526bc60ced2649dba2c528d5cd2259

There is an import function in SeedDMS, where you can import from a directory on the server, which minimizes the used bandwidth. Alternatively, you can use WebDAV to import a lot of files at the same time.

 

Tags: , , ,
Posted in Hosting | Comments Closed

I have been using TimeSheetNG for years to record the working time for myself, so that I can keep track on where my time is going, and who I can charge for which hours 🙂

I am using version 1.5.2 of TimeSheetNG, with some small modifications to make it easier for me to enter the minutes of an hour.

My instance is running on a Hostsharing server, which was just upgraded to Debian Jessie. This meant that I got a php warning, that mysql_connect is deprecated, and I should be using mysqli instead:

Deprecated: mysql_pconnect(): The mysql extension is deprecated and will be removed in the future: use mysqli or PDO instead in .../mysql.db.inc on line 10

So here is the patch that I applied to my TimeSheetNG 1.5.2, it should work for 1.5.3 as well:

https://gist.github.com/tpokorra/3b3f5ba2cdeaa7cb31e8888e65c017dd

 

 

Tags: , ,
Posted in Hosting, Software Development | Comments Closed

Hostsharing und Let’s encrypt   December 4th, 2015

Noch geht es nicht, bei Hostsharing automatisiert Let’s Encrypt Zertifikate anzulegen. Aber es kommt bald…

Jedoch kann ich heute schon mit Hilfe des 200 Zeilen Python Skriptes ACME-tiny in der Hostsharing Umgebung ein SSL Zertifikat erzeugen und mit LetsEncrypt erzeugen lassen.

Achtung: Bitte nur durchführen, wenn Du das ACME-tiny Skript verstehst! Es greift auf deinen privaten Schlüssel zu.

Hier die Schritte innerhalb eines Hostsharing Paketes:

Einmalig auszuführen:

mkdir ~/letsencrypt
cd ~/letsencrypt
openssl genrsa 4096 > account.key
# read and understand https://github.com/diafygi/acme-tiny
wget https://raw.githubusercontent.com/diafygi/acme-tiny/master/acme_tiny.py

Zum Erzeugen und Signieren der Zertifikate für eine Domain meinedomain.de, und gegebenenfalls von weiteren Subdomains, z.b. www:

Zuerst muss eine Konfigurationsdatei erstellt werden, mit dem Namen ~/letsencrypt/ssl.cnf. Sie kann so aussehen:

[req]
req_extensions = v3_req
distinguished_name = req_distinguished_name
prompt = no
 
[req_distinguished_name]
CN = www.meinedomain.de
 
[ v3_req ]
 
# Extensions to add to a certificate request
 
keyUsage = keyEncipherment, dataEncipherment
subjectAltName = @alt_names
 
[alt_names]
DNS.1 = meinedomain.de
DNS.2 = www.meinedomain.de

Dann sind folgende Befehle auszuführen:

domain=meinedomain.de
 
cd ~/letsencrypt
openssl genrsa 4096 > $domain.key
openssl req -new -sha256 -key $domain.key -config ssl.cnf > $domain.csr
domaindir=~/doms/$domain
challengedir=$domaindir/htdocs/.well-known/acme-challenge/
mkdir -p $challengedir
for subdomain in www
do
  cd $domaindir/subs/$subdomain && ln -s ../../htdocs/.well-known && cd -
  if [ -f $domaindir/subs/$subdomain/.htaccess ]
  then
    sed -i "s/^Redirect/#letsencrypt Redirect/g" $domaindir/subs/$subdomain/.htaccess
  fi
done
if [ -f $domaindir/htdocs/.htaccess ]
then
  sed -i "s/^Redirect/#letsencrypt Redirect/g" $domaindir/htdocs/.htaccess
fi
python acme_tiny.py --account-key ./account.key --csr ./$domain.csr --acme-dir $challengedir > ./$domain.crt
for subdomain in www
do
  rm -f $domaindir/subs/$subdomain/.well-known
  if [ -f $domaindir/subs/$subdomain/.htaccess ]
  then
    sed -i "s/^#letsencrypt Redirect/Redirect/g" $domaindir/subs/$subdomain/.htaccess
  fi
done
if [ -f $domaindir/htdocs/.htaccess ]
then
  sed -i "s/^#letsencrypt Redirect/Redirect/g" $domaindir/htdocs/.htaccess
fi
rm -Rf $challengedir
wget -O - https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem > ~/doms/$domain/etc/$domain.chain
cp $domain.key ~/doms/$domain/etc/$domain.key
cp $domain.crt ~/doms/$domain/etc/$domain.crt
ls -1 ~/doms/$domain/etc/*

Danach eine E-Mail an das Service Team von Hostsharing schicken, und um eine Einrichtung des Zertifikates bitten, mit Angabe wo die Zertifikats-Dateien gespeichert sind, und für welche Domain das Zertifikat bestimmt ist.

Und eine Erinnerung im Kalender setzen, dass das Zertifikat in 3 Monaten erneuert werden muss! 🙂

Tags: , ,
Posted in Hosting | Comments Closed

Bisher konnte man bei Hostsharing pro Paket nur ein SSL Zertifikat verwenden. Das ging zumindest an meiner Realität vorbei, da ich innerhalb meines Paketes verschiedenste Domains hoste.

Seit diesem Wochenende gibt es jetzt die Möglichkeit, über SNI mehrere SSL Zertifikate innerhalb eines Paketes, und sogar innerhalb einer Domain, zu verwenden: Siehe auch Hostsharing führt Server Name Indication (SNI) ein

Als nächster Schritt wird die automatische Verlängerung von SSL Zertifikaten mit letsencrypt geplant…

Ich möchte hier kurz beschreiben, wie ich für verschiedene Subdomains unterschiedliche SSL Zertifikate einrichten kann.

Wie aus einer E-Mail auf der Technikliste von Hostsharing zu lesen ist, müssen Subdomains in der Adminoberfläche von Hostsharing (oder auf der Kommandozeile mit hsscript) als eigenständige Domain angelegt werden (doms/meinesub.meinedomain.de/), und können nicht nur als Unterverzeichnis der Hauptdomain (doms/meinedomain.de/subs/meinesub) existieren.

Erst dachte ich, ich müsste unbedingt mit www.meinesub.meinedomain.de arbeiten, bis ich auf die Tatsache stieß, dass ich im Verzeichnis doms/meinesub.meinedomain.de/htdocs-ssl die Datei .htaccess löschen kann, und stattdessen meine Anwendung dort in htdocs-ssl installieren kann. Dann ist die Anwendung unter https://meinesub.meinedomain.de erreichbar!

Solange Lets Encrypt noch nicht bei Hostsharing läuft, kann man sich bei StartSSL auch für Subdomains kostenlose Zertifikate holen. Entsprechend der Anleitung im Hostsharing Wiki die Datei https.pem bauen, und dann in doms/meinesub.meinedomain.de/etc schieben, ebenfalls die Datei sub.class1.server.ca.pem, und dann eine freundliche E-Mail an den Hostsharing Service mit der Bitte zur Einrichtung des Zertifikats schicken 🙂

Tags: , , ,
Posted in Hosting | Comments Closed

I am a member of Hostsharing eG, which is a German company owned by its members with the aim to take the idea of Open Source to Webhosting. You don’t need to worry about backups or availability, because the webmasters do it all for you. But you still have SSH access and many other powerful tools to manage your applications that are running on the server.

Recently I had to fight off an intruder who had used a security leak in one of my php applications. He had added 700 MB somewhere, the quota command showed it, but all calls of du did not show the files. Finally I realised I have to run find / -group pac00 which shows all files that belong to my own user or my subusers, who all belong to the group e.g. pac00.

Another issue that I often have is that some files are created with PHP and they belong to user httpd. That means they don’t show up at all on the quota command, which is not fair for the other users, because everyone has to pay for the used webspace.
The first step is to actually find those files. I have written the script pac-du-httpd, which goes through all the users’ directories and prints the filenames that are owned by httpd to a file, and also prints the overall sum size of those files. You can find the file here: pac-du-httpd.sh

The next step is to change the ownership of those files. On the support mailing list (yes, Hostsharing eG has as much open communication as any other Open Source project) there is a very useful explanation of how to delete such files:
Support Anleitung
The basic idea is: first run the script as normal PHP with user httpd, and change the file permissions so that everybody can change them, and then to run the script again, to delete the file.
In my situation though I don’t want to delete the file, but would like to change the permission. I finally got this solution to work:

# first run as php file with httpd, to give permissions (chmod 0666) then run with cgi to recopy the file

$dh = opendir('./');

while( ($file = (readdir($dh))) !== FALSE )
{
if ($_SERVER["ORIG_SCRIPT_NAME"] == '/cgi-bin/phpstub')
{
copy ($file, $file.".tmp");
unlink ($file);
copy ($file.".tmp", $file);
unlink ($file.".tmp");
}
else
{
chmod($file, 0666);
}
}
?>

I first run the script to change the file permissions, and then I run it again via CGI so that it runs for the user that the files should belong to. I make a copy of the file, delete the original, and rename the copy back. This works perfectly. And for the future, I will let the PHP always run via CGI (see also Hostsharing documentation on PHP via CGI)

Another useful tool is pac-du-quota, which is referenced from the Hostsharing Documentation on Webspace.
I have extended it to also list the size of the MySQL databases, which is in my case also quite a bit of data…
Insert this in the function collectSpace just before “#Wenn User exisiteren aufsammeln”:

# Mysql Datenbanken
for DB in /var/lib/mysql/${PAC}_*; do
let COUNTER=COUNTER+1
TEMP=`du -s "$DB"`
SPACE[$COUNTER]=`echo $TEMP | sed -e 's/\([^ ]*\).*/\1/g'`
NAME[$COUNTER]="Mysql_`basename $DB`"
done
Tags: , , ,
Posted in Software Development | Comments Closed