I recently had to provide a working solution for Traditional Chinese for Hongkong.

It involved both ODBC on the Linux server to read correct data from a Progress database, and process that data in .net on Mono, but also on the client side to be able to read data through ODBC.

Here is the solution and also some warnings about what can go wrong:

For Mono, you need to have installed the RPM package mono-locale-extras, which installs the dll I18N.CJK.dll (CJK stands for Chinese Japanese Korean).

Make sure the Progress database is in codepage big-5.
proutil mydb -C convchar convert undefined -G 15
proutil mydb -C convchar convert big-5 -G 15
_progres -pf batch.pf -1 -b -rx -p upgrade.p -param $DLC/prolang/tch/big-5/_tran.df | cat
proutil mydb -C idxbuild all

Make sure that your pf files (or even the Progress startup.pf) contain these lines:
-cpinternal big-5
-cpstream big-5
-cprcodein undefined

For the Progress server, you need to set this variable before the server is started (note: if you have both sql and 4gl server, the first one needs all settings already):

Mono would rather expect BIG5 (without the dash), so we need to set another variable so that the default encoding is correct.
Before you start the server in the Mono environment, make sure that you export this variable:
export LANG=zh_TW.big5

The following command shows what languages are available on your Linux:
locale -a
Unfortunately there is no big5 for Hongkong, and zh_HK.big5hkscs does not seem to work for me. zh_TW.big5 works fine.

This code converts the data from the codepage coming through ODBC from the Progress database into Unicode:

FDBEncoding := System.Text.Encoding.Default;
sqlClientCodePage := Convert.ToInt32(System.Environment.GetEnvironmentVariable(‘SQL_CLIENT_CHARSET’));
FDBEncoding := System.Text.Encoding.GetEncoding(sqlClientCodePage);
except on exp: Exception do

function ConvertToUnicode(s: System.String): System.String;
bytes: array of byte;
bytes := Encoding.Default.GetBytes(s);
result := FDBEncoding.GetChars(bytes);

function ConvertFromUnicode(s: System.String): System.String;
bytes: array of byte;
bytes := FDBEncoding.GetBytes(s);
result := Encoding.Default.GetChars(bytes);

For the client to be able to access through ODBC, we also need to set the environment variable in Windows XP SQL_CLIENT_CHARSET=BIG-5
(Start/Settings/Control Panel/System/Advanced/Environment Variables/System or User Variables)

Tags: , , ,
Posted in Software Development | Comments Closed

BsDiff for Windows and .net   November 18th, 2007

This program is based on the excellent unix tool bsdiff from Colin Percival.

I made some adjustments, so that it would compile in C++ mode in Dev-C++ IDE (mingw compiler), using the libbzip2 package.

This version is based on bsdiff 4.1. The difference between 4.1 and 4.2 is only in the patch program, instead of using lseek and read he now uses pread, which I find this text about it:
Another thing to take note of is that Linux 2.2 introduces the pread() and pwrite() system calls. These allow a process to read and write from a specified position in a file. This is similar, but not identical, to using the lseek() system call followed by an ordinary read() or write() system call. In particular, concurrent access to a file (required for asynchronous I/O (AIO) support) requires the pread() and pwrite() system calls.

In the windows version, I had to adjust the code anyway, because I am not using pipes, but keep the file open for reading. I guess the bug of the official bsdiff had to do with the concurrent access of the several forks of the program to the file. That does not happen in the windows version at all, the file is opened only once.

For questions, write to Timotheus Pokorra.

Here are the sources (including the .dev files for Dev-C++): bsdiff_win_src.zip
Here are the 2 executables: bsdiff_win_exe.zip
The code is licensed under the BSD Protection License (see the original LICENSE file in bsdiff-4.1.tar.gz).

See also Andreas John’s bsdiff for Visual C++.

May 2007: I have ported the program to Delphi.net, and with small modifications (make it a proper dll) you can use it even with C# programs. If anyone is interested in the code, I will be happy to share it. Drop me an email…
June 2007: Ok, someone has dropped me an email, and here is the delphi.net version, with all required dlls. also read the readme.txt included in the archive, and the documentation in bsdiff.pas. If you have questions, ask!