Charles Explorer logo
🇬🇧

Compression of concatenated web pages using XBW

Publication at Faculty of Mathematics and Physics |
2007

Abstract

hod PPM. Coder offers choice of Huffman and arithmetic coding.XBW is modular program for lossless compression that enables testing various combinations of algorithms.

We obtained best results with XML parser creating dictionary of syllables or words combined with Burrows-Wheeler transform - hence the name XBW. The motivation for creating parser that handles non-valid XML and HTML files, has been system EGOTHOR for full-text searching.

On files of size approximately 20MB, formed by hundreds of web pages, we achieved twice the compression ratio of bzip2 while running only twice as long. For smaller files, XBW has very good results, compared with other programs, especially for languages with rich morphology such as Slovak or German.

For any big textual files, our program has good balance of compression and run time. Program XBW enables use of parser and coder with any implemented algorithm for compression. We have implemented Burrows-Wheeler transform which together with MTF and RLE forms block compres