圧縮ライブラリのsnappyのインストール(ついでにpython-snappyとgflagsとgtestも)

Goal

  • Google が提供している圧縮・伸張ライブラリ snappy をインストールする
  • 同梱されているベンチマーク実行用に gflags(Google C++ commandline flags module) と gtest(Google C++ testing framework)をインストール
  • Python バインディングの python-snappy をインストール

対象 OS は CentOS 6.4

About snappy

URL

https://code.google.com/p/snappy/

Google の社内システムで広く使われてきた圧縮・伸張ライブラリで、キーワードは

  • Fast
  • Stable
  • Robust
  • Free and open source software

オープンソース後は

  • Cassandra
  • Hadoop
  • Lucene

などでも著名な OSS で利用されている。

Prerequisite

snappy のビルドで aclocal, autoconf, libtoolize を使っているので以下をインストール

$ sudo yum install automake autoconf libtool

ソースコードの取得などのために以下をインストール

$ sudo yum install subversion unzip

Change Library Path

野良インストールするので ldconfig のパスに /usr/local/lib を追記

$ sudo vi /etc/ld.so.conf

include ld.so.conf.d/*.conf
/usr/local/lib # <- added

Install gflags

URL

https://code.google.com/p/gflags/

SUMMARY

The gflags package contains a library that implements commandline flags processing. As such it’s a replacement for getopt(). It has increased flexibility, including built-in support for C++ types like string, and the ability to define flags in the source file in which they’re used.

INSTALL

ステーブルの最新版と思しき2012年1月リリースの 2.0 をリリース

$ wget https://gflags.googlecode.com/files/gflags-2.0-no-svn-files.tar.gz
$ tar zxfv gflags-2.0-no-svn-files.tar.gz
$ cd gflags-2.0
$ ./configure
$ make
$ make check
...
==================
All 5 tests passed
==================
make[1]: Leaving directory `/home/vagrant/src/gflags-2.0'
$ sudo make install
$ sudo /sbin/ldconfig -v | grep gflag
        libgflags.so.2 -> libgflags.so.2.1.0
        libgflags_nothreads.so.2 -> libgflags_nothreads.so.2.1.0

Install gtest

URL

https://code.google.com/p/googletest/

SUMMARY

Google’s framework for writing C++ tests on a variety of platforms (Linux, Mac OS X, Windows, Cygwin, Windows CE, and Symbian). Based on the xUnit architecture. Supports automatic test discovery, a rich set of assertions, user-defined assertions, death tests, fatal and non-fatal failures, value- and type-parameterized tests, various options for running the tests, and XML test report generation.

INSTALL

どうなっていると正しくインストールされたことになるのか、今ひとつわからず。
インストール手順は README の “Generic Build Instructions” のセクションに書かれている。
stackoverflow の次の Q&A も参考にした

How to easily setup googleTest on Linux
http://stackoverflow.com/questions/13513905/how-to-easily-setup-googletest-on-linux

安定版の 1.6 ではなく 1.7 RC1 をインストールした

$ wget https://googletest.googlecode.com/files/gtest-1.7.0.zip
$ unzip gtest-1.7.0.zip
$ cd gtest-1.7.0
$ ./configure
$ make
$ sudo make install
...
echo "'make install' is dangerous and not supported. Instead, see README for how to integrate Google Test into your build system."
'make install' is dangerous and not supported. Instead, see README for how to integrate Google Test into your build system.
false
...

Build a test program against Google Test library

$ cd make
$ make
$ ls
gtest-all.o  gtest_main.a  gtest_main.o  Makefile  sample1.o  sample1_unittest  sample1_unittest.o
$ ./sample1_unittest
Running main() from gtest_main.cc
[==========] Running 6 tests from 2 test cases.
[----------] Global test environment set-up.
[----------] 3 tests from FactorialTest
[ RUN      ] FactorialTest.Negative
[       OK ] FactorialTest.Negative (0 ms)
[ RUN      ] FactorialTest.Zero
[       OK ] FactorialTest.Zero (0 ms)
[ RUN      ] FactorialTest.Positive
[       OK ] FactorialTest.Positive (0 ms)
[----------] 3 tests from FactorialTest (0 ms total)

[----------] 3 tests from IsPrimeTest
[ RUN      ] IsPrimeTest.Negative
[       OK ] IsPrimeTest.Negative (0 ms)
[ RUN      ] IsPrimeTest.Trivial
[       OK ] IsPrimeTest.Trivial (0 ms)
[ RUN      ] IsPrimeTest.Positive
[       OK ] IsPrimeTest.Positive (0 ms)
[----------] 3 tests from IsPrimeTest (0 ms total)

[----------] Global test environment tear-down
[==========] 6 tests from 2 test cases ran. (0 ms total)
[  PASSED  ] 6 tests.

# it seems OK

$ cd ..
$ sudo cp -av include/gtest /usr/include
`include/gtest/gtest_prod.h' -> `/usr/local/include/gtest/gtest_prod.h'
`include/gtest/gtest-param-test.h.pump' -> `/usr/local/include/gtest/gtest-param-test.h.pump'
`include/gtest/gtest-printers.h' -> `/usr/local/include/gtest/gtest-printers.h'
`include/gtest/gtest-death-test.h' -> `/usr/local/include/gtest/gtest-death-test.h'
`include/gtest/gtest-test-part.h' -> `/usr/local/include/gtest/gtest-test-part.h'
`include/gtest/gtest-typed-test.h' -> `/usr/local/include/gtest/gtest-typed-test.h'
`include/gtest/gtest-param-test.h' -> `/usr/local/include/gtest/gtest-param-test.h'
`include/gtest/gtest-spi.h' -> `/usr/local/include/gtest/gtest-spi.h'
`include/gtest/gtest_pred_impl.h' -> `/usr/local/include/gtest/gtest_pred_impl.h'
`include/gtest/gtest.h' -> `/usr/local/include/gtest/gtest.h'
`include/gtest/gtest-message.h' -> `/usr/local/include/gtest/gtest-message.h'
`include/gtest/internal/gtest-death-test-internal.h' -> `/usr/local/include/gtest/internal/gtest-death-test-internal.h'
`include/gtest/internal/gtest-type-util.h.pump' -> `/usr/local/include/gtest/internal/gtest-type-util.h.pump'
`include/gtest/internal/gtest-param-util-generated.h' -> `/usr/local/include/gtest/internal/gtest-param-util-generated.h'
`include/gtest/internal/gtest-linked_ptr.h' -> `/usr/local/include/gtest/internal/gtest-linked_ptr.h'
`include/gtest/internal/gtest-tuple.h' -> `/usr/local/include/gtest/internal/gtest-tuple.h'
`include/gtest/internal/gtest-param-util.h' -> `/usr/local/include/gtest/internal/gtest-param-util.h'
`include/gtest/internal/gtest-string.h' -> `/usr/local/include/gtest/internal/gtest-string.h'
`include/gtest/internal/gtest-tuple.h.pump' -> `/usr/local/include/gtest/internal/gtest-tuple.h.pump'
`include/gtest/internal/gtest-filepath.h' -> `/usr/local/include/gtest/internal/gtest-filepath.h'
`include/gtest/internal/gtest-param-util-generated.h.pump' -> `/usr/local/include/gtest/internal/gtest-param-util-generated.h.pump'
`include/gtest/internal/gtest-port.h' -> `/usr/local/include/gtest/internal/gtest-port.h'
`include/gtest/internal/gtest-type-util.h' -> `/usr/local/include/gtest/internal/gtest-type-util.h'
`include/gtest/internal/gtest-internal.h' -> `/usr/local/include/gtest/internal/gtest-internal.h'
$ sudo cp -av lib/.libs/* /usr/local/lib/
`lib/.libs/libgtest.a' -> `/usr/local/lib/libgtest.a'
`lib/.libs/libgtest.la' -> `/usr/local/lib/libgtest.la'
`lib/.libs/libgtest.lai' -> `/usr/local/lib/libgtest.lai'
`lib/.libs/libgtest_main.a' -> `/usr/local/lib/libgtest_main.a'
`lib/.libs/libgtest_main.la' -> `/usr/local/lib/libgtest_main.la'
`lib/.libs/libgtest_main.lai' -> `/usr/local/lib/libgtest_main.lai'
`lib/.libs/libgtest_main.so' -> `/usr/local/lib/libgtest_main.so'
`lib/.libs/libgtest_main.so.0' -> `/usr/local/lib/libgtest_main.so.0'
`lib/.libs/libgtest_main.so.0.0.0' -> `/usr/local/lib/libgtest_main.so.0.0.0'
`lib/.libs/libgtest.so' -> `/usr/local/lib/libgtest.so'
`lib/.libs/libgtest.so.0' -> `/usr/local/lib/libgtest.so.0'
`lib/.libs/libgtest.so.0.0.0' -> `/usr/local/lib/libgtest.so.0.0.0'
$ sudo ldconfig -v | grep libgtest
        libgtest_main.so.0 -> libgtest_main.so.0.0.0
        libgtest.so.0 -> libgtest.so.0.0.0

gtest のユーティリティプログラムを手動でコピーし、利用できるようにする。

$ cd scripts
$ sudo cp -pv fuse_gtest_files.py gen_gtest_pred_impl.py gtest-config pump.py /usr/local/bin/

1.7 で追加された機能

CHANGES ファイルから拾うと

Changes for 1.7.0:

* New feature: death tests are supported on OpenBSD and in iOS simulator now.
* New feature: Google Test now implements a protocol to allow a test runner to detect that a test program has exited prematurely and report it as a failure (before it would be falsely reported as a success if the exit code is 0).
* New feature: Test::RecordProperty() can now be used outside of the lifespan of a test method, in which case it will be attributed to the current test case or the test program in the XML report.
* New feature (potentially breaking): –gtest_list_tests now prints the type parameters and value parameters for each test.
* Improvement: char pointers and char arrays are now escaped properly in failure messages.
* Improvement: failure summary in XML reports now includes file and line information.
* Improvement: the XML element now has a timestamp attribute.
* Improvement: When –gtest_filter is specified, XML report now doesn’t contain information about tests that are filtered out.
* Fixed the bug where long –gtest_filter flag values are truncated in death tests.
* Potentially breaking change: RUN_ALL_TESTS() is now implemented as a function instead of a macro in order to work better with Clang.
* Compatibility fixes with C++ 11 and various platforms.
* Bug/warning fixes.

snappy のインストール

ようやく snappy のインストール、、、

URL

https://code.google.com/p/snappy/

SUMMARY

Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more.

INSTALL

単なる思いつきでレポジトリからコードを取得してインストールする。
リリース版からインストールすれば、下の autogen.sh は不要。

$ sudo yum install zlib-devel lzo-devel # for benchmarking
$ svn checkout http://snappy.googlecode.com/svn/trunk/ snappy-read-only
$ cd snappy-read-only
$ ./autogen.sh
$ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH
$ ./configure
...
checking for 'gtest-config'... checking for gtest-config... /usr/local/bin/gtest-config
/usr/local/bin/gtest-config
checking for Google Test... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for gflags... yes
checking if the compiler supports __builtin_expect... yes
checking if the compiler supports __builtin_ctzll... yes
checking for zlibVersion in -lz... yes
checking for lzo1x_1_15_compress in -llzo2... yes
checking for lzf_compress in -llzf... no
checking for fastlz_compress in -lfastlz... no
checking for qlz_compress in -lquicklz... no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating snappy-stubs-public.h
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands
...
$ make
$ sudo make install
$ sudo ldconfig -v | grep snap
        libsnappy.so.1 -> libsnappy.so.1.1.4

$ ldconfig -p | grep snappy
        libsnappy.so.1 (libc6,x86-64) => /usr/local/lib/libsnappy.so.1
        libsnappy.so.1 (libc6,x86-64) => /usr/lib/libsnappy.so.1
        libsnappy.so (libc6,x86-64) => /usr/local/lib/libsnappy.so
        libsnappy.so (libc6,x86-64) => /usr/lib/libsnappy.so

autogen.sh の実行について

$ ./autogen.sh: line 3: aclocal: command not foundというメッセージが表示された場合は automake をインストール

$ ./autogen.sh: line 5: libtoolize: command not foundというメッセージが表示された場合は libtool をインストール

configure 時の gtest/gflags のチェックについて

configure 時の後ろの方で gtest/gflags がインストールされているかチェックしている

...
checking for 'gtest-config'... checking for gtest-config... /usr/local/bin/gtest-config
/usr/local/bin/gtest-config
checking for Google Test... yes
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for gflags... yes
...

gtest はプログラム gtest-config が見つかるかチェックしている模様。
gflagspkg-config のサーチパスに libgflags.pc が見つかるかチェックしている模様。

$ pkg-config --exists --print-errors libgflags
Package libgflags was not found in the pkg-config search path.
Perhaps you should add the directory containing `libgflags.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libgflags' found
$ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:$PKG_CONFIG_PATH
$ pkg-config --exists --print-errors libgflags
$

今回は環境変数 PKG_CONFIG_PATH に /usr/local/lib/pkgconfig を無理やり突っ込んで解決したけれども、これが本筋なのかは不明。

pkg-config の概要をつかむには次の “Guide to pkg-config” もおすすめ

http://people.freedesktop.org/~dbn/pkg-config-guide.html

snappy のテスト

make すると snappy_unittest というユニットテスト・ベンチマークプログラムももれなくコンパイルされる。

$ ./snappy_unittest --help でオプションが表示される。

ユニットテストの実行

・ユニットテストと snappy のマイクロベンチマークを実行

$ ./snappy_unittest -norun_microbenchmarks # or $ ./snappy_unittest --run_microbenchmarks=false

・他の圧縮方式とのベンチマークを実行

$ ./snappy_unittest -norun_microbenchmarks -zlib -lzo testdata/*
testdata/alice29.snappy                  :
ZLIB:   [b 1M] bytes  88057 ->  77722 88.3%  comp  16.8 MB/s  uncomp 109.7 MB/s
LZO:    [b 1M] bytes  88057 ->  88631 100.7%  comp  42.6 MB/s  uncomp 1252.9 MB/s
SNAPPY: [b 4M] bytes  88057 ->  88066 100.0%  comp 2665.3 MB/s  uncomp 7187.8 MB/s
testdata/alice29.txt                     :
ZLIB:   [b 1M] bytes 152089 ->  54404 35.8%  comp  13.0 MB/s  uncomp 175.4 MB/s
LZO:    [b 1M] bytes 152089 ->  82721 54.4%  comp 112.7 MB/s  uncomp 275.3 MB/s
SNAPPY: [b 4M] bytes 152089 ->  88017 57.9%  comp 153.5 MB/s  uncomp 467.2 MB/s
...

Install Python Binding

最後に python-snappy のインストール

URL

https://github.com/andrix/python-snappy

SUMMARY

Python library for the snappy compression library from Google

INSTALL

CentOS 6.4 標準の Python 2.6 に対して pip 経由でインストールする。
Python のヘッダーファイルが必要なので python-devel もインストールする。

$ wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py -O - | sudo python
$ sudo easy_install pip
...
snappymodule.cc:28:20: error: Python.h: No such file or directory
$ sudo yum install python-devel
$ sudo pip install python-snappy
Downloading/unpacking python-snappy
  Downloading python-snappy-0.5.tar.gz
  Running setup.py egg_info for package python-snappy

Installing collected packages: python-snappy
  Running setup.py install for python-snappy
    building '_snappy' extension
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c snappymodule.cc -o build/temp.linux-x86_64-2.7/snappymodule.o
    cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for Ada/C/ObjC but not for C++ [enabled by default]
    snappymodule.cc:45:28: warning: ‘_state’ defined but not used [-Wunused-variable]
    gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c crc32c.c -o build/temp.linux-x86_64-2.7/crc32c.o
    g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro build/temp.linux-x86_64-2.7/snappymodule.o build/temp.linux-x86_64-2.7/crc32c.o -lsnappy -o build/lib.linux-x86_64-2.7/_snappy.so

Successfully installed python-snappy
Cleaning up...

USAGE

通常の圧縮・伸張

compressed = snappy.compress(text)
snappy.decompress(compressed))

ストリームに対して圧縮・伸張

仕様 : https://code.google.com/p/snappy/source/browse/trunk/framing_format.txt

text = "hello world!"
src, dst = StringIO(text), StringIO()
snappy.stream_compress(src, dst)
dst.seek(0)
src, dst = dst, StringIO()
snappy.stream_decompress(src, dst)
dst.getvalue()

コマンドラインから実行

$ python -m snappy -c < alice29.txt > alice29.snappy

References

Advertisements
Tagged with: , , , ,
Posted in linux, middleware

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Archives
  • RT @__apf__: How to write a research paper: a guide for software engineers & practitioners. docs.google.com/presentation/d… /cc @inwyrd 4 months ago
  • RT @HayatoChiba: 昔、自然と対話しながら数学に打ち込んだら何かを悟れるのではと思いたち、専門書1つだけ持ってパワースポットで名高い奈良の山奥に1週間籠ったことがある。しかし泊まった民宿にドカベンが全巻揃っていたため、水島新司と対話しただけで1週間過ぎた。 それ… 4 months ago
  • RT @googlecloud: Ever wonder what underwater fiber optic internet cables look like? Look no further than this deep dive w/ @NatAndLo: https… 4 months ago
  • @ijin UTC+01:00 な時間帯で生活しています、、、 10 months ago
  • RT @mattcutts: Google's world-class Site Reliability Engineering team wrote a new book: amazon.com/Site-Reliabili… It's about managing produc… 1 year ago
%d bloggers like this: