ITPub博客

首页 > IT基础架构 > 服务器/存储 > Ceph源码编译与打包

Ceph源码编译与打包

原创 服务器/存储 作者:jesselyu 时间:2017-04-07 19:31:12 0 删除 编辑
1.1clone repository

源码打包docker 7u机器:

lvjian100081200005.et15sqa

git clone --recursive

#checkout 出某一个版本

git clone –b v10.2.2

cd ceph

git status

git submodule update --init --recursive

git checkout master

1.2编译Ceph

解决依赖包:

#ceph相关,获取依赖的方法:

ceph.spec | grep -i require | sort -nk1 | uniq

sudo yum install libudev-devel libblkid-devel libuuid-devel libaio-devel fuse-devel \

xfsprogs-devel expat-devel fcgi-devel libatomic_ops-devel python-libs python-babel python-sphinx gperftools-devel \

boost-devel libunwind-devel libcurl-devel nss-devel Cython keyutils-libs-devel openssl-devel openldap-devel glibc-devel glibc-static -b current

#leveldb和rocksdb相关

sudo yum install snappy-devel bzip2-devel gflags-devel leveldb-devel

不需要特意安装rocksdb,因为已经在ceph目录下了。编译过程会报错,原因是“gflags namespace找成google“了。在main函数前增加namespace重定义:

//

// main

//

namespace gogle = gflags;

int main(int argc, char** argv) {

google::SetUsageMessage(std::string("\nUSAGE:\n") + std::string(argv[0]) +

" [OPTIONS]...");

google::ParseCommandLineFlags(&argc, &argv, false);

编译rocksdb:

sudo yum install devtoolset-2-gcc devtoolset-2-gcc-c++ -b current

sudo yum install devtoolset-2-binutils -b current

source /opt/rh/devtoolset-2/enable

export CC=/opt/rh/devtoolset-2/root/usr/bin/gcc

export CPP=/opt/rh/devtoolset-2/root/usr/bin/cpp

export CXX=/opt/rh/devtoolset-2/root/usr/bin/c++

make –j2

编译ceph:

./do_cmake.sh

cd build

cmake -DCMAKE_INSTALL_PREFIX=/opt/ceph -DCMAKE_C_FLAGS="-O0 -g3” ..

make –j24

如果是10.2.2版本,修改do_autogen.sh或者configure带参数,将安装目录放到/opt/ceph下面:

./configure \

--prefix=/opt/ceph --sbindir=/opt/ceph/sbin --localstatedir=/opt/ceph/var --sysconfdir=/opt/ceph/etc \

--with-debug $with_profiler --with-nss --without-cryptopp --with-radosgw \

修改ceph-detect-init的目录,将安装目录放到/opt/ceph下面,打开src/ceph-detect-init/Makefile.am

./configure \

--prefix=/opt/ceph --sbindir=/opt/ceph/sbin --localstatedir=/opt/ceph/var --sysconfdir=/opt/ceph/etc \

--with-debug $with_profiler --with-nss --without-cryptopp --with-radosgw \

./autogen.sh

./configure或者./do_autogen.sh -J -L -d 1

make –j24

make install DESTDIR=/opt/ceph

1.3RPM打包

RPM打包,依赖包解决:

#build RPM package,build RPM时遇到的依赖

sudo yum install valgrind-devel sharutils libxml2-devel libxslt-devel -b current

sudo yum install python-nose python-requests python-virtualenv python-devel python-setuptools Cython python-pip binutils-devel -b current

sudo yum install redhat-lsb-core yum-utils –b current

sudo yum install java-devel junit hdparm -b current

setuptools

#以下几个包在7u上没有,需要手动下载

sudo rpm –ivh jq-1.5-4.fc25.x86_64.rpm

sudo rpm –ivh yasm-1.3.0-3.fc24.x86_64.rpm

sudo rpm –ivh xmlstarlet-1.6.1-6.fc24.x86_64.rpm

sudo rpm –ivh oniguruma-6.0.0-1.fc25.x86_64.rpm

sudo rpm –ivh fcgi-devel-2.4.0-15.3.x86_64.rpm

sudo rpm –ivh lib64fcgi0-2.4.0-15.mga4.x86_64.rpm

sudo rpm –ivh fcgi-2.4.0-15.3.x86_64.rpm

sudo rpm –ivh jemalloc-4.2.1-1.fc26.x86_64.rpm

sudo rpm –ivh jemalloc-devel-4.2.1-1.fc26.x86_64.rpm

另外,需要将ceph.spec文件中的”WITH_PYTHON3=ON”改成”WITH_PYTHON3=OFF”。

./make_dist.sh

关闭debug package

echo “%debug_package %{nil}” >>.rpmmacros

修改ceph.spec文件:

%global _prefix /opt/ceph

%global _libexecdir %{_exec_prefix}/lib

%global __libtool_path ^%{buildroot}%{_libdir}/.*.la$

%global _sysconfdir %{_prefix}%{_sysconfdir}

%global _localstatedir %{_prefix}%{_localstatedir}

%debug_package %{nil}

#mv %{buildroot}/sbin/mount.ceph %{buildroot}/usr/sbin/mount.ceph

#mv %{buildroot}/sbin/mount.fuse.ceph %{buildroot}/usr/sbin/mount.fuse.ceph

mv %{buildroot}/sbin/mount.ceph %{buildroot}%{_prefix}/sbin/mount.ceph

mv %{buildroot}/sbin/mount.fuse.ceph %{buildroot}%{_prefix}/sbin/mount.fuse.ceph

mv %{buildroot}/usr/sbin/ceph-disk %{buildroot}%{_prefix}/sbin/ceph-disk

mv %{buildroot}/usr/bin/ceph-detect-init %{buildroot}%{_prefix}/bin/ceph-detect-init

rpmbuild –ba ceph.spec

用t-abs打包:

t-abs ceph.spec

sudo rpm -ivh librados2-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh libradosstriper1-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh python-rados-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh ceph-radosgw-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh ceph-base-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh ceph-common-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh librbd1-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh rbd-fuse-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh rbd-mirror-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh rbd-nbd-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh python-rbd-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh librgw2-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh libcephfs1-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh ceph-fuse-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh python-cephfs-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh ceph-libs-compat-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh python-ceph-compat-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh ceph-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh ceph-osd-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh ceph-mds-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh ceph-mon-10.2.2-0.rhel7.x86_64.rpm

#devel相关的包,可以不安装

sudo rpm -ivh librados2-devel-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh librgw2-devel-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh libradosstriper1-devel-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh librbd1-devel-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh libcephfs1-devel-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh ceph-devel-compat-10.2.2-0.rhel7.x86_64.rpm

#测试,debug以及java相关的包,可以不安装

sudo rpm -ivh ceph-test-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh ceph-debuginfo-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh libcephfs_jni1-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh libcephfs_jni1-devel-10.2.2-0.rhel7.x86_64.rpm

sudo rpm -ivh cephfs-java-10.2.2-0.rhel7.x86_64.rpm

1.4测试

在本地起ceph集群进行测试:

#起动vstart

./vstart.sh –d –n –x –l

./bin/ceph –s

如果本地文件系统是ext4,需要修改以下值:

filestore_max_xattr_value_size_other = 4086

osd max object name len = 1024

osd max object namespace len = 256

起动osd节点:

init-ceph start osd.*

本地做rados测试:

$rados -p rbd bench 30 write

Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 30 seconds or 0 objects

Object prefix: benchmark_data_lvjian100081200005.et15sqa_44248

sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)

0 0 0 0 0 0 - 0

1 16 38 22 87.9956 88 0.233292 0.179826

2 16 38 22 43.9967 0 - 0.179826

3 16 38 22 29.3311 0 - 0.179826

4 16 38 22 21.9984 0 - 0.179826

5 16 38 22 17.5987 0 - 0.179826

6 16 82 66 43.9966 35.2 0.136914 1.34638

7 16 82 66 37.7113 0 - 1.34638

8 16 82 66 32.9974 0 - 1.34638

9 16 82 66 29.331 0 - 1.34638

10 16 82 66 26.3979 0 - 1.34638

11 16 113 97 35.2699 24.8 0.16824 1.77199

12 16 113 97 32.3307 0 - 1.77199

13 16 113 97 29.8437 0 - 1.77199

14 16 113 97 27.712 0 - 1.77199

15 16 113 97 25.8646 0 - 1.77199

16 16 137 121 30.2476 19.2 0.0883345 2.09566

17 16 138 122 28.7035 4 0.164464 2.07983

18 16 138 122 27.1089 0 - 2.07983

19 16 138 122 25.6821 0 - 2.07983

2016-10-18 14:04:01.045958 min lat: 0.0330393 max lat: 5.18978 avg lat: 2.07983

sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s)

20 16 138 122 24.398 0 - 2.07983

21 16 149 133 25.3312 11 5.10087 2.3338

22 16 172 156 28.3612 92 0.181053 2.16421

23 16 172 156 27.1281 0 - 2.16421

24 16 172 156 25.9978 0 - 2.16421

25 16 172 156 24.9579 0 - 2.16421

26 16 172 156 23.9979 0 - 2.16421

27 16 203 187 27.7013 24.8 0.133892 2.24739

28 16 203 187 26.712 0 - 2.24739

29 16 203 187 25.7909 0 - 2.24739

30 16 203 187 24.9312 0 - 2.24739

31 16 203 187 24.127 0 - 2.24739

Total time run: 31.372199

Total writes made: 204

Write size: 4194304

Object size: 4194304

Bandwidth (MB/sec): 26.0103

Stddev Bandwidth: 23.3087

Max bandwidth (MB/sec): 92

Min bandwidth (MB/sec): 0

Average IOPS: 6

Stddev IOPS: 5

Max IOPS: 23

Min IOPS: 0

Average Latency(s): 2.46023

Stddev Latency(s): 2.47093

Max latency(s): 5.30316

Min latency(s): 0.0224793

创建rbd:

[jianshu.ljs@lvjian100081200005.et15sqa /home/jianshu.ljs/ceph_10.2.2/src]

$rbd create rbd_test_image_01 --size 1000

[jianshu.ljs@lvjian100081200005.et15sqa /home/jianshu.ljs/ceph_10.2.2/src]

$rbd ls

rbd_test_image_01

[jianshu.ljs@lvjian100081200005.et15sqa /home/jianshu.ljs/ceph_10.2.2/src]

$rbd info rbd_test_image_01

rbd image 'rbd_test_image_01':

size 1000 MB in 250 objects

order 22 (4096 kB objects)

block_name_prefix: rbd_data.102e74b0dc51

format: 2

features: layering, exclusive-lock, object-map, fast-diff, deep-flatten

flags:

来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/30088583/viewspace-2136827/,如需转载,请注明出处,否则将追究法律责任。

请登录后发表评论 登录
全部评论
资深Oracle RAC及Exadata部署实施运维专家,曾从事PostgreSQL内核开发与性能优化。丰富的大规模数据库管理架构设计经验。目前在阿里巴巴从事数据库架构工作以及新技术引进,对分布式存储,云计算及并行编程有一定研究。技术讨论请加微信:ljs521688

注册时间:2015-01-01

  • 博文量
    41
  • 访问量
    557959