swift分布式存储性能测试以及破坏性测试

2023-07-31 原文

延续上一个文章的做接下来的测试，环境是一个proxy和三个storage。

1、性能测试

测试一：五个线程，每个线程上传一个大小差不多一个G的文件

/data/swift/upload> du -sh test*.zip

987M test1.zip

987M test2.zip

987M test3.zip

987M test4.zip

987M test.zip

上传

swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container1 test.zip

swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container2 test1.zip

swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container3 test2.zip

swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container4 test3.zip

swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container5 test4.zip

共用时8:30:45

存储节点查看文件

/srv/node/sdb1> du -sh *

16K accounts

356M async_pending

129M containers

4.9G objects

0 tmp

查看上传的文件

/data/swift/upload> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container1

test.zip

/data/swift/upload> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container2

test1.zip

/data/swift/upload> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container3

test2.zip

/data/swift/upload> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container4

test3.zip

/data/swift/upload> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container5

test4.zip

测试二，十个线程：

/data/swift/upload/test100M> du -sh *

505M test10.zip

505M test1.zip

505M test2.zip

505M test3.zip

505M test4.zip

505M test5.zip

505M test6.zip

505M test7.zip

505M test8.zip

505M test9.zip

单个线程测试上传2.6M/s,下载5.1M/s

五线程测试：

上传

第一个用时03:20s,2.5M/s

第二个用时04:30s,1.85M/s

第三个用时05:50s,1.31M/s

第四个用时06:55s,1.2M/s

第五个用时07:00s,1.19M/s

下载

第一个用时30.263s,17.461 MB/s

第二个用时49.415s,10.691 MB/s

第三个用时54.724s,9.653 MB/s

第四个用时86.473s,6.108 MB/s

第五个用时97.919s,5.417 MB/s

十线程测试：

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container1 test1.zip

test1.zip

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container2 test2.zip

Object PUT Failed: http://192.168.1.211:8090:8090/v1/AUTH_system/container2/test2.zip 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource Could not be found.<

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container3 test3.zip

test3.zip

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container4 test4.zip

Object PUT Failed: http://192.168.1.211:8090:8090/v1/AUTH_system/container4/test4.zip 503 Internal Server Error [first 60 chars of response] <html><h1>Service Unavailable</h1><p>The server is currently

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container5 test5.zip

Object PUT Failed: http://192.168.1.211:8090:8090/v1/AUTH_system/container5/test5.zip 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource Could not be found.<

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container6 test6.zip

Object PUT Failed: http://192.168.1.211:8090:8090/v1/AUTH_system/container6/test6.zip 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource Could not be found.<

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container7 test7.zip

Object PUT Failed: http://192.168.1.211:8090:8090/v1/AUTH_system/container7/test7.zip 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource Could not be found.<

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container8 test8.zip

Object PUT Failed: http://192.168.1.211:8090:8090/v1/AUTH_system/container8/test8.zip 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource Could not be found.<

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container9 test9.zip

Object PUT Failed: http://192.168.1.211:8090:8090/v1/AUTH_system/container9/test9.zip 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource Could not be found.<

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container10 test10.zip

Object PUT Failed: http://192.168.1.211:8090:8090/v1/AUTH_system/container10/test10.zip 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource Could not be found.<

但是只有七个上传成功

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container1

test1.zip

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container2

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container3

test3.zip

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container4

test4.zip

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container5

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container6

test6.zip

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container7

test7.zip

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container8

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container9

test9.zip

/data/swift/upload/test100M> swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass list container10

test10.zip

上传

第一个用时00:39s,12M/s

第二个用时01:01s,8.19M/s

第三个用时03:37s,2.3M/s

第四个用时06:16s,1.32M/s

第五个用时06:16s,1.32M/s

第六个用时07:06s,1.17M/s

第七个用时07:06s,1.17M/s

下载

第一个用时73.207s,7.216 MB/s

第二个用时107.437s,4.916 MB/s

第三个用时107.581s,4.910 MB/s

第四个用时108.457s,4.870 MB/s

第五个用时117.852s,4.482 MB/s

第六个用时121.565s,4.345 MB/s

第七个用时122.600s,4.308 MB/s

第八个用时122.413s,4.315 MB/s

第九个用时145.383s,3.633 MB/s

第十个用时174.677s,3.024 MB/s

2、破坏性测试

测试一：测试关闭一个存储节点的服务之后上传文件会不会在开启该存储服务后同步

查看一个存储节点的服务

/srv/node/sdb1> ps -ef | grep swift

swift 5651 10735 5 17:16 ? 00:01:10 /usr/bin/python /usr/bin/swift-object-updater /etc/swift/object-server.conf

swift 8409 10727 3 17:38 ? 00:00:01 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf

was_wcm 8477 12319 0 17:38 pts/0 00:00:00 grep swift

swift 10723 1 0 Mar28 ? 00:00:50 /usr/bin/python /usr/bin/swift-container-updater /etc/swift/container-server.conf

swift 10724 1 0 Mar28 ? 00:00:05 /usr/bin/python /usr/bin/swift-account-auditor /etc/swift/account-server.conf

swift 10725 1 0 Mar28 ? 03:18:56 /usr/bin/python /usr/bin/swift-object-replicator /etc/swift/object-server.conf

swift 10726 1 1 Mar28 ? 04:23:53 /usr/bin/python /usr/bin/swift-container-replicator /etc/swift/container-server.conf

swift 10727 1 0 Mar28 ? 00:08:35 /usr/bin/python /usr/bin/swift-object-auditor /etc/swift/object-server.conf

swift 10728 1 0 Mar28 ? 00:01:47 /usr/bin/python /usr/bin/swift-object-expirer /etc/swift/object-expirer.conf

swift 10729 1 0 Mar28 ? 00:00:04 /usr/bin/python /usr/bin/swift-container-auditor /etc/swift/container-server.conf

swift 10730 1 0 Mar28 ? 00:00:00 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf

swift 10731 1 0 Mar28 ? 00:00:00 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf

swift 10732 1 0 Mar28 ? 00:00:02 /usr/bin/python /usr/bin/swift-account-reaper /etc/swift/account-server.conf

swift 10733 1 0 Mar28 ? 00:00:24 /usr/bin/python /usr/bin/swift-container-sync /etc/swift/container-server.conf

swift 10734 1 0 Mar28 ? 00:06:13 /usr/bin/python /usr/bin/swift-account-replicator /etc/swift/account-server.conf

swift 10735 1 0 Mar28 ? 00:00:06 /usr/bin/python /usr/bin/swift-object-updater /etc/swift/object-server.conf

swift 10736 1 0 Mar28 ? 00:00:00 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf

swift 10795 10730 0 Mar28 ? 00:18:29 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf

swift 10796 10730 0 Mar28 ? 00:17:56 /usr/bin/python /usr/bin/swift-container-server /etc/swift/container-server.conf

swift 10797 10736 0 Mar28 ? 02:58:11 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf

swift 10798 10736 0 Mar28 ? 03:16:19 /usr/bin/python /usr/bin/swift-object-server /etc/swift/object-server.conf

swift 10799 10731 0 Mar28 ? 00:04:15 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf

swift 10800 10731 0 Mar28 ? 00:04:30 /usr/bin/python /usr/bin/swift-account-server /etc/swift/account-server.conf

杀死服务

# kill -9 `ps -ef | grep -v grep | grep object-server | awk '{print $2}'`

上传文件

swift -Ahttp://192.168.1.211:8090/auth/v1.0 -U system:root -K testpass upload container1 apache_install

启动服务之前的三个存储文件对比

swift@server2:/srv/node/sdb1/objects> ll | wc -l

swift@server3:/srv/node/sdb1/objects> ll | wc -l

swift@server4:/srv/node/sdb1/objects> ll | wc -l

启动服务

swift@server2:/srv/node/sdb1/objects> sudo swift-init all restart

swift@server2:/srv/node/sdb1/objects> ll | wc -l

可以看到文件已经同步

测试二：测试删除一个存储节点物理服务器上的文件会不会在之后同步其他存储节点上的文件

删除一个存储节点的文件

swift@server2:/srv/node/sdb1/objects> rm -rf *

查看文件的同步变化