• Stars
    star
    131
  • Rank 275,867 (Top 6 %)
  • Language
    PHP
  • Created over 6 years ago
  • Updated over 6 years ago

Reviews

There are no reviews yet. Be the first to send feedback to the community and the maintainers!

Repository Details

基于lumen5.5开发的高性能高并发图片识别平台API源码

注意

如果重新开发这个项目,我可能会选择swoole,或者go之类的,所以参考参考就好了吧

基于LUMEN开发的高性能图片识别平台API接口源码

此项目是整个AI图片 识别项目 的API接口端

原本是作为商业项目开发的,但项目未上线就宣告结束了,出于共同学习的目的,开源此项目

因为项目并未真正上线运营,所以有一些不足或考虑不周全之处请自行忽略,若发现错误,还请指正。

用户管理系统&业务官网源码请移步至 https://github.com/hookover/aiocr-web.git

根据预期业务,将接口设计为:承载每天2000万张图片上传并相应处理的API

API接口 很少

/servers | 获取服务器列表

/user/login | 登录 /user/point?token=:token | 获取用户剩余积分

/decode/upload-base64?token=:token | 以base64字符串方式上传图片 /decode/upload?token=:token | 以form-data数据流方式上传图片

/result?token=:token | 获取识别结果 /report?token=:token | 若图片识别错误,可报错

接口的详细使用,请搭建https://github.com/hookover/aiocr-web.git 后点击API文档查看

技术目标

高并发、高性能、安全稳定、最佳实践方案

用到的技术栈

尽量RESTful规范

JWT,加密算法修改为了最快的SHA256

REDIS长连接,测试了memcached

高性能自增ID生成器:php_snowflke (测试每秒生成10万+不重复ID)

性能分析工具使用的是xhprof

APIDOC 根据代码注释自动生成API文档

MYSQL事件+存储过程 将上传图片日志表自动按天分区,并删除7天之前的分区

MYSQL事件+存储过程 自动统计每个用户/开发工程师每天产生的数据

LUMEN 层面

自定义Auth,将token验证放到了Auth服务模块中
定义了日志服务
统一的API响应错误代码机制
自增ID采用2个数据字段存储,并使用联合索引提升性能

   SQL语句优化    还有一些细节已经忘记了

安装

1、 为php安装此扩展 https://github.com/hookover/php_snowflake.git
2、 克隆此代码并composer安装代码依赖
3、 配置nginx/apache 参考lumen的配置
4、 cp .env.example .env
5、 项目数据表的创建和基础数据在这里: https://github.com/hookover/aiocr-web.git,建议先搭建业务系统再搭建API

代码层面优化

借助xhprof查看性能瓶颈,根据性能瓶颈提出解决方案,并实施,最终实现单个API请求处理在10ms左右,数据并发起来,业务逻辑也不会出现任何问题。

高并发是一个系统工程

个人认为,要想让整个系统实现高并发,需要从软件到硬件再到服务器架构配置做好全面协调,任何一个地方的短板都可能成为整个系统的短板。

代码层面无非就是减少对数据库的读写压力、将一些频繁数据放入缓存,PHP到缓存到数据库都通过长连接提高连接速度,数据库表合理的使用好索引,注意SQL质量和调整好一些耗费资源的算法。

服务器软件配置方面又会涉及到LINUX本身的优化,服务软件之间的协调,数据库之间的协调,负载均衡等。

硬件层面又需要考虑CPU瓶颈,硬盘读写性能,内存大小,服务器间网络通信状况等..

简单压测

开发机: Intel® Core™ i7-4770 CPU @ 3.40GHz × 8 + 15.6 GiB 内存

环境:ubuntu + nginx + php7 + php-fpm tcp

压测工具: ab

开发机上同时运行了phpstorm,网页等

nginx和php-fpm是使用的apt安装后的默认配置,未做深入调优

若需要下载代码自己压测,或者真正到线上运行, 还需要调整nginx、php、php-fpm、mysql、redis的配置,以便让这些服务器软件协同工作。

测试数据:

100并发1万次

Server Software:        nginx/1.12.2
Server Hostname:        api.decaptcha.com
Server Port:            80

Document Path:          /v2/decode/upload-base64?file=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAMAAAARCA0dGVyY4LQCQAAABFXKwAAAABJRU5ErkJggg==&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpZCI6MTk4NjkxOCwiZXhwIjoxNTIxNzAwMDY0fQ.tFbzUwqI5VU1oENTJWOqYLNGb5bg4Yh-L7rsIvRDe2M
Document Length:        82 bytes

Concurrency Level:      100
Time taken for tests:   10.254 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      2450000 bytes
HTML transferred:       820000 bytes
Requests per second:    975.25 [#/sec] (mean)
Time per request:       102.537 [ms] (mean)
Time per request:       1.025 [ms] (mean, across all concurrent requests)
Transfer rate:          233.34 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       4
Processing:     6  102 267.1     31    4582
Waiting:        6  102 267.1     31    4582
Total:          6  102 267.1     31    4583

200并发10万次

Server Software:        nginx/1.12.2
Server Hostname:        api.decaptcha.com
Server Port:            80

Document Path:          /v2/decode/upload-base64?file=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAMAAAARCA0dGVyY4LQCQAAABFXKwAAAABJRU5ErkJggg==&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpZCI6MTk4NjkxOCwiZXhwIjoxNTIxNzAwMDY0fQ.tFbzUwqI5VU1oENTJWOqYLNGb5bg4Yh-L7rsIvRDe2M
Document Length:        82 bytes

Concurrency Level:      200
Time taken for tests:   104.265 seconds
Complete requests:      100000
Failed requests:        17
   (Connect: 0, Receive: 0, Length: 17, Exceptions: 0)
Non-2xx responses:      17
Total transferred:      24501853 bytes
HTML transferred:       8201717 bytes
Requests per second:    959.09 [#/sec] (mean)
Time per request:       208.531 [ms] (mean)
Time per request:       1.043 [ms] (mean, across all concurrent requests)
Transfer rate:          229.49 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.2      0       3
Processing:     6  208 1630.6     43   60001
Waiting:        6  208 1630.6     43   60001
Total:          6  208 1630.6     44   60001

500并发,10万次

Server Software:        nginx/1.12.2
Server Hostname:        api.decaptcha.com
Server Port:            80

Document Path:          /v2/decode/upload-base64?file=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAMAAAARCA0dGVyY4LQCQAAABFXKwAAAABJRU5ErkJggg==&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpZCI6MTk4NjkxOCwiZXhwIjoxNTIxNzAwMDY0fQ.tFbzUwqI5VU1oENTJWOqYLNGb5bg4Yh-L7rsIvRDe2M
Document Length:        82 bytes

Concurrency Level:      500
Time taken for tests:   106.200 seconds
Complete requests:      100000
Failed requests:        114
   (Connect: 0, Receive: 0, Length: 114, Exceptions: 0)
Non-2xx responses:      114
Total transferred:      24511451 bytes
HTML transferred:       8210864 bytes
Requests per second:    941.62 [#/sec] (mean)
Time per request:       531.001 [ms] (mean)
Time per request:       1.062 [ms] (mean, across all concurrent requests)
Transfer rate:          225.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.7      0      14
Processing:     5  530 3091.5    151   63039
Waiting:        5  530 3091.5    151   63039
Total:          5  530 3091.9    151   63049

1000并发1万次

Server Software:        nginx/1.12.2
Server Hostname:        api.decaptcha.com
Server Port:            80

Document Path:          /v2/decode/upload-base64?file=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAMAAAARCA0dGVyY4LQCQAAABFXKwAAAABJRU5ErkJggg==&token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpZCI6MTk4NjkxOCwiZXhwIjoxNTIxNzAwMDY0fQ.tFbzUwqI5VU1oENTJWOqYLNGb5bg4Yh-L7rsIvRDe2M
Document Length:        82 bytes

Concurrency Level:      1000
Time taken for tests:   27.926 seconds
Complete requests:      10000
Failed requests:        64
   (Connect: 0, Receive: 0, Length: 64, Exceptions: 0)
Non-2xx responses:      64
Total transferred:      2456016 bytes
HTML transferred:       825824 bytes
Requests per second:    358.09 [#/sec] (mean)
Time per request:       2792.559 [ms] (mean)
Time per request:       2.793 [ms] (mean, across all concurrent requests)
Transfer rate:          85.89 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    2   6.7      0      27
Processing:    62 1170 3005.8    256   27883
Waiting:       62 1170 3005.8    256   27883
Total:         62 1172 3008.8    256   27906