Are you sure you want to delete this task? Once this task is deleted, it cannot be recovered.
Xin Yan 03f75ffd89 | 1 year ago | |
---|---|---|
Goods_Recognition.ipynb | 1 year ago | |
LICENSE | 1 year ago | |
README.md | 1 year ago |
近几年, 随着"新零售"概念的火热, 越来越多的品牌商及商超零售企业开始引进人工智能技术, 探索商品管理、成本控制、用户体验等多维度的数字化转型, 转型中所涉及的货架陈列分析、智能结算、智能库存管理、智能货柜、电商平台等以图搜图的场景背后的核心技术都离不开商品识别算法.
痛点问题
解决方案
PaddleClas团队开源的图像识别PP-ShiTu技术方案, 主要由主体检测、特征学习和向量检索三个模块组成, 是一个实用的轻量级通用图像识别系统. 基于此技术方案, 商超零售企业可实现大量商品的一键式智能化识别, 大大提高识别效率, 节省人工及时间成本.
此外, 当新品迭代更新时, PP-shitu无需重新训练模型, 能够做到"即增即用", 完美解决上述痛点问题, 大大提高了人工智能在商超零售行业的应用落地可能性.
PP-shitu技术方案可具体应用于例如: 商品结算、库存管理等关于商品识别的商超细分场景.
模型工具简介
飞桨图像识别套件PaddleClas是飞桨为工业界和学术界所准备的一个图像识别任务的工具集, 助力使用者训练出更好的视觉模型和应用落地.
而PP-ShiTu是一个实用的轻量级通用图像识别系统, 主要由主体检测、特征学习和向量检索三个模块组成. 该系统从骨干网络选择和调整、损失函数的选择、数据增强、学习率变换策略、正则化参数选择、预训练模型使用以及模型裁剪量化8个方面, 采用多种策略, 对各个模块的模型进行优化, 最终得到在CPU上仅0.2s即可完成10w+库的图像识别的系统.
环境要求
Python >= 3.6
PaddlePaddle >= 2.1
Linux 环境最佳
安装PaddleClas
# 克隆 PaddleClas
# 由于本项目已克隆并存储,故不重复克隆
## github仓库
#!git clone https://github.com/PaddlePaddle/PaddleClas.git -b release/2.3
## gitee仓库(推荐)
#!git clone https://gitee.com/paddlepaddle/PaddleClas.git -b release/2.3
# 安装 Python 依赖库
# PaddleClas 的 Python 依赖库在 `requirements.txt` 中给出,可通过如下命令安装:
%cd /code/PaddleClas/
!pip install --upgrade -r requirements.txt -i https://mirror.baidu.com/pypi/simple
数据集介绍
商品部分
东古酱油一品鲜
东古黄豆酱750G
东鹏特饮罐装
中华(硬)
中华(软)
乳酸菌600亿_2
乳酸菌600亿_3
乳酸菌600亿原味
乳酸菌600亿芒果
乳酸菌600亿芦荟
...
数据集格式
txt
文件指定, 训练数据集 train_list.txt
文件内容格式如下所示:# 采用"空格"作为分隔符号
...
train/10/1283.jpg 10 624
train/10/1284.jpg 10 625
train/10/1285.jpg 10 626
train/10/1286.jpg 10 627
...
验证数据集(本数据集中既是 gallery dataset, 也是 query dataset)test_list.txt 文件内容格式如下所示:
...
test/103/743.jpg 103 743
test/103/744.jpg 103 744
test/103/745.jpg 103 745
test/103/746.jpg 103 746
...
注:
├── classlabel.txt # 数据label和对应商品名称
├── gallery_label.txt # 图片地址及对应的商品名称
├── gallery # 底库图片
├── test # 测试集图片
├── test_list.txt # 测试集图片地址及对应的商品名称
├── train # 训练集图片
└── train_list.txt # 训练集图片地址及对应的商品名称
PP-ShiTu是一个实用的轻量级通用图像识别系统, 主要由主体检测、特征学习和向量检索三个模块组成. 该系统从骨干网络选择和调整、损失函数的选择、数据增强、学习率变换策略、正则化参数选择、预训练模型使用以及模型裁剪量化8个方面, 采用多种策略, 对各个模块的模型进行优化, 最终得到在CPU上仅0.2s即可完成10w+库的图像识别的系统.
主体检测
主体检测技术是目前应用非常广泛的一种检测技术, 它指的是检测出图片中一个或者多个主体的坐标位置, 然后将图像中的对应区域裁剪下来, 进行识别, 从而完成整个识别过程. 主体检测是识别任务的前序步骤, 可以有效提升识别精度.
考虑到商品识别实际应用场景中, 需要快速准确地获得识别结果, 故本项目选取适用于 CPU 或者移动端场景的轻量级主体检测模型PicoDet作为本项目主体检测部分的模型. 此模型融合了ATSS、Generalized Focal Loss、余弦学习率策略、Cycle-EMA、轻量级检测 head等一系列优化算法, 基于COCO train2017数据集进行大规模预训练, 最终inference模型大小(MB)仅30.1MB, mAP可达40.1%, 在cpu下单张图片预测耗时仅29.8ms, 完美符合本项目实际落地需求, 故在本项目中不对主体检测部分做适应性训练.
特征提取
特征提取是图像识别中的关键一环, 它的作用是将输入的图片转化为固定维度的特征向量, 用于后续的向量检索. 好的特征需要具备相似度保持性, 即在特征空间中, 相似度高的图片对其特征相似度要比较高(距离比较近), 相似度低的图片对, 其特征相似度要比较小(距离比较远). Deep Metric Learning用以研究如何通过深度学习的方法获得具有强表征能力的特征.
考虑到本项目的真实落地的场景中, 推理速度及预测准确率是考量模型好坏的重要指标, 所以本项目采用 PP_LCNet_x2_5 作为骨干网络, Neck 部分选用 Linear Layer, Head 部分选用 ArcMargin, Loss 部分选用 CELoss, 并结合度量学习arcmargin算法, 对高相似物体的区分效果远超单一模型, 能更好地适应 Intel CPU, 不仅准确率超越大模型ResNet50, 预测速度还能快3倍.
向量检索
向量检索技术在图像识别、图像检索中应用比较广泛. 其主要目标是, 对于给定的查询向量, 在已经建立好的向量库中, 与库中所有的待查询向量, 进行特征向量的相似度或距离计算, 得到相似度排序. 在图像识别系统中, 本项目使用 Faiss 对此部分进行支持. 在此过程中, 本项目选取 HNSW32 为检索算法, 使得检索精度、检索速度能够取得较好的平衡, 更为贴切本项目实际应用场景的使用需求.
本项目在主体检测部分直接采用PaddleDetection提供给得预训练模型, 故这里主要介绍特征提取部分的模型训练.
修改配置文件
首先, 在启动模型训练之前, 需要在配置文件中修改数据配置相关的内容, 主要包括数据集的地址以及类别数量. 对应到配置文件中的位置如下所示:
Head:
name: ArcMargin
embedding_size: 512
class_num: 358 #此处表示类别数
Train:
dataset:
name: ImageNetDataset
image_root: /dataset/ #此处表示train数据所在的目录
cls_label_path: /dataset/train_list.txt #此处表示train数据集label文件的地址
Query:
dataset:
name: VeriWild
image_root: /dataset/ #此处表示query数据集所在的目录
cls_label_path: /dataset/test_list.txt #此处表示query数据集label文件的地址
Gallery:
dataset:
name: VeriWild
image_root: /dataset/ #此处表示gallery数据集所在的目录
cls_label_path: /dataset/test_list.txt #此处表示gallery数据集label文件的地址
模型训练
%cd /code/PaddleClas
!python tools/train.py \
-c ./ppcls/configs/GeneralRecognition/GeneralRecognition_PPLCNet_x2_5.yaml \
-o Arch.Backbone.pretrained=True \
-o Global.device=gpu \
%cd /code/PaddleClas
!python tools/eval.py \
-c ./ppcls/configs/GeneralRecognition/GeneralRecognition_PPLCNet_x2_5.yaml \
-o Global.pretrained_model="/code/PaddleClas/output/RecModel/best_model"
由评估log可见, recall1为0.98232, 能够符合实际产业场景应用需求.
推理过程包括两个步骤: 1)导出推理模型, 2)获取特征向量
导出推理模型
PaddlePaddle框架保存的权重文件分为两种: 支持前向推理和反向梯度的训练模型 和 只支持前向推理的推理模型. 二者的区别是推理模型针对推理速度和显存做了优化, 裁剪了一些只在训练过程中才需要的tensor, 降低显存占用, 并进行了一些类似层融合, kernel选择的速度优化. 因此可执行如下命令导出推理模型.
%cd /code/PaddleClas
!python tools/export_model.py \
-c ./ppcls/configs/GeneralRecognition/GeneralRecognition_PPLCNet_x2_5.yaml \
-o Global.pretrained_model="output/RecModel/best_model"
生成的推理模型位于 inference 目录, 里面包含三个文件, 分别为 inference.pdmodel、inference.pdiparams、inference.pdiparams.info. 其中: inference.pdmodel 用来存储推理模型的结构, inference.pdiparams 和 inference.pdiparams.info 用来存储推理模型相关的参数信息.
获取特征向量
%cd /code/PaddleClas/deploy
!python python/predict_rec.py \
-c configs/inference_rec.yaml \
-o Global.rec_inference_model_dir="../inference" \
-o Global.infer_imgs="/dataset/test_image/sijibao.jpg"
这里串联主体检测、特征提取、向量检索, 从而构成一整套图像识别系统:
configs/build_general.yaml
文件内容:Global:
rec_inference_model_dir: "/code/PaddleClas/inference"
IndexProcess:
index_method: "HNSW32" # supported: HNSW32, IVF, Flat
image_root: "/code/dataset/"
index_dir: "/code/index"
data_file: "/dataset/gallery_label.txt"
执行如下代码:
# 建立索引库
%cd /code/PaddleClas/deploy
!python3 python/build_gallery.py \
-c configs/build_general.yaml \
-o IndexProcess.data_file="/dataset/gallery_label.txt" \
-o IndexProcess.index_dir="/code/index"
inference
模型并解压:%cd /code/deploy/
%mkdir models
%cd models
# 下载通用检测 inference 模型并解压
!wget https://paddle-imagenet-models-name.bj.bcebos.com/dygraph/rec/models/inference/picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer.tar && tar -xf picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer.tar
修改推理文件 configs/inference_general.yaml
内容:
Global:
infer_imgs: "/dataset/test_image/sijibao.jpg"
det_inference_model_dir: "./models/picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer"
rec_inference_model_dir: "/code/PaddleClas/inference"
rec_nms_thresold: 0.05
IndexProcess:
index_dir: "/code/index"
return_k: 5
score_thres: 0.5
运行下面的命令, 对图像 /dataset/test_image/sijibao.jpg
进行识别与检索:
#基于索引库的图像识别
%cd /code/PaddleClas/deploy
!python python/predict_system.py \
-c configs/inference_general.yaml \
-o Global.infer_imgs="/dataset/test_image/sijibao.jpg" \
-o IndexProcess.index_dir="/code/index"
/code/PaddleClas/deploy
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
'nearest': Image.NEAREST,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
'bilinear': Image.BILINEAR,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
'bicubic': Image.BICUBIC,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:39: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
'box': Image.BOX,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:40: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
'lanczos': Image.LANCZOS,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:41: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
'hamming': Image.HAMMING
2022-04-07 16:40:27 INFO:
===========================================================
== PaddleClas is powered by PaddlePaddle ! ==
===========================================================
== ==
== For more info please go to the following website. ==
== ==
== https://github.com/PaddlePaddle/PaddleClas ==
===========================================================
2022-04-07 16:40:27 INFO: DetPostProcess :
2022-04-07 16:40:27 INFO: DetPreProcess :
2022-04-07 16:40:27 INFO: transform_ops :
2022-04-07 16:40:27 INFO: DetResize :
2022-04-07 16:40:27 INFO: interp : 2
2022-04-07 16:40:27 INFO: keep_ratio : False
2022-04-07 16:40:27 INFO: target_size : [640, 640]
2022-04-07 16:40:27 INFO: DetNormalizeImage :
2022-04-07 16:40:27 INFO: is_scale : True
2022-04-07 16:40:27 INFO: mean : [0.485, 0.456, 0.406]
2022-04-07 16:40:27 INFO: std : [0.229, 0.224, 0.225]
2022-04-07 16:40:27 INFO: DetPermute :
2022-04-07 16:40:27 INFO: Global :
2022-04-07 16:40:27 INFO: batch_size : 1
2022-04-07 16:40:27 INFO: cpu_num_threads : 10
2022-04-07 16:40:27 INFO: det_inference_model_dir : ./models/picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer
2022-04-07 16:40:27 INFO: enable_benchmark : True
2022-04-07 16:40:27 INFO: enable_mkldnn : True
2022-04-07 16:40:27 INFO: enable_profile : False
2022-04-07 16:40:27 INFO: gpu_mem : 8000
2022-04-07 16:40:27 INFO: image_shape : [3, 640, 640]
2022-04-07 16:40:27 INFO: infer_imgs : /dataset/test_image/sijibao.jpg
2022-04-07 16:40:27 INFO: ir_optim : True
2022-04-07 16:40:27 INFO: labe_list : ['foreground']
2022-04-07 16:40:27 INFO: max_det_results : 5
2022-04-07 16:40:27 INFO: rec_inference_model_dir : /code/PaddleClas/inference
2022-04-07 16:40:27 INFO: rec_nms_thresold : 0.05
2022-04-07 16:40:27 INFO: threshold : 0.2
2022-04-07 16:40:27 INFO: use_fp16 : False
2022-04-07 16:40:27 INFO: use_gpu : True
2022-04-07 16:40:27 INFO: use_tensorrt : False
2022-04-07 16:40:27 INFO: IndexProcess :
2022-04-07 16:40:27 INFO: batch_size : 32
2022-04-07 16:40:27 INFO: data_file : /dataset/gallery_label.txt
2022-04-07 16:40:27 INFO: delimiter :
2022-04-07 16:40:27 INFO: dist_type : IP
2022-04-07 16:40:27 INFO: embedding_size : 512
2022-04-07 16:40:27 INFO: image_root : /dataset/
2022-04-07 16:40:27 INFO: index_dir : /code/index
2022-04-07 16:40:27 INFO: index_method : HNSW32
2022-04-07 16:40:27 INFO: index_operation : new
2022-04-07 16:40:27 INFO: return_k : 5
2022-04-07 16:40:27 INFO: score_thres : 0.5
2022-04-07 16:40:27 INFO: RecPostProcess : None
2022-04-07 16:40:27 INFO: RecPreProcess :
2022-04-07 16:40:27 INFO: transform_ops :
2022-04-07 16:40:27 INFO: ResizeImage :
2022-04-07 16:40:27 INFO: size : 224
2022-04-07 16:40:27 INFO: NormalizeImage :
2022-04-07 16:40:27 INFO: mean : [0.485, 0.456, 0.406]
2022-04-07 16:40:27 INFO: order :
2022-04-07 16:40:27 INFO: scale : 0.00392157
2022-04-07 16:40:27 INFO: std : [0.229, 0.224, 0.225]
2022-04-07 16:40:27 INFO: ToCHWImage : None
/code/PaddleClas/deploy/python/preprocess.py:65: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
'nearest': Image.NEAREST,
/code/PaddleClas/deploy/python/preprocess.py:66: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
'bilinear': Image.BILINEAR,
/code/PaddleClas/deploy/python/preprocess.py:67: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
'bicubic': Image.BICUBIC,
/code/PaddleClas/deploy/python/preprocess.py:68: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
'box': Image.BOX,
/code/PaddleClas/deploy/python/preprocess.py:69: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
'lanczos': Image.LANCZOS,
/code/PaddleClas/deploy/python/preprocess.py:70: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
'hamming': Image.HAMMING
Inference: 34.667015075683594 ms per batch image
[{'bbox': [131, 114, 356, 355], 'rec_docs': '四季宝花生酱', 'rec_scores': 0.85414684}]
其中 bbox 表示检测出的主体所在位置, rec_docs 表示索引库中与检测框最为相似的类别, rec_scores 表示对应的置信度.
检测的可视化结果也保存在 output 文件夹下, 对于本张图像, 识别结果可视化如下所示:
/dataset/test_image/recognition_2.jpg
进行识别,运行如下识别命令:
%cd /code/PaddleClas/deploy
!python python/predict_system.py \
-c configs/inference_general.yaml \
-o Global.infer_imgs="/dataset/test_image/recognition_2.jpg"
/code/PaddleClas/deploy
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
'nearest': Image.NEAREST,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
'bilinear': Image.BILINEAR,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
'bicubic': Image.BICUBIC,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:39: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
'box': Image.BOX,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:40: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
'lanczos': Image.LANCZOS,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:41: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
'hamming': Image.HAMMING
2022-04-07 16:41:11 INFO:
===========================================================
== PaddleClas is powered by PaddlePaddle ! ==
===========================================================
== ==
== For more info please go to the following website. ==
== ==
== https://github.com/PaddlePaddle/PaddleClas ==
===========================================================
2022-04-07 16:41:11 INFO: DetPostProcess :
2022-04-07 16:41:11 INFO: DetPreProcess :
2022-04-07 16:41:11 INFO: transform_ops :
2022-04-07 16:41:11 INFO: DetResize :
2022-04-07 16:41:11 INFO: interp : 2
2022-04-07 16:41:11 INFO: keep_ratio : False
2022-04-07 16:41:11 INFO: target_size : [640, 640]
2022-04-07 16:41:11 INFO: DetNormalizeImage :
2022-04-07 16:41:11 INFO: is_scale : True
2022-04-07 16:41:11 INFO: mean : [0.485, 0.456, 0.406]
2022-04-07 16:41:11 INFO: std : [0.229, 0.224, 0.225]
2022-04-07 16:41:11 INFO: DetPermute :
2022-04-07 16:41:11 INFO: Global :
2022-04-07 16:41:11 INFO: batch_size : 1
2022-04-07 16:41:11 INFO: cpu_num_threads : 10
2022-04-07 16:41:11 INFO: det_inference_model_dir : ./models/picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer
2022-04-07 16:41:11 INFO: enable_benchmark : True
2022-04-07 16:41:11 INFO: enable_mkldnn : True
2022-04-07 16:41:11 INFO: enable_profile : False
2022-04-07 16:41:11 INFO: gpu_mem : 8000
2022-04-07 16:41:11 INFO: image_shape : [3, 640, 640]
2022-04-07 16:41:11 INFO: infer_imgs : /dataset/test_image/recognition_2.jpg
2022-04-07 16:41:11 INFO: ir_optim : True
2022-04-07 16:41:11 INFO: labe_list : ['foreground']
2022-04-07 16:41:11 INFO: max_det_results : 5
2022-04-07 16:41:11 INFO: rec_inference_model_dir : /code/PaddleClas/inference
2022-04-07 16:41:11 INFO: rec_nms_thresold : 0.05
2022-04-07 16:41:11 INFO: threshold : 0.2
2022-04-07 16:41:11 INFO: use_fp16 : False
2022-04-07 16:41:11 INFO: use_gpu : True
2022-04-07 16:41:11 INFO: use_tensorrt : False
2022-04-07 16:41:11 INFO: IndexProcess :
2022-04-07 16:41:11 INFO: batch_size : 32
2022-04-07 16:41:11 INFO: data_file : /dataset/gallery_label.txt
2022-04-07 16:41:11 INFO: delimiter :
2022-04-07 16:41:11 INFO: dist_type : IP
2022-04-07 16:41:11 INFO: embedding_size : 512
2022-04-07 16:41:11 INFO: image_root : /dataset/
2022-04-07 16:41:11 INFO: index_dir : /code/index
2022-04-07 16:41:11 INFO: index_method : HNSW32
2022-04-07 16:41:11 INFO: index_operation : new
2022-04-07 16:41:11 INFO: return_k : 5
2022-04-07 16:41:11 INFO: score_thres : 0.5
2022-04-07 16:41:11 INFO: RecPostProcess : None
2022-04-07 16:41:11 INFO: RecPreProcess :
2022-04-07 16:41:11 INFO: transform_ops :
2022-04-07 16:41:11 INFO: ResizeImage :
2022-04-07 16:41:11 INFO: size : 224
2022-04-07 16:41:11 INFO: NormalizeImage :
2022-04-07 16:41:11 INFO: mean : [0.485, 0.456, 0.406]
2022-04-07 16:41:11 INFO: order :
2022-04-07 16:41:11 INFO: scale : 0.00392157
2022-04-07 16:41:11 INFO: std : [0.229, 0.224, 0.225]
2022-04-07 16:41:11 INFO: ToCHWImage : None
/code/PaddleClas/deploy/python/preprocess.py:65: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
'nearest': Image.NEAREST,
/code/PaddleClas/deploy/python/preprocess.py:66: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
'bilinear': Image.BILINEAR,
/code/PaddleClas/deploy/python/preprocess.py:67: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
'bicubic': Image.BICUBIC,
/code/PaddleClas/deploy/python/preprocess.py:68: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
'box': Image.BOX,
/code/PaddleClas/deploy/python/preprocess.py:69: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
'lanczos': Image.LANCZOS,
/code/PaddleClas/deploy/python/preprocess.py:70: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
'hamming': Image.HAMMING
Inference: 34.21902656555176 ms per batch image
[]
输出结果为空.
由于默认的索引库中不包含对应的索引信息, 所以这里的识别结果有误, 此时我们可以通过构建新的索引库的方式, 完成未知类别的图像识别.
当索引库中的图像无法覆盖我们实际识别的场景时, 即在预测未知类别的图像时, 只需要将对应类别的相似图像添加到索引库中, 从而完成对未知类别的图像识别, 这一过程是不需要重新训练的.
准备新的数据与标签
首先需要将与待检索图像相似的图像列表拷贝到索引库原始图像的文件夹. 这里将所有的底库图像数据都放在文件夹 /dataset/gallery/ 中.
然后需要编辑记录了图像路径和标签信息的文本文件, 这里 PaddleClas 将更正后的标签信息文件放在了 /dataset/gallery_update.txt 文件中. 可以与原来的 /dataset/gallery_label.txt 标签文件进行对比, 添加了小度充电宝和韩国进口火山泥的索引图像.
每一行的文本中, 第一个字段表示图像的相对路径, 第二个字段表示图像对应的标签信息, 中间用 \t 键分隔开
建立新的索引库
使用下面的命令构建 index 索引, 加速识别后的检索过程.
%cd /code/PaddleClas/deploy/
!python python/build_gallery.py \
-c configs/build_general.yaml \
-o IndexProcess.data_file="/dataset/gallery_update.txt" \
-o IndexProcess.index_dir="/code/index_update"
最终新的索引信息保存在文件夹 /code/index_update
中.
使用新的索引库, 对上述图像进行识别, 运行命令如下:
%cd /code/PaddleClas/deploy/
!python python/predict_system.py \
-c configs/inference_general.yaml \
-o Global.infer_imgs="/dataset/test_image/recognition_2.jpg" \
-o IndexProcess.index_dir="/code/index_update"
/code/PaddleClas/deploy
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:36: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
'nearest': Image.NEAREST,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:37: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
'bilinear': Image.BILINEAR,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:38: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
'bicubic': Image.BICUBIC,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:39: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
'box': Image.BOX,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:40: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
'lanczos': Image.LANCZOS,
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/vision/transforms/functional_pil.py:41: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
'hamming': Image.HAMMING
2022-04-07 16:42:20 INFO:
===========================================================
== PaddleClas is powered by PaddlePaddle ! ==
===========================================================
== ==
== For more info please go to the following website. ==
== ==
== https://github.com/PaddlePaddle/PaddleClas ==
===========================================================
2022-04-07 16:42:20 INFO: DetPostProcess :
2022-04-07 16:42:20 INFO: DetPreProcess :
2022-04-07 16:42:20 INFO: transform_ops :
2022-04-07 16:42:20 INFO: DetResize :
2022-04-07 16:42:20 INFO: interp : 2
2022-04-07 16:42:20 INFO: keep_ratio : False
2022-04-07 16:42:20 INFO: target_size : [640, 640]
2022-04-07 16:42:20 INFO: DetNormalizeImage :
2022-04-07 16:42:20 INFO: is_scale : True
2022-04-07 16:42:20 INFO: mean : [0.485, 0.456, 0.406]
2022-04-07 16:42:20 INFO: std : [0.229, 0.224, 0.225]
2022-04-07 16:42:20 INFO: DetPermute :
2022-04-07 16:42:20 INFO: Global :
2022-04-07 16:42:20 INFO: batch_size : 1
2022-04-07 16:42:20 INFO: cpu_num_threads : 10
2022-04-07 16:42:20 INFO: det_inference_model_dir : ./models/picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer
2022-04-07 16:42:20 INFO: enable_benchmark : True
2022-04-07 16:42:20 INFO: enable_mkldnn : True
2022-04-07 16:42:20 INFO: enable_profile : False
2022-04-07 16:42:20 INFO: gpu_mem : 8000
2022-04-07 16:42:20 INFO: image_shape : [3, 640, 640]
2022-04-07 16:42:20 INFO: infer_imgs : /code/dataset/test_image/recognition_2.jpg
2022-04-07 16:42:20 INFO: ir_optim : True
2022-04-07 16:42:20 INFO: labe_list : ['foreground']
2022-04-07 16:42:20 INFO: max_det_results : 5
2022-04-07 16:42:20 INFO: rec_inference_model_dir : /code/PaddleClas/inference
2022-04-07 16:42:20 INFO: rec_nms_thresold : 0.05
2022-04-07 16:42:20 INFO: threshold : 0.2
2022-04-07 16:42:20 INFO: use_fp16 : False
2022-04-07 16:42:20 INFO: use_gpu : True
2022-04-07 16:42:20 INFO: use_tensorrt : False
2022-04-07 16:42:20 INFO: IndexProcess :
2022-04-07 16:42:20 INFO: batch_size : 32
2022-04-07 16:42:20 INFO: data_file : /code/dataset/gallery_label.txt
2022-04-07 16:42:20 INFO: delimiter :
2022-04-07 16:42:20 INFO: dist_type : IP
2022-04-07 16:42:20 INFO: embedding_size : 512
2022-04-07 16:42:20 INFO: image_root : /dataset/
2022-04-07 16:42:20 INFO: index_dir : /code/index_update
2022-04-07 16:42:20 INFO: index_method : HNSW32
2022-04-07 16:42:20 INFO: index_operation : new
2022-04-07 16:42:20 INFO: return_k : 5
2022-04-07 16:42:20 INFO: score_thres : 0.5
2022-04-07 16:42:20 INFO: RecPostProcess : None
2022-04-07 16:42:20 INFO: RecPreProcess :
2022-04-07 16:42:20 INFO: transform_ops :
2022-04-07 16:42:20 INFO: ResizeImage :
2022-04-07 16:42:20 INFO: size : 224
2022-04-07 16:42:20 INFO: NormalizeImage :
2022-04-07 16:42:20 INFO: mean : [0.485, 0.456, 0.406]
2022-04-07 16:42:20 INFO: order :
2022-04-07 16:42:20 INFO: scale : 0.00392157
2022-04-07 16:42:20 INFO: std : [0.229, 0.224, 0.225]
2022-04-07 16:42:20 INFO: ToCHWImage : None
/code/PaddleClas/deploy/python/preprocess.py:65: DeprecationWarning: NEAREST is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.NEAREST or Dither.NONE instead.
'nearest': Image.NEAREST,
/code/PaddleClas/deploy/python/preprocess.py:66: DeprecationWarning: BILINEAR is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BILINEAR instead.
'bilinear': Image.BILINEAR,
/code/PaddleClas/deploy/python/preprocess.py:67: DeprecationWarning: BICUBIC is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BICUBIC instead.
'bicubic': Image.BICUBIC,
/code/PaddleClas/deploy/python/preprocess.py:68: DeprecationWarning: BOX is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.BOX instead.
'box': Image.BOX,
/code/PaddleClas/deploy/python/preprocess.py:69: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
'lanczos': Image.LANCZOS,
/code/PaddleClas/deploy/python/preprocess.py:70: DeprecationWarning: HAMMING is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.HAMMING instead.
'hamming': Image.HAMMING
Inference: 36.23652458190918 ms per batch image
[{'bbox': [2152, 179, 2916, 1709], 'rec_docs': '韩国进口火山泥', 'rec_scores': 0.5329636}, {'bbox': [572, 361, 1245, 1653], 'rec_docs': '小度充电宝', 'rec_scores': 0.5052846}]
由测试效果图可知, 模型对于未参与训练的商品及多个商品均有较好的识别效果:
PP-ShiTu
中检测模型采用的 PicoDet
算法, 在使用官方模型后, 如果不满足精度需求, 则可以参考此部分文档, 进行模型调优
对模型进行训练的话, 需要自行准备数据, 并对数据进行标注, 建议一个类别至少准备200张标注图像, 并将标注图像及groudtruth文件转成coco文件格式, 以方便使用PaddleDetection进行训练的时候, 请加载主体检测的预训练权重.
在使用官方模型后, 如果不满足精度需求, 则可以参考此部分文档, 进行模型调优
因为要对模型进行训练, 所以参照数据准备部分描述收集自己的数据集. 值得注意的是, 此部分需要准备大量的数据, 以保证识别模型效果.
RandomErasing
增强方法.backbone
, 一般来说, 越大的模型, 特征提取能力更强.Metric Learning
方法. 不同的Metric Learning
方法, 对不同的数据集效果可能不太一样, 建议尝试其他Loss
模型训练完成后, 参照测试代码进行检索库更新. 同时, 对整个pipeline进行测试, 如果精度不达预期, 则重复此步骤.
使用 PaddleServing 做服务化部署时, 需要将保存的 inference 模型转换为 Serving 模型.
模型转换
%cd /code/PaddleClas/deploy/
# 安装相关包
!pip3 install paddle-serving-client==0.7.0 -i https://mirror.baidu.com/pypi/simple
!pip3 install paddle-serving-app==0.7.0 -i https://mirror.baidu.com/pypi/simple
!pip3 install faiss-cpu==1.7.1post2 -i https://mirror.baidu.com/pypi/simple
!pip3 install paddle-serving-server-gpu==0.7.0.post102 -i https://mirror.baidu.com/pypi/simple
!pip3 install paddle-serving-server==0.7.0 # CPU
# 执行模型转换命令
!python3 -m paddle_serving_client.convert --dirname /code/PaddleClas/inference/ \
--model_filename inference.pdmodel \
--params_filename inference.pdiparams \
--serving_server ./inference_PPLCNet_serving/ \
--serving_client ./inference_PPLCNet_client
识别推理模型转换完成后, 会在当前文件夹多出 inference_PPLCNet_serving/ 和 inference_PPLCNet_client/ 的文件夹. 分别修改 inference_PPLCNet_serving/ 和 inference_PPLCNet_client/ 目录下的 serving_server_conf.prototxt 中的 alias 名字: 将 fetch_var 中的 alias_name 改为 features. 修改后的 serving_server_conf.prototxt 内容如下:
feed_var {
name: "x"
alias_name: "x"
is_lod_tensor: false
feed_type: 1
shape: 3
shape: 224
shape: 224
}
fetch_var {
name: "save_infer_model/scale_0.tmp_1"
alias_name: "features"
is_lod_tensor: false
fetch_type: 1
shape: 512
}
# 转换通用检测模型
%cd /code/PaddleClas/deploy/
!python -m paddle_serving_client.convert --dirname ./models/picodet_PPLCNet_x2_5_mainbody_lite_v1.0_infer/ \
--model_filename inference.pdmodel \
--params_filename inference.pdiparams \
--serving_server ./picodet_PPLCNet_x2_5_mainbody_lite_v1.0_serving/ \
--serving_client ./picodet_PPLCNet_x2_5_mainbody_lite_v1.0_client/
检测 inference 模型转换完成后, 会在当前文件夹多出 picodet_PPLCNet_x2_5_mainbody_lite_v1.0_serving/ 和 picodet_PPLCNet_x2_5_mainbody_lite_v1.0_client/ 的文件夹.
注意: 此处不需要修改 picodet_PPLCNet_x2_5_mainbody_lite_v1.0_serving/ 目录下的 serving_server_conf.prototxt 中的 alias 名字.
服务部署和请求
注意: 识别服务涉及到多个模型, 出于性能考虑采用 PipeLine 部署方式.
%cd /code/PaddleClas/deploy/paddleserving/recognition
/code/PaddleClas/deploy/paddleserving/recognition
paddleserving 目录包含启动 pipeline 服务和发送预测请求的代码, 包括:
__init__.py
config.yml # 启动服务的配置文件
pipeline_http_client.py # http方式发送pipeline预测请求的脚本
pipeline_rpc_client.py # rpc方式发送pipeline预测请求的脚本
recognition_web_service.py # 启动pipeline服务端的脚本
PaddleServing Python Pipeline提供了用户友好的多模型组合服务编程框架, 适用于多模型组合应用的场景. 其配置文件为YAML格式, 一般默认为config.yaml. 所以在启动服务前需要修改config.yaml配置文件.
这里主要修改 识别模型
和 检测模型
的路径以及相关的计算设备:
op:
rec:
···
#uci模型路径
model_config: /code/PaddleClas/deploy/inference_PPLCNet_serving
#计算硬件类型: 空缺时由devices决定(CPU/GPU),0=cpu, 1=gpu, 2=tensorRT, 3=arm cpu, 4=kunlun xpu
device_type: 1
#计算硬件ID,当devices为""或不写时为CPU预测;当devices为"0", "0,1,2"时为GPU预测,表示使用的GPU卡
devices: "0" # "0,1"
#client类型,包括brpc, grpc和local_predictor.local_predictor不启动Serving服务,进程内预测
client_type: local_predictor
#Fetch结果列表,以client_config中fetch_var的alias_name为准
fetch_list: ["features"]
det:
concurrency: 1
local_service_conf:
client_type: local_predictor
device_type: 1
devices: '0'
fetch_list:
- save_infer_model/scale_0.tmp_1
model_config: /code/PaddleClas/deploy/picodet_PPLCNet_x2_5_mainbody_lite_v1.0_serving
检索库地址
: ···
index_dir = "/code/dataset/index_update"
assert os.path.exists(os.path.join(
index_dir, "vector.index")), "vector.index not found ..."
assert os.path.exists(os.path.join(
index_dir, "id_map.pkl")), "id_map.pkl not found ... "
···
# 启动服务,运行日志保存在 log.txt
# 需要移步至终端执行此命令
!python3 recognition_web_service.py &>log.txt &
!python3 pipeline_http_client.py
{'err_no': 0, 'err_msg': '', 'key': ['result'], 'value': ["[{'bbox': [132, 118, 355, 357], 'rec_docs': '四季宝花生酱', 'rec_scores': 0.85405}]"], 'tensors': []}
Used Time: 0.16031885147094727s
# 此步需要移步至终端执行
!python3 -m paddle_serving_server.serve stop
本项目中用户提供了基于服务器的部署Demo方案. 用户可根据实际情况自行参考.
具体可以参考:袋鼯麻麻——智能零售结算平台
基于PP-ShiTu进行图像识别的训练、部署全流程教程
Jupyter Notebook
Dear OpenI User
Thank you for your continuous support to the Openl Qizhi Community AI Collaboration Platform. In order to protect your usage rights and ensure network security, we updated the Openl Qizhi Community AI Collaboration Platform Usage Agreement in January 2024. The updated agreement specifies that users are prohibited from using intranet penetration tools. After you click "Agree and continue", you can continue to use our services. Thank you for your cooperation and understanding.
For more agreement content, please refer to the《Openl Qizhi Community AI Collaboration Platform Usage Agreement》