• 😉Recent Update😉
• 🧁Project Context🧁
• Applicable Scene
• Pain Problem
• 🍑Realize function🍑
• 🍎Overall Architecture🍎
• 🐻Technical Route🐻
• • PP-ShiTu Introduce
  • Introduction to Data Set
  • List of Commodities
• Ablation experiments
• 🌍Deployment Mode🌍
• 💃bilibili Results Demonstrate💃
• ⛽️To Do List⛽️
• 🚀 Development Team🚀
• ☕Sponsor☕
• 🌟Thanks🌟
• ❤️Welcome to build together❤️
• Citation

English | 简体中文 | 启智OpenI | Gitee | Github

Remember to give it a !

😉Recent Update😉

• Release training code：Publish model training code and experimental results for adaptive tuning;
• Fixe Code Bug： Restart the service after the database is updated in Pipeline deployment mode;
• Improve Accuracy Greatly：The test accuracy of the self-collected data set is 99.098%；
• Upgrade Document：Provides detailed documentation of PP-ShiTu model training and deployment；
• Optimize Deployment Mode：The predicted speed increase is 65%, based on the overall CPU flow control at 0.9s；
• Upgrade Product Function：Add inventory information management function, provide one-click data visualization analysis platform;

🧁Project Context🧁

Currently in the process of actual operations of the retail industry, will produce a great human cost, such as guides, cleaning, settlement, and among them, especially need to spend a lot of labor cost and time cost in the identification of goods and settlement in the process of the price, and in the process, and so the customer need to wait in line. As a result, the retail industry has high labor costs and low work efficiency. Secondly, it also reduces the shopping experience of customers.

With the development of computer vision technology, as well as the unmanned and automated supermarket operation concept, the use of image recognition technology and target detection technology to achieve Automatic product identification and Automatic settlement demand, namely Automatic checkout system (ACO). The automatic checkout system based on computer vision can effectively reduce the operating cost of retail industry, improve the checkout efficiency of customers, so as to further enhance the user experience and happiness in the process of shopping.

Applicable Scene

ColugoMum——Intelligent Retail Rettlement PlatformCommitted to provide the largest offline retail experience store with retail settlement solution based on vision.

Pain Problem

1. Settlement efficiency is highly required：In the shopping scenario of large offline retail experience stores, if customers buy more goods, the traditional bar code settlement is adopted, which is inefficient and leads to poor shopping experience of customers；
2. Category update very fast：For in the new retail industry, new products are almost always updated on an hourly basis. Every time new products are added, the model has to be retrained so hard that it is impossible for a single model to keep pace；
3. Different products are very similar：Different flavors of the same drink, for example, are likely to have very similar packaging. And even for the same product, there are often considerable differences in the product images obtained under different circumstances;
4. Tens of thousands of commodity categories：There is no way to put all categories into the training set beforehand。

🍑Realize function🍑

ColugoMum realize automatic settlement of goods purchased by users in the retail process. We take advantage of the PaddleClas team's open source PP-ShiTu technology, precise positioning of customers to buy goods, and intelligent, automatic price settlement. When customers place their chosen products in the designated area, ColugoMum can accurately locate and identify each product, and can return a complete shopping list and the actual total price of goods that customers should pay. When the system has a new product increase, the system only need to update the retrieval database, without retraining the model.

This project is a lightweight general PP-ShiTu image recognition system provides the solid ground application cases, the new one of the retail industry and retail visual intelligent solution provides a very good basis and train of thought, especially for solving many categories, small sample, high similarity, and frequently updated the special image recognition scene pain difficulties provides reference of demonstrations, Greatly reduce the retail industry in the actual operation of the huge human cost, improve the retail industry unmanned, automation, intelligent level.

🍎Overall Architecture🍎

🐻Technical Route🐻

ColugoMum Based on PaddleClas as the main feature development suite, leveraging its open source PP-ShiTu for core feature development. Through PaddleInference, it was deployed in Jetson Nano, and was packaged based on QPT to develop an industrial-level intelligent retail settlement platform in line with actual application requirements.

PP-ShiTu Introduce

PP-ShiTu is a practical lightweight general image recognition system, which is mainly composed of three modules: subject detection, feature learning and vector retrieval. The system from the selection and adjustment of backbone networks, the choice of loss function, vector data, transform strategy, choice of regularization parameter, use the training model and quantitative model cutting eight aspects, use a variety of strategies, optimize the model of the various modules, finally got on the CPU is only 0.2 s to complete 10 w + library image recognition system.

The whole image recognition system is divided into three steps（See PP-ShiTu training module for details）：
(1) The candidate regions of image objects are detected by a target detection model;
(2) Feature extraction for each candidate region;
(3) Feature matching with images in the retrieval database, and extraction of recognition results.

For the new unknown category, there is no need to retrain the model, but only need to add the image of the category in the retrieval database and rebuild the retrieval database to recognize the category.

Introduction to Data Set

【The first one】:Products-10K Large Scale Product Recognition Dataset

【The second one】:RP2K: A Large-Scale Retail Product Dataset for Fine-Grained Image Classification

ColugoMum based on the above two data sets and combined with the actual characteristics of the retail scene, adaptive processing is carried out.

List of Commodities

东古酱油一品鲜
东古黄豆酱750G
东鹏特饮罐装
中华（硬）
中华（软）
乳酸菌600亿_2
乳酸菌600亿_3
乳酸菌600亿原味
乳酸菌600亿芒果
乳酸菌600亿芦荟
...

The processed dataset is now open source in AIStudio.

Ablation experiments

model	num epoch	batch size/gpu cards	learning rate	use cutout	use ssld	top1 recall	config
PP_LCNet_x2_5	400	256/4	0.01	N	N	98.189%	config
PP_LCNet_x2_5	400	256/4	0.01	Y	N	98.21%	config
PP_LCNet_x2_5	400	256/4	0.005	N	N	98.201%	config
PP_LCNet_x2_5	400	256/4	0.005	Y	N	98.29%	config
PP_LCNet_x2_5	400	256/4	0.001	Y	N	98.26%	config
PP_LCNet_x2_5	400	64/4	0.005	Y	Y	98.30%	config
PP_LCNet_x2_5	400	64/4	0.0025	Y	Y	98.37%	config
PP_LCNet_x2_5	400	64/4	0.002	N	Y	98.38%	config
PP_LCNet_x2_5	400	64/4	0.002	Y	Y	98.39%	config

Attention:

1. This experiment is based on GPU:Tesla V100* 4; CPU:Inter Xeon* 32; RAM:DDR4 128GB for training and testing;
2. The experiments are based on the above data set Retail Product Characteristics Study Data Set for training and testing;
3. Evaluation of RP2K and other large open source data sets of retail products will be carried out soon.

🌍Deployment Mode🌍

ColugoMum has been connected toJetson Nano, Windows, linux system.

• Windows
  [ColugoMum provides a relatively simple demo version]

  We use QPT for packaging.
  Download the project code, enter the QPT_client folder, and Click the "启动程序.exe".

• Linux
  Download the project code, enter the client folder, and run the following code to run it ：

    python client.py
  

• For details of the image recognition part deployment, you can see PP-ShiTu Development

• Wechat applet
  Open the wechat developer tool, import the AIContainer folder under the system folder and run it；

💃bilibili Results Demonstrate💃

• Main Interface

  

• Client Side Interface

• Applets Interface

• Big Data Visualization Analysis Interface

⛽️To Do List⛽️

number	complete degree	priority	category	Functional description
1	completed	★★★★★	Applets	Add inventory information display, add data analysis module
2	Doing	★★★★★	Applets	Initial function online
3	completed	★★★★★	Client Side	Jetson Nano Depth adaptation
4	planning	★★★★	Applets	Separation of functions for managers and customers
5	completed	★★★★	web	the establishment of web information management system
6	planning	★★★	Applets	Realize the automatic entry of commodity name
7	planning	★★	APP	Enabling deployment on the IOS and Android

🚀 Development Team🚀

Duty	Name
PM	X. Yan
Algorithm	X. Yan
Side of the front end	X. Yan
Applets front end	C. Shen
Back End	D. DU

☕Sponsor☕

A cup of coffee will refresh your mind, and product updates will be faster and better!

🌟Thanks🌟

• PaddleClas provides the image recognition core function development;
• QPT provides Windows side package;
• jkfx fixed some bugs.

❤️Welcome to build together❤️

We welcome you to contribute code or provide suggestions for "ColugoMum". Whether you have a bug, fix a bug, or add a new feature, feel free to submit Issue or Pull Requests.

Citation

@software{ColugoMum2021,
  author = {Xin Yan,  Chen Shen, and XuDong Du},
  title = {{ColugoMum: Intelligent Retail Settlement Platform}},
  howpublished = {\url{https://github.com/thomas-yanxin/Smart_container}},
  year = {2021}
}