In this project, we mainly implemented a Handwritten Mathematical Calculator on the PYNQ-Z2 FPGA platform. It uses the picture stored in the SD card or the USB camera as input. Then it will detects possible numbers or mathematical operators in the picture, extracts these elements separately, and resizes it to (32x32x1). The convolutional neural network will finally output an array of size (14x1). The meanings of these 14 numbers is shown in the below table:
Subsequently, we will accept the recognition result of the convolutional neural network, and finally calculate the result of the input.
This project is built for 2020 SEU-Xilinx International Summer School.
If you only want to implement this Calculator on your PYNQ-Z2 FPGA Platform, you just need to connect to the board, open a terminal and type:
sudo pip3 install --upgrade git+https://github.com/Microdent/Handwritten_Mathematical_Calculator_on_FPGA.git
Note: If you get the following error after executing the above command:
error: RPC failed; curl 18 transfer closed with outstanding read data remaining fatal: The remote end hung up unexpectedly fatal: early EOF fatal: index-pack failed
You need to execute the following command first and then execute the previous command
git config --global http.postBuffer 1073742000
Or you can install this offline
sudo pip3 install --upgrade .
This project consists of two parts: Software_Project and Hardware_Project.
The Software_Project includes the tensorflow model of this network, pre-trained network parameters and data sets. So if you want to understand more about the network structure, re-train your own parameters or train this network to detect others, you should look at the
If you want to rebuild the hardware of this project, you need to look the
Hardware_Project/ folder and follow my instructions below.
Here is a way to help you quickly implement the project on your own FPGA platform.
- Microsoft Visual Studio 2019 (or other C++ compiler)
- Vivado HLS 2019.2、Vivado 2019.2 and Vitis 2019.2 (Optional)
- PYNQ-Z2 FPGA Platform
- PYNQ-Z2 v2.5 images
- Ethernet cable and Micro-USB cable (Optional)
- A Webcam like Logi C270 (Optional)
This project is implemented by using Vivado HLS 2019.2 and Vivado 2019.2 tools, but it should be possible to be rebuilt on higher versions.
1. Generate HLS IP
All HLS-related files are placed under the
Please refer to the file
Hardware_Project/HLS_Version/README.md for specific operation steps.
2. Block Design in Vivado
After performing the operations in HLS and export the IP successfully, you need to create a hardware project in Vivado and create a Block Design. For specific steps, please refer to the file
3. Run on PYNQ-Z2
The files needed to run this project on PYNQ-Z2 are placed in the
After completing Block Design in Vivado and successfully generating bit、hwh
、and tcl files, rename them to hmc.xxx and put them in the
hw_bd/ folder. Finally, just run the HMC.ipynb file and enjoy your calculator.
Here are some test results: