PP-FormulaNet_plus is an enhanced version of the formula recognition model developed by the PaddleOCR Team, building upon the original PP-FormulaNet. Compared to the original version, PP-FormulaNet_plus utilizes a more diverse formula dataset during training, including sources such as Chinese dissertations, professional books, textbooks, exam papers, and mathematics journals. This expansion significantly improves the model’s recognition capabilities. The PP-FormulaNet_plus includes multiple versions: L, M, and S, where PP-FormulaNet_plus-L have added support for Chinese formulas and increased the maximum number of predicted tokens for formulas from 1,024 to 2,560, greatly enhancing the recognition performance for complex formulas. The key accuracy metrics are as follow:
Model
Backbone
En-BLEU↑
Zh-BLEU(%)↑
GPU Inference Time (ms)
UniMERNet
Donut Swin
85.91
43.50
2266.96
PP-FormulaNet-S
PPHGNetV2_B4
87.00
45.71
202.25
PP-FormulaNet-L
Vary_VIT_B
90.36
45.78
1976.52
PP-FormulaNet_plus-S
PPHGNetV2_B4
88.71
53.32
191.69
PP-FormulaNet_plus-M
PPHGNetV2_B6
91.45
89.76
1301.56
PP-FormulaNet_plus-L
Vary_VIT_B
92.22
90.64
1745.25
LaTeX-OCR
Hybrid ViT
74.55
39.96
1244.61
Note: En-BLEU and Zh-BLEU (%) represent the BLEU scores for English formulas and Chinese formulas, respectively. The evaluation dataset for English formulas includes simple and complex formulas from UniMERNet, as well as simple, intermediate, and complex formulas from PaddleX’s internally developed dataset. The evaluation dataset for Chinese formulas comes from PaddleX’s internally developed Chinese formula dataset.
Quick Start
Installation
PaddlePaddle
Please refer to the following commands to install PaddlePaddle using pip:
# for CUDA11.8
python -m pip install paddlepaddle-gpu==3.0.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu118/
# for CUDA12.6
python -m pip install paddlepaddle-gpu==3.0.0 -i https://www.paddlepaddle.org.cn/packages/stable/cu126/
# for CPU
python -m pip install paddlepaddle==3.0.0 -i https://www.paddlepaddle.org.cn/packages/stable/cpu/
You can also integrate the model inference of the formula recognition module into your project. Before running the following code, please download the
sample image
to your local machine.
from paddleocr import FormulaRecognition
model = FormulaRecognition(model_name="PP-FormulaNet_plus-L")
output = model.predict(input="4kkIUGxXMGozIg6U1BIxZ.png", batch_size=1)
for res in output:
res.print()
res.save_to_img(save_path="./output/")
res.save_to_json(save_path="./output/res.json")
Note: If you need to visualize the formula recognition module, you must install the LaTeX rendering environment by running the following command. Currently, visualization is only supported on Ubuntu. Other environments are not supported for now. For complex formulas, the LaTeX result may contain advanced representations that may not render successfully in Markdown or similar environments:
For details about usage command and descriptions of parameters, please refer to the
Document
.
Pipeline Usage
The ability of a single model is limited. But the pipeline consists of several models can provide more capacity to resolve difficult problems in real-world scenarios.
Formula Recognition Pipeline
The formula recognition pipeline is designed to solve formula recognition tasks by extracting formula information from images and outputting it in LaTeX source code format. And there are 4 modules in the pipeline:
If save_path is specified, the visualization results will be saved under
save_path
. The visualization output is shown below:
The command-line method is for quick experience. For project integration, also only a few codes are needed as well:
from paddleocr import FormulaRecognitionPipeline
pipeline = FormulaRecognitionPipeline(formula_recognition_model_name="PP-FormulaNet_plus-L")
output = pipeline.predict("./4HrLNUf2yKGI8CwN9axpt.png")
for res in output:
res.print() ## Print the structured output of the prediction
res.save_to_img(save_path="output") ## Save the formula visualization result of the current image.
res.save_to_json(save_path="output") ## Save the structured JSON result of the current image
For details about usage command and descriptions of parameters, please refer to the
Document
.
PP-StructureV3
Layout analysis is a technique used to extract structured information from document images. PP-StructureV3 includes the following six modules:
Layout Detection Module
General OCR Subline
Document Image Preprocessing Subline (Optional)
Table Recognition Subline (Optional)
Seal Recognition Subline (Optional)
Formula Recognition Subline (Optional)
Run a single command to quickly experience the PP-StructureV3 pipeline:
Just a few lines of code can experience the inference of the pipeline. Taking the PP-StructureV3 pipeline as an example:
from paddleocr import PPStructureV3
pipeline = PPStructureV3(
formula_recognition_model_name="PP-FormulaNet_plus-L",
use_doc_orientation_classify=False, # Use use_doc_orientation_classify to enable/disable document orientation classification model
use_doc_unwarping=False, # Use use_doc_unwarping to enable/disable document unwarping module
use_textline_orientation=False, # Use use_textline_orientation to enable/disable textline orientation classification model
device="gpu:0", # Use device to specify GPU for model inference
)
output = pipeline.predict("./bar8SvxUbAFdMmIMyZpTk.png")
for res in output:
res.print() # Print the structured prediction output
res.save_to_json(save_path="output") ## Save the current image's structured result in JSON format
res.save_to_markdown(save_path="output") ## Save the current image's result in Markdown format
The default model used in pipeline is
PP-FormulaNet_plus-L
. And you can also use the local model file by argument
formula_recognition_model_dir
. For details about usage command and descriptions of parameters, please refer to the
Document
.
PP-FormulaNet_plus-L huggingface.co is an AI model on huggingface.co that provides PP-FormulaNet_plus-L's model effect (), which can be used instantly with this PaddlePaddle PP-FormulaNet_plus-L model. huggingface.co supports a free trial of the PP-FormulaNet_plus-L model, and also provides paid use of the PP-FormulaNet_plus-L. Support call PP-FormulaNet_plus-L model through api, including Node.js, Python, http.
PP-FormulaNet_plus-L huggingface.co is an online trial and call api platform, which integrates PP-FormulaNet_plus-L's modeling effects, including api services, and provides a free online trial of PP-FormulaNet_plus-L, you can try PP-FormulaNet_plus-L online for free by clicking the link below.
PaddlePaddle PP-FormulaNet_plus-L online free url in huggingface.co:
PP-FormulaNet_plus-L is an open source model from GitHub that offers a free installation service, and any user can find PP-FormulaNet_plus-L on GitHub to install. At the same time, huggingface.co provides the effect of PP-FormulaNet_plus-L install, users can directly use PP-FormulaNet_plus-L installed effect in huggingface.co for debugging and trial. It also supports api for free installation.
PP-FormulaNet_plus-L install url in huggingface.co: