System for Handwritten Mathematical expressions extracted from smarthphone pictures and translation to LaTeX

Original Title: "Prototipo de sistema para el reconocimiento de texto en imágenes y su traducción a LaTex".

Description

This repo contains the capstone project developed during 1 year until June 2020.
The project is a system for Mathematical expressions recognition given an image taken from a smartphone.

Team Integrants:

• Barrera Pérez Carlos Tonatihu
• García Medina Juan Carlos
• Mendoza Jaimes Ian

Director:

• Cortés Galicia Jorge

State of the art

• Mathpix
• MyScript Nebo
• SESHAT

Papers and theory Behind:

(09 abril 2019)

• Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition PDF Link local Link

• CHROME dataset desctiption. link

• Image-to-Markup Generation with Coarse-to-Fine Attention PDF link Local link

• Introductory paper to the technique used in the above papers (newer methods) Paper

Specific Modules

All of the modules are stored within this same repo well organized inside folders.

• Image analysis module This process the taken picture from the smartphone and transform it to be as close as possible to the images from the Dataset.

• LaTeX Translation module Deep Learning module developed using TensorFlow 2.0 that takes as input the image previously processed by the Image analysis module.

• LaTeX tokenizer Module developed using Lex in python that takes a LaTeX sequence and returns a numeric sequence to be loaded later as a Tensor in TensorFlow.

• Android app for taking pictures containing handwritten mathematical expressions

• Web server Server that handles the above described modules communicating the android app with the database through a REST API.

• Users management module This module allows to have in the UI the translations organized in projects giving the user the ability to login anywhere and see his previous projects.