LibrasFlow

Context and problem

Communication between deaf and hearing people still runs into barriers when there is no interpreter, and many recognition tools need powerful hardware or a software install. LibrasFlow is our capstone project for the Systems Analysis and Development degree at IFSP Jacareí: a lightweight recognizer for the manual alphabet of Libras (Brazilian Sign Language) that works with an ordinary webcam, on a modest computer, in the browser.

The scope is the 21 static letters. H, J, K, X and Z involve movement and were left out. The work was written as a scientific paper, currently being prepared for submission.

My role

The project was done in a team of two with Carlos Henrique Moreira Gonçalves, advised by Prof. Tardelli Ronan Coelho Stekel, PhD, and Prof. Ana Paula Abrantes de Castro e Shiguemori, PhD. I was the lead developer: data preparation, training, the evaluation protocol (including the leak investigation) and the web app.

Solution and architecture

The model doesn't look at the raw image. MediaPipe finds 21 hand landmarks in each frame, and the classifier works only with their geometry.

1. 01CameraOne webcam frame
   ↓→
2. 02MediaPipe Hands21 hand landmarks
   ↓→
3. 03NormalizationCentered on the wrist and rescaled
   ↓→
4. 04ClassifierRandom Forest: from 42 values to a letter
   ↓→
5. 05Stability filterConfirms after 30 matching frames
   ↓→
6. 06Letter on screenReal-time text

• Features: the x and y coordinates of the 21 landmarks (42 values), centered on the wrist and divided by the largest absolute value. That reduces the effect of the hand's position and size in the image.
• Model: Random Forest with 100 trees, maximum depth 15 and at least 2 samples per leaf.
• Data: the public "Libras" dataset (Kaggle, Oliveira, 2022), with 34,374 samples across 21 classes. Each sample is a vector of hand landmarks, not a photo.
• Temporal confirmation: a letter only enters the text after 30 consecutive frames with the same prediction, which reduces flickering between similar letters.
• Virtual "LIMPAR" (clear) button: triggered by a gesture, when the index fingertip stays over the button area for 25 frames.

There are two versions. In the capstone version, Flask processes the video on the server and streams the result to the browser. In the public version, everything happens in the browser itself: MediaPipe runs in WebAssembly and classification uses an MLP network (63 → 128 → 64 → 21) exported as JSON, with 12-frame confirmation and a minimum confidence of 0.75. No image leaves the device.

The data leak

The first evaluation used a random 80/20 split, repeated 10 times, and reported 99.96% ± 0.02% accuracy. For a hard visual problem, that was too good, so we investigated.

For each test sample, we measured the distance to the closest training sample of the same class. 83.7% had a near-copy less than 0.05 away: consecutive frames from the same recordings. In part, the model was recognizing samples it had already seen.

1. 0134,374 samplesVectors of hand landmarks
   ↓→
2. 02DBSCAN (ε = 0.05)Joins near-identical samples
   ↓→
3. 036,270 groups5.5 samples per group on average
   ↓→
4. 04Split by groupEach group goes whole to training or to test

With the group split, the share of near-duplicate pairs between training and test dropped to 0%, and the typical distance between the two sets grew about 2.7 times. Accuracy settled at 97.01% ± 2.59% (minimum 92.7%, maximum 99.9%), with a macro F1 of 96.7%.

in the first evaluation, with the leak

99.96%

with training and test split by group

97.0%

on 3,812 images from another source

71.2%

The result depends on the grouping radius ε, so we ran a sensitivity analysis from 0.02 to 0.15. 0.05 is the largest value that keeps at least 10 groups per class and the planned 20% test share.

ε	Groups	Accuracy
0.02	30,266	99.97%
0.05 (adopted)	6,270	97.01%
0.10	961	88.13%
0.15	347	58.22% (unstable split)

Testing on images from another source

A leak-free protocol still tests on images from the same dataset. To measure generalization, we used the "Brazilian Sign Language Alphabet" dataset (Passos; Fernandes; Comunello, 2020): 4,411 images of 15 letters, from other authors and under other capture conditions, including worse lighting. This set was not used for training or for choosing hyperparameters.

MediaPipe didn't find the hand in 599 images (13.6%). On the remaining 3,812, accuracy was 71.2% (95% CI: 69.8%–72.6%), and balanced accuracy was 58.9%.

• M, N and E follow another sign language. In this dataset, those letters appear in their American Sign Language (ASL) form, which differs from the Libras form used in training. E was classified as S in 510 of 552 images. Without M and N, accuracy rises to 72.6%; without M, N and E, to 85.3% (95% CI: 84.0%–86.4%).
• The remaining errors are in fine details: pairs that differ only slightly, such as S/A, R/V, U/V and L/D.
• Depth (z) didn't help. With 63 features, internal accuracy didn't change significantly (97.84%; Wilcoxon test, p = 0.85), and the external test got worse (69.4%; McNemar test, p < 0.001). That is why the adopted model uses 42 features.
• The public version's MLP ties with the Random Forest on the internal test (96.4%) but drops to 59.4% on the external one. It is the working demo, not the validated model.

Real-time performance

Measured over 300 frames on a laptop with an i5-13420H, no dedicated GPU, and a 720p webcam:

Step	Average latency
Detection (MediaPipe)	16.7 ms
Classification (Random Forest)	25.6 ms
Full pipeline	40.4 ms (≈ 24.8 FPS)

With on-screen display, the observed end-to-end rate was 19.5 FPS.

Try it

The project's public version (in Portuguese), embedded here. Turn on the camera, show one hand and sign a letter of the manual alphabet: the ring around the letter closes when the recognition settles.

librasflow.ifsp.dev

Open live demo

Then click Ativar câmera

Runs in your browser: no image leaves your computer.

Open in a new tab↗(opens in a new tab)

The public version runs in the browser with an MLP network; the numbers in this case study come from the Random Forest.

Stack

• Computer vision: MediaPipe Hands and OpenCV.
• Model and evaluation: scikit-learn (Random Forest, MLP and DBSCAN), NumPy, pandas and SciPy.
• Capstone app: Flask, HTML5 and Tailwind CSS.
• Public version: MediaPipe Hand Landmarker in WebAssembly and the MLP exported as JSON, all in the browser.

Decisions and trade-offs

Hand landmarks instead of the image

With 42 numbers per frame, the model is small, runs in real time on a computer with no dedicated GPU and depends less on background and lighting.

trade-offIf MediaPipe doesn't detect the hand, there is no prediction. On the external test, that happened in 13.6% of the images.

Group before splitting

Splitting by sample measures memory. Splitting by group measures whether the model recognizes a pose it has never seen.

trade-offThe number looks less impressive and now depends on the grouping radius, which is why choosing ε needed a sensitivity analysis.

42 features, without depth

The z coordinate didn't improve the internal test and worsened the external one. Keeping x and y makes the model simpler and better outside its training data.

trade-offPairs that differ by fine details, such as R/V and U/V, are still the largest source of error.

Random Forest in the study, MLP in the demo

In the public version, inference happens in the browser with an MLP exported as JSON, and no image leaves the device.

trade-offOn the external test, the MLP reaches 59.4%, against 71.2% for the Random Forest. So the study's numbers come from the Random Forest, and the public version is presented as a demo.

Confirm a letter only after 30 frames

It keeps the phrase from filling up with wrong letters while the hand moves from one sign to the next.

trade-offSpelling is slower: each letter requires holding the sign for a moment.

Outcome

• A recognizer that runs in real time with an ordinary webcam and no dedicated GPU: 19.5 FPS end to end.
• A three-level evaluation: 99.96% with the leak, 97.0% with the group split and 71.2% on images from another source.
• An explanation for the drop: letters that follow another sign language and pairs that differ by fine details.
• A public version anyone can try in the browser.

Next steps named in the paper: evaluation across different users, Libras images with identified signers and dynamic signs with temporal models (LSTM or Transformers).

What I learned

• Accuracy close to 100% is a reason to investigate, not to celebrate.
• How training and test are split changed the result more than switching models: 99.96%, 97.0% or 71.2%, as the test samples get more "new".
• Generalizing takes varied data: other people, lighting, distance and angle. That is the main future work.