Problem Awareness

Traditional OCR works well on clean documents, but it is vulnerable when dealing with blurry or tilted images like license plates. Recognition rates drop sharply when spacing is irregular or parts are obscured. Additionally, Korean car license plates feature varied character arrangements and a mix of region names and numbers, posing challenges for standard OCR models.

This project adopted the perspective of viewing characters as objects rather than text. Each character is independently detected using YOLO-based object detection, and double verification is performed by conducting OCR on both the original and corrected images. Region names and numbers are separated using NMS and extremity-based line fitting, working robustly even in environments where traditional OCR fails.

3-Step Pipeline

The entire flow is controlled by a single function in detect.pycalled get_num(). Three ONNX models handle their respective specialties.

The first model identifies a single license plate area in the full image. It crops the one with the highest confidence among multiple candidates. The second model finds the four corners in the cropped image and performs perspective correction. The third model detects individual characters as objects across 75 classes. Unlike traditional OCR, it finds the position of each character on the license plate, allowing for recognition even if spacing is irregular or parts are obscured.

graph LR
    Input[Input Image] --> Plate[License Plate Detection]
    Plate --> Crop[License Plate Crop]
    Crop --> Vertex[Corner Detection]
    Vertex --> Warp[Perspective Correction]
    Warp --> OCR1[Warped OCR]
    OCR1 --> Validate1[Regex Validation]
    Validate1 -->|Pass| Output[Final Number]
    Validate1 -->|Fail| OCR2[Cropped OCR]
    Crop --> OCR2
    OCR2 --> Validate2[Regex Validation]
    Validate2 --> Output

Verification and Fallback: Balancing Stability and Efficiency

Perspective transformation isn't always perfect. There may be cases where corner detection fails or the license plate is tilted.

This project prioritizes OCR on warped images and returns results immediately if regex validation is passed. Only if it fails, additional OCR is performed by falling back to the original cropped image.

The method processes with a single OCR if warping succeeds, and falls back to cropped image only upon failure. It's designed to reduce unnecessary inferences while maintaining stability.

After character detection, NMS removes duplicate bounding boxes, and lines are drawn based on left and right endpoints to separate region names and number areas. Region names are corrected by comparing them to a hard-coded list of valid regions like Seoul, Gyeonggi, and Busan.

sequenceDiagram participant Detect as detect.py participant ONNX as ONNX Runtime participant Validate as validate_plate_num Detect->>ONNX: Recognize characters from warped image ONNX-->>Detect: Result Detect->>Validate: Regex Validation alt Pass Validation Validate-->>Detect: Return Final Number else Fail Validation Detect->>ONNX: Recognize characters from cropped image ONNX-->>Detect: Result Detect->>Validate: Regex Validation alt Pass Validation Validate-->>Detect: Return Final Number else Validate-->>Detect: Return Failure end end

User-Friendly GUI Design

What should be done if some images fail detection during batch processing? Ignoring them compromises data quality, and stopping entirely affects efficiency.

The PySide6-based GUI automatically pauses when detection fails, displaying the relevant image on screen. It halts processing until the user manually enters the license plate number and presses the confirm button.

This is designed for situations where accurate detection for all data is necessary, allowing users to make judgments rather than arbitrarily correcting incomplete parts.

Trade-offs and Design Philosophy

YOLO-based character detection is slower and heavier in performance compared to preprocessing-based OCR. There are also optimization issues remaining from using different models across multiple stages.

However, the essence of the project was to verify recognition rates in systems not utilizing preprocessing. By effectively employing preprocessing and applying a single model to detect and combine license plates and characters in the future, recognition rates could be preserved or improved while enhancing detection speeds.

Closing

This project marks my first experience in integrating machine learning into a service. It is a meaningful project that involved fine-tuning models and attempting new approaches.

I have been able to identify several areas for improvement upon reviewing with current perspectives. I plan to enhance the program in future updates for better performance and speed.

Through this project, I personally engaged machine learning with a service for the first time. Tuning the models and striving for new avenues provide it unique significance.