Vision and text model capabilities
The vision and text model layer turns images and text into signals for search and asset analysis. It provides shared image and text embeddings through CLIP, face locations and face embeddings through facial recognition, and detected text with scores through OCR. The machine learning service exposes these capabilities to the server, while automated asset analysis stores and uses the results.
Capability boundary
The model service accepts one image or one text value with a list of requested model tasks. It selects the model family for each task, prepares the input, runs the required stages, and returns task-specific results together with image dimensions when the input is an image.
The server sends requests through server/…/machine-learning.repository.ts. That boundary keeps model execution separate from product workflows. Search asks for embeddings, people analysis asks for faces, and OCR jobs ask for text regions and recognized text.
The main capabilities have these roles:
| Capability | Input | Product signal |
|---|---|---|
| CLIP | Image or text | A vector used to compare images and text for smart search |
| Facial recognition | Image | Face boxes, confidence scores, and face embeddings |
| OCR | Image | Text strings, polygon boxes, and detection and recognition scores |
The request and result flow is organized as follows.
flowchart LR
A[Asset or search text] --> B[Server model repository]
B --> C[Machine learning request]
C --> D[Input transform]
D --> E[Model pipeline]
E --> F[Structured result]
F --> G[Search, people, or OCR analysis]
Shared model abstraction
The shared model abstraction gives every model a common life cycle and runtime boundary. A model has a task and type identity, loads its model data when first used, and returns a result in the shape expected by the request. InferenceModel in machine-learning/…/base.py is the common contract.
Models can use more than one execution format. The service selects a format supported by the environment, downloads model data when it is not cached, and creates a session for inference. A model can warm its session before use, and the model cache reuses loaded instances across requests.
Model selection and dependencies
Model selection maps a model name, task, and type to one of the supported capability implementations. The model source map in machine-learning/…/constants.py identifies CLIP, InsightFace, and Paddle model families for visual, textual, detection, or recognition roles. The supported names include newer CLIP and Paddle OCR model families, so the map remains the contract that connects configured names to their model source.
Some roles depend on an earlier result. Facial recognition depends on face detection, and OCR recognition depends on text detection. The service runs independent stages first, then supplies their results to dependent stages. This lets one request express a complete pipeline without making the caller manage intermediate data.
References: machine-learning/…/constants.py• L4-92 machine-learning/…/constants.py• L167-182 machine-learning/…/recognition.py• L24-27 machine-learning/…/recognition.py• L23-25
Model loading and caching
Model loading keeps downloaded model data separate from live inference sessions. InferenceModel loads a model once under a load lock, while ModelCache returns the same model for matching model identity and graph options. Temporary models can expire after a time-to-live; preloaded models remain available until the cache is cleared.
This cache boundary also accounts for options that change a model graph. For example, OCR detection resolution is part of the cache key, so requests with different resolution choices do not share an incompatible model instance.
References: machine-learning/…/base.py• L27-92 machine-learning/…/base.py• L122-192 machine-learning/…/cache.py• L11-43 machine-learning/…/detection.py• L19-44
Input and output transforms
Transforms make source data match the model contract and make model results useful to the server. Shared image handling decodes bytes, arrays, or images to RGB data. Model-specific stages then resize, crop, pad, normalize, or change tensor layout as required by the selected model.
Text inputs are cleaned before tokenization. CLIP tokenizers pad or truncate text to the model context and can add a language code for multilingual models. Numeric vectors are serialized so the HTTP response can carry them without requiring the server to understand the model runtime. Detection and recognition results keep semantic fields such as boxes, landmarks, text, and confidence scores.
References: machine-learning/…/transforms.py• L15-79 machine-learning/…/transforms.py• L82-97 machine-learning/…/textual.py• L28-60 machine-learning/…/visual.py• L25-36
CLIP embeddings
CLIP provides a shared representation for visual and textual content. The visual encoder turns an asset image into a vector, and the textual encoder turns a search phrase into a vector in the same comparison space. The server can then compare the two forms without matching the words in the phrase directly to stored labels.
Visual encoding
The visual pipeline decodes an image, applies the preprocessing settings for the selected model, and returns one serialized embedding. Smart search stores that embedding for an asset and later compares it with a text embedding or with another asset embedding.
The visual preprocessing settings are part of the model contract. A change in image size, resize mode, interpolation, normalization, or model choice can change the meaning and dimension of the vector. The server therefore checks the selected CLIP model and keeps the search index dimension aligned with it.
References: machine-learning/…/visual.py• L25-36 machine-learning/…/visual.py• L62-89 machine-learning/…/transforms.py• L15-41 server/…/machine-learning.repository.ts• L205-209 server/…/smart-info.service.ts• L34-64
Text encoding
The textual pipeline cleans a search phrase, tokenizes it with the selected CLIP family, and returns a serialized embedding. OpenCLIP and multilingual CLIP use their own token input shapes, while multilingual OpenCLIP can add a language code when the model supports it. The search service caches these text embeddings during query handling and uses them to search stored asset embeddings.
References: machine-learning/…/textual.py• L28-60 machine-learning/…/textual.py• L101-142 server/…/machine-learning.repository.ts• L211-215 server/…/search.service.ts• L325-353
Facial recognition pipeline
Facial recognition converts an image into face records that can support people grouping and face search. It has two linked stages: detection finds candidate faces and landmarks, then recognition aligns each face and produces an embedding for that face.
Face detection
The detector prepares the image with aspect-preserving padding and pixel normalization. It decodes detector output into face boxes, scores, and five-point landmarks, then removes low-score and overlapping candidates. It maps the kept coordinates back from the detector canvas to the source image before the recognition stage receives them.
References: machine-learning/…/detection.py• L13-38 machine-learning/…/_ops.py• L18-56
Face recognition
The recognizer uses detector landmarks to align each face to a common face crop before it creates an embedding. Its result joins the embedding with the face box and detection score, so the server can both locate the face in the source image and compare it with known people.
The server scales returned boxes to the asset dimensions that it stores. It uses the embeddings for face search and keeps the face records linked to the asset and, when known, to a person.
References: machine-learning/…/recognition.py• L24-72 machine-learning/…/_ops.py• L59-73 server/…/machine-learning.repository.ts• L190-203 server/…/person.service.ts• L341-369 server/…/asset-face.table.ts• L40-88
OCR pipeline
OCR converts visible text in an image into searchable text and its position. It has two linked stages: detection finds text polygons, then recognition reads each cropped polygon and returns text with a recognition score.
Text detection
The detector scales an image to a model-friendly canvas and produces a probability map for text. It selects a compatible compiled canvas for the requested maximum resolution and skips images that are too small to yield a useful result. Post-processing turns the map into quadrilateral boxes and detection scores. The boxes are ordered from top to bottom and then from left to right, which gives the recognition stage a stable reading sequence.
References: machine-learning/…/detection.py• L19-36 machine-learning/…/detection.py• L46-59 machine-learning/…/detection.py• L61-92 machine-learning/…/detection.py• L111-117 machine-learning/…/postprocess.py• L45-52
Text recognition
The recognizer crops each detected polygon and corrects its perspective before resizing the crops for recognition. It groups similar crop shapes into batches, decodes the model output with a CTC decoder, and filters results below the recognition threshold. Before returning, it normalizes polygon coordinates to the image width and height so the server can store them in a stable range.
References: machine-learning/…/recognition.py• L23-55 machine-learning/…/recognition.py• L56-113 machine-learning/…/ctc.py• L62-85
OCR model compatibility
OCR supports both the current model-graph path and older model exports. The compatibility layer keeps older installations on the ONNX format and obtains their model files through the older OCR model source. Current model graphs use the shared model loading and session path.
The model source map in machine-learning/…/constants.py recognizes the supported Paddle OCR model names. The legacy path is used only when the older model setting is enabled. This keeps older exports available without making them the default for current model graphs.
References: machine-learning/…/legacy.py• L6-42 machine-learning/…/constants.py• L78-92 machine-learning/…/constants.py• L167-182 machine-learning/…/detection.py• L19-44 machine-learning/…/recognition.py• L23-45
Analysis outputs
The server turns model results into durable search data and user-visible asset data. The model service does not decide how a face is assigned to a person or how OCR text is searched. It provides the signals and scores that those server workflows use.
Search embeddings
Smart search creates an image embedding for each eligible asset and stores it in the search index. A text query uses the selected textual encoder to create a comparable embedding. A query can also use an existing asset embedding, which supports image-to-image search.
Changing the CLIP model can change both vector size and vector meaning. The smart-information workflow validates the model, updates the stored dimension when needed, and removes old embeddings when the model changes. Assets then need new image embeddings before the new model can give complete search results.
References: server/…/smart-info.service.ts• L67-118 server/…/search.service.ts• L325-350 server/…/machine-learning.repository.ts• L205-215
Face records
Face analysis stores each detected face with its asset location, image dimensions, visibility state, and optional person link. The face embedding supports later matching. The person workflow also uses the returned image dimensions to map model coordinates to the asset record.
References: server/…/person.service.ts• L324-369 server/…/asset-face.table.ts• L36-88 server/…/machine-learning.repository.ts• L68-74
OCR records
OCR analysis stores each recognized string with its quadrilateral, detection score, and recognition score. It also builds search tokens from the recognized text, so OCR contributes to text search as well as to an overlay on the asset.
References: server/…/ocr.service.ts• L31-55 server/…/ocr.service.ts• L57-80 server/…/asset-ocr.table.ts• L23-65
Change checklist
When these files or sections are changed, remember to consider these:
- Keep the Python model task and type identities aligned with the TypeScript request and response types in server/…/machine-learning.repository.ts.
- When a pipeline output changes, update the server consumer that stores or compares it. This includes smart-search vectors, face coordinate scaling, and OCR polygon normalization.
- When a CLIP model or preprocessing contract changes, keep the search vector dimension and existing embedding refresh path aligned with the new model.
- When a model graph option changes, keep it in the model cache key so requests cannot share an incompatible loaded model.
- When a detector output changes, preserve the dependency input expected by its recognition stage.
- When OCR decoding, coordinate, or score meaning changes, update
OcrService.parseOcrResultsand theasset_ocrcolumn contract together. - When legacy OCR support changes, keep the old export path separate from the current model-graph path until older installations no longer need it.
- When adding or removing a configured model name, update the model source map and the matching source-specific loading path together.
Important failure modes
| Trigger | Consequence | Guard |
|---|---|---|
| A model name does not map to a known source, task, and type | The prediction request cannot create the requested model | Validate the model name and use a supported model combination before sending work |
| A dependent recognition stage runs without its detector result | The machine learning request returns a bad-request error instead of a partial result | Send detection and recognition entries as one complete pipeline |
| A cached model is reused across incompatible graph options | Inference uses the wrong compiled shape or resolution | Include every graph-changing option in the model cache key |
| A CLIP model changes without matching search dimensions or fresh embeddings | Search can reject vectors or return incomplete results | Validate the model, align the stored dimension, and re-encode affected assets |
| Face landmarks or OCR polygons are interpreted in the wrong coordinate space | Stored overlays and later face matches point to the wrong image area | Preserve the detector-to-source scaling and OCR normalized-coordinate contracts |
| An asset has no usable preview or is hidden | Face or OCR analysis skips or fails the job | Keep asset eligibility checks before queuing or processing analysis |
| All configured machine learning servers fail health or prediction checks | The server cannot complete the analysis request | Keep more than one healthy endpoint when high availability is required and monitor the ping state |
| OCR detection or recognition scores fall below configured limits | Text is omitted from stored OCR results | Set thresholds with the expected model and image quality in mind |
| A configured Paddle model name is missing from the source map | OCR model selection returns no source and the request cannot start | Keep Paddle model names and legacy/current OCR loading paths aligned |
| Legacy OCR is enabled for a model export that does not match its expected language or size | Model download or inference can fail for older installations | Keep legacy model names in the expected version, language, and size form |
Key files
| File | Why to read it |
|---|---|
| machine-learning/…/main.py | Shows the request boundary and how independent and dependent model stages run. |
| machine-learning/…/base.py | Defines the shared model life cycle, model loading, runtime session boundary, and model format choice. |
| machine-learning/…/cache.py | Shows how loaded models are shared and how temporary models expire. |
| machine-learning/…/transforms.py | Defines common image, text, normalization, coordinate, and serialization steps. |
| machine-learning/…/constants.py | Maps model names to CLIP, facial-recognition, and OCR model sources and defines supported model families. |
| machine-learning/…/visual.py and textual.py | Show the two sides of the shared CLIP embedding space. |
| machine-learning/…/detection.py, recognition.py, and _ops.py | Show the detector-to-alignment-to-embedding face pipeline. |
| machine-learning/…/detection.py, recognition.py, ctc.py, and legacy.py | Show OCR box detection, text decoding, and support for older model exports. |
| server/…/machine-learning.repository.ts | Defines the server-to-model-service contract and product-facing result types. |
| server/…/smart-info.service.ts | Shows how image embeddings enter the smart-search index. |
| server/…/person.service.ts | Shows how face results become asset and person data. |
| server/…/ocr.service.ts | Shows how OCR results become stored text and search tokens. |