Architecting an AI Video Assessment Engine in Oracle APEX with Google Gemini Multimodal

A deep technical breakdown of handling binary media streams, two-stage AI orchestration, database state machines, and relational JSON parsing natively within Oracle APEX.

In enterprise applications, evaluating human communication—such as analyzing body language, vocal modulation, facial expressions, and speech relevance—has traditionally required standalone microservices built in Python or Node.js. Many development teams assume that because Oracle APEX is a database-centric platform, it is ill-suited to orchestrate heavy multimedia processing and multimodal artificial intelligence.

In Speech Nova, we challenged that convention. By leveraging the native capabilities of Oracle Database, APEX's internal web service engine, and Google Gemini's multimodal vision and speech processing, we engineered an end-to-end evaluation platform entirely within the database tier—eliminating intermediate server fleets and complex middleware pipelines.

High-Level Architectural Concept

The core engineering breakthrough is the division of labor: the client layer handles low-overhead local media capture, the Oracle Database manages binary storage and asynchronous state transitions, and Google Gemini performs multimodal inference across synchronized audio, video, and textual data.

1. Overcoming the Media Ingestion Bottleneck

Handling video recordings presents distinct challenges in enterprise database systems. A two-minute speech recording in high definition creates several megabytes of binary data. A common anti-pattern when integrating AI services is converting binary files into Base64-encoded strings and appending them directly inside a JSON request payload.

In practice, this approach fails in production environments:

Payload Inflation: Base64 encoding inflates file size by approximately 33%, causing excessive network transmission times.
Buffer Overflows: Large JSON strings push the memory limits of web service handlers and database CLOB buffers.
Timeout Risks: Combining media transmission and AI inference into a single synchronous request frequently hits gateway and HTTP connection timeouts.

To resolve this, Speech Nova isolates binary media storage from the evaluation pipeline. The application accepts the recording as an uninterrupted binary stream, committing it directly into a designated relational BLOB store before any external communication occurs.

2. The Two-Stage Multimodal Pipeline

Rather than calling Gemini’s content generation endpoint directly, our backend architecture implements a robust, two-stage decoupled pipeline.

Stage 1: Direct Binary Streaming to Google Gemini Files API

Instead of treating the AI as an immediate chat completion service, the database engine calls Gemini’s dedicated Files API. The Oracle APEX web service engine reads the BLOB directly from disk and streams it as raw binary data over an encrypted HTTP connection with dedicated media headers.

Google’s cloud infrastructure ingests, indexes, and provisions the video within its secure staging area, returning an immutable, temporary File URI. This decoupled step decouples the physical data upload from computational reasoning.

Stage 2: Multimodal Analysis via Gemini 3.1 Flash-Lite

With the File URI secured, the orchestrator triggers the evaluation phase. Instead of transmitting the video again, the request contains only the lightweight URI pointer alongside a highly structured system prompt.

Because gemini-3.1-flash-lite is natively multimodal, it processes temporal video frames and audio frequencies concurrently. The model assesses vocal cadences, facial expressions, and body language while cross-referencing the spoken content against the assigned topic for semantic relevance.

3. The 6-Dimension Communication Assessment Rubric

A common issue with generative AI evaluations is arbitrary scoring. To make assessments deterministic and actionable, our prompt instructs the model to evaluate the presentation across a calibrated 1-to-5 scale across six distinct criteria:

1. Facial Expressions: Quantifies natural eye contact with the camera, expressiveness, and emotional congruence with the topic.

2. Confidence Level: Evaluates physical composure, posture, purposeful hand gestures, and absence of nervous fidgeting.

3. Clarity of Communication: Evaluates diction, pronunciation, vocal modulation, and speech pacing.

4. Tone and Audience Engagement: Assesses vocal variation, enthusiasm, dynamic energy, and connection to the audience.

5. Conciseness and Relevance: Measures topic adherence relative to the presentation title, logical structure, avoiding repetition, and time management.

6. Language Proficiency and Fluency: Identifies grammatical precision, vocabulary variety, and active reduction of verbal fillers (such as "um", "uh", or "you know").

In addition to scoring, the pipeline extracts a verbatim transcript of the user's speech and generates an "Improved Transcript"—a restructured version showing how an executive coach would refine the same ideas for higher impact.

4. Database State Machine and Fault Tolerance

Network requests to external AI providers are non-deterministic. To maintain data integrity and support asynchronous frontends, the backend enforces a strict state machine on the evaluation record:

RECORDING INITIATED → UPLOADING → UPLOADED → EVALUATING → COMPLETED / FAILED

If any stage encounters an exception—whether an empty video stream, a network timeout, or an unexpected schema format—the transaction rolls back cleanly, records the full execution error backtrace into the database record, and marks the status as FAILED. This allows administrative auditing, automated alerting, and non-destructive retries without requiring the user to re-record.

5. Relational JSON Decomposition & Token Analytics

By enforcing strict JSON application output from the model, we prevent conversational filler and formatting inconsistencies. When the JSON payload arrives back in Oracle Database, it is decomposed immediately using native SQL/JSON operators:

Scalar Quantities: Ratings, scores, and categorical evaluations are unpacked directly into relational number and text columns. This makes the data immediately indexable, sortable, and aggregatable by database reporting engines.
Complex Arrays: Multi-valued attributes (strengths, developmental weaknesses, exercises, and actionable tips) are extracted as structured JSON collections, ready for detail rendering.

Token Observability & FinOps Governance

Enterprise AI architectures require precise cost tracking. Gemini's response metadata contains granular token consumption metrics, which the evaluator persists with every assessment:

Text Prompt Tokens: Cost of the evaluation rubric and scenario instructions.
Video Media Tokens: Cost generated by temporal video frame extraction.
Candidate & Reasoning Tokens: Cost of the model's analytical thinking and final evaluation payload.

Tracking token breakdowns per evaluation provides continuous visibility into operational costs, enabling fine-tuning of session duration policies based on real data.

6. Key Takeaways for Oracle APEX Developers

Oracle APEX is Fully Capable of Multimodal Workloads: You do not need secondary application servers or microservice stacks to handle video and AI pipelines. Native database tools and APEX web service components provide everything necessary to orchestrate modern media flows.
Decouple File Ingestion from Reasoning: When dealing with binary assets, always split upload and evaluation into two distinct operations. The Gemini Files API is purpose-built for this architectural pattern.
Enforce Structured JSON Contracts: Treat LLM responses as API contracts. Forcing application-level JSON mode allows Oracle's SQL/JSON operators to bridge unstructured intelligence into relational tables seamlessly.
Track Token Telemetry Natively: Instrumenting token observability at the database level ensures complete FinOps governance across your enterprise AI applications from day one.

Think APEX

Search This Blog