← Back to portfolio

Teach-by-Doing (TbD) V6 – Pathways-as-Data Engine Case Study

TbD V6 turns expert doing into executable, machine-readable “Pathways” for humans, AI agents, and robots. It is the production ingestion and understanding engine behind FreeFuse’s Pathways-as-Data (PAD) ecosystem, matured across six shipped versions to include AS9100-style traceability hooks: the compliance discipline aerospace and defense manufacturing runs on. Built for the people and machines that have to execute the work, not just read about it.

Role Systems Architect · ML Engineer · Product Co-Designer Domain Workflow Intelligence · Robotics-Ready Knowledge · Agents & Automation Stack GCP · Python · OpenCV · LLMs · Temporal Encoding · JSON/PAD

Overview

Teach-by-Doing (TbD) V6 is the ingestion and understanding engine for FreeFuse’s Pathways-as-Data platform. It converts expert behavior captured in video into a structured, temporal graph of steps, transitions, and metadata called a Pathway. The same artifact can be used as:

  • Human-facing documentation – clear, step-by-step instructions.
  • Training data – labeled trajectories for LLM agents and automation.
  • Robotics integration – a semantic “script” for embodied systems.
Users: SMEs, content creators, operations leads Consumers: learners, LLM agents, automation systems Artifact: Pathway.json (PAD schema)
Short version: TbD watches how an expert actually does the work, then produces a machine-readable “how-to” that machines and humans can both consume.

Problem Context

Most organizations have critical know-how trapped in unstructured formats: recorded Zoom calls, tutorial videos, tribal knowledge, and brittle SOPs nobody maintains. That creates three compounding gaps:

  • Data Gap: No consistent, labeled “unit of work” for training AI agents.
  • Tech Gap: Experts can’t (and shouldn’t) encode workflows as state machines.
  • Strategy Gap: Without a standard format, there is no ecosystem or marketplace for reusable workflows.

TbD is designed to bridge all three by turning unstructured “doing” into structured Pathways-as-Data that can sit at the heart of automation, continuous learning, and robotics.

Version Evolution (V1 → V6)

TbD V6 is the culmination of several iteration cycles. Each version added a missing layer of realism or capability:

  • V1 – Local MVP: Basic video ingestion, scene detection, and pathway.json export using a “bridge stack” of OpenCV + OCR. Proved the concept that video could be turned into steps.
  • V2 – GCP Fan-Out: Moved ingestion to a scalable cloud pattern with GCS, Pub/Sub, and per-segment processing, making longer videos feasible.
  • V3 – Semantic Enrichment: Added LLM-driven captions and step descriptions so the resulting pathways read like rich expert documentation instead of raw frames.
  • V4 – Temporal Grounding: Introduced a temporal encoder (LSTM-style) to capture sequence, dependencies, and multi-step context, not just isolated moments.
  • V5 – Compliance & Metadata: Added stronger metadata, depth perception, and hooks for compliance frameworks (e.g., AS9100-style traceability).
  • V6 – Pathways-as-Data Engine: Unified API, stable PAD schema, temporal embeddings, and a cohesive backend powering a Streamlit demo and downstream systems.

Architecture

TbD V6 is implemented as a cloud-native pipeline that ingests video, detects meaningful changes, enriches them with semantics and temporal context, and exports a single Pathway artifact that any downstream system can consume.

Teach-by-Doing V6 architecture diagram
TbD V6 architecture: video ingestion → segmentation (visual & temporal cues) → semantic description via LLM → temporal encoding → Pathways-as-Data export as Pathway.json for humans, LLM agents, and robotics workflows.

Key Responsibilities by Stage

  • Ingestion: Accepts source videos from SMEs (screen captures, task recordings) and normalizes them for processing.
  • Segmentation: Uses computer vision (optical flow, SSIM) and heuristic cues (UI changes, cursor movement, pauses) to identify candidate step boundaries.
  • Semantic Enrichment: Calls an LLM to describe each candidate step, expanding from “frame X to frame Y” into a natural-language explanation plus key fields (action, object, tools, preconditions).
  • Temporal Encoding: Runs a temporal encoder over the ordered steps to learn transitions, branching, and dependencies, yielding a contextual embedding for each node.
  • PAD Export: Writes out a Pathway JSON object with nodes, edges, timestamps, metadata, and temporal vectors, ready for orchestration, simulation, or documentation.

Demo Experience

The TbD V6 Streamlit demo exposes the core engine in a way that non-engineers can immediately understand: upload a clip of a task being performed, then inspect the generated pathway as a visual and JSON artifact.

TbD V6 Streamlit demo screenshot showing video upload and generated steps
TbD V6 Streamlit demo: upload a task video, run the pipeline, and inspect the generated steps, transitions, and metadata in a visual Pathway view.
TbD V6 JSON response screenshot showing Pathway.json structure
Pathway.json output: machine-readable representation of the workflow, including step descriptions, transitions, timestamps, and temporal embeddings.

Typical User Flow

  • An expert records themselves completing a task (e.g., configuring software, assembling a part).
  • The video is ingested into TbD V6 via the UI or API.
  • TbD segments the video into discrete steps, describes them, and encodes transitions.
  • A Pathway.json file is produced and can be viewed, edited, or passed downstream.
  • That Pathway can now power documentation, agent guidance, or robotics routines.

Implementation Highlights

1. Computer Vision + Heuristic Segmentation

  • Uses SSIM and optical flow to detect meaningful visual changes between frames.
  • Combines frame-level signals with domain heuristics (e.g., cursor pause, UI transitions) to infer step boundaries.

2. LLM-Based Semantic Layer

  • Frames around each boundary are summarized by an LLM, generating human-readable step descriptions.
  • Prompts extract structured fields (verb, object, tools, context) to populate PAD schema fields.

3. Temporal Encoder for Pathway Intelligence

  • A temporal model (e.g., LSTM-style encoder) processes the ordered steps to learn how they relate over time.
  • Produces embeddings that capture sequence, branching, and context, which is crucial for agents and robotics.

4. Pathways-as-Data Schema

  • Each pathway is stored as a JSON object with nodes, edges, timestamps, and metadata.
  • Supports multiple “views”: human documentation, analytics, and machine execution.

Impact & What This Demonstrates

TbD V6 demonstrates how to turn messy, unstructured behavioral data into a clean, standardized asset that can drive multiple lines of value:

  • Operational: Generate living SOPs without forcing experts to write documentation.
  • Training & AI: Supply rich, labeled trajectories for LLM agents and copilots.
  • Robotics & Automation: Provide high-level semantic scripts that can be mapped to skills.

From a skills perspective, TbD V6 showcases my ability to think and build at the level of platforms: designing schemas, temporal models, and pipelines that are useful to humans, software, and hardware all at once.

Assets