Vision & ML
On-device models read a camera feed for bodies, depth, objects and motion, and return the result as ordinary graph data.
Wire a Camera Input, or any raster source, into a vision node and the results arrive as points, masks and numbers. A dancer's joints become points to attach geometry to. A depth read becomes a Field to displace with. A detected object becomes a box to trigger from.
Models run on a worker thread, off the cook thread, so a slow prediction cannot stall playback. On a cook where the worker has not finished, the node re-emits its previous result unchanged, so the output lags the feed by however long the model takes. The depth model's own quality tiers are roughly 25 ms, 60 ms and 150 ms.
What you can read
Each vision node takes a raster and returns structured data.
Body tracking reads a whole person from one node. By default it detects up to 133 landmarks across body, both hands, face and feet, as a Collection of points carrying confidence, region and joint name on each. dense_face adds a 478-point face mesh, 52 blendshape coefficients and a face-presence score. dense_hands adds detailed per-finger hand landmarks. gestures adds a named hand gesture per hand. All twelve outputs are always present, so switching a region on or off never rewires the graph; a region that is off emits an empty value. smoothing is on by default and removes per-frame jitter, and a skeleton preview draws over the feed while the node is selected. Attach geometry to the landmarks, drive simulations, or measure angles between joints.
Depth estimates how far each pixel is from the camera, as a greyscale raster or Field. Use it for displacement, fog, parallax, or pushing 2D footage into 3D.
metricswitches to a model that reports absolute depth in metres.Object detection finds objects in frame and returns labelled boxes with confidence. Use the boxes to position things, or to fire a Trigger when something appears.
Surface normals estimate the facing direction of every pixel, for relighting flat footage.
Optical flow estimates per-frame motion as a flow map, a direction-coloured visualisation, a magnitude image and a
max_motionreading of the fastest thing in frame. Wire the flow map into Displace to warp along the motion. Unwired, it compares each frame against the previous one; wirereferenceto compare against an explicit frame instead.
Segmentation, which separates subject from background as a mask, is Apple Vision only. See platform notes below.
Optical flow is the odd one out here, and worth knowing separately: it is plain texture maths rather than a model, so it downloads nothing, costs no inference and runs on every platform including the browser. Its quality sets how much motion it can resolve at all, by choosing how far it coarsens the picture before matching: fast takes three steps, balanced five and accurate six, and motion that outruns the setting is simply not found. threshold zeroes flow the solve is not confident about, the ungated confidence still riding in the flow map's blue channel for you to read, and flow_scale multiplies the result to exaggerate or damp the warp downstream. Changing quality rebuilds the chain, so treat it as a decision rather than something to animate. Structure Tensor exposes the same local-orientation measurement flow uses internally, for edge and grain work of your own, and DIS Flow beside it is a measurement spike rather than a tool, kept for comparison.
Point and mask outputs are ordinary data. Body-tracking landmarks flow into the Points node, Iterator, simulations and Expressions exactly like any other Collection.
Trackers
The readers above look at one frame at a time. Trackers remember what they saw and follow it across frames, so an object keeps the same identity as it moves.
Point tracker: follows points through the footage over a sliding window, 16 frames by default. Feed the points into
query_pointsfrom upstream.Blob tracker: segments the subject, finds separate blobs and keeps each one's identity frame to frame, giving centroids, velocities, contours and a mask. Its
modelchooses the segmentation network.Vision tracker: follows an object seeded by
seed_inputfrom a detection node.
Each tracker has an enabled control. Switching it off clears the tracker's memory rather than freezing on the last result, so it starts clean when switched back on. Bypass behaves differently, passing the input straight through.
Both trackers are meant to take their starting points from the viewport, and neither connection is finished: the Point Tracker tool writes tracker_points while the node reads query_points, and the Vision Tracker tool writes regions while the node reads seed_regions. Points placed in the viewport are stored and never read. Seed both from an upstream pin instead.
Platform notes
On macOS, Vision Pose, Vision Segment, Vision Text, Vision Analyze and the Vision tracker use Apple's Vision framework and the Apple Neural Engine. All five are macOS-only, and person segmentation ships nowhere else as a node of its own. On other platforms, Body Tracking covers pose, and the Blob tracker's segmentation model produces a subject mask.
Depth has a second path on macOS that runs through CoreML directly. It is selected by the quality setting rather than by a backend dropdown: fast takes the CoreML path, and balanced and accurate fall back to the cross-platform model.
Vision nodes need a raster on their input, which on a live rig means a camera. A browser does supply one, so a camera reaches a web export, but the models behind depth, body tracking, object detection and normals do not: those run in the desktop app and the DNA Player. Optical flow, Motion Field and Structure Tensor are the exceptions that run anywhere. Inputs lists what each platform supports.
The cross-platform models are not bundled with DNA. The first time you use one, it downloads its weights, checks them against a recorded SHA-256, and caches them on disk. That first cook can pause while the download finishes; every cook after it reads the cache. The Apple Vision readers need no download, and neither do the texture-based motion nodes.
A worked chain
Camera → Body Tracking → scatter geometry on the joints with the Points node → render. As the performer moves, the points track their body and the geometry follows: a glowing skeleton, particles bursting from the hands, text pinned to the face.