Sequences

A sequence marks a time window of data that one machine part captured, so you can train on that window as one example. It names the part, the start and end times, and the components and methods to include, such as a camera’s GetImages and an arm’s JointPositions. It also carries tags that describe the whole window, such as pick-success or demo-3.

Sequences exist to feed sequence datasets, which you train on with a custom training script. To follow the whole path from capture to exported training files, see the sequences tutorial.

How a sequence works

A sequence is a saved filter over data you already captured, not a copy of it. When you read a sequence, Viam returns the images and readings that match its part, resources, and time window. Each sequence belongs to exactly one machine part, though it can include resources on that part’s remotes. This has four consequences:

  • Data must be captured during the window. A sequence only selects data that data capture recorded and synced. If capture wasn’t running on a resource while the sequence was open, the sequence has nothing from that resource.
  • Deleting data removes it from the sequence, not the sequence itself. If you delete images or readings inside the window, they no longer appear in the sequence, but the sequence stays, even if no data is left in it.
  • Deleting a sequence keeps its data. Deleting a sequence removes only the sequence. The images and readings it selected stay in Viam.
  • Editing the window changes the contents. If you change a sequence’s start time, end time, or resources, it selects a different set of data.

A sequence can hold two kinds of data:

  • Images, from the camera methods ReadImage and GetImages, or the vision service method CaptureAllFromCamera. These are the only binary data a sequence accepts. Other binary data, such as point clouds, is rejected. A sequence includes only JPEG and PNG images. Images in other formats, such as depth images, are left out of the sequence and its exports without an error, so a camera that captures only other formats produces a sequence with no images.
  • Readings from any other capture method, such as sensor readings or joint positions, stored as tabular data.

Sequence tags label the sequence as a whole. They are separate from the tags on individual images and readings. The capture-control module’s start_capture command sets both to the same values, and its start_sequence command sets only the sequence tags.

Where sequences fit in data management

Sequences sit between captured data and training:

  1. Capture. The data management service records images and readings from your machine’s components. See Capture and sync data.
  2. Mark sequences. A capture control sensor on the machine opens and closes sequences while the machine runs. You can also create a sequence afterward, from code, over data you already have.
  3. Sync. The data manager uploads captured data, and uploads each sequence after it closes.
  4. Collect. You add sequences to a sequence dataset. See Create a dataset.
  5. Export and train. You export the dataset as Parquet files, or run a custom training script on it. See Sequence dataset format.

When to use a sequence

Use a sequence when a training example is a stretch of time, not a single image. The meaning is in how the images and readings change over the window. For example:

  • Learning from demonstrations. Each sequence is one demonstration of a task, with the camera images and arm joint positions from start to finish.
  • Sequence classification. Each sequence is one event, tagged with its outcome, such as a successful or failed grasp.

Creating sequences

You can create a sequence as a machine records data, or afterward from data you already captured.

From a running machine

A machine records sequences through a capture control sensor. The data management service polls the sensor 10 times per second and reads a sequences list from its readings. A sequence opens the first time its entry appears in the list, and closes when the entry disappears. Changing an open entry’s tags or resources closes that sequence and opens a new one. If the sensor’s Readings call fails, every open sequence closes. After a sequence closes, the data manager uploads it on the next sync.

You can use the capture-control module, which opens a sequence when you send it a command, or write your own sensor. Start with the module. Write your own sensor only if the machine should decide by itself when to record, or if you need more than one sequence open at a time.

To record sequences with the capture-control module:

  1. Add the module’s sensor to your machine, list the components and methods to record, and point the data management service at the sensor. See Add the capture-control sensor and Point the data manager at the sensor.

  2. On the sensor’s Test section, find DoCommand, and send:

    { "start_capture": true, "frequency_hz": 2, "tags": ["demo-1"] }

    This starts capture on the listed components at 2 Hz and opens a sequence tagged demo-1. If capture is already running, send {"start_sequence": true, "tags": ["demo-1"]} instead, to open the sequence without changing capture.

  3. When the event ends, send {"stop_capture": true} to stop capture and close the sequence, or {"stop_sequence": true} to close only the sequence.

  4. Open the DATA tab and click SEQUENCES. The new sequence appears after the next sync.

You can send the same commands from code with DoCommand, or from a terminal with viam machines part run. See the sequences tutorial for both.

To record sequences from your own sensor, see Write your own capture control sensor. For every field the sensor can return, see the capture control sensor readings reference.

From existing data

To mark a window of data you already captured, create the sequence from code. Give it the machine part’s ID, the resources and methods to include, and a start and end time. Tags are optional.

  1. Connect a data client. See Set up a connection for the setup code. CreateSequence only needs an API key with access to the machine part. The other sequence methods, such as ListSequences and AddSequencesToDataset, need an organization API key.
  2. Find the part ID. At the top of the machine’s page, click the Live or Offline status dropdown, then click Part ID to copy it.
  3. Find when the data was captured. In the DATA tab, filter by the machine and the resource, and note the capture times of the first and last images or readings you want.
  4. Create the sequence. The window includes data captured exactly at the start and end times. The examples give both times in UTC. If the times you noted are in another time zone, convert them to UTC first.
from datetime import datetime, timezone

from viam.proto.app.data import SequenceResourceFilter

sequence_id = await data_client.create_sequence(
    part_id="<PART-ID>",
    resources=[
        SequenceResourceFilter(resource_name="my-camera", method_name="GetImages")
    ],
    sequence_tags=["demo-1"],
    start_time=datetime(2026, 9, 29, 14, 0, 0, tzinfo=timezone.utc),
    end_time=datetime(2026, 9, 29, 14, 0, 30, tzinfo=timezone.utc),
)

const sequenceId = await dataClient.createSequence(
  "<PART-ID>",
  [{ resourceName: "my-camera", methodName: "GetImages" }],
  ["demo-1"],
  new Date("2026-09-29T14:00:00Z"),
  new Date("2026-09-29T14:00:30Z"),
);

sequenceID, err := dataClient.CreateSequence(
    ctx,
    "<PART-ID>",
    []app.SequenceResourceFilter{
        {ResourceName: "my-camera", MethodName: "GetImages"},
    },
    []string{"demo-1"},
    time.Date(2026, 9, 29, 14, 0, 0, 0, time.UTC),
    time.Date(2026, 9, 29, 14, 0, 30, 0, time.UTC),
)

View sequences

  1. Go to the DATA tab and click SEQUENCES. The list shows each sequence’s time range, machine part, resources, and tags.
  2. Click a sequence to open it.
  3. Pick a resource, shown as <resource name> ยท <method>, to see its images or readings during the window.
  4. To work with that resource’s data in the rest of the DATA tab, click View in data gallery for images or View in query page for readings. Each opens the resource’s data, filtered to the sequence’s part and time window.

To copy a sequence’s ID, click the Sequence actions menu on its row and select Copy sequence ID.

From code, ListSequences lists the sequences in an organization and GetSequence returns one by ID. The Python, TypeScript, and Go SDKs have these methods. ListSequences returns one page at a time, 50 sequences by default, with a token for the next page. To list every sequence, pass that token back until it comes back empty. GetSequenceBinaryData returns the images inside a sequence, from Python or TypeScript. There is no API for a sequence’s readings. View them in the Viam app, or export a sequence dataset. See the data client API.

Edit and delete sequences

The Viam app can’t edit or delete a sequence, and the CLI has no sequence commands. Use the Python, TypeScript, or Go SDK.

  • Edit: UpdateSequence changes a sequence’s resources, sequence_tags, start_time, or end_time. Only the fields you list in its field mask change, and the field mask is required. The Python SDK’s update_sequence builds the field mask for you from the arguments you pass, and raises an error if you pass none of them. Pass sequence_tags=[] to clear a sequence’s tags.
  • Delete: DeleteSequence deletes a sequence by its ID. The images and readings in its window aren’t deleted.

See the data client API for each method’s parameters.

Add sequences to a dataset

To train on sequences, collect them into a sequence dataset. On a sequence’s detail page, click Add to dataset, or call AddSequencesToDataset from code. See Create a dataset to create the dataset, add sequences, and export it.

Limitations

  • Images are the only binary data. See How a sequence works.
  • A sequence belongs to one machine part. It can include resources on that part’s remotes. Name them without the remote prefix, for example cam, not my-remote:cam. To combine data from several parts, record a sequence on each part and add them all to one dataset.
  • A crash loses the open sequence. If viam-server stops uncleanly while a sequence is open, the data manager can’t tell when the sequence ended. It moves the sequence to failed/sequences/ in the capture directory and doesn’t upload it. A normal shutdown closes open sequences so they upload on the next sync.
  • Editing and deleting need an SDK. The Viam app and the CLI can’t edit or delete a sequence.
  • Managed training doesn’t accept sequence datasets. Train on them with a custom training script.

Next steps