Orchestration

Orchestration is how Vendo decides what runs, when it runs and in which order. Vendo orchestrates your Sources, Data Studio models, Identity Resolution, Attribution, Audiences and Destinations for you. You set the triggers. Vendo does the rest.

Use this page to learn why data arrives when it does and what depends on what. It also tells you what Vendo does when a job is slow or fails. In the app, the jobs show in Data → Integrations → Logs, and the graph shows in Data → Overview.

How Vendo Orchestrates, in Short

1. The graph. Vendo keeps one dependency graph for each account. The graph records which data each node reads and writes. See The Graph.
2. The trigger. A schedule, Sync Now, an API call or new upstream data starts a run. See What Starts a Run.
3. The job. Each run is a job. The job waits in a queue as Queued, runs in its own isolated container, and ends as Completed, Warning, Failed or Canceled. See How a Job Runs.
4. The next nodes. When a job ends as Completed or Warning, Vendo starts the downstream nodes that are ready. See How a Run Moves Through the Graph.
5. Errors. Vendo retries temporary errors, stops a stream that fails again and again, and fails a job that stops. See Errors, Retries and Stuck Jobs.

The Graph

The graph is a directed acyclic graph (DAG). Each item in the graph is a node. Each connection between two nodes is an edge.

A node is one part of your setup that runs and writes data.

An edge means “depends on”. Node B depends on node A when B reads data that A writes. The edge goes from A to B, so data flows from left to right.

You do not draw the edges. Vendo finds them from your setup:

• A model depends on each table that its SQL reads. It also depends on the Sources and models that you select as its inputs.
• A Destination depends on the model, Source or table that it sends.
• Identity Resolution depends on the models and Source tables in its identity mappings.

A SQL model that is saved as a view is a live view. It is in the graph, but it never runs, because it is always current. A node that reads a live view depends on the nodes that the view reads.

How Vendo Builds the Graph

Vendo builds and updates the graph automatically. It rebuilds the graph of an account when you do one of these actions:

• Add, change, pause or delete a Source.
• Add, change or delete a model. This includes its SQL, its inputs and its run policy.
• Add, change or delete a Destination. This includes the data that it sends and its trigger.
• Turn Identity Resolution on or off, or change an identity mapping.

Vendo also rebuilds the graph when the recorded lineage of a table changes.

One Graph for Each Account

Each account has its own graph. Vendo builds the graphs one account at a time. A node can read only data that belongs to its own account. Vendo rejects any input or output that crosses to a different account. No part of a graph is shared between accounts.

Nodes That Vendo Cannot Connect

Sometimes Vendo cannot connect a node correctly. Then Vendo stops the node and records the reason. The node shows Paused on the graph. The reason shows under Needs attention.

When you remove the cause, Vendo rebuilds the graph and the node becomes active again.

What Starts a Run

Each node has a trigger. The trigger sets what starts a new run of the node.

The trigger of a Source is on the Source page. The trigger of a model is in its Run policy card. The trigger of a Destination is in its Trigger setting.

Vendo sets the trigger of some nodes for you:

• Identity Resolution runs when any of its inputs has new data.
• Attribution runs on a daily schedule.
• A live view never runs.

How a Run Moves Through the Graph

When a node finishes, Vendo checks the nodes that depend on it. Vendo starts each downstream node that is ready. Nodes that do not depend on each other can run at the same time.

A downstream node starts only after its upstream node finishes successfully. A run that ends with the status Warning can still count as a success.

When an Upstream Job Fails

Vendo does not start the nodes that depend on a failed job. They do not run with incomplete data. Their tables keep the data from their last successful run. They can start again after the upstream node has a successful run.

A node with Wait for all upstreams also waits for its other inputs. It runs only when each required input has new data since its last successful run.

When an Upstream Job Is Slow

Downstream nodes wait for the upstream job to finish. Vendo does not start them while the upstream job is Queued or Running.

A node with Run on any upstream is different. It can run when another of its inputs has new data. That run uses the last successful data of the slow input.

When a Node Is Paused

A paused node does not run. Its downstream nodes also stop, because no active node writes their input. The graph shows them as Paused, with missing_input_producer under Needs attention. When you resume the node, Vendo rebuilds the graph and the downstream nodes become active again.

One Run at a Time

A node has a maximum of one active job (Queued, Pending or Running). If a new trigger arrives while the node has a job, Vendo does not start a second job at the same time:

• If the schedule time of a Source comes while its sync still runs, Vendo skips that scheduled run. It does not put the run in a queue behind the active sync. The next scheduled run starts as usual.
• If you click Sync Now while a sync runs, Vendo does not start a second sync. While a Source syncs, its button shows Syncing….

No Duplicate Runs

Vendo records which upstream data each run used. If a run already exists for the same node and the same input data, Vendo does not start another one. A repeated signal does not load the same data twice.

Manual Runs

Sync Now starts a run of a Source or a Destination. When a manual run finishes successfully, Vendo starts the downstream nodes that are ready, the same as after a scheduled run.

How a Job Runs

From Trigger to Job

A worker takes the job from the queue and runs it in its own isolated container. The job then shows Running.

Schedules

• A Source schedule repeats at a fixed interval, for example every 6 hours or every day.
• Vendo uses the time zone of the Source. If the Source has no time zone, Vendo uses UTC.
• When you change a schedule, Vendo keeps the start time of the existing schedule.
• A Source that is paused, or that uses Manual only, has no scheduled runs.

Queues and Capacity

Vendo runs jobs in two queues:

A large backfill uses only the backfill queue. Thus it does not delay your routine syncs. A backfill can wait for another backfill.

When many jobs are due at the same time, a job can stay Queued for some time before it starts.

Incremental Syncs

A routine sync reads only the new data since the last sync:

• The sync starts where the last sync stopped.
• It does not read the last 10 minutes, because the platform can still change that data.
• For some platforms, the sync reads a short period again to get late data, for example late conversions.

Backfills

A backfill loads historical data:

1. Vendo divides the date range into parts, for example 7, 30 or 90 days. The size of a part depends on the platform.
2. Vendo loads the parts one at a time, from the oldest to the newest.
3. If a part fails, the backfill stops. When you run the backfill again, Vendo skips the parts that loaded.
4. Downstream models and Destinations do not run after each part. They run one time, after the backfill ends as Completed or Warning.

Errors, Retries and Stuck Jobs

Temporary and Permanent Errors

Retries

• Models and Destinations: Vendo retries a step that has a temporary error up to 3 times. The wait before each retry increases, to a maximum of 2 minutes. If the platform tells Vendo how long to wait, Vendo waits that long, to a maximum of 15 minutes.
• Source syncs: Vendo does not retry a failed sync in the same job. The next scheduled sync is the retry. To retry now, click Sync Now.

A Stream That Fails Again and Again

A Source can have many streams, for example orders, customers and products. If one stream fails 15 times in a row, Vendo stops that stream. The other streams continue.

• About one time each day, Vendo tries the stopped stream again.
• The job shows the error of the stopped stream.
• To reset the count, reactivate the Source after you fix the cause.

Stuck Jobs

Cancel a Job

A Queued, Pending or Running job shows Cancel.

• When you cancel a running job, Vendo asks the job to stop. The job can show Running for a short time, until it stops. Then it shows Canceled.
• A job that did not start yet changes to Canceled immediately.
• The downstream nodes of a canceled sync do not run.

Completed, Warning or Failed

Downstream nodes run after a Warning, the same as after Completed.

Notifications

Vendo can tell you when:

• The credentials of a Source expire.
• A Source or a model becomes unhealthy, or recovers.
• A data check fails.
• A Destination sync fails.

You can also get a daily summary. Select in-app, email or Slack. See Notifications.

Example: Shopify and Meta Ads to a Destination

In this example, a SQL model joins Shopify orders with Meta Ads spend and writes a table. A Destination sends that table to an App.

If the model uses Run on any upstream, it runs after each Shopify sync and after each Meta Ads sync. Each run uses the latest successful data of the other Source.

See the Graph in Vendo

Data → Overview

Open Data → Overview to see the data graph of your account. If the account has no data graph yet, the page shows the older flow view.

The graph puts the nodes in lanes: Sources, Models, Vendo functions and Destinations. Each card shows the node name, its type, a health dot, its schedule or trigger and the time of its last run. When a node has an issue, the card also shows the first issue. Exports to the same Destination App show as one card.

The color of an edge shows the health of the node that feeds it.

Use these controls on the graph:

• Find a node… dims the cards that do not match the name that you type.
• Issues only shows the nodes that have an issue and their direct neighbors. The number of nodes with an issue shows next to it.
• Fullscreen makes the graph fill the screen. Exit returns to the page.
• Scroll, or drag the background, to move around the graph. Use the zoom buttons and the mini map for large graphs.
• You cannot drag the cards.

Click a card to show its lineage and open its details. The graph highlights the path of that node, and the bottom strip shows Showing lineage of with the node name. The details panel shows these sections:

• Needs attention: the issues of the node.
• Run: Last run, Result, Took, Trigger and State.
• Reads from: the upstream nodes. Click one to move to it.
• Feeds: the downstream nodes.
• Open source or Open details: opens the page of the node, where you can find Sync Now.

Click the background to close the details and clear the lineage.

The Older Flow View and Home

The Home dashboard shows the older flow view when your account has at least one Source, model or Destination. Data → Overview also shows it when your account has no data graph.

In the older flow view, you can drag a node to a different position. Scroll or drag the background to move around. You cannot zoom.

To save the older flow view as a PNG image, click Download as image (the download icon in the top-right corner). The image has a Vendo frame and a legend of the status colors: Active, Syncing, Pending, Paused and Error. The file name starts with data-flow and includes the date.

Logs and Job Statuses

Each run of a node is a job. Open Data → Integrations → Logs to see the jobs of your account. You can filter the jobs by status.

Open a job to see its details. A Queued, Pending or Running job shows Cancel. See Cancel a Job.

Troubleshooting

Find Why Data Is Late

1. Open Data → Overview.
2. Turn on Issues only.
3. Click the node that has late data.
4. Read Needs attention and Run in the details panel.
5. In Reads from, click the first upstream node that is not Healthy.
6. Click Open source or Open details.
7. Open Data → Integrations → Logs and find the last job of that node.
8. Fix the cause of the error.
9. If the upstream node is a Source, click Sync Now on the Source page.

When the upstream run completes, Vendo starts the downstream nodes that are ready.

Common Causes

Related

• Sources
• Destinations
• Data Studio
• Identity Merge
• Activity Logs
• Data Health