Building Pipelines

Introduction

Lobster pipelines are built from typed steps that pass data forward using a reference syntax. Each step's output becomes available to subsequent steps through explicit references like $step.stdout and $step.json. This lesson covers the piping syntax, step references, and how to write complete workflow definitions.

Key Concepts

• Step References: The $steps.<id>.<output> syntax for accessing previous step outputs
• $step.stdout: The standard text output of a step
• $step.json: The parsed JSON output of a step, enabling structured data passing
• Piping Syntax: How data flows from one step to the next through explicit input declarations
• .lobster Format: The file extension for Lobster workflow definitions (supports JSON and YAML)

Real World Context

A data engineering team builds a pipeline that extracts user analytics from their database, transforms the data into a report format, and loads it into their business intelligence dashboard. Each step produces output that the next step consumes. Using Lobster's reference syntax, they create a clean, readable pipeline where data flow is explicit and verifiable.

Deep Dive

Step Reference Syntax

Every step's output is available to subsequent steps via the $steps namespace:

yaml
steps:
  - id: fetch-users
    type: http
    config:
      url: "https://api.example.com/users"
      method: GET

  - id: count-active
    type: transform
    input: $steps.fetch-users.json
    config:
      expression: ".users | map(select(.active)) | length"

The second step references $steps.fetch-users.json, which provides the parsed JSON response from the first step. The transform step then applies a jq-like expression to count active users.

stdout vs json References

Steps produce both text and structured output:

yaml
steps:
  - id: run-tests
    type: shell
    config:
      command: "npm test -- --json"

  - id: parse-results
    type: transform
    input: $steps.run-tests.json
    config:
      expression: ".testResults | { passed: .numPassedTests, failed: .numFailedTests }"

  - id: format-report
    type: template
    input: $steps.run-tests.stdout
    config:
      template: "Test output:\n{{input}}"

The parse-results step uses .json to work with structured data. The format-report step uses .stdout to include the raw text output in a template. Choose the reference type based on whether you need structured or raw data.

Complete Pipeline Example

A full .lobster workflow file:

yaml
# deploy-staging.lobster
name: deploy-staging
version: 2.1.0
trigger: manual

steps:
  - id: pull-latest
    type: shell
    config:
      command: "git pull origin main"

  - id: run-tests
    type: shell
    config:
      command: "npm test"
    dependsOn: pull-latest

  - id: build
    type: shell
    config:
      command: "npm run build"
    dependsOn: run-tests

  - id: deploy
    type: shell
    config:
      command: "kubectl apply -f k8s/staging/"
    dependsOn: build

  - id: verify
    type: http
    config:
      url: "https://staging.example.com/health"
      expectedStatus: 200
    dependsOn: deploy

This pipeline pulls the latest code, runs tests, builds the project, deploys to Kubernetes, and verifies the deployment health. Each step explicitly declares its dependency using dependsOn, making the execution order unambiguous.

JSON Format

The same pipeline in JSON:

json
{
  "name": "deploy-staging",
  "version": "2.1.0",
  "trigger": "manual",
  "steps": [
    {
      "id": "pull-latest",
      "type": "shell",
      "config": { "command": "git pull origin main" }
    },
    {
      "id": "run-tests",
      "type": "shell",
      "config": { "command": "npm test" },
      "dependsOn": "pull-latest"
    }
  ]
}

Both JSON and YAML formats are fully supported. Choose whichever your team prefers.

Common Pitfalls

• Referencing a step that has not run yet: Step references are resolved in order. You cannot reference $steps.deploy.stdout from the build step because deploy runs after build.
• Using .json on non-JSON output: If a step produces plain text, referencing .json will cause a parse error. Use .stdout for text output.
• Forgetting dependsOn for parallel steps: Without explicit dependencies, Lobster may attempt to run steps in parallel if they have no data dependencies.

Best Practices

• Use descriptive step IDs that make references self-documenting (e.g., $steps.fetch-users.json is clearer than $steps.step1.json).
• Prefer .json references for structured data to enable downstream transformations and validations.
• Add dependsOn explicitly even when data references already imply the dependency, for clarity.

Summary

• Step references use $steps.<id>.<output> syntax to pass data between steps
• .stdout provides raw text output; .json provides parsed structured data
• Workflow files use the .lobster extension and support both JSON and YAML formats
• The dependsOn field explicitly declares execution order between steps
• Always use descriptive step IDs to make references self-documenting