Code Generation

Spage's code generation is one of its most powerful features, transforming Ansible playbooks into high-performance, standalone Go programs. This process compiles your playbook into an executable binary that can run without any external dependencies.

Overview

When you run spage generate, Spage performs several transformations:

1. Parse and validate the playbook YAML
2. Build a dependency graph of tasks and their relationships
3. Generate Go source code that represents the execution graph
4. Create a complete Go program with a main function
5. Format the code using go fmt for readability

The generated code is a complete, self-contained Go program that can be compiled and executed independently.

Generated Code Structure

Every generated program follows the same structure:

package main

import (
    "os"
    "github.com/AlexanderGrooff/spage/cmd"
    "github.com/AlexanderGrooff/spage/pkg"
    "github.com/AlexanderGrooff/spage/pkg/common"
    "github.com/AlexanderGrooff/spage/pkg/modules"
)

var GeneratedGraph = pkg.Graph{
    RequiredInputs: []string{},
    Vars:           map[string]interface{}{},
    Nodes:          [][]pkg.GraphNode{},
    Handlers:       []pkg.GraphNode{},
}

func main() {
    localCmd := cmd.NewLocalExecutorCmd(GeneratedGraph)
    temporalCmd := cmd.NewTemporalExecutorCmd(GeneratedGraph)

    rootCmd := localCmd
    rootCmd.AddCommand(temporalCmd)

    if err := rootCmd.Execute(); err != nil {
        common.LogError("Failed to run playbook", map[string]interface{}{
            "error": err.Error(),
        })
        os.Exit(1)
    }
}

Core Data Structures

Graph

The GeneratedGraph is the central data structure that represents your entire playbook:

type Graph struct {
    RequiredInputs []string               // Variables that must be provided at runtime
    Vars           map[string]interface{} // Global playbook variables
    Nodes          [][]GraphNode          // Execution levels (parallel tasks)
    Handlers       []GraphNode            // Handler tasks
    PlaybookPath   string                 // Path to the original playbook
}

GraphNode

Each GraphNode represents either a task or a collection of tasks:

type GraphNode interface {
    String() string
    ToCode() string
    GetVariableUsage() ([]string, error)
    ConstructClosure(*HostContext, *config.Config) *Closure
    ExecuteModule(*Closure) chan TaskResult
    RevertModule(*Closure) chan TaskResult
    ShouldExecute(*Closure) (bool, error)
}

Task

Individual tasks are represented by the Task struct:

type Task struct {
    *TaskParams
}

type TaskParams struct {
    Id           int                 `json:"id"`
    Name         string              `json:"name"`
    Module       string              `json:"module"`
    Params       ModuleInput         `json:"params"`
    Register     string              `json:"register"`
    When         JinjaExpressionList `json:"when"`
    Before       string              `json:"before"`
    After        string              `json:"after"`
    Become       bool                `json:"become"`
    BecomeUser   string              `json:"become_user"`
    IgnoreErrors JinjaExpression     `json:"ignore_errors"`
    FailedWhen   JinjaExpressionList `json:"failed_when"`
    ChangedWhen  JinjaExpressionList `json:"changed_when"`
    // ... additional fields
}

MetaTask

Complex task structures like blocks, includes, and roles are represented by MetaTask:

type MetaTask struct {
    *TaskParams
    Children []GraphNode `json:"children"` // Regular tasks in the block
    Rescue   []GraphNode `json:"rescue"`   // Rescue tasks (error handling)
    Always   []GraphNode `json:"always"`   // Always tasks (cleanup)
}

Execution Levels

Spage organizes tasks into execution levels for parallel execution:

Nodes: [][]pkg.GraphNode{
    []pkg.GraphNode{
        // Level 0: Tasks that can run in parallel
        &pkg.Task{...},
        &pkg.Task{...},
    },
    []pkg.GraphNode{
        // Level 1: Tasks that depend on Level 0 completion
        &pkg.Task{...},
    },
    []pkg.GraphNode{
        // Level 2: Tasks that depend on Level 1 completion
        &pkg.Task{...},
    },
}

Tasks within the same level can execute in parallel, while tasks in different levels execute sequentially.

info

When running in sequential mode, the execution levels are not used. The order of execution is instead based on the task ID.

Module Input Generation

Each module's parameters are converted to strongly-typed Go structs:

// Example: command module
&pkg.Task{TaskParams: &pkg.TaskParams{
    Id: 1, 
    Name: "install package", 
    Module: "command", 
    Register: "", 
    Params: pkg.ModuleInput{
        Actual: modules.CommandInput{
            Execute: "apt install nginx", 
            Revert: "apt remove nginx"
        }
    }
}}

Variable Handling

Spage automatically detects variable usage and generates the RequiredInputs list:

RequiredInputs: []string{
    "package_name",
    "config_path",
},

Variables that are:

• Used in templates but not provided by tasks
• Not defined in playbook variables
• Not system facts

Are automatically added to RequiredInputs and must be provided at runtime.

Example: Simple Playbook

Given this playbook:

- name: Install nginx
  apt:
    name: nginx
    state: present

- name: Start nginx
  systemd:
    name: nginx
    state: started
    enabled: true

Spage generates:

var GeneratedGraph = pkg.Graph{
    RequiredInputs: []string{},
    Vars:           map[string]interface{}{},
    Nodes: [][]pkg.GraphNode{
        []pkg.GraphNode{
            &pkg.Task{TaskParams: &pkg.TaskParams{
                Id: 1, 
                Name: "Install nginx", 
                Module: "apt", 
                Register: "", 
                Params: pkg.ModuleInput{
                    Actual: modules.AptInput{
                        Name: "nginx", 
                        State: "present"
                    }
                }
            }},
        },
        []pkg.GraphNode{
            &pkg.Task{TaskParams: &pkg.TaskParams{
                Id: 2, 
                Name: "Start nginx", 
                Module: "systemd", 
                Register: "", 
                Params: pkg.ModuleInput{
                    Actual: modules.SystemdInput{
                        Name: "nginx", 
                        State: "started", 
                        Enabled: true
                    }
                }
            }},
        },
    },
    Handlers: []pkg.GraphNode{},
}

Example: Complex Block with Error Handling

For a more complex playbook with blocks and error handling:

- name: Deploy application
  block:
    - name: Create config directory
      file:
        path: /etc/myapp
        state: directory

    - name: Copy configuration
      copy:
        src: config.yaml
        dest: /etc/myapp/config.yaml

  rescue:
    - name: Cleanup on failure
      file:
        path: /etc/myapp
        state: absent

  always:
    - name: Log deployment attempt
      debug:
        msg: "Deployment completed"

Spage generates a MetaTask with nested structure:

[]pkg.GraphNode{
    &pkg.MetaTask{TaskParams: &pkg.TaskParams{
        Id: 1, 
        Name: "Deploy application", 
        Module: "block"
    }, Children: []pkg.GraphNode{
        &pkg.Task{TaskParams: &pkg.TaskParams{
            Id: 1, 
            Name: "Create config directory", 
            Module: "file", 
            Params: pkg.ModuleInput{
                Actual: modules.FileInput{
                    Path: "/etc/myapp", 
                    State: "directory"
                }
            }
        }},
        &pkg.Task{TaskParams: &pkg.TaskParams{
            Id: 2, 
            Name: "Copy configuration", 
            Module: "copy", 
            Params: pkg.ModuleInput{
                Actual: modules.CopyInput{
                    Src: "config.yaml", 
                    Dest: "/etc/myapp/config.yaml"
                }
            }
        }},
    }, Rescue: []pkg.GraphNode{
        &pkg.Task{TaskParams: &pkg.TaskParams{
            Id: 3, 
            Name: "Cleanup on failure", 
            Module: "file", 
            Params: pkg.ModuleInput{
                Actual: modules.FileInput{
                    Path: "/etc/myapp", 
                    State: "absent"
                }
            }
        }},
    }, Always: []pkg.GraphNode{
        &pkg.Task{TaskParams: &pkg.TaskParams{
            Id: 4, 
            Name: "Log deployment attempt", 
            Module: "debug", 
            Params: pkg.ModuleInput{
                Actual: modules.DebugInput{
                    Msg: "Deployment completed"
                }
            }
        }},
    }},
}

Conditional Execution

Tasks with when conditions generate Jinja expression structures:

- name: Install package
  apt:
    name: "{{ package_name }}"
  when: ansible_os_family == "Debian"

Generates:

&pkg.Task{TaskParams: &pkg.TaskParams{
    Id: 1, 
    Name: "Install package", 
    Module: "apt", 
    Params: pkg.ModuleInput{
        Actual: modules.AptInput{
            Name: "{{ package_name }}"
        }
    },
    When: pkg.JinjaExpressionList{
        pkg.JinjaExpression{Expression: "ansible_os_family == \"Debian\""}
    }
}}

Variable Dependencies

When tasks use variables from other tasks, Spage automatically creates dependencies:

- name: Get file info
  stat:
    path: /etc/hosts
  register: hosts_info

- name: Display file info
  debug:
    msg: "File exists: {{ hosts_info.stat.exists }}"

This creates a dependency where the second task must wait for the first:

Nodes: [][]pkg.GraphNode{
    []pkg.GraphNode{
        // Level 0: File info gathering
        &pkg.Task{TaskParams: &pkg.TaskParams{
            Id: 1, 
            Name: "Get file info", 
            Module: "stat", 
            Register: "hosts_info",
            Params: pkg.ModuleInput{
                Actual: modules.StatInput{Path: "/etc/hosts"}
            }
        }},
    },
    []pkg.GraphNode{
        // Level 1: Depends on hosts_info variable
        &pkg.Task{TaskParams: &pkg.TaskParams{
            Id: 2, 
            Name: "Display file info", 
            Module: "debug", 
            Params: pkg.ModuleInput{
                Actual: modules.DebugInput{
                    Msg: "File exists: {{ hosts_info.stat.exists }}"
                }
            }
        }},
    },
}

Benefits of Code Generation

Performance

• Compiled execution: No runtime parsing or interpretation
• Type safety: All parameters are strongly typed at compile time
• Optimized execution: Go's compiler optimizes the generated code

Reliability

• Dependency validation: Missing variables are caught at generation time
• Syntax validation: Invalid playbooks are rejected before generation
• Type checking: Module parameters are validated against their schemas

Portability

• Single binary: No external dependencies required
• Cross-platform: Compile for any target architecture
• Offline execution: No network requirements for playbook execution

Debugging

• Source code visibility: Generated code is human-readable
• Go tooling: Use standard Go debugging and profiling tools
• Clear error messages: Go's error handling provides detailed failure information

Best Practices

Variable Management

• Define all required variables in your playbook or inventory
• Use descriptive variable names that make the generated code readable
• Avoid complex variable dependencies that create deep execution levels

Module Selection

• Use native Spage modules when available for better performance
• Fall back to ansible_python module for unsupported Ansible modules
• Consider the trade-offs between compatibility and performance

Code Organization

• Keep playbooks focused and modular
• Use blocks to group related tasks logically
• Minimize the use of complex Jinja expressions that generate verbose code

Testing

• Test generated code in your target environment before deployment
• Use spage generate in CI/CD pipelines to catch issues early
• Validate that all required inputs are properly provided at runtime

Advanced Features

Custom Execution Engines

The generated code supports multiple execution engines:

func main() {
    localCmd := cmd.NewLocalExecutorCmd(GeneratedGraph)
    temporalCmd := cmd.NewTemporalExecutorCmd(GeneratedGraph)
    
    rootCmd := localCmd
    rootCmd.AddCommand(temporalCmd)
    
    // Choose execution engine at runtime
    rootCmd.Execute()
}

Handler Integration

Handlers are automatically integrated into the generated code:

Handlers: []pkg.GraphNode{
    &pkg.Task{TaskParams: &pkg.TaskParams{
        Id: 10, 
        Name: "restart nginx", 
        Module: "systemd", 
        IsHandler: true,
        Params: pkg.ModuleInput{
            Actual: modules.SystemdInput{
                Name: "nginx", 
                State: "restarted"
            }
        }
    }},
}

Real-World Example

Here's an example of actual generated code from a complex playbook with blocks, error handling, and conditional execution:

var GeneratedGraph = pkg.Graph{
    RequiredInputs: []string{},
    Vars:           map[string]interface{}{},
    Nodes: [][]pkg.GraphNode{
        []pkg.GraphNode{
            &pkg.MetaTask{TaskParams: &pkg.TaskParams{
                Id: 1, 
                Name: "block with rescue only", 
                Module: "block"
            }, Children: []pkg.GraphNode{
                &pkg.Task{TaskParams: &pkg.TaskParams{
                    Id: 0, 
                    Name: "task 1", 
                    Module: "command", 
                    Params: pkg.ModuleInput{
                        Actual: modules.CommandInput{
                            Execute: "/bin/false", 
                            Revert: ""
                        }
                    }
                }},
            }, Rescue: []pkg.GraphNode{
                &pkg.Task{TaskParams: &pkg.TaskParams{
                    Id: 0, 
                    Name: "rescue file", 
                    Module: "file", 
                    Params: pkg.ModuleInput{
                        Actual: modules.FileInput{
                            Path: "/tmp/spage/rescue_file.txt", 
                            State: "touch"
                        }
                    }
                }},
            }},
        },
        []pkg.GraphNode{
            &pkg.Task{TaskParams: &pkg.TaskParams{
                Id: 2, 
                Name: "rescue file exists", 
                Module: "stat", 
                Register: "rescue_file_stat", 
                Params: pkg.ModuleInput{
                    Actual: modules.StatInput{
                        Path: "/tmp/spage/rescue_file.txt"
                    }
                }
            }},
        },
    },
    Handlers: []pkg.GraphNode{},
}

This example shows:

• MetaTask with rescue handling: A block that fails and triggers its rescue section
• Variable registration: The rescue task creates a file that's later checked
• Execution levels: Tasks are organized into sequential execution levels
• Complex nesting: Block structure with children and rescue tasks

This code generation approach makes Spage uniquely powerful, combining the familiarity of Ansible syntax with the performance and reliability of compiled Go code.