Python Sandbox

Run Python code in a sandboxed environment

Overview

You can run Python code in a sandboxed environment using the Python Sandbox tool. This tool allows you to execute Python code securely and perform advanced and flexible operations.

Key Features

• PYTHON_SANDBOX_RUN
  • This tool allows you to run Python code in a sandboxed environment.
  • You can add additional packages or libraries in the configuration.

Example: Merge Step Results in Python

- id: input1
  name: Input 1
  tool: INPUT_TEXT
  input:
    - name: value
      value: "1"
- id: input2
  tool: INPUT_TEXT
  input:
    - name: value
      value: "2"
- id: merge
  tool: PYTHON_SANDBOX_RUN
  needs:
    - input1
    - input2
  input:
    - name: code
      value: int("{{steps.input1.result}}") + int("{{steps.input2.result}}")
    - name: data_type
      value: NUMBER

Large tables: tables

A step output travels between steps as JSON. A DataFrame with hundreds of thousands of rows does not survive that: serializing it exhausts the sandbox’s memory before the step can return.

Every Python step is given a tables helper for this. tables.write(df) publishes the frame to storage and returns a small marker — a few hundred bytes naming the stored table — which becomes the step’s result. The next step calls tables.read(...) on that marker and gets the DataFrame back. The rows never become JSON.

- id: load
  tool: PYTHON_SANDBOX_RUN
  input:
    - name: code
      value: |
        import json
        import pandas as pd
        df = pd.read_csv(steps.input.result.url)
        json.dumps(tables.write(df))
    - name: data_type
      value: OBJECT
- id: pivot
  tool: PYTHON_SANDBOX_RUN
  needs: [load]
  input:
    - name: code
      value: |
        import json
        df = tables.read(steps.load.result.data)
        pv = df.pivot_table(index=["rep_month", "branch_cd"],
                            columns="product_cd", values="amt_00",
                            aggfunc="sum")
        json.dumps(tables.write(pv, name="pivot"))
    - name: data_type
      value: OBJECT

The marker is a dict, so the step has to return it the way it returns any other structured result: json.dumps(...) as the last expression, with data_type: OBJECT. Without that the step’s data_type defaults to STRING and the next step receives the dict’s Python text, which tables.read cannot open.

tables.write(df, name="result", fmt="csv", index=None) publishes a frame and returns its marker.

• name — the stored object’s name, so one step can publish several tables. Letters, digits, ., _ and - only, with no .. and no leading or trailing dot.
• fmt — "csv" (the default) or "parquet". Parquet needs pyarrow, which is not installed in every sandbox image; tables.write says so if it is missing.
• index — whether to write the frame’s index as columns. The default, None, writes it when it carries data: the grouping keys of groupby(...).agg(...) or pivot_table, a DatetimeIndex from resample. A plain positional index is dropped, so a filtered frame does not gain a stray column. Pass True or False to decide it yourself.

Columns must be single-level. Flatten a pivot with several values first: df.columns = ["_".join(str(part) for part in column) for column in df.columns.to_flat_index()].

df must be a DataFrame, not a Series. df.groupby([...])["amt"].sum() returns a Series — call .to_frame() on it first; its grouping keys are a named index and are written as columns.

tables.read(marker) returns the DataFrame. The frame comes back flat: everything tables.write serialized is a column, including levels that were an index on the way out. marker["__jinbatable"]["columns"] lists them, along with rows and sizeBytes, so a later step can see the shape without fetching the table. rows is also checked against the frame that actually parsed: an object cut short mid-upload is refused here rather than handed on as a quietly shorter table.

tables.is_table(value) reports whether a value is a marker, for a step that accepts either a table or inline rows.

Size limit. One tables.write call is bounded by what the deployment accepts in a single upload, and that bound belongs to the deployment rather than to the tool: on the managed service the platform ingress rejects a request body over 32 MiB, while a self-hosted deployment may have a different ceiling, or none, and may instead cut a slow upload on a request timeout. tables.write reports whichever happened — a frame refused for its size, or a connection closed before any status came back — and names the limit when the deployment has declared one. Either way: aggregate or filter before writing, or publish several smaller tables with different names. Reading is not bounded this way: a table already in storage can be read back whole, subject only to the sandbox’s memory — which for a very large frame is the binding constraint, because tables.read materializes the whole table at once.

Notes:

• A table is readable by later steps of the same flow, including reused steps replaying an earlier run’s output. It is not readable from another flow.
• Inside a forEach, each iteration gets its own storage slot, so every iteration can call tables.write(df) with the default name without overwriting its siblings.
• tables.read also accepts a marker arriving as the item of a forEach.
• Small results need none of this: a step that returns rows still returns rows.