# Add a recording example Source: https://docs.optexity.com/api-reference/add_example/add-a-recording-example /openapi.json post /api/v1/add_example Creates a new recording/automation example under the authenticated user. # Update a recording example Source: https://docs.optexity.com/api-reference/add_example/update-a-recording-example /openapi.json post /api/v1/update_example Updates the automation, description, and endpoint name of an existing recording example. # Callback Reference Source: https://docs.optexity.com/api-reference/callback Complete reference for all callback schemas ### Callback Response The callback response is a JSON object with the following fields: * `status`: string indicating the status of the callback. Can be `queued`, `allocated`, `running`, `success`, `failed`, `cancelled` * `task_id`: string indicating the task ID * `recording_id`: string indicating the recording ID * `endpoint_name`: string indicating the endpoint name * `output_data`: list of output data objects. Each object can be a `dict` or a `string`. * `error`: string indicating the error message. If there is no error, it will be `null`. * `final_screenshot`: base64 encoded string of the final screenshot * `downloads`: list of download objects. Each object has a `url` and a `filename` field. * `downloads_with_metadata`: optional list included only when at least one download has metadata. Each object has `url`, `filename`, and `metadata` (`null` when that file has none). Omitted entirely when no download metadata was set. ```json theme={null} { "task_id": "123", "recording_id": "456", "endpoint_name": "login-flow", "output_data": [{ "name": "email", "value": "test@example.com" }, "test_output_data"], "error": "An error occurred", "status": "success", "final_screenshot": "base64 encoded string", "downloads": [ { "url": "signed url", "filename": "filename" } ], "downloads_with_metadata": [ { "url": "signed url", "filename": "filename", "metadata": { "doc_id": "123" } } ] } ``` # Get Recordings Source: https://docs.optexity.com/api-reference/dashboard/get-recordings /openapi.json get /api/v1/get_recordings # Get Tasks Source: https://docs.optexity.com/api-reference/dashboard/get-tasks /openapi.json get /api/v1/get_tasks # Get Workspaces Source: https://docs.optexity.com/api-reference/dashboard/get-workspaces /openapi.json get /api/v1/get_workspaces # Submit an automation task Source: https://docs.optexity.com/api-reference/inference/submit-an-automation-task /openapi.json post /api/v1/inference Queue a browser automation task for asynchronous execution. Returns immediately with a `task_id`; the automation runs in the background. Use `priority` to control queue position within your login group — lower numbers run first, `null`/omitted runs last. # Update a recording Source: https://docs.optexity.com/api-reference/recording/update-a-recording /openapi.json patch /api/v1/update_recording/{recording_id} Updates the endpoint name, description, or other metadata of an existing recording. # Live Stream Endpoint Source: https://docs.optexity.com/api-reference/stream-endpoint Stream the live browser view of a running task into your own dashboard ## GET /api/v1/tasks//stream Returns a short-lived WebSocket URL that streams the live browser view of a running task. Pair it with a noVNC client (e.g. `@novnc/novnc`) to embed the live screen in your own dashboard. ## Description While a task is running, its browser is rendered in a headed Chromium inside the worker container and exposed over VNC via `websockify`. This endpoint resolves a task to a WebSocket URL (`wss://...`) you can pass directly to a noVNC `RFB` client to render the live view in a ``. The URL is only valid while the task is actively running. Once the task finishes (success / failure / cancellation) the upstream WebSocket closes and the endpoint returns an error. ## Authentication Requires an API key in the `x-api-key` header. The same key used for `POST /api/v1/inference` works here. ## Parameters ### Path Parameters * **`task_id`** `string` *required* UUID of the task whose live stream you want. Obtain this from the `task_id` field in your task creation response or task listing. ### Headers * **`x-api-key`** `string` *required* Your Optexity API key. ## Code Examples ### Fetch the stream URL ```bash cURL theme={null} curl -X GET \ "https://api.optexity.com/api/v1/tasks//stream" \ -H "x-api-key: $OPTEXITY_API_KEY" ``` ```python Python theme={null} import os import requests task_id = "bf982ff9-f0af-4598-97be-2eb99c917eb0" resp = requests.get( f"https://api.optexity.com/api/v1/tasks/{task_id}/stream", headers={"x-api-key": os.environ["OPTEXITY_API_KEY"]}, ) resp.raise_for_status() stream_url = resp.json()["stream_url"] print(stream_url) ``` ```javascript JavaScript theme={null} const taskId = "bf982ff9-f0af-4598-97be-2eb99c917eb0"; const res = await fetch( `https://api.optexity.com/api/v1/tasks/${taskId}/stream`, { headers: { "x-api-key": process.env.OPTEXITY_API_KEY } }, ); if (!res.ok) { const err = await res.json().catch(() => ({})); throw new Error(err.detail || `HTTP ${res.status}`); } const { stream_url } = await res.json(); console.log(stream_url); ``` ## Success Response (200 OK) ```json theme={null} { "stream_url": "wss://stream.optexity.com/tasks//websockify?token=..." } ``` **Response Fields:** | Field | Type | Description | | - | - | - | | `stream_url` | string | A `wss://` WebSocket URL that speaks the [noVNC](https://github.com/novnc/noVNC) wire protocol. Pass it directly to a `RFB` constructor. | The URL is short-lived (it embeds a signed token) and is only useful while the task is running. Re-fetch a fresh URL each time you (re)connect. ## Error Responses ### 401 Unauthorized ```json theme={null} { "detail": "Invalid or missing API key" } ``` ### 404 Not Found ```json theme={null} { "detail": "Task not found" } ``` Returned when the task UUID doesn't exist or doesn't belong to your account. ### 409 Conflict ```json theme={null} { "detail": "Stream not available: task is not running" } ``` Returned when the task is queued, completed, failed, or cancelled — there is no live browser to stream. ## Frontend Integration Use `@novnc/novnc`'s `RFB` class. It attaches to a plain `
` and renders the live view into a `` inside it. The full reference implementation is in `LiveStreamViewer.jsx` in the Optexity dashboard. > [!NOTE] > > Install with `npm install @novnc/novnc`. ### Reference implementations ```jsx React theme={null} import { useEffect, useRef, useState } from "react"; import RFB from "@novnc/novnc"; const INFERENCE_API_BASE_URL = "https://api.optexity.com"; async function getStreamUrl(taskId, apiKey) { const res = await fetch( `${INFERENCE_API_BASE_URL}/api/v1/tasks/${taskId}/stream`, { headers: { "x-api-key": apiKey } }, ); if (!res.ok) { const err = await res.json().catch(() => ({})); throw new Error(err.detail || `HTTP ${res.status}`); } return res.json(); } export function LiveStreamViewer({ taskId, apiKey }) { const containerRef = useRef(null); const rfbRef = useRef(null); const [state, setState] = useState("loading"); const [error, setError] = useState(null); useEffect(() => { let cancelled = false; (async () => { try { const { stream_url } = await getStreamUrl(taskId, apiKey); if (cancelled || !containerRef.current) return; setState("connecting"); const rfb = new RFB(containerRef.current, stream_url, { wsProtocols: ["binary"], credentials: {}, }); rfb.viewOnly = true; rfb.scaleViewport = true; rfb.resizeSession = false; rfb.addEventListener("connect", () => setState("connected")); rfb.addEventListener("disconnect", (e) => { setState(e.detail?.clean ? "disconnected" : "error"); if (!e.detail?.clean) setError("Connection lost"); }); rfb.addEventListener("credentialsrequired", () => rfb.sendCredentials({}), ); rfb.addEventListener("securityfailure", () => { setError("Security negotiation failed"); setState("error"); }); rfbRef.current = rfb; } catch (err) { if (!cancelled) { setError(err.message); setState("error"); } } })(); return () => { cancelled = true; if (rfbRef.current) { rfbRef.current.disconnect(); rfbRef.current = null; } }; }, [taskId, apiKey]); return (
{state !== "connected" && (
{state === "loading" && "Initializing stream…"} {state === "connecting" && "Connecting to browser…"} {state === "disconnected" && "Stream ended"} {state === "error" && (error || "Stream error")}
)}
); } ``` ```html Vanilla JS theme={null} Optexity Live Stream
Initializing stream…
``` ```vue Vue 3 theme={null} ``` ```typescript Angular theme={null} import { Component, ElementRef, Input, OnDestroy, OnInit, ViewChild, } from "@angular/core"; // @ts-ignore — @novnc/novnc ships ESM without bundled types import RFB from "@novnc/novnc"; const INFERENCE_API_BASE_URL = "https://api.optexity.com"; type StreamState = "loading" | "connecting" | "connected" | "disconnected" | "error"; @Component({ selector: "live-stream-viewer", template: `
Initializing stream… Connecting to browser… Stream ended {{ error || "Stream error" }}
`, }) export class LiveStreamViewerComponent implements OnInit, OnDestroy { @Input() taskId!: string; @Input() apiKey!: string; @ViewChild("container", { static: true }) container!: ElementRef; state: StreamState = "loading"; error: string | null = null; private rfb: any = null; private cancelled = false; async ngOnInit() { try { const res = await fetch( `${INFERENCE_API_BASE_URL}/api/v1/tasks/${this.taskId}/stream`, { headers: { "x-api-key": this.apiKey } }, ); if (!res.ok) { const err = await res.json().catch(() => ({})); throw new Error(err.detail || `HTTP ${res.status}`); } const { stream_url } = await res.json(); if (this.cancelled) return; this.state = "connecting"; this.rfb = new RFB(this.container.nativeElement, stream_url, { wsProtocols: ["binary"], credentials: {}, }); this.rfb.viewOnly = true; this.rfb.scaleViewport = true; this.rfb.resizeSession = false; this.rfb.addEventListener("connect", () => (this.state = "connected")); this.rfb.addEventListener("disconnect", (e: any) => { this.state = e.detail?.clean ? "disconnected" : "error"; if (!e.detail?.clean) this.error = "Connection lost"; }); this.rfb.addEventListener("credentialsrequired", () => this.rfb.sendCredentials({}), ); this.rfb.addEventListener("securityfailure", () => { this.error = "Security negotiation failed"; this.state = "error"; }); } catch (err: any) { if (!this.cancelled) { this.error = err.message; this.state = "error"; } } } ngOnDestroy() { this.cancelled = true; this.rfb?.disconnect(); this.rfb = null; } } ``` ### Key `RFB` options | Option | Recommended | Why | | - | - | - | | `viewOnly` | `true` | Read-only preview — disables input forwarding so a dashboard viewer can't accidentally take over the browser. | | `scaleViewport` | `true` | Scales the remote framebuffer to fit your container without resizing the actual browser. | | `resizeSession` | `false` | Don't try to negotiate a different remote display size — the worker runs at a fixed 1920×1080. | | `wsProtocols` | `["binary"]` | Required for `websockify`'s binary subprotocol. | | `credentials` | `{}` | The stream is auth'd via the signed token in the URL — no VNC password is needed. | ### Lifecycle tips * Fetch a **fresh** `stream_url` on every (re)connect attempt — the token is short-lived. * Call `rfb.disconnect()` on component unmount to release the underlying WebSocket. * Handle the `disconnect` event: `e.detail.clean === true` typically means the task finished gracefully (show "Stream ended"); `clean === false` is a genuine network/auth failure (show retry). * The endpoint returns `409` while the task is queued — poll task status first and only call `/stream` once the task is `running`. ## Related * [Inference Endpoint](/api-reference/inference-endpoint) — Submitting tasks * [noVNC](https://github.com/novnc/noVNC) — Upstream client library # Get callback payload Source: https://docs.optexity.com/api-reference/task_details/get-callback-payload /openapi.json post /api/v1/get_callback_data Returns the full task result payload in the same shape sent to callback URLs. Pass `task_id` as a single string for the single-task shape (`{success, data}`, 404 if not found), or as an array for the batch shape (`{success, data: [...], found_task_count, not_found_task_ids}`, always 200). Batches are capped at 50 `task_id`s per request. `endpoint_name` is optional — when supplied it's echoed back onto every payload as-is; when omitted it's resolved server-side per task from its recording. Pass full_logs=true to also include for_loop_status. # Get task download URLs Source: https://docs.optexity.com/api-reference/task_details/get-task-download-urls /openapi.json get /api/v1/get_downloads Returns presigned download URLs for all files produced by the task. # Get task output data Source: https://docs.optexity.com/api-reference/task_details/get-task-output-data /openapi.json get /api/v1/get_output_data Returns the structured output data produced by the automation. # Get task status Source: https://docs.optexity.com/api-reference/task_details/get-task-status /openapi.json get /api/v1/task_status Returns the current status and lifecycle timestamps for a task. # Get task status (single or batch) Source: https://docs.optexity.com/api-reference/task_details/get-task-status-single-or-batch /openapi.json post /api/v1/task_status Returns the current status and lifecycle timestamps for one or more tasks. Pass `task_id` as a single string for the single-task shape (`{success, task}`, 404 if not found), or as an array for the batch shape (`{success, tasks, found_task_count, not_found_task_ids}`, always 200). Batches are capped at 50 `task_id`s per request. # Trigger task callback Source: https://docs.optexity.com/api-reference/task_details/trigger-task-callback /openapi.json post /api/v1/initiate_callback Accepts the task result for delivery to the configured callback URLs and returns without waiting for them. Delivery retries with exponential backoff for up to 5 minutes per URL. Returns immediately if no callback URL is set. # Validate an API key Source: https://docs.optexity.com/api-reference/validate-an-api-key /openapi.json post /api/v1/validate_api_key Checks whether an API key belongs to a registered user. Returns `valid: true` on success. # Agentic Tasks Source: https://docs.optexity.com/docs/action-types/agentic-tasks Using AI agents for complex browser interactions For interactions too complex or unpredictable for static automations, use AI agents that autonomously navigate and interact based on goals. ## When to Use | Use Agentic For | Use Static Actions For | | - | - | | Unpredictable UI layouts | Known, stable elements | | Complex navigation paths | Simple click/type actions | | Handling popups/modals | Performance-critical paths | | Sites with frequent changes | Cost-sensitive automations | | CAPTCHAs and verification | Deterministic workflows | ## AgenticTask Use `agentic_task` when AI should autonomously accomplish a goal: ```json theme={null} { "interaction_action": { "agentic_task": { "task": "Navigate to settings and enable two-factor authentication", "max_steps": 15, "backend": "browser_use", "use_vision": false, "keep_alive": true } } } ``` ### Properties | Property | Type | Default | Description | | - | - | - | - | | `task` | `str` | Required | Natural language goal description | | `max_steps` | `int` | Required | Maximum actions the agent can take | | `backend` | `"browser_use" \| "browserbase"` | Required | Agent backend | | `use_vision` | `bool` | `False` | Include screenshots for agent | | `keep_alive` | `bool` | `True` | Keep browser session after task | ### Uploading a downloaded file A `download_file` node earlier in the same run saves a URL under `/tmp` and passes that path to this agent as an allowed upload file. Reference it in `task` with the generated variable, which defaults to `{downloaded_file_path[0]}`. See [Downloads & Files](/docs/advanced/downloads-files) for the action schema. Tell the agent to use its `upload_file` action on the page's file input. Clicking a control that opens the operating-system file dialog does not attach the file. ### max\_steps Guidelines | Task Complexity | Suggested max\_steps | | - | - | | Simple (1-2 clicks) | 3-5 | | Medium (navigate + fill form) | 10-15 | | Complex (multi-page workflow) | 20-30 | > [!WARNING] > > Higher `max_steps` means longer execution time and higher LLM costs. ### Writing Good Task Descriptions **Good:** ```json theme={null} {"task": "Click 'Account Settings' in the sidebar, scroll down, click 'Security'"} ``` ```json theme={null} {"task": "1. Close any popups 2. Click search 3. Search 'laptop' 4. Click first result"} ``` **Poor:** ```json theme={null} {"task": "Do the thing"} ``` ```json theme={null} {"task": "Complete the form"} ``` ### Vision Mode Enable `use_vision` for visual elements without good text labels: ```json theme={null} { "agentic_task": { "task": "Click on the red 'Sale' banner", "max_steps": 5, "use_vision": true } } ``` | Use Vision For | Avoid Vision For | | - | - | | Image-based navigation | Text-based navigation | | Visual verification | Speed-critical tasks | | Elements without ARIA labels | Cost minimization | *** ## CloseOverlayPopup Specialized action for dismissing popups, modals, and overlays: ```json theme={null} { "interaction_action": { "close_overlay_popup": { "max_steps": 5 } } } ``` ### Default Behavior | Property | Default | | - | - | | `task` | Comprehensive popup dismissal prompt | | `max_steps` | `5` | | `use_vision` | `True` | | `keep_alive` | `True` | ### What It Handles * Cookie consent banners * Privacy policy notices * Newsletter signup prompts * Age verification gates * Promotional popups * Blocking overlays *** ## Variables in Agentic Tasks Use parameter substitution in task descriptions: ```json theme={null} { "agentic_task": { "task": "Search for '{search_query[0]}' and filter to items under ${price_max[0]}", "max_steps": 10 } } ``` *** ## Combining with Static Actions Best pattern: use static actions for predictable steps, agentic for uncertainty: ```json theme={null} [ { "type": "action_node", "interaction_action": { "input_text": { "command": "get_by_label(\"Email\")", "input_text": "{email[0]}" } } }, { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Sign In\")" } } }, { "type": "action_node", "interaction_action": { "agentic_task": { "task": "Navigate to Reports and find the Monthly Summary", "max_steps": 10 } } }, { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Download\")", "expect_download": true } } } ] ``` *** ## Best Practices | Practice | Recommendation | | - | - | | Start with static | Use agentic only where needed | | Keep tasks focused | Break complex goals into smaller tasks | | Start low on max\_steps | Increase if agent can't complete | | Review execution logs | Refine task descriptions based on results | | Use vision selectively | Only when visual context is necessary | When field labels are not known at authoring time, fetch values first with [Dynamic Form Mapping](/docs/action-types/dynamic-form-mapping) and pass `{form_values[0]}` into the agentic task prompt. # Assert Locator Node Source: https://docs.optexity.com/docs/action-types/assertion-action Check Playwright locator visibility and store the result as a boolean variable Use `assert_locator_node` to check whether a specific element is present on the page and store the result. It evaluates a Playwright locator within a timeout and writes a boolean into `output_variable_name` (`true` if the assertion passes, `false` if it does not). You then branch on that variable with an [`if_else_node`](/docs/building-automations/if-else-node). This is the lowest-overhead way to capture element presence—no LLM call, no extraction step, just a direct visibility check that becomes a reusable variable. ## Structure ```json theme={null} { "type": "assert_locator_node", "locator": "get_by_role(\"alert\", name=\"Error\")", "assertion": "to_be_visible", "output_variable_name": "error_visible", "timeout": 5.0 } ``` The node above stores `error_visible = [true]` or `error_visible = [false]`. Reference it later as `{error_visible[0]}`. ## Properties | Property | Type | Default | Description | | - | - | - | - | | `type` | `"assert_locator_node"` | Required | Node discriminator | | `locator` | `str` | Required | Playwright locator command (`page.` style) | | `assertion` | `"to_be_visible" \| "to_be_hidden"` | Required | Condition to test | | `output_variable_name` | `str` | Required | Variable to store the boolean result, as a single-element list (e.g. `[true]`) | | `timeout` | `float` | `5.0` | Seconds to wait for the assertion before treating it as failed | The result is stored as a single-element list, matching the convention used by other variables, so you reference it with index `[0]` (e.g. `{error_visible[0]}`). ## Assertions | Assertion | Result is `true` when | | - | - | | `to_be_visible` | The element exists in the DOM and is visible within `timeout` seconds | | `to_be_hidden` | The element is absent or hidden within `timeout` seconds | If the assertion does not pass within `timeout`, the variable is set to `false` (no error is raised). If the locator does not resolve at all, the result is also `false`. ## Examples ### Dismiss an element only when it appears Check whether an error alert is visible, then dismiss it with a following `if_else_node`: ```json theme={null} [ { "type": "assert_locator_node", "locator": "get_by_role(\"alert\", name=\"Error\")", "assertion": "to_be_visible", "output_variable_name": "error_visible", "timeout": 5.0 }, { "type": "if_else_node", "condition": "{error_visible[0]}", "if_nodes": [ { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Dismiss\")" } } } ], "else_nodes": [] } ] ``` ### Wait for a spinner to disappear, then branch Wait up to 10 seconds for a loading spinner to be hidden, then submit or fail: ```json theme={null} [ { "type": "assert_locator_node", "locator": "get_by_test_id(\"loading-spinner\")", "assertion": "to_be_hidden", "output_variable_name": "spinner_gone", "timeout": 10.0 }, { "type": "if_else_node", "condition": "{spinner_gone[0]}", "if_nodes": [ { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Submit\")" } } } ], "else_nodes": [ { "type": "action_node", "misc_action": { "fail_state": { "message": "Loading spinner did not disappear within 10 seconds" } } } ] } ] ``` ### Variable substitution in the locator `locator` supports `{variable[index]}` substitution, so you can target elements dynamically: ```json theme={null} { "type": "assert_locator_node", "locator": "get_by_text(\"{order_id[0]}\")", "assertion": "to_be_visible", "output_variable_name": "order_found", "timeout": 5.0 } ``` ## assert\_locator\_node vs if\_else\_node | | `assert_locator_node` | `if_else_node` | | - | - | - | | **Role** | Captures element visibility as a boolean variable | Branches on a Python expression over a variable | | **Condition** | Playwright locator visibility | Python expression on a variable | | **Requires extraction?** | No | Yes — needs a variable to branch on (e.g. from `assert_locator_node` or an `extraction_action`) | | **Cost** | Zero (no LLM) | Zero (expression eval) | | **Best for** | Element present/absent checks | Acting on a captured boolean or comparing extracted values | `assert_locator_node` produces the boolean; `if_else_node` consumes it. Pair them to run steps conditionally on element presence, or use the variable in any later condition. ## Locator Syntax The `locator` field uses the same Playwright command syntax as interaction actions—omit the leading `page.`: ```json theme={null} { "locator": "get_by_role(\"button\", name=\"Submit\")" } { "locator": "get_by_label(\"Email\")" } { "locator": "get_by_text(\"Order confirmed\")" } { "locator": "locator(\"#error-banner\")" } ``` For full locator syntax guidance see [Locators](/docs/advanced/locators). # Count Locator Action Source: https://docs.optexity.com/docs/action-types/count-locator-action Count how many elements a Playwright locator matches on the current page Use `misc_action.count_locator` to count Playwright locator matches on the current page and store the integer in `generated_variables`—useful for pagination checks, empty-state branches, or capping loops. ## Overview * **Use when**: You need the match count as a variable (e.g. decide whether to paginate, skip an empty table, or compare against a threshold) without iterating over the matches. * **Execution**: Resolves `locator` against the live page, waits for the first match to attach (optional timeout), then waits until the match count is stable for 1s before storing the result. * **Same counting semantics as locator for-loops**: See [For Loop Node](/docs/building-automations/for-loop-node) for `locator_timeout` and the stable-count wait. ## Properties | Property | Type | Default | Description | | - | - | - | - | | `locator` | `str` | Required | Playwright locator command evaluated against `page`, e.g. `get_by_role("row")` | | `name` | `str` | Required | Key written into `generated_variables` as a single-element list `[count]`. Accepts placeholders (e.g. `"rows_{index}"`) | | `locator_timeout` | `float` | `5.0` | Seconds to wait for the first match to attach before counting; `0` skips the attach wait | | `output_variable_name` | `str \| None` | `None` | When set, also append the count to task `output_data` under this key. Accepts placeholders | ## JSON Example ```json theme={null} { "type": "action_node", "misc_action": { "count_locator": { "locator": "get_by_role(\"row\")", "name": "row_count" } } } ``` After this action, `{row_count[0]}` is available in subsequent nodes (including `if_else_node` conditions and `set_variable` expressions). To also include the count in the task output payload, set `output_variable_name`: ```json theme={null} { "misc_action": { "count_locator": { "locator": "get_by_role(\"row\")", "name": "row_count", "output_variable_name": "row_count" } } } ``` ## Waiting for Matches Playwright's `count()` does not auto-wait. Without `locator_timeout`, an asynchronously rendered table can count as empty. Defaults match locator for-loops: wait up to 5s for the first match, then require the count to stay unchanged for 1s so streaming rows are included. ```json theme={null} { "misc_action": { "count_locator": { "locator": "locator(\"table tbody tr\")", "name": "result_rows", "locator_timeout": 10.0 } } } ``` Zero matches is a valid result: the action stores `0` (with a warning) rather than failing the run. ## Count Locator vs Locator For-Loop | | `misc_action.count_locator` | `for_loop_node` with `locator` | | - | - | - | | Purpose | Store match count as a variable | Iterate once per match | | Output | `generated_variables[name] = [count]` | Body runs `count` times; `{locator[index]}` expands to `.nth(N)` | | Typical use | Branching, thresholds, logging | Process every row/item | > [!TIP] > > Use `count_locator` when you only need the number. Use a locator for-loop when you need to act on each match. # Dynamic Form Mapping Source: https://docs.optexity.com/docs/action-types/dynamic-form-mapping Extract dynamic form fields, ask your endpoint for values, and fill the form with an agentic task Use `dynamic_form_mapping_action` when a form's fields are not known at authoring time. The action extracts the field keys from the current page, POSTs them to your endpoint, and stores the returned mapping so a later `agentic_task` can fill the form. ## Overview * **Use when**: Questionnaires, eligibility forms, or other pages whose labels change per run, and your backend already knows how to map those labels to values. * **How it works**: LLM extraction reads the page, Optexity POSTs the extracted keys (plus optional screenshot, axtree, and live-stream URL) to your endpoint, then waits for a JSON mapping in the HTTP response. * **Next step**: An `agentic_task` whose prompt references `{form_values[0]}` (or whatever you set as `output_variable_name`). This action does **not** send the live browser session. Your endpoint receives JSON only. ## Properties | Property | Type | Default | Description | | - | - | - | - | | `extraction_instructions` | `str` | Required | Prompt telling the LLM which fields to extract | | `extraction_format` | `dict` | Required | Schema of the extracted keys, same as `extraction_action.llm` | | `source` | `list["axtree" \| "screenshot"]` | `["axtree"]` | Page context used for extraction | | `callback_url` | `FormMappingCallbackUrl` | Required | Endpoint that receives extracted keys and returns values | | `include_screenshot` | `bool` | `true` | Attach a base64 PNG of the current page | | `full_page_screenshot` | `bool` | `false` | Capture the full scrolled page instead of the viewport, for both LLM extraction (`source` includes `"screenshot"`) and the payload. Ignored for the payload when `include_screenshot` is `false` | | `include_axtree` | `bool` | `false` | Attach the page accessibility tree | | `include_live_stream_url` | `bool` | `false` | Attach a dashboard URL to the live browser stream | | `max_wait_time` | `float` | `60` | Seconds to wait for the HTTP response. Must be `> 0` and `<= 120`. Supported range is about 30–60s (ALB and most proxies idle-timeout around 60s) | | `output_variable_name` | `str` | `"form_values"` | Generated variable that stores the returned mapping | | `llm_model_name` | `str \| None` | `None` | Vertex model ID. Falls back to the task model | | `llm_provider` | `str \| None` | `None` | Deprecated — prefix the provider in `llm_model_name` instead | Default `before_sleep_time` is `3.0` seconds and default `end_sleep_time` is `0.0`, matching extraction actions. ## JSON Example ```json theme={null} { "type": "action_node", "dynamic_form_mapping_action": { "extraction_instructions": "Extract the question fields from the patient questionnaire. Ignore the provider questionnaire.", "extraction_format": { "form_fields": "List[str]" }, "callback_url": { "url": "https://customer.example.com/form-values" } } } ``` ## Extraction Schema `extraction_format` describes the **keys** you send to your endpoint, not the values that come back. Use one list or one object of lists — the same type-annotated dict as [LLM extraction](/docs/action-types/extraction-action). ```json theme={null} "extraction_format": { "form_fields": "List[str]" } ``` ```json theme={null} "extraction_instructions": "Extract the question fields from the patient questionnaire. Ignore the provider questionnaire." ``` The extracted JSON is POSTed as `extracted_fields` with no extra transformation, so your mapper can key off whatever schema you defined. ## Callback URL `callback_url` is this action's endpoint. It is separate from the task completion callback. | Field | Type | Description | | - | - | - | | `url` | `str` | HTTPS endpoint that receives the POST and returns the mapping | | `api_key` | `str \| None` | Sent as `x-api-key`. Cannot be combined with username/password | | `username` | `str \| None` | HTTP basic auth username | | `password` | `str \| None` | HTTP basic auth password | `url` is SSRF-checked (no private/internal hosts). Redirects are not followed. `{variable[0]}` substitution works on all four fields. ## Request Payload Optexity POSTs JSON. Optional fields are present and set to `null` when disabled, so the schema is stable: ```json theme={null} { "task_id": "abc123", "extracted_fields": { "form_fields": ["Date of Birth", "Member ID"] }, "screenshot": "", "axtree": null, "live_stream_url": null } ``` | Field | When set | Notes | | - | - | - | | `extracted_fields` | Always | LLM extraction result, same shape as `extraction_format` | | `screenshot` | `include_screenshot: true` | Raw base64 PNG (no `data:image/png;base64,` prefix). Viewport unless `full_page_screenshot` is `true` | | `axtree` | `include_axtree: true` | Accessibility tree string | | `live_stream_url` | `include_live_stream_url: true` | Dashboard page at `/task-logs?task_id=...` (no `&hitl=true`). The live tab works for any running task without a HITL notify; `&hitl=true` only adds the Done button used by `human_in_loop_action`. Requires an Optexity login; useful for a human, not for a machine mapper | Turn screenshot sending off when your mapper only needs the extracted keys: ```json theme={null} { "dynamic_form_mapping_action": { "include_screenshot": false, "full_page_screenshot": false, "include_axtree": false, "include_live_stream_url": false } } ``` ## Response Contract The HTTP response body **is** the mapping: a JSON object whose values are strings, numbers, or booleans. Nested objects, arrays, and `null` are rejected. Serialized size is capped at 64KB. ```json theme={null} { "Date of Birth": "1990-01-01", "Member ID": "ABC123" } ``` Optexity stores that body as a JSON string in a one-element list: ```text theme={null} generated_variables.form_values = ['{"Date of Birth":"1990-01-01","Member ID":"ABC123"}'] ``` Use `{form_values[0]}` in later nodes. The variable is overwritten each time this action runs (including inside a loop). You do not need to declare `form_values` under `generated_parameters`. Non-2xx, redirect, non-JSON, invalid shape, oversized body, or a timeout raises `DynamicFormMappingException` and fails the task. Keep `max_wait_time` in the 30–60s range; values above 120s are rejected. This action holds one HTTP request — it is not a HITL-style poll, so it is not suitable for a human sitting on the live stream deciding values. ## Filling the Form Follow the mapping action with an `agentic_task` whose prompt includes the stored JSON: ```json theme={null} { "type": "action_node", "interaction_action": { "agentic_task": { "task": "Fill every visible field using this mapping. Do not submit.\n{form_values[0]}", "max_steps": 15, "backend": "browser_use" } } } ``` See [Agentic Tasks](/docs/action-types/agentic-tasks) for `max_steps` and task phrasing. ## Complete Example ```json theme={null} { "url": "https://portal.example.com/questionnaire", "parameters": { "input_parameters": { "username": ["user@example.com"] }, "generated_parameters": {} }, "nodes": [ { "type": "action_node", "dynamic_form_mapping_action": { "extraction_instructions": "Extract the question fields from the patient questionnaire. Ignore the provider questionnaire.", "extraction_format": { "form_fields": "List[str]" }, "callback_url": { "url": "https://customer.example.com/form-values", "api_key": "your-api-key" }, "include_screenshot": true, "full_page_screenshot": false, "max_wait_time": 60, "output_variable_name": "form_values" } }, { "type": "action_node", "interaction_action": { "agentic_task": { "task": "Fill the form using {form_values[0]}. Do not submit.", "max_steps": 15, "backend": "browser_use" } } }, { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Submit\")", "prompt_instructions": "Click submit after the form is filled" } } } ] } ``` ## Multiple Mappings Set `output_variable_name` when two mapping actions run in the same automation so the second does not overwrite the first: ```json theme={null} { "output_variable_name": "patient_form_values" } ``` Then reference `{patient_form_values[0]}` in the matching `agentic_task`. # Extraction Actions Source: https://docs.optexity.com/docs/action-types/extraction-action Capturing data from web pages Extraction actions capture data from web pages during automation—essential for scraping, validation, and feeding dynamic values into subsequent actions. ## Extraction Types | Type | Purpose | Best For | | - | - | - | | `llm` | AI-powered structured data extraction | Tables, forms, text content | | `locator` | Extract text directly via Playwright locator | Single values, fast extraction, no LLM tokens | | `network_call` | Capture API/AJAX responses | API data, JSON responses | | `api_call` | Call an external REST API directly | Webhooks, triggering jobs, polling for async results | | `screenshot` | Save visual snapshot | Receipts, proofs, visual records | | `state` | Capture page state (URL, title, storage, cookies) | Debugging auth, navigation validation | | `python_script` | Run custom Python code to extract data | Complex parsing, computed values | | `two_fa_action` | Wait for and extract 2FA code | 2FA codes | ## Common Properties These properties sit on the `extraction_action` object itself, regardless of extraction type: | Property | Type | Default | Description | | - | - | - | - | | `allow_none` | `bool` | `false` | Allow `output_variable_names` to resolve to `null` without failing the workflow | ### `allow_none` By default, if you declare `output_variable_names` and any of those variables comes back `null`, the workflow **fails immediately** with an error like: ``` Extraction produced null value(s) in variable 'price' and allow_none is False: [None] ``` Set `allow_none: true` on the `extraction_action` when the value may legitimately be absent on some pages: ```json theme={null} { "type": "action_node", "extraction_action": { "allow_none": true, "llm": { "source": ["axtree"], "extraction_format": { "product_name": "str", "price": "str", "availability": "str" }, "extraction_instructions": "Extract product details from the product page. Return null for any field not visible on the page.", "output_variable_names": ["price"] } } } ``` > [!WARNING] > > Only use `output_variable_names` when you need to reference the extracted value in a later action (e.g. `{price[0]}`). If you only need the value in the output data and won't reference it downstream, omit `output_variable_names` entirely — the null-check is skipped and the value is simply stored in `OutputData`. *** ## LLM Extraction The most powerful extraction method. Uses AI to parse page content into structured data. ```json theme={null} { "extraction_action": { "llm": { "source": ["axtree"], "extraction_format": { "product_name": "str", "price": "str", "availability": "str" }, "extraction_instructions": "Extract product details from the product page" } } } ``` ### Properties | Property | Type | Default | Description | | - | - | - | - | | `source` | `list["axtree" \| "screenshot"]` | `["axtree"]` | Data sources to analyze | | `extraction_format` | `dict` | Required | Expected output structure | | `extraction_instructions` | `str` | Required | What to extract | | `output_variable_names` | `list[str]` | `None` | Store specific keys as workflow variables for use in later actions. Only set this when you need to reference the value downstream. Null values fail the workflow unless `allow_none: true` is set. | | `include_full_page` | `bool` | `false` | Capture the full scrollable page (not just the visible viewport) when `source` includes `"screenshot"` | | `llm_model_name` | `str \| None` | `None` | Vertex model ID, e.g. `"gemini-4-argon"`. Falls back to the task model, then `LLM_MODEL`. See [Model Configuration](/docs/advanced/model-configuration) | ### Source Selection | Source | Best For | | - | - | | `["axtree"]` | Text, tables, forms (default, fastest) | | `["screenshot"]` | Charts, images, visual layouts | | `["axtree", "screenshot"]` | Complex pages needing both | ### Extraction Format Define output structure with type hints: ```json theme={null} { "extraction_format": { "title": "str", "items": "List[str]", "count": "str" } } ``` > [!NOTE] > > Only `str` and `List[str]` are supported types. ### Storing as Variables Use `output_variable_names` to make extracted values available for subsequent actions: ```json theme={null} { "extraction_action": { "llm": { "extraction_format": { "order_ids": "List[str]", "total": "str" }, "extraction_instructions": "Extract order IDs from the table", "output_variable_names": ["order_ids"] } } } ``` After this action, use `{order_ids[0]}`, `{order_ids[index]}`, or iterate with `for_loop_node`. ### Writing Good Instructions **Good examples:** ```json theme={null} {"extraction_instructions": "Extract all authorization numbers from the Auth Nbr column in the Authorizations table"} ``` ```json theme={null} {"extraction_instructions": "From the patient info section, extract: name (shown as 'Name:'), DOB, and member ID"} ``` **Poor examples:** ```json theme={null} {"extraction_instructions": "Get the data"} ``` ```json theme={null} {"extraction_instructions": "Extract the numbers"} ``` > [!TIP] > > Be specific about where data appears, what it looks like, and expected format. *** ## Locator Extraction Extract text from a specific element on the page using a Playwright locator — no LLM tokens consumed. If the locator fails, it can fall back to LLM extraction. ```json theme={null} { "extraction_action": { "locator": { "command": "get_by_role(\"cell\", name=\"Authorization Number\").locator(\"+ td\")", "output_variable_name": "auth_number", "extraction_format": { "auth_number": "str" } } } } ``` ### Properties | Property | Type | Default | Description | | - | - | - | - | | `command` | `str` | Required | Playwright locator command to find the element | | `output_variable_name` | `str \| None` | `None` | Variable name to store the extracted text. When omitted, the value is stored under an auto-generated key (`node{index}_output`) and `extraction_format` must contain exactly one field. Accepts loop placeholders (e.g. `"patient_{row}"`) to store one value per iteration — see [For Loop Node](/docs/building-automations/for-loop-node) | | `extraction_format` | `dict` | Required | Must contain `output_variable_name` as a key (or exactly one field when `output_variable_name` is omitted). Templated keys are expanded alongside `output_variable_name` | | `extraction_instructions` | `str \| None` | `None` | LLM fallback instructions if the locator fails | | `llm_model_name` | `str \| None` | `None` | Vertex model ID used for the fallback, e.g. `"gemini-4-argon"`. Falls back to the task model, then `LLM_MODEL` | | `llm_provider` | `str \| None` | `None` | Deprecated — prefix the provider in `llm_model_name` instead | ### Fallback Behavior If the locator fails to find the element or find text content, two outcomes are possible: * **With `extraction_instructions`** — falls back to LLM extraction automatically * **Without `extraction_instructions`** — variable is set to `None` ```json theme={null} { "extraction_action": { "locator": { "command": "get_by_label(\"Total Amount\")", "output_variable_name": "total_amount", "extraction_format": { "total_amount": "str" }, "extraction_instructions": "Extract the total amount shown on the invoice page" } } } ``` ### When to Use Locator vs LLM | Situation | Use | | - | - | | Element has a stable, reliable locator | `locator` (faster, no cost) | | Page structure changes often | `llm` | | Single known value to extract | `locator` with LLM fallback | | Multiple fields at once | `llm` | > [!TIP] > > Always provide `extraction_instructions` as a fallback. This makes the extraction resilient if the page structure changes. *** ## Network Call Extraction Capture data from API requests and responses: ```json theme={null} { "extraction_action": { "network_call": { "url_pattern": "https://inference-api.optexity.com/orders" } } } ``` ### Properties | Property | Type | Default | Description | | - | - | - | - | | `url_pattern` | `str \| None` | `None` | URL substring to match | | `extract_from` | `"request" \| "response"` | `None` | Extract from request or response | | `download_from` | `"request" \| "response"` | `None` | Download as file | | `download_filename` | `str \| None` | Auto-generated | Filename for download | > [!TIP] > > Use `network_call` to *intercept* requests the page already makes. Use `api_call` (below) to *initiate* your own HTTP request to any external endpoint. *** ## API Call Extraction Make an outbound REST API call directly from the automation—useful for hitting webhooks, triggering backend jobs, enriching data from a third-party service, or polling an async endpoint until it's ready. The full response is stored as a variable for use in later actions. ```json theme={null} { "extraction_action": { "api_call": { "url": "https://inference-api.optexity.com/v1/orders", "method": "GET" } } } ``` ### Properties | Property | Type | Default | Description | | - | - | - | - | | `url` | `str` | Required | Endpoint to call | | `method` | `"GET" \| "POST" \| "PUT" \| "PATCH" \| "DELETE"` | `"GET"` | HTTP method | | `headers` | `dict[str, str]` | `{}` | Request headers | | `body` | `dict \| str \| None` | `None` | Request body. A `dict` is sent as JSON; a `str` is sent as raw content | | `query_params` | `dict[str, str]` | `{}` | URL query parameters | | `output_variable_names` | `list[str]` | `["api_result"]` | Variable name(s) to store the response under | | `timeout` | `float` | `30.0` | Request timeout in seconds | | `poll_condition` | `str \| None` | `None` | Expression to re-poll until satisfied (see [Polling](#polling)) | | `poll_interval` | `float` | `5.0` | Seconds to wait between poll attempts | | `max_poll_attempts` | `int` | `10` | Maximum number of poll attempts | ### Response Shape The stored variable holds a dict with the following keys: | Key | Type | Description | | - | - | - | | `status_code` | `int \| null` | HTTP status code (`null` on a connection error or timeout) | | `headers` | `dict[str, str]` | Response headers | | `body` | `any` | Parsed JSON if the response is JSON, otherwise the raw text | | `error` | `str` | Present only on failure—`"timeout"` or `"http_error"` | ### Using the Response Reference fields of the response in later actions with dot-path syntax: `{var.field}`, `{var.nested.field}`, and `{var.array[0].field}`. Both object keys and array indices are supported. ```json theme={null} { "extraction_action": { "api_call": { "url": "https://inference-api.optexity.com/v1/customers", "method": "POST", "headers": { "Authorization": "Bearer {api_token}" }, "body": { "email": "{customer_email}" }, "output_variable_names": ["create_result"] } } } ``` After this action, `{create_result.body.id}` resolves to the new customer's ID, and `{create_result.status_code}` resolves to `201`. > [!NOTE] > > Dot-path resolution (`{var.field}`) applies only to dict-valued variables such as API responses. The existing list-indexing format `{var[0]}` from `llm` extraction is unaffected. ### Polling For asynchronous endpoints, set `poll_condition` to keep re-requesting until the condition is met (or `max_poll_attempts` is reached). The condition is a Python-style boolean expression evaluated against the response dict, supporting both top-level keys and dot-paths: ```json theme={null} { "extraction_action": { "api_call": { "url": "https://inference-api.optexity.com/v1/jobs/{job_id}", "poll_condition": "body.status == 'completed'", "poll_interval": 10.0, "max_poll_attempts": 30 } } } ``` Example conditions: | Condition | Meaning | | - | - | | `status_code == 200` | Stop once the request returns HTTP 200 | | `body.status == 'completed'` | Stop once the response body's `status` field is `"completed"` | | `body.progress >= 100` | Stop once `progress` reaches 100 | If the condition is never met within `max_poll_attempts`, the last response is stored and the automation continues. *** ## Screenshot Extraction Save a screenshot for later analysis: ```json theme={null} { "extraction_action": { "screenshot": { "filename": "confirmation.png", "full_page": true } } } ``` | Property | Type | Default | Description | | - | - | - | - | | `filename` | `str` | Required | Output filename | | `full_page` | `bool` | `True` | Entire page or viewport only | *** ## State Extraction Capture page state, including URL/title plus browser storage and cookies: ```json theme={null} { "extraction_action": { "state": {} } } ``` ### Output `state` extraction appends an `OutputData.json_data` object with the following shape: | Key | Type | Description | | - | - | - | | `page_url` | `str` | Current page URL | | `page_title` | `str` | Current page title | | `local_storage` | `dict[str, str \| null]` | All `localStorage` key/value pairs | | `session_storage` | `dict[str, str \| null]` | All `sessionStorage` key/value pairs | | `cookies` | `list[dict]` | Cookies from the current browser context | | `document_cookie` | `str` | `document.cookie` string for the current page | ## Python Script Extraction Run a custom Python function to extract data from the current page's accessibility tree or perform any computed logic. ```json theme={null} { "extraction_action": { "python_script": { "script": "async def code_fn(axtree, browser):\n rows = [r.strip() for r in axtree.split('\\n') if r.strip()]\n return {'row_count': str(len(rows))}", "extraction_format": { "row_count": "str" }, "output_variable_names": ["row_count"] } } } ``` ### Properties | Property | Type | Default | Description | | - | - | - | - | | `script` | `str` | Required | Python source defining an `async def code_fn(axtree, browser)` function. Must return a `dict` matching `extraction_format`, or `None`. | | `extraction_format` | `dict \| None` | `None` | Expected output structure (same `str` / `List[str]` types as `llm` extraction) | | `output_variable_names` | `list[str] \| None` | `None` | Keys from the returned dict to store as workflow variables | > [!NOTE] > > The `code_fn` function receives the page's accessibility tree as a plain string (`axtree`) and the Playwright `browser` object. Return `None` to skip storing any output. *** ## Two-Factor Authentication Extraction Wait for and extract 2FA code: ```json theme={null} { "extraction_action": { "two_fa_action": { "action": "email_two_fa_action", "output_variable_name": "two_fa_code" } } } ``` ### Properties | Property | Type | Default | Description | | - | - | - | - | | `action` | `"email_two_fa_action" \| "slack_two_fa_action" \| "sms_two_fa_action"` | Required | The type of 2FA action to use | | `output_variable_name` | `str` | Required | The name of the variable to store the 2FA code in | | `instructions` | `str` | `None` | Optional Custom instructions for code extraction | | `max_wait_time` | `float` | `300.0` | The maximum time to wait for the 2FA code | | `check_interval` | `float` | `10.0` | The interval to check for the 2FA code | ### Action Types | Action Type | Description | | - | - | | `email_two_fa_action` | Wait for and extract 2FA code from email | | `slack_two_fa_action` | Wait for and extract 2FA code from Slack | | `sms_two_fa_action` | Wait for and extract 2FA code from SMS via Twilio | For more information on how to use the 2FA code in your automation, please refer to the [Two-Factor Authentication Integration](/docs/advanced/two-fa-integration) documentation. *** ## Timing Extraction actions have different timing defaults to allow pages to fully load: | Property | Default for Extractions | | - | - | | `before_sleep_time` | `3.0` seconds | | `end_sleep_time` | `0.0` seconds | Override if needed: ```json theme={null} { "type": "action_node", "extraction_action": { "llm": { ... } }, "before_sleep_time": 5.0 } ``` *** ## When to Use Each Type | Scenario | Recommended | | - | - | | Extract text/tables from page | `llm` with `axtree` | | Extract a single known element | `locator` | | Extract with locator + LLM fallback | `locator` with `extraction_instructions` | | Charts, images, visual content | `llm` with `screenshot` | | Intercept API data the page requests | `network_call` | | Call an external API / webhook directly | `api_call` | | Poll an async endpoint until ready | `api_call` with `poll_condition` | | Visual proof/documentation | `screenshot` | | Validate navigation | `state` | | Complex parsing / computed values | `python_script` | | Value may be null on some pages | Any type + `allow_none: true` on `extraction_action` | # Human in the Loop Source: https://docs.optexity.com/docs/action-types/human-in-loop Pause an automation and hand control to a human, then resume automatically The `human_in_loop_action` lets you pause an automation at any point so a human can take over the browser session. Once the human signals they're done, the automation resumes from where it left off. ## Overview * **Use when**: A step requires human judgment, sensitive credentials, or manual verification that cannot be automated. * **How it works**: The agent pauses, the task owner receives a magic-link email to a live browser stream, the human completes the step, then clicks a "Done" button to resume the agent. * **Timeout**: If no completion signal arrives within `max_wait_time` seconds, the task fails with a `HumanInLoopTimeoutException`. The hard task timeout is **10 minutes**, so `max_wait_time` must be less than `600`. ## Properties | Property | Type | Unit | Description | | - | - | - | - | | `max_wait_time` | `float` | seconds | Maximum time to wait for human completion before timing out | ## JSON Example ```json theme={null} { "type": "action_node", "human_in_loop_action": { "max_wait_time": 300 } } ``` The example above waits up to **5 minutes** for the human to finish. ## The HITL Flow When the agent reaches a `human_in_loop_action` node: 1. **Agent pauses** — execution halts at this node. 2. **Email sent** — Optexity sends a magic-link (OTP) email to the task owner. 3. **Human takes over** — the link opens the live browser stream at the task-logs page. 4. **Human signals done** — the task owner clicks the "Done" button in the dashboard. 5. **Agent resumes** — the agent picks up execution at the next node. If the human does not click done within `max_wait_time` seconds, the task fails. ## Complete Automation Example ```json theme={null} { "url": "https://secure.example.com/login", "parameters": { "input_parameters": { "username": ["user@example.com"] } }, "nodes": [ { "type": "action_node", "interaction_action": { "input_text": { "command": "get_by_label(\"Username\")", "input_text": "{username[0]}", "prompt_instructions": "Enter the username" } } }, { "type": "action_node", "human_in_loop_action": { "max_wait_time": 300 } }, { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Submit\")", "prompt_instructions": "Click submit after human completes the step" } } } ] } ``` In this example the agent fills in the username, then pauses for up to 5 minutes so the human can enter a password or complete a sensitive step, then resumes and clicks Submit. ## Timeout Behavior If `max_wait_time` elapses without a completion signal, the agent raises `HumanInLoopTimeoutException` and the task is marked failed. Choose a value that gives the human enough time to respond — common values: | Scenario | Suggested `max_wait_time` | | - | - | | Quick manual entry (\~1 min) | `120` | | Short review (\~3 min) | `180` | | Longer step (\~8 min) | `480` | > [!WARNING] > > The hard task timeout is **10 minutes (600 seconds)**. Setting `max_wait_time` > to `600` or above will cause the task to time out before the HITL window > closes. Always keep `max_wait_time` below `600` ## Best Practices ### Place HITL nodes at natural breakpoints Put the pause node immediately before the step that requires human action, not after. The agent stops at the HITL node; all prior nodes will have already executed. ### Keep `max_wait_time` realistic Setting an extremely large timeout (e.g. hours) keeps the cloud worker occupied. If you expect a long human delay, consider splitting the automation into two separate runs instead. ### Only one HITL pause per task at a time A second `human_in_loop_action` node will not trigger while the first is already active. Design your automation so HITL nodes are sequential, not concurrent. ## Next Steps * See [Timing & Retries](/docs/advanced/timing-retries) for retry strategies around HITL failures * See [Agentic Tasks](/docs/action-types/agentic-tasks) for AI-driven steps that can precede a human handoff # Interaction Actions Source: https://docs.optexity.com/docs/action-types/interaction-action Clicking, typing, and navigating web pages Interaction actions represent user interactions with the browser: clicking buttons, filling forms, navigating pages, and more. ## Available Actions | Action | Purpose | | - | - | | `click_element` | Click buttons, links, elements | | `input_text` | Type into text fields | | `select_option` | Select from dropdowns | | `check` | Check/uncheck checkboxes | | `upload_file` | Upload files | | `go_to_url` | Navigate to a URL | | `go_back` | Go back in history | | `close_tabs_until` | Close tabs until condition met | | `download_url_as_pdf` | Save page as PDF | | `download_file` | Download a URL to disk for a later upload | | `key_press` | Press keyboard keys | | `agentic_task` | AI agent for complex tasks | | `close_overlay_popup` | Dismiss popups/modals | ## Common Properties All element-targeting actions share these properties: | Property | Type | Default | Description | | - | - | - | - | | `command` | `str \| None` | `None` | Playwright locator (e.g., `get_by_role("button")`) | | `xpath` | `str \| None` | `None` | XPath selector (alternative to command) | | `prompt_instructions` | `str` | Required | Description for AI fallback | | `skip_prompt` | `bool` | `False` | Skip AI if locator fails (for optional elements) | | `assert_locator_presence` | `bool` | `False` | Skip action if element doesn't exist | | `max_tries` | `int` | `10` | Maximum retry attempts | | `max_timeout_seconds_per_try` | `float` | `1.0` | Timeout per attempt | > [!TIP] > > Use `command` for deterministic element finding (fast, no LLM tokens). The AI uses `prompt_instructions` as fallback when locators fail. *** ## Click Element ```json theme={null} { "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Submit\")", "prompt_instructions": "Click the submit button" } } } ``` ### Click Properties | Property | Type | Default | Description | | - | - | - | - | | `double_click` | `bool` | `False` | Double-click instead of single | | `expect_download` | `bool` | `False` | Click triggers file download | | `download_filename` | `str \| None` | Auto-generated | Filename for download | | `download_metadata` | `dict \| None` | `None` | Freeform JSON stored with the download | | `download_timeout_seconds` | `float \| None` | Global default (200s) | How long to wait for the file after the click | ### Examples **Double-click:** ```json theme={null} { "click_element": { "command": "get_by_role(\"row\", name=\"Item 1\")", "double_click": true } } ``` **Download trigger:** ```json theme={null} { "click_element": { "command": "get_by_role(\"button\", name=\"Export\")", "expect_download": true, "download_filename": "report.pdf" } } ``` *** ## Input Text ```json theme={null} { "interaction_action": { "input_text": { "command": "get_by_label(\"Email\")", "input_text": "{email[0]}", "prompt_instructions": "Enter the email address" } } } ``` ### Input Properties | Property | Type | Default | Description | | - | - | - | - | | `input_text` | `str \| None` | `None` | Text to enter (supports variables) | | `fill_or_type` | `"fill" \| "type"` | `"fill"` | How to enter text | | `is_slider` | `bool` | `False` | Element is a slider | | `press_enter` | `bool` | `False` | Press Enter after input | ### Fill vs Type | Mode | Behavior | Use When | | - | - | - | | `fill` | Sets value instantly | Standard form fields | | `type` | Types character by character | Autocomplete, search with suggestions | **Autocomplete example:** ```json theme={null} { "input_text": { "command": "get_by_label(\"Search\")", "input_text": "laptop", "fill_or_type": "type" } } ``` **Slider example:** ```json theme={null} { "input_text": { "command": "get_by_role(\"slider\", name=\"Price\")", "input_text": "500", "is_slider": true } } ``` > [!NOTE] > > If `input_text` references an empty variable, the action is skipped automatically. *** ## Select Option ```json theme={null} { "interaction_action": { "select_option": { "command": "get_by_label(\"Country\")", "select_values": ["United States"], "prompt_instructions": "Select country" } } } ``` ### Select Properties | Property | Type | Default | Description | | - | - | - | - | | `select_values` | `list[str]` | Required | Values to select | | `expect_download` | `bool` | `False` | Selection triggers download | | `download_filename` | `str \| None` | Auto-generated | Filename for download | | `download_metadata` | `dict \| None` | `None` | Freeform JSON stored with the download | | `download_timeout_seconds` | `float \| None` | Global default (200s) | How long to wait for the file after the selection | > [!TIP] > > Optexity supports fuzzy matching for select values—"United States" will match "UNITED STATES OF AMERICA". *** ## Check (Checkbox/Radio) ```json theme={null} { "interaction_action": { "check": { "command": "get_by_label(\"I agree to the terms\")", "prompt_instructions": "Check the terms agreement" } } } ``` *** ## Navigation Actions ### Go to URL | Property | Type | Default | Description | | - | - | - | - | | `url` | `str` | Required | URL to navigate to | | `new_tab` | `bool` | `False` | Open in new tab | ```json theme={null} { "interaction_action": { "go_to_url": { "url": "https://example.com/dashboard", "new_tab": true } } } ``` ### Go Back ```json theme={null} { "interaction_action": { "go_back": {} } } ``` ### Close Tabs Until Close tabs until reaching a specific URL or tab index: | Property | Type | Default | Description | | - | - | - | - | | `matching_url` | `str \| None` | `None` | Stop at this URL | | `tab_index` | `int \| None` | `None` | Stop at this tab index (0-based) | ```json theme={null} { "interaction_action": { "close_tabs_until": { "matching_url": "https://example.com/dashboard" } } } ``` *** ## File Operations ### Upload File The file source must be **exactly one** of `file_path` (local file) or `file_url` (public `http(s)://` URL — downloaded to a temp file just before upload, then cleaned up). | Property | Type | Description | | - | - | - | | `file_path` | `str \| None` | Absolute or relative local path | | `file_url` | `str \| None` | Public `http://` or `https://` URL; downloaded with a 120s timeout. If the download fails (network error or non-2xx response), the automation fails. | Upload from a local path: ```json theme={null} { "interaction_action": { "upload_file": { "command": "get_by_label(\"Upload Document\")", "file_path": "/path/to/document.pdf", "prompt_instructions": "Upload the document" } } } ``` Upload from a public URL: ```json theme={null} { "interaction_action": { "upload_file": { "command": "get_by_label(\"Upload Document\")", "file_url": "https://example.com/files/document.pdf", "prompt_instructions": "Upload the document" } } } ``` ### Download Page as PDF Capture the current page or a specific URL as PDF: | Property | Type | Default | Description | | - | - | - | - | | `download_filename` | `str \| None` | Auto-generated | Filename for PDF | | `url` | `str \| None` | Current page | URL to download | ```json theme={null} { "interaction_action": { "download_url_as_pdf": { "download_filename": "page_snapshot.pdf" } } } ``` ### Download File Download an `http(s)` URL and leave the file on disk. A later `agentic_task` in the same run can upload it. This is separate from `download_url_as_pdf`, which saves into the task downloads directory and does not pass the path to the agent. | Property | Type | Default | Description | | - | - | - | - | | `file_url` | `str` | Required | Public `http://` or `https://` URL. Supports `{variable[index]}`. 120s timeout. The automation fails if the download errors or returns a non-2xx status. | | `output_variable_name` | `str` | `downloaded_file_path` | Generated variable that holds the absolute path. Read it as `{output_variable_name[0]}`. | The file is written under `/tmp` as `optexity_download_.` and is not deleted when the node finishes. The extension comes from the URL path, `Content-Disposition`, or `Content-Type`, in that order. The same path is added to the next `agentic_task`'s allowed upload files. ```json theme={null} { "interaction_action": { "download_file": { "file_url": "{file_url[0]}" } } } ``` The default variable is `{downloaded_file_path[0]}`. Set `output_variable_name` to use a different name; the `[0]` stays, because the value is stored as a one-item list. *** ## Handling New Tabs When an action opens a new tab, set `expect_new_tab` on the action node: ```json theme={null} { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Open Details\")", "prompt_instructions": "Click to open in new tab" } }, "expect_new_tab": true } ``` This automatically waits up to 10 seconds for the new tab and switches to it. *** ## Retry Configuration For slow-loading elements, increase retry attempts: ```json theme={null} { "interaction_action": { "max_tries": 15, "max_timeout_seconds_per_try": 2.0, "click_element": { "command": "get_by_role(\"button\", name=\"Submit\")", "prompt_instructions": "Click submit" } } } ``` > [!TIP] > > Increase `max_tries` rather than timeout per try. This finds elements faster when they appear while still allowing for slow pages. # LLM Query Action Source: https://docs.optexity.com/docs/action-types/llm-query-action Run a direct LLM query without browser context `misc_action.llm_query` sends a prompt directly to an LLM and stores the structured response—no browser state, axtree, or screenshot is involved. ## Overview * **Use when**: You need to transform, classify, or reason over data already in memory (e.g., reformat an extracted date, classify extracted text, compute a derived value). * **Execution**: The action sends `prompt_instructions` to the configured LLM, parses the response according to `output_format`, and stores results in `output_data`. Specified fields are optionally promoted to `generated_variables`. * **No browser access**: Unlike `extraction_action.llm`, this action does not read the current page. Reference earlier extracted values via `{variable_name[index]}` in your prompt. ## Properties | Property | Type | Default | Description | | - | - | - | - | | `output_format` | `dict` | Required | Expected output structure as a type-annotated dict | | `prompt_instructions` | `str` | Required | The full prompt to send to the LLM | | `output_variable_names` | `list[str] \| None` | `None` | Promote response fields to `generated_variables` | | `llm_model_name` | `str \| None` | `None` | Vertex model ID, e.g. `"gemini-4-argon"`. Falls back to the task model, then `LLM_MODEL` | | `llm_provider` | `str \| None` | `None` | Deprecated — prefix the provider in `llm_model_name` instead | ## JSON Example ```json theme={null} { "type": "action_node", "misc_action": { "llm_query": { "output_format": { "category": "str", "confidence": "str" }, "prompt_instructions": "Classify the following support ticket as 'billing', 'technical', or 'other'. Ticket: {ticket_text[0]}", "output_variable_names": ["category"] } } } ``` ## Output Format Define the expected response structure with Python type hints: ```json theme={null} { "output_format": { "summary": "str", "status": "str", "items": "List[str]" } } ``` > [!NOTE] > > Only `str` and `List[str]` are supported types in `output_format`. ## Using Variables in Prompts Reference extracted values from earlier nodes using `{variable_name[index]}`: ```json theme={null} { "misc_action": { "llm_query": { "output_format": { "normalized_date": "str" }, "prompt_instructions": "Reformat this date to ISO 8601 (YYYY-MM-DD): {raw_date[0]}", "output_variable_names": ["normalized_date"] } } } ``` After this action, `{normalized_date[0]}` is available in subsequent nodes. ## Storing Output as Variables Set `output_variable_names` to promote response fields into `generated_variables`: ```json theme={null} { "misc_action": { "llm_query": { "output_format": { "next_url": "str", "has_more_pages": "str" }, "prompt_instructions": "Given these pagination links: {page_links[0]}, return the next page URL and whether more pages exist ('true'/'false').", "output_variable_names": ["next_url", "has_more_pages"] } } } ``` > [!WARNING] > > Every key in `output_variable_names` must appear in `output_format`. Validation fails at schema load time if a key is missing. ## Choosing a Model Set `llm_model_name` to a Vertex model ID: | Model | Notes | | - | - | | `gemini-3.5-flash-lite` | Default; cost-effective | | `gemini-3.6-flash` | Stronger, still fast | | `gemini-4-argon` | Harder reasoning over extracted data | | `claude-sonnet-5-5` | Strong reasoning | Omit `llm_model_name` to use the task model, which itself falls back to the `LLM_MODEL` environment variable. See [Model Configuration](/docs/advanced/model-configuration). ## LLM Query vs LLM Extraction | | `misc_action.llm_query` | `extraction_action.llm` | | - | - | - | | Browser context | None — prompt only | axtree and/or screenshot of current page | | Typical use | Transform / classify in-memory data | Extract data from the page the browser is on | | Input source | Variables embedded in the prompt | Live page content | | Output destination | `output_data` + optional `generated_variables` | `output_data` + optional `generated_variables` | > [!TIP] > > Use `misc_action.llm_query` after an extraction step to post-process or validate extracted data without consuming a browser round-trip. # Python Script Action Source: https://docs.optexity.com/docs/action-types/python-script-action Execute custom Python code against the live browser page `python_script_action` lets you run arbitrary Python code against the live Playwright `page` object—covering anything that standard interaction actions cannot express. ## Overview * **Use when**: Built-in actions are insufficient (complex DOM manipulation, custom scrolling, dispatching browser events, reading element properties). * **Execution**: The runner `exec`s your script, finds an async function named `code_fn`, and calls it with the current Playwright `page`. * **No return value**: The value returned by `code_fn` is discarded. To extract data with custom code, use [`extraction_action.python_script`](#python-script-extraction). To emit a file, add a [`ctx`](#the-script-context-ctx) argument and call `ctx.save_download()`. ## Properties | Property | Type | Description | | - | - | - | | `execution_code` | `str` | Python source that defines an async `code_fn(page)` | `execution_code` supports the same `{variable[0]}` / `{index}` substitution as other actions, so a `python_script_action` inside a `for_loop_node` can read `{index}` directly. ## Script Contract Your script **must** define an async function named `code_fn` that accepts a single `page` argument (a Playwright [`Page`](https://playwright.dev/python/docs/api/class-page)): ```python theme={null} async def code_fn(page): # interact with the page here ``` The function is called as `await code_fn(page)`. Any return value is ignored. ## JSON Example ```json theme={null} { "type": "action_node", "python_script_action": { "execution_code": "async def code_fn(page):\n await page.evaluate(\"window.scrollTo(0, document.body.scrollHeight)\")" } } ``` ## Common Patterns ### Scroll to the bottom of the page ```python theme={null} async def code_fn(page): await page.evaluate("window.scrollTo(0, document.body.scrollHeight)") ``` ### Wait for a custom JavaScript condition ```python theme={null} async def code_fn(page): await page.wait_for_function("() => document.readyState === 'complete'") ``` ### Dispatch a custom browser event ```python theme={null} async def code_fn(page): await page.evaluate(""" document.querySelector('#upload-input').dispatchEvent( new Event('change', { bubbles: true }) ) """) ``` ### Interact with a shadow DOM element ```python theme={null} async def code_fn(page): await page.evaluate(""" document.querySelector('my-component').shadowRoot .querySelector('button.confirm') .click() """) ``` > [!WARNING] > > `execution_code` is evaluated with `exec()`. Only run scripts from trusted sources. *** ## Python Script Extraction To extract data from the page using custom code, use `python_script` inside an `extraction_action` instead. The contract is different: the function receives `(axtree, browser)` and **must return a dict** containing the extracted values. ```json theme={null} { "type": "action_node", "extraction_action": { "python_script": { "script": "async def code_fn(axtree, browser):\n page = await browser.get_current_page()\n text = await page.locator('.order-id').inner_text()\n return {\"order_id\": text.strip()}", "output_variable_names": ["order_id"] } } } ``` ### Properties | Property | Type | Default | Description | | - | - | - | - | | `script` | `str` | Required | Python source defining `async def code_fn(axtree, browser)` | | `extraction_format` | `dict \| None` | `None` | Expected output structure (required when `output_variable_names` is set) | | `output_variable_names` | `list[str] \| None` | `None` | Promote returned dict keys to `generated_variables` | | `timeout_seconds` | `float \| None` | `None` | Fail the node if the script runs longer than this. Unset means the script is bounded only by the task's `max_timeout_in_minutes` | ### Script Contract ```python theme={null} async def code_fn(axtree, browser): # axtree: string representation of the page accessibility tree # browser: the Browser instance (use browser.get_current_page() for the Playwright page) return {"key": "value"} # must return a dict ``` The returned dict is stored as `OutputData` and, if `output_variable_names` is set, the specified keys are promoted to `generated_variables` for use in later nodes. > [!TIP] > > All keys in `output_variable_names` must exist in `extraction_format`, or schema validation will fail. > [!WARNING] > > Put `import` statements **inside** `code_fn`, not at the top of the script. The > script is `exec`'d with separate globals and locals, so a module-level import > lands somewhere `code_fn`'s body cannot see it and you get `NameError` at > runtime. > > ```python theme={null} > async def code_fn(axtree, browser): > import json # correct > ... > ``` *** ## The Script Context (`ctx`) Both script types can request an extra `ctx` argument by **naming it** in the signature. Add `ctx` (or `context`) and you get it; omit it and nothing changes. ```python theme={null} async def code_fn(axtree, browser, ctx): # extraction async def code_fn(page, ctx): # python_script_action ``` Opting in is by parameter name, not position — an existing third parameter called something else will never receive the context. ### `ctx.save_download()` — emit a file as a task download Previously only `click_element` / `select_option` with `expect_download` could produce a downloadable file, so scripts that had already built the bytes had to push them back into the page as a `Blob`, inject an `` anchor, and add a second node to click it. `ctx.save_download()` removes that round trip. ```python theme={null} async def code_fn(axtree, browser, ctx): page = await ctx.get_page() csv = await fetch_export(page) await ctx.save_download( "patient_demographics.csv", csv, metadata={"provider": "{provider_name[0]}", "kind": "export"}, ) return {"saved": 1} ``` ```python theme={null} await ctx.save_download(filename, content=None, *, path=None, metadata=None) ``` | Argument | Description | | - | - | | `filename` | Required. Sanitized (path separators and control characters replaced, whitespace collapsed, truncated to 150 chars) and de-duplicated with a `_2`, `_3`, … suffix if it already exists | | `content` | `bytes` or `str` held in memory. Text is encoded as UTF-8 | | `path` | A file already on disk, **moved** into the downloads directory. Provide exactly one of `content` or `path` | | `metadata` | Freeform JSON stored against the final filename, same as `download_metadata` on a click. Values support `{variable[0]}` substitution, resolved when the file is saved | Returns the final `Path`, which may differ from `filename` after sanitizing or de-duplication. Raises if the resulting file is empty or missing; nothing partial is left behind. The file is uploaded with the task's other downloads and its metadata is returned as `downloads_with_metadata` — identical to the `expect_download` path, because it uses the same download registry. See [Downloads & Files](/docs/advanced/downloads-files). > [!NOTE] > > Unlike `expect_download`, `filename` is required — there is no browser-supplied > name to fall back on, and defaulting to a UUID would only hide mistakes. ### `ctx.state` — share data between script nodes Each script node is `exec`'d with fresh globals, so nothing survives between nodes by default. `ctx.state` is a plain dict scoped to the run — use it instead of stashing work lists on `window`, which costs a JS round trip per read and is lost on navigation. ```python theme={null} # prep node async def code_fn(axtree, browser, ctx): ctx.state["labs"] = labs # any Python object return {"iterations": list(range(len(labs)))} # per-iteration node async def code_fn(axtree, browser, ctx): lab = ctx.state["labs"][{index}] ``` ### Other members | Member | Description | | - | - | | `await ctx.get_page()` | The live Playwright page | | `ctx.variables` | Variables produced by earlier nodes, before substitution | | `ctx.input_parameters` / `ctx.unique_parameters` | The task's parameters | | `ctx.downloads_dir` | The task's downloads directory as a `Path` | | `ctx.log(msg, level="info")` | Log to the task logs | `ctx.log` tags each line with the current step index (e.g. `[python_script step=12] ...`) so lines from different nodes or loop iterations can be told apart, and writes it via the standard logger — pass `level="warning"` or `level="error"` for anything more severe than routine progress. These lines land in `optexity.log`, viewable in the dashboard's task logs "Logs" panel. *** ## Action vs Extraction — Which to Use | Scenario | Use | | - | - | | Save a computed or fetched file as a task download | either, with `ctx.save_download()` | | Scroll, dispatch events, manipulate DOM | `python_script_action` | | Extract and store data from the page | `extraction_action.python_script` | | Simple text / table extraction | `extraction_action.llm` | # AWS Secrets Manager Integration Source: https://docs.optexity.com/docs/advanced/aws-secrets-manager Store and retrieve secrets securely using AWS Secrets Manager Use AWS Secrets Manager to store sensitive data like passwords and API keys. Values are retrieved at runtime without exposing secrets in your workflow. ## Setup ### Create a Secret in AWS 1. Go to the [AWS Secrets Manager console](https://console.aws.amazon.com/secretsmanager) 2. Click **Store a new secret** 3. Choose **Other type of secret** (key/value pairs or plain text) 4. Name your secret (e.g. `my-app/prod/login`) 5. Note the secret name and the AWS region it was created in ### Create IAM Credentials 1. Go to the [IAM console](https://console.aws.amazon.com/iam) 2. Create a user or role with the following policy: ```json theme={null} { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "secretsmanager:GetSecretValue", "Resource": "arn:aws:secretsmanager:::secret:*" } ] } ``` 3. Generate an access key and copy the credentials ### Configure Environment Add them directly on the **Integrations** page of the dashboard, or add to your `.env` file: ```bash theme={null} AWS_ACCESS_KEY_ID=your_access_key_id AWS_SECRET_ACCESS_KEY=your_secret_access_key ``` *** ## Usage Move parameters from `input_parameters` to `secure_parameters`: **Before:** ```json theme={null} { "input_parameters": { "password": ["password_value"] } } ``` **After (plain string secret):** ```json theme={null} { "secure_parameters": { "password": [{ "amazon_secrets_manager": { "secret_name": "my-app/prod/login", "region_name": "us-east-1", "key": "password" } }] } } ``` **After (JSON secret — extract a single key):** If your secret is stored as a JSON object like `{"username": "admin", "password": "s3cr3t"}`, use the `key` field to pluck the value you need: ```json theme={null} { "secure_parameters": { "password": [{ "amazon_secrets_manager": { "secret_name": "my-app/prod/login", "region_name": "us-east-1", "key": "password" } }] } } ``` *** ## Properties | Property | Type | Default | Description | | - | - | - | - | | `secret_name` | `str` | Required | Name or ARN of the secret in AWS Secrets Manager | | `region_name` | `str` | Required | AWS region where the secret is stored (e.g. `"us-east-1"`) | | `key` | `str` | `null` | Key to extract from the secret (plain string or JSON object) | | `type` | `str` | `null` | Set to `"totp_secret"` to generate TOTP codes | | `digits` | `int` | `null` | Required when `type` is `"totp_secret"` (e.g. `6`) | *** ## TOTP from AWS Secrets Manager Store a TOTP secret in AWS Secrets Manager and generate codes at runtime: ```json theme={null} { "secure_parameters": { "auth_code": [{ "amazon_secrets_manager": { "type": "totp_secret", "secret_name": "my-app/prod/totp", "region_name": "us-east-1", "digits": 6 } }] } } ``` If the TOTP secret is stored inside a JSON object, combine `key` with `type: "totp_secret"`: ```json theme={null} { "secure_parameters": { "auth_code": [{ "amazon_secrets_manager": { "type": "totp_secret", "secret_name": "my-app/prod/login", "region_name": "us-east-1", "key": "totp_secret", "digits": 6 } }] } } ``` > [!TIP] > > See [TOTP Integration](/docs/advanced/totp-integration) for more 2FA options. *** ## Revoking Access Deactivate or delete the IAM access key from the AWS console at any time to immediately revoke access. # Best Practices Source: https://docs.optexity.com/docs/advanced/best-practices Tips for building reliable and maintainable automations Best practices learned from real-world Optexity deployments. ## Design Principles | Principle | Practice | | - | - | | Start simple | Build minimum viable automation first | | Iterate | Add complexity incrementally | | Test with real data | Validate on realistic inputs | | Document complexity | Comment non-obvious logic | *** ## Locator Strategy ### Priority Order | Priority | Method | Example | | - | - | - | | 1. Best | Role-based | `get_by_role("button", name="Submit")` | | 2. Good | Label-based | `get_by_label("Email Address")` | | 3. Okay | Test ID | `get_by_test_id("submit-btn")` | | 4. Last resort | CSS/XPath | `locator("#submit-form-btn")` | ### Always Provide Fallback ```json theme={null} { "click_element": { "command": "get_by_role(\"button\", name=\"Continue\")", "prompt_instructions": "Click the green Continue button at the bottom of the form" } } ``` ### Avoid Dynamic IDs **Bad:** ```json theme={null} {"command": "locator(\"#btn_12345_submit\")"} ``` **Good:** ```json theme={null} {"command": "get_by_role(\"button\", name=\"Submit\")"} ``` *** ## Naming Conventions | Good | Bad | | - | - | | `patient_date_of_birth` | `dob` | | `authorization_number` | `num` | | `search_query` | `q` | *** ## Timing Guidelines | Scenario | Recommendation | | - | - | | Before extraction | Wait 3+ seconds for page to load | | After navigation | Wait 1-2 seconds for render | | Dynamic content | Increase retry count | | Form submission | Wait for confirmation | ```json theme={null} { "type": "action_node", "extraction_action": { ... }, "before_sleep_time": 5.0 } ``` *** ## Error Handling ### Optional Elements Use `assert_locator_presence` for elements that may not appear: ```json theme={null} { "click_element": { "command": "get_by_role(\"button\", name=\"Accept Cookies\")", "assert_locator_presence": true } } ``` ### Skip Optional Steps ```json theme={null} { "click_element": { "command": "get_by_role(\"button\", name=\"Skip Tutorial\")", "skip_prompt": true, "assert_locator_presence": true } } ``` *** ## Security | Practice | Example | | - | - | | Never hardcode credentials | Use `secure_parameters` | | Use 1Password integration | Store in vault, retrieve at runtime | | Avoid logging sensitive data | Don't include in prompt instructions | *** ## Performance | Tip | Benefit | | - | - | | Minimize waits | Start conservative, optimize later | | Use static over agentic | Faster, cheaper, more reliable | | Batch extractions | One LLM call for multiple fields | | Increase retries over timeouts | Faster when element appears | *** ## Debugging ### Common Issues | Symptom | Solution | | - | - | | "Element not found" | Increase `before_sleep_time` | | Wrong element clicked | Wait for page to settle | | Empty extraction | Increase wait before extraction | | Random failures | Increase retry count | ### Debug with Screenshots ```json theme={null} { "type": "action_node", "extraction_action": { "screenshot": { "filename": "debug_before_submit.png", "full_page": true } } } ``` *** ## Pre-Deployment Checklist * [ ] All locators have `prompt_instructions` * [ ] Sensitive data in `secure_parameters` * [ ] Appropriate wait times for page loads * [ ] Optional elements use `assert_locator_presence` * [ ] Error cases considered * [ ] Tested on realistic data * [ ] Variable names are descriptive * [ ] Complex logic is commented # Callbacks Source: https://docs.optexity.com/docs/advanced/callbacks Receive notifications when automations complete Set up callbacks to receive automation results without polling. ## Overview Optexity sends HTTP POST requests to your callback URL when automations complete. | Environment | Configuration | | - | - | | Local | `LOCAL_CALLBACK_URL` environment variable | | Production | Set in dashboard | *** ## Local Development Set the environment variable: ```bash theme={null} LOCAL_CALLBACK_URL=http://localhost:3000/receive_callback ``` Requirements: * URL must be accessible from the machine running the automation * Endpoint must be unauthenticated *** ## Production Configure in the Optexity dashboard: * Callback URL must be publicly accessible * Supports API key authentication * Supports username/password authentication *** ## Callback Payload ```json theme={null} { "task_id": "abc123", "recording_id": "rec456", "endpoint_name": "login-flow", "status": "success", "output_data": [ {"name": "email", "value": "test@example.com"} ], "error": null, "final_screenshot": "base64_encoded_string", "downloads": [ {"url": "signed_url", "filename": "report.pdf"} ], "downloads_with_metadata": [ { "url": "signed_url", "filename": "report.pdf", "metadata": {"doc_id": "123", "kind": "report"} } ] } ``` | Field | Type | Description | | - | - | - | | `task_id` | `str` | Unique task identifier | | `status` | `str` | `queued`, `allocated`, `running`, `success`, `failed`, `cancelled` | | `output_data` | `list` | Extracted data objects | | `error` | `str \| null` | Error message if failed | | `final_screenshot` | `str` | Base64 screenshot | | `downloads` | `list` | Download URLs and filenames | | `downloads_with_metadata` | `list \| omitted` | All downloads with optional `metadata`; only present when at least one file has metadata | Set `download_metadata` on `expect_download` actions to populate this. See [Downloads & Files](/docs/advanced/downloads-files). *** ## Authentication | Type | Fields Required | | - | - | | API Key | `api_key` | | Basic Auth | `username`, `password` | > [!NOTE] > > See [Callback Reference](/api-reference/callback) for complete schema details. # Downloads & Files Source: https://docs.optexity.com/docs/advanced/downloads-files Handling file downloads and uploads in automations Optexity can download files from websites and upload files to forms. ## Download Methods | Method | Trigger | | - | - | | Click download link | `click_element` with `expect_download=true` | | Select export option | `select_option` with `expect_download=true` | | Save page as PDF | `download_url_as_pdf` | | Download a URL for a later upload | `download_file` | | Save bytes from a script | `ctx.save_download()` in a `python_script` | *** ## Click to Download ```json theme={null} { "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Download Report\")", "expect_download": true, "download_filename": "monthly_report.pdf" } } } ``` | Property | Type | Default | Description | | - | - | - | - | | `expect_download` | `bool` | `False` | Action triggers download | | `download_filename` | `str \| None` | Auto-generated UUID | Filename for download | | `download_metadata` | `dict \| None` | `None` | Freeform JSON stored with the download | | `download_timeout_seconds` | `float \| None` | Global default (200s) | How long to wait for the file after the action | *** ## Download Metadata Attach freeform JSON to a download when using `expect_download`. Values support the same `{variable}` substitution as other action fields: ```json theme={null} { "interaction_action": { "click_element": { "command": "get_by_text(\"{doc_ids[index]}\")", "expect_download": true, "download_filename": "doc_{doc_ids[index]}.pdf", "download_metadata": { "doc_id": "{doc_ids[index]}", "source": "ehr", "kind": "report" } } } } ``` Metadata is keyed by the final filename and returned on callbacks as optional `downloads_with_metadata` (see [Callbacks](/docs/advanced/callbacks)). On interaction actions, only `click_element` / `select_option` with `expect_download` support this field; from a Python script, pass `metadata=` to `ctx.save_download()`. *** ## Save a File from a Python Script When a script has already fetched or assembled the bytes — an API response, a CSV it built, pages it reassembled into a PDF — it can register the file directly instead of routing it back through the browser: ```json theme={null} { "type": "action_node", "extraction_action": { "python_script": { "script": "async def code_fn(axtree, browser, ctx):\n page = await ctx.get_page()\n csv = await page.evaluate(\"() => fetch('/export.csv').then(r => r.text())\")\n await ctx.save_download('export_{index}.csv', csv, metadata={'kind': 'export'})\n return {\"saved\": 1}", "output_variable_names": ["saved"], "extraction_format": {"saved": "int"} } } } ``` The file is uploaded and its metadata persisted exactly as for `expect_download` downloads. This replaces the older workaround of base64-ing bytes into a `Blob`, injecting an `` anchor, and adding a second `click_element` node to click it — which cost a screenshot, an axtree capture and a trajectory upload per file. See [Python Script Action](/docs/action-types/python-script-action#the-script-context-ctx) for the full `ctx` reference. *** ## Select to Download ```json theme={null} { "interaction_action": { "select_option": { "command": "get_by_label(\"Export Format\")", "select_values": ["CSV"], "expect_download": true, "download_filename": "data.csv" } } } ``` *** ## Save Page as PDF Capture current page or specific URL as PDF: ```json theme={null} { "interaction_action": { "download_url_as_pdf": { "download_filename": "page_snapshot.pdf" } } } ``` *** ## Download a File for Upload Use `download_file` when a later `agentic_task` needs to upload a file from a URL. The file stays on disk, its path is stored in a generated variable, and that path is passed to the agent as an allowed upload file. `download_url_as_pdf` does not do this. It saves into the task downloads directory and does not expose the path to the agent. | Property | Type | Default | Description | | - | - | - | - | | `file_url` | `str` | Required | Public `http://` or `https://` URL. Supports `{variable[index]}`. 120s timeout. The automation fails on a network error or a non-2xx response. | | `output_variable_name` | `str` | `downloaded_file_path` | Name of the generated variable. The path is `{output_variable_name[0]}`. | The file is `/tmp/optexity_download_.`. It is not deleted when the node finishes. The extension comes from the URL path, `Content-Disposition`, or `Content-Type`, in that order. ```json theme={null} { "type": "action_node", "interaction_action": { "download_file": { "file_url": "{file_url[0]}" } } } ``` ```json theme={null} { "type": "action_node", "interaction_action": { "agentic_task": { "task": "Upload the local file {downloaded_file_path[0]} using the upload_file action on the file input. Do not click a button that opens the system file dialog.", "max_steps": 15, "backend": "browser_use" } } } ``` Set `output_variable_name` to change the variable name. The index stays `[0]`. *** ## Multiple Downloads Use `for_loop_node` to download multiple files: ```json theme={null} { "type": "for_loop_node", "variable_name": "doc_ids", "nodes": [{ "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_text(\"{doc_ids[index]}\")", "expect_download": true, "download_filename": "doc_{doc_ids[index]}.pdf" } } }] } ``` > [!TIP] > > Use variable substitution in `download_filename` to create unique filenames. When the download links are on the page but their identifiers aren't known in advance, iterate a locator instead of a variable and use the loop index for the filename: ```json theme={null} { "type": "for_loop_node", "locator": "locator(\"tr.invoiceRow\")", "index_variable_name": "row", "nodes": [{ "type": "action_node", "interaction_action": { "click_element": { "command": "{locator[row]}.get_by_role(\"link\", name=\"Download\")", "expect_download": true, "download_filename": "invoice_{row}.pdf" } } }] } ``` See [For Loop Node](/docs/building-automations/for-loop-node) for the locator loop's placeholders and timing. *** ## Upload Files The file source must be **exactly one** of `file_path` or `file_url`. Setting both, or neither, is rejected by the schema. Upload from a local path: ```json theme={null} { "interaction_action": { "upload_file": { "command": "get_by_label(\"Upload Document\")", "file_path": "/path/to/document.pdf", "prompt_instructions": "Upload the document" } } } ``` Upload from a public URL (downloaded to a temp file just before upload, then cleaned up): ```json theme={null} { "interaction_action": { "upload_file": { "command": "get_by_label(\"Upload Document\")", "file_url": "https://example.com/files/document.pdf", "prompt_instructions": "Upload the document" } } } ``` | Property | Description | | - | - | | `command` / `xpath` | Locator for file input | | `file_path` | Absolute or relative local path (supports variables) | | `file_url` | Public `http://` or `https://` URL (supports variables). 120s download timeout. The automation fails if the download errors or returns a non-2xx status. The file extension is derived from the URL path, `Content-Disposition`, or `Content-Type` (in that order) so file inputs that check extensions still validate. | *** ## Download Storage Downloaded files are stored in the task's downloads directory: ``` /tmp/optexity/{task_id}/downloads/ ├── monthly_report.pdf ├── data.csv └── doc_123.pdf ``` *** ## Waiting for Downloads Optexity automatically waits for downloads when `expect_download=true`. The default wait is 200 seconds (`DOWNLOAD_TIMEOUT_SECONDS`). Override it per node with `download_timeout_seconds` for slow or large files: ```json theme={null} { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_text(\"Download Large File\")", "expect_download": true, "download_timeout_seconds": 600 } } } ``` You can still add `end_sleep_time` on the node for extra post-action delay after the download completes. *** ## Failure on Missing Download When a node expects a file but none is produced, the task **fails** rather than silently continuing. This applies to both download mechanisms: **`expect_download=true`** (`click_element` / `select_option`): | Condition | Task error | | - | - | | No file appears within the download timeout | `could not download file when expect download is true` | | A file appears but is empty or missing after being saved | `file appeared but was empty/missing after move` | **`download_url_as_pdf`:** | Condition | Task error | | - | - | | No URL could be resolved for the page | `could not download file for download_url_as_pdf: no URL found` | | The request returns a non-OK HTTP status | `could not download file for download_url_as_pdf: HTTP ` | | The saved PDF is empty or missing | `file appeared but was empty/missing after move` | > [!TIP] > > If a download is slow, increase `end_sleep_time` (see below) so the file has > time to finish before the timeout triggers a failure. *** ## Best Practices | Practice | Recommendation | | - | - | | Set `expect_download` | Always when download is expected | | Use descriptive filenames | Include IDs or dates | | Handle large files | Increase `end_sleep_time` | | Validate upload paths | Ensure absolute paths are accessible | # Locators Source: https://docs.optexity.com/docs/advanced/locators Finding elements on the page reliably Optexity provides multiple ways to locate elements. Understanding when to use each method is key to building robust automations. ## Locator Methods | Method | Field | When to Use | | - | - | - | | **Playwright Command** | `command` | Preferred—robust, fast, no LLM tokens | | **XPath** | `xpath` | Complex DOM traversal | | **AI Fallback** | `prompt_instructions` | Always provide—used when locators fail | *** ## Playwright Commands ### Role-Based (Recommended) Most resilient—doesn't depend on CSS classes or IDs: ```json theme={null} {"command": "get_by_role(\"button\", name=\"Submit\")"} {"command": "get_by_role(\"link\", name=\"Learn More\")"} {"command": "get_by_role(\"textbox\", name=\"Email\")"} {"command": "get_by_role(\"checkbox\", name=\"Remember me\")"} {"command": "get_by_role(\"combobox\", name=\"Country\")"} ``` ### Label-Based Great for form fields: ```json theme={null} {"command": "get_by_label(\"Email address\")"} {"command": "get_by_label(\"Password\")"} ``` ### Text-Based Find by visible text: ```json theme={null} {"command": "get_by_text(\"Welcome back\")"} {"command": "get_by_text(\"Welcome\", exact=True)"} ``` ### Test ID For elements with `data-testid`: ```json theme={null} {"command": "get_by_test_id(\"login-button\")"} ``` ### CSS Selector Direct CSS access: ```json theme={null} {"command": "locator(\"#email-input\")"} {"command": "locator(\".submit-button\")"} {"command": "locator(\"[data-action='submit']\")"} ``` ### Chaining Narrow down to specific elements: ```json theme={null} {"command": "get_by_role(\"navigation\").get_by_role(\"link\", name=\"Home\")"} {"command": "locator(\"form#login\").get_by_role(\"button\", name=\"Submit\")"} ``` ### Iframes Access elements inside iframes: ```json theme={null} {"command": "locator('#login-iframe').content_frame.get_by_role('button', name='Submit')"} ``` *** ## XPath Locators Use when Playwright locators can't express the query: ```json theme={null} { "interaction_action": { "click_element": { "xpath": "//table[@id='results']//tr[contains(@class, 'active')]/td[3]/button", "prompt_instructions": "Click the action button in the active row" } } } ``` ### Common Patterns ```json theme={null} {"xpath": "//button[text()='Submit']"} {"xpath": "//div[contains(text(), 'Welcome')]"} {"xpath": "//input[@type='email']"} {"xpath": "//ul/li[3]/a"} ``` > [!WARNING] > > XPath locators are more fragile than Playwright commands. Prefer `command` when possible. *** ## Prompt Instructions Natural language description used when locators fail: ```json theme={null} { "click_element": { "command": "get_by_role(\"button\", name=\"Continue\")", "prompt_instructions": "Click the green Continue button at the bottom of the checkout form" } } ``` ### Writing Good Instructions **Good:** ```json theme={null} {"prompt_instructions": "Click the blue 'Add to Cart' button below the product price"} {"prompt_instructions": "Click Submit next to the Cancel button at the form bottom"} {"prompt_instructions": "Click on order number {order_id[0]} in the orders table"} ``` **Poor:** ```json theme={null} {"prompt_instructions": "Click the button"} {"prompt_instructions": "Click it"} ``` > [!TIP] > > Include visual details, position, and context in prompt instructions. *** ## Locator Selection Strategy ``` 1. Element has role + accessible name? → get_by_role() 2. Form element with label? → get_by_label() 3. Has data-testid? → get_by_test_id() 4. Unique visible text? → get_by_text() 5. Unique ID or class? → locator() with CSS 6. Complex DOM traversal? → xpath 7. Dynamic or variable? → prompt_instructions primarily ``` *** ## Dynamic Elements Use variables in locators: ```json theme={null} {"command": "get_by_text(\"{order_id[0]}\")"} {"command": "get_by_role(\"link\", name=\"{product[index]}\")"} ``` Always provide `prompt_instructions` as backup: ```json theme={null} {"prompt_instructions": "Click on the row containing order {order_id[0]}"} ``` *** ## assert\_locator\_presence Verify element exists before acting (for optional steps): ```json theme={null} { "click_element": { "command": "get_by_role(\"button\", name=\"Proceed\")", "prompt_instructions": "Click Proceed if visible", "assert_locator_presence": true } } ``` When `true`: * Checks if locator finds element * Acts only if the element is present * Fails the automation with a clear error reason if the element is not found # Model Configuration Source: https://docs.optexity.com/docs/advanced/model-configuration Choose the LLM models your automations run on Every LLM call in Optexity runs on [Google Vertex AI](https://cloud.google.com/vertex-ai): Gemini first, with Claude as the fallback. Calls are authenticated with Application Default Credentials, so there are no model API keys. ## Setup 1. In your Google Cloud project, enable the **Vertex AI API**. 2. In **Vertex AI → Model Garden**, enable the Claude models used as fallbacks (`claude-haiku-5-5`, `claude-sonnet-5-5`, `claude-opus-5-5`). 3. Sign in: `gcloud auth application-default login` locally, or set `GOOGLE_APPLICATION_CREDENTIALS` to a service-account key file on a server. ## Environment Variables | Variable | Type | Default | Description | | - | - | - | - | | `GOOGLE_CLOUD_PROJECT` | `str` | — | Google Cloud project for every LLM call | | `GOOGLE_CLOUD_LOCATION` | `str` | `us` | Vertex AI location: `us` (data stays in the US) or `global` (about 10% cheaper for Claude) | | `GOOGLE_CREDENTIALS_JSON` | `str \| None` | `None` | Service-account key as JSON text, for runtimes that can't mount a key file (used when `GOOGLE_APPLICATION_CREDENTIALS` isn't set) | | `LLM_MODEL` | `str \| None` | `None` | Model for calls that don't choose one (default: the `lite` tier) | | `LLM_MODEL_FALLBACK` | `str \| None` | `None` | Backup model for `LLM_MODEL` and `llm_model_name` overrides | | `LLM_TIER_LITE` / `LLM_TIER_STANDARD` / `LLM_TIER_PRO` | `str \| None` | `None` | Comma-separated models that replace a tier | | `LLM_SERVICE_NAME`, `LLM_ENVIRONMENT` | `str` | `optexity`, `dev` | Tags on every call, for cost and usage reports | | `LLM_CLOUD_LOGGING` | `bool` | `false` | Also send one log entry per LLM call to Google Cloud Logging | | `LLM_LOG_CONTENT` | `bool` | `false` | Add the prompt and response text to each entry; images and files appear as placeholders, and text over \~180k characters loses its middle | ## Tiers | Tier | Models, in fallback order | Used by | | - | - | - | | `lite` | `gemini-3.5-flash-lite`, `claude-haiku-5-5` | Default for every call that doesn't choose a model | | `standard` | `gemini-3.6-flash`, `claude-sonnet-5-5` | — | | `pro` | `gemini-3.1-pro-preview`, `claude-opus-5-5` (`us`: `gemini-4-argon`, `claude-opus-5-5`) | Captcha solving | ## Model IDs Use Vertex model IDs, such as `gemini-3.6-flash`, `gemini-4-argon` or `claude-sonnet-5-5`. Model strings saved in older automations keep working: `gemini/gemini-2.5-flash` and `anthropic/claude-sonnet-4-6` are converted, and models Vertex no longer serves run on a current model of the same strength (for example `gemini-2.5-pro` runs on the `pro` tier). Models Vertex doesn't offer, such as OpenAI models, run on the default models. Optexity logs a warning in both cases; update the saved value when you next edit the automation. ## Resolution Order | Priority | Source | Example | | - | - | - | | 1 | Action-level `llm_model_name` | `extraction_action.llm.llm_model_name` | | 2 | Task-level `llm_model_name` | Top-level field on the automation | | 3 | `LLM_MODEL` environment variable | `gemini-3.6-flash` | | 4 | The `lite` tier | `gemini-3.5-flash-lite` | ### Action-Level Override Give a single action a stronger (or cheaper) model than the rest of the automation: ```json theme={null} { "type": "action_node", "extraction_action": { "llm": { "extraction_format": { "invoice_total": "str" }, "extraction_instructions": "Read the invoice total from the summary table.", "llm_model_name": "gemini-4-argon" } } } ``` Actions that accept `llm_model_name`: `extraction_action.llm`, `extraction_action.pdf`, `extraction_action.locator`, `misc_action.llm_query`, `dynamic_form_mapping_action`, and `interaction_action.captcha`. > [!NOTE] > > Captcha solving uses the `pro` tier unless the captcha action sets `llm_model_name`, because captcha grids need a stronger vision model. ## Fallbacks A call that hits a rate limit, an outage or a model that can't serve it moves to the next model automatically: * A model chosen with `llm_model_name` or `LLM_MODEL` falls back to `LLM_MODEL_FALLBACK` if set, otherwise to the other models in its tier (the `standard` tier if the model isn't in one). * Tier calls (default calls, captcha) fall back within their tier. * Temporary errors are retried once on the same model, and output that doesn't match the requested format is retried up to twice, before moving on. ## Cost Tracking Token usage is reported per task from Vertex AI's token counts. Cost is estimated from list prices (non-global locations such as `us` are priced 10% higher); the billed amount is in Google Cloud Billing, where calls are tagged with `service`, `feature`, `environment` and `tier` labels. ## Migrating from LiteLLM | Before | After | | - | - | | `GOOGLE_API_KEY`, `LLM_MODEL_API_KEY`, `LLM_MODEL_FALLBACK_API_KEY`, provider keys | Removed — set `GOOGLE_CLOUD_PROJECT` and sign in with Application Default Credentials | | `LLM_MODEL=gemini/gemini-3.5-flash-lite` | `LLM_MODEL=gemini-3.5-flash-lite`, or leave it unset | | `"llm_provider": "gemini", "llm_model_name": "gemini-2.5-flash"` | `"llm_model_name": "gemini-3.6-flash"` | `llm_provider` is deprecated. Automations that still set it keep working. # 1Password Integration Source: https://docs.optexity.com/docs/advanced/onepassword Store and retrieve secrets securely using 1Password Use 1Password to store sensitive data like passwords and API keys. Values are retrieved at runtime without exposing secrets in your workflow. ## Setup ### Create an Item in 1Password 1. Go to [my.1password.com](https://my.1password.com/home) 2. Open an existing vault or click **New Vault** to create one 3. Click **New Item** and choose a category (e.g. **Login** or **Password**) 4. Fill in the fields you want to retrieve (e.g. a field named `password`) 5. Save the item and note the **vault name**, **item name**, and **field name** — you'll use these in `secure_parameters` ### Get Service Account Token 1. Go to [my.1password.com](https://my.1password.com/home) 2. Create a service account 3. Copy the token [https://www.youtube.com/embed/5LHseHgkPs4](https://www.youtube.com/embed/5LHseHgkPs4) ### Configure Environment Add it directly on the **Integrations** page of the dashboard, or if you are running locally, add to your `.env` file: ```bash theme={null} OP_SERVICE_ACCOUNT_TOKEN=your_service_account_token ``` *** ## Usage Move parameters from `input_parameters` to `secure_parameters`: **Before:** ```json theme={null} { "input_parameters": { "password": ["password_value"] } } ``` **After:** ```json theme={null} { "secure_parameters": { "password": [{ "onepassword": { "vault_name": "my_vault", "item_name": "my_login", "field_name": "password" } }] } } ``` *** ## Properties | Property | Description | | - | - | | `vault_name` | 1Password vault name | | `item_name` | Item name in vault | | `field_name` | Field to retrieve | | `type` | Set to `"totp_secret"` to generate TOTP codes | | `digits` | Required when `type` is `"totp_secret"` (e.g., `6`) | *** ## TOTP from 1Password Retrieve TOTP secret and generate codes: ```json theme={null} { "secure_parameters": { "auth_code": [{ "onepassword": { "type": "totp_secret", "vault_name": "vault", "item_name": "login", "field_name": "totp_secret", "digits": 6 } }] } } ``` > [!TIP] > > See [TOTP Integration](/docs/advanced/totp-integration) for more 2FA options. *** ## Revoking Access Revoke the service account token anytime from the 1Password dashboard. # Orchestration Source: https://docs.optexity.com/docs/advanced/orchestration Coordinate multiple automations with callbacks Orchestrate complex workflows by chaining automations using callbacks instead of polling. ## Architecture ``` Your Server Optexity │ │ ├──POST /inference────────────>│ │ ├── Execute automation │<─────────POST /callback──────┤ │ │ └──Process result, continue────> ``` *** ## Minimal Example ```python theme={null} import asyncio import httpx from fastapi import FastAPI, Request from contextlib import asynccontextmanager callbacks: dict[str, dict] = {} CALLBACK_TIMEOUT = 600 async def run_workflow(items: list[str]): async with httpx.AsyncClient() as client: for item in items: resp = await client.post( "https://inference.optexity.com/api/v1/inference", json={ "endpoint_name": "my-automation", "input_parameters": {"email": [item]}, "unique_parameter_names": ["email"] }, ) task_id = resp.json()["task_id"] callbacks[task_id] = {"event": asyncio.Event(), "data": None} await asyncio.wait_for( callbacks[task_id]["event"].wait(), timeout=CALLBACK_TIMEOUT, ) result = callbacks[task_id]["data"] print("Completed:", result) callbacks.pop(task_id, None) @asynccontextmanager async def lifespan(app: FastAPI): task = asyncio.create_task(run_workflow(["a@test.com", "b@test.com"])) yield task.cancel() app = FastAPI(lifespan=lifespan) @app.post("/receive_callback") async def receive_callback(req: Request): payload = await req.json() task_id = payload.get("task_id") entry = callbacks.get(task_id) if entry: entry["data"] = payload entry["event"].set() return {"ok": True} ``` *** ## Dependencies ```bash theme={null} pip install fastapi uvicorn httpx asyncio ``` *** ## Running ```bash theme={null} uvicorn main:app --reload --port 4000 ``` *** ## How It Works | Component | Purpose | | - | - | | `callbacks` dict | Stores pending tasks with asyncio Events | | `run_workflow` | Starts tasks, waits for callbacks | | `/receive_callback` | Receives Optexity notifications, signals completion | ### Flow 1. **Start task**: POST to Optexity inference API 2. **Register callback**: Create asyncio Event for task\_id 3. **Wait**: Event.wait() suspends until callback received 4. **Callback received**: Event.set() wakes workflow 5. **Process**: Use result, continue to next task *** ## Benefits | Benefit | Description | | - | - | | No polling | Tasks complete asynchronously | | Efficient | Lightweight asyncio coordination | | Scalable | Extend for parallel tasks | | Reliable | Timeout handling included | > [!WARNING] > > Your callback URL must be publicly accessible from Optexity's servers. # Proxy Setup Source: https://docs.optexity.com/docs/advanced/proxy-setup Configure proxies for automations Use proxies to run automations through different IP addresses. ## Local Development Set environment variables: ```bash theme={null} PROXY_URL=http://proxy-server:8080 PROXY_USERNAME=proxy-user # optional PROXY_PASSWORD=proxy-pass # optional PROXY_PROVIDER=other # optional, default: oxylabs ``` *** ## Production No configuration needed. Optexity automatically uses proxies in production. *** ## Enabling Proxy in Inference Set `use_proxy: true` in the inference request: ```json theme={null} { "endpoint_name": "extract_price_stockanalysis", "input_parameters": { "search_term": ["NVDA"] }, "unique_parameter_names": [], "use_proxy": true } ``` *** ## Proxy Providers ### Webshare [Webshare](https://webshare.io/) provides 10 free proxies for testing. ```bash theme={null} PROXY_URL=http://your-webshare-proxy:port PROXY_USERNAME=your-username PROXY_PASSWORD=your-password ``` # Timing & Retries Source: https://docs.optexity.com/docs/advanced/timing-retries Controlling execution timing and handling failures Web automation requires careful timing. Pages load at different speeds, elements appear dynamically, and networks can be slow. ## Timing Controls | Level | Properties | | - | - | | **Automation** | `max_retries` | | **Action Node** | `before_sleep_time`, `end_sleep_time`, `expect_new_tab`, `max_new_tab_wait_time` | | **Interaction Action** | `max_tries`, `max_timeout_seconds_per_try` | ## Sleep Times ### before\_sleep\_time Wait **before** executing the action. Use when page needs to load or settle. ```json theme={null} { "type": "action_node", "interaction_action": { ... }, "before_sleep_time": 3.0 } ``` ### end\_sleep\_time Wait **after** action completes. Use when subsequent actions depend on this action's effects. ```json theme={null} { "type": "action_node", "interaction_action": { ... }, "end_sleep_time": 2.0 } ``` ### Default Values | Action Type | `before_sleep_time` | `end_sleep_time` | | - | - | - | | Interaction | `0.0` | `1.0` | | Extraction | `3.0` | `0.0` | | Assertion | `0.0` | `0.0` | | 2FA | `0.0` | `0.0` | > [!NOTE] > > Sleep times must be between 0 and 10 seconds. *** ## Automation-Level Retries `max_retries` on the automation reruns the **entire automation** when an unexpected error occurs (e.g. browser crash, network failure). | Property | Type | Default | Description | | - | - | - | - | | `max_retries` | `int` | `0` | Total run attempts. `1` = no retry, `1` = one retry, etc. | ```json theme={null} { "url": "https://example.com", "max_retries": 3, "nodes": [ ... ] } ``` > [!NOTE] > > `AssertionError` failures (e.g. failed assertions) are **never** retried regardless of `max_retries`. Retries only apply to unexpected runtime errors. *** ## Element-Level Retry Configuration Control how Optexity retries finding elements. | Property | Type | Default | Description | | - | - | - | - | | `max_tries` | `int` | `10` | Maximum retry attempts | | `max_timeout_seconds_per_try` | `float` | `1.0` | Timeout per attempt | ```json theme={null} { "interaction_action": { "max_tries": 15, "max_timeout_seconds_per_try": 2.0, "click_element": { ... } } } ``` ### How Retries Work 1. Attempt to find element using `command` or `xpath` 2. If not found within timeout, retry 3. After all tries exhausted, use AI with `prompt_instructions` 4. If AI can't find it, action fails > [!TIP] > > Increase `max_tries` rather than timeout per try. This finds elements faster when they appear while still handling slow pages. *** ## Handling New Tabs ### expect\_new\_tab Set when action opens a new browser tab: ```json theme={null} { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"link\", name=\"Open Report\")" } }, "expect_new_tab": true } ``` When `expect_new_tab=True`: * `max_new_tab_wait_time` automatically set to `10.0` * Automation waits for new tab * Focus switches to new tab *** ## Common Patterns ### Slow-Loading Pages ```json theme={null} { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Search\")" } }, "end_sleep_time": 5.0 } ``` ### Dynamic AJAX Content ```json theme={null} { "type": "action_node", "interaction_action": { "max_tries": 15, "max_timeout_seconds_per_try": 1.0, "click_element": { "command": "get_by_text(\"Results loaded\")" } }, "before_sleep_time": 2.0 } ``` ### Optional Elements ```json theme={null} { "type": "action_node", "interaction_action": { "max_tries": 3, "click_element": { "command": "get_by_role(\"button\", name=\"Dismiss\")", "skip_prompt": true, "assert_locator_presence": true } } } ``` *** ## Troubleshooting | Symptom | Likely Cause | Solution | | - | - | - | | "Element not found" | Page not loaded | Increase `before_sleep_time` | | Clicking wrong element | Page still loading | Increase `before_sleep_time` | | Missing extracted data | Content not rendered | Increase wait before extraction | | Next action fails | Previous effect not ready | Increase `end_sleep_time` | | Random failures | Race conditions | Increase retries and timeouts | *** ## Best Practices | Practice | Recommendation | | - | - | | Start conservative | Use longer waits initially, optimize later | | Use defaults | Let action-type defaults handle most cases | | Wait before extraction | Ensure page stability | | Wait after navigation | Give pages time to load | | Increase retries | Prefer more tries over longer timeouts | # Time-Based One-Time Password (TOTP) Integration Source: https://docs.optexity.com/docs/advanced/totp-integration Handle TOTP codes in automations Generate TOTP codes from authenticator apps (Google Authenticator, Microsoft Authenticator, Authy) for 2FA. > [!NOTE] > > If you are looking for 2FA codes from Email or Slack, please refer to the [Two-Factor Authentication Integration](/docs/advanced/two-fa-integration) documentation. ## Get Your TOTP Secret Follow this guide to extract your TOTP secret: [TOTP Secret Extraction Guide](https://cavalloj.medium.com/totp-secret-extraction-from-qr-codes-ee097b4c687f) *** ## Direct TOTP Secret Provide the TOTP secret directly in `secure_parameters`: ```json theme={null} { "secure_parameters": { "auth_code": [{ "totp": { "totp_secret": "YOUR_BASE32_SECRET", "digits": 6 } }] } } ``` | Property | Type | Default | Description | | - | - | - | - | | `totp_secret` | `str` | Required | Base32-encoded TOTP secret | | `digits` | `int` | `6` | Number of digits in code | *** ## TOTP via 1Password (Recommended) Store the TOTP secret in 1Password for better security: ```json theme={null} { "secure_parameters": { "auth_code": [{ "onepassword": { "type": "totp_secret", "vault_name": "my_vault", "item_name": "my_login", "field_name": "totp_secret", "digits": 6 } }] } } ``` > [!TIP] > > See [1Password Integration](/docs/advanced/onepassword) for setup instructions. *** ## Using in Automations Reference the TOTP code like any parameter: ```json theme={null} { "type": "action_node", "interaction_action": { "input_text": { "command": "get_by_label(\"Verification Code\")", "input_text": "{auth_code[0]}", "prompt_instructions": "Enter the 2FA code" } } } ``` *** ## Complete Example Here's a complete automation that authenticates with a TOTP-protected service: ```json theme={null} { "url": "https://sample.com/", "parameters": { "input_parameters": { "username": [ "provider_username" ] }, "secure_parameters": { "auth_code": [ { "totp": { "totp_secret": "YUNFZZZH" } } ] }, "generated_parameters": {} }, "nodes": [ { "type": "action_node", "interaction_action": { "input_text": { "command": "get_by_test_id(\"totp\")", "prompt_instructions": "Enter the 2FA code", "input_text": "{auth_code[0]}" } } } ] } ``` This example: * Uses a direct TOTP secret (`YUNFZZZH`) from `secure_parameters` * Accepts a `username` as an `input_parameter` * Enters the generated TOTP code into the input field identified by `test_id="totp"` # Two-Factor Authentication Integration Source: https://docs.optexity.com/docs/advanced/two-fa-integration Fetch 2FA codes from Gmail, Slack, and Twilio SMS for automations Automatically fetch 2FA verification codes from email, Slack, or Twilio SMS messages—no manual code entry required. > [!NOTE] > > For TOTP codes from authenticator apps, see [TOTP Integration](/docs/advanced/totp-integration). ## Methods | Method | Source | Use Case | | - | - | - | | **Email** | Gmail inbox | Codes sent via email | | **Slack** | Slack workspace | Codes sent via Slack | | **SMS** | Twilio SMS | Codes sent via text | All methods follow the same pattern: 1. Fetch messages from the source (email inbox, Slack channel, or Twilio SMS) 2. Extract the 2FA code from matching messages 3. Store the code in the specified `output_variable_name` 4. Use `{output_variable_name}` in subsequent actions to input the code *** ## Setup 1. Go to the [Optexity dashboard](https://app.optexity.com) → **Integrations** 2. Click the desired integration → **Connect** 3. Follow the prompts to grant access ### Twilio SMS 1. Log in to your Twilio account and open the Twilio Console 2. Go to **API Credentials** and copy your **Account SID** and **Auth Token** 3. Add or select a Twilio phone number that will send or receive verification texts 4. Connect the Twilio integration in Optexity using the same credentials and phone number *** ## Email 2FA Fetch verification codes from email messages: ```json theme={null} { "type": "action_node", "extraction_action": { "two_fa_action": { "action": { "type": "email_two_fa_action", "receiver_email_address": "user@example.com", "sender_email_address": "noreply@example.com", "integration_email_address": "automation-inbox@example.com" }, "output_variable_name": "auth_code" } } } ``` ### Properties | Property | Type | Required | Description | | - | - | - | - | | `receiver_email_address` | `str` | Yes | Recipient mailbox to search for 2FA emails | | `sender_email_address` | `str` | Yes | Sender to filter (e.g., `noreply@site.com`) | | `integration_email_address` | `str \| null` | No | Connected integration mailbox. Defaults to `receiver_email_address` if omitted | > [!TIP] > > Use `integration_email_address` when the connected mailbox is different from the mailbox receiving the 2FA email (for example, alias/forwarding setups). *** ## Slack 2FA Fetch verification codes from Slack messages: ```json theme={null} { "type": "action_node", "extraction_action": { "two_fa_action": { "action": { "type": "slack_two_fa_action", "slack_workspace_domain": "mycompany.slack.com", "channel_name": "security-codes", "sender_name": "Security Bot" }, "output_variable_name": "auth_code" } } } ``` ### Properties | Property | Type | Required | Description | | - | - | - | - | | `slack_workspace_domain` | `str` | Yes | Workspace domain (identifies integration) | | `channel_name` | `str` | Yes | Channel containing the 2FA messages | | `sender_name` | `str` | Yes | Name of the bot/user sending the codes | *** ## SMS 2FA Fetch verification codes from Twilio SMS messages: ```json theme={null} { "type": "action_node", "extraction_action": { "two_fa_action": { "action": { "type": "sms_two_fa_action", "from_number": "+18777804236", "to_number": "+19897878948" }, "output_variable_name": "auth_code" } }, "before_sleep_time": 3, "end_sleep_time": 0 } ``` ### Properties | Property | Type | Required | Description | | - | - | - | - | | `from_number` | `str` | Yes | Twilio phone number sending the SMS code | | `to_number` | `str` | Yes | Destination phone number receiving the SMS | *** ## Common Properties These apply to Email, Slack, and SMS 2FA: | Property | Type | Default | Description | | - | - | - | - | | `output_variable_name` | `str` | — | Variable name to store the extracted code | | `instructions` | `str` | `None` | Optional custom instructions for code extraction | | `max_wait_time` | `float` | `300.0` | Maximum wait time in seconds | | `check_interval` | `float` | `10.0` | Polling interval in seconds | > [!TIP] > > The action polls for the 2FA code every `check_interval` seconds until the code arrives or `max_wait_time` is reached. *** ## Using the Code Reference the extracted code in subsequent actions: ```json theme={null} { "type": "action_node", "interaction_action": { "input_text": { "command": "get_by_label(\"Verification Code\")", "input_text": "{auth_code[0]}" } } } ``` # Action Node Source: https://docs.optexity.com/docs/building-automations/action-node Single atomic actions in automations An `action_node` represents a single atomic action. Each node contains exactly one action type. ## Structure ```json theme={null} { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Submit\")", "prompt_instructions": "Click the submit button" } }, "before_sleep_time": 0.0, "end_sleep_time": 1.0 } ``` ## Action Types Each action node contains exactly one of: | Action | Purpose | Documentation | | - | - | - | | `interaction_action` | Click, type, select, navigate | [Interaction Actions](/docs/action-types/interaction-action) | | `extraction_action` | Extract data, screenshots | [Extraction Actions](/docs/action-types/extraction-action) | | `assertion_action` | Verify page conditions | [Assertion Actions](/docs/action-types/assertion-action) | | `python_script_action` | Custom Python code | [Python Scripts](/docs/action-types/python-script-action) | | `dynamic_form_mapping_action` | Extract form keys, fetch values from your endpoint | [Dynamic Form Mapping](/docs/action-types/dynamic-form-mapping) | | `sleep_action` | Pure wait / timing step | [Sleep Action](/docs/action-types/sleep-action) | ## Timing Properties | Property | Type | Default | Description | | - | - | - | - | | `before_sleep_time` | `float` | `0.0` (extractions: `3.0`) | Seconds to wait before action | | `end_sleep_time` | `float` | `1.0` (extractions: `0.0`) | Seconds to wait after action | | `expect_new_tab` | `bool` | `False` | Action opens a new tab | | `max_new_tab_wait_time` | `float` | `0.0` (if expect\_new\_tab: `10.0`) | Max wait for new tab | | `include_full_page` | `bool` | `False` | Include elements outside the viewport in this node's accessibility tree. Index prediction, extraction, and other axtree reads use the full page when this is `true` | ### Default Timing by Action Type | Action Type | `before_sleep_time` | `end_sleep_time` | | - | - | - | | Interaction | `0.0` | `1.0` | | Extraction | `3.0` | `0.0` | | Dynamic form mapping | `3.0` | `0.0` | | Assertion | `0.0` | `0.0` | | 2FA | `0.0` | `0.0` | ## Examples ### Basic Click ```json theme={null} { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Continue\")", "prompt_instructions": "Click continue" } } } ``` ### With Custom Timing ```json theme={null} { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Submit\")" } }, "before_sleep_time": 2.0, "end_sleep_time": 3.0 } ``` ### Sleep-Only Node Use a `sleep_action` when you want to pause without performing any browser interaction. `sleep_time` is specified in seconds: ```json theme={null} { "type": "action_node", "sleep_action": { "sleep_time": 5.0 } } ``` ### Fail State Node A `fail_state_action` is used to handle failure states in the automation. It will stop the automation, raise an exception with the provided failure message and mark it as failed. ```json theme={null} { "type": "action_node", "fail_state_action": { "failure_message": "Automation completed at one of the failure states." } } ``` `fail_state_action` can also be used with variables. ```json theme={null} { "type": "action_node", "fail_state_action": { "failure_message": "Error in login process for user {username[0]}" } } ``` ### Opens New Tab ```json theme={null} { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"link\", name=\"View Details\")" } }, "expect_new_tab": true } ``` > [!TIP] > > See [Timing & Retries](/docs/advanced/timing-retries) for detailed timing configuration. # Automation Structure Source: https://docs.optexity.com/docs/building-automations/automation-structure Understanding the structure of Optexity automations Optexity uses a declarative model for browser automation. Define **what** actions to perform, and Optexity handles the **how** using AI-assisted element location. ## Overview ``` Automation ├── url # Starting point ├── browser_channel # "chromium" or "chrome" ├── os_emulation # OS to emulate: "windows", "linux", or null ├── max_retries # Total run attempts on failure (default: 1) ├── expected_downloads # Number of files to download ├── enable_browser_alerts # Log native JS dialogs (alert/confirm/prompt) as they're auto-accepted ├── reuse_page_if_already_on_url # Skip navigation when already on url (dedicated only) ├── parameters # Input, secure, and generated variables └── nodes[] # Sequence of actions ├── action_node ├── for_loop_node ├── if_else_node └── private_node ``` ## Complete Example ```json theme={null} { "url": "https://example.com/login", "browser_channel": "chromium", "os_emulation": null, "max_retries": 1, "expected_downloads": 0, "parameters": { "input_parameters": { "email": ["user@example.com"] }, "secure_parameters": {}, "generated_parameters": {} }, "nodes": [ { "type": "action_node", "interaction_action": { "input_text": { "command": "get_by_label(\"Email\")", "input_text": "{email[0]}" } } } ] } ``` ## Properties | Property | Type | Default | Description | | - | - | - | - | | `url` | `str` | — | Starting URL—use the URL closest to your target to reduce navigation steps | | `browser_channel` | `"chromium" \| "chrome"` | `"chromium"` | Browser to use | | `os_emulation` | `"windows" \| "linux" \| null` | `null` | OS user-agent to emulate; `null` uses the default system OS | | `max_retries` | `int` | `0` | Total number of run attempts. `0` = no retry, `1` = one retry, etc. | | `expected_downloads` | `int` | `0` | Number of expected downloads (automation waits for completion) | | `enable_browser_alerts` | `bool` | `false` | Log native JS dialogs (`alert`, `confirm`, `prompt`) as they're encountered. See [Browser Alerts](#browser-alerts) | | `reuse_page_if_already_on_url` | `bool` | `false` | Start on the existing page instead of navigating, when a dedicated browser is already on `url`. See [Reuse Page If Already On URL](#reuse-page-if-already-on-url) | | `parameters` | `Parameters` | — | Input, secure, and generated variables | | `nodes` | `list[action_node \| for_loop_node \| if_else_node \| private_node]` | — | Ordered list of actions | ## Parameters Three types of parameters control data flow: | Type | Purpose | Example | | - | - | - | | `input_parameters` | Values provided before execution | Username, search queries | | `secure_parameters` | Sensitive data from secure storage | Passwords, API keys | | `generated_parameters` | Values extracted during execution | Order IDs, confirmation numbers | ```json theme={null} { "parameters": { "input_parameters": { "username": ["admin@example.com"] }, "secure_parameters": { "password": [{ "onepassword": { "vault_name": "vault", "item_name": "login", "field_name": "password" } }] }, "generated_parameters": { "order_ids": [] } } } ``` > [!TIP] > > See [Parameters](/docs/building-automations/parameters) for detailed usage. ## Node Types | Node | Purpose | Documentation | | - | - | - | | `action_node` | Single atomic action | [Action Node](/docs/building-automations/action-node) | | `for_loop_node` | Iterate over list values or locator matches | [For Loop Node](/docs/building-automations/for-loop-node) | | `if_else_node` | Conditional execution | [If Else Node](/docs/building-automations/if-else-node) | | `private_node` | Closed-source handler, runs only on Optexity's servers | [Private Node](/docs/building-automations/private-node) | ## Browser Channel | Channel | Use When | | - | - | | `"chromium"` | Default, works for most sites | | `"chrome"` | Site requires Chrome specifically, or automation is unreliable with Chromium | ```json theme={null} { "browser_channel": "chrome" } ``` ## Expected Downloads Set this to wait for file downloads to complete: ```json theme={null} { "expected_downloads": 3 } ``` The automation waits until all expected files are downloaded before completing. ## Browser Alerts Native JS dialogs (`alert`, `confirm`, `prompt`) are always auto-accepted so they never block the automation. Set `enable_browser_alerts` to also log each dialog's type and message when one is triggered: ```json theme={null} { "enable_browser_alerts": true } ``` > [!TIP] > > Turn this on while building or debugging an automation against a site that pops native dialogs, so you can see in the logs when and why one fired. Leave it off otherwise. ## OS Emulation Override the OS reported in the browser's user-agent string: | Value | Use When | | - | - | | `null` | Default — uses the host system's OS (recommended for most sites) | | `"windows"` | Site behaves differently on Windows (e.g. Windows-only portals) | | `"linux"` | Site behaves differently on Linux | ```json theme={null} { "os_emulation": "windows" } ``` > [!TIP] > > Only set `os_emulation` when a site serves different content or layouts based on the OS. Leave it as `null` otherwise. ## Max Retries Control how many times the full automation reruns when an unexpected error occurs: ```json theme={null} { "max_retries": 3 } ``` | Value | Behavior | | - | - | | `1` (default) | Runs once; fails immediately on error | | `2` | Runs up to twice (1 retry) | | `3` | Runs up to three times (2 retries) | > [!NOTE] > > `max_retries` is the **total number of attempts**, not the number of extra retries. `AssertionError` failures are never retried regardless of this value. > [!TIP] > > See [Timing & Retries](/docs/advanced/timing-retries) for controlling per-element retry behavior (`max_tries`) vs. full-automation retries (`max_retries`). ## Reuse Page If Already On URL Every run navigates to `url` before the first node executes, even on a warm [dedicated instance](/docs/inference/dedicated-instances). Set `reuse_page_if_already_on_url` to leave the existing page untouched when the reused browser is **already sitting on `url`**: ```json theme={null} { "reuse_page_if_already_on_url": true } ``` Use this only for portals that break when their page is reloaded — some session-bound portals throw an error or invalidate the session on any refresh, so the safest thing is not to navigate at all. | Condition | Behavior | | - | - | | Dedicated run, browser already on `url` | Nodes start on the existing page; no navigation | | URL differs, or the page is unresponsive | Normal navigation to `url` | | Non-dedicated run | Normal navigation to `url` (flag has no effect) | The URL comparison is exact on scheme, host, path, query string, and fragment — only a trailing slash on the path is ignored. A hash-routed portal on `#/dashboard` therefore does not match a `url` of `#/home`. Anything that doesn't match falls back to normal navigation, so enabling the flag can never leave a run stranded. > [!WARNING] > > Your first node now runs against whatever the previous task left behind — an open modal, a filled form, a scrolled list. Make it tolerant of a mid-session page, or pair it with an [if-else node](/docs/building-automations/if-else-node) that resets the portal to a known state. # For Loop Node Source: https://docs.optexity.com/docs/building-automations/for-loop-node Iterating over list values or page locator matches in automations Use `for_loop_node` to repeat actions for each value in a list, or for each element matching a Playwright locator on the page—processing search results, downloading multiple files, or clicking through table rows. Exactly one of `variable_name` or `locator` is required. ## Structure ### Variable loop ```json theme={null} { "type": "for_loop_node", "variable_name": "order_ids", "nodes": [ { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_text(\"{order_ids[index]}\")", "prompt_instructions": "Click order {order_ids[index]}" } } } ], "reset_nodes": [], "on_error_in_loop": "raise" } ``` ### Locator loop ```json theme={null} { "type": "for_loop_node", "locator": "get_by_role(\"row\")", "index_variable_name": "row", "nodes": [ { "type": "action_node", "interaction_action": { "click_element": { "command": "{locator[row]}.get_by_role(\"button\", name=\"Edit\")" } } } ], "reset_nodes": [], "on_error_in_loop": "raise" } ``` At loop start the runtime waits up to `locator_timeout` for the first match to attach, then waits until the match count stays unchanged for 1 second (so slowly streaming rows are not missed), snapshots that count once, and iterates `0 .. count-1`. An empty match set runs zero iterations with a warning (same as an empty list variable — the run does not fail). ## Properties | Property | Type | Default | Description | | - | - | - | - | | `variable_name` | `str \| None` | `None` | Parameter list to iterate over. Mutually exclusive with `locator` | | `locator` | `str \| None` | `None` | Playwright locator command whose matches are iterated. Mutually exclusive with `variable_name` | | `index_variable_name` | `str` | `"index"` | Placeholder name used in `{var[]}` / `{locator[]}` and bare `{}` | | `locator_timeout` | `float` | `5.0` | Locator loops only: seconds to wait for the first match to attach; after that, count must stay stable for 1s before the loop starts | | `max_iterations` | `int \| null` | `null` | Cap on iterations; extra items are skipped with a warning | | `nodes` | `list[action_node \| for_loop_node \| if_else_node \| assert_locator_node]` | — | Actions for each iteration | | `reset_nodes` | `list[action_node \| for_loop_node \| if_else_node \| assert_locator_node]` | `[]` | Actions to run after each iteration | | `on_error_in_loop` | `"continue" \| "break" \| "raise"` | `"raise"` | Error handling behavior | ## The Index Variable ### Variable loops Inside a variable loop, `{variable[index]}` references the current iteration's value. The token `index` comes from `index_variable_name` (default `"index"`): ``` order_ids = ["ORD-001", "ORD-002", "ORD-003"] Iteration 1: {order_ids[index]} → "ORD-001" Iteration 2: {order_ids[index]} → "ORD-002" Iteration 3: {order_ids[index]} → "ORD-003" ``` | Pattern | Expands to | | - | - | | `{order_ids[index]}` | Current list value | | `{index_of(order_ids)}` | Current numeric index (`0`, `1`, …) | | `{index}` | Current numeric index (same as `index_of`, using `index_variable_name`) | ### Locator loops Locator loops use the same shape as variable loops: `{locator[index]}` is the current match (like `{order_ids[index]}`), and bare `{index}` is the numeric index: ``` locator = get_by_role("row") (3 matches) Iteration 1: {locator[index]} → get_by_role("row").nth(0) Iteration 2: {locator[index]} → get_by_role("row").nth(1) Iteration 3: {locator[index]} → get_by_role("row").nth(2) ``` With `index_variable_name: "row"`: | Pattern | Expands to | | - | - | | `{locator[row]}` | Current match command (`.nth()`) | | `{row}` | Current numeric index (`0`, `1`, …) | | `{locator[row]}.get_by_role("button")` | Chained command on the current match | There is no `{index_of(locator)}` counterpart to `{index_of()}`. It would carry no per-loop name, so in nested locator loops the outer loop would bind the inner loop's occurrences. Use the bare `{}` form, which is scoped per level. ### Using Multiple Variables in a Loop You can iterate over **multiple parameters in parallel** by passing a **comma-separated list** to `variable_name`. **Important:** The **first** in `variable_name` is the **main loop variable** that controls: * How many iterations run (loop length) * Which index is used for each step All additional names in the list are **synced to the same index**. In other words: * `variable_name: "primary_var,secondary_var"` → loop length comes from `primary_var` * `{primary_var[index]}` and `{secondary_var[index]}` both use the **same** `index` on each iteration For each iteration, `{variable[index]}` is expanded for **every** name in the list: ```json theme={null} { "type": "for_loop_node", "variable_name": "primary_var,secondary_var", "nodes": [ { "type": "action_node", "interaction_action": { "input_text": { "command": "get_by_role(\"textbox\", name=\"Primary Field\")", "input_text": "{primary_var[index]}" } } }, { "type": "action_node", "interaction_action": { "input_text": { "command": "get_by_role(\"textbox\", name=\"Secondary Field\")", "input_text": "{secondary_var[index]}" } } } ] } ``` If your parameters are: * `primary_var = ["Value A1", "Value A2"]` * `secondary_var = ["Value B1", "Value B2"]` Then: * Iteration 1 uses `{primary_var[index]}` → `"Value A1"` and `{secondary_var[index]}` → `"Value B1"` * Iteration 2 uses `{primary_var[index]}` → `"Value A2"` and `{secondary_var[index]}` → `"Value B2"` ## Waiting for Matches Locator loops only. `count()` does not auto-wait, and the sleep after an action returns as soon as the page has loaded — which is immediately for an SPA that re-renders without navigating. Counting straight away would see zero rows and skip the loop body silently, so a locator loop: 1. Waits up to `locator_timeout` (default 5s) for the first match to attach 2. Then waits until the match count stays unchanged for **1 second**, so rows that appear shortly after the first paint are included | Situation | Behavior | | - | - | | Matches appear and count stays stable for 1s | Loop runs over that stable count | | No match within `locator_timeout` | Zero iterations, warning `No matching locator found`, run does not fail | | Locator command doesn't resolve (browser/page gone) | Raises, rather than silently looping zero times | | More matches than `max_iterations` | Extra items skipped, logged as a warning | Raise `locator_timeout` for slow grids; set it to `0` to skip the first-match wait when rows are already on the page (the 1s stability wait still applies). ## Storing One Value Per Iteration `output_variable_name` is expanded like any other placeholder, so template it to give each iteration its own variable: ```json theme={null} { "type": "action_node", "extraction_action": { "locator": { "command": "{locator[row]}.locator(\"td.PatientNameCell\")", "output_variable_name": "patient_{row}", "extraction_format": { "patient_{row}": "str" } } } } ``` Without the placeholder every iteration overwrites the same variable and only the last row survives. Extracted values are appended to the task's output data on every iteration either way — template the name only when you need to reference a specific row's value later. ## Nested Loops `for_loop_node` can contain another `for_loop_node`. Give each loop a distinct `index_variable_name` so the outer index stays usable inside the inner loop: ```json theme={null} { "type": "for_loop_node", "variable_name": "pages", "index_variable_name": "page_i", "nodes": [ { "type": "for_loop_node", "variable_name": "items", "index_variable_name": "item_i", "nodes": [ { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_text(\"{items[item_i]}\")", "prompt_instructions": "Page {page_i} ({pages[page_i]}), item {item_i} ({items[item_i]})" } } } ], "reset_nodes": [], "on_error_in_loop": "raise" } ], "reset_nodes": [], "on_error_in_loop": "raise" } ``` | Omitted `index_variable_name`? | Behavior | | - | - | | Both loops use default `"index"` | `{pages[index]}` and `{items[index]}` still work. Bare `{index}` in the inner body is bound by the **outer** loop. Prefer `{index_of(items)}` for the inner numeric index, or set distinct names. | | Distinct names (`page_i`, `item_i`) | Outer and inner indexes can both be referenced clearly inside the innermost body. | The inner loop's list must already exist when that loop starts (for example, extract `items` inside the outer iteration before the inner `for_loop_node`). An inner loop's `locator` is itself expanded by the outer loop, so a locator loop can be scoped to the outer iteration — `"locator": "{locator[row]}.locator(\"td\")"` inside a loop over `tr` elements iterates that row's cells. The inner count is taken from the already-scoped locator. ## Data Sources Loop variables can come from: | Source | When to Use | | - | - | | `input_parameters` | Known values before execution | | `generated_parameters` | Values extracted during automation | | `secure_parameters` | Sensitive values from secure storage | For dynamic iteration, extract values first: ```json theme={null} [ { "type": "action_node", "extraction_action": { "llm": { "extraction_format": { "item_ids": "List[str]" }, "output_variable_names": ["item_ids"], "extraction_instructions": "Extract all item IDs" } } }, { "type": "for_loop_node", "variable_name": "item_ids", "nodes": [ ... ] } ] ``` ## Reset Nodes Actions that run **after each iteration** to return the browser to a known state. Essential for loops that navigate away from the starting page. Reset nodes also receive the current iteration's placeholder bindings. ```json theme={null} { "type": "for_loop_node", "variable_name": "documents", "nodes": [ { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_text(\"{documents[index]}\")" } } } ], "reset_nodes": [ { "type": "action_node", "interaction_action": { "close_tabs_until": { "matching_url": "https://example.com/documents" } } } ] } ``` ### Reset Strategy Recommendations | Action | Reliability | Use When | | - | - | - | | `close_tabs_until` | High | Actions open new tabs | | `go_to_url` | High | Navigate to known URL | | `click_element` (navbar) | Medium | Persistent navigation exists | | `go_back` | Low | Avoid—fails on errors | > [!WARNING] > > Avoid `go_back` for reset nodes. If an error occurs mid-loop, the browser may be on an unexpected page. ## Error Handling | Behavior | Description | | - | - | | `"raise"` | Stop automation on error (default) | | `"continue"` | Skip failed iteration, continue to next | | `"break"` | Stop loop, continue with next node outside loop | ```json theme={null} { "type": "for_loop_node", "variable_name": "files", "on_error_in_loop": "continue", "nodes": [ ... ] } ``` > [!TIP] > > Use `"continue"` when some items may fail but you want to process as many as possible (e.g., downloading files where some may be missing). ## Common Patterns ### Click Every Matching Element ```json theme={null} { "type": "for_loop_node", "locator": "get_by_role(\"checkbox\")", "nodes": [{ "type": "action_node", "interaction_action": { "click_element": { "command": "{locator[index]}" } } }] } ``` ### Download Multiple Files ```json theme={null} { "type": "for_loop_node", "variable_name": "doc_links", "nodes": [{ "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"link\", name=\"{doc_links[index]}\")", "expect_download": true, "download_filename": "{doc_links[index]}.pdf" } } }] } ``` ### Download One File Per Matched Row When the rows are on the page but their identifiers aren't known in advance, loop the locator and use the index for the filename. `{row}` works in any field, not just `command`: ```json theme={null} { "type": "for_loop_node", "locator": "locator(\"tr.invoiceRow\")", "index_variable_name": "row", "on_error_in_loop": "continue", "nodes": [{ "type": "action_node", "interaction_action": { "click_element": { "command": "{locator[row]}.get_by_role(\"link\", name=\"Download\")", "expect_download": true, "download_filename": "invoice_{row}.pdf", "prompt_instructions": "Download the invoice on row {row}" } } }] } ``` Three matched rows download `invoice_0.pdf`, `invoice_1.pdf`, `invoice_2.pdf`. Keep the index in `download_filename` — a constant name makes every iteration overwrite the previous file. ### Process Items and Return ```json theme={null} { "type": "for_loop_node", "variable_name": "item_ids", "nodes": [ { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_text(\"{item_ids[index]}\")" } } }, { "type": "action_node", "extraction_action": { "llm": { "extraction_format": { "details": "str" }, "extraction_instructions": "Extract item details" } } } ], "reset_nodes": [{ "type": "action_node", "interaction_action": { "go_to_url": { "url": "https://example.com/items" } } }] } ``` ## Limitations * Variable loops require the list to be populated before loop execution * Locator loops wait for a stable `count()` (unchanged for 1s) once at start; DOM changes mid-loop do not change how many iterations run. Set `max_iterations` as a guard and re-check state inside the loop rather than relying on the initial count * Use `{locator[index]}` for the current match and bare `{index}` for the numeric index * When nesting locator loops, give each a distinct `index_variable_name` and reference `{locator[row]}` / `{locator[cell]}` so each level keeps its own match token * Locator loops that mutate the matched set (remove/add rows) should use `reset_nodes` and/or prefer extracting a stable list into a variable loop * Deep nesting is allowed; keep `index_variable_name` distinct per nesting level when you need both indexes # If Else Node Source: https://docs.optexity.com/docs/building-automations/if-else-node Conditional branching based on extracted data or runtime conditions Use `if_else_node` for conditional execution based on runtime conditions—handling different page states, optional elements, or branching logic. To determine condition logic, use **extraction actions** to capture data from the page and store it as variables. These variables are then referenced in `if_else_node` conditions to make branching decisions based on the extracted data. ## Structure ```json theme={null} { "type": "if_else_node", "condition": "has_captcha[0]", "if_nodes": [ { "type": "action_node", "interaction_action": { "agentic_task": { "task": "Solve the captcha", "max_steps": 10, "backend": "browser_use" } } } ], "else_nodes": [ { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Continue\")" } } } ] } ``` **Note:** Variables like `has_captcha` are extracted from the page using an extraction action (see [Using Extraction Nodes to Set Conditions](#using-extraction-nodes-to-set-conditions) below). ## Properties | Property | Type | Description | | - | - | - | | `condition` | `str` | Python-like expression to evaluate | | `if_nodes` | `list[action_node \| if_else_node \| for_loop_node]` | Actions when condition is true | | `else_nodes` | `list[...]` | Actions when condition is false (optional) | ## Using Extraction Nodes to Set Conditions **Extraction nodes** allow you to capture data from the page and store it as variables. These variables can then be referenced in `if_else_node` conditions to make branching decisions based on the extracted data. ### How It Works 1. **Extract data** using an `extraction_action` and specify `output_variable_names` to create a variable 2. **Reference the variable** in an `if_else_node` condition using the syntax `variable_name[0]` 3. **Branch logic** executes different paths based on the extracted value ### Variable Naming and Syntax * Variables are referenced using array index syntax: `variable_name[0]` * The `[0]` accesses the first (or only) element of the extracted value * Multiple extractions create arrays: `variable_name[0]`, `variable_name[1]`, etc. ## Condition Syntax Conditions are Python-like expressions that can reference parameters: ```json theme={null} { "condition": "is_login_screen[0]" } ``` ```json theme={null} { "condition": "has_2fa[0] == 'true'" } ``` ```json theme={null} { "condition": "verification_needed[0] is not None and verification_needed[0] != 'skip'" } ``` ### Operators | Operator | Example | | - | - | | Equality | `var[0] == 'value'` | | Inequality | `var[0] != 'value'` | | None check | `var[0] is not None` | | Boolean | `var[0] == 'true'` | | Logical AND | `a[0] == 'x' and b[0] == 'y'` | | Logical OR | `a[0] == 'x' or b[0] == 'y'` | ## Examples ### Extract and Check Page State ```json theme={null} [ { "type": "action_node", "extraction_action": { "llm": { "extraction_format": { "is_login_page": "bool" }, "extraction_instructions": "Check if the current page is a login page asking for username or password. If yes, return 'is_login_page' as true, otherwise return false", "output_variable_names": [ "is_login_page" ], "llm_model_name": "gemini-4-argon" } }, "before_sleep_time": 3, "end_sleep_time": 0 }, { "type": "if_else_node", "condition": "is_login_page[0]", "if_nodes": [ { "type": "action_node", "interaction_action": { "input_text": { "command": "get_by_label(\"Username\")", "input_text": "{username[0]}" } } } ], "else_nodes": [ { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Next\")" } } } ] } ] ``` ### Handle Optional 2FA ```json theme={null} { "type": "if_else_node", "condition": "requires_2fa[0] == 'true'", "if_nodes": [ { "type": "action_node", "interaction_action": { "input_text": { "command": "get_by_label(\"Verification Code\")", "input_text": "{auth_code[0]}" } } } ], "else_nodes": [] } ``` ### Branch Based on Extracted Value ```json theme={null} [ { "type": "action_node", "extraction_action": { "llm": { "extraction_format": { "status": "str" }, "extraction_instructions": "Extract the order status", "output_variable_names": ["status"] } } }, { "type": "if_else_node", "condition": "status[0] == 'pending'", "if_nodes": [ { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Approve\")" } } } ], "else_nodes": [] } ] ``` ### Nested Conditions ```json theme={null} { "type": "if_else_node", "condition": "user_type[0] == 'admin'", "if_nodes": [ { "type": "if_else_node", "condition": "access_level[0] == 'full'", "if_nodes": [ ... ], "else_nodes": [ ... ] } ], "else_nodes": [] } ``` > [!NOTE] > > `else_nodes` is optional. If omitted or empty, nothing happens when condition is false. # Local Setup Source: https://docs.optexity.com/docs/building-automations/local-setup Follow these steps to set up your Optexity account, grab your API key, and install the toolchain locally. ### 1. Create an Account Head to [dashboard.optexity.com](https://dashboard.optexity.com) and sign up for a free account. ### 2. Get Your API Key Once logged in, navigate to the **API Keys** section in your dashboard and create a new key. ### 3. Install the Recorder Extension Install the **Optexity Recorder** extension from the [Chrome Web Store](https://chromewebstore.google.com/detail/optexity-recorder/pbaganbicadeoacahamnbgohafchgakp). This extension captures your browser interactions and converts them into automation workflows. ### Prerequisites * Python 3.11+ * Git ## Create and Activate a Python Environment (Optional) Choose **one** of the options below. #### Option A – Conda (includes Python 3.11 and Node.js) ```bash theme={null} conda create -n optexity python=3.11 conda activate optexity ``` Install miniconda here: [https://docs.conda.io/projects/conda/en/stable/user-guide/install/index.html#installing-in-silent-mode](https://docs.conda.io/projects/conda/en/stable/user-guide/install/index.html#installing-in-silent-mode) #### Option B – Python `venv` ```bash theme={null} python3 -m venv .venv source .venv/bin/activate ``` ## Installation ### Quick Installation (from PyPI) Install Optexity directly from PyPI: ```bash theme={null} pip install optexity optexity install-browsers ``` **OR** ### Installation from Source If you want to clone and edit from source: ```bash theme={null} git clone git@github.com:Optexity/optexity.git cd optexity pip install -e . optexity install-browsers ``` ## Set required environment variables: ```bash theme={null} OPTEXITY_API_KEY=YOUR_OPTEXITY_API_KEY # API key used for authenticated requests GOOGLE_CLOUD_PROJECT=YOUR_GCP_PROJECT_ID # Google Cloud project used for every LLM call DEPLOYMENT=dev # or "prod" in production ``` Optexity runs Gemini and Claude on Google Vertex AI. In your Google Cloud project, enable the Vertex AI API and enable the Claude models in Model Garden, then sign in with Application Default Credentials: ```bash theme={null} gcloud auth application-default login ``` ### Using a Different LLM Optexity defaults to the `lite` tier (`gemini-3.5-flash-lite`, falling back to `claude-haiku-5-5`). Set `LLM_MODEL` to use another Vertex model: ```bash theme={null} LLM_MODEL=gemini-3.6-flash ``` See [Model Configuration](/docs/advanced/model-configuration) for fallbacks, per-task overrides, and the full list of settings. # Parameters Source: https://docs.optexity.com/docs/building-automations/parameters Passing data in, extracting data out, and using dynamic values in automations Parameters control data flow in Optexity automations. They enable you to run the same automation with different inputs and capture outputs for use in subsequent steps. ## Parameter Types | Type | Purpose | When Set | Example Use | | - | - | - | - | | `input_parameters` | Data you provide | Before execution | Username, search queries | | `generated_parameters` | Data extracted during execution | During automation | Order IDs, confirmation numbers | | `secure_parameters` | Sensitive data from secure storage | Retrieved at runtime | Passwords, API keys, TOTP codes | ## Defining Parameters All parameters must be declared in the automation's `parameters` object: ```json theme={null} { "parameters": { "input_parameters": { "username": ["example_username"], "search_queries": ["query1", "query2"] }, "secure_parameters": { "password": [{ "onepassword": { "vault_name": "vault", "item_name": "login", "field_name": "password" } }] }, "generated_parameters": { "order_ids": [] } } } ``` > [!WARNING] > > Values must always be lists: `["value"]` not `"value"`. ## Accessing Parameters Use `{variable_name[index]}` syntax to reference values: | Syntax | Use Case | Example | | - | - | - | | `{username[0]}` | First (or only) value | Single email address | | `{items[1]}` | Second value | Specific list item | | `{order_ids[index]}` | Current loop iteration | In `for_loop_node` | ### Where Substitution Works | Field | Example | | - | - | | `input_text` | `"{email[0]}"` | | `prompt_instructions` | `"Click order {order_id[0]}"` | | `command` | `get_by_text("{item[0]}")` | | `xpath` | `//td[text()='{id[0]}']` | | `select_values` | `["{country[0]}"]` | | `task` (agentic) | `"Search for {query[0]}"` | ## Input Parameters Values provided before execution. Declare placeholder values in the automation; actual values are passed in the inference request. **In automation:** ```json theme={null} "input_parameters": { "family_name": ["placeholder"], "first_name": ["placeholder"] } ``` **In inference request:** ```json theme={null} "input_parameters": { "family_name": ["Smith"], "first_name": ["John"] } ``` ## Generated Parameters Values extracted during execution. Initialize as empty lists; they're populated by extraction actions. ```json theme={null} "generated_parameters": { "order_ids": [], "confirmation": [] } ``` Populate using `output_variable_names` in extraction actions: ```json theme={null} { "extraction_action": { "llm": { "extraction_format": { "order_ids": "List[str]" }, "extraction_instructions": "Extract all order IDs from the table", "output_variable_names": ["order_ids"] } } } ``` After extraction, use `{order_ids[index]}` in subsequent actions or iterate with `for_loop_node`. > [!NOTE] > > Only `str` and `List[str]` types can be stored as variables. Other types can be extracted but won't be available for substitution. ## Secure Parameters Sensitive values retrieved from secure storage at runtime. Never hardcode passwords in automations. ### 1Password Integration ```json theme={null} "secure_parameters": { "password": [{ "onepassword": { "type": "raw", "vault_name": "my_vault", "item_name": "my_login", "field_name": "password" } }] } ``` ### TOTP Codes Generate 2FA codes from a TOTP secret: ```json theme={null} "secure_parameters": { "auth_code": [{ "totp": { "totp_secret": "BASE32SECRET" } }] } ``` Or retrieve TOTP secret from 1Password: ```json theme={null} "secure_parameters": { "auth_code": [{ "onepassword": { "type": "totp_secret", "vault_name": "vault", "item_name": "login", "field_name": "totp_secret", "digits": 6 } }] } ``` > [!TIP] > > See [1Password Integration](/docs/advanced/onepassword) and [TOTP Integration](/docs/advanced/totp-integration) for setup instructions. ## Loop Index Variable In `for_loop_node`, use `{variable[index]}` for the current iteration value: ```json theme={null} { "type": "for_loop_node", "variable_name": "product_ids", "nodes": [{ "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_text(\"{product_ids[index]}\")", "prompt_instructions": "Click product {product_ids[index]}" } } }] } ``` If `product_ids = ["PROD-1", "PROD-2", "PROD-3"]`: * Iteration 1: `{product_ids[index]}` → `"PROD-1"` * Iteration 2: `{product_ids[index]}` → `"PROD-2"` * Iteration 3: `{product_ids[index]}` → `"PROD-3"` A `for_loop_node` can also iterate a locator instead of a parameter, in which case `{locator[index]}` is the current match rather than a value. See [For Loop Node](/docs/building-automations/for-loop-node). ## Complete Example This automation logs in, extracts order IDs, and processes each order: ```json theme={null} { "url": "https://orders.example.com", "parameters": { "input_parameters": { "username": ["admin@example.com"] }, "secure_parameters": { "password": [{ "onepassword": { "type": "raw", "vault_name": "vault", "item_name": "orders", "field_name": "password" } }] }, "generated_parameters": { "order_ids": [] } }, "nodes": [ { "type": "action_node", "interaction_action": { "input_text": { "command": "get_by_label(\"Email\")", "input_text": "{username[0]}" } } }, { "type": "action_node", "interaction_action": { "input_text": { "command": "get_by_label(\"Password\")", "input_text": "{password[0]}" } } }, { "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_role(\"button\", name=\"Sign In\")" } } }, { "type": "action_node", "extraction_action": { "llm": { "extraction_format": { "order_ids": "List[str]" }, "extraction_instructions": "Extract all order IDs", "output_variable_names": ["order_ids"] } } }, { "type": "for_loop_node", "variable_name": "order_ids", "nodes": [{ "type": "action_node", "interaction_action": { "click_element": { "command": "get_by_text(\"{order_ids[index]}\")", "prompt_instructions": "Click order {order_ids[index]}" } } }] } ] } ``` ## Best Practices | Practice | Example | | - | - | | Use descriptive names | `patient_date_of_birth` not `dob` | | Use placeholder values in automation | `["placeholder@test.com"]` | | Pass real values in inference request | Actual user credentials | | Initialize generated params as empty | `"order_ids": []` | | Always use lists | `["value"]` not `"value"` | # Private Node Source: https://docs.optexity.com/docs/building-automations/private-node Closed-source node handlers that run only on Optexity's cloud infrastructure A `private_node` runs a closed-source handler built, maintained, and updated by Optexity — typically a portal-specific integration (e.g. an EHR or claims portal) that's too specialized or proprietary to ship in the public SDK. Inside an automation it behaves like any other node: it takes `inputs`, runs, and can bind its result to `output_variable_names` for later nodes to use. > [!WARNING] > > `private_node` handlers are closed-source. You cannot view their implementation, and a `private_node` will not run against a local install of the public `optexity` SDK — it only executes on Optexity's cloud infrastructure, where the private handler package is installed. ## Structure ```json theme={null} { "type": "private_node", "handler": "ecw.list_lab_orders", "inputs": { "patient_id": "{patient_id[0]}" }, "output_variable_names": ["lab_orders"] } ``` ## Properties | Property | Type | Default | Description | | - | - | - | - | | `handler` | `str` | — | Registered handler name, in `.` form | | `inputs` | `dict[str, Any]` | `{}` | Inputs passed to the handler; the shape is defined by the handler, not the public schema | | `output_variable_names` | `list[str] \| None` | `null` | Variable names the handler's return value is bound to, same as an extraction node | | `before_sleep_time` | `float` | `0.0` | Seconds to wait before running the handler | | `end_sleep_time` | `float` | `0.0` | Seconds to wait after running the handler | ## Examples Private handlers are grouped into portals — one integration per stateful session (login, cached tokens, base URL). A few handlers that exist today: | Handler | Purpose | | - | - | | `ecw.resolve_patient_id` | Resolve a patient's internal ID from search criteria | | `ecw.list_lab_orders` | List a patient's lab orders | | `ecw.download_lab_reports` | Download lab report files for a patient | | `ecw.list_di_orders` | List a patient's diagnostic imaging orders | | `ecw.list_patient_docs` | List documents on a patient's chart | | `ecw.list_progress_notes` | List a patient's progress notes | ```json theme={null} { "type": "private_node", "handler": "ecw.resolve_patient_id", "inputs": { "first_name": "{first_name[0]}", "last_name": "{last_name[0]}", "dob": "{dob[0]}" }, "output_variable_names": ["patient_id"] } ``` Like any other node, a `private_node` can be placed inside a [for-loop node](/docs/building-automations/for-loop-node) or [if-else node](/docs/building-automations/if-else-node), and its outputs can be referenced by later nodes using `{variable_name}` interpolation. > [!NOTE] > > Optexity builds, maintains, and updates all private node handlers. Available handlers depend on what's registered in your Optexity deployment — contact the Optexity team if you need a handler for a portal that isn't listed here, or if an existing handler needs updating (e.g. after a portal changes its UI). # FAQs Source: https://docs.optexity.com/docs/faqs/faqs Frequently asked questions about Optexity ## General > [accordion] What is Optexity? > > Optexity is a platform for building and running browser automations using AI-assisted element location and robust execution. ## Parameters & Credentials > [accordion] Do I need to build new automation for each user or login credentials? > > No. Use [Parameters](/docs/building-automations/parameters) to run the same automation with different values. Pass credentials at runtime via the inference request. > [accordion] How do I access parameter values? > > Use the `{parameter_name[index]}` syntax. For single values use `{username[0]}`. In loops, use `{variable[index]}` for the current iteration. See [Parameters](/docs/building-automations/parameters) for details. > [accordion] How do I handle passwords and secrets securely? > > Use `secure_parameters` instead of hardcoding credentials. Optexity retrieves values at runtime from secure storage like 1Password or generates TOTP codes. Your secrets are never exposed in the workflow. > > | Provider | Use Case | Documentation | > | - | - | - | > | 1Password | Passwords, API keys | [1Password Integration](/docs/advanced/onepassword) | > | TOTP | 2FA codes | [TOTP Integration](/docs/advanced/totp-integration) | > > ```json theme={null} > "secure_parameters": { > "password": [{ > "onepassword": { > "vault_name": "my_vault", > "item_name": "my_login", > "field_name": "password" > } > }] > } > ``` ## Control Flow > [accordion] How do I iterate over multiple items on a page? > > Use `for_loop_node`. Set `variable_name` to iterate over values from input parameters or extracted data, or set `locator` to iterate over every element a Playwright locator matches when you don't know the count in advance (e.g. every row of a results table). See [For Loop Node](/docs/building-automations/for-loop-node). > [accordion] How do I handle conditional logic? > > Use `if_else_node` to execute different actions based on conditions. See [If Else Node](/docs/building-automations/if-else-node). ## Two-Factor Authentication > [accordion] Can Optexity handle 2FA/MFA? > > Yes. Optexity integrates with authenticator apps (Google Authenticator, Microsoft Authenticator, Authy). See [TOTP Integration](/docs/advanced/totp-integration). > [accordion] Do you handle OTP over email? > > This is a work in progress. Contact us for details. ## Data & Downloads > [accordion] How do I access downloaded files? > > Use the `GET /api/v1/get_downloads` endpoint. Downloaded files are stored in cloud storage with signed URLs. See [Downloads & Files](/docs/advanced/downloads-files). > [accordion] Where is my extracted data stored? > > Extracted data is stored in Optexity cloud storage. Retrieve it via `GET /api/v1/get_output_data` or receive it in callbacks. > [accordion] Do I have to poll to get results? > > No. Set up a callback URL in the dashboard to receive data automatically when automation completes. See [Callbacks](/docs/advanced/callbacks). ## Form Interactions > [accordion] How do I fill a field and select from a dropdown that appears? > > Use two sequential actions: first `input_text` to type, then `click_element` to select. Add `before_sleep_time` to wait for the dropdown. > > ```json theme={null} > [ > { > "type": "action_node", > "interaction_action": { > "input_text": { > "command": "get_by_role(\"textbox\", name=\"Search\")", > "input_text": "{search_term[0]}" > } > }, > "end_sleep_time": 1 > }, > { > "type": "action_node", > "interaction_action": { > "click_element": { > "prompt_instructions": "Click on the dropdown option '{search_term[0]}'." > } > }, > "before_sleep_time": 2 > } > ] > ``` > [accordion] How do I select from a native dropdown? > > Use `select_option` for native `