Troubleshooting¶

Look up the symptom; each entry gives the cause and the fix. First thing, almost always:

rayito doctor --template rayito-base           # add --launch to try a real sandbox

Messages stay in Spanish

The SDK and CLI messages are written in Spanish, so the literal texts quoted on this page (sin región: …, denegado: …, actualiza la imagen) appear in Spanish. Each entry says what they mean.

`rayito doctor`¶

Each check ends in OK, WARN, FAIL or SKIP. A denied permission shows up as <operation> denegado: … ("denied"); with rayito --json doctor, details.operation names the call AWS rejected and details.action the permission that is missing. The first seven are read-only; checks 8 to 10 need a RUNNING sandbox of the image, or --launch to create a 300 s one that is killed at the end.

#	Check	Typical `FAIL` (or `WARN`)	Fix
1	`credentials`	no credentials, expired token, no region; `WARN` if the region does not have Lambda MicroVMs	`aws sso login --profile <your-profile>`; export `AWS_PROFILE` and `AWS_REGION` with one of the ten regions
2	`managed-images`	`AccessDenied`, unknown endpoint or a region without the service	update `boto3` (`pip install -U boto3`); change region
3	`quotas`	`WARN`: a MicroVM quota below the default value (new accounts); `SKIP` if the permission is missing	request the increase in Service Quotas
4	`iam-simulation`	`WARN` for `lambda:PassNetworkConnector`; `SKIP` without `iam:SimulatePrincipalPolicy`	it is a false positive of the simulator: ignore it. Other `implicitDeny`: review the policy
5	`bucket`	403/404 on the artifacts bucket, or a bucket in another region; `SKIP` without `--bucket` or `RAYITO_BUCKET`	grant `s3:ListBucket`; use a bucket in the same region
6	`image-gate`	the image does not exist or its version cannot be launched	`rayito image publish …`; if the build failed, look at its `stateReason`
7	`sandboxes`	more than 10 `RUNNING` sandboxes of that image (`WARN` from 1 to 10): they bill	`rayito sandbox list`; `rayito sandbox kill --all --template <image>`
8	`token`	`AccessDenied` when minting the proxy token; `SKIP` if there is no `RUNNING` sandbox	grant `lambda:CreateMicrovmAuthToken`; use `--launch`
9	`agent`	`rayd responde pero no está agent_ready` (rayd responds but is not agent_ready), or no response (`UNAVAILABLE`, proxy 403); `WARN` with `kernel_ready=false`, `hook_anomalies` greater than 0 or `imds_blocked=false` on a `-caps` image; `SKIP` without a token	wait a few seconds; republish the image; with `hook_anomalies`, check who mints `allPorts` tokens (Security)
10	`compatibility`	the image's `rayd` is older than the one the SDK requires (compatibility table)	publish the image from your SDK's release (Set up AWS)
—	`efs-network` (only with `--efs-vpc-id` and `--efs-subnet-ids`)	the VPC or the subnets are not suitable for `rayito stack deploy efs-volumes`	fix what it names; it creates nothing (EFS volumes in your VPC)

`rayito setup`¶

Every error comes out as rayito: <message> on stderr (with --json, also in the document: error: {"code", "message"} and exitCode). The messages are in Spanish, like the rest of the CLI; the table glosses each one. Nothing has been created after a --dry-run; the rest of the errors can be retried: rayito setup observes the account again and resumes whatever was left half done.

Code	Exit	What happens	What to do
`unsupported_region`	2	the region has no Lambda MicroVMs (the message lists the ten that do)	pick one of them with `--region` or `AWS_REGION`
`bucket_unavailable`	2	the bucket (`--bucket`, `RAYITO_BUCKET` or the IAM stack's) exists but is not accessible with these credentials (S3 answers 403: it usually belongs to another account)	pass another name with `--bucket` (if it does not exist, `setup` creates it) or, if it is yours, grant `s3:ListBucket` on it
`bucket_taken`	2	the default name `rayito-artifacts-<account>-<region>` is already used by a bucket that the `rayito-m0-artifacts` stack did not create: it may be one of yours that was kept when the stack was deleted, or another account's; `setup` never adopts such a bucket	if it is yours, pass it with `--bucket <name>` (it reuses it); if not, pick another name with `--bucket` and `setup` will create it
`artifact_stack_conflict`	2	the `rayito-m0-artifacts` stack already manages another bucket and you asked for a new one with `--bucket` or `RAYITO_BUCKET`	pass `--bucket <the stack's bucket>`, or delete the stack first (`aws cloudformation delete-stack --stack-name rayito-m0-artifacts`; the bucket it managed is kept)
`bucket_region`	2	the bucket is in another region and the build would fail with `S3_CROSS_REGION_ACCESS_DENIED`	use a bucket in the same region as the account and `setup` region
`artifact_missing`	2	the `--artifact` file does not exist	fix the path
`artifact_variant`	2	`--artifact` is not the `full` variant zip without EFS	use the release's `rayito-image.zip`; the other variants are published with `rayito image publish`
`needs_confirmation`	2	there is something to create and there is no terminal (or you use `--json`)	confirm with `--yes`, or run it from a terminal
`stack_newer`	2	a Rayito newer than yours deployed the `rayito-m0-iam` stack and the plan changes its bucket: updating it would go back to your older template	run `rayito setup` with that version or a later one (for example, upgrade the SDK in CI), or pass the bucket the stack already has
`stack_blocked`	1	the `rayito-m0-iam` or `rayito-m0-artifacts` stack (the message names it) is in a state that cannot be reused or waited for (`ROLLBACK_COMPLETE`, `*_FAILED`, `DELETE_IN_PROGRESS`)	`aws cloudformation delete-stack --stack-name <the stack>`, wait for it to disappear and run `rayito setup` again; if it is the bucket's and it failed on creation, first look at `aws cloudformation describe-stack-events` (usually an S3 permission or an SCP)
`stack_incomplete`	1	the stack has no output Rayito reads: `BuildRoleArn` in `rayito-m0-iam` (`infra/iam.yaml` template) or `BucketName` in `rayito-m0-artifacts` (`infra/artifact-bucket.yaml` template)	deploy Rayito's template (or delete the stack and let `rayito setup` create it)
`release_download`	1	`rayito-image.zip` or `SHA256SUMS` could not be downloaded from the release (`HTTP <status>`, `tiempo agotado` (timeout) or `error de red` (network error))	check the network or proxy (`HTTPS_PROXY`); or download the zip, verify it and pass `--artifact`
`checksum_missing`	1	the release's `SHA256SUMS` does not list `rayito-image.zip`; nothing is published	use the release of your SDK's version; or `--artifact` with a verified zip
`checksum_mismatch`	1	the zip's sha256 does not match `SHA256SUMS`; nothing is published	download again; if it persists, do not publish that file and verify it
`image_build_failed`	1	the `rayito-base` image was not launchable after the build	run `rayito setup` again (it will ask to confirm a new `publish`); if it repeats, `rayito doctor` and `rayito image list` show the state

Other failures: a rayito doctor FAIL at the end leaves the exit at 1 with la cuenta no está lista (the account is not ready) (fix it with rayito doctor and repeat); a "no" at the confirmation exits with 1 without creating anything; no region, no profile or no credentials exit with 2 like the rest of the CLI; an untranslated AWS error (for example a permission denied when creating the stack) exits with 1 and its code. rayito setup needs administrator or bootstrap credentials once (CloudFormation, IAM and, if it creates the bucket, S3: s3:CreateBucket, s3:PutEncryptionConfiguration, s3:PutBucketPublicAccessBlock, s3:PutBucketOwnershipControls, s3:PutLifecycleConfiguration, s3:PutBucketTagging, s3:GetBucketPolicy and s3:PutBucketPolicy); the CallerPolicy policy is not enough. The bucket it creates is kept when its stack is deleted: to remove it, empty and delete it by hand. When a later command (Sandbox.create(), rayito agent deploy) says there is no image or bucket, the message ends with the hint "si la cuenta o la región aún no están preparadas, ejecuta rayito setup" (if the account or region are not prepared yet, run rayito setup).

If setup fails with AccessDenied or an explicit deny on s3:CreateBucket or cloudformation:CreateStack (or stack_blocked with that reason in describe-stack-events) because an SCP in your organization requires tags, run it again with the ones it asks for, one --tag for each (the values here are examples):

rayito setup --tag Owner=platform --tag Project=agents

Credentials and region¶

ExpiredTokenException, Token has expired, UnauthorizedSSOTokenError. The SSO session expired. Sign in again with aws sso login --profile <your-profile>. S3 URLs signed with temporary credentials stop working when those credentials expire, even if the URL says otherwise.

sin región: pasa --region o exporta AWS_REGION ("no region: pass --region or export AWS_REGION"; CLI, exit code 2). The SDK and the CLI need a region: export AWS_REGION=us-east-1 (both SDKs and the CLI read AWS_REGION; if it is missing, they use AWS_DEFAULT_REGION and then the profile's region).

AuthenticationException / AuthenticationError when connecting. The access token is not the sandbox's (you lost it or it belongs to another). It cannot be recovered: create a new sandbox and store sbx.access_token next to the id.

The image¶

UnimplementedError: … actualiza la imagen (or "publica una imagen nueva"), meaning "update the image" / "publish a new image". The image's rayd is older than the feature you ask for (server-side deadline, S3 URLs, metrics history, Deno kernels...). Publish the image from your SDK's release and check with rayito doctor (compatibility check).

UnimplementedError naming rayito-base-poly or rayito-base-caps. The feature needs another image variant: bash, JavaScript and TypeScript live in rayito-base-poly; the network policy, in rayito-base-caps (Images).

The image build fails. rayito image publish prints the stateReasons and the last lines of the build log (group /rayito/<image> in CloudWatch). Usual causes: the build role cannot read the bucket, or the zip is of another variant (--variant).

Building rayd on macOS or Windows. You need Linux or WSL2 (on macOS, a Linux VM). To get started you do not need it: publish the release's signed rayito-image.zip (Set up AWS).

Quotas and pacing¶

RateLimitException / RateLimitError with ThrottlingException. The Lambda MicroVMs APIs have per-account, per-region quotas: 5 run-microvm per second, 2 suspend, 5 resume, 10 terminate, 50 proxy tokens and 100 get-microvm. The SDK already rate-limits per process; several processes at once can exceed them. Space out the launches or use a pool.

QuotaExceededException / QuotaExceededError. You have reached the region's memory quota (for example 1,024 GB in us-east-1, counting RUNNING and SUSPENDED sandboxes). Kill sandboxes or request an increase.

RateLimitException inside the sandbox. More than 256 live processes and PTYs, more than 8 code contexts or more than 16 active transfers. Release the ones you do not use.

Files and S3¶

A missing key gives 403 instead of 404. Your credentials lack s3:ListBucket on the bucket (scoped to the prefix). Without it, S3 cannot say "does not exist" and the import waits until it expires. Add it (IAM).

UnimplementedError on upload_url / download_url. There is no transfer bucket: pass transfer=S3Staging("amzn-s3-demo-bucket") or export RAYITO_TRANSFER_BUCKET.

The E2B shim does not accept transfer=. That is on purpose (E2B does not have it): in rayito.e2b the bucket comes from RAYITO_TRANSFER_BUCKET (and RAYITO_TRANSFER_PREFIX, RAYITO_TRANSFER_REGION).

FileUploadException with wrong_region. The bucket is in a different region than the sandbox: use one in the same region or S3Staging(region=).

Execution¶

A cell or a silent command stops halfway or takes too long. Auto-suspend counts the endpoint's traffic: something that prints nothing for max_idle_seconds (300 s by default) can be suspended while it runs. The SDK reattaches on resume, but raise max_idle_seconds or pass idle=None for long, silent tasks (Pause and resume). In the MCP server: RAYITO_MCP_IDLE_SECONDS=0.

SandboxException: output_truncated. A command produced output and nobody read it within 30 s: the agent retains 64 chunks of 32 KiB and then cuts it off. Read the handle (by iterating it or with wait()) or redirect the output to a file.

TimeoutException on a command. The default timeout is 60 s. For long processes, timeout=None (TypeScript: timeoutMs: 0) or background=True.

execution.error.name == "ExecutionTimeout". The cell exceeded its timeout (300 s by default): raise it.

<defunct> processes in ps after launching a daemon. With rayd 0.6.0 or earlier, a process that daemonizes (double fork, nohup … & inside a subshell, mount-s3 without --foreground) was left as a zombie hanging off rayd, which is the sandbox's PID 1. rayd 0.6.1 reaps them as soon as they finish without touching your commands' exit code (What's new in 0.6.1); with an older image, republish it on rayd 0.6.1. Zombies use no CPU or memory, only an entry in the process table.

Network¶

UnimplementedError when passing network= or allow_internet_access=False. The network policy is only enforced on rayito-base-caps; on another image the SDK terminates the sandbox before returning it, so you are not handed one with open network.

A client fails with the policy active even though the host is allowed. Hostname rules go through a local proxy; clients that do not honor HTTPS_PROXY (raw sockets, ssh, git://) fail closed (Outbound network).

403 or 502 on get_host(port). The proxy headers are missing (host.headers) or nothing is listening on that port (Ports and host).

TypeScript¶

InvalidArgumentError: <función> necesita el paquete opcional '<paquete>': instala npm install <paquete> (the function needs the optional package; install it). The AWS clients for the optional features (secrets and gateway, index, optional stacks, events, EFS volumes, custom domain, OpenTelemetry...) are optional peerDependencies of the package: they are only loaded when you use that feature. Install the one the message names (for example, npm i @aws-sdk/client-secrets-manager for secrets= or gateways=).

await using does not compile. You need TypeScript ≥ 5.2 with "lib": ["ES2022", "ESNext.Disposable"]. Without await using, use try { … } finally { await sbx.kill(); }.

Agent¶

Failures of sbx.agent.run()/.stream() are AgentException (TypeScript: AgentError) with a closed reason; the full table is in Errors.

reason="runtime_missing". The image does not ship the runtime (opencode is not on the PATH). Create the sandbox from an image built with AgentTemplate (Agent templates).

reason="busy". A run/stream is already in progress in that sandbox: one at a time per sandbox. Wait for it to finish or use another sandbox.

reason="model_error". The model responded with an error. Check that the gateway secret holds the full credential (for Bedrock, Bearer <key>, which expires: rotate it with sbx.gateways.refresh()) and that the model is in the preset's models= (Agent in the sandbox).

Agent deployed to AgentCore or with CDK. Failures of rayito agent deploy --target agentcore and of AgentCoreFleet are AgentCoreException (TypeScript: AgentCoreError) with a closed code (Errors and Deploy to AgentCore); those of the RayitoAgent and RayitoMicrovmImage constructs, at synth time or in cdk deploy, are in the CDK troubleshooting.

I do not see the HTTP Request: ... lines from httpx in the agent logs. This is on purpose: the agent host raises the httpx and httpcore loggers to WARNING because at INFO they log the full URL, and a URL can carry a credential in its query. If you need them to debug, lower them from the agent code (Security).

400 to a malformed request. A request line that is not HTTP (a port scanner, a client speaking another protocol) gets a 400 and the host writes a single - -> 400 line to its log; later invocations are not affected. Before 0.17 that line was followed by an AttributeError with its traceback, which was noise and not a failure.

Secrets¶

An AWS SDK v3 logger prints the secrets

Rayito never writes a secret's value or name to its logs. In Python, since 0.17.1 import rayito drops the DEBUG records of the botocore, boto3 and s3transfer loggers that carried AWS parameters and bodies (the list): the rest of the DEBUG output still comes through, without the SecretString. In TypeScript, a logger on the AWS SDK v3 client does include those bodies with the SecretString in the clear: do not set one with secrets.

SecretNotFoundException with a secret created from the shim. The shim lowercases names (Secret.create("OpenAI", …) stores rayito/openai); the native SDK does not. Refer to it in lowercase.

SecretException with InvalidRequestException when recreating a name. AWS takes ≈ 20–30 s to release a deleted name. create already retries for up to 60 s; if that runs out, wait and try again.

Unexpected bills¶

Sandboxes nobody killed. rayito sandbox list shows them with their age; rayito sandbox kill --all terminates them. To keep it from happening again, always use with / await using, a tight timeout and the server-side deadline with on_timeout="kill", which is enforced even if your process dies.

Accumulated image versions. Each one costs ≈ $0.04 per week (Pricing): rayito image prune --dry-run and then rayito image prune --keep 5.

Forgotten AgentCore agents. With no sessions they charge no compute, but they keep charging for the image in ECR and the logs (Pricing). rayito agent list --target agentcore shows them and rayito agent delete <name> --target agentcore removes the runtime, its stack and its log group.

Forgotten CDK stacks. An agent or an image from the constructs is deleted with cdk destroy, not with the CLI. The bootstrap assets (the agent's code, the image zip and, with localDocker(), the agent's images) stay in the bootstrap bucket and repository until cdk gc (Deleting).