Intelligent Compute
Seqera Intelligent Compute is in preview. Seqera must enable it on a per-workspace basis before you can use it. Contact your account manager to request access for one or more workspaces.
Intelligent Compute may assign different CPU and memory values to tasks than your pipeline's process directives specify. The scheduler picks the most cost-effective instance shape that meets each task's resource request.
Intelligent Compute is a scheduling service that runs Nextflow pipelines on Seqera-managed EC2 instances in your AWS account. It allocates compute resources based on what each task needs rather than what the pipeline requests. This reduces cost and improves utilization across a run. Intelligent Compute is available only on AWS Cloud compute environments.
The standard AWS Cloud compute environment runs each pipeline on a single EC2 instance with a local executor. Intelligent Compute runs pipelines on a cluster of EC2 instances that scales beyond a single instance.
When you enable Intelligent Compute on an AWS Cloud compute environment, Seqera provisions and manages the following resources in your AWS account on first use:
- EC2 instances that run your pipeline tasks, launched as Spot or On-Demand instances according to the provisioning model
- A security group per VPC named
seqera-sched-vm, unless the compute environment specifies its own security groups - An IAM role and instance profile per compute environment and bucket configuration, attached to the EC2 instances
- An S3 gateway VPC endpoint, when the selected subnets are private and the VPC has none
- CloudWatch log groups under
/seqera(for example,/seqera/platform)
All managed resources use the seqera-sched- prefix. Seqera creates them on first use and removes them automatically when no longer needed.
Compute environments created before Intelligent Compute moved to EC2 instances run on Seqera-managed Amazon ECS clusters and keep doing so. New compute environments always use EC2 instances.
IAM permissions
Intelligent Compute requires two IAM policies attached to the same IAM user or role that Seqera uses to access your AWS account:
- AWS Cloud policy — required for all AWS Cloud compute environments. If you have already set up an AWS Cloud compute environment, this policy is already in place.
- Intelligent Compute policy — additional permissions required specifically for Intelligent Compute.
AWS Cloud policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AwsCloudCreate",
"Effect": "Allow",
"Action": [
"iam:CreateRole",
"iam:AddRoleToInstanceProfile",
"iam:CreateInstanceProfile",
"iam:AttachRolePolicy",
"iam:PutRolePolicy",
"iam:TagRole",
"iam:TagInstanceProfile"
],
"Resource": [
"arn:aws:iam::*:role/TowerForge-*",
"arn:aws:iam::*:instance-profile/TowerForge-*"
]
},
{
"Sid": "AwsCloudCreatePassRole",
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": "arn:aws:iam::*:role/TowerForge-*"
},
{
"Sid": "AwsCloudLaunchEC2",
"Effect": "Allow",
"Action": [
"ec2:CreateTags",
"ec2:DeleteTags",
"ec2:DescribeInstances",
"ec2:RunInstances",
"ec2:TerminateInstances"
],
"Resource": "*"
},
{
"Sid": "AwsCloudLaunchLogs",
"Effect": "Allow",
"Action": [
"logs:GetLogEvents"
],
"Resource": "arn:aws:logs:*:*:log-group:*:log-stream:*"
},
{
"Sid": "AwsCloudLaunchS3",
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "*"
},
{
"Sid": "AwsCloudDelete",
"Effect": "Allow",
"Action": [
"iam:GetRole",
"iam:ListAttachedRolePolicies",
"iam:ListRolePolicies",
"iam:DeleteRole",
"iam:DeleteInstanceProfile",
"iam:RemoveRoleFromInstanceProfile",
"iam:DetachRolePolicy",
"iam:DeleteRolePolicy"
],
"Resource": [
"arn:aws:iam::*:role/TowerForge-*",
"arn:aws:iam::*:instance-profile/TowerForge-*"
]
},
{
"Sid": "AwsCloudRead",
"Effect": "Allow",
"Action": [
"ec2:DescribeInstanceTypes",
"ec2:DescribeKeyPairs",
"ec2:DescribeVpcs",
"ec2:DescribeImages",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"s3:ListAllMyBuckets"
],
"Resource": "*"
},
{
"Sid": "AwsCloudUserdataCheck",
"Effect": "Allow",
"Action": [
"ec2:GetConsoleOutput"
],
"Resource": "*"
},
{
"Sid": "OptionalLineageIntegrationSNSAndS3",
"Effect": "Allow",
"Action": [
"sns:CreateTopic",
"sns:SetTopicAttributes",
"sns:Subscribe",
"sns:ConfirmSubscription",
"sns:Unsubscribe",
"sns:DeleteTopic",
"s3:CreateBucket",
"s3:GetBucketNotification",
"s3:PutBucketNotification",
"s3:GetBucketLocation",
"s3:ListBucket",
"s3:GetObject"
],
"Resource": [
"arn:aws:sns:*:*:seqera-lineage-*",
"arn:aws:s3:::seqera-lineage-*",
"arn:aws:s3:::seqera-lineage-*/*"
]
}
]
}
Download aws-cloud-full-policy.json
Intelligent Compute policy
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "ECSScopedOperations",
"Effect": "Allow",
"Action": [
"ecs:CreateCluster",
"ecs:DeleteCluster",
"ecs:DescribeClusters",
"ecs:PutClusterCapacityProviders",
"ecs:CreateCapacityProvider",
"ecs:DeleteCapacityProvider",
"ecs:DescribeCapacityProviders",
"ecs:RunTask",
"ecs:StopTask",
"ecs:DescribeTasks",
"ecs:DescribeContainerInstances",
"ecs:UpdateContainerInstancesState",
"ecs:TagResource"
],
"Resource": "arn:aws:ecs:*:*:*/seqera-sched-*"
},
{
"Sid": "ECSUnscopedOperations",
"Effect": "Allow",
"Action": [
"ecs:RegisterTaskDefinition",
"ecs:DeregisterTaskDefinition",
"ecs:DescribeTaskDefinition",
"ecs:ListTaskDefinitions",
"ecs:ListTaskDefinitionFamilies",
"ecs:ListTasks"
],
"Resource": "*"
},
{
"Sid": "IAMRoleManagement",
"Effect": "Allow",
"Action": [
"iam:CreateRole",
"iam:GetRole",
"iam:DeleteRole",
"iam:PutRolePolicy",
"iam:DeleteRolePolicy",
"iam:ListRolePolicies",
"iam:AttachRolePolicy",
"iam:DetachRolePolicy",
"iam:ListAttachedRolePolicies",
"iam:CreateInstanceProfile",
"iam:GetInstanceProfile",
"iam:AddRoleToInstanceProfile",
"iam:ListInstanceProfilesForRole",
"iam:RemoveRoleFromInstanceProfile",
"iam:DeleteInstanceProfile"
],
"Resource": [
"arn:aws:iam::*:role/seqera-sched-*",
"arn:aws:iam::*:instance-profile/seqera-sched-*"
]
},
{
"Sid": "PassRoleToECS",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": [
"arn:aws:iam::*:role/seqera-sched-*",
"arn:aws:iam::*:role/TowerForge-*"
],
"Condition": {
"StringEquals": {
"iam:PassedToService": [
"ecs-tasks.amazonaws.com",
"ecs.amazonaws.com",
"ec2.amazonaws.com"
]
}
}
},
{
"Sid": "ServiceLinkedRoles",
"Effect": "Allow",
"Action": "iam:CreateServiceLinkedRole",
"Resource": "arn:aws:iam::*:role/aws-service-role/*",
"Condition": {
"StringEquals": {
"iam:AWSServiceName": [
"ecs.amazonaws.com",
"ecs-compute.amazonaws.com",
"autoscaling.amazonaws.com",
"spot.amazonaws.com"
]
}
}
},
{
"Sid": "CloudWatchLogs",
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:DeleteLogGroup",
"logs:PutRetentionPolicy",
"logs:DescribeLogStreams",
"logs:GetLogEvents",
"logs:TagResource"
],
"Resource": "arn:aws:logs:*:*:log-group:/seqera/*"
},
{
"Sid": "EC2NetworkDiscovery",
"Effect": "Allow",
"Action": [
"ec2:DescribeImages",
"ec2:DescribeVpcs",
"ec2:DescribeSubnets",
"ec2:DescribeSecurityGroups",
"ec2:DescribeRouteTables",
"ec2:DescribeVpcEndpoints",
"ec2:DescribeInstances",
"ec2:CreateSecurityGroup",
"ec2:CreateVpcEndpoint",
"ec2:AuthorizeSecurityGroupEgress",
"ec2:CreateTags"
],
"Resource": "*"
},
{
"Sid": "ECRAccess",
"Effect": "Allow",
"Action": [
"ecr:GetAuthorizationToken",
"ecr:BatchCheckLayerAvailability",
"ecr:GetDownloadUrlForLayer",
"ecr:BatchGetImage"
],
"Resource": "*"
},
{
"Sid": "S3Access",
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:ListAllMyBuckets"
],
"Resource": "*"
},
{
"Sid": "VMEC2Operations",
"Effect": "Allow",
"Action": [
"ec2:DescribeInstanceTypes",
"ec2:RunInstances",
"ec2:TerminateInstances",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:RevokeSecurityGroupEgress",
"ec2:DeleteSecurityGroup"
],
"Resource": "*"
},
{
"Sid": "ASGEC2Operations",
"Effect": "Allow",
"Action": [
"ec2:CreateLaunchTemplate",
"ec2:DeleteLaunchTemplate"
],
"Resource": "*"
},
{
"Sid": "ASGManagement",
"Effect": "Allow",
"Action": [
"autoscaling:CreateAutoScalingGroup",
"autoscaling:UpdateAutoScalingGroup",
"autoscaling:DeleteAutoScalingGroup",
"autoscaling:CreateOrUpdateTags"
],
"Resource": "arn:aws:autoscaling:*:*:*/seqera-sched-*"
},
{
"Sid": "ASGDescribe",
"Effect": "Allow",
"Action": "autoscaling:DescribeAutoScalingGroups",
"Resource": "*"
},
{
"Sid": "SSMECSOptimizedAmi",
"Effect": "Allow",
"Action": "ssm:GetParameter",
"Resource": "arn:aws:ssm:*:*:parameter/aws/service/ecs/optimized-ami/*"
},
{
"Sid": "CostExplorer",
"Effect": "Allow",
"Action": "ce:GetCostAndUsage",
"Resource": "*"
}
]
}
Download aws-cloud-intelligent-compute-policy.json
Permission groups
| Group | Purpose |
|---|---|
ECSScopedOperations | Create, delete, describe, and tag ECS clusters, capacity providers, and tasks, and drain a faulty container instance so it stops receiving tasks. Scoped to seqera-sched-* resources. Required only for ECS-backed compute environments. |
ECSUnscopedOperations | Register, deregister, list, and describe ECS task definitions. ECS task definition APIs do not support resource-level permissions. Required only for ECS-backed compute environments. |
IAMRoleManagement | Create, update, and delete IAM roles and instance profiles scoped to seqera-sched-*. Seqera creates the EC2 instance role and instance profile on first use. ECS-backed compute environments also use an execution role, an infrastructure role, and per-cluster instance and task roles. |
PassRoleToECS | Pass seqera-sched-* and TowerForge-* roles to EC2, ECS, and ECS tasks. Required to attach the instance profile to EC2 instances, and roles to ECS infrastructure and task definitions. |
ServiceLinkedRoles | Create service-linked roles for Spot, ECS, and autoscaling. AWS creates the Spot role on the first Spot instance launch. Required only if these roles do not already exist in your account. |
CloudWatchLogs | Create and manage log groups under /seqera (for example, /seqera/platform), and read log events. Task stdout and stderr are written to CloudWatch. |
EC2NetworkDiscovery | Describe VPCs, subnets, security groups, route tables, and instances. Create security groups, VPC endpoints, and tags. Used for VPC auto-discovery, network setup, and tracking the instances of a cluster. |
ECRAccess | Authorize ECR and pull container images. ECS tasks pull images from ECR. Required only for ECS-backed compute environments. |
S3Access | Read objects and list buckets. Used to read Fusion trace files and pipeline work directory content. |
VMEC2Operations | Describe instance types, launch and terminate EC2 instances, and manage the rules of the seqera-sched-vm security group. Required for all compute environments. |
ASGEC2Operations | Create or delete EC2 launch templates. Required only for Auto Scaling Group-backed ECS clusters. |
ASGManagement | Create, update, and delete Auto Scaling Groups scoped to seqera-sched-*. Required only for Auto Scaling Group-backed ECS clusters. |
ASGDescribe | Describe Auto Scaling Groups. Required only for Auto Scaling Group-backed ECS clusters. |
SSMECSOptimizedAmi | Read the ECS-optimized AMI ID from SSM Parameter Store. Used to look up the latest Amazon Linux 2023 ECS-optimized AMI. Required only for ECS-backed compute environments. |
CostExplorer | Query ce:GetCostAndUsage. Used to display cost post pipeline launch, after receiving data from AWS Cost Explorer with a 24-48 delay. If this permission is absent, cost predictions do not appear. |
Conditional statements:
ECSScopedOperations,ECSUnscopedOperations,ECRAccess, andSSMECSOptimizedAmiare required only while you have compute environments that run on Amazon ECS. New compute environments run on EC2 instances and don't use them.ASGEC2Operations,ASGManagement, andASGDescribeare required only if Auto Scaling Group-backed ECS clusters are enabled. You can omit them for Managed Instances deployments.ServiceLinkedRolesis required only if the listed service-linked roles do not already exist in your AWS account.CostExploreris required only if you want cost predictions at pipeline launch.
Create and attach the IAM policies
Both policies must be attached to the IAM user or role that Seqera uses to access your AWS account before you create the compute environment. Create each policy as follows:
- Open the AWS IAM console.
- Select Policies under Access management, then select Create policy.
- Select the JSON tab, paste the policy JSON, then select Next.
- Enter a name (for example,
SeqeraAwsCloudPolicy), then select Create policy. - Repeat steps 2–4 for the Intelligent Compute policy (for example,
SeqeraIntelligentComputePolicy). - Attach both policies to the IAM user or role that Seqera uses to access your AWS account.
Set up an AWS Cloud compute environment with Intelligent Compute
You need the following:
- Intelligent Compute enabled for your workspace by Seqera. Contact your account manager to request access.
- AWS credentials with both the standard AWS Cloud permissions and the Intelligent Compute permissions attached.
- In your Seqera workspace, select Compute Environments, then select Add compute environment.
- Enter a name and select AWS Cloud as the platform.
- Select your AWS credentials.
- Select the Region where Seqera provisions compute resources.
- Enter a Work directory (S3 URI, for example
s3://my-bucket/work). - Under Compute Mode, enable the Seqera Intelligent Compute toggle.
- Configure the Intelligent Compute options as needed.
- Select Add.
Seqera validates credentials and configuration on save. On first use, it provisions the required IAM role, instance profile, and EC2 instances in your account. Instances and associated resources are removed automatically when no longer needed.
S3 bucket access
Intelligent Compute mints a dedicated IAM role per cluster in your AWS account and attaches it to the ECS tasks and EC2 instances that run your pipeline. The role's S3 permissions are derived from the compute environment's Allow buckets list:
- Empty list (default): the role grants S3 access account-wide (
"Resource": "*"). Every bucket in the account is reachable. - Populated list: the role grants access to the listed buckets only, plus the compute environment work directory and the work directory of the run being launched, which are added automatically.
Declaring buckets is opt-in least privilege: a compute environment that lists no buckets keeps account-wide access.
On each in-scope bucket the role grants:
| Actions | Resource |
|---|---|
s3:GetObject, s3:PutObject, s3:DeleteObject, s3:GetObjectTagging, s3:PutObjectTagging | arn:aws:s3:::<BUCKET_NAME>/* |
s3:ListBucket, s3:GetBucketLocation | arn:aws:s3:::<BUCKET_NAME> |
A populated Allow buckets list is the complete allow-list. A pipeline that reads or writes a bucket missing from the list fails with AccessDenied at task runtime — not at compute environment validation or pipeline launch. List every bucket your pipelines touch: reference or public data (for example, s3://ngi-igenomes), inputs staged from elsewhere, and the outdir. The compute environment and run work directories are added for you.
How entries are interpreted:
- An entry can be an
s3://URI or a bare bucket name. Entries for other storage providers (gs://,az://) and local paths are ignored. - A key prefix is discarded.
s3://shared-bucket/team-agrants access to all ofshared-bucket, not just theteam-aprefix. - An entry that is not a valid S3 bucket name is dropped.
Access buckets in another AWS account
Cross-account S3 access requires a grant on both sides: an identity policy in the compute account and a bucket policy in the account that owns the bucket. Adding the bucket to Allow buckets covers the first half only — the bucket owner must grant access to your compute account for the second.
For example, to read a reference dataset in s3://reference-data-shared, owned by account 444455556666, from a compute environment running in account 111122223333:
-
Add
s3://reference-data-sharedto Allow buckets on the compute environment. -
Ask the owner of account
444455556666to attach this bucket policy toreference-data-shared:{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SeqeraComputeAccount",
"Effect": "Allow",
"Principal": { "AWS": "arn:aws:iam::111122223333:root" },
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:GetObjectTagging",
"s3:PutObjectTagging",
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": [
"arn:aws:s3:::reference-data-shared",
"arn:aws:s3:::reference-data-shared/*"
]
}
]
}For read-only data, drop
s3:PutObject,s3:DeleteObject, ands3:PutObjectTagging.
The policy above delegates the decision to your account's IAM. If the bucket owner would rather name the specific principals, use "Principal": "*" with a condition on the role names Intelligent Compute creates:
"Condition": {
"ArnLike": {
"aws:PrincipalArn": [
"arn:aws:iam::111122223333:role/seqera-sched-task-role-clu-*",
"arn:aws:iam::111122223333:role/seqera-sched-instance-v2-clu-*",
"arn:aws:iam::111122223333:role/seqera-sched-vm-instance-*"
]
}
}
Each cluster gets its own role and the cluster identifier varies per cluster, but the prefixes are stable:
| Role | Name pattern | Used by |
|---|---|---|
| ECS task role | seqera-sched-task-role-clu-* | Task containers and Fusion on ECS-backed compute environments |
| ECS instance role | seqera-sched-instance-v2-clu-* | EC2 hosts of ECS-backed compute environments |
| EC2 instance role | seqera-sched-vm-instance-* | EC2 instances of new compute environments |
Limitations
- Buckets encrypted with a customer-managed KMS key (SSE-KMS) are not supported. The minted role carries no
kmspermissions for S3, and no bucket policy can supply them — KMS access must be granted on the identity side. Unencrypted buckets and buckets using SSE-S3 (AES256) work. The KMS for S3 policy documented for a custom instance profile does not apply here: Intelligent Compute always uses the role it mints and ignores a custom Instance Profile ARN. - Interrupted multipart uploads are not cleaned up. Uploads succeed, but the role cannot call
s3:AbortMultipartUpload, so a failed or retried upload can leave incomplete parts that still incur storage cost. Add an S3 lifecycle rule to the bucket to expire incomplete multipart uploads. - Specific object versions are not readable. The role does not grant
s3:GetObjectVersion, so on a versioned bucket only the current version of an object is accessible. - Buckets with ACLs enabled may reject cross-account writes. The role does not grant
s3:PutObjectAcl, so a bucket that requiresbucket-owner-full-controlon writes from another account cannot be written to. Set the bucket's object ownership to bucket-owner-enforced, which disables ACLs and is the default for buckets created after April 2023. - Access is granted per bucket, not per prefix. Key prefixes in Allow buckets entries are ignored, so a compute environment cannot be restricted to part of a bucket.
Resource metrics
The Metrics tab for a run on Intelligent Compute shows three resource values for CPU and memory: Requested, Allocated, and Used.
| Metric | Source | What it represents |
|---|---|---|
| Requested | Pipeline process directives | The CPU and memory your pipeline asked for, as written in your process directives (for example, cpus = 4, memory = 8 GB). |
| Allocated | Scheduler decision | The CPU and memory the scheduler assigned to the task container. May differ from Requested when the scheduler picks a more cost-effective instance shape that still meets the task's requirements. |
| Used | Nextflow trace data | The CPU and memory the task consumed, measured from the Nextflow trace metrics (pcpu × realtime for CPU, peakRss for memory). Absent for tasks that did not produce trace data. |
How to read the numbers:
- If Requested is much higher than Allocated, the scheduler found a more efficient instance shape than your directives implied.
- If Allocated is much higher than Used, the task ran with idle headroom.
- If Used is close to Allocated, resource utilization is near-optimal for that task.
- If Allocated matches Requested, confirm whether Seqera Intelligent Compute has been set up with a prediction model.
Configuration options
Configure the following options when you enable Intelligent Compute on a compute environment.
| Option | Values | Default | Description |
|---|---|---|---|
| Seqera Intelligent Compute | Enabled / Disabled | Disabled | Enables the Intelligent Compute scheduler for this compute environment. This option only appears if Intelligent Compute is enabled for your workspace. |
| Provisioning model | spotFirst, spot, ondemand | spotFirst | Instance procurement strategy. spotFirst uses Spot instances and falls back to On-Demand if Spot capacity is unavailable. spot uses Spot instances only. ondemand uses On-Demand instances only. |
| Max spot attempts | Integer, 1–10 | 3 | Total number of Spot provisioning attempts for a task, including the first, before falling back to On-Demand with spotFirst. Only visible when Provisioning model is spot or spotFirst. Set to 1 for a single attempt with no retry. Compute environments created before this setting was introduced are not backfilled and keep their current behavior. |
| Instance types | Comma-separated EC2 instance type identifiers (for example, m5.xlarge, c5.2xlarge) | Empty | Restricts which instance types the scheduler can select. When empty, the scheduler picks the most cost-effective type for each task. |
| Prediction model | none, qr/v1, qr/v2 | none | For private preview, use qr/v2. |
| Fusion snapshots | Enabled / Disabled | Disabled | When enabled, interrupted tasks (for example, after a Spot reclaim) resume from a Fusion snapshot instead of restarting from scratch. |
| Use NVMe instance storage | Enabled / Disabled | Disabled | Restrict Intelligent Compute to instance types that provide local SSD (NVMe) storage for faster I/O. |
| Enable warm pool | Enabled / Disabled | Disabled | Keep a pool of idle EC2 instances available to reduce task start latency. Not available on compute environments that run on Amazon ECS. |
| Desired warm VMs | Integer | — | Target number of idle VMs to keep warm. Must be greater than zero. Only visible when Enable warm pool is enabled. |
| Scale-to-zero timeout | Seconds | — | Seconds of inactivity after which the warm pool scales to zero. Set to 0 to never scale to zero. Only visible when Enable warm pool is enabled. |
Spot retries and process.maxRetries
Intelligent Compute and Nextflow retry tasks at two independent layers.
The scheduler retries a task on Spot capacity up to the Max spot attempts value. These retries happen inside the scheduler. Neither Nextflow nor Seqera Platform sees them. With spotFirst, the scheduler can also move a task from Spot to On-Demand capacity. Nextflow does not see this transition either.
A Spot capacity failure reaches Nextflow only after the scheduler exhausts its attempts. Nextflow then counts one task error. That error consumes a process.maxRetries attempt when the process sets errorStrategy = 'retry'.
Max spot attempts is a separate setting from the aws.batch.maxSpotAttempts Nextflow configuration property described in AWS Spot interruption management. That property applies to AWS Batch compute environments only and has no effect on Intelligent Compute.