2024-07-08T16:48:09.3750041Z Current runner version: '2.317.0'
2024-07-08T16:48:09.3756195Z Runner name: 'i-00726d0d1e8699347'
2024-07-08T16:48:09.3757086Z Runner group name: 'Default'
2024-07-08T16:48:09.3758062Z Machine name: 'ip-10-0-79-130'
2024-07-08T16:48:09.3763139Z ##[group]GITHUB_TOKEN Permissions
2024-07-08T16:48:09.3765226Z Actions: read
2024-07-08T16:48:09.3765968Z Attestations: read
2024-07-08T16:48:09.3766535Z Checks: read
2024-07-08T16:48:09.3767073Z Contents: read
2024-07-08T16:48:09.3767597Z Deployments: read
2024-07-08T16:48:09.3768161Z Discussions: read
2024-07-08T16:48:09.3768727Z Issues: read
2024-07-08T16:48:09.3769289Z Metadata: read
2024-07-08T16:48:09.3769805Z Packages: read
2024-07-08T16:48:09.3770367Z Pages: read
2024-07-08T16:48:09.3770895Z PullRequests: read
2024-07-08T16:48:09.3771468Z RepositoryProjects: read
2024-07-08T16:48:09.3772158Z SecurityEvents: read
2024-07-08T16:48:09.3772741Z Statuses: read
2024-07-08T16:48:09.3773279Z ##[endgroup]
2024-07-08T16:48:09.3776559Z Secret source: Actions
2024-07-08T16:48:09.3777429Z Prepare workflow directory
2024-07-08T16:48:09.7213398Z Prepare all required actions
2024-07-08T16:48:09.7399841Z Getting action download info
2024-07-08T16:48:09.8910760Z Download action repository 'pytorch/test-infra@main' (SHA:42f716d5f8e9e9215f663a172ae233a6008ebbcb)
2024-07-08T16:48:10.2577618Z Download action repository 'pytorch/pytorch@main' (SHA:63743b223c0b69f4dbbcfd349f523eee3f5a12c2)
2024-07-08T16:48:12.9991742Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0)
2024-07-08T16:48:13.1506421Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2024-07-08T16:48:13.4262679Z Getting action download info
2024-07-08T16:48:13.5099446Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c)
2024-07-08T16:48:13.6931983Z Getting action download info
2024-07-08T16:48:13.7965182Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2024-07-08T16:48:13.9354939Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (c8ab2e8b637515b6488931f5e59f23848aae9991)
2024-07-08T16:48:13.9357034Z ##[group] Inputs
2024-07-08T16:48:13.9357526Z   build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-07-08T16:48:13.9366240Z   test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-07-08T16:48:13.9375528Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:48:13.9376782Z   sync-tag:
2024-07-08T16:48:13.9377642Z   timeout-minutes: 240
2024-07-08T16:48:13.9377993Z   use-gha:
2024-07-08T16:48:13.9378286Z   dashboard-tag:
2024-07-08T16:48:13.9378621Z   s3-bucket: gha-artifacts
2024-07-08T16:48:13.9378992Z   aws-role-to-assume:
2024-07-08T16:48:13.9379337Z ##[endgroup]
2024-07-08T16:48:13.9380212Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-07-08T16:48:13.9937018Z A job started hook has been configured by the self-hosted runner administrator
2024-07-08T16:48:14.0073484Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh'
2024-07-08T16:48:14.0083748Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:48:14.0084282Z ##[endgroup]
2024-07-08T16:48:14.6230487Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2024-07-08T16:48:14.6231239Z with:
2024-07-08T16:48:14.6232023Z   github-secret: ***
2024-07-08T16:48:14.6233063Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2024-07-08T16:48:14.6234126Z   activate-with-label: false
2024-07-08T16:48:14.6234561Z   label: with-ssh
2024-07-08T16:48:14.6234957Z   remove-existing-keys: true
2024-07-08T16:48:14.6235392Z   fail-silently: true
2024-07-08T16:48:14.6235770Z env:
2024-07-08T16:48:14.6236114Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:48:14.6236576Z ##[endgroup]
2024-07-08T16:48:14.7105093Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info.
2024-07-08T16:48:14.7106317Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys
2024-07-08T16:48:14.7180660Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2024-07-08T16:48:14.7181283Z with:
2024-07-08T16:48:14.7181633Z   submodules: recursive
2024-07-08T16:48:14.7182032Z   fetch-depth: 0
2024-07-08T16:48:14.7182398Z env:
2024-07-08T16:48:14.7182738Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:48:14.7183143Z ##[endgroup]
2024-07-08T16:48:14.7376264Z ##[group]Run retry () {
2024-07-08T16:48:14.7376721Z [36;1mretry () {[0m
2024-07-08T16:48:14.7377300Z [36;1m  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)[0m
2024-07-08T16:48:14.7377916Z [36;1m}[0m
2024-07-08T16:48:14.7378288Z [36;1mecho "${GITHUB_WORKSPACE}"[0m
2024-07-08T16:48:14.7378786Z [36;1mif [ -z "${NO_SUDO}" ]; then[0m
2024-07-08T16:48:14.7379333Z [36;1m  retry sudo rm -rf "${GITHUB_WORKSPACE}"[0m
2024-07-08T16:48:14.7379853Z [36;1melse[0m
2024-07-08T16:48:14.7380262Z [36;1m  retry rm -rf "${GITHUB_WORKSPACE}"[0m
2024-07-08T16:48:14.7380752Z [36;1mfi[0m
2024-07-08T16:48:14.7381161Z [36;1mmkdir "${GITHUB_WORKSPACE}"[0m
2024-07-08T16:48:14.7389168Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:48:14.7389743Z env:
2024-07-08T16:48:14.7390145Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:48:14.7390593Z   NO_SUDO:
2024-07-08T16:48:14.7391034Z ##[endgroup]
2024-07-08T16:48:14.7414827Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-07-08T16:48:16.8651620Z ##[group]Run malfet/checkout@silent-checkout
2024-07-08T16:48:16.8652178Z with:
2024-07-08T16:48:16.8652568Z   ref: c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:48:16.8653078Z   fetch-depth: 0
2024-07-08T16:48:16.8653467Z   submodules: recursive
2024-07-08T16:48:16.8654010Z   quiet-checkout: true
2024-07-08T16:48:16.8654432Z   repository: pytorch/pytorch
2024-07-08T16:48:16.8654993Z   token: ***
2024-07-08T16:48:16.8655356Z   ssh-strict: true
2024-07-08T16:48:16.8655755Z   persist-credentials: true
2024-07-08T16:48:16.8656185Z   clean: true
2024-07-08T16:48:16.8656590Z   sparse-checkout-cone-mode: true
2024-07-08T16:48:16.8657056Z   lfs: false
2024-07-08T16:48:16.8657429Z   set-safe-directory: true
2024-07-08T16:48:16.8657839Z env:
2024-07-08T16:48:16.8658182Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:48:16.8658597Z ##[endgroup]
2024-07-08T16:48:16.9573902Z Syncing repository: pytorch/pytorch
2024-07-08T16:48:16.9575719Z ##[group]Getting Git version info
2024-07-08T16:48:16.9576552Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-07-08T16:48:16.9577511Z [command]/usr/bin/git version
2024-07-08T16:48:16.9578055Z git version 2.40.1
2024-07-08T16:48:16.9579814Z ##[endgroup]
2024-07-08T16:48:16.9591450Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/68a7ee48-fff3-4a09-9e83-d29ac364649b' before making global git config changes
2024-07-08T16:48:16.9592780Z Adding repository directory to the temporary git global config as a safe directory
2024-07-08T16:48:16.9593967Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-07-08T16:48:16.9622506Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2024-07-08T16:48:16.9626538Z ##[group]Initializing the repository
2024-07-08T16:48:16.9629618Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-07-08T16:48:16.9650695Z hint: Using 'master' as the name for the initial branch. This default branch name
2024-07-08T16:48:16.9651783Z hint: is subject to change. To configure the initial branch name to use in all
2024-07-08T16:48:16.9652650Z hint: of your new repositories, which will suppress this warning, call:
2024-07-08T16:48:16.9653362Z hint:
2024-07-08T16:48:16.9654139Z hint: 	git config --global init.defaultBranch <name>
2024-07-08T16:48:16.9654691Z hint:
2024-07-08T16:48:16.9655276Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2024-07-08T16:48:16.9656180Z hint: 'development'. The just-created branch can be renamed via this command:
2024-07-08T16:48:16.9657247Z hint:
2024-07-08T16:48:16.9657665Z hint: 	git branch -m <name>
2024-07-08T16:48:16.9658479Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2024-07-08T16:48:16.9660494Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2024-07-08T16:48:16.9685348Z ##[endgroup]
2024-07-08T16:48:16.9686136Z ##[group]Disabling automatic garbage collection
2024-07-08T16:48:16.9689097Z [command]/usr/bin/git config --local gc.auto 0
2024-07-08T16:48:16.9712543Z ##[endgroup]
2024-07-08T16:48:16.9713064Z ##[group]Setting up auth
2024-07-08T16:48:16.9718557Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-07-08T16:48:16.9739289Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-07-08T16:48:16.9934552Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-07-08T16:48:16.9954475Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-07-08T16:48:17.0147942Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-07-08T16:48:17.0190375Z ##[endgroup]
2024-07-08T16:48:17.0191105Z ##[group]Fetching the repository
2024-07-08T16:48:17.0196370Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2024-07-08T16:48:19.4717533Z remote: Enumerating objects: 991888
2024-07-08T16:48:19.4718330Z remote: Enumerating objects: 994111, done.
2024-07-08T16:48:19.4719380Z remote: Counting objects:   0% (1/2223)
2024-07-08T16:48:19.4720295Z remote: Counting objects:   1% (23/2223)
2024-07-08T16:48:19.4721175Z remote: Counting objects:   2% (45/2223)
2024-07-08T16:48:19.4721972Z remote: Counting objects:   3% (67/2223)
2024-07-08T16:48:19.4722635Z remote: Counting objects:   4% (89/2223)
2024-07-08T16:48:19.4723422Z remote: Counting objects:   5% (112/2223)
2024-07-08T16:48:19.4724024Z remote: Counting objects:   6% (134/2223)
2024-07-08T16:48:19.4724729Z remote: Counting objects:   7% (156/2223)
2024-07-08T16:48:19.4725314Z remote: Counting objects:   8% (178/2223)
2024-07-08T16:48:19.4726454Z remote: Counting objects:   9% (201/2223)
2024-07-08T16:48:19.4727227Z remote: Counting objects:  10% (223/2223)
2024-07-08T16:48:19.4728022Z remote: Counting objects:  11% (245/2223)
2024-07-08T16:48:19.4728872Z remote: Counting objects:  12% (267/2223)
2024-07-08T16:48:19.4729595Z remote: Counting objects:  13% (289/2223)
2024-07-08T16:48:19.4730295Z remote: Counting objects:  14% (312/2223)
2024-07-08T16:48:19.4731103Z remote: Counting objects:  15% (334/2223)
2024-07-08T16:48:19.4731834Z remote: Counting objects:  16% (356/2223)
2024-07-08T16:48:19.4732612Z remote: Counting objects:  17% (378/2223)
2024-07-08T16:48:19.4733391Z remote: Counting objects:  18% (401/2223)
2024-07-08T16:48:19.4734164Z remote: Counting objects:  19% (423/2223)
2024-07-08T16:48:19.4734944Z remote: Counting objects:  20% (445/2223)
2024-07-08T16:48:19.4735669Z remote: Counting objects:  21% (467/2223)
2024-07-08T16:48:19.4736439Z remote: Counting objects:  22% (490/2223)
2024-07-08T16:48:19.4737225Z remote: Counting objects:  23% (512/2223)
2024-07-08T16:48:19.4737816Z remote: Counting objects:  24% (534/2223)
2024-07-08T16:48:19.4738462Z remote: Counting objects:  25% (556/2223)
2024-07-08T16:48:19.4739253Z remote: Counting objects:  26% (578/2223)
2024-07-08T16:48:19.4739955Z remote: Counting objects:  27% (601/2223)
2024-07-08T16:48:19.4740956Z remote: Counting objects:  28% (623/2223)
2024-07-08T16:48:19.4741698Z remote: Counting objects:  29% (645/2223)
2024-07-08T16:48:19.4742315Z remote: Counting objects:  30% (667/2223)
2024-07-08T16:48:19.4743016Z remote: Counting objects:  31% (690/2223)
2024-07-08T16:48:19.4743622Z remote: Counting objects:  32% (712/2223)
2024-07-08T16:48:19.4744251Z remote: Counting objects:  33% (734/2223)
2024-07-08T16:48:19.4744942Z remote: Counting objects:  34% (756/2223)
2024-07-08T16:48:19.4745575Z remote: Counting objects:  35% (779/2223)
2024-07-08T16:48:19.4746254Z remote: Counting objects:  36% (801/2223)
2024-07-08T16:48:19.4746862Z remote: Counting objects:  37% (823/2223)
2024-07-08T16:48:19.4747587Z remote: Counting objects:  38% (845/2223)
2024-07-08T16:48:19.4748242Z remote: Counting objects:  39% (867/2223)
2024-07-08T16:48:19.4748920Z remote: Counting objects:  40% (890/2223)
2024-07-08T16:48:19.4749508Z remote: Counting objects:  41% (912/2223)
2024-07-08T16:48:19.4750088Z remote: Counting objects:  42% (934/2223)
2024-07-08T16:48:19.4750815Z remote: Counting objects:  43% (956/2223)
2024-07-08T16:48:19.4751473Z remote: Counting objects:  44% (979/2223)
2024-07-08T16:48:19.4752053Z remote: Counting objects:  45% (1001/2223)
2024-07-08T16:48:19.4752659Z remote: Counting objects:  46% (1023/2223)
2024-07-08T16:48:19.4753261Z remote: Counting objects:  47% (1045/2223)
2024-07-08T16:48:19.4753854Z remote: Counting objects:  48% (1068/2223)
2024-07-08T16:48:19.4754441Z remote: Counting objects:  49% (1090/2223)
2024-07-08T16:48:19.4755172Z remote: Counting objects:  50% (1112/2223)
2024-07-08T16:48:19.4755764Z remote: Counting objects:  51% (1134/2223)
2024-07-08T16:48:19.4756360Z remote: Counting objects:  52% (1156/2223)
2024-07-08T16:48:19.4756953Z remote: Counting objects:  53% (1179/2223)
2024-07-08T16:48:19.4757543Z remote: Counting objects:  54% (1201/2223)
2024-07-08T16:48:19.4758141Z remote: Counting objects:  55% (1223/2223)
2024-07-08T16:48:19.4758733Z remote: Counting objects:  56% (1245/2223)
2024-07-08T16:48:19.4759317Z remote: Counting objects:  57% (1268/2223)
2024-07-08T16:48:19.4759909Z remote: Counting objects:  58% (1290/2223)
2024-07-08T16:48:19.4760505Z remote: Counting objects:  59% (1312/2223)
2024-07-08T16:48:19.4761186Z remote: Counting objects:  60% (1334/2223)
2024-07-08T16:48:19.4761775Z remote: Counting objects:  61% (1357/2223)
2024-07-08T16:48:19.4762369Z remote: Counting objects:  62% (1379/2223)
2024-07-08T16:48:19.4762985Z remote: Counting objects:  63% (1401/2223)
2024-07-08T16:48:19.4763578Z remote: Counting objects:  64% (1423/2223)
2024-07-08T16:48:19.4764178Z remote: Counting objects:  65% (1445/2223)
2024-07-08T16:48:19.4764770Z remote: Counting objects:  66% (1468/2223)
2024-07-08T16:48:19.4765369Z remote: Counting objects:  67% (1490/2223)
2024-07-08T16:48:19.4766326Z remote: Counting objects:  68% (1512/2223)
2024-07-08T16:48:19.4767122Z remote: Counting objects:  69% (1534/2223)
2024-07-08T16:48:19.4767716Z remote: Counting objects:  70% (1557/2223)
2024-07-08T16:48:19.4768308Z remote: Counting objects:  71% (1579/2223)
2024-07-08T16:48:19.4768915Z remote: Counting objects:  72% (1601/2223)
2024-07-08T16:48:19.4769729Z remote: Counting objects:  73% (1623/2223)
2024-07-08T16:48:19.4770494Z remote: Counting objects:  74% (1646/2223)
2024-07-08T16:48:19.4771179Z remote: Counting objects:  75% (1668/2223)
2024-07-08T16:48:19.4771790Z remote: Counting objects:  76% (1690/2223)
2024-07-08T16:48:19.4772389Z remote: Counting objects:  77% (1712/2223)
2024-07-08T16:48:19.4772976Z remote: Counting objects:  78% (1734/2223)
2024-07-08T16:48:19.4773690Z remote: Counting objects:  79% (1757/2223)
2024-07-08T16:48:19.4774304Z remote: Counting objects:  80% (1779/2223)
2024-07-08T16:48:19.4774897Z remote: Counting objects:  81% (1801/2223)
2024-07-08T16:48:19.4775487Z remote: Counting objects:  82% (1823/2223)
2024-07-08T16:48:19.4776081Z remote: Counting objects:  83% (1846/2223)
2024-07-08T16:48:19.4776673Z remote: Counting objects:  84% (1868/2223)
2024-07-08T16:48:19.4777265Z remote: Counting objects:  85% (1890/2223)
2024-07-08T16:48:19.4777859Z remote: Counting objects:  86% (1912/2223)
2024-07-08T16:48:19.4778531Z remote: Counting objects:  87% (1935/2223)
2024-07-08T16:48:19.4779134Z remote: Counting objects:  88% (1957/2223)
2024-07-08T16:48:19.4779739Z remote: Counting objects:  89% (1979/2223)
2024-07-08T16:48:19.4780330Z remote: Counting objects:  90% (2001/2223)
2024-07-08T16:48:19.4780921Z remote: Counting objects:  91% (2023/2223)
2024-07-08T16:48:19.4781529Z remote: Counting objects:  92% (2046/2223)
2024-07-08T16:48:19.4782120Z remote: Counting objects:  93% (2068/2223)
2024-07-08T16:48:19.4782712Z remote: Counting objects:  94% (2090/2223)
2024-07-08T16:48:19.4783310Z remote: Counting objects:  95% (2112/2223)
2024-07-08T16:48:19.4783902Z remote: Counting objects:  96% (2135/2223)
2024-07-08T16:48:19.4784502Z remote: Counting objects:  97% (2157/2223)
2024-07-08T16:48:19.4785088Z remote: Counting objects:  98% (2179/2223)
2024-07-08T16:48:19.4785680Z remote: Counting objects:  99% (2201/2223)
2024-07-08T16:48:19.4786286Z remote: Counting objects: 100% (2223/2223)
2024-07-08T16:48:19.4786999Z remote: Counting objects: 100% (2223/2223), done.
2024-07-08T16:48:19.5159100Z remote: Compressing objects:   0% (1/1068)
2024-07-08T16:48:19.5758784Z remote: Compressing objects:   1% (11/1068)
2024-07-08T16:48:19.7134068Z remote: Compressing objects:   2% (22/1068)
2024-07-08T16:48:19.7940207Z remote: Compressing objects:   3% (33/1068)
2024-07-08T16:48:19.8597028Z remote: Compressing objects:   4% (43/1068)
2024-07-08T16:48:19.8722950Z remote: Compressing objects:   5% (54/1068)
2024-07-08T16:48:19.8865170Z remote: Compressing objects:   6% (65/1068)
2024-07-08T16:48:19.9070212Z remote: Compressing objects:   7% (75/1068)
2024-07-08T16:48:19.9317771Z remote: Compressing objects:   8% (86/1068)
2024-07-08T16:48:19.9318501Z remote: Compressing objects:   9% (97/1068)
2024-07-08T16:48:19.9416245Z remote: Compressing objects:  10% (107/1068)
2024-07-08T16:48:19.9518476Z remote: Compressing objects:  11% (118/1068)
2024-07-08T16:48:19.9519032Z remote: Compressing objects:  12% (129/1068)
2024-07-08T16:48:19.9574936Z remote: Compressing objects:  13% (139/1068)
2024-07-08T16:48:19.9576004Z remote: Compressing objects:  14% (150/1068)
2024-07-08T16:48:19.9632102Z remote: Compressing objects:  15% (161/1068)
2024-07-08T16:48:19.9692823Z remote: Compressing objects:  16% (171/1068)
2024-07-08T16:48:19.9693370Z remote: Compressing objects:  17% (182/1068)
2024-07-08T16:48:19.9693913Z remote: Compressing objects:  18% (193/1068)
2024-07-08T16:48:19.9724378Z remote: Compressing objects:  19% (203/1068)
2024-07-08T16:48:19.9724921Z remote: Compressing objects:  20% (214/1068)
2024-07-08T16:48:19.9725454Z remote: Compressing objects:  21% (225/1068)
2024-07-08T16:48:19.9726128Z remote: Compressing objects:  22% (235/1068)
2024-07-08T16:48:19.9726661Z remote: Compressing objects:  23% (246/1068)
2024-07-08T16:48:19.9727201Z remote: Compressing objects:  24% (257/1068)
2024-07-08T16:48:19.9738425Z remote: Compressing objects:  25% (267/1068)
2024-07-08T16:48:19.9745942Z remote: Compressing objects:  26% (278/1068)
2024-07-08T16:48:19.9746485Z remote: Compressing objects:  27% (289/1068)
2024-07-08T16:48:19.9747236Z remote: Compressing objects:  28% (300/1068)
2024-07-08T16:48:19.9747773Z remote: Compressing objects:  29% (310/1068)
2024-07-08T16:48:19.9748302Z remote: Compressing objects:  30% (321/1068)
2024-07-08T16:48:19.9752558Z remote: Compressing objects:  31% (332/1068)
2024-07-08T16:48:19.9753374Z remote: Compressing objects:  32% (342/1068)
2024-07-08T16:48:19.9797268Z remote: Compressing objects:  33% (353/1068)
2024-07-08T16:48:19.9809266Z remote: Compressing objects:  34% (364/1068)
2024-07-08T16:48:19.9818726Z remote: Compressing objects:  35% (374/1068)
2024-07-08T16:48:19.9828987Z remote: Compressing objects:  36% (385/1068)
2024-07-08T16:48:19.9847392Z remote: Compressing objects:  37% (396/1068)
2024-07-08T16:48:19.9876800Z remote: Compressing objects:  38% (406/1068)
2024-07-08T16:48:19.9897181Z remote: Compressing objects:  39% (417/1068)
2024-07-08T16:48:19.9902587Z remote: Compressing objects:  40% (428/1068)
2024-07-08T16:48:19.9915595Z remote: Compressing objects:  41% (438/1068)
2024-07-08T16:48:19.9947997Z remote: Compressing objects:  42% (449/1068)
2024-07-08T16:48:19.9953585Z remote: Compressing objects:  43% (460/1068)
2024-07-08T16:48:19.9969559Z remote: Compressing objects:  44% (470/1068)
2024-07-08T16:48:19.9977412Z remote: Compressing objects:  45% (481/1068)
2024-07-08T16:48:19.9986285Z remote: Compressing objects:  46% (492/1068)
2024-07-08T16:48:19.9999398Z remote: Compressing objects:  47% (502/1068)
2024-07-08T16:48:20.0016446Z remote: Compressing objects:  48% (513/1068)
2024-07-08T16:48:20.0033893Z remote: Compressing objects:  49% (524/1068)
2024-07-08T16:48:20.0050104Z remote: Compressing objects:  50% (534/1068)
2024-07-08T16:48:20.0068944Z remote: Compressing objects:  51% (545/1068)
2024-07-08T16:48:20.0080188Z remote: Compressing objects:  52% (556/1068)
2024-07-08T16:48:20.0088016Z remote: Compressing objects:  53% (567/1068)
2024-07-08T16:48:20.0099115Z remote: Compressing objects:  54% (577/1068)
2024-07-08T16:48:20.0107271Z remote: Compressing objects:  55% (588/1068)
2024-07-08T16:48:20.0110879Z remote: Compressing objects:  56% (599/1068)
2024-07-08T16:48:20.0114149Z remote: Compressing objects:  57% (609/1068)
2024-07-08T16:48:20.0117722Z remote: Compressing objects:  58% (620/1068)
2024-07-08T16:48:20.0121921Z remote: Compressing objects:  59% (631/1068)
2024-07-08T16:48:20.0125976Z remote: Compressing objects:  60% (641/1068)
2024-07-08T16:48:20.0128491Z remote: Compressing objects:  61% (652/1068)
2024-07-08T16:48:20.0131211Z remote: Compressing objects:  62% (663/1068)
2024-07-08T16:48:20.0131754Z remote: Compressing objects:  63% (673/1068)
2024-07-08T16:48:20.0133901Z remote: Compressing objects:  64% (684/1068)
2024-07-08T16:48:20.0136598Z remote: Compressing objects:  65% (695/1068)
2024-07-08T16:48:20.0139379Z remote: Compressing objects:  66% (705/1068)
2024-07-08T16:48:20.0139927Z remote: Compressing objects:  67% (716/1068)
2024-07-08T16:48:20.0141746Z remote: Compressing objects:  68% (727/1068)
2024-07-08T16:48:20.0142294Z remote: Compressing objects:  69% (737/1068)
2024-07-08T16:48:20.0144408Z remote: Compressing objects:  70% (748/1068)
2024-07-08T16:48:20.0144954Z remote: Compressing objects:  71% (759/1068)
2024-07-08T16:48:20.0145535Z remote: Compressing objects:  72% (769/1068)
2024-07-08T16:48:20.0146065Z remote: Compressing objects:  73% (780/1068)
2024-07-08T16:48:20.0146607Z remote: Compressing objects:  74% (791/1068)
2024-07-08T16:48:20.0147137Z remote: Compressing objects:  75% (801/1068)
2024-07-08T16:48:20.0147666Z remote: Compressing objects:  76% (812/1068)
2024-07-08T16:48:20.0148359Z remote: Compressing objects:  77% (823/1068)
2024-07-08T16:48:20.0149003Z remote: Compressing objects:  78% (834/1068)
2024-07-08T16:48:20.0150389Z remote: Compressing objects:  79% (844/1068)
2024-07-08T16:48:20.0152322Z remote: Compressing objects:  80% (855/1068)
2024-07-08T16:48:20.0154696Z remote: Compressing objects:  81% (866/1068)
2024-07-08T16:48:20.0158139Z remote: Compressing objects:  82% (876/1068)
2024-07-08T16:48:20.0161523Z remote: Compressing objects:  83% (887/1068)
2024-07-08T16:48:20.0164175Z remote: Compressing objects:  84% (898/1068)
2024-07-08T16:48:20.0166675Z remote: Compressing objects:  85% (908/1068)
2024-07-08T16:48:20.0169410Z remote: Compressing objects:  86% (919/1068)
2024-07-08T16:48:20.0169986Z remote: Compressing objects:  87% (930/1068)
2024-07-08T16:48:20.0172009Z remote: Compressing objects:  88% (940/1068)
2024-07-08T16:48:20.0174436Z remote: Compressing objects:  89% (951/1068)
2024-07-08T16:48:20.0177311Z remote: Compressing objects:  90% (962/1068)
2024-07-08T16:48:20.0179795Z remote: Compressing objects:  91% (972/1068)
2024-07-08T16:48:20.0180341Z remote: Compressing objects:  92% (983/1068)
2024-07-08T16:48:20.0182282Z remote: Compressing objects:  93% (994/1068)
2024-07-08T16:48:20.0182831Z remote: Compressing objects:  94% (1004/1068)
2024-07-08T16:48:20.0185036Z remote: Compressing objects:  95% (1015/1068)
2024-07-08T16:48:20.0185584Z remote: Compressing objects:  96% (1026/1068)
2024-07-08T16:48:20.0187652Z remote: Compressing objects:  97% (1036/1068)
2024-07-08T16:48:20.0188258Z remote: Compressing objects:  98% (1047/1068)
2024-07-08T16:48:20.0190120Z remote: Compressing objects:  99% (1058/1068)
2024-07-08T16:48:20.0190776Z remote: Compressing objects: 100% (1068/1068)
2024-07-08T16:48:20.0191353Z remote: Compressing objects: 100% (1068/1068), done.
2024-07-08T16:48:40.2831621Z remote: Total 994111 (delta 1386), reused 1837 (delta 1149), pack-reused 991888
2024-07-08T16:49:01.9614268Z [command]/usr/bin/git rev-parse --verify --quiet c8ab2e8b637515b6488931f5e59f23848aae9991^{object}
2024-07-08T16:49:01.9637773Z c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:49:01.9642042Z ##[endgroup]
2024-07-08T16:49:01.9642661Z ##[group]Determining the checkout info
2024-07-08T16:49:01.9644935Z ##[endgroup]
2024-07-08T16:49:01.9645486Z ##[group]Checking out the ref
2024-07-08T16:49:01.9647022Z [command]/usr/bin/git checkout --quiet --force c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:49:03.1759907Z ##[endgroup]
2024-07-08T16:49:03.1760516Z ##[group]Setting up auth for fetching submodules
2024-07-08T16:49:03.1764459Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2024-07-08T16:49:03.1804878Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2024-07-08T16:49:03.1827608Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2024-07-08T16:49:03.1846816Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2024-07-08T16:49:03.1862495Z ##[endgroup]
2024-07-08T16:49:03.1863026Z ##[group]Fetching submodules
2024-07-08T16:49:03.1865959Z [command]/usr/bin/git submodule sync --recursive
2024-07-08T16:49:03.2064143Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2024-07-08T16:49:03.2268317Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2024-07-08T16:49:03.2269849Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2024-07-08T16:49:03.2271339Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2024-07-08T16:49:03.2273111Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2024-07-08T16:49:03.2274923Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2024-07-08T16:49:03.2276698Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2024-07-08T16:49:03.2278584Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2024-07-08T16:49:03.2280641Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib'
2024-07-08T16:49:03.2283293Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2024-07-08T16:49:03.2285835Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2024-07-08T16:49:03.2289162Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2024-07-08T16:49:03.2291481Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2024-07-08T16:49:03.2293903Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2024-07-08T16:49:03.2296530Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2024-07-08T16:49:03.2298930Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2024-07-08T16:49:03.2301626Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2024-07-08T16:49:03.2304460Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2024-07-08T16:49:03.2307082Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2024-07-08T16:49:03.2309959Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2024-07-08T16:49:03.2312795Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2024-07-08T16:49:03.2315805Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2024-07-08T16:49:03.2318835Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2024-07-08T16:49:03.2322264Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc'
2024-07-08T16:49:03.2325369Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2024-07-08T16:49:03.2329114Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2024-07-08T16:49:03.2332306Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2024-07-08T16:49:03.2335992Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp'
2024-07-08T16:49:03.2339236Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2024-07-08T16:49:03.2342748Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2024-07-08T16:49:03.2346282Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2024-07-08T16:49:03.2350039Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2024-07-08T16:49:03.2353785Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2024-07-08T16:49:03.2357526Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2024-07-08T16:49:03.2361275Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2024-07-08T16:49:03.2365318Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2024-07-08T16:49:03.2383467Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2024-07-08T16:49:04.5158854Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2024-07-08T16:49:04.7273571Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2024-07-08T16:49:04.9059473Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2024-07-08T16:49:05.1538727Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2024-07-08T16:49:07.1862589Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2024-07-08T16:49:19.1545255Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2024-07-08T16:49:19.5700676Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'...
2024-07-08T16:49:20.1155841Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2024-07-08T16:49:20.7575814Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2024-07-08T16:49:21.9216694Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2024-07-08T16:49:23.6480834Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2024-07-08T16:49:29.5316608Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2024-07-08T16:49:30.8906970Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2024-07-08T16:49:32.6211769Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2024-07-08T16:49:33.8984816Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2024-07-08T16:49:34.0638237Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2024-07-08T16:49:34.4607520Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2024-07-08T16:49:34.7780458Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2024-07-08T16:49:35.8472001Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2024-07-08T16:49:36.2041627Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2024-07-08T16:49:36.4780721Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2024-07-08T16:49:37.9585043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'...
2024-07-08T16:49:38.6688490Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2024-07-08T16:49:39.3766837Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2024-07-08T16:49:45.1713085Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2024-07-08T16:49:47.7786310Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'...
2024-07-08T16:49:54.0973180Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2024-07-08T16:49:54.3573454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2024-07-08T16:50:03.0420208Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2024-07-08T16:50:03.1986483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2024-07-08T16:50:03.3759612Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2024-07-08T16:50:04.4216903Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2024-07-08T16:50:04.7485984Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2024-07-08T16:50:05.4163413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2024-07-08T16:50:05.8355573Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2024-07-08T16:50:05.8439685Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2024-07-08T16:50:05.8499831Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2024-07-08T16:50:05.8681830Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2024-07-08T16:50:05.8987797Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2024-07-08T16:50:06.6670211Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2'
2024-07-08T16:50:06.6840904Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2024-07-08T16:50:06.7193155Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006'
2024-07-08T16:50:06.8042565Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5'
2024-07-08T16:50:06.8312400Z Submodule path 'third_party/cudnn_frontend': checked out '98ca4e1941fe3263f128f74f10063a3ea35c7019'
2024-07-08T16:50:07.2567145Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49'
2024-07-08T16:50:07.4680191Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2024-07-08T16:50:07.5318333Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e'
2024-07-08T16:50:07.5329246Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2024-07-08T16:50:07.5332615Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2024-07-08T16:50:07.5334426Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass'
2024-07-08T16:50:07.5336226Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2024-07-08T16:50:07.5337914Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2024-07-08T16:50:07.5354564Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2024-07-08T16:50:08.5566289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2024-07-08T16:50:09.1457942Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'...
2024-07-08T16:50:10.8472985Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2024-07-08T16:50:11.8466876Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2024-07-08T16:50:12.1783901Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2024-07-08T16:50:12.2611945Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2024-07-08T16:50:12.6006876Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2024-07-08T16:50:12.6535972Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2024-07-08T16:50:12.6618674Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2024-07-08T16:50:12.7516578Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2024-07-08T16:50:12.7794864Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281'
2024-07-08T16:50:12.7855505Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2024-07-08T16:50:12.8154384Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2024-07-08T16:50:12.8346231Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509'
2024-07-08T16:50:12.8701785Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-07-08T16:50:12.8790759Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b'
2024-07-08T16:50:12.8799466Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2024-07-08T16:50:12.8815016Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2024-07-08T16:50:22.8671952Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67'
2024-07-08T16:50:22.8820076Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2024-07-08T16:50:22.9606396Z Submodule path 'third_party/kineto': checked out '9d33b110176854afb0649586674ca435b9d44f1d'
2024-07-08T16:50:22.9619267Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog'
2024-07-08T16:50:22.9621832Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2024-07-08T16:50:22.9623940Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2024-07-08T16:50:22.9638905Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'...
2024-07-08T16:50:23.4728049Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2024-07-08T16:50:24.7317731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2024-07-08T16:50:25.7809722Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2024-07-08T16:50:25.7819341Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-07-08T16:50:25.7821534Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-07-08T16:50:25.7823571Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-07-08T16:50:25.7825305Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-07-08T16:50:25.7827048Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-07-08T16:50:25.7829291Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-07-08T16:50:25.7831142Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-07-08T16:50:25.7833296Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-07-08T16:50:25.7850608Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'...
2024-07-08T16:50:26.8461748Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'...
2024-07-08T16:50:27.2144534Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'...
2024-07-08T16:50:28.4241939Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'...
2024-07-08T16:50:28.6980529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'...
2024-07-08T16:50:29.1886028Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'...
2024-07-08T16:50:30.1758505Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'...
2024-07-08T16:50:35.9369746Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'...
2024-07-08T16:50:36.2868277Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2024-07-08T16:50:36.3004280Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2024-07-08T16:50:36.3296446Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2024-07-08T16:50:36.3391532Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2024-07-08T16:50:36.3401141Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-07-08T16:50:36.3417389Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'...
2024-07-08T16:50:36.6658791Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2024-07-08T16:50:36.6794863Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2024-07-08T16:50:36.7126877Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2024-07-08T16:50:36.7927084Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2024-07-08T16:50:36.8047370Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2024-07-08T16:50:36.8339422Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50'
2024-07-08T16:50:36.8827055Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2024-07-08T16:50:36.9129815Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2024-07-08T16:50:36.9318230Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290'
2024-07-08T16:50:37.0168792Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2024-07-08T16:50:37.2855531Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781'
2024-07-08T16:50:37.2879324Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2024-07-08T16:50:37.2912091Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2024-07-08T16:50:37.2913547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2024-07-08T16:50:37.6428957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2024-07-08T16:50:38.6224432Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6'
2024-07-08T16:50:38.6501699Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242'
2024-07-08T16:50:38.7028957Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2024-07-08T16:50:38.7042532Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-07-08T16:50:38.7049959Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest'
2024-07-08T16:50:38.7052106Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-07-08T16:50:38.7054000Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-07-08T16:50:38.7056131Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-07-08T16:50:38.7058154Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-07-08T16:50:38.7060067Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-07-08T16:50:38.7061645Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-07-08T16:50:38.7074113Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'...
2024-07-08T16:50:39.0976040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'...
2024-07-08T16:50:40.1869529Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'...
2024-07-08T16:50:40.6293236Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'...
2024-07-08T16:50:46.2792605Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'...
2024-07-08T16:50:46.5311547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'...
2024-07-08T16:50:46.7333554Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'...
2024-07-08T16:50:47.0815499Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'...
2024-07-08T16:50:53.3348555Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2024-07-08T16:50:53.3679282Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2024-07-08T16:50:53.3799031Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2024-07-08T16:50:53.4630633Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2024-07-08T16:50:53.4721302Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2024-07-08T16:50:53.4827596Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2024-07-08T16:50:53.4937568Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2024-07-08T16:50:53.4946682Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-07-08T16:50:53.4948751Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-07-08T16:50:53.4963886Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'...
2024-07-08T16:50:55.2239775Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'...
2024-07-08T16:50:56.4125351Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2024-07-08T16:50:56.4487231Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2024-07-08T16:50:56.8094908Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2024-07-08T16:50:56.8174123Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe'
2024-07-08T16:50:57.0335445Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2024-07-08T16:50:57.0348633Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2024-07-08T16:50:57.0350508Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2024-07-08T16:50:57.0367712Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2024-07-08T16:50:57.7532751Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2024-07-08T16:50:58.7861438Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2024-07-08T16:50:58.8470208Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2024-07-08T16:50:58.8526557Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2024-07-08T16:50:58.8608912Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2024-07-08T16:50:58.8881847Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b'
2024-07-08T16:50:58.9103252Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2024-07-08T16:50:58.9436136Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b'
2024-07-08T16:50:58.9638179Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2024-07-08T16:50:58.9649177Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2024-07-08T16:50:58.9651035Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2024-07-08T16:50:58.9652760Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2024-07-08T16:50:58.9654524Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2024-07-08T16:50:58.9670987Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2024-07-08T16:50:59.9319142Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2024-07-08T16:51:00.1281962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2024-07-08T16:51:01.1572920Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2024-07-08T16:51:02.2303978Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2024-07-08T16:51:02.2416909Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2024-07-08T16:51:02.2983954Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2024-07-08T16:51:02.3208471Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2024-07-08T16:51:02.3218046Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-07-08T16:51:02.3233976Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2024-07-08T16:51:02.5218985Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2024-07-08T16:51:02.5242015Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2024-07-08T16:51:02.5440827Z Entering 'android/libs/fbjni'
2024-07-08T16:51:02.5469017Z Entering 'third_party/FP16'
2024-07-08T16:51:02.5496546Z Entering 'third_party/FXdiv'
2024-07-08T16:51:02.5521523Z Entering 'third_party/NNPACK'
2024-07-08T16:51:02.5547828Z Entering 'third_party/VulkanMemoryAllocator'
2024-07-08T16:51:02.5575228Z Entering 'third_party/XNNPACK'
2024-07-08T16:51:02.5612372Z Entering 'third_party/benchmark'
2024-07-08T16:51:02.5639969Z Entering 'third_party/cpp-httplib'
2024-07-08T16:51:02.5664814Z Entering 'third_party/cpuinfo'
2024-07-08T16:51:02.5692330Z Entering 'third_party/cudnn_frontend'
2024-07-08T16:51:02.5717878Z Entering 'third_party/cutlass'
2024-07-08T16:51:02.5749659Z Entering 'third_party/eigen'
2024-07-08T16:51:02.5777308Z Entering 'third_party/fbgemm'
2024-07-08T16:51:02.5804989Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-07-08T16:51:02.5830834Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-07-08T16:51:02.5857958Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-07-08T16:51:02.5887719Z Entering 'third_party/fbgemm/third_party/googletest'
2024-07-08T16:51:02.5912957Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-07-08T16:51:02.5942189Z Entering 'third_party/flatbuffers'
2024-07-08T16:51:02.5971887Z Entering 'third_party/fmt'
2024-07-08T16:51:02.5998134Z Entering 'third_party/foxi'
2024-07-08T16:51:02.6024637Z Entering 'third_party/gemmlowp/gemmlowp'
2024-07-08T16:51:02.6054246Z Entering 'third_party/gloo'
2024-07-08T16:51:02.6080368Z Entering 'third_party/googletest'
2024-07-08T16:51:02.6109373Z Entering 'third_party/ideep'
2024-07-08T16:51:02.6134116Z Entering 'third_party/ideep/mkl-dnn'
2024-07-08T16:51:02.6163782Z Entering 'third_party/ittapi'
2024-07-08T16:51:02.6190990Z Entering 'third_party/kineto'
2024-07-08T16:51:02.6219151Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-07-08T16:51:02.6246471Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-07-08T16:51:02.6275293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-07-08T16:51:02.6301072Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-07-08T16:51:02.6327931Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-07-08T16:51:02.6352830Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-07-08T16:51:02.6382682Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-07-08T16:51:02.6410536Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-07-08T16:51:02.6437111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-07-08T16:51:02.6465143Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-07-08T16:51:02.6492880Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-07-08T16:51:02.6518663Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-07-08T16:51:02.6545481Z Entering 'third_party/mimalloc'
2024-07-08T16:51:02.6573952Z Entering 'third_party/nccl/nccl'
2024-07-08T16:51:02.6602005Z Entering 'third_party/nlohmann'
2024-07-08T16:51:02.6627537Z Entering 'third_party/onnx'
2024-07-08T16:51:02.6665169Z Entering 'third_party/onnx/third_party/benchmark'
2024-07-08T16:51:02.6696040Z Entering 'third_party/onnx/third_party/pybind11'
2024-07-08T16:51:02.6722820Z Entering 'third_party/opentelemetry-cpp'
2024-07-08T16:51:02.6753818Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-07-08T16:51:02.6778797Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-07-08T16:51:02.6806642Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-07-08T16:51:02.6834354Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-07-08T16:51:02.6860915Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-07-08T16:51:02.6888422Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-07-08T16:51:02.6915670Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-07-08T16:51:02.6940796Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-07-08T16:51:02.6970194Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-07-08T16:51:02.6996752Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-07-08T16:51:02.7037994Z Entering 'third_party/pocketfft'
2024-07-08T16:51:02.7067578Z Entering 'third_party/protobuf'
2024-07-08T16:51:02.7095555Z Entering 'third_party/protobuf/third_party/benchmark'
2024-07-08T16:51:02.7120879Z Entering 'third_party/protobuf/third_party/googletest'
2024-07-08T16:51:02.7149804Z Entering 'third_party/psimd'
2024-07-08T16:51:02.7176685Z Entering 'third_party/pthreadpool'
2024-07-08T16:51:02.7201863Z Entering 'third_party/pybind11'
2024-07-08T16:51:02.7230255Z Entering 'third_party/python-peachpy'
2024-07-08T16:51:02.7258343Z Entering 'third_party/sleef'
2024-07-08T16:51:02.7284854Z Entering 'third_party/tensorpipe'
2024-07-08T16:51:02.7312079Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-07-08T16:51:02.7338084Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-07-08T16:51:02.7362647Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-07-08T16:51:02.7387607Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-07-08T16:51:02.7413933Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-07-08T16:51:02.7452568Z ##[endgroup]
2024-07-08T16:51:02.7454768Z ##[group]Persisting credentials for submodules
2024-07-08T16:51:02.7457833Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2024-07-08T16:51:02.7663100Z Entering 'android/libs/fbjni'
2024-07-08T16:51:02.7696336Z Entering 'third_party/FP16'
2024-07-08T16:51:02.7732035Z Entering 'third_party/FXdiv'
2024-07-08T16:51:02.7764051Z Entering 'third_party/NNPACK'
2024-07-08T16:51:02.7798684Z Entering 'third_party/VulkanMemoryAllocator'
2024-07-08T16:51:02.7834528Z Entering 'third_party/XNNPACK'
2024-07-08T16:51:02.7879805Z Entering 'third_party/benchmark'
2024-07-08T16:51:02.7915601Z Entering 'third_party/cpp-httplib'
2024-07-08T16:51:02.7950911Z Entering 'third_party/cpuinfo'
2024-07-08T16:51:02.7986498Z Entering 'third_party/cudnn_frontend'
2024-07-08T16:51:02.8021265Z Entering 'third_party/cutlass'
2024-07-08T16:51:02.8060471Z Entering 'third_party/eigen'
2024-07-08T16:51:02.8097302Z Entering 'third_party/fbgemm'
2024-07-08T16:51:02.8132377Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-07-08T16:51:02.8165894Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-07-08T16:51:02.8199465Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-07-08T16:51:02.8238731Z Entering 'third_party/fbgemm/third_party/googletest'
2024-07-08T16:51:02.8271001Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-07-08T16:51:02.8305098Z Entering 'third_party/flatbuffers'
2024-07-08T16:51:02.8343289Z Entering 'third_party/fmt'
2024-07-08T16:51:02.8379173Z Entering 'third_party/foxi'
2024-07-08T16:51:02.8413627Z Entering 'third_party/gemmlowp/gemmlowp'
2024-07-08T16:51:02.8448422Z Entering 'third_party/gloo'
2024-07-08T16:51:02.8480878Z Entering 'third_party/googletest'
2024-07-08T16:51:02.8515132Z Entering 'third_party/ideep'
2024-07-08T16:51:02.8548980Z Entering 'third_party/ideep/mkl-dnn'
2024-07-08T16:51:02.8586809Z Entering 'third_party/ittapi'
2024-07-08T16:51:02.8622972Z Entering 'third_party/kineto'
2024-07-08T16:51:02.8659076Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-07-08T16:51:02.8693259Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-07-08T16:51:02.8728706Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-07-08T16:51:02.8762545Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-07-08T16:51:02.8796662Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-07-08T16:51:02.8829985Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-07-08T16:51:02.8865012Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-07-08T16:51:02.8899323Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-07-08T16:51:02.8931307Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-07-08T16:51:02.8964031Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-07-08T16:51:02.9000499Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-07-08T16:51:02.9034547Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-07-08T16:51:02.9068511Z Entering 'third_party/mimalloc'
2024-07-08T16:51:02.9105001Z Entering 'third_party/nccl/nccl'
2024-07-08T16:51:02.9137699Z Entering 'third_party/nlohmann'
2024-07-08T16:51:02.9174398Z Entering 'third_party/onnx'
2024-07-08T16:51:02.9218839Z Entering 'third_party/onnx/third_party/benchmark'
2024-07-08T16:51:02.9253674Z Entering 'third_party/onnx/third_party/pybind11'
2024-07-08T16:51:02.9289810Z Entering 'third_party/opentelemetry-cpp'
2024-07-08T16:51:02.9324504Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-07-08T16:51:02.9358852Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-07-08T16:51:02.9393035Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-07-08T16:51:02.9424759Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-07-08T16:51:02.9458323Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-07-08T16:51:02.9491482Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-07-08T16:51:02.9525830Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-07-08T16:51:02.9559076Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-07-08T16:51:02.9593946Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-07-08T16:51:02.9628040Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-07-08T16:51:02.9674983Z Entering 'third_party/pocketfft'
2024-07-08T16:51:02.9709208Z Entering 'third_party/protobuf'
2024-07-08T16:51:02.9746316Z Entering 'third_party/protobuf/third_party/benchmark'
2024-07-08T16:51:02.9781128Z Entering 'third_party/protobuf/third_party/googletest'
2024-07-08T16:51:02.9813797Z Entering 'third_party/psimd'
2024-07-08T16:51:02.9849383Z Entering 'third_party/pthreadpool'
2024-07-08T16:51:02.9881120Z Entering 'third_party/pybind11'
2024-07-08T16:51:02.9915286Z Entering 'third_party/python-peachpy'
2024-07-08T16:51:02.9950359Z Entering 'third_party/sleef'
2024-07-08T16:51:02.9984053Z Entering 'third_party/tensorpipe'
2024-07-08T16:51:03.0018878Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-07-08T16:51:03.0052538Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-07-08T16:51:03.0086770Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-07-08T16:51:03.0120366Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-07-08T16:51:03.0153751Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-07-08T16:51:03.0198895Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2024-07-08T16:51:03.0397624Z Entering 'android/libs/fbjni'
2024-07-08T16:51:03.0429676Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2024-07-08T16:51:03.0443332Z Entering 'third_party/FP16'
2024-07-08T16:51:03.0473571Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2024-07-08T16:51:03.0484722Z Entering 'third_party/FXdiv'
2024-07-08T16:51:03.0516580Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2024-07-08T16:51:03.0530522Z Entering 'third_party/NNPACK'
2024-07-08T16:51:03.0561712Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2024-07-08T16:51:03.0575240Z Entering 'third_party/VulkanMemoryAllocator'
2024-07-08T16:51:03.0604099Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2024-07-08T16:51:03.0615615Z Entering 'third_party/XNNPACK'
2024-07-08T16:51:03.0647287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2024-07-08T16:51:03.0670469Z Entering 'third_party/benchmark'
2024-07-08T16:51:03.0701874Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2024-07-08T16:51:03.0715238Z Entering 'third_party/cpp-httplib'
2024-07-08T16:51:03.0747427Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2024-07-08T16:51:03.0758858Z Entering 'third_party/cpuinfo'
2024-07-08T16:51:03.0790668Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2024-07-08T16:51:03.0804148Z Entering 'third_party/cudnn_frontend'
2024-07-08T16:51:03.0835519Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2024-07-08T16:51:03.0847701Z Entering 'third_party/cutlass'
2024-07-08T16:51:03.0878154Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2024-07-08T16:51:03.0897434Z Entering 'third_party/eigen'
2024-07-08T16:51:03.0928553Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2024-07-08T16:51:03.0944400Z Entering 'third_party/fbgemm'
2024-07-08T16:51:03.0975805Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2024-07-08T16:51:03.0989760Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-07-08T16:51:03.1021994Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2024-07-08T16:51:03.1033724Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-07-08T16:51:03.1065083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2024-07-08T16:51:03.1077849Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-07-08T16:51:03.1108743Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2024-07-08T16:51:03.1126172Z Entering 'third_party/fbgemm/third_party/googletest'
2024-07-08T16:51:03.1156569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2024-07-08T16:51:03.1169621Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-07-08T16:51:03.1199954Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2024-07-08T16:51:03.1213840Z Entering 'third_party/flatbuffers'
2024-07-08T16:51:03.1244667Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2024-07-08T16:51:03.1258817Z Entering 'third_party/fmt'
2024-07-08T16:51:03.1290963Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2024-07-08T16:51:03.1304332Z Entering 'third_party/foxi'
2024-07-08T16:51:03.1335026Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2024-07-08T16:51:03.1347486Z Entering 'third_party/gemmlowp/gemmlowp'
2024-07-08T16:51:03.1380836Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2024-07-08T16:51:03.1393644Z Entering 'third_party/gloo'
2024-07-08T16:51:03.1425305Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2024-07-08T16:51:03.1438350Z Entering 'third_party/googletest'
2024-07-08T16:51:03.1469908Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2024-07-08T16:51:03.1481564Z Entering 'third_party/ideep'
2024-07-08T16:51:03.1513158Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2024-07-08T16:51:03.1524874Z Entering 'third_party/ideep/mkl-dnn'
2024-07-08T16:51:03.1556932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2024-07-08T16:51:03.1575009Z Entering 'third_party/ittapi'
2024-07-08T16:51:03.1605891Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2024-07-08T16:51:03.1619494Z Entering 'third_party/kineto'
2024-07-08T16:51:03.1650479Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2024-07-08T16:51:03.1662126Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-07-08T16:51:03.1694015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2024-07-08T16:51:03.1705931Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-07-08T16:51:03.1737120Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2024-07-08T16:51:03.1750213Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-07-08T16:51:03.1781373Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2024-07-08T16:51:03.1794198Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-07-08T16:51:03.1824473Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2024-07-08T16:51:03.1835232Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-07-08T16:51:03.1866249Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2024-07-08T16:51:03.1877957Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-07-08T16:51:03.1909007Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2024-07-08T16:51:03.1923198Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-07-08T16:51:03.1956032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2024-07-08T16:51:03.1969421Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-07-08T16:51:03.2000454Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2024-07-08T16:51:03.2012341Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-07-08T16:51:03.2043040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2024-07-08T16:51:03.2055380Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-07-08T16:51:03.2086264Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2024-07-08T16:51:03.2099877Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-07-08T16:51:03.2130028Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2024-07-08T16:51:03.2143550Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-07-08T16:51:03.2175528Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2024-07-08T16:51:03.2189790Z Entering 'third_party/mimalloc'
2024-07-08T16:51:03.2221111Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2024-07-08T16:51:03.2233887Z Entering 'third_party/nccl/nccl'
2024-07-08T16:51:03.2264959Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2024-07-08T16:51:03.2277135Z Entering 'third_party/nlohmann'
2024-07-08T16:51:03.2309130Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2024-07-08T16:51:03.2321428Z Entering 'third_party/onnx'
2024-07-08T16:51:03.2353056Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2024-07-08T16:51:03.2377717Z Entering 'third_party/onnx/third_party/benchmark'
2024-07-08T16:51:03.2408803Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2024-07-08T16:51:03.2420643Z Entering 'third_party/onnx/third_party/pybind11'
2024-07-08T16:51:03.2452081Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2024-07-08T16:51:03.2467605Z Entering 'third_party/opentelemetry-cpp'
2024-07-08T16:51:03.2499596Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2024-07-08T16:51:03.2513258Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-07-08T16:51:03.2544263Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2024-07-08T16:51:03.2557389Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-07-08T16:51:03.2588113Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2024-07-08T16:51:03.2601319Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-07-08T16:51:03.2633983Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2024-07-08T16:51:03.2644967Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-07-08T16:51:03.2676680Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2024-07-08T16:51:03.2691161Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-07-08T16:51:03.2722295Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2024-07-08T16:51:03.2734408Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-07-08T16:51:03.2763314Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2024-07-08T16:51:03.2774812Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-07-08T16:51:03.2805194Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2024-07-08T16:51:03.2816337Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-07-08T16:51:03.2847017Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2024-07-08T16:51:03.2860687Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-07-08T16:51:03.2892579Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2024-07-08T16:51:03.2906827Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-07-08T16:51:03.2937924Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2024-07-08T16:51:03.2966968Z Entering 'third_party/pocketfft'
2024-07-08T16:51:03.2997149Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2024-07-08T16:51:03.3010350Z Entering 'third_party/protobuf'
2024-07-08T16:51:03.3042030Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2024-07-08T16:51:03.3056530Z Entering 'third_party/protobuf/third_party/benchmark'
2024-07-08T16:51:03.3087472Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2024-07-08T16:51:03.3099772Z Entering 'third_party/protobuf/third_party/googletest'
2024-07-08T16:51:03.3131471Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2024-07-08T16:51:03.3145789Z Entering 'third_party/psimd'
2024-07-08T16:51:03.3178485Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2024-07-08T16:51:03.3190899Z Entering 'third_party/pthreadpool'
2024-07-08T16:51:03.3222001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2024-07-08T16:51:03.3235115Z Entering 'third_party/pybind11'
2024-07-08T16:51:03.3264913Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2024-07-08T16:51:03.3277914Z Entering 'third_party/python-peachpy'
2024-07-08T16:51:03.3309870Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2024-07-08T16:51:03.3322764Z Entering 'third_party/sleef'
2024-07-08T16:51:03.3353597Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2024-07-08T16:51:03.3367025Z Entering 'third_party/tensorpipe'
2024-07-08T16:51:03.3397603Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2024-07-08T16:51:03.3409095Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-07-08T16:51:03.3439586Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2024-07-08T16:51:03.3452727Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-07-08T16:51:03.3483517Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2024-07-08T16:51:03.3495222Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-07-08T16:51:03.3526697Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2024-07-08T16:51:03.3540513Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-07-08T16:51:03.3570880Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2024-07-08T16:51:03.3582432Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-07-08T16:51:03.3613410Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2024-07-08T16:51:03.4601355Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2024-07-08T16:51:03.4796121Z Entering 'android/libs/fbjni'
2024-07-08T16:51:03.4822143Z Entering 'third_party/FP16'
2024-07-08T16:51:03.4851492Z Entering 'third_party/FXdiv'
2024-07-08T16:51:03.4878189Z Entering 'third_party/NNPACK'
2024-07-08T16:51:03.4907358Z Entering 'third_party/VulkanMemoryAllocator'
2024-07-08T16:51:03.4937336Z Entering 'third_party/XNNPACK'
2024-07-08T16:51:03.4975722Z Entering 'third_party/benchmark'
2024-07-08T16:51:03.5001900Z Entering 'third_party/cpp-httplib'
2024-07-08T16:51:03.5030662Z Entering 'third_party/cpuinfo'
2024-07-08T16:51:03.5059736Z Entering 'third_party/cudnn_frontend'
2024-07-08T16:51:03.5085530Z Entering 'third_party/cutlass'
2024-07-08T16:51:03.5119401Z Entering 'third_party/eigen'
2024-07-08T16:51:03.5149992Z Entering 'third_party/fbgemm'
2024-07-08T16:51:03.5179067Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-07-08T16:51:03.5207984Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-07-08T16:51:03.5234819Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-07-08T16:51:03.5266704Z Entering 'third_party/fbgemm/third_party/googletest'
2024-07-08T16:51:03.5294120Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-07-08T16:51:03.5321921Z Entering 'third_party/flatbuffers'
2024-07-08T16:51:03.5353806Z Entering 'third_party/fmt'
2024-07-08T16:51:03.5383849Z Entering 'third_party/foxi'
2024-07-08T16:51:03.5410722Z Entering 'third_party/gemmlowp/gemmlowp'
2024-07-08T16:51:03.5438553Z Entering 'third_party/gloo'
2024-07-08T16:51:03.5469428Z Entering 'third_party/googletest'
2024-07-08T16:51:03.5495504Z Entering 'third_party/ideep'
2024-07-08T16:51:03.5522454Z Entering 'third_party/ideep/mkl-dnn'
2024-07-08T16:51:03.5555479Z Entering 'third_party/ittapi'
2024-07-08T16:51:03.5582739Z Entering 'third_party/kineto'
2024-07-08T16:51:03.5610373Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-07-08T16:51:03.5638279Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-07-08T16:51:03.5666654Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-07-08T16:51:03.5692527Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-07-08T16:51:03.5719659Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-07-08T16:51:03.5745148Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-07-08T16:51:03.5774990Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-07-08T16:51:03.5802088Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-07-08T16:51:03.5830287Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-07-08T16:51:03.5856918Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-07-08T16:51:03.5886923Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-07-08T16:51:03.5914006Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-07-08T16:51:03.5943379Z Entering 'third_party/mimalloc'
2024-07-08T16:51:03.5969644Z Entering 'third_party/nccl/nccl'
2024-07-08T16:51:03.5996722Z Entering 'third_party/nlohmann'
2024-07-08T16:51:03.6026290Z Entering 'third_party/onnx'
2024-07-08T16:51:03.6063295Z Entering 'third_party/onnx/third_party/benchmark'
2024-07-08T16:51:03.6090280Z Entering 'third_party/onnx/third_party/pybind11'
2024-07-08T16:51:03.6119027Z Entering 'third_party/opentelemetry-cpp'
2024-07-08T16:51:03.6147897Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-07-08T16:51:03.6175732Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-07-08T16:51:03.6203128Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-07-08T16:51:03.6231440Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-07-08T16:51:03.6260854Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-07-08T16:51:03.6285145Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-07-08T16:51:03.6313271Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-07-08T16:51:03.6340681Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-07-08T16:51:03.6368332Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-07-08T16:51:03.6395290Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-07-08T16:51:03.6437018Z Entering 'third_party/pocketfft'
2024-07-08T16:51:03.6467061Z Entering 'third_party/protobuf'
2024-07-08T16:51:03.6497753Z Entering 'third_party/protobuf/third_party/benchmark'
2024-07-08T16:51:03.6526205Z Entering 'third_party/protobuf/third_party/googletest'
2024-07-08T16:51:03.6553547Z Entering 'third_party/psimd'
2024-07-08T16:51:03.6582482Z Entering 'third_party/pthreadpool'
2024-07-08T16:51:03.6612159Z Entering 'third_party/pybind11'
2024-07-08T16:51:03.6639015Z Entering 'third_party/python-peachpy'
2024-07-08T16:51:03.6665735Z Entering 'third_party/sleef'
2024-07-08T16:51:03.6693679Z Entering 'third_party/tensorpipe'
2024-07-08T16:51:03.6721021Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-07-08T16:51:03.6749333Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-07-08T16:51:03.6775066Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-07-08T16:51:03.6800143Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-07-08T16:51:03.6827560Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-07-08T16:51:03.6864523Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2024-07-08T16:51:03.7068335Z Entering 'android/libs/fbjni'
2024-07-08T16:51:03.7098630Z Entering 'third_party/FP16'
2024-07-08T16:51:03.7125479Z Entering 'third_party/FXdiv'
2024-07-08T16:51:03.7153885Z Entering 'third_party/NNPACK'
2024-07-08T16:51:03.7182895Z Entering 'third_party/VulkanMemoryAllocator'
2024-07-08T16:51:03.7208494Z Entering 'third_party/XNNPACK'
2024-07-08T16:51:03.7246987Z Entering 'third_party/benchmark'
2024-07-08T16:51:03.7273076Z Entering 'third_party/cpp-httplib'
2024-07-08T16:51:03.7299647Z Entering 'third_party/cpuinfo'
2024-07-08T16:51:03.7327089Z Entering 'third_party/cudnn_frontend'
2024-07-08T16:51:03.7353898Z Entering 'third_party/cutlass'
2024-07-08T16:51:03.7385951Z Entering 'third_party/eigen'
2024-07-08T16:51:03.7415416Z Entering 'third_party/fbgemm'
2024-07-08T16:51:03.7443578Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-07-08T16:51:03.7470902Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-07-08T16:51:03.7497704Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-07-08T16:51:03.7530413Z Entering 'third_party/fbgemm/third_party/googletest'
2024-07-08T16:51:03.7556509Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-07-08T16:51:03.7585759Z Entering 'third_party/flatbuffers'
2024-07-08T16:51:03.7614247Z Entering 'third_party/fmt'
2024-07-08T16:51:03.7642367Z Entering 'third_party/foxi'
2024-07-08T16:51:03.7670453Z Entering 'third_party/gemmlowp/gemmlowp'
2024-07-08T16:51:03.7696723Z Entering 'third_party/gloo'
2024-07-08T16:51:03.7724609Z Entering 'third_party/googletest'
2024-07-08T16:51:03.7753889Z Entering 'third_party/ideep'
2024-07-08T16:51:03.7781039Z Entering 'third_party/ideep/mkl-dnn'
2024-07-08T16:51:03.7814362Z Entering 'third_party/ittapi'
2024-07-08T16:51:03.7841204Z Entering 'third_party/kineto'
2024-07-08T16:51:03.7870459Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-07-08T16:51:03.7896500Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-07-08T16:51:03.7923688Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-07-08T16:51:03.7951539Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-07-08T16:51:03.7977133Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-07-08T16:51:03.8004353Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-07-08T16:51:03.8033489Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-07-08T16:51:03.8062490Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-07-08T16:51:03.8089551Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-07-08T16:51:03.8116825Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-07-08T16:51:03.8145799Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-07-08T16:51:03.8172895Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-07-08T16:51:03.8202136Z Entering 'third_party/mimalloc'
2024-07-08T16:51:03.8231333Z Entering 'third_party/nccl/nccl'
2024-07-08T16:51:03.8258527Z Entering 'third_party/nlohmann'
2024-07-08T16:51:03.8285290Z Entering 'third_party/onnx'
2024-07-08T16:51:03.8326563Z Entering 'third_party/onnx/third_party/benchmark'
2024-07-08T16:51:03.8354927Z Entering 'third_party/onnx/third_party/pybind11'
2024-07-08T16:51:03.8384251Z Entering 'third_party/opentelemetry-cpp'
2024-07-08T16:51:03.8413856Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-07-08T16:51:03.8442141Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-07-08T16:51:03.8469545Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-07-08T16:51:03.8496296Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-07-08T16:51:03.8523975Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-07-08T16:51:03.8552018Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-07-08T16:51:03.8577952Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-07-08T16:51:03.8604414Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-07-08T16:51:03.8633114Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-07-08T16:51:03.8662779Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-07-08T16:51:03.8705271Z Entering 'third_party/pocketfft'
2024-07-08T16:51:03.8730966Z Entering 'third_party/protobuf'
2024-07-08T16:51:03.8761469Z Entering 'third_party/protobuf/third_party/benchmark'
2024-07-08T16:51:03.8786855Z Entering 'third_party/protobuf/third_party/googletest'
2024-07-08T16:51:03.8814211Z Entering 'third_party/psimd'
2024-07-08T16:51:03.8843470Z Entering 'third_party/pthreadpool'
2024-07-08T16:51:03.8871228Z Entering 'third_party/pybind11'
2024-07-08T16:51:03.8897909Z Entering 'third_party/python-peachpy'
2024-07-08T16:51:03.8926729Z Entering 'third_party/sleef'
2024-07-08T16:51:03.8955889Z Entering 'third_party/tensorpipe'
2024-07-08T16:51:03.8982973Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-07-08T16:51:03.9010819Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-07-08T16:51:03.9036287Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-07-08T16:51:03.9062595Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-07-08T16:51:03.9089664Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-07-08T16:51:03.9128415Z ##[endgroup]
2024-07-08T16:51:03.9155077Z [command]/usr/bin/git log -1 --format='%H'
2024-07-08T16:51:03.9176245Z 'c8ab2e8b637515b6488931f5e59f23848aae9991'
2024-07-08T16:51:03.9351106Z Prepare all required actions
2024-07-08T16:51:03.9351571Z Getting action download info
2024-07-08T16:51:04.0806468Z ##[group]Run ./.github/actions/setup-linux
2024-07-08T16:51:04.0806897Z env:
2024-07-08T16:51:04.0807185Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:04.0807539Z ##[endgroup]
2024-07-08T16:51:04.0866548Z ##[group]Run set -euo pipefail
2024-07-08T16:51:04.0866986Z [36;1mset -euo pipefail[0m
2024-07-08T16:51:04.0867373Z [36;1mfunction get_ec2_metadata() {[0m
2024-07-08T16:51:04.0867912Z [36;1m  # Pulled from instance metadata endpoint for EC2[0m
2024-07-08T16:51:04.0868795Z [36;1m  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html[0m
2024-07-08T16:51:04.0869556Z [36;1m  category=$1[0m
2024-07-08T16:51:04.0870076Z [36;1m  # If it is GCP runner (runner name contains gcp), do not run this[0m
2024-07-08T16:51:04.0870718Z [36;1m  runner_name_str=i-00726d0d1e8699347[0m
2024-07-08T16:51:04.0871211Z [36;1m  if [[ -f /.inarc ]]; then[0m
2024-07-08T16:51:04.0871724Z [36;1m    echo "ARC Runner, no info on ec2 metadata"[0m
2024-07-08T16:51:04.0872297Z [36;1m  elif [[ $runner_name_str == *"gcp"* ]]; then[0m
2024-07-08T16:51:04.0872992Z [36;1m    echo "Runner is from Google Cloud Platform, No info on ec2 metadata"[0m
2024-07-08T16:51:04.0873605Z [36;1m  else[0m
2024-07-08T16:51:04.0874108Z [36;1m    curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"[0m
2024-07-08T16:51:04.0874687Z [36;1m  fi[0m
2024-07-08T16:51:04.0874980Z [36;1m}[0m
2024-07-08T16:51:04.0875355Z [36;1mecho "ami-id: $(get_ec2_metadata ami-id)"[0m
2024-07-08T16:51:04.0875951Z [36;1mecho "instance-id: $(get_ec2_metadata instance-id)"[0m
2024-07-08T16:51:04.0876606Z [36;1mecho "instance-type: $(get_ec2_metadata instance-type)"[0m
2024-07-08T16:51:04.0877177Z [36;1mecho "system info $(uname -a)"[0m
2024-07-08T16:51:04.0885251Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:51:04.0885900Z env:
2024-07-08T16:51:04.0886190Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:04.0886542Z ##[endgroup]
2024-07-08T16:51:04.0953877Z ami-id: ami-0ce0c36d7a00b20e2
2024-07-08T16:51:04.0995143Z instance-id: i-00726d0d1e8699347
2024-07-08T16:51:04.1035493Z instance-type: g5.4xlarge
2024-07-08T16:51:04.1042496Z system info Linux ip-10-0-79-130.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
2024-07-08T16:51:04.1062785Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)"  >> $GITHUB_OUTPUT
2024-07-08T16:51:04.1063708Z [36;1mecho "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)"  >> $GITHUB_OUTPUT[0m
2024-07-08T16:51:04.1071564Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:51:04.1072060Z env:
2024-07-08T16:51:04.1072354Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:04.1072710Z ##[endgroup]
2024-07-08T16:51:04.1133167Z ##[group]Run if systemctl is-active --quiet docker; then
2024-07-08T16:51:04.1133778Z [36;1mif systemctl is-active --quiet docker; then[0m
2024-07-08T16:51:04.1134324Z [36;1m    echo "Docker daemon is running...";[0m
2024-07-08T16:51:04.1134887Z [36;1melse[0m
2024-07-08T16:51:04.1135379Z [36;1m    echo "Starting docker deamon..." && sudo systemctl start docker;[0m
2024-07-08T16:51:04.1135968Z [36;1mfi[0m
2024-07-08T16:51:04.1143388Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:51:04.1143880Z env:
2024-07-08T16:51:04.1144167Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:04.1144519Z ##[endgroup]
2024-07-08T16:51:04.1179279Z Docker daemon is running...
2024-07-08T16:51:04.1222768Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-07-08T16:51:04.1223324Z with:
2024-07-08T16:51:04.1223602Z   shell: bash
2024-07-08T16:51:04.1223896Z   timeout_minutes: 5
2024-07-08T16:51:04.1224225Z   max_attempts: 3
2024-07-08T16:51:04.1224545Z   retry_wait_seconds: 30
2024-07-08T16:51:04.1226034Z   command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
    --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"

2024-07-08T16:51:04.1227472Z   polling_interval_seconds: 1
2024-07-08T16:51:04.1227851Z   warning_on_retry: true
2024-07-08T16:51:04.1228200Z   continue_on_error: false
2024-07-08T16:51:04.1228536Z env:
2024-07-08T16:51:04.1228800Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:04.1229156Z   AWS_RETRY_MODE: standard
2024-07-08T16:51:04.1229504Z   AWS_MAX_ATTEMPTS: 5
2024-07-08T16:51:04.1229837Z   AWS_DEFAULT_REGION: us-east-1
2024-07-08T16:51:04.1230208Z ##[endgroup]
2024-07-08T16:51:04.9642958Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-07-08T16:51:04.9643986Z Configure a credential helper to remove this warning. See
2024-07-08T16:51:04.9646455Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-07-08T16:51:04.9647268Z
2024-07-08T16:51:04.9647477Z Login Succeeded
2024-07-08T16:51:05.1719827Z Command completed after 1 attempt(s).
2024-07-08T16:51:05.1760102Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2024-07-08T16:51:05.1760942Z [36;1menv | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"[0m
2024-07-08T16:51:05.1761607Z [36;1menv | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"[0m
2024-07-08T16:51:05.1769743Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:51:05.1770250Z env:
2024-07-08T16:51:05.1770541Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:05.1770895Z ##[endgroup]
2024-07-08T16:51:05.1829524Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-07-08T16:51:05.1830289Z [36;1m# ignore expansion of "docker ps -q" since it could be empty[0m
2024-07-08T16:51:05.1830889Z [36;1m# shellcheck disable=SC2046[0m
2024-07-08T16:51:05.1831363Z [36;1mdocker stop $(docker ps -q) || true[0m
2024-07-08T16:51:05.1831861Z [36;1m# Prune all of the docker images[0m
2024-07-08T16:51:05.1832318Z [36;1mdocker system prune -af[0m
2024-07-08T16:51:05.1839271Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:51:05.1839777Z env:
2024-07-08T16:51:05.1840053Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:05.1840393Z ##[endgroup]
2024-07-08T16:51:05.2066669Z "docker stop" requires at least 1 argument.
2024-07-08T16:51:05.2067306Z See 'docker stop --help'.
2024-07-08T16:51:05.2067553Z
2024-07-08T16:51:05.2067779Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2024-07-08T16:51:05.2068159Z
2024-07-08T16:51:05.2068307Z Stop one or more running containers
2024-07-08T16:51:05.2220120Z Total reclaimed space: 0B
2024-07-08T16:51:05.2250346Z ##[group]Run set +e
2024-07-08T16:51:05.2250699Z [36;1mset +e[0m
2024-07-08T16:51:05.2251000Z [36;1mset -x[0m
2024-07-08T16:51:05.2251300Z [36;1m[0m
2024-07-08T16:51:05.2251617Z [36;1mPT_DOMAIN=download.pytorch.org[0m
2024-07-08T16:51:05.2252416Z [36;1m# TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400,[0m
2024-07-08T16:51:05.2253486Z [36;1m# cleaning this up once the issue is fixed. There are more than one resolved IP here, the last[0m
2024-07-08T16:51:05.2254364Z [36;1m# one is returned at random[0m
2024-07-08T16:51:05.2254911Z [36;1mRESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1)[0m
2024-07-08T16:51:05.2255420Z [36;1m[0m
2024-07-08T16:51:05.2255751Z [36;1mif [ -z "${RESOLVED_IP}" ]; then[0m
2024-07-08T16:51:05.2256368Z [36;1m  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..."[0m
2024-07-08T16:51:05.2257133Z [36;1m  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1)[0m
2024-07-08T16:51:05.2257682Z [36;1m[0m
2024-07-08T16:51:05.2258019Z [36;1m  if [ -z "${RESOLVED_IP}" ]; then[0m
2024-07-08T16:51:05.2258577Z [36;1m    echo "Couldn't resolve ${PT_DOMAIN}, exiting..."[0m
2024-07-08T16:51:05.2259091Z [36;1m    exit 1[0m
2024-07-08T16:51:05.2259406Z [36;1m  fi[0m
2024-07-08T16:51:05.2259704Z [36;1mfi[0m
2024-07-08T16:51:05.2260078Z [36;1m[0m
2024-07-08T16:51:05.2260443Z [36;1mif grep -r "${PT_DOMAIN}" /etc/hosts; then[0m
2024-07-08T16:51:05.2260961Z [36;1m  # Clean up any old records first[0m
2024-07-08T16:51:05.2261478Z [36;1m  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts[0m
2024-07-08T16:51:05.2261943Z [36;1mfi[0m
2024-07-08T16:51:05.2262220Z [36;1m[0m
2024-07-08T16:51:05.2262660Z [36;1mecho "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts[0m
2024-07-08T16:51:05.2263213Z [36;1mcat /etc/hosts[0m
2024-07-08T16:51:05.2270823Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:51:05.2271321Z env:
2024-07-08T16:51:05.2271607Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:05.2271962Z ##[endgroup]
2024-07-08T16:51:05.2291414Z + PT_DOMAIN=download.pytorch.org
2024-07-08T16:51:05.2295085Z ++ dig -4 +short download.pytorch.org
2024-07-08T16:51:05.2295526Z ++ tail -n1
2024-07-08T16:51:05.2357507Z + RESOLVED_IP=18.160.10.22
2024-07-08T16:51:05.2357921Z + '[' -z 18.160.10.22 ']'
2024-07-08T16:51:05.2358358Z + grep -r download.pytorch.org /etc/hosts
2024-07-08T16:51:05.2363051Z 18.160.10.76 download.pytorch.org
2024-07-08T16:51:05.2363725Z + sudo sed -i /download.pytorch.org/d /etc/hosts
2024-07-08T16:51:05.2459814Z + echo '18.160.10.22 download.pytorch.org'
2024-07-08T16:51:05.2460516Z + sudo tee -a /etc/hosts
2024-07-08T16:51:05.2799514Z 18.160.10.22 download.pytorch.org
2024-07-08T16:51:05.2812764Z + cat /etc/hosts
2024-07-08T16:51:05.2817245Z 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
2024-07-08T16:51:05.2831770Z ::1         localhost6 localhost6.localdomain6
2024-07-08T16:51:05.2832283Z 18.160.10.22 download.pytorch.org
2024-07-08T16:51:05.3038193Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2024-07-08T16:51:05.3038796Z with:
2024-07-08T16:51:05.3039881Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:05.3041205Z   docker-build-dir: .ci/docker
2024-07-08T16:51:05.3041579Z   working-directory: .
2024-07-08T16:51:05.3042055Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-07-08T16:51:05.3042575Z   force-push: false
2024-07-08T16:51:05.3042873Z env:
2024-07-08T16:51:05.3043143Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:05.3043481Z ##[endgroup]
2024-07-08T16:51:05.3060993Z ##[group]Run set -ex
2024-07-08T16:51:05.3061352Z [36;1mset -ex[0m
2024-07-08T16:51:05.3061818Z [36;1m[0m
2024-07-08T16:51:05.3062374Z [36;1m# If the docker build directory or the build script doesn't exist, the action will[0m
2024-07-08T16:51:05.3063340Z [36;1m# gracefully return the docker image name as it is.  Pulling docker image in Linux[0m
2024-07-08T16:51:05.3064143Z [36;1m# job could then download the pre-built image as usual[0m
2024-07-08T16:51:05.3064886Z [36;1mif [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then[0m
2024-07-08T16:51:05.3065566Z [36;1m  echo "skip=true" >> "${GITHUB_OUTPUT}"[0m
2024-07-08T16:51:05.3066200Z [36;1m  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"[0m
2024-07-08T16:51:05.3066876Z [36;1m[0m
2024-07-08T16:51:05.3067386Z [36;1m  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."[0m
2024-07-08T16:51:05.3068009Z [36;1m  exit 0[0m
2024-07-08T16:51:05.3068315Z [36;1melse[0m
2024-07-08T16:51:05.3068695Z [36;1m  echo "skip=false" >> "${GITHUB_OUTPUT}"[0m
2024-07-08T16:51:05.3069152Z [36;1mfi[0m
2024-07-08T16:51:05.3069440Z [36;1m[0m
2024-07-08T16:51:05.3069923Z [36;1mif [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then[0m
2024-07-08T16:51:05.3070773Z [36;1m  # The docker image name already includes the ECR prefix and tag, so we can just[0m
2024-07-08T16:51:05.3071546Z [36;1m  # use it as it is, but first let's extract the tag[0m
2024-07-08T16:51:05.3072255Z [36;1m  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')[0m
2024-07-08T16:51:05.3072985Z [36;1m  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"[0m
2024-07-08T16:51:05.3073697Z [36;1m  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"[0m
2024-07-08T16:51:05.3074261Z [36;1melse[0m
2024-07-08T16:51:05.3074710Z [36;1m  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")[0m
2024-07-08T16:51:05.3075382Z [36;1m  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"[0m
2024-07-08T16:51:05.3076277Z [36;1m  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"[0m
2024-07-08T16:51:05.3077027Z [36;1mfi[0m
2024-07-08T16:51:05.3084626Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:51:05.3085130Z env:
2024-07-08T16:51:05.3085413Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:05.3086221Z   REPO_NAME: pytorch
2024-07-08T16:51:05.3087389Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:05.3088591Z   DOCKER_BUILD_DIR: .ci/docker
2024-07-08T16:51:05.3089106Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-07-08T16:51:05.3089630Z ##[endgroup]
2024-07-08T16:51:05.3109212Z + [[ ! -d .ci/docker ]]
2024-07-08T16:51:05.3109654Z + [[ ! -f .ci/docker/build.sh ]]
2024-07-08T16:51:05.3110045Z + echo skip=false
2024-07-08T16:51:05.3112108Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2024-07-08T16:51:05.3115899Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:05.3117651Z ++ awk -F '[:,]' '{print $2}'
2024-07-08T16:51:05.3121002Z + DOCKER_TAG=52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:05.3121689Z + echo docker-tag=52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:05.3123390Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:05.3150353Z ##[group]Run set +e
2024-07-08T16:51:05.3150726Z [36;1mset +e[0m
2024-07-08T16:51:05.3151035Z [36;1mset -x[0m
2024-07-08T16:51:05.3151327Z [36;1m[0m
2024-07-08T16:51:05.3151616Z [36;1mlogin() {[0m
2024-07-08T16:51:05.3152288Z [36;1m  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"[0m
2024-07-08T16:51:05.3153020Z [36;1m}[0m
2024-07-08T16:51:05.3153296Z [36;1m[0m
2024-07-08T16:51:05.3153581Z [36;1mretry () {[0m
2024-07-08T16:51:05.3153986Z [36;1m  $*  || (sleep 1 && $*) || (sleep 2 && $*)[0m
2024-07-08T16:51:05.3154443Z [36;1m}[0m
2024-07-08T16:51:05.3154724Z [36;1m[0m
2024-07-08T16:51:05.3155048Z [36;1mretry login "${DOCKER_REGISTRY}"[0m
2024-07-08T16:51:05.3155472Z [36;1m[0m
2024-07-08T16:51:05.3156053Z [36;1m# Check if image already exists, if it does then skip building it[0m
2024-07-08T16:51:05.3156763Z [36;1mif docker manifest inspect "${DOCKER_IMAGE}"; then[0m
2024-07-08T16:51:05.3157279Z [36;1m  exit 0[0m
2024-07-08T16:51:05.3157584Z [36;1mfi[0m
2024-07-08T16:51:05.3157862Z [36;1m[0m
2024-07-08T16:51:05.3158409Z [36;1m# NB: This part requires a full checkout. Otherwise, the merge base will[0m
2024-07-08T16:51:05.3159257Z [36;1m# be empty.  The default action would be to continue rebuild the image[0m
2024-07-08T16:51:05.3160003Z [36;1mif [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then[0m
2024-07-08T16:51:05.3160672Z [36;1m  # if we're on the base branch then use the parent commit[0m
2024-07-08T16:51:05.3161403Z [36;1m  MERGE_BASE=$(git rev-parse HEAD~)[0m
2024-07-08T16:51:05.3161843Z [36;1melse[0m
2024-07-08T16:51:05.3162320Z [36;1m  # otherwise we're on a PR, so use the most recent base commit[0m
2024-07-08T16:51:05.3163008Z [36;1m  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")[0m
2024-07-08T16:51:05.3163535Z [36;1mfi[0m
2024-07-08T16:51:05.3163826Z [36;1m[0m
2024-07-08T16:51:05.3164156Z [36;1mif [[ -z "${MERGE_BASE}" ]]; then[0m
2024-07-08T16:51:05.3164655Z [36;1m  echo "rebuild=true" >> "${GITHUB_OUTPUT}"[0m
2024-07-08T16:51:05.3165120Z [36;1m[0m
2024-07-08T16:51:05.3166006Z [36;1m  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."[0m
2024-07-08T16:51:05.3166794Z [36;1m  exit 0[0m
2024-07-08T16:51:05.3167103Z [36;1mfi[0m
2024-07-08T16:51:05.3167392Z [36;1m[0m
2024-07-08T16:51:05.3167850Z [36;1mif ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then[0m
2024-07-08T16:51:05.3168826Z [36;1m  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"[0m
2024-07-08T16:51:05.3169620Z [36;1m  exit 1[0m
2024-07-08T16:51:05.3169932Z [36;1mfi[0m
2024-07-08T16:51:05.3170218Z [36;1m[0m
2024-07-08T16:51:05.3170714Z [36;1mPREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")[0m
2024-07-08T16:51:05.3171664Z [36;1m# If no image exists but the hash is the same as the previous hash then we should error out here[0m
2024-07-08T16:51:05.3172552Z [36;1mif [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then[0m
2024-07-08T16:51:05.3173519Z [36;1m  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"[0m
2024-07-08T16:51:05.3174620Z [36;1m  echo "         Will re-build docker image to store in local cache, TTS may be longer"[0m
2024-07-08T16:51:05.3175374Z [36;1mfi[0m
2024-07-08T16:51:05.3175661Z [36;1m[0m
2024-07-08T16:51:05.3176030Z [36;1mecho "rebuild=true" >> "${GITHUB_OUTPUT}"[0m
2024-07-08T16:51:05.3183661Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:51:05.3184165Z env:
2024-07-08T16:51:05.3184437Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:05.3184814Z   DOCKER_BUILD_DIR: .ci/docker
2024-07-08T16:51:05.3185283Z   BASE_REVISION: c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:51:05.3186560Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:05.3187812Z   DOCKER_TAG: 52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:05.3188429Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-07-08T16:51:05.3188955Z ##[endgroup]
2024-07-08T16:51:05.3211192Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-07-08T16:51:05.3211856Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-07-08T16:51:05.3213316Z + aws ecr get-login-password --region us-east-1
2024-07-08T16:51:05.3214087Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-07-08T16:51:05.7331549Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-07-08T16:51:05.7332364Z Configure a credential helper to remove this warning. See
2024-07-08T16:51:05.7333366Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-07-08T16:51:05.7333876Z
2024-07-08T16:51:05.7334005Z Login Succeeded
2024-07-08T16:51:05.7344856Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:05.9820778Z {
2024-07-08T16:51:05.9821156Z 	"schemaVersion": 2,
2024-07-08T16:51:05.9821874Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2024-07-08T16:51:05.9822585Z 	"config": {
2024-07-08T16:51:05.9823079Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2024-07-08T16:51:05.9823636Z 		"size": 48700,
2024-07-08T16:51:05.9824174Z 		"digest": "sha256:6cbb2e66e2081153c16c8c4674e997a9c177a07ce94b8ce56e51c1f9cd5e49ea"
2024-07-08T16:51:05.9824821Z 	},
2024-07-08T16:51:05.9825182Z 	"layers": [
2024-07-08T16:51:05.9825555Z 		{
2024-07-08T16:51:05.9826079Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9826876Z 			"size": 28580681,
2024-07-08T16:51:05.9827493Z 			"digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a"
2024-07-08T16:51:05.9828116Z 		},
2024-07-08T16:51:05.9828356Z 		{
2024-07-08T16:51:05.9828792Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9829434Z 			"size": 7943451,
2024-07-08T16:51:05.9830172Z 			"digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d"
2024-07-08T16:51:05.9830999Z 		},
2024-07-08T16:51:05.9831354Z 		{
2024-07-08T16:51:05.9831948Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9832991Z 			"size": 55728572,
2024-07-08T16:51:05.9833767Z 			"digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450"
2024-07-08T16:51:05.9834663Z + exit 0
2024-07-08T16:51:05.9835027Z 		},
2024-07-08T16:51:05.9835345Z 		{
2024-07-08T16:51:05.9835937Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9836679Z 			"size": 186,
2024-07-08T16:51:05.9837381Z 			"digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195"
2024-07-08T16:51:05.9838280Z 		},
2024-07-08T16:51:05.9838661Z 		{
2024-07-08T16:51:05.9839300Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9840137Z 			"size": 6886,
2024-07-08T16:51:05.9841049Z 			"digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80"
2024-07-08T16:51:05.9842244Z 		},
2024-07-08T16:51:05.9842697Z 		{
2024-07-08T16:51:05.9843363Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9844209Z 			"size": 1329001756,
2024-07-08T16:51:05.9845033Z 			"digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108"
2024-07-08T16:51:05.9846331Z 		},
2024-07-08T16:51:05.9846701Z 		{
2024-07-08T16:51:05.9847354Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9848168Z 			"size": 62501,
2024-07-08T16:51:05.9848776Z 			"digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198"
2024-07-08T16:51:05.9849401Z 		},
2024-07-08T16:51:05.9849650Z 		{
2024-07-08T16:51:05.9850097Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9850783Z 			"size": 1684,
2024-07-08T16:51:05.9851330Z 			"digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a"
2024-07-08T16:51:05.9851935Z 		},
2024-07-08T16:51:05.9852200Z 		{
2024-07-08T16:51:05.9852645Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9853193Z 			"size": 1523,
2024-07-08T16:51:05.9853760Z 			"digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1"
2024-07-08T16:51:05.9854400Z 		},
2024-07-08T16:51:05.9854662Z 		{
2024-07-08T16:51:05.9855099Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9855807Z 			"size": 2528295403,
2024-07-08T16:51:05.9856388Z 			"digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5"
2024-07-08T16:51:05.9857008Z 		},
2024-07-08T16:51:05.9857262Z 		{
2024-07-08T16:51:05.9857709Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9858273Z 			"size": 86016,
2024-07-08T16:51:05.9858832Z 			"digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b"
2024-07-08T16:51:05.9859457Z 		},
2024-07-08T16:51:05.9859721Z 		{
2024-07-08T16:51:05.9860170Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9860714Z 			"size": 1820,
2024-07-08T16:51:05.9861255Z 			"digest": "sha256:2500215611d5758c06825a0a9efd3cf2a94157786f1fb66368142514d05e818a"
2024-07-08T16:51:05.9861858Z 		},
2024-07-08T16:51:05.9862118Z 		{
2024-07-08T16:51:05.9862547Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9863106Z 			"size": 246735740,
2024-07-08T16:51:05.9863688Z 			"digest": "sha256:e392141629a352bc998c7f30b09f3dcefaba9325a0d56481a9f39b07b3ef043a"
2024-07-08T16:51:05.9864310Z 		},
2024-07-08T16:51:05.9864560Z 		{
2024-07-08T16:51:05.9865006Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9865572Z 			"size": 544,
2024-07-08T16:51:05.9866111Z 			"digest": "sha256:60a6b13946fd71c99c7555039bf976d4c76d237e4d5e527955deecccb093340e"
2024-07-08T16:51:05.9866729Z 		},
2024-07-08T16:51:05.9866993Z 		{
2024-07-08T16:51:05.9867438Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9867988Z 			"size": 1281,
2024-07-08T16:51:05.9868555Z 			"digest": "sha256:d2c0e45285aec8c7cd21ba22540c0a8e63dae9674502e56c7ca9debeb5e51e18"
2024-07-08T16:51:05.9869182Z 		},
2024-07-08T16:51:05.9869443Z 		{
2024-07-08T16:51:05.9869875Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9870434Z 			"size": 480,
2024-07-08T16:51:05.9870985Z 			"digest": "sha256:9a09c9d2f87071f644e8b5b8724ab22ea6109df7495bc37b6b44e407fdf9364d"
2024-07-08T16:51:05.9871602Z 		},
2024-07-08T16:51:05.9871864Z 		{
2024-07-08T16:51:05.9872303Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9872871Z 			"size": 91711742,
2024-07-08T16:51:05.9873430Z 			"digest": "sha256:cad71e8326b7605e491fed6b297ce9e6794240e6af9dc37016c55890dc840c55"
2024-07-08T16:51:05.9874048Z 		},
2024-07-08T16:51:05.9874309Z 		{
2024-07-08T16:51:05.9874755Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9875393Z 			"size": 3234,
2024-07-08T16:51:05.9875950Z 			"digest": "sha256:73516a730d124ac0c9d25e74f7a093fca6a12d1ab68bc908b3fbd3bbf1bce74b"
2024-07-08T16:51:05.9876578Z 		},
2024-07-08T16:51:05.9876844Z 		{
2024-07-08T16:51:05.9877273Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9877825Z 			"size": 1903,
2024-07-08T16:51:05.9878374Z 			"digest": "sha256:3aef13fe65250278425f5fe3753041806a9638b9b11eebd9573019ed8f87006a"
2024-07-08T16:51:05.9878971Z 		},
2024-07-08T16:51:05.9879238Z 		{
2024-07-08T16:51:05.9879694Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9880264Z 			"size": 698,
2024-07-08T16:51:05.9880914Z 			"digest": "sha256:58051158c0cbc3107d8e370aa25ee334552b1b8d6000c4fbfd1131560976fda2"
2024-07-08T16:51:05.9881540Z 		},
2024-07-08T16:51:05.9881799Z 		{
2024-07-08T16:51:05.9882239Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9882785Z 			"size": 2780134558,
2024-07-08T16:51:05.9883364Z 			"digest": "sha256:980bc6bf56d39e2c42fe2dea37c6b12da6949f1d8b4a1400f1c31d0849d76227"
2024-07-08T16:51:05.9883986Z 		},
2024-07-08T16:51:05.9884241Z 		{
2024-07-08T16:51:05.9884693Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9885266Z 			"size": 379,
2024-07-08T16:51:05.9886028Z 			"digest": "sha256:4bbd66984c291bd37777cf7789d2713ecba43abcde7bf1f21b4ddbc90fd5c335"
2024-07-08T16:51:05.9886748Z 		},
2024-07-08T16:51:05.9887017Z 		{
2024-07-08T16:51:05.9887478Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9888055Z 			"size": 11879,
2024-07-08T16:51:05.9888605Z 			"digest": "sha256:08c10256971903f54da2a24ff00659f5228e857cd373af7546337ec9672b3c04"
2024-07-08T16:51:05.9889230Z 		},
2024-07-08T16:51:05.9889512Z 		{
2024-07-08T16:51:05.9889968Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9890704Z 			"size": 803,
2024-07-08T16:51:05.9891284Z 			"digest": "sha256:ae32e42b591398cc08d31cea0fda33dabcde703c32c00e911d3543079e366512"
2024-07-08T16:51:05.9891940Z 		},
2024-07-08T16:51:05.9892197Z 		{
2024-07-08T16:51:05.9892654Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9893209Z 			"size": 106,
2024-07-08T16:51:05.9893746Z 			"digest": "sha256:68354870b7a062846bb7238f964c02731ed78725f676c5fce6cef37a56c39881"
2024-07-08T16:51:05.9894339Z 		},
2024-07-08T16:51:05.9894608Z 		{
2024-07-08T16:51:05.9895051Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9895603Z 			"size": 503,
2024-07-08T16:51:05.9896125Z 			"digest": "sha256:61b35c224f83587fae9233bc9ab3a259406a1e11091301ac431f3417689eda6b"
2024-07-08T16:51:05.9896737Z 		},
2024-07-08T16:51:05.9897004Z 		{
2024-07-08T16:51:05.9897447Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9898021Z 			"size": 121477159,
2024-07-08T16:51:05.9898676Z 			"digest": "sha256:7c79b55aab49d00c4034a76e1c7adce6366a28dabcd2597f8aa2bf0a9bab7a36"
2024-07-08T16:51:05.9899313Z 		},
2024-07-08T16:51:05.9899568Z 		{
2024-07-08T16:51:05.9900024Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9900596Z 			"size": 109,
2024-07-08T16:51:05.9901152Z 			"digest": "sha256:97f2ee2e8876f533f00ffa73c7c6e3e07ff46547313e22e3a498d5e68c6d0491"
2024-07-08T16:51:05.9901762Z 		},
2024-07-08T16:51:05.9902036Z 		{
2024-07-08T16:51:05.9902481Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9903034Z 			"size": 488,
2024-07-08T16:51:05.9903580Z 			"digest": "sha256:f26b1339c7341a19fdb8235c3151940a02a09b7285fc276984b9b99e66af207f"
2024-07-08T16:51:05.9904191Z 		},
2024-07-08T16:51:05.9904453Z 		{
2024-07-08T16:51:05.9904877Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9905437Z 			"size": 296,
2024-07-08T16:51:05.9905983Z 			"digest": "sha256:a938af61c80579a1e709b9dc55da6ee0bef6c40f59a77e05ea69d3f073e51d49"
2024-07-08T16:51:05.9906683Z 		},
2024-07-08T16:51:05.9906933Z 		{
2024-07-08T16:51:05.9907369Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9907926Z 			"size": 103,
2024-07-08T16:51:05.9908474Z 			"digest": "sha256:c2d76dae598c19b172bedea619c30e8824690ba4cc004bd57ac5359f59b2b49e"
2024-07-08T16:51:05.9909136Z 		},
2024-07-08T16:51:05.9909394Z 		{
2024-07-08T16:51:05.9909829Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9910381Z 			"size": 1471,
2024-07-08T16:51:05.9910915Z 			"digest": "sha256:07a096d308549868371eee718dffbb151596525722a6da6e78785989861e5f4c"
2024-07-08T16:51:05.9911517Z 		},
2024-07-08T16:51:05.9911773Z 		{
2024-07-08T16:51:05.9912199Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9912751Z 			"size": 446890143,
2024-07-08T16:51:05.9913307Z 			"digest": "sha256:9617411c9c34c85541a851c657fc6543ddc7693fed31eebd102da2e4b9ebf015"
2024-07-08T16:51:05.9913909Z 		},
2024-07-08T16:51:05.9914162Z 		{
2024-07-08T16:51:05.9914597Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9915149Z 			"size": 160,
2024-07-08T16:51:05.9915691Z 			"digest": "sha256:fcaf380e209a44e733b12588ab1199dd891a0b214fbbfaa4b35cce50a72ffd89"
2024-07-08T16:51:05.9916296Z 		},
2024-07-08T16:51:05.9916543Z 		{
2024-07-08T16:51:05.9916971Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9917601Z 			"size": 563,
2024-07-08T16:51:05.9918129Z 			"digest": "sha256:335862b6874147290c1f9253299cff44aa5ec60ebf193ac66d99e8cd04f3a0b9"
2024-07-08T16:51:05.9918738Z 		},
2024-07-08T16:51:05.9918990Z 		{
2024-07-08T16:51:05.9919418Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9919970Z 			"size": 35874454,
2024-07-08T16:51:05.9920540Z 			"digest": "sha256:f4973d9e8cd610fbf370bcae1f2e80607ef3e5255e2f2a61fc2d68030fa13592"
2024-07-08T16:51:05.9921242Z 		},
2024-07-08T16:51:05.9921491Z 		{
2024-07-08T16:51:05.9921934Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9922490Z 			"size": 104,
2024-07-08T16:51:05.9923019Z 			"digest": "sha256:5b44615090ccf9ad56f77de7f3efc605931728b69914e9d1a1d4d94ab096dfa0"
2024-07-08T16:51:05.9923629Z 		},
2024-07-08T16:51:05.9923882Z 		{
2024-07-08T16:51:05.9924319Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9924867Z 			"size": 423,
2024-07-08T16:51:05.9925418Z 			"digest": "sha256:dc088c962a355886f55a959c7c0e4a9eeb5e88b57a405d1eeceec04719e7c5a4"
2024-07-08T16:51:05.9926211Z 		},
2024-07-08T16:51:05.9926464Z 		{
2024-07-08T16:51:05.9926891Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9927441Z 			"size": 20262062,
2024-07-08T16:51:05.9928007Z 			"digest": "sha256:0f0eedef185df7996ba4c40eee4a6eb434e19312ab9d8a12006885afe9ae4a7c"
2024-07-08T16:51:05.9928629Z 		},
2024-07-08T16:51:05.9928874Z 		{
2024-07-08T16:51:05.9929308Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9929856Z 			"size": 437,
2024-07-08T16:51:05.9930370Z 			"digest": "sha256:815f7a234f8172bf60220cf8452157ae8f2676a065ff812b3a768b9937423409"
2024-07-08T16:51:05.9930969Z 		},
2024-07-08T16:51:05.9931217Z 		{
2024-07-08T16:51:05.9931643Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9932180Z 			"size": 698,
2024-07-08T16:51:05.9932709Z 			"digest": "sha256:58051158c0cbc3107d8e370aa25ee334552b1b8d6000c4fbfd1131560976fda2"
2024-07-08T16:51:05.9933319Z 		},
2024-07-08T16:51:05.9933574Z 		{
2024-07-08T16:51:05.9934003Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9934547Z 			"size": 142,
2024-07-08T16:51:05.9935079Z 			"digest": "sha256:2be212e55519305eb7dde42982dd940790824ce7f89aa5a88f2f7bd231833db7"
2024-07-08T16:51:05.9935688Z 		},
2024-07-08T16:51:05.9935934Z 		{
2024-07-08T16:51:05.9936370Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9937017Z 			"size": 136,
2024-07-08T16:51:05.9937563Z 			"digest": "sha256:e01cceedac021b8393b030c76fad084843fee5a8dc57f4516ee1ca0e3cdf7d54"
2024-07-08T16:51:05.9938233Z 		},
2024-07-08T16:51:05.9938484Z 		{
2024-07-08T16:51:05.9938909Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9939447Z 			"size": 4912026187,
2024-07-08T16:51:05.9940001Z 			"digest": "sha256:5aa920a2a3625e56c20900d7a2e3e012c80fe9f47f083ec73731a592213277a7"
2024-07-08T16:51:05.9940603Z 		},
2024-07-08T16:51:05.9940853Z 		{
2024-07-08T16:51:05.9941274Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9941827Z 			"size": 191,
2024-07-08T16:51:05.9942352Z 			"digest": "sha256:277d6ed6a725f86728a2c88caf77b869449740395153933a6196bc6b058b6826"
2024-07-08T16:51:05.9942939Z 		},
2024-07-08T16:51:05.9943193Z 		{
2024-07-08T16:51:05.9943618Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9944165Z 			"size": 564,
2024-07-08T16:51:05.9944700Z 			"digest": "sha256:238bb91c34c5e4a59df803fcfdff53451e7157483b52dfe34a057c7d6d5614a9"
2024-07-08T16:51:05.9945310Z 		},
2024-07-08T16:51:05.9945561Z 		{
2024-07-08T16:51:05.9945995Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9946539Z 			"size": 43163075,
2024-07-08T16:51:05.9947086Z 			"digest": "sha256:d12781f845588c6e53f9bd073ec643a8f4692b3284c46288b5a7bb65bcf126e0"
2024-07-08T16:51:05.9947756Z 		},
2024-07-08T16:51:05.9947998Z 		{
2024-07-08T16:51:05.9948433Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9948981Z 			"size": 106,
2024-07-08T16:51:05.9949515Z 			"digest": "sha256:0da1d2512da882ac68cb4f462c7e1337283c15840a110df6af727808eb5472ee"
2024-07-08T16:51:05.9950111Z 		},
2024-07-08T16:51:05.9950360Z 		{
2024-07-08T16:51:05.9950787Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9951335Z 			"size": 1210,
2024-07-08T16:51:05.9951877Z 			"digest": "sha256:c5cc709139b86630c136f6d5aa8178ca1770f1fe92f6761b5e4ead662470304e"
2024-07-08T16:51:05.9952482Z 		},
2024-07-08T16:51:05.9952735Z 		{
2024-07-08T16:51:05.9953169Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9953727Z 			"size": 698,
2024-07-08T16:51:05.9954260Z 			"digest": "sha256:58051158c0cbc3107d8e370aa25ee334552b1b8d6000c4fbfd1131560976fda2"
2024-07-08T16:51:05.9954855Z 		},
2024-07-08T16:51:05.9955104Z 		{
2024-07-08T16:51:05.9955531Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9956078Z 			"size": 139,
2024-07-08T16:51:05.9956607Z 			"digest": "sha256:93f1924e1938f8b5c5c35f45820a33d3896b2fe31860c479af02450cd8d92abf"
2024-07-08T16:51:05.9957195Z 		},
2024-07-08T16:51:05.9957450Z 		{
2024-07-08T16:51:05.9957878Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9958468Z 			"size": 118,
2024-07-08T16:51:05.9959010Z 			"digest": "sha256:d5331131aafd8107cdbd9ded5ead70e5b8bc126d11557c3fd0ee8b8381a8bf6b"
2024-07-08T16:51:05.9959635Z 		},
2024-07-08T16:51:05.9959883Z 		{
2024-07-08T16:51:05.9960313Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9960946Z 			"size": 1922508692,
2024-07-08T16:51:05.9961505Z 			"digest": "sha256:9157dbafa6bce3d9a6483359ba90f25885488ccc978053204ce1649669adcf96"
2024-07-08T16:51:05.9962114Z 		},
2024-07-08T16:51:05.9962360Z 		{
2024-07-08T16:51:05.9962796Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9963344Z 			"size": 173,
2024-07-08T16:51:05.9963889Z 			"digest": "sha256:0c2f066daef8df477f67ef61462db1ce4f1a0c216a5726b824b3652fcc7c94fe"
2024-07-08T16:51:05.9964497Z 		},
2024-07-08T16:51:05.9964751Z 		{
2024-07-08T16:51:05.9965181Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9965898Z 			"size": 906,
2024-07-08T16:51:05.9966451Z 			"digest": "sha256:bf4cdb0324374c09a6a6618dbf06bfbf1c91b3e06f1399205c6ffbe1f2550601"
2024-07-08T16:51:05.9967162Z 		},
2024-07-08T16:51:05.9967421Z 		{
2024-07-08T16:51:05.9967843Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9968445Z 			"size": 698,
2024-07-08T16:51:05.9968975Z 			"digest": "sha256:58051158c0cbc3107d8e370aa25ee334552b1b8d6000c4fbfd1131560976fda2"
2024-07-08T16:51:05.9969578Z 		},
2024-07-08T16:51:05.9969822Z 		{
2024-07-08T16:51:05.9970253Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9970806Z 			"size": 135,
2024-07-08T16:51:05.9971334Z 			"digest": "sha256:16f104223c35d4e53afc77729a6984ff740ecea2217078a32913437e14b822fb"
2024-07-08T16:51:05.9971923Z 		},
2024-07-08T16:51:05.9972173Z 		{
2024-07-08T16:51:05.9972605Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9973159Z 			"size": 32,
2024-07-08T16:51:05.9973705Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-07-08T16:51:05.9974321Z 		},
2024-07-08T16:51:05.9974586Z 		{
2024-07-08T16:51:05.9975013Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9975571Z 			"size": 156,
2024-07-08T16:51:05.9976104Z 			"digest": "sha256:01d06d370ea464526737501928b7c647ef4e4d9110eed487a898cc28eb1d56aa"
2024-07-08T16:51:05.9976702Z 		},
2024-07-08T16:51:05.9976943Z 		{
2024-07-08T16:51:05.9977383Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9978018Z 			"size": 1839,
2024-07-08T16:51:05.9978542Z 			"digest": "sha256:514e1f285315445de65858389610d03507edcd16d592103d9f94c20f93adf7b6"
2024-07-08T16:51:05.9979131Z 		},
2024-07-08T16:51:05.9979382Z 		{
2024-07-08T16:51:05.9979817Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9980361Z 			"size": 7529783,
2024-07-08T16:51:05.9980921Z 			"digest": "sha256:e83fe93022c44063b37e9b7f0710cb2d94ff0da9291ff50fba2f3093302d61f4"
2024-07-08T16:51:05.9981549Z 		},
2024-07-08T16:51:05.9981804Z 		{
2024-07-08T16:51:05.9982246Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9982793Z 			"size": 162,
2024-07-08T16:51:05.9983323Z 			"digest": "sha256:729564988a365999967d2838602e51d6fb28ecdbb398330f0259afbee699b9f3"
2024-07-08T16:51:05.9983917Z 		},
2024-07-08T16:51:05.9984164Z 		{
2024-07-08T16:51:05.9984603Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9985161Z 			"size": 7940,
2024-07-08T16:51:05.9985727Z 			"digest": "sha256:19e8b19d9b3e657e1a331be4763e8dfc28af216701dbab8cd9bce0d9149e6668"
2024-07-08T16:51:05.9986344Z 		},
2024-07-08T16:51:05.9986603Z 		{
2024-07-08T16:51:05.9987042Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9987595Z 			"size": 8070,
2024-07-08T16:51:05.9988144Z 			"digest": "sha256:930584c906245fc6a5e96327cb0d48384d6b95b04bca6c728c14d4023d34b99d"
2024-07-08T16:51:05.9988752Z 		},
2024-07-08T16:51:05.9989004Z 		{
2024-07-08T16:51:05.9989440Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9990002Z 			"size": 302,
2024-07-08T16:51:05.9990546Z 			"digest": "sha256:44b0076546be3d37af8c5262d7c05dc73cf867e37d54250bf3d6c727f104f1a0"
2024-07-08T16:51:05.9991167Z 		},
2024-07-08T16:51:05.9991417Z 		{
2024-07-08T16:51:05.9991861Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9992423Z 			"size": 7629102,
2024-07-08T16:51:05.9992971Z 			"digest": "sha256:10ce8b9cd377e74bdf575c1424c82a5c7711368adaf20bf5c0853b54f2b44603"
2024-07-08T16:51:05.9993602Z 		},
2024-07-08T16:51:05.9993861Z 		{
2024-07-08T16:51:05.9994302Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9994854Z 			"size": 108,
2024-07-08T16:51:05.9995422Z 			"digest": "sha256:e2feb35cbe3f045d7dcdec46a5cc99e5aea0389b0513fb2a1638443a1cf28fb0"
2024-07-08T16:51:05.9996053Z 		},
2024-07-08T16:51:05.9996308Z 		{
2024-07-08T16:51:05.9996730Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9997342Z 			"size": 54145774,
2024-07-08T16:51:05.9997907Z 			"digest": "sha256:fdf9f04399544a2e09e12c6a0373212b59ddb10463c34124f298b0b06b84352b"
2024-07-08T16:51:05.9998530Z 		},
2024-07-08T16:51:05.9998819Z 		{
2024-07-08T16:51:05.9999267Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:05.9999832Z 			"size": 470,
2024-07-08T16:51:06.0000367Z 			"digest": "sha256:35290ca2e6025bd3c70880230f46317ef8432db0669dc56ca8d586d7cabf67ad"
2024-07-08T16:51:06.0001060Z 		},
2024-07-08T16:51:06.0001328Z 		{
2024-07-08T16:51:06.0001784Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:06.0002349Z 			"size": 1374858708,
2024-07-08T16:51:06.0002950Z 			"digest": "sha256:ec7e4facda86a415dfdba6a01ff5465e3da81d7a4261a465357aaaadddd3a79d"
2024-07-08T16:51:06.0003605Z 		},
2024-07-08T16:51:06.0003870Z 		{
2024-07-08T16:51:06.0004307Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:06.0004888Z 			"size": 106,
2024-07-08T16:51:06.0005474Z 			"digest": "sha256:af1dbfbab64ab46793e4539f2ccd79f431abc54a13122b2ee0c3337b98a7548a"
2024-07-08T16:51:06.0006287Z 		},
2024-07-08T16:51:06.0006550Z 		{
2024-07-08T16:51:06.0007008Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:06.0007583Z 			"size": 558,
2024-07-08T16:51:06.0008142Z 			"digest": "sha256:44afb2fb888d0ee52f686345f5313ff6bfc7b528ae763dc62885bda21466817c"
2024-07-08T16:51:06.0008959Z 		},
2024-07-08T16:51:06.0009224Z 		{
2024-07-08T16:51:06.0009675Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:06.0010225Z 			"size": 46248562,
2024-07-08T16:51:06.0010795Z 			"digest": "sha256:055b36a703f024c568b98dc39c05a811af62e231ee4509e425bb16fd9352e0da"
2024-07-08T16:51:06.0011415Z 		},
2024-07-08T16:51:06.0011676Z 		{
2024-07-08T16:51:06.0012120Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:06.0012679Z 			"size": 111,
2024-07-08T16:51:06.0013229Z 			"digest": "sha256:1cb19548583ae345ef49ebe0ddaafa9a00f26099333d6a21e580ddbc19c46874"
2024-07-08T16:51:06.0013847Z 		},
2024-07-08T16:51:06.0014109Z 		{
2024-07-08T16:51:06.0014557Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:06.0015130Z 			"size": 32,
2024-07-08T16:51:06.0015678Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-07-08T16:51:06.0016307Z 		},
2024-07-08T16:51:06.0016581Z 		{
2024-07-08T16:51:06.0017031Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:06.0017588Z 			"size": 32,
2024-07-08T16:51:06.0018141Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-07-08T16:51:06.0018765Z 		},
2024-07-08T16:51:06.0019021Z 		{
2024-07-08T16:51:06.0019466Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:06.0020030Z 			"size": 32,
2024-07-08T16:51:06.0020596Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-07-08T16:51:06.0021212Z 		},
2024-07-08T16:51:06.0021475Z 		{
2024-07-08T16:51:06.0021927Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2024-07-08T16:51:06.0022493Z 			"size": 32,
2024-07-08T16:51:06.0023023Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2024-07-08T16:51:06.0023658Z 		}
2024-07-08T16:51:06.0023915Z 	]
2024-07-08T16:51:06.0024157Z }
2024-07-08T16:51:06.0154748Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/}
2024-07-08T16:51:06.0155284Z [36;1mtag=${ECR_DOCKER_IMAGE##*/}[0m
2024-07-08T16:51:06.0155837Z [36;1mecho "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}"[0m
2024-07-08T16:51:06.0164076Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:51:06.0164582Z env:
2024-07-08T16:51:06.0164859Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:06.0166156Z   ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:06.0167332Z ##[endgroup]
2024-07-08T16:51:06.0189831Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:06.0239097Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2024-07-08T16:51:06.0239684Z with:
2024-07-08T16:51:06.0240835Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:06.0242156Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-07-08T16:51:06.0242676Z env:
2024-07-08T16:51:06.0242958Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:06.0243314Z ##[endgroup]
2024-07-08T16:51:06.0259419Z ##[group]Run set -x
2024-07-08T16:51:06.0259762Z [36;1mset -x[0m
2024-07-08T16:51:06.0260072Z [36;1mset +e[0m
2024-07-08T16:51:06.0260370Z [36;1m[0m
2024-07-08T16:51:06.0260658Z [36;1mlogin() {[0m
2024-07-08T16:51:06.0261311Z [36;1m  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"[0m
2024-07-08T16:51:06.0262034Z [36;1m}[0m
2024-07-08T16:51:06.0262314Z [36;1m[0m
2024-07-08T16:51:06.0262619Z [36;1mretry () {[0m
2024-07-08T16:51:06.0263015Z [36;1m  $*  || (sleep 1 && $*) || (sleep 2 && $*)[0m
2024-07-08T16:51:06.0263458Z [36;1m}[0m
2024-07-08T16:51:06.0263734Z [36;1m[0m
2024-07-08T16:51:06.0264164Z [36;1mretry login "${DOCKER_REGISTRY}"[0m
2024-07-08T16:51:06.0264634Z [36;1m[0m
2024-07-08T16:51:06.0264937Z [36;1mset -e[0m
2024-07-08T16:51:06.0265438Z [36;1m# ignore output since only exit code is used for conditional[0m
2024-07-08T16:51:06.0266141Z [36;1m# only pull docker image if it's not available locally[0m
2024-07-08T16:51:06.0266907Z [36;1mif ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then[0m
2024-07-08T16:51:06.0267595Z [36;1m  retry docker pull "${DOCKER_IMAGE}"[0m
2024-07-08T16:51:06.0268089Z [36;1mfi[0m
2024-07-08T16:51:06.0275545Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:51:06.0276046Z env:
2024-07-08T16:51:06.0276324Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:51:06.0277471Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:06.0278774Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-07-08T16:51:06.0279306Z ##[endgroup]
2024-07-08T16:51:06.0299318Z + set +e
2024-07-08T16:51:06.0299924Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-07-08T16:51:06.0300550Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-07-08T16:51:06.0301647Z + aws ecr get-login-password --region us-east-1
2024-07-08T16:51:06.0302411Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2024-07-08T16:51:06.4359038Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2024-07-08T16:51:06.4359951Z Configure a credential helper to remove this warning. See
2024-07-08T16:51:06.4360865Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2024-07-08T16:51:06.4361426Z
2024-07-08T16:51:06.4361549Z Login Succeeded
2024-07-08T16:51:06.4368436Z + set -e
2024-07-08T16:51:06.4369774Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:06.4492893Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:06.4495021Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:51:06.6635099Z 52b65ca8de123a7d92fdc1594a96fa3040229cab: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks
2024-07-08T16:51:06.6636883Z 7a2c55901189: Pulling fs layer
2024-07-08T16:51:06.6637717Z 224fe954d725: Pulling fs layer
2024-07-08T16:51:06.6638673Z 75722010b82e: Pulling fs layer
2024-07-08T16:51:06.6639551Z d527cbbb87e3: Pulling fs layer
2024-07-08T16:51:06.6642223Z b57676e46aee: Pulling fs layer
2024-07-08T16:51:06.6642753Z a8c1e85b5e14: Pulling fs layer
2024-07-08T16:51:06.6643315Z a41a8d1c11c8: Pulling fs layer
2024-07-08T16:51:06.6643889Z 0c1227890755: Pulling fs layer
2024-07-08T16:51:06.6644424Z d8d1234baab3: Pulling fs layer
2024-07-08T16:51:06.6644925Z 7ed32bc8e469: Pulling fs layer
2024-07-08T16:51:06.6645415Z b57676e46aee: Waiting
2024-07-08T16:51:06.6646183Z ec1e7978c1fe: Pulling fs layer
2024-07-08T16:51:06.6646687Z 2500215611d5: Pulling fs layer
2024-07-08T16:51:06.6647177Z a8c1e85b5e14: Waiting
2024-07-08T16:51:06.6647551Z e392141629a3: Pulling fs layer
2024-07-08T16:51:06.6647925Z 60a6b13946fd: Pulling fs layer
2024-07-08T16:51:06.6648295Z d2c0e45285ae: Pulling fs layer
2024-07-08T16:51:06.6648671Z 9a09c9d2f870: Pulling fs layer
2024-07-08T16:51:06.6649075Z a41a8d1c11c8: Waiting
2024-07-08T16:51:06.6649538Z cad71e8326b7: Pulling fs layer
2024-07-08T16:51:06.6650011Z 0c1227890755: Waiting
2024-07-08T16:51:06.6650456Z 73516a730d12: Pulling fs layer
2024-07-08T16:51:06.6650932Z d8d1234baab3: Waiting
2024-07-08T16:51:06.6651371Z 3aef13fe6525: Pulling fs layer
2024-07-08T16:51:06.6651941Z 58051158c0cb: Pulling fs layer
2024-07-08T16:51:06.6652303Z 7ed32bc8e469: Waiting
2024-07-08T16:51:06.6652624Z ec1e7978c1fe: Waiting
2024-07-08T16:51:06.6652952Z 980bc6bf56d3: Pulling fs layer
2024-07-08T16:51:06.6653312Z 2500215611d5: Waiting
2024-07-08T16:51:06.6653639Z 4bbd66984c29: Pulling fs layer
2024-07-08T16:51:06.6654005Z d527cbbb87e3: Waiting
2024-07-08T16:51:06.6654316Z e392141629a3: Waiting
2024-07-08T16:51:06.6654648Z 08c102569719: Pulling fs layer
2024-07-08T16:51:06.6655058Z d2c0e45285ae: Waiting
2024-07-08T16:51:06.6655378Z 60a6b13946fd: Waiting
2024-07-08T16:51:06.6655708Z ae32e42b5913: Pulling fs layer
2024-07-08T16:51:06.6656077Z cad71e8326b7: Waiting
2024-07-08T16:51:06.6656407Z 68354870b7a0: Pulling fs layer
2024-07-08T16:51:06.6656770Z 73516a730d12: Waiting
2024-07-08T16:51:06.6657094Z 61b35c224f83: Pulling fs layer
2024-07-08T16:51:06.6657466Z 9a09c9d2f870: Waiting
2024-07-08T16:51:06.6657781Z 4bbd66984c29: Waiting
2024-07-08T16:51:06.6658128Z 7c79b55aab49: Pulling fs layer
2024-07-08T16:51:06.6658534Z 980bc6bf56d3: Waiting
2024-07-08T16:51:06.6658859Z 3aef13fe6525: Waiting
2024-07-08T16:51:06.6659173Z 08c102569719: Waiting
2024-07-08T16:51:06.6659477Z 58051158c0cb: Waiting
2024-07-08T16:51:06.6659807Z 97f2ee2e8876: Pulling fs layer
2024-07-08T16:51:06.6660164Z 68354870b7a0: Waiting
2024-07-08T16:51:06.6660476Z 61b35c224f83: Waiting
2024-07-08T16:51:06.6660785Z 97f2ee2e8876: Waiting
2024-07-08T16:51:06.6661100Z 7c79b55aab49: Waiting
2024-07-08T16:51:06.6661426Z ae32e42b5913: Waiting
2024-07-08T16:51:06.6661750Z f26b1339c734: Pulling fs layer
2024-07-08T16:51:06.6662120Z a938af61c805: Pulling fs layer
2024-07-08T16:51:06.6662492Z c2d76dae598c: Pulling fs layer
2024-07-08T16:51:06.6662859Z 07a096d30854: Pulling fs layer
2024-07-08T16:51:06.6663229Z 9617411c9c34: Pulling fs layer
2024-07-08T16:51:06.6663614Z fcaf380e209a: Pulling fs layer
2024-07-08T16:51:06.6663992Z 335862b68741: Pulling fs layer
2024-07-08T16:51:06.6664370Z f4973d9e8cd6: Pulling fs layer
2024-07-08T16:51:06.6664726Z f26b1339c734: Waiting
2024-07-08T16:51:06.6665055Z 5b44615090cc: Pulling fs layer
2024-07-08T16:51:06.6665411Z a938af61c805: Waiting
2024-07-08T16:51:06.6665735Z dc088c962a35: Pulling fs layer
2024-07-08T16:51:06.6666109Z 0f0eedef185d: Pulling fs layer
2024-07-08T16:51:06.6666483Z 815f7a234f81: Pulling fs layer
2024-07-08T16:51:06.6666841Z c2d76dae598c: Waiting
2024-07-08T16:51:06.6667165Z 2be212e55519: Pulling fs layer
2024-07-08T16:51:06.6667538Z e01cceedac02: Pulling fs layer
2024-07-08T16:51:06.6667915Z 5aa920a2a362: Pulling fs layer
2024-07-08T16:51:06.6668283Z 277d6ed6a725: Pulling fs layer
2024-07-08T16:51:06.6668633Z f4973d9e8cd6: Waiting
2024-07-08T16:51:06.6668944Z 5b44615090cc: Waiting
2024-07-08T16:51:06.6669276Z 238bb91c34c5: Pulling fs layer
2024-07-08T16:51:06.6669742Z d12781f84558: Pulling fs layer
2024-07-08T16:51:06.6670102Z 335862b68741: Waiting
2024-07-08T16:51:06.6670416Z 815f7a234f81: Waiting
2024-07-08T16:51:06.6670748Z 0da1d2512da8: Pulling fs layer
2024-07-08T16:51:06.6671111Z dc088c962a35: Waiting
2024-07-08T16:51:06.6671444Z c5cc709139b8: Pulling fs layer
2024-07-08T16:51:06.6671806Z 0f0eedef185d: Waiting
2024-07-08T16:51:06.6672136Z 93f1924e1938: Pulling fs layer
2024-07-08T16:51:06.6672506Z d5331131aafd: Pulling fs layer
2024-07-08T16:51:06.6672857Z 2be212e55519: Waiting
2024-07-08T16:51:06.6673169Z e01cceedac02: Waiting
2024-07-08T16:51:06.6673496Z 9157dbafa6bc: Pulling fs layer
2024-07-08T16:51:06.6673853Z 0da1d2512da8: Waiting
2024-07-08T16:51:06.6674164Z d12781f84558: Waiting
2024-07-08T16:51:06.6674472Z c5cc709139b8: Waiting
2024-07-08T16:51:06.6674792Z 0c2f066daef8: Pulling fs layer
2024-07-08T16:51:06.6675147Z 9617411c9c34: Waiting
2024-07-08T16:51:06.6675473Z bf4cdb032437: Pulling fs layer
2024-07-08T16:51:06.6675832Z bf4cdb032437: Waiting
2024-07-08T16:51:06.6676159Z 16f104223c35: Pulling fs layer
2024-07-08T16:51:06.6676528Z 4f4fb700ef54: Pulling fs layer
2024-07-08T16:51:06.6676883Z 5aa920a2a362: Waiting
2024-07-08T16:51:06.6677264Z 01d06d370ea4: Pulling fs layer
2024-07-08T16:51:06.6677643Z 514e1f285315: Pulling fs layer
2024-07-08T16:51:06.6678002Z 4f4fb700ef54: Waiting
2024-07-08T16:51:06.6678337Z e83fe93022c4: Pulling fs layer
2024-07-08T16:51:06.6678693Z 93f1924e1938: Waiting
2024-07-08T16:51:06.6679015Z 01d06d370ea4: Waiting
2024-07-08T16:51:06.6679330Z 0c2f066daef8: Waiting
2024-07-08T16:51:06.6679655Z 729564988a36: Pulling fs layer
2024-07-08T16:51:06.6680030Z 19e8b19d9b3e: Pulling fs layer
2024-07-08T16:51:06.6680408Z 930584c90624: Pulling fs layer
2024-07-08T16:51:06.6680917Z 44b0076546be: Pulling fs layer
2024-07-08T16:51:06.6681296Z 10ce8b9cd377: Pulling fs layer
2024-07-08T16:51:06.6681662Z 729564988a36: Waiting
2024-07-08T16:51:06.6681976Z 19e8b19d9b3e: Waiting
2024-07-08T16:51:06.6682317Z e2feb35cbe3f: Pulling fs layer
2024-07-08T16:51:06.6682711Z fdf9f0439954: Pulling fs layer
2024-07-08T16:51:06.6683066Z 930584c90624: Waiting
2024-07-08T16:51:06.6683390Z 35290ca2e602: Pulling fs layer
2024-07-08T16:51:06.6683746Z e2feb35cbe3f: Waiting
2024-07-08T16:51:06.6684066Z 10ce8b9cd377: Waiting
2024-07-08T16:51:06.6684384Z fdf9f0439954: Waiting
2024-07-08T16:51:06.6684695Z 514e1f285315: Waiting
2024-07-08T16:51:06.6685005Z 9157dbafa6bc: Waiting
2024-07-08T16:51:06.6685317Z 44b0076546be: Waiting
2024-07-08T16:51:06.6685808Z 16f104223c35: Waiting
2024-07-08T16:51:06.6686162Z ec7e4facda86: Pulling fs layer
2024-07-08T16:51:06.6686534Z 35290ca2e602: Waiting
2024-07-08T16:51:06.6686870Z af1dbfbab64a: Pulling fs layer
2024-07-08T16:51:06.6687245Z 44afb2fb888d: Pulling fs layer
2024-07-08T16:51:06.6687613Z 055b36a703f0: Pulling fs layer
2024-07-08T16:51:06.6687987Z 1cb19548583a: Pulling fs layer
2024-07-08T16:51:06.6688343Z ec7e4facda86: Waiting
2024-07-08T16:51:06.6688660Z af1dbfbab64a: Waiting
2024-07-08T16:51:06.6688987Z 44afb2fb888d: Waiting
2024-07-08T16:51:06.6689299Z 055b36a703f0: Waiting
2024-07-08T16:51:06.6689611Z 1cb19548583a: Waiting
2024-07-08T16:51:06.7986105Z 224fe954d725: Verifying Checksum
2024-07-08T16:51:06.7986591Z 224fe954d725: Download complete
2024-07-08T16:51:06.8759400Z d527cbbb87e3: Verifying Checksum
2024-07-08T16:51:06.8776073Z d527cbbb87e3: Download complete
2024-07-08T16:51:06.9606337Z b57676e46aee: Download complete
2024-07-08T16:51:07.0142767Z 7a2c55901189: Download complete
2024-07-08T16:51:07.0774130Z a41a8d1c11c8: Download complete
2024-07-08T16:51:07.1606656Z 0c1227890755: Verifying Checksum
2024-07-08T16:51:07.1607163Z 0c1227890755: Download complete
2024-07-08T16:51:07.2625930Z d8d1234baab3: Download complete
2024-07-08T16:51:07.2811332Z 75722010b82e: Verifying Checksum
2024-07-08T16:51:07.2811904Z 75722010b82e: Download complete
2024-07-08T16:51:07.3624672Z ec1e7978c1fe: Verifying Checksum
2024-07-08T16:51:07.3625284Z ec1e7978c1fe: Download complete
2024-07-08T16:51:07.4650118Z 2500215611d5: Download complete
2024-07-08T16:51:08.0325206Z 7a2c55901189: Pull complete
2024-07-08T16:51:08.3342016Z 224fe954d725: Pull complete
2024-07-08T16:51:09.3187967Z 75722010b82e: Pull complete
2024-07-08T16:51:09.4081257Z d527cbbb87e3: Pull complete
2024-07-08T16:51:09.4835234Z b57676e46aee: Pull complete
2024-07-08T16:51:09.9914393Z e392141629a3: Verifying Checksum
2024-07-08T16:51:09.9914876Z e392141629a3: Download complete
2024-07-08T16:51:10.1178554Z 60a6b13946fd: Verifying Checksum
2024-07-08T16:51:10.1179006Z 60a6b13946fd: Download complete
2024-07-08T16:51:10.1858744Z d2c0e45285ae: Verifying Checksum
2024-07-08T16:51:10.1859279Z d2c0e45285ae: Download complete
2024-07-08T16:51:10.2629309Z 9a09c9d2f870: Verifying Checksum
2024-07-08T16:51:10.2629832Z 9a09c9d2f870: Download complete
2024-07-08T16:51:11.2385839Z cad71e8326b7: Verifying Checksum
2024-07-08T16:51:11.2386333Z cad71e8326b7: Download complete
2024-07-08T16:51:11.3070575Z 73516a730d12: Download complete
2024-07-08T16:51:11.3868218Z 3aef13fe6525: Verifying Checksum
2024-07-08T16:51:11.3868851Z 3aef13fe6525: Download complete
2024-07-08T16:51:11.4614445Z 58051158c0cb: Verifying Checksum
2024-07-08T16:51:11.4614874Z 58051158c0cb: Download complete
2024-07-08T16:51:20.3137913Z a8c1e85b5e14: Verifying Checksum
2024-07-08T16:51:20.3138405Z a8c1e85b5e14: Download complete
2024-07-08T16:51:20.3909370Z 4bbd66984c29: Verifying Checksum
2024-07-08T16:51:20.3910051Z 4bbd66984c29: Download complete
2024-07-08T16:51:20.4686601Z 08c102569719: Verifying Checksum
2024-07-08T16:51:20.4689265Z 08c102569719: Download complete
2024-07-08T16:51:20.5694533Z ae32e42b5913: Verifying Checksum
2024-07-08T16:51:20.5695259Z ae32e42b5913: Download complete
2024-07-08T16:51:20.6442900Z 68354870b7a0: Download complete
2024-07-08T16:51:20.7127205Z 61b35c224f83: Verifying Checksum
2024-07-08T16:51:20.7127757Z 61b35c224f83: Download complete
2024-07-08T16:51:21.9940666Z 7c79b55aab49: Verifying Checksum
2024-07-08T16:51:21.9941269Z 7c79b55aab49: Download complete
2024-07-08T16:51:22.0773020Z 97f2ee2e8876: Verifying Checksum
2024-07-08T16:51:22.0773668Z 97f2ee2e8876: Download complete
2024-07-08T16:51:22.1522826Z f26b1339c734: Download complete
2024-07-08T16:51:22.2566360Z a938af61c805: Verifying Checksum
2024-07-08T16:51:22.2567057Z a938af61c805: Download complete
2024-07-08T16:51:22.3500453Z c2d76dae598c: Verifying Checksum
2024-07-08T16:51:22.3501019Z c2d76dae598c: Download complete
2024-07-08T16:51:22.4282443Z 07a096d30854: Verifying Checksum
2024-07-08T16:51:22.4282929Z 07a096d30854: Download complete
2024-07-08T16:51:26.9497844Z 9617411c9c34: Verifying Checksum
2024-07-08T16:51:26.9498428Z 9617411c9c34: Download complete
2024-07-08T16:51:27.0369003Z fcaf380e209a: Verifying Checksum
2024-07-08T16:51:27.0369587Z fcaf380e209a: Download complete
2024-07-08T16:51:27.1157229Z 335862b68741: Verifying Checksum
2024-07-08T16:51:27.1157687Z 335862b68741: Download complete
2024-07-08T16:51:27.5247053Z f4973d9e8cd6: Verifying Checksum
2024-07-08T16:51:27.5247655Z f4973d9e8cd6: Download complete
2024-07-08T16:51:27.6016207Z 5b44615090cc: Verifying Checksum
2024-07-08T16:51:27.6016673Z 5b44615090cc: Download complete
2024-07-08T16:51:27.7446831Z dc088c962a35: Verifying Checksum
2024-07-08T16:51:27.7447418Z dc088c962a35: Download complete
2024-07-08T16:51:28.0057449Z 0f0eedef185d: Verifying Checksum
2024-07-08T16:51:28.0058076Z 0f0eedef185d: Download complete
2024-07-08T16:51:28.1199316Z 815f7a234f81: Verifying Checksum
2024-07-08T16:51:28.1199930Z 815f7a234f81: Download complete
2024-07-08T16:51:28.1999293Z 2be212e55519: Download complete
2024-07-08T16:51:28.2888934Z e01cceedac02: Verifying Checksum
2024-07-08T16:51:28.2889691Z e01cceedac02: Download complete
2024-07-08T16:51:32.6045054Z 7ed32bc8e469: Verifying Checksum
2024-07-08T16:51:32.6045838Z 7ed32bc8e469: Download complete
2024-07-08T16:51:32.6804494Z 277d6ed6a725: Verifying Checksum
2024-07-08T16:51:32.6805061Z 277d6ed6a725: Download complete
2024-07-08T16:51:32.7578131Z 238bb91c34c5: Verifying Checksum
2024-07-08T16:51:32.7578917Z 238bb91c34c5: Download complete
2024-07-08T16:51:33.0831949Z a8c1e85b5e14: Pull complete
2024-07-08T16:51:33.2456379Z d12781f84558: Verifying Checksum
2024-07-08T16:51:33.2456837Z d12781f84558: Download complete
2024-07-08T16:51:33.3218452Z a41a8d1c11c8: Pull complete
2024-07-08T16:51:33.3284524Z 0da1d2512da8: Verifying Checksum
2024-07-08T16:51:33.3285147Z 0da1d2512da8: Download complete
2024-07-08T16:51:33.4029277Z c5cc709139b8: Verifying Checksum
2024-07-08T16:51:33.4029707Z c5cc709139b8: Download complete
2024-07-08T16:51:33.4795052Z 93f1924e1938: Verifying Checksum
2024-07-08T16:51:33.4795568Z 93f1924e1938: Download complete
2024-07-08T16:51:33.5517863Z d5331131aafd: Verifying Checksum
2024-07-08T16:51:33.5518379Z d5331131aafd: Download complete
2024-07-08T16:51:33.5607872Z 0c1227890755: Pull complete
2024-07-08T16:51:33.7800931Z d8d1234baab3: Pull complete
2024-07-08T16:51:39.3133128Z 980bc6bf56d3: Verifying Checksum
2024-07-08T16:51:39.3133619Z 980bc6bf56d3: Download complete
2024-07-08T16:51:39.3855804Z 0c2f066daef8: Verifying Checksum
2024-07-08T16:51:39.3857784Z 0c2f066daef8: Download complete
2024-07-08T16:51:39.4599037Z bf4cdb032437: Verifying Checksum
2024-07-08T16:51:39.4599530Z bf4cdb032437: Download complete
2024-07-08T16:51:39.5376099Z 16f104223c35: Verifying Checksum
2024-07-08T16:51:39.5376598Z 16f104223c35: Download complete
2024-07-08T16:51:39.5450187Z 4f4fb700ef54: Download complete
2024-07-08T16:51:39.6322975Z 01d06d370ea4: Verifying Checksum
2024-07-08T16:51:39.6323570Z 01d06d370ea4: Download complete
2024-07-08T16:51:39.7091195Z 514e1f285315: Verifying Checksum
2024-07-08T16:51:39.7091804Z 514e1f285315: Download complete
2024-07-08T16:51:39.8422286Z e83fe93022c4: Verifying Checksum
2024-07-08T16:51:39.8422834Z e83fe93022c4: Download complete
2024-07-08T16:51:39.9057381Z 729564988a36: Verifying Checksum
2024-07-08T16:51:39.9057883Z 729564988a36: Download complete
2024-07-08T16:51:40.0147998Z 19e8b19d9b3e: Verifying Checksum
2024-07-08T16:51:40.0148461Z 19e8b19d9b3e: Download complete
2024-07-08T16:51:40.0887728Z 930584c90624: Verifying Checksum
2024-07-08T16:51:40.0888180Z 930584c90624: Download complete
2024-07-08T16:51:40.1676613Z 44b0076546be: Verifying Checksum
2024-07-08T16:51:40.1677113Z 44b0076546be: Download complete
2024-07-08T16:51:40.2888083Z 10ce8b9cd377: Verifying Checksum
2024-07-08T16:51:40.2888607Z 10ce8b9cd377: Download complete
2024-07-08T16:51:40.3810451Z e2feb35cbe3f: Download complete
2024-07-08T16:51:40.9757938Z fdf9f0439954: Verifying Checksum
2024-07-08T16:51:40.9758412Z fdf9f0439954: Download complete
2024-07-08T16:51:41.0899858Z 35290ca2e602: Verifying Checksum
2024-07-08T16:51:41.0900353Z 35290ca2e602: Download complete
2024-07-08T16:52:04.6703219Z 9157dbafa6bc: Verifying Checksum
2024-07-08T16:52:04.6705111Z 9157dbafa6bc: Download complete
2024-07-08T16:52:04.7519300Z af1dbfbab64a: Verifying Checksum
2024-07-08T16:52:04.7520870Z af1dbfbab64a: Download complete
2024-07-08T16:52:04.8291497Z 44afb2fb888d: Verifying Checksum
2024-07-08T16:52:04.8292035Z 44afb2fb888d: Download complete
2024-07-08T16:52:06.0939019Z 055b36a703f0: Verifying Checksum
2024-07-08T16:52:06.0939490Z 055b36a703f0: Download complete
2024-07-08T16:52:06.1981212Z 1cb19548583a: Download complete
2024-07-08T16:52:09.9489375Z ec7e4facda86: Verifying Checksum
2024-07-08T16:52:09.9489845Z ec7e4facda86: Download complete
2024-07-08T16:52:32.5880997Z 5aa920a2a362: Verifying Checksum
2024-07-08T16:52:32.5881480Z 5aa920a2a362: Download complete
2024-07-08T16:52:49.1470034Z 7ed32bc8e469: Pull complete
2024-07-08T16:52:49.3712453Z ec1e7978c1fe: Pull complete
2024-07-08T16:52:49.5814502Z 2500215611d5: Pull complete
2024-07-08T16:52:56.8393611Z e392141629a3: Pull complete
2024-07-08T16:52:57.0555216Z 60a6b13946fd: Pull complete
2024-07-08T16:52:57.2939680Z d2c0e45285ae: Pull complete
2024-07-08T16:52:57.5102436Z 9a09c9d2f870: Pull complete
2024-07-08T16:52:59.8891717Z cad71e8326b7: Pull complete
2024-07-08T16:53:00.1139886Z 73516a730d12: Pull complete
2024-07-08T16:53:00.3387077Z 3aef13fe6525: Pull complete
2024-07-08T16:53:00.5662724Z 58051158c0cb: Pull complete
2024-07-08T16:53:49.6970496Z 980bc6bf56d3: Pull complete
2024-07-08T16:53:49.8958311Z 4bbd66984c29: Pull complete
2024-07-08T16:53:50.1326664Z 08c102569719: Pull complete
2024-07-08T16:53:50.3470501Z ae32e42b5913: Pull complete
2024-07-08T16:53:50.5136874Z 68354870b7a0: Pull complete
2024-07-08T16:53:50.6976935Z 61b35c224f83: Pull complete
2024-07-08T16:53:54.1853283Z 7c79b55aab49: Pull complete
2024-07-08T16:53:54.3615349Z 97f2ee2e8876: Pull complete
2024-07-08T16:53:54.4728047Z f26b1339c734: Pull complete
2024-07-08T16:53:54.6327469Z a938af61c805: Pull complete
2024-07-08T16:53:54.8568357Z c2d76dae598c: Pull complete
2024-07-08T16:53:55.0897686Z 07a096d30854: Pull complete
2024-07-08T16:54:03.3334402Z 9617411c9c34: Pull complete
2024-07-08T16:54:03.4917050Z fcaf380e209a: Pull complete
2024-07-08T16:54:03.6908045Z 335862b68741: Pull complete
2024-07-08T16:54:04.6706274Z f4973d9e8cd6: Pull complete
2024-07-08T16:54:04.8569574Z 5b44615090cc: Pull complete
2024-07-08T16:54:05.0208549Z dc088c962a35: Pull complete
2024-07-08T16:54:05.4952996Z 0f0eedef185d: Pull complete
2024-07-08T16:54:05.6453433Z 815f7a234f81: Pull complete
2024-07-08T16:54:06.0832489Z 2be212e55519: Pull complete
2024-07-08T16:54:06.2764929Z e01cceedac02: Pull complete
2024-07-08T16:54:44.7924568Z 5aa920a2a362: Pull complete
2024-07-08T16:54:44.9194355Z 277d6ed6a725: Pull complete
2024-07-08T16:54:44.9573361Z 238bb91c34c5: Pull complete
2024-07-08T16:54:46.9044460Z d12781f84558: Pull complete
2024-07-08T16:54:46.9401032Z 0da1d2512da8: Pull complete
2024-07-08T16:54:47.0047583Z c5cc709139b8: Pull complete
2024-07-08T16:54:47.1327037Z 93f1924e1938: Pull complete
2024-07-08T16:54:47.1608923Z d5331131aafd: Pull complete
2024-07-08T16:55:21.6179529Z 9157dbafa6bc: Pull complete
2024-07-08T16:55:21.8127596Z 0c2f066daef8: Pull complete
2024-07-08T16:55:22.0427774Z bf4cdb032437: Pull complete
2024-07-08T16:55:22.4169124Z 16f104223c35: Pull complete
2024-07-08T16:55:22.6454422Z 4f4fb700ef54: Pull complete
2024-07-08T16:55:22.8678136Z 01d06d370ea4: Pull complete
2024-07-08T16:55:23.0873096Z 514e1f285315: Pull complete
2024-07-08T16:55:23.4606719Z e83fe93022c4: Pull complete
2024-07-08T16:55:23.6912320Z 729564988a36: Pull complete
2024-07-08T16:55:23.9142360Z 19e8b19d9b3e: Pull complete
2024-07-08T16:55:24.1403783Z 930584c90624: Pull complete
2024-07-08T16:55:24.3582231Z 44b0076546be: Pull complete
2024-07-08T16:55:25.4068341Z 10ce8b9cd377: Pull complete
2024-07-08T16:55:25.5675688Z e2feb35cbe3f: Pull complete
2024-07-08T16:55:27.6428679Z fdf9f0439954: Pull complete
2024-07-08T16:55:27.8614046Z 35290ca2e602: Pull complete
2024-07-08T16:55:41.2853222Z ec7e4facda86: Pull complete
2024-07-08T16:55:41.3489306Z af1dbfbab64a: Pull complete
2024-07-08T16:55:41.3814255Z 44afb2fb888d: Pull complete
2024-07-08T16:55:42.0894565Z 055b36a703f0: Pull complete
2024-07-08T16:55:42.1626519Z 1cb19548583a: Pull complete
2024-07-08T16:55:42.7622402Z Digest: sha256:5937e650467bc1196abb8a39571fc2470742e1a0827fa4cc78d216bf0b182f34
2024-07-08T16:55:42.8082233Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:55:42.8307425Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:55:42.8339211Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT"
2024-07-08T16:55:42.8340137Z [36;1mecho "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT"[0m
2024-07-08T16:55:42.8348148Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:55:42.8348655Z env:
2024-07-08T16:55:42.8348935Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:55:42.8349277Z ##[endgroup]
2024-07-08T16:55:42.8502358Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2024-07-08T16:55:42.8502901Z with:
2024-07-08T16:55:42.8503190Z   driver-version: 550.54.15
2024-07-08T16:55:42.8503532Z env:
2024-07-08T16:55:42.8503805Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:55:42.8504156Z ##[endgroup]
2024-07-08T16:55:42.8622720Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-07-08T16:55:42.8623285Z with:
2024-07-08T16:55:42.8623569Z   timeout_minutes: 10
2024-07-08T16:55:42.8623897Z   max_attempts: 3
2024-07-08T16:55:42.8655058Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then
          YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo"
        else
          # Amazon Linux 2
          YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"
        fi

        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        # Install nvidia-driver package if not installed
        status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)"
        if [ ! $? = 0 ] || [ ! "$status" = installed ]; then
          sudo apt-get install -y nvidia-docker2
          sudo systemctl restart docker
        fi
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e

            nvidia-smi
            # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in
            # the case where the driver has already crashed as it still can get the driver version
            # and some basic information like the bus ID.  However, the rest of the information
            # would be missing (ERR!), for example:
            #
            # +-----------------------------------------------------------------------------+
            # | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
            # |-------------------------------+----------------------+----------------------+
            # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
            # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
            # |                               |                      |               MIG M. |
            # |===============================+======================+======================|
            # |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |
            # |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |
            # |                               |                      |                 ERR! |
            # +-------------------------------+----------------------+----------------------+
            #
            # +-----------------------------------------------------------------------------+
            # | Processes:                                                                  |
            # |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
            # |        ID   ID                                                   Usage      |
            # |=============================================================================|
            # +-----------------------------------------------------------------------------+
            #
            # This should be reported as a failure instead as it will guarantee to fail when
            # Docker tries to run with --gpus all
            #
            # So, the correct check here is to query one of the missing piece of info like
            # GPU name, so that the command can fail accordingly
            nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}"

2024-07-08T16:55:42.8686452Z   retry_wait_seconds: 10
2024-07-08T16:55:42.8686836Z   polling_interval_seconds: 1
2024-07-08T16:55:42.8687213Z   warning_on_retry: true
2024-07-08T16:55:42.8687582Z   continue_on_error: false
2024-07-08T16:55:42.8687935Z env:
2024-07-08T16:55:42.8688216Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:55:42.8688572Z   DRIVER_VERSION: 550.54.15
2024-07-08T16:55:42.8688925Z ##[endgroup]
2024-07-08T16:55:42.9211737Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run ==
2024-07-08T16:55:42.9213393Z + pre_install_nvidia_driver_amzn2
2024-07-08T16:55:42.9214026Z + sudo yum remove -y nvidia-driver-latest-dkms
2024-07-08T16:55:43.2217625Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2024-07-08T16:55:43.2624100Z No Match for argument: nvidia-driver-latest-dkms
2024-07-08T16:55:43.2861183Z No Packages marked for removal
2024-07-08T16:55:43.2979498Z + install_nvidia_driver_common
2024-07-08T16:55:43.2980730Z + echo 'Before installing NVIDIA driver'
2024-07-08T16:55:43.2981243Z + lspci
2024-07-08T16:55:43.2981543Z Before installing NVIDIA driver
2024-07-08T16:55:43.3060054Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-07-08T16:55:43.3061039Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-07-08T16:55:43.3062329Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-07-08T16:55:43.3063275Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-07-08T16:55:43.3064043Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061
2024-07-08T16:55:43.3064803Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-07-08T16:55:43.3065542Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1)
2024-07-08T16:55:43.3066321Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-07-08T16:55:43.3066915Z + lsmod
2024-07-08T16:55:43.3073911Z Module                  Size  Used by
2024-07-08T16:55:43.3074488Z nvidia_modeset       1339392  0
2024-07-08T16:55:43.3074951Z veth                   16384  0
2024-07-08T16:55:43.3075361Z nvidia_uvm           4599808  0
2024-07-08T16:55:43.3075997Z nvidia              53989376  2 nvidia_uvm,nvidia_modeset
2024-07-08T16:55:43.3076571Z drm                   421888  1 nvidia
2024-07-08T16:55:43.3077369Z i2c_core               77824  2 nvidia,drm
2024-07-08T16:55:43.3078050Z backlight              16384  1 nvidia_modeset
2024-07-08T16:55:43.3078647Z xt_conntrack           16384  1
2024-07-08T16:55:43.3079179Z ipt_MASQUERADE         16384  1
2024-07-08T16:55:43.3079773Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2024-07-08T16:55:43.3080443Z nf_conntrack_netlink    49152  0
2024-07-08T16:55:43.3081190Z nfnetlink              16384  2 nf_conntrack_netlink
2024-07-08T16:55:43.3081879Z xfrm_user              45056  1
2024-07-08T16:55:43.3082428Z xfrm_algo              16384  1 xfrm_user
2024-07-08T16:55:43.3083014Z iptable_nat            16384  1
2024-07-08T16:55:43.3083413Z nf_conntrack_ipv4      16384  3
2024-07-08T16:55:43.3083976Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2024-07-08T16:55:43.3084465Z nf_nat_ipv4            16384  1 iptable_nat
2024-07-08T16:55:43.3085179Z nf_nat                 32768  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2024-07-08T16:55:43.3086692Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2024-07-08T16:55:43.3087801Z xt_addrtype            16384  2
2024-07-08T16:55:43.3088313Z iptable_filter         16384  1
2024-07-08T16:55:43.3088701Z br_netfilter           24576  0
2024-07-08T16:55:43.3089115Z bridge                172032  1 br_netfilter
2024-07-08T16:55:43.3089731Z stp                    16384  1 bridge
2024-07-08T16:55:43.3090160Z llc                    16384  2 bridge,stp
2024-07-08T16:55:43.3090578Z overlay                86016  0
2024-07-08T16:55:43.3090963Z sunrpc                393216  1
2024-07-08T16:55:43.3091339Z dm_mirror              28672  0
2024-07-08T16:55:43.3091742Z dm_region_hash         20480  1 dm_mirror
2024-07-08T16:55:43.3092234Z dm_log                 20480  2 dm_region_hash,dm_mirror
2024-07-08T16:55:43.3092749Z dm_mod                143360  2 dm_log,dm_mirror
2024-07-08T16:55:43.3093201Z dax                    69632  1 dm_mod
2024-07-08T16:55:43.3093726Z crc32_pclmul           16384  0
2024-07-08T16:55:43.3094115Z ghash_clmulni_intel    16384  0
2024-07-08T16:55:43.3094554Z pcbc                   16384  0
2024-07-08T16:55:43.3094931Z aesni_intel           188416  0
2024-07-08T16:55:43.3095325Z aes_x86_64             20480  1 aesni_intel
2024-07-08T16:55:43.3095755Z mousedev               24576  0
2024-07-08T16:55:43.3096165Z crypto_simd            16384  1 aesni_intel
2024-07-08T16:55:43.3096626Z glue_helper            16384  1 aesni_intel
2024-07-08T16:55:43.3097049Z evdev                  20480  3
2024-07-08T16:55:43.3097559Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2024-07-08T16:55:43.3098104Z psmouse                32768  0
2024-07-08T16:55:43.3098478Z button                 16384  0
2024-07-08T16:55:43.3098842Z ena                   139264  0
2024-07-08T16:55:43.3099222Z ptp                    20480  1 ena
2024-07-08T16:55:43.3099623Z pps_core               20480  1 ptp
2024-07-08T16:55:43.3100026Z crc32c_intel           24576  0
2024-07-08T16:55:43.3100404Z autofs4                49152  2
2024-07-08T16:55:43.3100774Z + modinfo nvidia
2024-07-08T16:55:43.3101442Z filename:       /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2024-07-08T16:55:43.3102192Z alias:          char-major-195-*
2024-07-08T16:55:43.3102586Z version:        550.54.15
2024-07-08T16:55:43.3102940Z supported:      external
2024-07-08T16:55:43.3103285Z license:        NVIDIA
2024-07-08T16:55:43.3103665Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-07-08T16:55:43.3104158Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-07-08T16:55:43.3104627Z srcversion:     833721318DA517F0C2FEC97
2024-07-08T16:55:43.3105108Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-07-08T16:55:43.3105612Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-07-08T16:55:43.3106114Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-07-08T16:55:43.3106606Z depends:        i2c-core,drm
2024-07-08T16:55:43.3107062Z retpoline:      Y
2024-07-08T16:55:43.3107388Z name:           nvidia
2024-07-08T16:55:43.3107952Z vermagic:       4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions
2024-07-08T16:55:43.3108616Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-07-08T16:55:43.3109256Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-07-08T16:55:43.3109863Z parm:           NVreg_ResmanDebugLevel:int
2024-07-08T16:55:43.3110311Z parm:           NVreg_RmLogonRC:int
2024-07-08T16:55:43.3110751Z parm:           NVreg_ModifyDeviceFiles:int
2024-07-08T16:55:43.3111212Z parm:           NVreg_DeviceFileUID:int
2024-07-08T16:55:43.3111655Z parm:           NVreg_DeviceFileGID:int
2024-07-08T16:55:43.3112102Z parm:           NVreg_DeviceFileMode:int
2024-07-08T16:55:43.3112685Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-07-08T16:55:43.3113247Z parm:           NVreg_UsePageAttributeTable:int
2024-07-08T16:55:43.3113736Z parm:           NVreg_EnablePCIeGen3:int
2024-07-08T16:55:43.3114184Z parm:           NVreg_EnableMSI:int
2024-07-08T16:55:43.3114609Z parm:           NVreg_TCEBypassMode:int
2024-07-08T16:55:43.3115076Z parm:           NVreg_EnableStreamMemOPs:int
2024-07-08T16:55:43.3115615Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-07-08T16:55:43.3116208Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-07-08T16:55:43.3116828Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-07-08T16:55:43.3117446Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-07-08T16:55:43.3118050Z parm:           NVreg_DynamicPowerManagement:int
2024-07-08T16:55:43.3118668Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-07-08T16:55:43.3119280Z parm:           NVreg_EnableGpuFirmware:int
2024-07-08T16:55:43.3119774Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-07-08T16:55:43.3120319Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-07-08T16:55:43.3121000Z parm:           NVreg_EnableUserNUMAManagement:int
2024-07-08T16:55:43.3121505Z parm:           NVreg_MemoryPoolSize:int
2024-07-08T16:55:43.3121969Z parm:           NVreg_KMallocHeapMaxSize:int
2024-07-08T16:55:43.3122462Z parm:           NVreg_VMallocHeapMaxSize:int
2024-07-08T16:55:43.3122941Z parm:           NVreg_IgnoreMMIOCheck:int
2024-07-08T16:55:43.3123390Z parm:           NVreg_NvLinkDisable:int
2024-07-08T16:55:43.3123903Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-07-08T16:55:43.3124434Z parm:           NVreg_RegisterPCIDriver:int
2024-07-08T16:55:43.3124921Z parm:           NVreg_EnableResizableBar:int
2024-07-08T16:55:43.3125405Z parm:           NVreg_EnableDbgBreakpoint:int
2024-07-08T16:55:43.3126118Z parm:           NVreg_EnableNonblockingOpen:int
2024-07-08T16:55:43.3126608Z parm:           NVreg_RegistryDwords:charp
2024-07-08T16:55:43.3127114Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-07-08T16:55:43.3127592Z parm:           NVreg_RmMsg:charp
2024-07-08T16:55:43.3128016Z parm:           NVreg_GpuBlacklist:charp
2024-07-08T16:55:43.3128491Z parm:           NVreg_TemporaryFilePath:charp
2024-07-08T16:55:43.3128967Z parm:           NVreg_ExcludedGpus:charp
2024-07-08T16:55:43.3129428Z parm:           NVreg_DmaRemapPeerMmio:int
2024-07-08T16:55:43.3129914Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-07-08T16:55:43.3130399Z parm:           NVreg_ImexChannelCount:int
2024-07-08T16:55:43.3130864Z parm:           rm_firmware_active:charp
2024-07-08T16:55:43.3131279Z + HAS_NVIDIA_DRIVER=0
2024-07-08T16:55:43.3131666Z ++ command -v nvidia-smi
2024-07-08T16:55:43.3132117Z + '[' -x /usr/bin/nvidia-smi ']'
2024-07-08T16:55:43.3132485Z + set +e
2024-07-08T16:55:43.3132993Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2024-07-08T16:55:45.2961686Z + INSTALLED_DRIVER_VERSION=550.54.15
2024-07-08T16:55:45.2962224Z + NVIDIA_SMI_STATUS=0
2024-07-08T16:55:45.2962836Z + '[' 0 -ne 0 ']'
2024-07-08T16:55:45.2963331Z + '[' 550.54.15 '!=' 550.54.15 ']'
2024-07-08T16:55:45.2964175Z + HAS_NVIDIA_DRIVER=1
2024-07-08T16:55:45.2965177Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation'
2024-07-08T16:55:45.2966279Z + set -e
2024-07-08T16:55:45.2966582Z + '[' 1 -eq 0 ']'
2024-07-08T16:55:45.2966914Z + post_install_nvidia_driver_common
2024-07-08T16:55:45.2967578Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation
2024-07-08T16:55:45.2968241Z + sudo modprobe nvidia
2024-07-08T16:55:45.3063033Z + echo 'After installing NVIDIA driver'
2024-07-08T16:55:45.3063461Z + lspci
2024-07-08T16:55:45.3063753Z After installing NVIDIA driver
2024-07-08T16:55:45.3144346Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2024-07-08T16:55:45.3145311Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2024-07-08T16:55:45.3146571Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2024-07-08T16:55:45.3147672Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2024-07-08T16:55:45.3148751Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061
2024-07-08T16:55:45.3149826Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2024-07-08T16:55:45.3150849Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1)
2024-07-08T16:55:45.3151972Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2024-07-08T16:55:45.3152986Z + lsmod
2024-07-08T16:55:45.3159790Z Module                  Size  Used by
2024-07-08T16:55:45.3160287Z nvidia_modeset       1339392  0
2024-07-08T16:55:45.3160795Z veth                   16384  0
2024-07-08T16:55:45.3161438Z nvidia_uvm           4599808  0
2024-07-08T16:55:45.3162008Z nvidia              53989376  2 nvidia_uvm,nvidia_modeset
2024-07-08T16:55:45.3162504Z drm                   421888  1 nvidia
2024-07-08T16:55:45.3163086Z i2c_core               77824  2 nvidia,drm
2024-07-08T16:55:45.3163677Z backlight              16384  1 nvidia_modeset
2024-07-08T16:55:45.3164296Z xt_conntrack           16384  1
2024-07-08T16:55:45.3164717Z ipt_MASQUERADE         16384  1
2024-07-08T16:55:45.3165224Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2024-07-08T16:55:45.3166116Z nf_conntrack_netlink    49152  0
2024-07-08T16:55:45.3166716Z nfnetlink              16384  2 nf_conntrack_netlink
2024-07-08T16:55:45.3167336Z xfrm_user              45056  1
2024-07-08T16:55:45.3167881Z xfrm_algo              16384  1 xfrm_user
2024-07-08T16:55:45.3168369Z iptable_nat            16384  1
2024-07-08T16:55:45.3168913Z nf_conntrack_ipv4      16384  3
2024-07-08T16:55:45.3169484Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2024-07-08T16:55:45.3170122Z nf_nat_ipv4            16384  1 iptable_nat
2024-07-08T16:55:45.3170743Z nf_nat                 32768  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2024-07-08T16:55:45.3171725Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2024-07-08T16:55:45.3172577Z xt_addrtype            16384  2
2024-07-08T16:55:45.3172960Z iptable_filter         16384  1
2024-07-08T16:55:45.3173346Z br_netfilter           24576  0
2024-07-08T16:55:45.3173749Z bridge                172032  1 br_netfilter
2024-07-08T16:55:45.3174192Z stp                    16384  1 bridge
2024-07-08T16:55:45.3174617Z llc                    16384  2 bridge,stp
2024-07-08T16:55:45.3175040Z overlay                86016  0
2024-07-08T16:55:45.3175406Z sunrpc                393216  1
2024-07-08T16:55:45.3175777Z dm_mirror              28672  0
2024-07-08T16:55:45.3176179Z dm_region_hash         20480  1 dm_mirror
2024-07-08T16:55:45.3176664Z dm_log                 20480  2 dm_region_hash,dm_mirror
2024-07-08T16:55:45.3177268Z dm_mod                143360  2 dm_log,dm_mirror
2024-07-08T16:55:45.3177715Z dax                    69632  1 dm_mod
2024-07-08T16:55:45.3178124Z crc32_pclmul           16384  0
2024-07-08T16:55:45.3178529Z ghash_clmulni_intel    16384  0
2024-07-08T16:55:45.3179069Z pcbc                   16384  0
2024-07-08T16:55:45.3179452Z aesni_intel           188416  0
2024-07-08T16:55:45.3179851Z aes_x86_64             20480  1 aesni_intel
2024-07-08T16:55:45.3180266Z mousedev               24576  0
2024-07-08T16:55:45.3180669Z crypto_simd            16384  1 aesni_intel
2024-07-08T16:55:45.3181133Z glue_helper            16384  1 aesni_intel
2024-07-08T16:55:45.3181563Z evdev                  20480  3
2024-07-08T16:55:45.3182055Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2024-07-08T16:55:45.3182593Z psmouse                32768  0
2024-07-08T16:55:45.3182970Z button                 16384  0
2024-07-08T16:55:45.3183333Z ena                   139264  0
2024-07-08T16:55:45.3183709Z ptp                    20480  1 ena
2024-07-08T16:55:45.3184111Z pps_core               20480  1 ptp
2024-07-08T16:55:45.3184512Z crc32c_intel           24576  0
2024-07-08T16:55:45.3184883Z autofs4                49152  2
2024-07-08T16:55:45.3185255Z + modinfo nvidia
2024-07-08T16:55:45.3185932Z filename:       /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2024-07-08T16:55:45.3186632Z alias:          char-major-195-*
2024-07-08T16:55:45.3187012Z version:        550.54.15
2024-07-08T16:55:45.3187362Z supported:      external
2024-07-08T16:55:45.3187712Z license:        NVIDIA
2024-07-08T16:55:45.3188099Z firmware:       nvidia/550.54.15/gsp_tu10x.bin
2024-07-08T16:55:45.3188669Z firmware:       nvidia/550.54.15/gsp_ga10x.bin
2024-07-08T16:55:45.3189132Z srcversion:     833721318DA517F0C2FEC97
2024-07-08T16:55:45.3189601Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2024-07-08T16:55:45.3190098Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2024-07-08T16:55:45.3190594Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2024-07-08T16:55:45.3191085Z depends:        i2c-core,drm
2024-07-08T16:55:45.3191452Z retpoline:      Y
2024-07-08T16:55:45.3191759Z name:           nvidia
2024-07-08T16:55:45.3192322Z vermagic:       4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions
2024-07-08T16:55:45.3192976Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2024-07-08T16:55:45.3193622Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2024-07-08T16:55:45.3194211Z parm:           NVreg_ResmanDebugLevel:int
2024-07-08T16:55:45.3194658Z parm:           NVreg_RmLogonRC:int
2024-07-08T16:55:45.3195102Z parm:           NVreg_ModifyDeviceFiles:int
2024-07-08T16:55:45.3195560Z parm:           NVreg_DeviceFileUID:int
2024-07-08T16:55:45.3195992Z parm:           NVreg_DeviceFileGID:int
2024-07-08T16:55:45.3196435Z parm:           NVreg_DeviceFileMode:int
2024-07-08T16:55:45.3196967Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2024-07-08T16:55:45.3197534Z parm:           NVreg_UsePageAttributeTable:int
2024-07-08T16:55:45.3198011Z parm:           NVreg_EnablePCIeGen3:int
2024-07-08T16:55:45.3198443Z parm:           NVreg_EnableMSI:int
2024-07-08T16:55:45.3198871Z parm:           NVreg_TCEBypassMode:int
2024-07-08T16:55:45.3199334Z parm:           NVreg_EnableStreamMemOPs:int
2024-07-08T16:55:45.3199873Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2024-07-08T16:55:45.3200457Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2024-07-08T16:55:45.3201139Z parm:           NVreg_EnableS0ixPowerManagement:int
2024-07-08T16:55:45.3201754Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2024-07-08T16:55:45.3202363Z parm:           NVreg_DynamicPowerManagement:int
2024-07-08T16:55:45.3203033Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2024-07-08T16:55:45.3203633Z parm:           NVreg_EnableGpuFirmware:int
2024-07-08T16:55:45.3204123Z parm:           NVreg_EnableGpuFirmwareLogs:int
2024-07-08T16:55:45.3204665Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2024-07-08T16:55:45.3205217Z parm:           NVreg_EnableUserNUMAManagement:int
2024-07-08T16:55:45.3205955Z parm:           NVreg_MemoryPoolSize:int
2024-07-08T16:55:45.3206521Z parm:           NVreg_KMallocHeapMaxSize:int
2024-07-08T16:55:45.3207021Z parm:           NVreg_VMallocHeapMaxSize:int
2024-07-08T16:55:45.3207496Z parm:           NVreg_IgnoreMMIOCheck:int
2024-07-08T16:55:45.3207950Z parm:           NVreg_NvLinkDisable:int
2024-07-08T16:55:45.3208453Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2024-07-08T16:55:45.3208985Z parm:           NVreg_RegisterPCIDriver:int
2024-07-08T16:55:45.3209470Z parm:           NVreg_EnableResizableBar:int
2024-07-08T16:55:45.3209962Z parm:           NVreg_EnableDbgBreakpoint:int
2024-07-08T16:55:45.3210460Z parm:           NVreg_EnableNonblockingOpen:int
2024-07-08T16:55:45.3210949Z parm:           NVreg_RegistryDwords:charp
2024-07-08T16:55:45.3211458Z parm:           NVreg_RegistryDwordsPerDevice:charp
2024-07-08T16:55:45.3211933Z parm:           NVreg_RmMsg:charp
2024-07-08T16:55:45.3212355Z parm:           NVreg_GpuBlacklist:charp
2024-07-08T16:55:45.3212827Z parm:           NVreg_TemporaryFilePath:charp
2024-07-08T16:55:45.3213311Z parm:           NVreg_ExcludedGpus:charp
2024-07-08T16:55:45.3213766Z parm:           NVreg_DmaRemapPeerMmio:int
2024-07-08T16:55:45.3214253Z parm:           NVreg_RmNvlinkBandwidth:charp
2024-07-08T16:55:45.3214737Z parm:           NVreg_ImexChannelCount:int
2024-07-08T16:55:45.3215196Z parm:           rm_firmware_active:charp
2024-07-08T16:55:45.3215600Z + set +e
2024-07-08T16:55:45.3215915Z + nvidia-smi
2024-07-08T16:55:46.8814674Z Mon Jul  8 16:55:46 2024
2024-07-08T16:55:46.8815553Z +-----------------------------------------------------------------------------------------+
2024-07-08T16:55:46.8816377Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-07-08T16:55:46.8817157Z |-----------------------------------------+------------------------+----------------------+
2024-07-08T16:55:46.8817948Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-07-08T16:55:46.8818849Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-07-08T16:55:46.8819545Z |                                         |                        |               MIG M. |
2024-07-08T16:55:46.8820090Z |=========================================+========================+======================|
2024-07-08T16:55:46.8948886Z |   0  NVIDIA A10G                    Off |   00000000:00:1E.0 Off |                    0 |
2024-07-08T16:55:46.8963407Z |  0%   35C    P0             57W /  300W |       0MiB /  23028MiB |      0%      Default |
2024-07-08T16:55:46.8964073Z |                                         |                        |                  N/A |
2024-07-08T16:55:46.8964771Z +-----------------------------------------+------------------------+----------------------+
2024-07-08T16:55:46.8965356Z
2024-07-08T16:55:46.8966225Z +-----------------------------------------------------------------------------------------+
2024-07-08T16:55:46.8966865Z | Processes:                                                                              |
2024-07-08T16:55:46.8967577Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-07-08T16:55:46.8968276Z |        ID   ID                                                               Usage      |
2024-07-08T16:55:46.8968835Z |=========================================================================================|
2024-07-08T16:55:46.8969479Z |  No running processes found                                                             |
2024-07-08T16:55:46.8970231Z +-----------------------------------------------------------------------------------------+
2024-07-08T16:55:47.5133549Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2024-07-08T16:55:49.0722003Z NVIDIA A10G
2024-07-08T16:55:49.5121941Z + NVIDIA_SMI_STATUS=0
2024-07-08T16:55:49.5122776Z + '[' 0 -eq 0 ']'
2024-07-08T16:55:49.5123206Z + echo 'INFO: Ignoring allowed status 0'
2024-07-08T16:55:49.5123910Z + set -e
2024-07-08T16:55:49.5124214Z INFO: Ignoring allowed status 0
2024-07-08T16:55:49.5126189Z == Installing nvidia container toolkit for amzn2 ==
2024-07-08T16:55:49.5128446Z + sudo yum install -y yum-utils
2024-07-08T16:55:49.8134245Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2024-07-08T16:55:51.3370697Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2024-07-08T16:55:51.3371409Z Nothing to do
2024-07-08T16:55:51.4887220Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]]
2024-07-08T16:55:51.4888083Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2024-07-08T16:55:51.4889122Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2024-07-08T16:55:51.8212796Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2024-07-08T16:55:51.8476410Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2024-07-08T16:55:51.8477607Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2024-07-08T16:55:51.8478549Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2024-07-08T16:55:51.8596233Z + sudo yum install -y nvidia-docker2
2024-07-08T16:55:52.1748775Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2024-07-08T16:55:53.6368799Z Package nvidia-docker2-2.14.0-1.noarch already installed and latest version
2024-07-08T16:55:53.6369464Z Nothing to do
2024-07-08T16:55:53.7855959Z + sudo systemctl restart docker
2024-07-08T16:56:29.9653362Z Command completed after 1 attempt(s).
2024-07-08T16:56:29.9708998Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84
2024-07-08T16:56:29.9709774Z [36;1mpython3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84[0m
2024-07-08T16:56:29.9710471Z [36;1mpython3 -m tools.stats.monitor > usage_log.txt 2>&1 &[0m
2024-07-08T16:56:29.9711149Z [36;1mecho "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"[0m
2024-07-08T16:56:29.9718874Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:56:29.9719376Z env:
2024-07-08T16:56:29.9719663Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:56:29.9720127Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:56:29.9720619Z ##[endgroup]
2024-07-08T16:56:30.1932361Z Defaulting to user installation because normal site-packages is not writeable
2024-07-08T16:56:30.2082293Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1)
2024-07-08T16:56:30.2185302Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84)
2024-07-08T16:56:30.3149089Z Prepare all required actions
2024-07-08T16:56:30.3149643Z Getting action download info
2024-07-08T16:56:30.4102028Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2024-07-08T16:56:30.6162257Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a)
2024-07-08T16:56:30.7507218Z ##[group]Run ./.github/actions/download-build-artifacts
2024-07-08T16:56:30.7507922Z with:
2024-07-08T16:56:30.7508418Z   name: linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-07-08T16:56:30.7509104Z   s3-bucket: gha-artifacts
2024-07-08T16:56:30.7509600Z env:
2024-07-08T16:56:30.7509992Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:56:30.7510665Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:56:30.7511406Z ##[endgroup]
2024-07-08T16:56:31.7876171Z ##[group]Run seemethere/download-artifact-s3@v4
2024-07-08T16:56:31.7876670Z with:
2024-07-08T16:56:31.7877036Z   name: linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-07-08T16:56:31.7877491Z   s3-bucket: gha-artifacts
2024-07-08T16:56:31.7877832Z   region: us-east-1
2024-07-08T16:56:31.7878127Z env:
2024-07-08T16:56:31.7878399Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:56:31.7878840Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:56:31.7879318Z ##[endgroup]
2024-07-08T16:56:32.1954523Z (node:28512) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-07-08T16:56:32.1955333Z
2024-07-08T16:56:32.1955606Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-07-08T16:56:32.1956339Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-07-08T16:56:32.1957441Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-07-08T16:56:32.2716405Z Found 1 objects with prefix pytorch/pytorch/9843060221/linux-focal-cuda12.1-py3.10-gcc9-sm86/
2024-07-08T16:56:32.2717569Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-07-08T16:56:48.8179123Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2024-07-08T16:56:48.8184690Z Artifact download has finished successfully
2024-07-08T16:56:48.8332478Z ##[group]Run unzip -o artifacts.zip
2024-07-08T16:56:48.8332914Z [36;1munzip -o artifacts.zip[0m
2024-07-08T16:56:48.8340606Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:56:48.8341095Z env:
2024-07-08T16:56:48.8341358Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:56:48.8341814Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:56:48.8342290Z ##[endgroup]
2024-07-08T16:56:48.8371664Z Archive:  artifacts.zip
2024-07-08T16:56:48.8373320Z    creating: dist/
2024-07-08T16:56:50.8142197Z   inflating: dist/torch-2.5.0a0+gitc8ab2e8-cp310-cp310-linux_x86_64.whl
2024-07-08T16:56:50.8143542Z    creating: build/custom_test_artifacts/
2024-07-08T16:56:50.8144705Z    creating: build/custom_test_artifacts/custom-op-build/
2024-07-08T16:56:50.8146137Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2024-07-08T16:56:50.8147773Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2024-07-08T16:56:50.8150557Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml
2024-07-08T16:56:50.8152609Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/
2024-07-08T16:56:50.8153580Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake
2024-07-08T16:56:50.8154572Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/
2024-07-08T16:56:50.8155538Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-07-08T16:56:50.8156668Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c
2024-07-08T16:56:50.8157939Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out
2024-07-08T16:56:50.8158950Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-07-08T16:56:50.8159950Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-07-08T16:56:50.8161190Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp
2024-07-08T16:56:50.8162362Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out
2024-07-08T16:56:50.8163504Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin
2024-07-08T16:56:50.8164648Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake
2024-07-08T16:56:50.8166026Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin
2024-07-08T16:56:50.8167208Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake
2024-07-08T16:56:50.8168249Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-07-08T16:56:50.8169261Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-07-08T16:56:50.8202472Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii
2024-07-08T16:56:50.8240110Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp
2024-07-08T16:56:50.8241618Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id
2024-07-08T16:56:50.8284415Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii
2024-07-08T16:56:50.8285955Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c
2024-07-08T16:56:50.8287381Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu
2024-07-08T16:56:50.8288848Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c
2024-07-08T16:56:50.8290254Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx
2024-07-08T16:56:50.8291652Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin
2024-07-08T16:56:50.8293047Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin
2024-07-08T16:56:50.8294577Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c
2024-07-08T16:56:50.8295928Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o
2024-07-08T16:56:50.8297203Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin
2024-07-08T16:56:50.8298439Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c
2024-07-08T16:56:50.8299658Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin
2024-07-08T16:56:50.8300900Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c
2024-07-08T16:56:50.8302094Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o
2024-07-08T16:56:50.8303317Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu
2024-07-08T16:56:50.8352203Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out
2024-07-08T16:56:50.8409191Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin
2024-07-08T16:56:50.8410380Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake
2024-07-08T16:56:50.8411383Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2024-07-08T16:56:50.8412252Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2024-07-08T16:56:50.8413158Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache
2024-07-08T16:56:50.8414098Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2024-07-08T16:56:50.8415136Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts
2024-07-08T16:56:50.8416293Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make
2024-07-08T16:56:50.8417400Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make
2024-07-08T16:56:50.8418457Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt
2024-07-08T16:56:50.8419536Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake
2024-07-08T16:56:50.8420636Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make
2024-07-08T16:56:50.8421711Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake
2024-07-08T16:56:50.8422796Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make
2024-07-08T16:56:50.8423867Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make
2024-07-08T16:56:50.8436555Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d
2024-07-08T16:56:50.8573814Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o
2024-07-08T16:56:50.8574813Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2024-07-08T16:56:50.8575908Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts
2024-07-08T16:56:50.8577111Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make
2024-07-08T16:56:50.8578269Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make
2024-07-08T16:56:50.8579368Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt
2024-07-08T16:56:50.8580603Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake
2024-07-08T16:56:50.8581739Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make
2024-07-08T16:56:50.8582887Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake
2024-07-08T16:56:50.8584020Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make
2024-07-08T16:56:50.8585133Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make
2024-07-08T16:56:50.8599083Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d
2024-07-08T16:56:50.8678699Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o
2024-07-08T16:56:50.8679892Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake
2024-07-08T16:56:50.8681045Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt
2024-07-08T16:56:50.8682106Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks
2024-07-08T16:56:50.8683079Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2
2024-07-08T16:56:50.8683974Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake
2024-07-08T16:56:50.8684870Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc
2024-07-08T16:56:50.8685866Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt
2024-07-08T16:56:50.8686646Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile
2024-07-08T16:56:50.8687452Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake
2024-07-08T16:56:50.8800760Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so
2024-07-08T16:56:50.8861350Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops
2024-07-08T16:56:50.8862085Z    creating: build/custom_test_artifacts/jit-hook-build/
2024-07-08T16:56:50.8862831Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2024-07-08T16:56:50.8863655Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2024-07-08T16:56:50.8868958Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml
2024-07-08T16:56:50.8869879Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/
2024-07-08T16:56:50.8870803Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake
2024-07-08T16:56:50.8871787Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/
2024-07-08T16:56:50.8872796Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-07-08T16:56:50.8873890Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c
2024-07-08T16:56:50.8874979Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out
2024-07-08T16:56:50.8875964Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-07-08T16:56:50.8876950Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-07-08T16:56:50.8878091Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp
2024-07-08T16:56:50.8879248Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out
2024-07-08T16:56:50.8880377Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin
2024-07-08T16:56:50.8881578Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake
2024-07-08T16:56:50.8882858Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin
2024-07-08T16:56:50.8884014Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake
2024-07-08T16:56:50.8885034Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-07-08T16:56:50.8886163Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-07-08T16:56:50.8923016Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii
2024-07-08T16:56:50.8961931Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp
2024-07-08T16:56:50.8963441Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id
2024-07-08T16:56:50.9005545Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii
2024-07-08T16:56:50.9007066Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c
2024-07-08T16:56:50.9008595Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu
2024-07-08T16:56:50.9010038Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c
2024-07-08T16:56:50.9011448Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx
2024-07-08T16:56:50.9012814Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin
2024-07-08T16:56:50.9014184Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin
2024-07-08T16:56:50.9015553Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c
2024-07-08T16:56:50.9016883Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o
2024-07-08T16:56:50.9018157Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin
2024-07-08T16:56:50.9019379Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c
2024-07-08T16:56:50.9020590Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin
2024-07-08T16:56:50.9039141Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c
2024-07-08T16:56:50.9040525Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o
2024-07-08T16:56:50.9042374Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu
2024-07-08T16:56:50.9074462Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out
2024-07-08T16:56:50.9132310Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin
2024-07-08T16:56:50.9133489Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake
2024-07-08T16:56:50.9134462Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2024-07-08T16:56:50.9135310Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2024-07-08T16:56:50.9136194Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache
2024-07-08T16:56:50.9137128Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2024-07-08T16:56:50.9138323Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts
2024-07-08T16:56:50.9139494Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make
2024-07-08T16:56:50.9140609Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make
2024-07-08T16:56:50.9141646Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt
2024-07-08T16:56:50.9142728Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake
2024-07-08T16:56:50.9143823Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make
2024-07-08T16:56:50.9144919Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake
2024-07-08T16:56:50.9145998Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make
2024-07-08T16:56:50.9147079Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make
2024-07-08T16:56:50.9160159Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d
2024-07-08T16:56:50.9222059Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o
2024-07-08T16:56:50.9224106Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake
2024-07-08T16:56:50.9226016Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt
2024-07-08T16:56:50.9227740Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks
2024-07-08T16:56:50.9229345Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2
2024-07-08T16:56:50.9230939Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake
2024-07-08T16:56:50.9232516Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc
2024-07-08T16:56:50.9233352Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt
2024-07-08T16:56:50.9234107Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile
2024-07-08T16:56:50.9234898Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake
2024-07-08T16:56:50.9276497Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks
2024-07-08T16:56:50.9277240Z    creating: build/custom_test_artifacts/custom-backend-build/
2024-07-08T16:56:50.9277989Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2024-07-08T16:56:50.9278875Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2024-07-08T16:56:50.9284148Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml
2024-07-08T16:56:50.9285126Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/
2024-07-08T16:56:50.9286225Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake
2024-07-08T16:56:50.9287264Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/
2024-07-08T16:56:50.9288275Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/
2024-07-08T16:56:50.9289417Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c
2024-07-08T16:56:50.9290562Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out
2024-07-08T16:56:50.9291775Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/
2024-07-08T16:56:50.9292912Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/
2024-07-08T16:56:50.9294112Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp
2024-07-08T16:56:50.9295442Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out
2024-07-08T16:56:50.9296625Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin
2024-07-08T16:56:50.9297799Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake
2024-07-08T16:56:50.9299002Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin
2024-07-08T16:56:50.9300205Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake
2024-07-08T16:56:50.9301281Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/
2024-07-08T16:56:50.9302321Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/
2024-07-08T16:56:50.9337817Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii
2024-07-08T16:56:50.9375816Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp
2024-07-08T16:56:50.9377427Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id
2024-07-08T16:56:50.9419978Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii
2024-07-08T16:56:50.9421436Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c
2024-07-08T16:56:50.9422895Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu
2024-07-08T16:56:50.9424407Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c
2024-07-08T16:56:50.9425878Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx
2024-07-08T16:56:50.9427310Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin
2024-07-08T16:56:50.9428748Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin
2024-07-08T16:56:50.9430174Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c
2024-07-08T16:56:50.9431552Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o
2024-07-08T16:56:50.9432883Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin
2024-07-08T16:56:50.9434159Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c
2024-07-08T16:56:50.9435428Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin
2024-07-08T16:56:50.9436717Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c
2024-07-08T16:56:50.9437945Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o
2024-07-08T16:56:50.9439203Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu
2024-07-08T16:56:50.9490967Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out
2024-07-08T16:56:50.9548353Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin
2024-07-08T16:56:50.9549604Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake
2024-07-08T16:56:50.9550767Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2024-07-08T16:56:50.9551682Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2024-07-08T16:56:50.9552655Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache
2024-07-08T16:56:50.9553682Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2024-07-08T16:56:50.9554818Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts
2024-07-08T16:56:50.9556073Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make
2024-07-08T16:56:50.9557274Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make
2024-07-08T16:56:50.9558415Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt
2024-07-08T16:56:50.9559595Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake
2024-07-08T16:56:50.9560979Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make
2024-07-08T16:56:50.9562175Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake
2024-07-08T16:56:50.9563347Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make
2024-07-08T16:56:50.9564527Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make
2024-07-08T16:56:50.9565909Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d
2024-07-08T16:56:50.9680974Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o
2024-07-08T16:56:50.9682155Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2024-07-08T16:56:50.9683336Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts
2024-07-08T16:56:50.9684636Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make
2024-07-08T16:56:50.9686002Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make
2024-07-08T16:56:50.9687205Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt
2024-07-08T16:56:50.9688449Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake
2024-07-08T16:56:50.9689676Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make
2024-07-08T16:56:50.9690918Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake
2024-07-08T16:56:50.9692155Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make
2024-07-08T16:56:50.9693381Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make
2024-07-08T16:56:50.9706706Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d
2024-07-08T16:56:50.9760078Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o
2024-07-08T16:56:50.9761434Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake
2024-07-08T16:56:50.9762576Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt
2024-07-08T16:56:50.9763613Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks
2024-07-08T16:56:50.9764723Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2
2024-07-08T16:56:50.9765820Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake
2024-07-08T16:56:50.9766789Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc
2024-07-08T16:56:50.9767718Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt
2024-07-08T16:56:50.9768573Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile
2024-07-08T16:56:50.9769437Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake
2024-07-08T16:56:50.9867546Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so
2024-07-08T16:56:50.9908898Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend
2024-07-08T16:56:50.9909544Z    creating: build/lib/
2024-07-08T16:56:50.9995282Z   inflating: build/lib/libprotobuf-lite.a
2024-07-08T16:56:51.0433668Z   inflating: build/lib/libprotobuf.a
2024-07-08T16:56:51.0443119Z   inflating: build/lib/libpthreadpool.a
2024-07-08T16:56:51.0451266Z   inflating: build/lib/libcpuinfo.a
2024-07-08T16:56:51.0459067Z   inflating: build/lib/libcpuinfo_internals.a
2024-07-08T16:56:51.0459564Z   inflating: build/lib/libclog.a
2024-07-08T16:56:51.0477315Z   inflating: build/lib/libnnpack.a
2024-07-08T16:56:51.0479458Z   inflating: build/lib/libnnpack_reference_layers.a
2024-07-08T16:56:51.0542214Z   inflating: build/lib/libgtest.a
2024-07-08T16:56:51.0614210Z   inflating: build/lib/libbenchmark.a
2024-07-08T16:56:51.0677026Z   inflating: build/lib/libasmjit.a
2024-07-08T16:56:51.0683820Z   inflating: build/lib/libittnotify.a
2024-07-08T16:56:51.0710691Z   inflating: build/lib/libtensorpipe_uv.a
2024-07-08T16:56:51.0833262Z   inflating: build/lib/libgloo.a
2024-07-08T16:56:51.0854366Z   inflating: build/lib/libfmt.a
2024-07-08T16:56:51.0943091Z   inflating: build/lib/libc10.so
2024-07-08T16:56:51.0944706Z   inflating: build/lib/libcaffe2_nvrtc.so
2024-07-08T16:56:51.0945193Z   inflating: build/lib/libfoxi_loader.a
2024-07-08T16:56:51.0946817Z   inflating: build/lib/libtorch_global_deps.so
2024-07-08T16:56:51.0965504Z   inflating: build/lib/libpytorch_qnnpack.a
2024-07-08T16:56:51.0983908Z   inflating: build/lib/libgmock.a
2024-07-08T16:56:51.0984474Z   inflating: build/lib/libgtest_main.a
2024-07-08T16:56:51.0985011Z   inflating: build/lib/libbenchmark_main.a
2024-07-08T16:56:51.1470769Z   inflating: build/lib/libprotoc.a
2024-07-08T16:56:51.1842077Z   inflating: build/lib/libgloo_cuda.a
2024-07-08T16:56:52.1575986Z   inflating: build/lib/libdnnl.a
2024-07-08T16:56:52.2125998Z   inflating: build/lib/libtensorpipe.a
2024-07-08T16:56:52.2178079Z   inflating: build/lib/libc10_cuda.so
2024-07-08T16:56:52.2178537Z   inflating: build/lib/libgmock_main.a
2024-07-08T16:56:52.3381700Z   inflating: build/lib/libfbgemm.a
2024-07-08T16:56:52.3627956Z   inflating: build/lib/libtensorpipe_cuda.a
2024-07-08T16:56:52.4125106Z   inflating: build/lib/libkineto.a
2024-07-08T16:56:52.4302754Z   inflating: build/lib/libXNNPACK.a
2024-07-08T16:56:52.4344252Z   inflating: build/lib/libonnx_proto.a
2024-07-08T16:56:52.5008982Z   inflating: build/lib/libonnx.a
2024-07-08T16:56:54.8328626Z   inflating: build/lib/libtorch_cpu.so
2024-07-08T16:56:54.8332524Z   inflating: build/lib/libunbox_lib.a
2024-07-08T16:56:54.8336347Z   inflating: build/lib/libshm.so
2024-07-08T16:56:56.8263697Z   inflating: build/lib/libtorch_cuda.so
2024-07-08T16:56:56.8265119Z   inflating: build/lib/libtorch.so
2024-07-08T16:56:56.8266573Z   inflating: build/lib/libc10d_cuda_test.so
2024-07-08T16:56:57.6223966Z   inflating: build/lib/libtorch_cuda_linalg.so
2024-07-08T16:56:57.8120520Z   inflating: build/lib/libtorch_python.so
2024-07-08T16:56:57.8188591Z   inflating: build/lib/libtorchbind_test.so
2024-07-08T16:56:57.8208287Z   inflating: build/lib/libjitbackend_test.so
2024-07-08T16:56:57.8232656Z   inflating: build/lib/libbackend_with_compiler.so
2024-07-08T16:56:57.8265980Z   inflating: build/lib/libnnapi_backend.so
2024-07-08T16:56:57.8266451Z    creating: build/bin/
2024-07-08T16:56:57.8314744Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test
2024-07-08T16:56:57.8364953Z   inflating: build/bin/c10_DeviceGuard_test
2024-07-08T16:56:57.8413310Z   inflating: build/bin/c10_Device_test
2024-07-08T16:56:57.8470644Z   inflating: build/bin/c10_DispatchKeySet_test
2024-07-08T16:56:57.8522413Z   inflating: build/bin/c10_Scalar_test
2024-07-08T16:56:57.8570028Z   inflating: build/bin/c10_StreamGuard_test
2024-07-08T16:56:57.8619180Z   inflating: build/bin/c10_SymInt_test
2024-07-08T16:56:57.8672421Z   inflating: build/bin/c10_InlineDeviceGuard_test
2024-07-08T16:56:57.8726130Z   inflating: build/bin/c10_InlineStreamGuard_test
2024-07-08T16:56:57.8780672Z   inflating: build/bin/c10_SizesAndStrides_test
2024-07-08T16:56:57.8849329Z   inflating: build/bin/c10_cow_test
2024-07-08T16:56:57.8900881Z   inflating: build/bin/c10_Bitset_test
2024-07-08T16:56:57.8948375Z   inflating: build/bin/c10_ConstexprCrc_test
2024-07-08T16:56:57.8996732Z   inflating: build/bin/c10_DeadlockDetection_test
2024-07-08T16:56:57.9045780Z   inflating: build/bin/c10_Half_test
2024-07-08T16:56:57.9100093Z   inflating: build/bin/c10_LeftRight_test
2024-07-08T16:56:57.9153626Z   inflating: build/bin/c10_Metaprogramming_test
2024-07-08T16:56:57.9201768Z   inflating: build/bin/c10_Synchronized_test
2024-07-08T16:56:57.9255411Z   inflating: build/bin/c10_ThreadLocal_test
2024-07-08T16:56:57.9305679Z   inflating: build/bin/c10_TypeIndex_test
2024-07-08T16:56:57.9354739Z   inflating: build/bin/c10_TypeList_test
2024-07-08T16:56:57.9402030Z   inflating: build/bin/c10_TypeTraits_test
2024-07-08T16:56:57.9453363Z   inflating: build/bin/c10_accumulate_test
2024-07-08T16:56:57.9507124Z   inflating: build/bin/c10_bfloat16_test
2024-07-08T16:56:57.9556208Z   inflating: build/bin/c10_bit_cast_test
2024-07-08T16:56:57.9611091Z   inflating: build/bin/c10_complex_math_test
2024-07-08T16:56:57.9665121Z   inflating: build/bin/c10_complex_test
2024-07-08T16:56:57.9716275Z   inflating: build/bin/c10_exception_test
2024-07-08T16:56:57.9764754Z   inflating: build/bin/c10_flags_test
2024-07-08T16:56:57.9813290Z   inflating: build/bin/c10_generic_math_test
2024-07-08T16:56:57.9971702Z   inflating: build/bin/c10_intrusive_ptr_test
2024-07-08T16:56:58.0020902Z   inflating: build/bin/c10_irange_test
2024-07-08T16:56:58.0072674Z   inflating: build/bin/c10_lazy_test
2024-07-08T16:56:58.0128172Z   inflating: build/bin/c10_logging_test
2024-07-08T16:56:58.0202098Z   inflating: build/bin/c10_optional_test
2024-07-08T16:56:58.0262721Z   inflating: build/bin/c10_ordered_preserving_dict_test
2024-07-08T16:56:58.0314919Z   inflating: build/bin/c10_registry_test
2024-07-08T16:56:58.0460925Z   inflating: build/bin/c10_small_vector_test
2024-07-08T16:56:58.0510390Z   inflating: build/bin/c10_ssize_test
2024-07-08T16:56:58.0560570Z   inflating: build/bin/c10_string_util_test
2024-07-08T16:56:58.0617466Z   inflating: build/bin/c10_string_view_test
2024-07-08T16:56:58.0666364Z   inflating: build/bin/c10_tempfile_test
2024-07-08T16:56:58.0713412Z   inflating: build/bin/c10_intrusive_ptr_benchmark
2024-07-08T16:56:58.0767786Z   inflating: build/bin/c10_typeid_test
2024-07-08T16:56:58.1200599Z   inflating: build/bin/protoc-3.13.0.0
2024-07-08T16:56:58.1634724Z   inflating: build/bin/protoc
2024-07-08T16:56:58.1685876Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test
2024-07-08T16:56:58.1737132Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream
2024-07-08T16:56:58.1788806Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device
2024-07-08T16:56:58.1839087Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes
2024-07-08T16:56:58.1889663Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads
2024-07-08T16:56:58.1940543Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks
2024-07-08T16:56:58.1988778Z   inflating: build/bin/c10_cuda_CUDATest
2024-07-08T16:56:58.2040675Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block
2024-07-08T16:56:58.2358928Z   inflating: build/bin/vec_test_all_types_DEFAULT
2024-07-08T16:56:58.2691288Z   inflating: build/bin/vec_test_all_types_AVX512
2024-07-08T16:56:58.3037265Z   inflating: build/bin/vec_test_all_types_AVX2
2024-07-08T16:56:58.3087952Z   inflating: build/bin/BackoffTest
2024-07-08T16:56:58.3139031Z   inflating: build/bin/FileStoreTest
2024-07-08T16:56:58.3193333Z   inflating: build/bin/TCPStoreTest
2024-07-08T16:56:58.3245118Z   inflating: build/bin/HashStoreTest
2024-07-08T16:56:58.3257823Z   inflating: build/bin/ProcessGroupMPITest
2024-07-08T16:56:58.3310355Z   inflating: build/bin/test_edge_op_registration
2024-07-08T16:56:58.3314236Z   inflating: build/bin/torch_shm_manager
2024-07-08T16:56:58.3316752Z   inflating: build/bin/example_allreduce
2024-07-08T16:56:58.3370138Z   inflating: build/bin/test_dist_autograd
2024-07-08T16:56:58.3437185Z   inflating: build/bin/test_cpp_rpc
2024-07-08T16:56:58.3439508Z   inflating: build/bin/parallel_benchmark
2024-07-08T16:56:58.3504515Z   inflating: build/bin/test_mobile_nnc
2024-07-08T16:56:58.3513686Z   inflating: build/bin/aot_model_compiler_test
2024-07-08T16:56:58.3846000Z   inflating: build/bin/test_lazy
2024-07-08T16:56:58.4991597Z   inflating: build/bin/test_api
2024-07-08T16:56:58.5063517Z   inflating: build/bin/Dict_test
2024-07-08T16:56:58.5113611Z   inflating: build/bin/Dimname_test
2024-07-08T16:56:58.5176067Z   inflating: build/bin/MaybeOwned_test
2024-07-08T16:56:58.5232085Z   inflating: build/bin/NamedTensor_test
2024-07-08T16:56:58.5289037Z   inflating: build/bin/apply_utils_test
2024-07-08T16:56:58.5346375Z   inflating: build/bin/atest
2024-07-08T16:56:58.5407283Z   inflating: build/bin/basic
2024-07-08T16:56:58.5460386Z   inflating: build/bin/broadcast_test
2024-07-08T16:56:58.5509833Z   inflating: build/bin/cpu_allocator_test
2024-07-08T16:56:58.5565826Z   inflating: build/bin/cpu_generator_test
2024-07-08T16:56:58.5617138Z   inflating: build/bin/cpu_profiling_allocator_test
2024-07-08T16:56:58.5707296Z   inflating: build/bin/cpu_rng_test
2024-07-08T16:56:58.5755836Z   inflating: build/bin/dispatch_key_set_test
2024-07-08T16:56:58.5804245Z   inflating: build/bin/dlconvertor_test
2024-07-08T16:56:58.5860933Z   inflating: build/bin/extension_backend_test
2024-07-08T16:56:58.5914326Z   inflating: build/bin/half_test
2024-07-08T16:56:58.6007478Z   inflating: build/bin/ivalue_test
2024-07-08T16:56:58.6055656Z   inflating: build/bin/lazy_tensor_test
2024-07-08T16:56:58.6108952Z   inflating: build/bin/math_kernel_test
2024-07-08T16:56:58.6161598Z   inflating: build/bin/memory_format_test
2024-07-08T16:56:58.6212696Z   inflating: build/bin/memory_overlapping_test
2024-07-08T16:56:58.6264347Z   inflating: build/bin/mobile_memory_cleanup
2024-07-08T16:56:58.6318775Z   inflating: build/bin/native_test
2024-07-08T16:56:58.6367714Z   inflating: build/bin/operator_name_test
2024-07-08T16:56:58.6417676Z   inflating: build/bin/operators_test
2024-07-08T16:56:58.6468254Z   inflating: build/bin/packedtensoraccessor_test
2024-07-08T16:56:58.6533067Z   inflating: build/bin/pow_test
2024-07-08T16:56:58.6588451Z   inflating: build/bin/quantized_test
2024-07-08T16:56:58.6637725Z   inflating: build/bin/reduce_ops_test
2024-07-08T16:56:58.6686959Z   inflating: build/bin/reportMemoryUsage_test
2024-07-08T16:56:58.6741630Z   inflating: build/bin/scalar_tensor_test
2024-07-08T16:56:58.6798499Z   inflating: build/bin/scalar_test
2024-07-08T16:56:58.6849005Z   inflating: build/bin/StorageUtils_test
2024-07-08T16:56:58.6899993Z   inflating: build/bin/stride_properties_test
2024-07-08T16:56:58.6976216Z   inflating: build/bin/tensor_iterator_test
2024-07-08T16:56:58.7028559Z   inflating: build/bin/test_parallel
2024-07-08T16:56:58.7031207Z   inflating: build/bin/thread_init_test
2024-07-08T16:56:58.7085398Z   inflating: build/bin/type_ptr_test
2024-07-08T16:56:58.7143516Z   inflating: build/bin/type_test
2024-07-08T16:56:58.7194593Z   inflating: build/bin/undefined_tensor_test
2024-07-08T16:56:58.7195265Z   inflating: build/bin/verify_api_visibility
2024-07-08T16:56:58.7263012Z   inflating: build/bin/legacy_vmap_test
2024-07-08T16:56:58.7313068Z   inflating: build/bin/weakref_test
2024-07-08T16:56:58.7363042Z   inflating: build/bin/wrapdim_test
2024-07-08T16:56:58.7413350Z   inflating: build/bin/xla_tensor_test
2024-07-08T16:56:58.7471455Z   inflating: build/bin/IListRef_test
2024-07-08T16:56:58.7574986Z   inflating: build/bin/List_test
2024-07-08T16:56:58.7639039Z   inflating: build/bin/KernelFunction_test
2024-07-08T16:56:58.7757739Z   inflating: build/bin/kernel_function_legacy_test
2024-07-08T16:56:58.7850813Z   inflating: build/bin/kernel_function_test
2024-07-08T16:56:58.7975132Z   inflating: build/bin/kernel_lambda_legacy_test
2024-07-08T16:56:58.8075560Z   inflating: build/bin/kernel_lambda_test
2024-07-08T16:56:58.8134549Z   inflating: build/bin/kernel_stackbased_test
2024-07-08T16:56:58.8228109Z   inflating: build/bin/make_boxed_from_unboxed_functor_test
2024-07-08T16:56:58.8277231Z   inflating: build/bin/CppSignature_test
2024-07-08T16:56:58.8324484Z   inflating: build/bin/op_allowlist_test
2024-07-08T16:56:58.8378603Z   inflating: build/bin/backend_fallback_test
2024-07-08T16:56:58.8671255Z   inflating: build/bin/op_registration_test
2024-07-08T16:56:58.8732824Z   inflating: build/bin/inline_container_test
2024-07-08T16:56:58.8784317Z   inflating: build/bin/cuda_apply_test
2024-07-08T16:56:58.8833449Z   inflating: build/bin/cuda_allocator_test
2024-07-08T16:56:58.8890708Z   inflating: build/bin/cuda_atomic_ops_test
2024-07-08T16:56:58.8943655Z   inflating: build/bin/cuda_caching_host_allocator_test
2024-07-08T16:56:58.9011367Z   inflating: build/bin/cuda_complex_math_test
2024-07-08T16:56:58.9069330Z   inflating: build/bin/cuda_complex_test
2024-07-08T16:56:58.9117498Z   inflating: build/bin/cuda_device_test
2024-07-08T16:56:58.9172549Z   inflating: build/bin/cuda_cub_test
2024-07-08T16:56:58.9221884Z   inflating: build/bin/cuda_dlconvertor_test
2024-07-08T16:56:58.9284848Z   inflating: build/bin/cuda_distributions_test
2024-07-08T16:56:58.9339759Z   inflating: build/bin/cuda_generator_test
2024-07-08T16:56:58.9388482Z   inflating: build/bin/cuda_half_test
2024-07-08T16:56:58.9437366Z   inflating: build/bin/cuda_integer_divider_test
2024-07-08T16:56:58.9485000Z   inflating: build/bin/cuda_optional_test
2024-07-08T16:56:58.9535093Z   inflating: build/bin/cuda_packedtensoraccessor_test
2024-07-08T16:56:58.9586775Z   inflating: build/bin/cuda_reportMemoryUsage_test
2024-07-08T16:56:58.9635406Z   inflating: build/bin/cuda_allocatorTraceTracker_test
2024-07-08T16:56:58.9693682Z   inflating: build/bin/cuda_stream_test
2024-07-08T16:56:58.9741452Z   inflating: build/bin/cuda_cudnn_test
2024-07-08T16:56:58.9792498Z   inflating: build/bin/cuda_vectorized_test
2024-07-08T16:56:58.9806490Z   inflating: build/bin/tutorial_tensorexpr
2024-07-08T16:56:58.9870494Z   inflating: build/bin/ProcessGroupGlooTest
2024-07-08T16:56:58.9925801Z   inflating: build/bin/ProcessGroupGlooAsyncTest
2024-07-08T16:56:58.9987147Z   inflating: build/bin/ProcessGroupNCCLTest
2024-07-08T16:56:59.0047345Z   inflating: build/bin/ProcessGroupNCCLErrorsTest
2024-07-08T16:56:59.0853970Z   inflating: build/bin/test_tensorexpr
2024-07-08T16:56:59.1414469Z   inflating: build/bin/test_jit
2024-07-08T16:56:59.1415141Z    creating: .additional_ci_files/
2024-07-08T16:56:59.1470376Z   inflating: .additional_ci_files/test-times.json
2024-07-08T16:56:59.1689462Z   inflating: .additional_ci_files/test-class-times.json
2024-07-08T16:56:59.1714925Z ##[group]Run rm artifacts.zip
2024-07-08T16:56:59.1715341Z [36;1mrm artifacts.zip[0m
2024-07-08T16:56:59.1724256Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:56:59.1724766Z env:
2024-07-08T16:56:59.1725062Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:56:59.1725524Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:56:59.1726249Z ##[endgroup]
2024-07-08T16:56:59.2426255Z ##[group]Run df -H
2024-07-08T16:56:59.2426589Z [36;1mdf -H[0m
2024-07-08T16:56:59.2434101Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:56:59.2434595Z env:
2024-07-08T16:56:59.2434882Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:56:59.2435342Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:56:59.2435831Z ##[endgroup]
2024-07-08T16:56:59.2462433Z Filesystem      Size  Used Avail Use% Mounted on
2024-07-08T16:56:59.2462999Z devtmpfs         34G     0   34G   0% /dev
2024-07-08T16:56:59.2463698Z tmpfs            34G     0   34G   0% /dev/shm
2024-07-08T16:56:59.2464169Z tmpfs            34G  455k   34G   1% /run
2024-07-08T16:56:59.2464657Z tmpfs            34G     0   34G   0% /sys/fs/cgroup
2024-07-08T16:56:59.2465312Z /dev/nvme0n1p1  162G   43G  119G  27% /
2024-07-08T16:56:59.2503343Z Prepare all required actions
2024-07-08T16:56:59.2503787Z Getting action download info
2024-07-08T16:56:59.3592729Z ##[group]Run ./.github/actions/download-td-artifacts
2024-07-08T16:56:59.3593199Z with:
2024-07-08T16:56:59.3593471Z env:
2024-07-08T16:56:59.3593760Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:56:59.3594211Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:56:59.3594701Z ##[endgroup]
2024-07-08T16:56:59.3626182Z ##[group]Run seemethere/download-artifact-s3@v4
2024-07-08T16:56:59.3626618Z with:
2024-07-08T16:56:59.3626898Z   name: td_results
2024-07-08T16:56:59.3627219Z   s3-bucket: gha-artifacts
2024-07-08T16:56:59.3627582Z   region: us-east-1
2024-07-08T16:56:59.3627880Z env:
2024-07-08T16:56:59.3628166Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:56:59.3628631Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:56:59.3629118Z ##[endgroup]
2024-07-08T16:56:59.7678048Z (node:28537) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2024-07-08T16:56:59.7679407Z
2024-07-08T16:56:59.7679937Z Please migrate your code to use AWS SDK for JavaScript (v3).
2024-07-08T16:56:59.7681495Z For more information, check the migration guide at https://a.co/7PzMCcy
2024-07-08T16:56:59.7683129Z (Use `node --trace-warnings ...` to show where the warning was created)
2024-07-08T16:56:59.8445829Z Found 0 objects with prefix pytorch/pytorch/9843060221/td_results/
2024-07-08T16:56:59.8452343Z Artifact download has finished successfully
2024-07-08T16:56:59.8591975Z ##[group]Run mkdir -p .additional_ci_files
2024-07-08T16:56:59.8592464Z [36;1mmkdir -p .additional_ci_files[0m
2024-07-08T16:56:59.8593040Z [36;1mmv td_results.json .additional_ci_files/td_results.json[0m
2024-07-08T16:56:59.8600875Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:56:59.8601381Z env:
2024-07-08T16:56:59.8601655Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:56:59.8602112Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:56:59.8602587Z ##[endgroup]
2024-07-08T16:56:59.8636866Z mv: cannot stat 'td_results.json': No such file or directory
2024-07-08T16:56:59.8649160Z ##[error]Process completed with exit code 1.
2024-07-08T16:56:59.8680863Z ##[group]Run .github/scripts/parse_ref.py
2024-07-08T16:56:59.8681350Z [36;1m.github/scripts/parse_ref.py[0m
2024-07-08T16:56:59.8688584Z shell: /usr/bin/bash -e {0}
2024-07-08T16:56:59.8688943Z env:
2024-07-08T16:56:59.8689217Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:56:59.8689675Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:56:59.8690147Z ##[endgroup]
2024-07-08T16:56:59.8916649Z Prepare all required actions
2024-07-08T16:56:59.8953360Z ##[group]Run ./.github/actions/get-workflow-job-id
2024-07-08T16:56:59.8953816Z with:
2024-07-08T16:56:59.8954371Z   github-token: ***
2024-07-08T16:56:59.8954678Z env:
2024-07-08T16:56:59.8954958Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:56:59.8955416Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:56:59.8955901Z ##[endgroup]
2024-07-08T16:56:59.8971382Z ##[group]Run set -eux
2024-07-08T16:56:59.8971721Z [36;1mset -eux[0m
2024-07-08T16:56:59.8972310Z [36;1mpython3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"[0m
2024-07-08T16:56:59.8979824Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:56:59.8980315Z env:
2024-07-08T16:56:59.8980601Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:56:59.8981061Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:56:59.8981697Z   GITHUB_TOKEN: ***
2024-07-08T16:56:59.8982003Z ##[endgroup]
2024-07-08T16:56:59.9001139Z + python3 .github/scripts/get_workflow_job_id.py 9843060221 i-00726d0d1e8699347
2024-07-08T16:57:01.2598184Z setting job-id=27175320579
2024-07-08T16:57:01.2599889Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-07-08T16:57:01.2825232Z Prepare all required actions
2024-07-08T16:57:01.2825717Z Getting action download info
2024-07-08T16:57:01.3923584Z ##[group]Run ./.github/actions/filter-test-configs
2024-07-08T16:57:01.3924042Z with:
2024-07-08T16:57:01.3924489Z   github-token: ***
2024-07-08T16:57:01.3932610Z   test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2024-07-08T16:57:01.3941200Z   job-name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-07-08T16:57:01.3941918Z env:
2024-07-08T16:57:01.3942207Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:57:01.3942669Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:57:01.3943148Z ##[endgroup]
2024-07-08T16:57:01.3983445Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-07-08T16:57:01.3984118Z with:
2024-07-08T16:57:01.3984393Z   shell: bash
2024-07-08T16:57:01.3984694Z   timeout_minutes: 10
2024-07-08T16:57:01.3985022Z   max_attempts: 5
2024-07-08T16:57:01.3985330Z   retry_wait_seconds: 30
2024-07-08T16:57:01.3986449Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2024-07-08T16:57:01.3987623Z   polling_interval_seconds: 1
2024-07-08T16:57:01.3987995Z   warning_on_retry: true
2024-07-08T16:57:01.3988344Z   continue_on_error: false
2024-07-08T16:57:01.3988688Z env:
2024-07-08T16:57:01.3988969Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:57:01.3989422Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:57:01.3990056Z   GITHUB_TOKEN: ***
2024-07-08T16:57:01.3990377Z ##[endgroup]
2024-07-08T16:57:01.4480830Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2024-07-08T16:57:01.6629513Z Defaulting to user installation because normal site-packages is not writeable
2024-07-08T16:57:01.6779903Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1)
2024-07-08T16:57:01.6926180Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1)
2024-07-08T16:57:01.6934550Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19)
2024-07-08T16:57:01.7134360Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7)
2024-07-08T16:57:01.7145553Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12)
2024-07-08T16:57:01.7166645Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.7.4)
2024-07-08T16:57:02.4479817Z Command completed after 1 attempt(s).
2024-07-08T16:57:02.4519820Z ##[group]Run set -x
2024-07-08T16:57:02.4520154Z [36;1mset -x[0m
2024-07-08T16:57:02.4520452Z [36;1m[0m
2024-07-08T16:57:02.4521124Z [36;1m# Use relative path here as this could be checked out anywhere, not necessarily[0m
2024-07-08T16:57:02.4521797Z [36;1m# in runner workspace[0m
2024-07-08T16:57:02.4522328Z [36;1mpython3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"[0m
2024-07-08T16:57:02.4530411Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:57:02.4530905Z env:
2024-07-08T16:57:02.4531176Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:57:02.4531636Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:57:02.4532121Z ##[endgroup]
2024-07-08T16:57:02.4556814Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2024-07-08T16:57:02.4752449Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2024-07-08T16:57:02.4753033Z [36;1mecho "Workflow: ${GITHUB_WORKFLOW}"[0m
2024-07-08T16:57:02.4753506Z [36;1mecho "Job name: ${JOB_NAME}"[0m
2024-07-08T16:57:02.4753916Z [36;1m[0m
2024-07-08T16:57:02.4754487Z [36;1m# Use relative path here as this could be checked out anywhere, not necessarily[0m
2024-07-08T16:57:02.4755174Z [36;1m# in runner workspace[0m
2024-07-08T16:57:02.4755803Z [36;1mpython3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \[0m
2024-07-08T16:57:02.4756446Z [36;1m  --workflow "${GITHUB_WORKFLOW}" \[0m
2024-07-08T16:57:02.4756908Z [36;1m  --job-name "${JOB_NAME}" \[0m
2024-07-08T16:57:02.4765371Z [36;1m  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \[0m
2024-07-08T16:57:02.4774207Z [36;1m  --selected-test-configs "" \[0m
2024-07-08T16:57:02.4774675Z [36;1m  --pr-number "${PR_NUMBER}" \[0m
2024-07-08T16:57:02.4775103Z [36;1m  --tag "${TAG}" \[0m
2024-07-08T16:57:02.4775510Z [36;1m  --event-name "${EVENT_NAME}" \[0m
2024-07-08T16:57:02.4776005Z [36;1m  --schedule "${SCHEDULE}" \[0m
2024-07-08T16:57:02.4776453Z [36;1m  --branch "${HEAD_BRANCH}"[0m
2024-07-08T16:57:02.4783680Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:57:02.4784176Z env:
2024-07-08T16:57:02.4784466Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:57:02.4784941Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:57:02.4785631Z   GITHUB_TOKEN: ***
2024-07-08T16:57:02.4786294Z   JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-07-08T16:57:02.4787024Z   PR_NUMBER:
2024-07-08T16:57:02.4787323Z   TAG:
2024-07-08T16:57:02.4787611Z   EVENT_NAME: push
2024-07-08T16:57:02.4787923Z   SCHEDULE:
2024-07-08T16:57:02.4788219Z   HEAD_BRANCH:
2024-07-08T16:57:02.4788523Z ##[endgroup]
2024-07-08T16:57:02.4807790Z Workflow: inductor
2024-07-08T16:57:02.4808772Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-07-08T16:57:02.6494668Z ##[group]Run echo "Filtered matrix:"
2024-07-08T16:57:02.6495132Z [36;1mecho "Filtered matrix:"[0m
2024-07-08T16:57:02.6503239Z [36;1mecho "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}"[0m
2024-07-08T16:57:02.6511760Z [36;1m[0m
2024-07-08T16:57:02.6512029Z [36;1mecho[0m
2024-07-08T16:57:02.6512409Z [36;1mecho "Is the current job unstable? False"[0m
2024-07-08T16:57:02.6512944Z [36;1m[0m
2024-07-08T16:57:02.6513220Z [36;1mecho[0m
2024-07-08T16:57:02.6513583Z [36;1mecho "Is keep-going label set? False"[0m
2024-07-08T16:57:02.6514024Z [36;1m[0m
2024-07-08T16:57:02.6514295Z [36;1mecho[0m
2024-07-08T16:57:02.6514617Z [36;1mecho "Renabled issues? "[0m
2024-07-08T16:57:02.6522512Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:57:02.6523015Z env:
2024-07-08T16:57:02.6523298Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:57:02.6523754Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:57:02.6524233Z ##[endgroup]
2024-07-08T16:57:02.6544159Z Filtered matrix:
2024-07-08T16:57:02.6554002Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]}
2024-07-08T16:57:02.6562152Z
2024-07-08T16:57:02.6562305Z Is the current job unstable? False
2024-07-08T16:57:02.6562591Z
2024-07-08T16:57:02.6562867Z Is keep-going label set? False
2024-07-08T16:57:02.6563123Z
2024-07-08T16:57:02.6563256Z Renabled issues?
2024-07-08T16:57:02.6617342Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2024-07-08T16:57:02.6618209Z [36;1mecho "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"[0m
2024-07-08T16:57:02.6625921Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:57:02.6626408Z env:
2024-07-08T16:57:02.6626691Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:57:02.6627150Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:57:02.6627633Z   JOB_TIMEOUT: 240
2024-07-08T16:57:02.6627929Z ##[endgroup]
2024-07-08T16:57:02.6695402Z ##[group]Run set -x
2024-07-08T16:57:02.6695885Z [36;1mset -x[0m
2024-07-08T16:57:02.6696186Z [36;1m[0m
2024-07-08T16:57:02.6696545Z [36;1mif [[ $TEST_CONFIG == 'multigpu' ]]; then[0m
2024-07-08T16:57:02.6697089Z [36;1m  TEST_COMMAND=.ci/pytorch/multigpu-test.sh[0m
2024-07-08T16:57:02.6697644Z [36;1melif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then[0m
2024-07-08T16:57:02.6698151Z [36;1m  TEST_COMMAND=.ci/onnx/test.sh[0m
2024-07-08T16:57:02.6698560Z [36;1melse[0m
2024-07-08T16:57:02.6698925Z [36;1m  TEST_COMMAND=.ci/pytorch/test.sh[0m
2024-07-08T16:57:02.6699337Z [36;1mfi[0m
2024-07-08T16:57:02.6699612Z [36;1m[0m
2024-07-08T16:57:02.6700079Z [36;1m# detached container should get cleaned up by teardown_ec2_linux[0m
2024-07-08T16:57:02.6700830Z [36;1m# TODO: Stop building test binaries as part of the build phase[0m
2024-07-08T16:57:02.6701502Z [36;1m# Used for GPU_FLAG since that doesn't play nice[0m
2024-07-08T16:57:02.6702085Z [36;1m# shellcheck disable=SC2086,SC2090[0m
2024-07-08T16:57:02.6702552Z [36;1mcontainer_name=$(docker run \[0m
2024-07-08T16:57:02.6702962Z [36;1m  ${GPU_FLAG:-} \[0m
2024-07-08T16:57:02.6703337Z [36;1m  -e BUILD_ENVIRONMENT \[0m
2024-07-08T16:57:02.6703735Z [36;1m  -e PR_NUMBER \[0m
2024-07-08T16:57:02.6704090Z [36;1m  -e GITHUB_ACTIONS \[0m
2024-07-08T16:57:02.6704476Z [36;1m  -e GITHUB_REPOSITORY \[0m
2024-07-08T16:57:02.6704878Z [36;1m  -e GITHUB_WORKFLOW \[0m
2024-07-08T16:57:02.6705258Z [36;1m  -e GITHUB_JOB \[0m
2024-07-08T16:57:02.6705614Z [36;1m  -e GITHUB_RUN_ID \[0m
2024-07-08T16:57:02.6705998Z [36;1m  -e GITHUB_RUN_NUMBER \[0m
2024-07-08T16:57:02.6706399Z [36;1m  -e GITHUB_RUN_ATTEMPT \[0m
2024-07-08T16:57:02.6706791Z [36;1m  -e JOB_ID \[0m
2024-07-08T16:57:02.6707119Z [36;1m  -e JOB_NAME \[0m
2024-07-08T16:57:02.6707474Z [36;1m  -e BASE_SHA \[0m
2024-07-08T16:57:02.6707811Z [36;1m  -e BRANCH \[0m
2024-07-08T16:57:02.6708139Z [36;1m  -e SHA1 \[0m
2024-07-08T16:57:02.6708475Z [36;1m  -e AWS_DEFAULT_REGION \[0m
2024-07-08T16:57:02.6708877Z [36;1m  -e IN_WHEEL_TEST \[0m
2024-07-08T16:57:02.6709248Z [36;1m  -e SHARD_NUMBER \[0m
2024-07-08T16:57:02.6709604Z [36;1m  -e TEST_CONFIG \[0m
2024-07-08T16:57:02.6709969Z [36;1m  -e NUM_TEST_SHARDS \[0m
2024-07-08T16:57:02.6710360Z [36;1m  -e REENABLED_ISSUES \[0m
2024-07-08T16:57:02.6710776Z [36;1m  -e CONTINUE_THROUGH_ERROR \[0m
2024-07-08T16:57:02.6711193Z [36;1m  -e VERBOSE_TEST_LOGS \[0m
2024-07-08T16:57:02.6711591Z [36;1m  -e NO_TEST_TIMEOUT \[0m
2024-07-08T16:57:02.6711975Z [36;1m  -e NO_TD \[0m
2024-07-08T16:57:02.6712313Z [36;1m  -e TD_DISTRIBUTED \[0m
2024-07-08T16:57:02.6712683Z [36;1m  -e PR_LABELS \[0m
2024-07-08T16:57:02.6713089Z [36;1m  -e MAX_JOBS="$(nproc --ignore=2)" \[0m
2024-07-08T16:57:02.6713540Z [36;1m  -e SCCACHE_BUCKET \[0m
2024-07-08T16:57:02.6713931Z [36;1m  -e SCCACHE_S3_KEY_PREFIX \[0m
2024-07-08T16:57:02.6714329Z [36;1m  -e XLA_CUDA \[0m
2024-07-08T16:57:02.6714726Z [36;1m  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \[0m
2024-07-08T16:57:02.6715238Z [36;1m  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \[0m
2024-07-08T16:57:02.6715761Z [36;1m  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \[0m
2024-07-08T16:57:02.6716267Z [36;1m  -e SKIP_SCCACHE_INITIALIZATION=1 \[0m
2024-07-08T16:57:02.6716735Z [36;1m  -e HUGGING_FACE_HUB_TOKEN \[0m
2024-07-08T16:57:02.6717154Z [36;1m  -e DASHBOARD_TAG \[0m
2024-07-08T16:57:02.6717625Z [36;1m  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \[0m
2024-07-08T16:57:02.6718169Z [36;1m  --security-opt seccomp=unconfined \[0m
2024-07-08T16:57:02.6718746Z [36;1m  --cap-add=SYS_PTRACE \[0m
2024-07-08T16:57:02.6719141Z [36;1m  --ipc=host \[0m
2024-07-08T16:57:02.6719508Z [36;1m  --shm-size="${SHM_SIZE}" \[0m
2024-07-08T16:57:02.6719905Z [36;1m  --tty \[0m
2024-07-08T16:57:02.6720219Z [36;1m  --detach \[0m
2024-07-08T16:57:02.6720575Z [36;1m  --name="${container_name}" \[0m
2024-07-08T16:57:02.6721105Z [36;1m  --user jenkins \[0m
2024-07-08T16:57:02.6721658Z [36;1m  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \[0m
2024-07-08T16:57:02.6722216Z [36;1m  -w /var/lib/jenkins/workspace \[0m
2024-07-08T16:57:02.6722656Z [36;1m  "${DOCKER_IMAGE}"[0m
2024-07-08T16:57:02.6722996Z [36;1m)[0m
2024-07-08T16:57:02.6723406Z [36;1m# Propagate download.pytorch.org IP to container[0m
2024-07-08T16:57:02.6724314Z [36;1mgrep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts"[0m
2024-07-08T16:57:02.6725285Z [36;1mecho "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"[0m
2024-07-08T16:57:02.6726492Z [36;1mdocker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"[0m
2024-07-08T16:57:02.6733802Z shell: /usr/bin/bash -e {0}
2024-07-08T16:57:02.6734164Z env:
2024-07-08T16:57:02.6734445Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:57:02.6734909Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:57:02.6735524Z   BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-07-08T16:57:02.6736039Z   PR_NUMBER:
2024-07-08T16:57:02.6736378Z   GITHUB_REPOSITORY: pytorch/pytorch
2024-07-08T16:57:02.6736813Z   GITHUB_WORKFLOW: inductor
2024-07-08T16:57:02.6737173Z   GITHUB_JOB: test
2024-07-08T16:57:02.6737507Z   GITHUB_RUN_ID: 9843060221
2024-07-08T16:57:02.6737877Z   GITHUB_RUN_NUMBER: 78798
2024-07-08T16:57:02.6738231Z   GITHUB_RUN_ATTEMPT: 1
2024-07-08T16:57:02.6738575Z   JOB_ID: 27175320579
2024-07-08T16:57:02.6739230Z   JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-07-08T16:57:02.6739957Z   BRANCH: main
2024-07-08T16:57:02.6740314Z   SHA1: c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:57:02.6740835Z   BASE_SHA: c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:57:02.6741320Z   TEST_CONFIG: inductor_huggingface
2024-07-08T16:57:02.6741727Z   SHARD_NUMBER: 1
2024-07-08T16:57:02.6742039Z   NUM_TEST_SHARDS: 1
2024-07-08T16:57:02.6742375Z   REENABLED_ISSUES:
2024-07-08T16:57:02.6742732Z   CONTINUE_THROUGH_ERROR: False
2024-07-08T16:57:02.6743132Z   VERBOSE_TEST_LOGS: False
2024-07-08T16:57:02.6743481Z   NO_TEST_TIMEOUT: False
2024-07-08T16:57:02.6743824Z   NO_TD: False
2024-07-08T16:57:02.6744131Z   TD_DISTRIBUTED: False
2024-07-08T16:57:02.6744550Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2024-07-08T16:57:02.6745026Z   SCCACHE_S3_KEY_PREFIX: inductor
2024-07-08T16:57:02.6745405Z   SHM_SIZE: 2g
2024-07-08T16:57:02.6746480Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:57:02.6747639Z   XLA_CUDA:
2024-07-08T16:57:02.6748118Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2024-07-08T16:57:02.6748733Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2024-07-08T16:57:02.6749183Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2024-07-08T16:57:02.6749608Z   DASHBOARD_TAG:
2024-07-08T16:57:02.6750113Z   HUGGING_FACE_HUB_TOKEN: ***
2024-07-08T16:57:02.6750487Z ##[endgroup]
2024-07-08T16:57:02.6770413Z + [[ inductor_huggingface == \m\u\l\t\i\g\p\u ]]
2024-07-08T16:57:02.6771131Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]]
2024-07-08T16:57:02.6771650Z + TEST_COMMAND=.ci/pytorch/test.sh
2024-07-08T16:57:02.6777729Z +++ nproc --ignore=2
2024-07-08T16:57:02.6790111Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9843060221 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:57:13.8429897Z + container_name=5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:57:13.8432207Z + docker exec -i 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71 sudo bash -c '/bin/cat >> /etc/hosts'
2024-07-08T16:57:13.8433354Z + grep download.pytorch.org /etc/hosts
2024-07-08T16:57:13.9133256Z + echo DOCKER_CONTAINER_ID=5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:57:13.9135616Z ++ echo dist/torch-2.5.0a0+gitc8ab2e8-cp310-cp310-linux_x86_64.whl
2024-07-08T16:57:13.9137576Z + docker exec -t 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71 sh -c 'pip install dist/torch-2.5.0a0+gitc8ab2e8-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh'
2024-07-08T16:57:14.3107523Z Processing ./dist/torch-2.5.0a0+gitc8ab2e8-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitc8ab2e8)
2024-07-08T16:57:14.6244948Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitc8ab2e8->torch==2.5.0a0+gitc8ab2e8) (3.13.1)
2024-07-08T16:57:14.6250199Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitc8ab2e8->torch==2.5.0a0+gitc8ab2e8) (4.12.2)
2024-07-08T16:57:14.6252220Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitc8ab2e8->torch==2.5.0a0+gitc8ab2e8) (1.12.1)
2024-07-08T16:57:14.6255954Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitc8ab2e8->torch==2.5.0a0+gitc8ab2e8) (2.8.8)
2024-07-08T16:57:14.6259813Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitc8ab2e8->torch==2.5.0a0+gitc8ab2e8) (3.1.4)
2024-07-08T16:57:14.6262244Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitc8ab2e8->torch==2.5.0a0+gitc8ab2e8) (2024.6.1)
2024-07-08T16:57:14.6287707Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitc8ab2e8->torch==2.5.0a0+gitc8ab2e8) (3.3.0)
2024-07-08T16:57:14.6347094Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitc8ab2e8->torch==2.5.0a0+gitc8ab2e8) (1.21.2)
2024-07-08T16:57:14.7251277Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitc8ab2e8->torch==2.5.0a0+gitc8ab2e8) (2.1.5)
2024-07-08T16:57:14.7410263Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitc8ab2e8->torch==2.5.0a0+gitc8ab2e8) (1.3.0)
2024-07-08T16:57:16.0549580Z Installing collected packages: torch
2024-07-08T16:57:24.7485228Z [31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2024-07-08T16:57:24.7487098Z timm 0.9.7 requires torchvision, which is not installed.[0m[31m
2024-07-08T16:57:24.7487771Z [0mSuccessfully installed torch-2.5.0a0+gitc8ab2e8
2024-07-08T16:57:24.8449590Z ++ dirname .ci/pytorch/test.sh
2024-07-08T16:57:24.8452703Z + source .ci/pytorch/common.sh
2024-07-08T16:57:24.8455297Z +++ dirname .ci/pytorch/common.sh
2024-07-08T16:57:24.8461475Z ++ source .ci/pytorch/common_utils.sh
2024-07-08T16:57:24.8464151Z +++ declare -f -t trap_add
2024-07-08T16:57:24.8469395Z ++ set -ex
2024-07-08T16:57:24.8471500Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]]
2024-07-08T16:57:24.8472231Z ++ BUILD_TEST_LIBTORCH=0
2024-07-08T16:57:24.8472866Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]]
2024-07-08T16:57:24.8473455Z ++ stat -c %u /var/lib/jenkins/workspace
2024-07-08T16:57:24.8481030Z + WORKSPACE_ORIGINAL_OWNER_ID=1000
2024-07-08T16:57:24.8481466Z + trap_add cleanup_workspace EXIT
2024-07-08T16:57:24.8482047Z + trap_add_cmd=cleanup_workspace
2024-07-08T16:57:24.8483030Z + shift
2024-07-08T16:57:24.8483700Z + for trap_add_name in "$@"
2024-07-08T16:57:24.8488151Z +++ trap -p EXIT
2024-07-08T16:57:24.8489351Z ++ eval 'extract_trap_cmd '
2024-07-08T16:57:24.8490097Z +++ extract_trap_cmd
2024-07-08T16:57:24.8490598Z +++ printf '%s\n' ''
2024-07-08T16:57:24.8491050Z ++ printf '%s\n' cleanup_workspace
2024-07-08T16:57:24.8492132Z + trap -- '
2024-07-08T16:57:24.8492475Z cleanup_workspace' EXIT
2024-07-08T16:57:24.8493089Z + sudo chown -R jenkins /var/lib/jenkins/workspace
2024-07-08T16:57:25.2823781Z + git config --global --add safe.directory /var/lib/jenkins/workspace
2024-07-08T16:57:25.2835578Z + echo 'Environment variables:'
2024-07-08T16:57:25.2836124Z Environment variables:
2024-07-08T16:57:25.2836453Z + env
2024-07-08T16:57:25.2841389Z INSTALLED_DB=yes
2024-07-08T16:57:25.2841936Z NV_LIBCUBLAS_VERSION=12.1.3.1-1
2024-07-08T16:57:25.2842479Z NVIDIA_VISIBLE_DEVICES=all
2024-07-08T16:57:25.2843010Z NV_NVML_DEV_VERSION=12.1.105-1
2024-07-08T16:57:25.2843847Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-07-08T16:57:25.2846302Z CONTINUE_THROUGH_ERROR=False
2024-07-08T16:57:25.2847061Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1
2024-07-08T16:57:25.2847795Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1
2024-07-08T16:57:25.2848511Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-07-08T16:57:25.2849162Z HOSTNAME=5517005639eb
2024-07-08T16:57:25.2850201Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_4a32ed44-6883-44df-af8b-38aebb8abf4f
2024-07-08T16:57:25.2851007Z GITHUB_ACTION=__self
2024-07-08T16:57:25.2851363Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-07-08T16:57:25.2855111Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
2024-07-08T16:57:25.2859839Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1
2024-07-08T16:57:25.2860353Z NV_NVTX_VERSION=12.1.105-1
2024-07-08T16:57:25.2860707Z GITHUB_RUN_NUMBER=78798
2024-07-08T16:57:25.2861066Z TEST_CONFIG=inductor_huggingface
2024-07-08T16:57:25.2861476Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-07-08T16:57:25.2861945Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-07-08T16:57:25.2862397Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1
2024-07-08T16:57:25.2862836Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1
2024-07-08T16:57:25.2863507Z NV_LIBNPP_VERSION=12.1.0.40-1
2024-07-08T16:57:25.2863898Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2024-07-08T16:57:25.2864387Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-07-08T16:57:25.2864883Z GITHUB_REF_TYPE=branch
2024-07-08T16:57:25.2865223Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-07-08T16:57:25.2865596Z NCCL_VERSION=2.17.1-1
2024-07-08T16:57:25.2865977Z BASE_SHA=c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:57:25.2866519Z XLA_CUDA=
2024-07-08T16:57:25.2867024Z HUGGING_FACE_HUB_TOKEN=***
2024-07-08T16:57:25.2868703Z ***
2024-07-08T16:57:25.2868996Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-07-08T16:57:25.2869419Z GITHUB_REPOSITORY_ID=65600975
2024-07-08T16:57:25.2869800Z GITHUB_ACTIONS=true
2024-07-08T16:57:25.2870145Z NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:57:25.2870653Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1
2024-07-08T16:57:25.2871181Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1
2024-07-08T16:57:25.2871647Z SHA1=c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:57:25.2872151Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-07-08T16:57:25.2872620Z GITHUB_SHA=c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:57:25.2873315Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main
2024-07-08T16:57:25.2873924Z UCC_HOME=/usr
2024-07-08T16:57:25.2874258Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1
2024-07-08T16:57:25.2874645Z VERBOSE_TEST_LOGS=False
2024-07-08T16:57:25.2874988Z NVIDIA_PRODUCT_NAME=CUDA
2024-07-08T16:57:25.2875441Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1
2024-07-08T16:57:25.2875890Z GITHUB_REF=refs/heads/main
2024-07-08T16:57:25.2876270Z NV_CUDA_CUDART_VERSION=12.1.105-1
2024-07-08T16:57:25.2876644Z SHARD_NUMBER=1
2024-07-08T16:57:25.2876950Z GITHUB_REF_PROTECTED=true
2024-07-08T16:57:25.2877283Z HOME=/var/lib/jenkins
2024-07-08T16:57:25.2877648Z GITHUB_API_URL=https://api.github.com
2024-07-08T16:57:25.2878082Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-07-08T16:57:25.2878557Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-07-08T16:57:25.2879032Z SCCACHE_S3_KEY_PREFIX=inductor
2024-07-08T16:57:25.2879390Z CUDA_VERSION=12.1.1
2024-07-08T16:57:25.2879800Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1
2024-07-08T16:57:25.2880223Z NUM_TEST_SHARDS=1
2024-07-08T16:57:25.2880518Z UCX_HOME=/usr
2024-07-08T16:57:25.2881123Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1
2024-07-08T16:57:25.2882194Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_4a32ed44-6883-44df-af8b-38aebb8abf4f
2024-07-08T16:57:25.2883400Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-07-08T16:57:25.2884605Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_4a32ed44-6883-44df-af8b-38aebb8abf4f
2024-07-08T16:57:25.2885984Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-07-08T16:57:25.2886638Z GITHUB_EVENT_NAME=push
2024-07-08T16:57:25.2886972Z DASHBOARD_TAG=
2024-07-08T16:57:25.2887266Z GITHUB_RUN_ID=9843060221
2024-07-08T16:57:25.2887714Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1
2024-07-08T16:57:25.2888240Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1
2024-07-08T16:57:25.2889209Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_4a32ed44-6883-44df-af8b-38aebb8abf4f
2024-07-08T16:57:25.2890075Z GITHUB_ACTOR=pytorchmergebot
2024-07-08T16:57:25.2890480Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1
2024-07-08T16:57:25.2890854Z PR_NUMBER=
2024-07-08T16:57:25.2891126Z GITHUB_RUN_ATTEMPT=1
2024-07-08T16:57:25.2891455Z ANACONDA_PYTHON_VERSION=3.10
2024-07-08T16:57:25.2891890Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-07-08T16:57:25.2892342Z TERM=xterm
2024-07-08T16:57:25.2892663Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1
2024-07-08T16:57:25.2893077Z INSTALLED_VISION=yes
2024-07-08T16:57:25.2893386Z BRANCH=main
2024-07-08T16:57:25.2893670Z OPENSSL_ROOT_DIR=/opt/openssl
2024-07-08T16:57:25.2894208Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-07-08T16:57:25.2894627Z CUDA_PATH=/usr/local/cuda
2024-07-08T16:57:25.2895370Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-07-08T16:57:25.2896104Z GITHUB_SERVER_URL=https://github.com
2024-07-08T16:57:25.2896585Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-07-08T16:57:25.2897042Z REENABLED_ISSUES=
2024-07-08T16:57:25.2897341Z SHLVL=1
2024-07-08T16:57:25.2897670Z MAX_JOBS=14
2024-07-08T16:57:25.2897984Z NV_CUDA_LIB_VERSION=12.1.1-1
2024-07-08T16:57:25.2898336Z NVARCH=x86_64
2024-07-08T16:57:25.2898627Z GITHUB_ACTOR_ID=97764156
2024-07-08T16:57:25.2899083Z GITHUB_WORKFLOW_SHA=c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:57:25.2899580Z GITHUB_REF_NAME=main
2024-07-08T16:57:25.2899975Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1
2024-07-08T16:57:25.2900627Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-07-08T16:57:25.2901190Z GITHUB_JOB=test
2024-07-08T16:57:25.2901586Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1
2024-07-08T16:57:25.2902140Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-07-08T16:57:25.2902640Z NO_TEST_TIMEOUT=False
2024-07-08T16:57:25.2902965Z TD_DISTRIBUTED=False
2024-07-08T16:57:25.2903336Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1
2024-07-08T16:57:25.2903765Z GITHUB_REPOSITORY=pytorch/pytorch
2024-07-08T16:57:25.2904175Z NV_NVPROF_VERSION=12.1.105-1
2024-07-08T16:57:25.2904543Z GITHUB_RETENTION_DAYS=90
2024-07-08T16:57:25.2904890Z OPENSSL_DIR=/opt/openssl
2024-07-08T16:57:25.2905230Z GITHUB_ACTION_REPOSITORY=
2024-07-08T16:57:25.2906253Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-07-08T16:57:25.2907310Z GITHUB_BASE_REF=
2024-07-08T16:57:25.2907635Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-07-08T16:57:25.2907992Z CI=true
2024-07-08T16:57:25.2908308Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1
2024-07-08T16:57:25.2908726Z GITHUB_REPOSITORY_OWNER=pytorch
2024-07-08T16:57:25.2909095Z JOB_ID=27175320579
2024-07-08T16:57:25.2909398Z INSTALLED_PROTOBUF=yes
2024-07-08T16:57:25.2909722Z GITHUB_HEAD_REF=
2024-07-08T16:57:25.2910022Z GITHUB_ACTION_REF=
2024-07-08T16:57:25.2910442Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-07-08T16:57:25.2910915Z GITHUB_WORKFLOW=inductor
2024-07-08T16:57:25.2911274Z DEBIAN_FRONTEND=noninteractive
2024-07-08T16:57:25.2912176Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_4a32ed44-6883-44df-af8b-38aebb8abf4f
2024-07-08T16:57:25.2913014Z NO_TD=False
2024-07-08T16:57:25.2913322Z SKIP_SCCACHE_INITIALIZATION=1
2024-07-08T16:57:25.2913684Z _=/usr/bin/env
2024-07-08T16:57:25.2914157Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2024-07-08T16:57:25.2988230Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2024-07-08T16:57:25.2989846Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2024-07-08T16:57:25.2991100Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2024-07-08T16:57:25.2992067Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2024-07-08T16:57:25.2992668Z + BUILD_DIR=build
2024-07-08T16:57:25.2993041Z + BUILD_RENAMED_DIR=build_renamed
2024-07-08T16:57:25.2993490Z + BUILD_BIN_DIR=build/bin
2024-07-08T16:57:25.2993830Z + SHARD_NUMBER=1
2024-07-08T16:57:25.2994230Z + NUM_TEST_SHARDS=1
2024-07-08T16:57:25.2994562Z + export VALGRIND=ON
2024-07-08T16:57:25.2994877Z + VALGRIND=ON
2024-07-08T16:57:25.2995353Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]]
2024-07-08T16:57:25.2995836Z + [[ 0 == \1 ]]
2024-07-08T16:57:25.2996128Z + [[ False == \1 ]]
2024-07-08T16:57:25.2996607Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]]
2024-07-08T16:57:25.2997130Z ++ realpath build/custom_test_artifacts
2024-07-08T16:57:25.3002734Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2024-07-08T16:57:25.3003820Z + [[ -n '' ]]
2024-07-08T16:57:25.3004178Z + echo 'Environment variables'
2024-07-08T16:57:25.3004555Z Environment variables
2024-07-08T16:57:25.3004863Z + env
2024-07-08T16:57:25.3009432Z INSTALLED_DB=yes
2024-07-08T16:57:25.3010003Z NV_LIBCUBLAS_VERSION=12.1.3.1-1
2024-07-08T16:57:25.3010489Z NVIDIA_VISIBLE_DEVICES=all
2024-07-08T16:57:25.3010975Z NV_NVML_DEV_VERSION=12.1.105-1
2024-07-08T16:57:25.3011956Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-07-08T16:57:25.3012741Z CONTINUE_THROUGH_ERROR=False
2024-07-08T16:57:25.3013398Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1
2024-07-08T16:57:25.3014068Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1
2024-07-08T16:57:25.3014799Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-07-08T16:57:25.3015289Z HOSTNAME=5517005639eb
2024-07-08T16:57:25.3016255Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_4a32ed44-6883-44df-af8b-38aebb8abf4f
2024-07-08T16:57:25.3017086Z GITHUB_ACTION=__self
2024-07-08T16:57:25.3017435Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2024-07-08T16:57:25.3021145Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526
2024-07-08T16:57:25.3025007Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1
2024-07-08T16:57:25.3025514Z NV_NVTX_VERSION=12.1.105-1
2024-07-08T16:57:25.3038835Z GITHUB_RUN_NUMBER=78798
2024-07-08T16:57:25.3039215Z TEST_CONFIG=inductor_huggingface
2024-07-08T16:57:25.3039628Z GITHUB_REPOSITORY_OWNER_ID=21003710
2024-07-08T16:57:25.3040117Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2024-07-08T16:57:25.3040574Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1
2024-07-08T16:57:25.3041137Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1
2024-07-08T16:57:25.3041565Z NV_LIBNPP_VERSION=12.1.0.40-1
2024-07-08T16:57:25.3041965Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot
2024-07-08T16:57:25.3042457Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2024-07-08T16:57:25.3042927Z GITHUB_REF_TYPE=branch
2024-07-08T16:57:25.3043270Z TORCH_CUDA_ARCH_LIST=Maxwell
2024-07-08T16:57:25.3043639Z NCCL_VERSION=2.17.1-1
2024-07-08T16:57:25.3044032Z BASE_SHA=c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:57:25.3044480Z XLA_CUDA=
2024-07-08T16:57:25.3044967Z HUGGING_FACE_HUB_TOKEN=***
2024-07-08T16:57:25.3045364Z ***
2024-07-08T16:57:25.3045958Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2024-07-08T16:57:25.3046408Z GITHUB_REPOSITORY_ID=65600975
2024-07-08T16:57:25.3046774Z GITHUB_ACTIONS=true
2024-07-08T16:57:25.3047098Z NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:57:25.3047595Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1
2024-07-08T16:57:25.3048128Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1
2024-07-08T16:57:25.3048599Z SHA1=c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:57:25.3049104Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2024-07-08T16:57:25.3049587Z GITHUB_SHA=c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:57:25.3050283Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/heads/main
2024-07-08T16:57:25.3050895Z UCC_HOME=/usr
2024-07-08T16:57:25.3051220Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1
2024-07-08T16:57:25.3051620Z VERBOSE_TEST_LOGS=False
2024-07-08T16:57:25.3051962Z NVIDIA_PRODUCT_NAME=CUDA
2024-07-08T16:57:25.3052409Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1
2024-07-08T16:57:25.3052872Z GITHUB_REF=refs/heads/main
2024-07-08T16:57:25.3053423Z NV_CUDA_CUDART_VERSION=12.1.105-1
2024-07-08T16:57:25.3053803Z SHARD_NUMBER=1
2024-07-08T16:57:25.3054106Z GITHUB_REF_PROTECTED=true
2024-07-08T16:57:25.3054455Z HOME=/var/lib/jenkins
2024-07-08T16:57:25.3054817Z GITHUB_API_URL=https://api.github.com
2024-07-08T16:57:25.3055257Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2024-07-08T16:57:25.3055728Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2024-07-08T16:57:25.3056209Z SCCACHE_S3_KEY_PREFIX=inductor
2024-07-08T16:57:25.3056703Z CUDA_VERSION=12.1.1
2024-07-08T16:57:25.3057116Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1
2024-07-08T16:57:25.3057557Z NUM_TEST_SHARDS=1
2024-07-08T16:57:25.3057863Z UCX_HOME=/usr
2024-07-08T16:57:25.3058371Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1
2024-07-08T16:57:25.3059433Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_4a32ed44-6883-44df-af8b-38aebb8abf4f
2024-07-08T16:57:25.3060657Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_huggingface, 1, 1, linux.g5.4xlarge.nvidia.gpu)
2024-07-08T16:57:25.3061870Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_4a32ed44-6883-44df-af8b-38aebb8abf4f
2024-07-08T16:57:25.3062973Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2024-07-08T16:57:25.3063614Z GITHUB_EVENT_NAME=push
2024-07-08T16:57:25.3063953Z DASHBOARD_TAG=
2024-07-08T16:57:25.3064255Z GITHUB_RUN_ID=9843060221
2024-07-08T16:57:25.3064709Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1
2024-07-08T16:57:25.3065232Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1
2024-07-08T16:57:25.3066223Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_4a32ed44-6883-44df-af8b-38aebb8abf4f
2024-07-08T16:57:25.3067093Z GITHUB_ACTOR=pytorchmergebot
2024-07-08T16:57:25.3067500Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1
2024-07-08T16:57:25.3067870Z PR_NUMBER=
2024-07-08T16:57:25.3068157Z GITHUB_RUN_ATTEMPT=1
2024-07-08T16:57:25.3068483Z VALGRIND=ON
2024-07-08T16:57:25.3068785Z ANACONDA_PYTHON_VERSION=3.10
2024-07-08T16:57:25.3069221Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2024-07-08T16:57:25.3069687Z TERM=xterm
2024-07-08T16:57:25.3070019Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1
2024-07-08T16:57:25.3070426Z INSTALLED_VISION=yes
2024-07-08T16:57:25.3070749Z BRANCH=main
2024-07-08T16:57:25.3071045Z OPENSSL_ROOT_DIR=/opt/openssl
2024-07-08T16:57:25.3071451Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2024-07-08T16:57:25.3071876Z CUDA_PATH=/usr/local/cuda
2024-07-08T16:57:25.3072625Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2024-07-08T16:57:25.3073427Z GITHUB_SERVER_URL=https://github.com
2024-07-08T16:57:25.3073918Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2024-07-08T16:57:25.3074377Z REENABLED_ISSUES=
2024-07-08T16:57:25.3074682Z SHLVL=1
2024-07-08T16:57:25.3074954Z MAX_JOBS=14
2024-07-08T16:57:25.3075261Z NV_CUDA_LIB_VERSION=12.1.1-1
2024-07-08T16:57:25.3075617Z NVARCH=x86_64
2024-07-08T16:57:25.3075916Z GITHUB_ACTOR_ID=97764156
2024-07-08T16:57:25.3076373Z GITHUB_WORKFLOW_SHA=c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:57:25.3076867Z GITHUB_REF_NAME=main
2024-07-08T16:57:25.3077263Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1
2024-07-08T16:57:25.3077924Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2024-07-08T16:57:25.3078495Z GITHUB_JOB=test
2024-07-08T16:57:25.3078893Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1
2024-07-08T16:57:25.3079456Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2024-07-08T16:57:25.3079966Z NO_TEST_TIMEOUT=False
2024-07-08T16:57:25.3080288Z TD_DISTRIBUTED=False
2024-07-08T16:57:25.3080663Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1
2024-07-08T16:57:25.3081185Z GITHUB_REPOSITORY=pytorch/pytorch
2024-07-08T16:57:25.3081603Z NV_NVPROF_VERSION=12.1.105-1
2024-07-08T16:57:25.3081967Z GITHUB_RETENTION_DAYS=90
2024-07-08T16:57:25.3082314Z OPENSSL_DIR=/opt/openssl
2024-07-08T16:57:25.3082747Z GITHUB_ACTION_REPOSITORY=
2024-07-08T16:57:25.3083782Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-07-08T16:57:25.3084831Z GITHUB_BASE_REF=
2024-07-08T16:57:25.3085158Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2024-07-08T16:57:25.3085530Z CI=true
2024-07-08T16:57:25.3086145Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1
2024-07-08T16:57:25.3086554Z GITHUB_REPOSITORY_OWNER=pytorch
2024-07-08T16:57:25.3087049Z JOB_ID=27175320579
2024-07-08T16:57:25.3087367Z INSTALLED_PROTOBUF=yes
2024-07-08T16:57:25.3087691Z GITHUB_HEAD_REF=
2024-07-08T16:57:25.3088002Z GITHUB_ACTION_REF=
2024-07-08T16:57:25.3088440Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2024-07-08T16:57:25.3088914Z GITHUB_WORKFLOW=inductor
2024-07-08T16:57:25.3089266Z DEBIAN_FRONTEND=noninteractive
2024-07-08T16:57:25.3090168Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_4a32ed44-6883-44df-af8b-38aebb8abf4f
2024-07-08T16:57:25.3090982Z NO_TD=False
2024-07-08T16:57:25.3091286Z SKIP_SCCACHE_INITIALIZATION=1
2024-07-08T16:57:25.3091638Z _=/usr/bin/env
2024-07-08T16:57:25.3091971Z + echo 'Testing pytorch'
2024-07-08T16:57:25.3092321Z Testing pytorch
2024-07-08T16:57:25.3092637Z + export LANG=C.UTF-8
2024-07-08T16:57:25.3092979Z + LANG=C.UTF-8
2024-07-08T16:57:25.3093273Z + PR_NUMBER=
2024-07-08T16:57:25.3093623Z + [[ inductor_huggingface == \d\e\f\a\u\l\t ]]
2024-07-08T16:57:25.3094153Z + [[ inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]]
2024-07-08T16:57:25.3094669Z + [[ inductor_huggingface == \s\l\o\w ]]
2024-07-08T16:57:25.3095298Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]]
2024-07-08T16:57:25.3095982Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]]
2024-07-08T16:57:25.3096520Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-07-08T16:57:25.3096990Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2024-07-08T16:57:25.3097442Z + [[ inductor_huggingface == *crossref* ]]
2024-07-08T16:57:25.3098018Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]]
2024-07-08T16:57:25.3098638Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]]
2024-07-08T16:57:25.3099276Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-07-08T16:57:25.3099831Z + pip_install --user ninja==1.10.2
2024-07-08T16:57:25.3100362Z + pip install --progress-bar off --user ninja==1.10.2
2024-07-08T16:57:25.6742956Z Collecting ninja==1.10.2
2024-07-08T16:57:25.6919699Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2024-07-08T16:57:25.7061265Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2024-07-08T16:57:27.0530737Z Installing collected packages: ninja
2024-07-08T16:57:27.0599572Z [33m  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2024-07-08T16:57:27.0602753Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.[0m[33m
2024-07-08T16:57:27.0640918Z [0mSuccessfully installed ninja-1.10.2
2024-07-08T16:57:27.1644451Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-07-08T16:57:27.1646949Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-07-08T16:57:27.1648477Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]]
2024-07-08T16:57:27.1649032Z + install_tlparse
2024-07-08T16:57:27.1649426Z + pip_install --user tlparse==0.3.7
2024-07-08T16:57:27.1649981Z + pip install --progress-bar off --user tlparse==0.3.7
2024-07-08T16:57:27.5188574Z Collecting tlparse==0.3.7
2024-07-08T16:57:27.5339560Z   Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes)
2024-07-08T16:57:27.5445788Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB)
2024-07-08T16:57:28.9292061Z Installing collected packages: tlparse
2024-07-08T16:57:28.9651207Z Successfully installed tlparse-0.3.7
2024-07-08T16:57:29.0577687Z ++ python -m site --user-base
2024-07-08T16:57:29.0729077Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2024-07-08T16:57:29.0731178Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]]
2024-07-08T16:57:29.0731936Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]]
2024-07-08T16:57:29.0732622Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]]
2024-07-08T16:57:29.0733707Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass'
2024-07-08T16:57:29.0734951Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass
2024-07-08T16:57:29.0735750Z + cd test
2024-07-08T16:57:29.0736351Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2024-07-08T16:57:30.5530359Z + [[ inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2024-07-08T16:57:30.5530994Z + [[ inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2024-07-08T16:57:30.5533692Z + DYNAMO_BENCHMARK_FLAGS=()
2024-07-08T16:57:30.5534161Z + [[ inductor_huggingface == *dynamo_eager* ]]
2024-07-08T16:57:30.5534641Z + [[ inductor_huggingface == *aot_eager* ]]
2024-07-08T16:57:30.5535121Z + [[ inductor_huggingface == *aot_inductor* ]]
2024-07-08T16:57:30.5535608Z + [[ inductor_huggingface == *inductor* ]]
2024-07-08T16:57:30.5536062Z + [[ inductor_huggingface != *perf* ]]
2024-07-08T16:57:30.5536636Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor)
2024-07-08T16:57:30.5537084Z + [[ inductor_huggingface == *dynamic* ]]
2024-07-08T16:57:30.5537549Z + [[ inductor_huggingface == *cpu_inductor* ]]
2024-07-08T16:57:30.5538058Z + [[ inductor_huggingface == *cpu_aot_inductor* ]]
2024-07-08T16:57:30.5538575Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2024-07-08T16:57:30.5562503Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]]
2024-07-08T16:57:30.5563239Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]]
2024-07-08T16:57:30.5564056Z + cd test
2024-07-08T16:57:30.5564492Z + python -c 'import torch; print(torch.__config__.show())'
2024-07-08T16:57:31.9076954Z PyTorch built with:
2024-07-08T16:57:31.9077678Z   - GCC 9.4
2024-07-08T16:57:31.9078132Z   - C++ Version: 201703
2024-07-08T16:57:31.9079217Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-07-08T16:57:31.9080292Z   - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
2024-07-08T16:57:31.9081077Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-07-08T16:57:31.9081592Z   - LAPACK is enabled (usually provided by MKL)
2024-07-08T16:57:31.9082079Z   - NNPACK is enabled
2024-07-08T16:57:31.9082449Z   - CPU capability usage: AVX2
2024-07-08T16:57:31.9082864Z   - CUDA Runtime 12.1
2024-07-08T16:57:31.9083383Z   - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86
2024-07-08T16:57:31.9083989Z   - CuDNN 90.1  (built against CUDA 12.4)
2024-07-08T16:57:31.9084420Z   - Magma 2.6.1
2024-07-08T16:57:31.9091928Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF,
2024-07-08T16:57:31.9098575Z
2024-07-08T16:57:32.1878122Z + cd test
2024-07-08T16:57:32.1878831Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2024-07-08T16:57:33.3935664Z ATen/Parallel:
2024-07-08T16:57:33.3936169Z 	at::get_num_threads() : 8
2024-07-08T16:57:33.3936696Z 	at::get_num_interop_threads() : 16
2024-07-08T16:57:33.3937253Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2024-07-08T16:57:33.3937805Z 	omp_get_max_threads() : 8
2024-07-08T16:57:33.3938914Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2024-07-08T16:57:33.3939716Z 	mkl_get_max_threads() : 8
2024-07-08T16:57:33.3940331Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67)
2024-07-08T16:57:33.3940961Z std::thread::hardware_concurrency() : 16
2024-07-08T16:57:33.3941383Z Environment variables:
2024-07-08T16:57:33.3941741Z 	OMP_NUM_THREADS : [not set]
2024-07-08T16:57:33.3942113Z 	MKL_NUM_THREADS : [not set]
2024-07-08T16:57:33.3942490Z ATen parallel backend: OpenMP
2024-07-08T16:57:33.3942740Z
2024-07-08T16:57:33.6409808Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]]
2024-07-08T16:57:33.6410605Z + [[ inductor_huggingface == *backward* ]]
2024-07-08T16:57:33.6411080Z + [[ inductor_huggingface == *xla* ]]
2024-07-08T16:57:33.6411588Z + [[ inductor_huggingface == *executorch* ]]
2024-07-08T16:57:33.6412178Z + [[ inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]]
2024-07-08T16:57:33.6412862Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]]
2024-07-08T16:57:33.6413428Z + [[ inductor_huggingface == distributed ]]
2024-07-08T16:57:33.6414000Z + [[ inductor_huggingface == *inductor_distributed* ]]
2024-07-08T16:57:33.6414641Z + [[ inductor_huggingface == *inductor-halide* ]]
2024-07-08T16:57:33.6415260Z + [[ inductor_huggingface == *inductor-micro-benchmark* ]]
2024-07-08T16:57:33.6415860Z + [[ inductor_huggingface == *huggingface* ]]
2024-07-08T16:57:33.6416350Z + install_torchvision
2024-07-08T16:57:33.6416692Z + local orig_preload
2024-07-08T16:57:33.6417055Z + local commit
2024-07-08T16:57:33.6417379Z ++ get_pinned_commit vision
2024-07-08T16:57:33.6417794Z ++ cat .github/ci_commit_pins/vision.txt
2024-07-08T16:57:33.6422708Z + commit=d23a6e1664d20707c11781299611436e1f0c104f
2024-07-08T16:57:33.6423193Z + orig_preload=
2024-07-08T16:57:33.6423576Z + '[' -n '' ']'
2024-07-08T16:57:33.6424474Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-07-08T16:57:33.6425971Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-07-08T16:57:33.9649979Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2024-07-08T16:57:33.9657222Z   Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-upjje6hl
2024-07-08T16:57:33.9670738Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-upjje6hl
2024-07-08T16:57:35.3671982Z   Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f'
2024-07-08T16:57:35.3690271Z   Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f
2024-07-08T16:57:36.6063003Z   Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f
2024-07-08T16:57:36.8497671Z   Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f
2024-07-08T16:57:39.1378714Z   Preparing metadata (setup.py) ... [?25l- \ done
2024-07-08T16:57:39.1429554Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2)
2024-07-08T16:57:39.1433848Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitc8ab2e8)
2024-07-08T16:57:39.1438569Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0)
2024-07-08T16:57:39.1640761Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1)
2024-07-08T16:57:39.1645972Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2)
2024-07-08T16:57:39.1648329Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1)
2024-07-08T16:57:39.1651761Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8)
2024-07-08T16:57:39.1654760Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4)
2024-07-08T16:57:39.1657863Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1)
2024-07-08T16:57:39.2529770Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5)
2024-07-08T16:57:39.2683163Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0)
2024-07-08T16:57:39.2756739Z Building wheels for collected packages: torchvision
2024-07-08T16:58:52.7175716Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done
2024-07-08T16:58:52.7202499Z [?25h  Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116366 sha256=3d413f4180628cb47744a688c6be279ba6019b73ef4ecd4934834f853af33c7e
2024-07-08T16:58:52.7204869Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96
2024-07-08T16:58:52.7239975Z Successfully built torchvision
2024-07-08T16:58:53.9223416Z Installing collected packages: torchvision
2024-07-08T16:58:54.3065227Z Successfully installed torchvision-0.19.0a0+d23a6e1
2024-07-08T16:58:54.4211863Z + '[' -n '' ']'
2024-07-08T16:58:54.4212245Z + id=0
2024-07-08T16:58:54.4212546Z + test_dynamo_benchmark huggingface 0
2024-07-08T16:58:54.4213292Z ++ pwd
2024-07-08T16:58:54.4215861Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-07-08T16:58:54.4216590Z + local suite=huggingface
2024-07-08T16:58:54.4216927Z + shift
2024-07-08T16:58:54.4217220Z + local shard_id=0
2024-07-08T16:58:54.4217524Z + shift
2024-07-08T16:58:54.4220287Z + [[ inductor_huggingface == *perf_compare* ]]
2024-07-08T16:58:54.4220891Z + [[ inductor_huggingface == *perf* ]]
2024-07-08T16:58:54.4221361Z + [[ inductor_huggingface == *cpu_inductor* ]]
2024-07-08T16:58:54.4221852Z + [[ inductor_huggingface == *cpu_aot_inductor* ]]
2024-07-08T16:58:54.4222352Z + [[ inductor_huggingface == *aot_inductor* ]]
2024-07-08T16:58:54.4223081Z + test_single_dynamo_benchmark inference huggingface 0 --inference --bfloat16
2024-07-08T16:58:54.4223944Z ++ pwd
2024-07-08T16:58:54.4224390Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2024-07-08T16:58:54.4225048Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2024-07-08T16:58:54.4230599Z + local name=inference
2024-07-08T16:58:54.4231315Z + shift
2024-07-08T16:58:54.4231740Z + local suite=huggingface
2024-07-08T16:58:54.4232182Z + shift
2024-07-08T16:58:54.4232546Z + local shard_id=0
2024-07-08T16:58:54.4232865Z + shift
2024-07-08T16:58:54.4233320Z + partition_flags=()
2024-07-08T16:58:54.4233657Z + local partition_flags
2024-07-08T16:58:54.4234039Z + [[ -n 1 ]]
2024-07-08T16:58:54.4234345Z + [[ -n 0 ]]
2024-07-08T16:58:54.4234930Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2024-07-08T16:58:54.4235612Z + [[ inductor_huggingface == *perf_compare* ]]
2024-07-08T16:58:54.4236091Z + [[ inductor_huggingface == *perf* ]]
2024-07-08T16:58:54.4236562Z + [[ inductor_huggingface == *aot_inductor* ]]
2024-07-08T16:58:54.4238167Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv
2024-07-08T16:58:57.0698480Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
2024-07-08T16:58:57.0700125Z   warnings.warn(
2024-07-08T16:58:57.1054639Z
2024-07-08T16:58:57.1055573Z config.json:   0% 0.00/694 [00:00<?, ?B/s]
2024-07-08T16:58:57.1056353Z config.json: 100% 694/694 [00:00<00:00, 12.0MB/s]
2024-07-08T16:59:07.1190721Z Traceback (most recent call last):
2024-07-08T16:59:07.1192611Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
2024-07-08T16:59:07.1193637Z     six.raise_from(e, None)
2024-07-08T16:59:07.1194027Z   File "<string>", line 3, in raise_from
2024-07-08T16:59:07.1195302Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 462, in _make_request
2024-07-08T16:59:07.1196471Z     httplib_response = conn.getresponse()
2024-07-08T16:59:07.1197396Z   File "/opt/conda/envs/py_3.10/lib/python3.10/http/client.py", line 1375, in getresponse
2024-07-08T16:59:07.1198188Z     response.begin()
2024-07-08T16:59:07.1198757Z   File "/opt/conda/envs/py_3.10/lib/python3.10/http/client.py", line 318, in begin
2024-07-08T16:59:07.1199666Z     version, status, reason = self._read_status()
2024-07-08T16:59:07.1200436Z   File "/opt/conda/envs/py_3.10/lib/python3.10/http/client.py", line 279, in _read_status
2024-07-08T16:59:07.1201555Z     line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
2024-07-08T16:59:07.1202481Z   File "/opt/conda/envs/py_3.10/lib/python3.10/socket.py", line 705, in readinto
2024-07-08T16:59:07.1203131Z     return self._sock.recv_into(b)
2024-07-08T16:59:07.1203860Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ssl.py", line 1307, in recv_into
2024-07-08T16:59:07.1204718Z     return self.read(nbytes, buffer)
2024-07-08T16:59:07.1205470Z   File "/opt/conda/envs/py_3.10/lib/python3.10/ssl.py", line 1163, in read
2024-07-08T16:59:07.1206599Z     return self._sslobj.read(len, buffer)
2024-07-08T16:59:07.1207115Z TimeoutError: The read operation timed out
2024-07-08T16:59:07.1207553Z
2024-07-08T16:59:07.1207888Z During handling of the above exception, another exception occurred:
2024-07-08T16:59:07.1208330Z
2024-07-08T16:59:07.1208475Z Traceback (most recent call last):
2024-07-08T16:59:07.1250510Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
2024-07-08T16:59:07.1251555Z     resp = conn.urlopen(
2024-07-08T16:59:07.1252674Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 801, in urlopen
2024-07-08T16:59:07.1254155Z     retries = retries.increment(
2024-07-08T16:59:07.1255205Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/urllib3/util/retry.py", line 552, in increment
2024-07-08T16:59:07.1256054Z     raise six.reraise(type(error), error, _stacktrace)
2024-07-08T16:59:07.1256980Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/urllib3/packages/six.py", line 770, in reraise
2024-07-08T16:59:07.1257725Z     raise value
2024-07-08T16:59:07.1258609Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
2024-07-08T16:59:07.1260003Z     httplib_response = self._make_request(
2024-07-08T16:59:07.1260943Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 469, in _make_request
2024-07-08T16:59:07.1262040Z     self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
2024-07-08T16:59:07.1263089Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 358, in _raise_timeout
2024-07-08T16:59:07.1263919Z     raise ReadTimeoutError(
2024-07-08T16:59:07.1264827Z urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)
2024-07-08T16:59:07.1265542Z
2024-07-08T16:59:07.1265835Z During handling of the above exception, another exception occurred:
2024-07-08T16:59:07.1266269Z
2024-07-08T16:59:07.1266411Z Traceback (most recent call last):
2024-07-08T16:59:07.1267442Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1722, in _get_metadata_or_catch_error
2024-07-08T16:59:07.1268644Z     metadata = get_hf_file_metadata(url=url, proxies=proxies, timeout=etag_timeout, headers=headers)
2024-07-08T16:59:07.1269858Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
2024-07-08T16:59:07.1270698Z     return fn(*args, **kwargs)
2024-07-08T16:59:07.1271664Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1645, in get_hf_file_metadata
2024-07-08T16:59:07.1272543Z     r = _request_wrapper(
2024-07-08T16:59:07.1273430Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 372, in _request_wrapper
2024-07-08T16:59:07.1274286Z     response = _request_wrapper(
2024-07-08T16:59:07.1275211Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 395, in _request_wrapper
2024-07-08T16:59:07.1276217Z     response = get_session().request(method=method, url=url, **params)
2024-07-08T16:59:07.1277186Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
2024-07-08T16:59:07.1277967Z     resp = self.send(prep, **send_kwargs)
2024-07-08T16:59:07.1278794Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
2024-07-08T16:59:07.1279535Z     r = adapter.send(request, **kwargs)
2024-07-08T16:59:07.1280418Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/utils/_http.py", line 66, in send
2024-07-08T16:59:07.1281363Z     return super().send(request, *args, **kwargs)
2024-07-08T16:59:07.1282223Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/requests/adapters.py", line 713, in send
2024-07-08T16:59:07.1282972Z     raise ReadTimeout(e, request=request)
2024-07-08T16:59:07.1284290Z requests.exceptions.ReadTimeout: (ReadTimeoutError("HTTPSConnectionPool(host='huggingface.co', port=443): Read timed out. (read timeout=10)"), '(Request ID: f75e69b1-bb0b-4f88-8996-05b3e99d1b45)')
2024-07-08T16:59:07.1285320Z
2024-07-08T16:59:07.1285880Z The above exception was the direct cause of the following exception:
2024-07-08T16:59:07.1286374Z
2024-07-08T16:59:07.1286528Z Traceback (most recent call last):
2024-07-08T16:59:07.1287411Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/hub.py", line 398, in cached_file
2024-07-08T16:59:07.1288420Z     resolved_file = hf_hub_download(
2024-07-08T16:59:07.1289355Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
2024-07-08T16:59:07.1290207Z     return fn(*args, **kwargs)
2024-07-08T16:59:07.1291120Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1221, in hf_hub_download
2024-07-08T16:59:07.1291986Z     return _hf_hub_download_to_cache_dir(
2024-07-08T16:59:07.1293128Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1325, in _hf_hub_download_to_cache_dir
2024-07-08T16:59:07.1294280Z     _raise_on_head_call_error(head_call_error, force_download, local_files_only)
2024-07-08T16:59:07.1295472Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/huggingface_hub/file_download.py", line 1826, in _raise_on_head_call_error
2024-07-08T16:59:07.1296383Z     raise LocalEntryNotFoundError(
2024-07-08T16:59:07.1297916Z huggingface_hub.utils._errors.LocalEntryNotFoundError: An error happened while trying to locate the file on the Hub and we cannot find the requested files in the local cache. Please check your connection and try again or make sure your Internet connection is on.
2024-07-08T16:59:07.1299428Z
2024-07-08T16:59:07.1299722Z The above exception was the direct cause of the following exception:
2024-07-08T16:59:07.1300173Z
2024-07-08T16:59:07.1300314Z Traceback (most recent call last):
2024-07-08T16:59:07.1301023Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 404, in <module>
2024-07-08T16:59:07.1301804Z     AutoConfig.from_pretrained("t5-small"),
2024-07-08T16:59:07.1302878Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1103, in from_pretrained
2024-07-08T16:59:07.1304146Z     config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
2024-07-08T16:59:07.1305428Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/configuration_utils.py", line 634, in get_config_dict
2024-07-08T16:59:07.1306521Z     config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
2024-07-08T16:59:07.1307702Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/configuration_utils.py", line 689, in _get_config_dict
2024-07-08T16:59:07.1308588Z     resolved_config_file = cached_file(
2024-07-08T16:59:07.1309489Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/utils/hub.py", line 442, in cached_file
2024-07-08T16:59:07.1310276Z     raise EnvironmentError(
2024-07-08T16:59:07.1311657Z OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like t5-small is not the path to a directory containing a file named config.json.
2024-07-08T16:59:07.1313579Z Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'.
2024-07-08T16:59:07.6628974Z + cleanup_workspace
2024-07-08T16:59:07.6630046Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.'
2024-07-08T16:59:07.6631135Z sudo may print the following warning message that can be ignored. The chown command will still run.
2024-07-08T16:59:07.6632053Z + echo '    sudo: setrlimit(RLIMIT_STACK): Operation not permitted'
2024-07-08T16:59:07.6632722Z     sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2024-07-08T16:59:07.6633516Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42'
2024-07-08T16:59:07.6634382Z For more details refer to https://github.com/sudo-project/sudo/issues/42
2024-07-08T16:59:07.6635067Z + sudo chown -R 1000 /var/lib/jenkins/workspace
2024-07-08T16:59:08.1747875Z ##[error]Process completed with exit code 1.
2024-07-08T16:59:08.1824599Z Prepare all required actions
2024-07-08T16:59:08.1825203Z Getting action download info
2024-07-08T16:59:08.3184776Z ##[group]Run ./.github/actions/pytest-cache-upload
2024-07-08T16:59:08.3185241Z with:
2024-07-08T16:59:08.3185530Z   cache_dir: .pytest_cache
2024-07-08T16:59:08.3185882Z   shard: 1
2024-07-08T16:59:08.3186223Z   sha: c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:59:08.3186702Z   test_config: inductor_huggingface
2024-07-08T16:59:08.3187263Z   job_identifier: inductor_linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-07-08T16:59:08.3187793Z env:
2024-07-08T16:59:08.3188070Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:08.3188529Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:08.3189269Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:08.3189907Z ##[endgroup]
2024-07-08T16:59:08.3228890Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2024-07-08T16:59:08.3229487Z with:
2024-07-08T16:59:08.3229750Z   shell: bash
2024-07-08T16:59:08.3230071Z   timeout_minutes: 5
2024-07-08T16:59:08.3230401Z   max_attempts: 5
2024-07-08T16:59:08.3230719Z   retry_wait_seconds: 30
2024-07-08T16:59:08.3231165Z   command: set -eu
python3 -m pip install boto3==1.19.12

2024-07-08T16:59:08.3231692Z   polling_interval_seconds: 1
2024-07-08T16:59:08.3232070Z   warning_on_retry: true
2024-07-08T16:59:08.3232422Z   continue_on_error: false
2024-07-08T16:59:08.3232756Z env:
2024-07-08T16:59:08.3233037Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:08.3233492Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:08.3234269Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:08.3234892Z ##[endgroup]
2024-07-08T16:59:08.5898215Z Defaulting to user installation because normal site-packages is not writeable
2024-07-08T16:59:09.8375661Z Collecting boto3==1.19.12
2024-07-08T16:59:09.8616222Z   Downloading boto3-1.19.12-py3-none-any.whl (131 kB)
2024-07-08T16:59:11.2953942Z Collecting botocore<1.23.0,>=1.22.12
2024-07-08T16:59:11.3036342Z   Downloading botocore-1.22.12-py3-none-any.whl (8.1 MB)
2024-07-08T16:59:11.4715882Z Collecting s3transfer<0.6.0,>=0.5.0
2024-07-08T16:59:11.4756925Z   Downloading s3transfer-0.5.2-py3-none-any.whl (79 kB)
2024-07-08T16:59:11.5131849Z Collecting jmespath<1.0.0,>=0.7.1
2024-07-08T16:59:11.5178050Z   Downloading jmespath-0.10.0-py2.py3-none-any.whl (24 kB)
2024-07-08T16:59:11.5680027Z Collecting python-dateutil<3.0.0,>=2.1
2024-07-08T16:59:11.5721195Z   Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
2024-07-08T16:59:11.5850373Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.19)
2024-07-08T16:59:11.6399625Z Collecting six>=1.5
2024-07-08T16:59:11.6460642Z   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
2024-07-08T16:59:11.7170148Z Installing collected packages: six, python-dateutil, jmespath, botocore, s3transfer, boto3
2024-07-08T16:59:12.3827719Z Successfully installed boto3-1.19.12 botocore-1.22.12 jmespath-0.10.0 python-dateutil-2.9.0.post0 s3transfer-0.5.2 six-1.16.0
2024-07-08T16:59:13.3752650Z Command completed after 1 attempt(s).
2024-07-08T16:59:13.3795893Z ##[group]Run python3 .github/scripts/pytest_cache.py \
2024-07-08T16:59:13.3796517Z [36;1mpython3 .github/scripts/pytest_cache.py \[0m
2024-07-08T16:59:13.3796993Z [36;1m  --upload \[0m
2024-07-08T16:59:13.3797414Z [36;1m  --cache_dir $GITHUB_WORKSPACE/$CACHE_DIR \[0m
2024-07-08T16:59:13.3797929Z [36;1m  --pr_identifier $GITHUB_REF \[0m
2024-07-08T16:59:13.3798413Z [36;1m  --job_identifier $JOB_IDENTIFIER \[0m
2024-07-08T16:59:13.3798848Z [36;1m  --sha $SHA \[0m
2024-07-08T16:59:13.3799222Z [36;1m  --test_config $TEST_CONFIG \[0m
2024-07-08T16:59:13.3799760Z [36;1m  --shard $SHARD \[0m
2024-07-08T16:59:13.3800171Z [36;1m  --repo $REPO \[0m
2024-07-08T16:59:13.3800679Z [36;1m  --temp_dir $RUNNER_TEMP \[0m
2024-07-08T16:59:13.3808857Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:59:13.3809472Z env:
2024-07-08T16:59:13.3809746Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:13.3810211Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:13.3810939Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:13.3811578Z   CACHE_DIR: .pytest_cache
2024-07-08T16:59:13.3812072Z   JOB_IDENTIFIER: inductor_linux-focal-cuda12.1-py3.10-gcc9-sm86
2024-07-08T16:59:13.3812649Z   SHA: c8ab2e8b637515b6488931f5e59f23848aae9991
2024-07-08T16:59:13.3813110Z   TEST_CONFIG: inductor_huggingface
2024-07-08T16:59:13.3813505Z   SHARD: 1
2024-07-08T16:59:13.3813789Z   REPO: pytorch/pytorch
2024-07-08T16:59:13.3814123Z ##[endgroup]
2024-07-08T16:59:13.5142939Z PR identifier for `refs/heads/main` is `96e092540d6b3c4076e3d2bc6f1f9013`
2024-07-08T16:59:13.5145787Z Uploading cache with args Namespace(bucket=None, cache_dir='/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache', download=False, job_identifier='inductor_linux-focal-cuda12.1-py3.10-gcc9-sm86', pr_identifier='refs/heads/main', repo='pytorch/pytorch', sha='c8ab2e8b637515b6488931f5e59f23848aae9991', shard='1', temp_dir='/home/ec2-user/actions-runner/_work/_temp', test_config='inductor_huggingface', upload=True)
2024-07-08T16:59:13.5148560Z The pytest cache dir `/home/ec2-user/actions-runner/_work/pytorch/pytorch/.pytest_cache` does not exist. Skipping upload
2024-07-08T16:59:13.5314348Z ##[group]Run cat test/**/*_toprint.log || true
2024-07-08T16:59:13.5314876Z [36;1mcat test/**/*_toprint.log || true[0m
2024-07-08T16:59:13.5322990Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:59:13.5323481Z env:
2024-07-08T16:59:13.5323771Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:13.5324231Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:13.5324955Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:13.5325706Z ##[endgroup]
2024-07-08T16:59:13.5387599Z cat: test/**/*_toprint.log: No such file or directory
2024-07-08T16:59:13.5415765Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2024-07-08T16:59:13.5416230Z [36;1mkill "$MONITOR_SCRIPT_PID"[0m
2024-07-08T16:59:13.5422898Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:59:13.5423392Z env:
2024-07-08T16:59:13.5423660Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:13.5424116Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:13.5424835Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:13.5425471Z   MONITOR_SCRIPT_PID: 28481
2024-07-08T16:59:13.5425818Z ##[endgroup]
2024-07-08T16:59:13.5611040Z Prepare all required actions
2024-07-08T16:59:13.5611492Z Getting action download info
2024-07-08T16:59:13.7004614Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32)
2024-07-08T16:59:13.8532069Z ##[group]Run ./.github/actions/upload-test-artifacts
2024-07-08T16:59:13.8532560Z with:
2024-07-08T16:59:13.8533103Z   file-suffix: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_27175320579
2024-07-08T16:59:13.8533771Z   s3-bucket: gha-artifacts
2024-07-08T16:59:13.8534116Z env:
2024-07-08T16:59:13.8534396Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:13.8534849Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:13.8535579Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:13.8536206Z ##[endgroup]
2024-07-08T16:59:13.8577963Z ##[group]Run # Remove any previous test jsons if they exist
2024-07-08T16:59:13.8578617Z [36;1m# Remove any previous test jsons if they exist[0m
2024-07-08T16:59:13.8579136Z [36;1mrm -f test-jsons-*.zip[0m
2024-07-08T16:59:13.8579660Z [36;1mzip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'[0m
2024-07-08T16:59:13.8587324Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:59:13.8587824Z env:
2024-07-08T16:59:13.8588110Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:13.8588759Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:13.8589498Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:13.8590399Z   FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_27175320579
2024-07-08T16:59:13.8591058Z ##[endgroup]
2024-07-08T16:59:13.8792164Z   adding: test/allowlist_for_publicAPI.json (deflated 79%)
2024-07-08T16:59:13.8824465Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2024-07-08T16:59:13.8825480Z   adding: test/minioptest_failures_dict.json (deflated 70%)
2024-07-08T16:59:13.8831888Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2024-07-08T16:59:13.8860823Z ##[group]Run # Remove any previous test reports if they exist
2024-07-08T16:59:13.8861464Z [36;1m# Remove any previous test reports if they exist[0m
2024-07-08T16:59:13.8861989Z [36;1mrm -f test-reports-*.zip[0m
2024-07-08T16:59:13.8862590Z [36;1mzip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'[0m
2024-07-08T16:59:13.8870351Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:59:13.8870838Z env:
2024-07-08T16:59:13.8871120Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:13.8871571Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:13.8872293Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:13.8873173Z   FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_27175320579
2024-07-08T16:59:13.8873813Z ##[endgroup]
2024-07-08T16:59:13.9029501Z 	zip warning: zip file empty
2024-07-08T16:59:13.9055759Z ##[group]Run # Remove any previous usage logs if they exist
2024-07-08T16:59:13.9056389Z [36;1m# Remove any previous usage logs if they exist[0m
2024-07-08T16:59:13.9056885Z [36;1mrm -f logs-*.zip[0m
2024-07-08T16:59:13.9057546Z [36;1m# this workflow is also run in bazel build test, but we dont generate usage reports for it[0m
2024-07-08T16:59:13.9058338Z [36;1m# so check to see if the file exists first[0m
2024-07-08T16:59:13.9058832Z [36;1mif [ -f 'usage_log.txt' ]; then[0m
2024-07-08T16:59:13.9059360Z [36;1m    zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt'[0m
2024-07-08T16:59:13.9059840Z [36;1mfi[0m
2024-07-08T16:59:13.9060209Z [36;1mif ls test/**/*.log 1> /dev/null 2>&1; then[0m
2024-07-08T16:59:13.9060776Z [36;1m    zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log'[0m
2024-07-08T16:59:13.9061264Z [36;1mfi[0m
2024-07-08T16:59:13.9068537Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:59:13.9069033Z env:
2024-07-08T16:59:13.9069418Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:13.9069877Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:13.9070604Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:13.9071489Z   FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_27175320579
2024-07-08T16:59:13.9072128Z ##[endgroup]
2024-07-08T16:59:13.9103830Z   adding: usage_log.txt (deflated 92%)
2024-07-08T16:59:13.9166971Z ##[group]Run # Remove any previous debugging artifacts if they exist
2024-07-08T16:59:13.9167689Z [36;1m# Remove any previous debugging artifacts if they exist[0m
2024-07-08T16:59:13.9168235Z [36;1mrm -f debug-*.zip[0m
2024-07-08T16:59:13.9168621Z [36;1mif [ -d 'test/debug' ]; then[0m
2024-07-08T16:59:13.9169120Z [36;1m  zip -r "debug-${FILE_SUFFIX}.zip" test/debug[0m
2024-07-08T16:59:13.9169608Z [36;1mfi[0m
2024-07-08T16:59:13.9176233Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:59:13.9176772Z env:
2024-07-08T16:59:13.9177061Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:13.9177515Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:13.9178247Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:13.9179142Z   FILE_SUFFIX: test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_27175320579
2024-07-08T16:59:13.9179908Z ##[endgroup]
2024-07-08T16:59:13.9253142Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-07-08T16:59:13.9253570Z with:
2024-07-08T16:59:13.9253858Z   s3-bucket: gha-artifacts
2024-07-08T16:59:13.9254292Z   s3-prefix: pytorch/pytorch/9843060221/1/artifact

2024-07-08T16:59:13.9254768Z   retention-days: 14
2024-07-08T16:59:13.9255099Z   if-no-files-found: warn
2024-07-08T16:59:13.9255465Z   path: test-jsons-*.zip
2024-07-08T16:59:13.9255808Z   name: artifact
2024-07-08T16:59:13.9256117Z   region: us-east-1
2024-07-08T16:59:13.9256414Z env:
2024-07-08T16:59:13.9256700Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:13.9257171Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:13.9257894Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:13.9258525Z ##[endgroup]
2024-07-08T16:59:14.2297626Z NOTE: s3-prefix specified, ignoring name parameter
2024-07-08T16:59:14.2298231Z With the provided path, there will be 1 file uploaded
2024-07-08T16:59:14.2298874Z Uploading to s3 prefix: pytorch/pytorch/9843060221/1/artifact
2024-07-08T16:59:14.2324070Z Starting upload of test-jsons-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_27175320579.zip
2024-07-08T16:59:14.3524106Z Finished upload of test-jsons-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_27175320579.zip
2024-07-08T16:59:14.3646802Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-07-08T16:59:14.3647244Z with:
2024-07-08T16:59:14.3647541Z   s3-bucket: gha-artifacts
2024-07-08T16:59:14.3647973Z   s3-prefix: pytorch/pytorch/9843060221/1/artifact

2024-07-08T16:59:14.3648463Z   retention-days: 14
2024-07-08T16:59:14.3648809Z   if-no-files-found: error
2024-07-08T16:59:14.3649184Z   path: test-reports-*.zip
2024-07-08T16:59:14.3649554Z   name: artifact
2024-07-08T16:59:14.3649866Z   region: us-east-1
2024-07-08T16:59:14.3650180Z env:
2024-07-08T16:59:14.3650471Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:14.3650953Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:14.3651683Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:14.3652321Z ##[endgroup]
2024-07-08T16:59:14.6721074Z NOTE: s3-prefix specified, ignoring name parameter
2024-07-08T16:59:14.6721761Z With the provided path, there will be 1 file uploaded
2024-07-08T16:59:14.6722372Z Uploading to s3 prefix: pytorch/pytorch/9843060221/1/artifact
2024-07-08T16:59:14.6747378Z Starting upload of test-reports-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_27175320579.zip
2024-07-08T16:59:14.7715464Z Finished upload of test-reports-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_27175320579.zip
2024-07-08T16:59:14.7837838Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-07-08T16:59:14.7838280Z with:
2024-07-08T16:59:14.7838573Z   s3-bucket: gha-artifacts
2024-07-08T16:59:14.7839005Z   s3-prefix: pytorch/pytorch/9843060221/1/artifact

2024-07-08T16:59:14.7839480Z   retention-days: 14
2024-07-08T16:59:14.7839829Z   if-no-files-found: ignore
2024-07-08T16:59:14.7840192Z   path: logs-*.zip
2024-07-08T16:59:14.7840510Z   name: artifact
2024-07-08T16:59:14.7840982Z   region: us-east-1
2024-07-08T16:59:14.7841291Z env:
2024-07-08T16:59:14.7841571Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:14.7842029Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:14.7842753Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:14.7843389Z ##[endgroup]
2024-07-08T16:59:15.0867697Z NOTE: s3-prefix specified, ignoring name parameter
2024-07-08T16:59:15.0868334Z With the provided path, there will be 1 file uploaded
2024-07-08T16:59:15.0868957Z Uploading to s3 prefix: pytorch/pytorch/9843060221/1/artifact
2024-07-08T16:59:15.0894527Z Starting upload of logs-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_27175320579.zip
2024-07-08T16:59:15.2252138Z Finished upload of logs-test-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu_27175320579.zip
2024-07-08T16:59:15.2398353Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-07-08T16:59:15.2398792Z with:
2024-07-08T16:59:15.2399074Z   s3-bucket: gha-artifacts
2024-07-08T16:59:15.2399501Z   s3-prefix: pytorch/pytorch/9843060221/1/artifact

2024-07-08T16:59:15.2399969Z   retention-days: 14
2024-07-08T16:59:15.2400308Z   if-no-files-found: ignore
2024-07-08T16:59:15.2400661Z   path: debug-*.zip
2024-07-08T16:59:15.2401073Z   name: artifact
2024-07-08T16:59:15.2401378Z   region: us-east-1
2024-07-08T16:59:15.2401670Z env:
2024-07-08T16:59:15.2401951Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:15.2402415Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:15.2403142Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:15.2403765Z ##[endgroup]
2024-07-08T16:59:15.5368287Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded.
2024-07-08T16:59:15.5519897Z ##[group]Run # shellcheck disable=SC2156
2024-07-08T16:59:15.5520392Z [36;1m# shellcheck disable=SC2156[0m
2024-07-08T16:59:15.5521253Z [36;1mfind . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;[0m
2024-07-08T16:59:15.5529410Z shell: /usr/bin/bash -e {0}
2024-07-08T16:59:15.5529760Z env:
2024-07-08T16:59:15.5530050Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:15.5530509Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:15.5531226Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:15.5531853Z ##[endgroup]
2024-07-08T16:59:15.7109388Z ##[group]Run seemethere/upload-artifact-s3@v5
2024-07-08T16:59:15.7109843Z with:
2024-07-08T16:59:15.7110311Z   name: coredumps-inductor_huggingface-1-1-linux.g5.4xlarge.nvidia.gpu
2024-07-08T16:59:15.7110893Z   retention-days: 14
2024-07-08T16:59:15.7111225Z   if-no-files-found: ignore
2024-07-08T16:59:15.7111590Z   path: ./**/core.[1-9]*
2024-07-08T16:59:15.7111950Z   s3-bucket: gha-artifacts
2024-07-08T16:59:15.7112301Z   region: us-east-1
2024-07-08T16:59:15.7112589Z env:
2024-07-08T16:59:15.7112861Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:15.7113313Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:15.7114035Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:15.7114651Z ##[endgroup]
2024-07-08T16:59:24.3948398Z No files were found with the provided path: ./**/core.[1-9]*. No artifacts will be uploaded.
2024-07-08T16:59:24.4140080Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2024-07-08T16:59:24.4140648Z with:
2024-07-08T16:59:24.4140911Z env:
2024-07-08T16:59:24.4141200Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:24.4141654Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:24.4142395Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:24.4143027Z ##[endgroup]
2024-07-08T16:59:24.4162155Z ##[group]Run set -eou pipefail
2024-07-08T16:59:24.4162569Z [36;1mset -eou pipefail[0m
2024-07-08T16:59:24.4162926Z [36;1m[0m
2024-07-08T16:59:24.4163443Z [36;1mecho "Holding runner for 2 hours until all ssh sessions have logged out"[0m
2024-07-08T16:59:24.4164090Z [36;1mfor _ in $(seq 1440); do[0m
2024-07-08T16:59:24.4164574Z [36;1m    # Break if no ssh session exists anymore[0m
2024-07-08T16:59:24.4165082Z [36;1m    if [ "$(who)" = "" ]; then[0m
2024-07-08T16:59:24.4165495Z [36;1m      break[0m
2024-07-08T16:59:24.4166090Z [36;1m    fi[0m
2024-07-08T16:59:24.4166392Z [36;1m    echo "."[0m
2024-07-08T16:59:24.4166724Z [36;1m    sleep 5[0m
2024-07-08T16:59:24.4167050Z [36;1mdone[0m
2024-07-08T16:59:24.4174641Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:59:24.4175132Z env:
2024-07-08T16:59:24.4175416Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:24.4175876Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:24.4176725Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:24.4177353Z ##[endgroup]
2024-07-08T16:59:24.4196737Z Holding runner for 2 hours until all ssh sessions have logged out
2024-07-08T16:59:24.4230141Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2024-07-08T16:59:24.4230893Z [36;1m# ignore expansion of "docker ps -q" since it could be empty[0m
2024-07-08T16:59:24.4231477Z [36;1m# shellcheck disable=SC2046[0m
2024-07-08T16:59:24.4231951Z [36;1mdocker stop $(docker ps -q) || true[0m
2024-07-08T16:59:24.4232425Z [36;1m# Prune all of the docker images[0m
2024-07-08T16:59:24.4232872Z [36;1mdocker system prune -af[0m
2024-07-08T16:59:24.4239625Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:59:24.4240117Z env:
2024-07-08T16:59:24.4240394Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:24.4240956Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:24.4241692Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:24.4242325Z ##[endgroup]
2024-07-08T16:59:25.0088727Z 5517005639eb
2024-07-08T16:59:25.3888255Z Deleted Containers:
2024-07-08T16:59:25.3888944Z 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:25.3889475Z
2024-07-08T16:59:30.3412306Z Deleted Images:
2024-07-08T16:59:30.3414189Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:52b65ca8de123a7d92fdc1594a96fa3040229cab
2024-07-08T16:59:30.3416593Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:5937e650467bc1196abb8a39571fc2470742e1a0827fa4cc78d216bf0b182f34
2024-07-08T16:59:30.3418104Z deleted: sha256:6cbb2e66e2081153c16c8c4674e997a9c177a07ce94b8ce56e51c1f9cd5e49ea
2024-07-08T16:59:30.3418933Z deleted: sha256:22dbf24b6d03330b05574e39d8495351a1f29f15a4d85131e86d77f69c88d166
2024-07-08T16:59:30.3419761Z deleted: sha256:c4ec097bf492215c1519c69d3f734816b11d09e9ae9a7cd254f6c23648fc884f
2024-07-08T16:59:30.3420591Z deleted: sha256:630e638a01efa2b625ae62a5db97e5543e84a1fed1acc77e18fa5c0aafb23b69
2024-07-08T16:59:30.3421432Z deleted: sha256:b1d0031f4b22e11c8076b04cfb1a208af638947f213cac69ecbed3b969bc83ad
2024-07-08T16:59:30.3422279Z deleted: sha256:83d11bea190decba29e19f673cc29dc3d0adffb66d795f93f240d83b7f3ba068
2024-07-08T16:59:30.3423114Z deleted: sha256:10416f0e8e638de9c067a36b7f3ec94e2e314ec5462ce83335a9b960b91b3700
2024-07-08T16:59:30.3424112Z deleted: sha256:ac66a038e82bb36dc786b0000387183544925296d2cfefb368364cfe7ab029ea
2024-07-08T16:59:30.3424939Z deleted: sha256:6e1ba50b929d01943d60a7a529eada0cf4f56279969c652591eed077c829fa73
2024-07-08T16:59:30.3425762Z deleted: sha256:4815c45d3c40a22356deb10290795cd18611ab25de720cbabcf50d115d371079
2024-07-08T16:59:30.3426586Z deleted: sha256:65e5add7c494aa3faa8e558b57538a707e5ec54ae42f3a7583ca7abf3bbffe50
2024-07-08T16:59:30.3427411Z deleted: sha256:da75255295cbac4d1ee140a3c5212f5118cc45702e99d3e243b5f0ac08696e08
2024-07-08T16:59:30.3428244Z deleted: sha256:440b278e1c1eed9cd89c6d17ad7ca15fcb66317e25a5c581e147b8efdb0fb7c4
2024-07-08T16:59:30.3429075Z deleted: sha256:143b7f87ce14aab1d97e0647ca9cfa8cd35b83034f363a0ba8daae564d5db4c2
2024-07-08T16:59:30.3429894Z deleted: sha256:65f11c90da3942a0953f897aa147c5cf9c4a38182ca5f1a3597f40d7e8335414
2024-07-08T16:59:30.3430690Z deleted: sha256:1d0f0c10e705b261f7b9a18e602560e97b96173a9724319b0a3305ec6304480e
2024-07-08T16:59:30.3431668Z deleted: sha256:d8c19ccab785ef6174ed6897c4e5d19503f07b60691035eb9d81111f9df7f8af
2024-07-08T16:59:30.3432645Z deleted: sha256:64a7f00ca842e8da941b1d7d80aaa0b3961c49583064611be8234450b03efff5
2024-07-08T16:59:30.3433474Z deleted: sha256:caf16525e8d4cf3834020f8df2f76fed44820d4bb6aba1d9dd2f27c9ef393118
2024-07-08T16:59:30.3434292Z deleted: sha256:ecefb98156f537ce27b3a5154d0484a841ed73bc6ae669a23830d995b1a8923c
2024-07-08T16:59:30.3435116Z deleted: sha256:cf695ad7a0e7368f371b9108e8bc89e7bbe4b427e8f15773bf4ad2c685b7d44d
2024-07-08T16:59:30.3436063Z deleted: sha256:1a535401903930da091c30eb66612202191cffc593f31db7e562af13c9043ad2
2024-07-08T16:59:30.3436922Z deleted: sha256:936c6d96fe8def86d38ef527f85fbb2b83af546974925ec0ba3c1d42ce18135a
2024-07-08T16:59:30.3437742Z deleted: sha256:131e412b330060972391b81a43cbd07c111ea78b76701e6701dc924727bcb3b3
2024-07-08T16:59:30.3438539Z deleted: sha256:3900df42b5d4f84283ba35433cd908942216b9a715094e166a77aebc9c300874
2024-07-08T16:59:30.3439360Z deleted: sha256:57e75b0c2aa4c37fd4a7312b23213147efa92d3d4d67235e8d5458de86b42947
2024-07-08T16:59:30.3440172Z deleted: sha256:2e6b148139549ee21544548b9cdb3adf4949d71ac6b412a676f59a53a2973bf6
2024-07-08T16:59:30.3441117Z deleted: sha256:5cdc11d2df61777674d94df70829f8b563e0456a56a913b524b7e7b95fcc9523
2024-07-08T16:59:30.3441953Z deleted: sha256:b7a16a49d5b8640a559d2ded9ecad0c0b100353e7ba773218513cd496d5601b2
2024-07-08T16:59:30.3442775Z deleted: sha256:3054afd68713b2b4864dbbbc0e85604514f7a2d76c071f3b9615cb379cc9671b
2024-07-08T16:59:30.3443882Z deleted: sha256:19cca7263e7714589132bcf2799e0a2b03537ee7d11e97dd188e9a485e9d77b1
2024-07-08T16:59:30.3444834Z deleted: sha256:a1c878fe030137af597c6e5729dc05d8587957486c05376107b5eacd8ca35416
2024-07-08T16:59:30.3445938Z deleted: sha256:4ffe30f717f5f4d727018c87dad28602e3ac0ca35c7f09178212a17a94cacbbf
2024-07-08T16:59:30.3446794Z deleted: sha256:f47a9faa56cebd33d5e61d777d0c40a90e6ea08d9a3240cdae90fe6bcf6902b7
2024-07-08T16:59:30.3447666Z deleted: sha256:58e36abfaed10ccac5399ffb5a6bbe44f229d8f89efcfa9faba14efdf27f6995
2024-07-08T16:59:30.3448525Z deleted: sha256:a420fddf94bf4ca5d7ef282f14fd669af5088b39e5cad8a60a4d61c2fcd54081
2024-07-08T16:59:30.3449352Z deleted: sha256:4452b8eab012fea47378a21d05be50291bdf723eac135c70f48393dd65edfccb
2024-07-08T16:59:30.3450178Z deleted: sha256:c0802ea22b68a7b4de48842d77e9d20d2021ef161a96785fe664c9d5b784f4ee
2024-07-08T16:59:30.3450985Z deleted: sha256:28ce24c7f004c35eebe8fe9b46fa303853829061a592b6ab2263f076615cdb7c
2024-07-08T16:59:30.3451798Z deleted: sha256:f37c3b37b91076c3ff858ad7745a40639cf6dbc66a44ff8d58db681532b173ab
2024-07-08T16:59:30.3452593Z deleted: sha256:81b8a7216b960082684715097beae9b818aa563c156e53973071ee8784edc7db
2024-07-08T16:59:30.3453402Z deleted: sha256:1e63cd2361f9b47b22c1d52ae0b61d2adba5e80e20965db7f7ffd53516966e1e
2024-07-08T16:59:30.3454231Z deleted: sha256:fdb9019efb350f03b8f1a9e185c671b8712bb7312b0bcfd53ca7e71adffa8aa8
2024-07-08T16:59:30.3455052Z deleted: sha256:e4c604519d50531d12e9f91a45dd32e81fdd12c0332ffa60872cd16a0eaa55fb
2024-07-08T16:59:30.3456037Z deleted: sha256:5ac8a37ff152d87a2f220d5d4c8ea0b791cfbfb2d395d2b9f4dcf0d9e518abe3
2024-07-08T16:59:30.3456877Z deleted: sha256:1a4a9da9e63174a71b0acc0bccf4e42c579af74e03ed8976243410156b705aa3
2024-07-08T16:59:30.3457706Z deleted: sha256:881b7d9831ece30387b871242f02be20a99d0c2e91bd54d68a3a4223fb0fa6ce
2024-07-08T16:59:30.3458528Z deleted: sha256:cac2d08d22da007f6587004afb18df9dea137599bd3e35102d17a6dbe7ad05f4
2024-07-08T16:59:30.3459358Z deleted: sha256:40ab9162d8ba32cf1aa1d710596dfb84f70c90a6e5d93ec0c0169b9811c65075
2024-07-08T16:59:30.3460157Z deleted: sha256:80f0188177a48f3a3f91463915c42db76f86551690dbf86e826ce335076f2d20
2024-07-08T16:59:30.3460979Z deleted: sha256:4ad0fb46e2e1d523dd3c9ad9fa80e9dbe8902f681cdf7d82aa47b2b6be12fa1b
2024-07-08T16:59:30.3461803Z deleted: sha256:589b28e48382e970247e066aca0d23c01c003087496a0c8c461bc7935e47c2c0
2024-07-08T16:59:30.3462613Z deleted: sha256:d8974ee3b7073df5b6aa774def601d5bcee6aa311e3b8f29a10b3293a61ed3aa
2024-07-08T16:59:30.3463445Z deleted: sha256:01b9cdb7c907b8a7be96abf7e1bb88062f11ca82536d89988efdffd7ebb239eb
2024-07-08T16:59:30.3464267Z deleted: sha256:0e9dfb6eb94324c49423b288802d00a6cd2e069c3f6748c9ab5d2c621c7ff6ac
2024-07-08T16:59:30.3465097Z deleted: sha256:b7fd9118078f0f832ab60dab4a66d54a71af5c46f86bea66a8fa8b8a67a6ffa6
2024-07-08T16:59:30.3465917Z deleted: sha256:4319144fd84f3356357dc0fd2fd1b7361bc19995d6d111a1e4544ab2a9d4f0ba
2024-07-08T16:59:30.3466711Z deleted: sha256:90748aa0801d6c9dc979757e3daa12e5a93a8294b3f4df05f6eb54f2189b13c0
2024-07-08T16:59:30.3467607Z deleted: sha256:78586a85aacbfe956b61c801b5b96f33e895eb8af47f87a1ac40885dc86fe778
2024-07-08T16:59:30.3468430Z deleted: sha256:94d38deb2a8608090f8deb7f1660e31d0a20c07294429cf282cc902373c16ca3
2024-07-08T16:59:30.3469256Z deleted: sha256:6ac15aca64743daac247bcd0009f8311e93d827c3d35ce7a42717b80d942438b
2024-07-08T16:59:30.3470078Z deleted: sha256:8ff547723ff7ad989e34c786e72d696f01bee049f0933fc676b8114b75c6d3fb
2024-07-08T16:59:30.3470964Z deleted: sha256:aaaef137d8ace48e6be105c515d2b975f14585299fb2bc0f48caffc25ff052c2
2024-07-08T16:59:30.3471797Z deleted: sha256:bdce421e573aab0427a1c48a2225514c4a6380daa464976f9c2ef599a9ed12a2
2024-07-08T16:59:30.3472635Z deleted: sha256:9baabcd02d19023ad028491d5d65ead89d17ef240c4f262aefacc7a07b08df51
2024-07-08T16:59:30.3473456Z deleted: sha256:f453a04240783c1cc796485b83884aa737dbe658d9af221cea56ae6f843fadcc
2024-07-08T16:59:30.3474261Z deleted: sha256:a54ed15ed684af63ba3f2b61184e48be7b977c802ce84b42a2a4aa1b558f7dfe
2024-07-08T16:59:30.3475076Z deleted: sha256:f91929262592643bac4f2cfe778b78b9039a644db5f06744837a3802cc8a4874
2024-07-08T16:59:30.3475888Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8
2024-07-08T16:59:30.3476711Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00
2024-07-08T16:59:30.3477509Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f
2024-07-08T16:59:30.3478316Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64
2024-07-08T16:59:30.3479141Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88
2024-07-08T16:59:30.3479952Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d
2024-07-08T16:59:30.3480851Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25
2024-07-08T16:59:30.3481740Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9
2024-07-08T16:59:30.3482587Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29
2024-07-08T16:59:30.3483421Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037
2024-07-08T16:59:30.3484228Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9
2024-07-08T16:59:30.3484723Z
2024-07-08T16:59:30.3484871Z Total reclaimed space: 31.44GB
2024-07-08T16:59:30.3517595Z ##[group]Run set +e
2024-07-08T16:59:30.3517986Z [36;1mset +e[0m
2024-07-08T16:59:30.3518279Z [36;1mset -x[0m
2024-07-08T16:59:30.3518575Z [36;1m[0m
2024-07-08T16:59:30.3518855Z [36;1mnvidia-smi[0m
2024-07-08T16:59:30.3519474Z [36;1m# NB: Surprisingly, nvidia-smi command returns successfully with return code 0 even in[0m
2024-07-08T16:59:30.3520440Z [36;1m# the case where the driver has already crashed as it still can get the driver version[0m
2024-07-08T16:59:30.3521532Z [36;1m# and some basic information like the bus ID.  However, the rest of the information[0m
2024-07-08T16:59:30.3522270Z [36;1m# would be missing (ERR!), for example:[0m
2024-07-08T16:59:30.3522725Z [36;1m#[0m
2024-07-08T16:59:30.3523147Z [36;1m# +-----------------------------------------------------------------------------+[0m
2024-07-08T16:59:30.3523925Z [36;1m# | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |[0m
2024-07-08T16:59:30.3524708Z [36;1m# |-------------------------------+----------------------+----------------------+[0m
2024-07-08T16:59:30.3525462Z [36;1m# | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |[0m
2024-07-08T16:59:30.3526587Z [36;1m# | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |[0m
2024-07-08T16:59:30.3527323Z [36;1m# |                               |                      |               MIG M. |[0m
2024-07-08T16:59:30.3527909Z [36;1m# |===============================+======================+======================|[0m
2024-07-08T16:59:30.3528547Z [36;1m# |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |[0m
2024-07-08T16:59:30.3529402Z [36;1m# |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |[0m
2024-07-08T16:59:30.3530092Z [36;1m# |                               |                      |                 ERR! |[0m
2024-07-08T16:59:30.3530717Z [36;1m# +-------------------------------+----------------------+----------------------+[0m
2024-07-08T16:59:30.3531229Z [36;1m#[0m
2024-07-08T16:59:30.3531644Z [36;1m# +-----------------------------------------------------------------------------+[0m
2024-07-08T16:59:30.3532292Z [36;1m# | Processes:                                                                  |[0m
2024-07-08T16:59:30.3533008Z [36;1m# |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |[0m
2024-07-08T16:59:30.3533711Z [36;1m# |        ID   ID                                                   Usage      |[0m
2024-07-08T16:59:30.3534314Z [36;1m# |=============================================================================|[0m
2024-07-08T16:59:30.3546907Z [36;1m# +-----------------------------------------------------------------------------+[0m
2024-07-08T16:59:30.3547635Z [36;1m#[0m
2024-07-08T16:59:30.3548196Z [36;1m# This should be reported as a failure instead as it will guarantee to fail when[0m
2024-07-08T16:59:30.3548919Z [36;1m# Docker tries to run with --gpus all[0m
2024-07-08T16:59:30.3549359Z [36;1m#[0m
2024-07-08T16:59:30.3549893Z [36;1m# So, the correct check here is to query one of the missing piece of info like[0m
2024-07-08T16:59:30.3550659Z [36;1m# GPU name, so that the command can fail accordingly[0m
2024-07-08T16:59:30.3551344Z [36;1mnvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0[0m
2024-07-08T16:59:30.3551916Z [36;1mNVIDIA_SMI_STATUS=$?[0m
2024-07-08T16:59:30.3552277Z [36;1m[0m
2024-07-08T16:59:30.3552881Z [36;1m# These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action[0m
2024-07-08T16:59:30.3553794Z [36;1mif [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then[0m
2024-07-08T16:59:30.3554623Z [36;1m  echo "NVIDIA driver installation has failed, shutting down the runner..."[0m
2024-07-08T16:59:30.3555327Z [36;1m  .github/scripts/stop_runner_service.sh[0m
2024-07-08T16:59:30.3555772Z [36;1mfi[0m
2024-07-08T16:59:30.3556052Z [36;1m[0m
2024-07-08T16:59:30.3556744Z [36;1m# For runner with multiple GPUs, we also want to confirm that the number of GPUs are the[0m
2024-07-08T16:59:30.3557711Z [36;1m# power of 2, i.e. 1, 2, 4, or 8. This is to avoid flaky test issue when one GPU fails[0m
2024-07-08T16:59:30.3558508Z [36;1m# https://github.com/pytorch/test-infra/issues/4000[0m
2024-07-08T16:59:30.3559101Z [36;1mGPU_COUNT=$(nvidia-smi --list-gpus | wc -l)[0m
2024-07-08T16:59:30.3559584Z [36;1mNVIDIA_SMI_STATUS=$?[0m
2024-07-08T16:59:30.3559952Z [36;1m[0m
2024-07-08T16:59:30.3560562Z [36;1m# These are acceptable return code from nvidia-smi as copied from setup-nvidia GitHub action[0m
2024-07-08T16:59:30.3561584Z [36;1mif [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then[0m
2024-07-08T16:59:30.3562411Z [36;1m  echo "NVIDIA driver installation has failed, shutting down the runner..."[0m
2024-07-08T16:59:30.3563110Z [36;1m  .github/scripts/stop_runner_service.sh[0m
2024-07-08T16:59:30.3563575Z [36;1mfi[0m
2024-07-08T16:59:30.3563858Z [36;1m[0m
2024-07-08T16:59:30.3564213Z [36;1m# Check the GPU count to be a power of 2[0m
2024-07-08T16:59:30.3565051Z [36;1mif [ "$GPU_COUNT" -le 8 ] && [ "$GPU_COUNT" -ne 1 ] && [ "$GPU_COUNT" -ne 2 ] && [ "$GPU_COUNT" -ne 4 ] && [ "$GPU_COUNT" -ne 8 ]; then[0m
2024-07-08T16:59:30.3566327Z [36;1m  echo "NVIDIA driver detects $GPU_COUNT GPUs. The runner has a broken GPU, shutting it down..."[0m
2024-07-08T16:59:30.3567113Z [36;1m  .github/scripts/stop_runner_service.sh[0m
2024-07-08T16:59:30.3567569Z [36;1mfi[0m
2024-07-08T16:59:30.3575061Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:59:30.3575557Z env:
2024-07-08T16:59:30.3575921Z   GIT_DEFAULT_BRANCH: main
2024-07-08T16:59:30.3576380Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2024-07-08T16:59:30.3577109Z   DOCKER_CONTAINER_ID: 5517005639eb2a5523bdd8699b890ed51972774d391d8acbbc153c14ab664a71
2024-07-08T16:59:30.3577874Z   RUNNER_WORKSPACE: /home/ec2-user/actions-runner/_work/pytorch
2024-07-08T16:59:30.3578390Z ##[endgroup]
2024-07-08T16:59:30.3597479Z + nvidia-smi
2024-07-08T16:59:31.9293770Z Mon Jul  8 16:59:31 2024
2024-07-08T16:59:31.9294673Z +-----------------------------------------------------------------------------------------+
2024-07-08T16:59:31.9295529Z | NVIDIA-SMI 550.54.15              Driver Version: 550.54.15      CUDA Version: 12.4     |
2024-07-08T16:59:31.9296325Z |-----------------------------------------+------------------------+----------------------+
2024-07-08T16:59:31.9297116Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2024-07-08T16:59:31.9298014Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2024-07-08T16:59:31.9298717Z |                                         |                        |               MIG M. |
2024-07-08T16:59:31.9299269Z |=========================================+========================+======================|
2024-07-08T16:59:31.9429054Z |   0  NVIDIA A10G                    Off |   00000000:00:1E.0 Off |                    0 |
2024-07-08T16:59:31.9430636Z |  0%   33C    P0             56W /  300W |       0MiB /  23028MiB |      5%      Default |
2024-07-08T16:59:31.9431603Z |                                         |                        |                  N/A |
2024-07-08T16:59:31.9432302Z +-----------------------------------------+------------------------+----------------------+
2024-07-08T16:59:31.9432936Z
2024-07-08T16:59:31.9433692Z +-----------------------------------------------------------------------------------------+
2024-07-08T16:59:31.9434330Z | Processes:                                                                              |
2024-07-08T16:59:31.9435019Z |  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
2024-07-08T16:59:31.9435907Z |        ID   ID                                                               Usage      |
2024-07-08T16:59:31.9436477Z |=========================================================================================|
2024-07-08T16:59:31.9437125Z |  No running processes found                                                             |
2024-07-08T16:59:31.9437877Z +-----------------------------------------------------------------------------------------+
2024-07-08T16:59:32.5481341Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2024-07-08T16:59:34.1126078Z NVIDIA A10G
2024-07-08T16:59:34.5566518Z + NVIDIA_SMI_STATUS=0
2024-07-08T16:59:34.5567058Z + '[' 0 -ne 0 ']'
2024-07-08T16:59:34.5570923Z ++ nvidia-smi --list-gpus
2024-07-08T16:59:34.5571371Z ++ wc -l
2024-07-08T16:59:36.5771166Z + GPU_COUNT=1
2024-07-08T16:59:36.5771836Z + NVIDIA_SMI_STATUS=0
2024-07-08T16:59:36.5772737Z + '[' 0 -ne 0 ']'
2024-07-08T16:59:36.5773200Z + '[' 1 -le 8 ']'
2024-07-08T16:59:36.5773544Z + '[' 1 -ne 1 ']'
2024-07-08T16:59:36.5909524Z Post job cleanup.
2024-07-08T16:59:36.5959835Z Post job cleanup.
2024-07-08T16:59:36.6767426Z [command]/usr/bin/git version
2024-07-08T16:59:36.6803375Z git version 2.40.1
2024-07-08T16:59:36.6840919Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/29313eb5-600b-40be-b67a-13b0bc419d01' before making global git config changes
2024-07-08T16:59:36.6842154Z Adding repository directory to the temporary git global config as a safe directory
2024-07-08T16:59:36.6846549Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2024-07-08T16:59:36.6874492Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2024-07-08T16:59:36.6904265Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2024-07-08T16:59:36.7121392Z Entering 'android/libs/fbjni'
2024-07-08T16:59:36.7158872Z Entering 'third_party/FP16'
2024-07-08T16:59:36.7194791Z Entering 'third_party/FXdiv'
2024-07-08T16:59:36.7232100Z Entering 'third_party/NNPACK'
2024-07-08T16:59:36.7268922Z Entering 'third_party/VulkanMemoryAllocator'
2024-07-08T16:59:36.7306088Z Entering 'third_party/XNNPACK'
2024-07-08T16:59:36.7353177Z Entering 'third_party/benchmark'
2024-07-08T16:59:36.7388219Z Entering 'third_party/cpp-httplib'
2024-07-08T16:59:36.7425273Z Entering 'third_party/cpuinfo'
2024-07-08T16:59:36.7462393Z Entering 'third_party/cudnn_frontend'
2024-07-08T16:59:36.7500177Z Entering 'third_party/cutlass'
2024-07-08T16:59:36.7542982Z Entering 'third_party/eigen'
2024-07-08T16:59:36.7580639Z Entering 'third_party/fbgemm'
2024-07-08T16:59:36.7618397Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-07-08T16:59:36.7652961Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-07-08T16:59:36.7688737Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-07-08T16:59:36.7726728Z Entering 'third_party/fbgemm/third_party/googletest'
2024-07-08T16:59:36.7759960Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-07-08T16:59:36.7796871Z Entering 'third_party/flatbuffers'
2024-07-08T16:59:36.7835043Z Entering 'third_party/fmt'
2024-07-08T16:59:36.7869117Z Entering 'third_party/foxi'
2024-07-08T16:59:36.7904527Z Entering 'third_party/gemmlowp/gemmlowp'
2024-07-08T16:59:36.7940975Z Entering 'third_party/gloo'
2024-07-08T16:59:36.7973943Z Entering 'third_party/googletest'
2024-07-08T16:59:36.8008788Z Entering 'third_party/ideep'
2024-07-08T16:59:36.8041022Z Entering 'third_party/ideep/mkl-dnn'
2024-07-08T16:59:36.8080102Z Entering 'third_party/ittapi'
2024-07-08T16:59:36.8115680Z Entering 'third_party/kineto'
2024-07-08T16:59:36.8150823Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-07-08T16:59:36.8185155Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-07-08T16:59:36.8220184Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-07-08T16:59:36.8254540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-07-08T16:59:36.8290039Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-07-08T16:59:36.8321919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-07-08T16:59:36.8358282Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-07-08T16:59:36.8390399Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-07-08T16:59:36.8424921Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-07-08T16:59:36.8459655Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-07-08T16:59:36.8495253Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-07-08T16:59:36.8527373Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-07-08T16:59:36.8561461Z Entering 'third_party/mimalloc'
2024-07-08T16:59:36.8597180Z Entering 'third_party/nccl/nccl'
2024-07-08T16:59:36.8631826Z Entering 'third_party/nlohmann'
2024-07-08T16:59:36.8668777Z Entering 'third_party/onnx'
2024-07-08T16:59:36.8713950Z Entering 'third_party/onnx/third_party/benchmark'
2024-07-08T16:59:36.8748883Z Entering 'third_party/onnx/third_party/pybind11'
2024-07-08T16:59:36.8786020Z Entering 'third_party/opentelemetry-cpp'
2024-07-08T16:59:36.8820655Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-07-08T16:59:36.8855206Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-07-08T16:59:36.8889274Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-07-08T16:59:36.8922907Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-07-08T16:59:36.8957430Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-07-08T16:59:36.8992279Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-07-08T16:59:36.9024314Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-07-08T16:59:36.9057450Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-07-08T16:59:36.9091269Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-07-08T16:59:36.9125772Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-07-08T16:59:36.9175158Z Entering 'third_party/pocketfft'
2024-07-08T16:59:36.9210039Z Entering 'third_party/protobuf'
2024-07-08T16:59:36.9245083Z Entering 'third_party/protobuf/third_party/benchmark'
2024-07-08T16:59:36.9278953Z Entering 'third_party/protobuf/third_party/googletest'
2024-07-08T16:59:36.9315443Z Entering 'third_party/psimd'
2024-07-08T16:59:36.9348235Z Entering 'third_party/pthreadpool'
2024-07-08T16:59:36.9384047Z Entering 'third_party/pybind11'
2024-07-08T16:59:36.9418337Z Entering 'third_party/python-peachpy'
2024-07-08T16:59:36.9452340Z Entering 'third_party/sleef'
2024-07-08T16:59:36.9484189Z Entering 'third_party/tensorpipe'
2024-07-08T16:59:36.9517926Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-07-08T16:59:36.9553080Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-07-08T16:59:36.9585023Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-07-08T16:59:36.9619214Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-07-08T16:59:36.9652582Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-07-08T16:59:36.9702473Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2024-07-08T16:59:36.9719544Z http.https://github.com/.extraheader
2024-07-08T16:59:36.9727803Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2024-07-08T16:59:36.9755898Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2024-07-08T16:59:36.9960356Z Entering 'android/libs/fbjni'
2024-07-08T16:59:36.9981495Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0007275Z Entering 'third_party/FP16'
2024-07-08T16:59:37.0028342Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0050886Z Entering 'third_party/FXdiv'
2024-07-08T16:59:37.0071692Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0093375Z Entering 'third_party/NNPACK'
2024-07-08T16:59:37.0115703Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0141129Z Entering 'third_party/VulkanMemoryAllocator'
2024-07-08T16:59:37.0162317Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0186041Z Entering 'third_party/XNNPACK'
2024-07-08T16:59:37.0207766Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0242121Z Entering 'third_party/benchmark'
2024-07-08T16:59:37.0264202Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0287213Z Entering 'third_party/cpp-httplib'
2024-07-08T16:59:37.0307938Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0330635Z Entering 'third_party/cpuinfo'
2024-07-08T16:59:37.0352171Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0375669Z Entering 'third_party/cudnn_frontend'
2024-07-08T16:59:37.0396415Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0420274Z Entering 'third_party/cutlass'
2024-07-08T16:59:37.0439807Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0470661Z Entering 'third_party/eigen'
2024-07-08T16:59:37.0493327Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0518326Z Entering 'third_party/fbgemm'
2024-07-08T16:59:37.0540099Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0563961Z Entering 'third_party/fbgemm/third_party/asmjit'
2024-07-08T16:59:37.0585438Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0607497Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2024-07-08T16:59:37.0628234Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0652595Z Entering 'third_party/fbgemm/third_party/cutlass'
2024-07-08T16:59:37.0673501Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0701342Z Entering 'third_party/fbgemm/third_party/googletest'
2024-07-08T16:59:37.0721676Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0743714Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2024-07-08T16:59:37.0764774Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0789127Z Entering 'third_party/flatbuffers'
2024-07-08T16:59:37.0811277Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0837142Z Entering 'third_party/fmt'
2024-07-08T16:59:37.0857716Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0881816Z Entering 'third_party/foxi'
2024-07-08T16:59:37.0904199Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0928453Z Entering 'third_party/gemmlowp/gemmlowp'
2024-07-08T16:59:37.0949746Z http.https://github.com/.extraheader
2024-07-08T16:59:37.0971621Z Entering 'third_party/gloo'
2024-07-08T16:59:37.0993553Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1017383Z Entering 'third_party/googletest'
2024-07-08T16:59:37.1038956Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1062976Z Entering 'third_party/ideep'
2024-07-08T16:59:37.1084449Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1105919Z Entering 'third_party/ideep/mkl-dnn'
2024-07-08T16:59:37.1127877Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1155969Z Entering 'third_party/ittapi'
2024-07-08T16:59:37.1177574Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1199163Z Entering 'third_party/kineto'
2024-07-08T16:59:37.1221800Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1244814Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2024-07-08T16:59:37.1266889Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1290458Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2024-07-08T16:59:37.1311191Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1336176Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2024-07-08T16:59:37.1357005Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1380540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2024-07-08T16:59:37.1400937Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1425141Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2024-07-08T16:59:37.1446424Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1468314Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2024-07-08T16:59:37.1489708Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1512941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2024-07-08T16:59:37.1534360Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1557667Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2024-07-08T16:59:37.1579165Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1601241Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2024-07-08T16:59:37.1622875Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1647811Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2024-07-08T16:59:37.1667743Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1693323Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2024-07-08T16:59:37.1715337Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1737991Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2024-07-08T16:59:37.1758946Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1784787Z Entering 'third_party/mimalloc'
2024-07-08T16:59:37.1807679Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1830197Z Entering 'third_party/nccl/nccl'
2024-07-08T16:59:37.1852502Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1877127Z Entering 'third_party/nlohmann'
2024-07-08T16:59:37.1898890Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1922442Z Entering 'third_party/onnx'
2024-07-08T16:59:37.1943922Z http.https://github.com/.extraheader
2024-07-08T16:59:37.1978536Z Entering 'third_party/onnx/third_party/benchmark'
2024-07-08T16:59:37.1999577Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2022218Z Entering 'third_party/onnx/third_party/pybind11'
2024-07-08T16:59:37.2042646Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2066832Z Entering 'third_party/opentelemetry-cpp'
2024-07-08T16:59:37.2089514Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2113404Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2024-07-08T16:59:37.2134090Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2157114Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2024-07-08T16:59:37.2178196Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2201070Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2024-07-08T16:59:37.2221760Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2245043Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2024-07-08T16:59:37.2266892Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2290070Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2024-07-08T16:59:37.2311457Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2333638Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2024-07-08T16:59:37.2352641Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2373631Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2024-07-08T16:59:37.2394522Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2416504Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2024-07-08T16:59:37.2436769Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2461971Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2024-07-08T16:59:37.2481554Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2507879Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2024-07-08T16:59:37.2529328Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2566820Z Entering 'third_party/pocketfft'
2024-07-08T16:59:37.2588634Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2611931Z Entering 'third_party/protobuf'
2024-07-08T16:59:37.2633317Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2656997Z Entering 'third_party/protobuf/third_party/benchmark'
2024-07-08T16:59:37.2677355Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2701273Z Entering 'third_party/protobuf/third_party/googletest'
2024-07-08T16:59:37.2722066Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2746295Z Entering 'third_party/psimd'
2024-07-08T16:59:37.2768524Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2790880Z Entering 'third_party/pthreadpool'
2024-07-08T16:59:37.2813240Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2835934Z Entering 'third_party/pybind11'
2024-07-08T16:59:37.2857034Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2879357Z Entering 'third_party/python-peachpy'
2024-07-08T16:59:37.2900419Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2921283Z Entering 'third_party/sleef'
2024-07-08T16:59:37.2943909Z http.https://github.com/.extraheader
2024-07-08T16:59:37.2967450Z Entering 'third_party/tensorpipe'
2024-07-08T16:59:37.2990435Z http.https://github.com/.extraheader
2024-07-08T16:59:37.3013711Z Entering 'third_party/tensorpipe/third_party/googletest'
2024-07-08T16:59:37.3034663Z http.https://github.com/.extraheader
2024-07-08T16:59:37.3055311Z Entering 'third_party/tensorpipe/third_party/libnop'
2024-07-08T16:59:37.3076522Z http.https://github.com/.extraheader
2024-07-08T16:59:37.3099064Z Entering 'third_party/tensorpipe/third_party/libuv'
2024-07-08T16:59:37.3119098Z http.https://github.com/.extraheader
2024-07-08T16:59:37.3141769Z Entering 'third_party/tensorpipe/third_party/pybind11'
2024-07-08T16:59:37.3162252Z http.https://github.com/.extraheader
2024-07-08T16:59:37.3184855Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2024-07-08T16:59:37.3205383Z http.https://github.com/.extraheader
2024-07-08T16:59:37.3332065Z A job completed hook has been configured by the self-hosted runner administrator
2024-07-08T16:59:37.3349404Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh'
2024-07-08T16:59:37.3355954Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2024-07-08T16:59:37.3356471Z ##[endgroup]
2024-07-08T16:59:37.9561403Z Cleaning up orphan processes