resource "ibm_cm_validation" generate 504 HTTP Error. Multiple schematics validation workspaces are created and an HTTP 400 error is generate at the end

[fberzollaibm]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform IBM Provider Version

Terraform v1.10.5
on darwin_arm64

Affected Resource(s)

• ibm_cm_validation

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please share a link to the ZIP file.
resource "ibm_cm_validation" "my_cm_version_grafana_validation" {
 
  provider = ibm.target_account
 
  version_locator         = ibm_cm_version.cm_version_grafana.version_locator
  revalidate_if_validated = false
  mark_version_consumable = true
 
  environment_variables {
    name = "__netrc__"
    value = "[['${local.grit_region}.git.cloud.ibm.com', 'oauth', '${local.catalog_git_token}']]"
    secure = true
  }
 
  override_values = {
    ibmcloud_api_key                = "a"
    workload_git_token              = local.catalog_git_token
    ibm_schematics_workload_id      = "a"
    workload_env                    = "a"
    workload_app_component          = "a"
    dry_run                         = true
    workload_git_region_override    = true
    workload_git_region_overwritten = local.grit_region
  }
}
 

Debug Output

2025-02-13T09:48:43.263Z [INFO]  provider.terraform-provider-ibm_v1.75.2: 2025/02/13 09:48:43 [DEBUG] POST https://cm.globalcatalog.cloud.ibm.com/api/v1-beta/versions/020a73e3-5571-430d-8bba-cb5a471500f2.bbd5f569-b3e0-4ec7-bcd0-f51364bb9bef/validation/install (status: 504): retrying in 1s (10 left): timestamp=2025-02-13T09:48:43.263Z
2025-02-13T09:48:43.370Z [INFO]  provider.terraform-provider-ibm_v1.75.2: 2025/02/13 09:48:43 [Debug] Considering retry attempt; status_code=504, method=POST, url=https://cm.globalcatalog.cloud.ibm.com/api/v1-beta/versions/020a73e3-5571-430d-8bba-cb5a471500f2.82349a2e-b811-419e-b735-2ba4793189e0/validation/install, error=nil: timestamp=2025-02-13T09:48:43.369Z
2025-02-13T09:48:43.371Z [INFO]  provider.terraform-provider-ibm_v1.75.2: 2025/02/13 09:48:43 [Debug] Retry will be attempted: timestamp=2025-02-13T09:48:43.369Z
2025-02-13T09:48:43.371Z [INFO]  provider.terraform-provider-ibm_v1.75.2: 2025/02/13 09:48:43 [DEBUG] POST https://cm.globalcatalog.cloud.ibm.com/api/v1-beta/versions/020a73e3-5571-430d-8bba-cb5a471500f2.82349a2e-b811-419e-b735-2ba4793189e0/validation/install (status: 504): retrying in 1s (10 left): timestamp=2025-02-13T09:48:43.371Z
2025-02-13T09:48:43.476Z [INFO]  provider.terraform-provider-ibm_v1.75.2: 2025/02/13 09:48:43 [DEBUG] Status is in_progress: timestamp=2025-02-13T09:48:43.475Z
 

Expected Behavior

Only one validation schematics workspace should be created and the template will not fail.

Actual Behavior

Multiple validation schematics workspaces are created for the same ressource and at the end an error message occured

id: terraform-9f0a5d50
summary: |
  GetValidationStatusWithContext failed Cannot update status for version 020a73e3-5571-430d-8bba-cb5a471500f2.9d646c71-b02a-4658-bc73-2e2a19512476 using workspace us-south.workspace.globalcatalog-collection.697f841d.  The specified workspace is not for this version
  {
      "StatusCode": 400,
      "Headers": {
          "Cache-Control": [
              "no-cache, no-store, max-age=0, must-revalidate"
          ],
          "Cf-Cache-Status": [
              "DYNAMIC"
          ],
          "Cf-Ray": [
              "9113f00f389218d9-FRA"
          ],
          "Content-Length": [
              "314"
          ],
          "Content-Type": [
              "application/json; charset=utf-8"
          ],
          "Date": [
              "Thu, 13 Feb 2025 09:55:03 GMT"
          ],
          "Server": [
              "cloudflare"
          ],
          "Strict-Transport-Security": [
              "max-age=31536000"
          ],
          "Transaction-Id": [
              "78dd5b43-dd27-4cbd-9b5b-cafd3ecd3686"
          ],
          "Vary": [
              "Accept-Encoding"
          ],
          "X-Cm-Instance-Id": [
              "globalcatalog-eu-de-prod-content-mgmt-86c8f77784-wjgz9"
          ],
          "X-Content-Type-Options": [
              "nosniff"
          ],
          "X-Correlation-Id": [
              "78dd5b43-dd27-4cbd-9b5b-cafd3ecd3686"
          ],
          "X-Envoy-Upstream-Service-Time": [
              "30437"
          ],
          "X-Global-Transaction-Id": [
              "78dd5b43-dd27-4cbd-9b5b-cafd3ecd3686"
          ],
          "X-Request-Id": [
              "78dd5b43-dd27-4cbd-9b5b-cafd3ecd3686"
          ]
      },
      "Result": {
          "code": 400,
          "global_transaction_id": "78dd5b43-dd27-4cbd-9b5b-cafd3ecd3686",
          "message": "Cannot update status for version 020a73e3-5571-430d-8bba-cb5a471500f2.9d646c71-b02a-4658-bc73-2e2a19512476 using workspace us-south.workspace.globalcatalog-collection.697f841d.  The specified workspace is not for this version"
      },
      "RawResult": null
  }
severity: error
resource: ibm_cm_validation
operation: create
component:
  name: github.com/IBM-Cloud/terraform-provider-ibm
  version: 1.75.2
---
with ibm_cm_validation.my_cm_version_grafana_validation[0],
  on p15-grafana.tf line 547, in resource "ibm_cm_validation" "my_cm_version_grafana_validation":
547: resource "ibm_cm_validation" "my_cm_version_grafana_validation"
 

Steps to Reproduce

Important Factoids

References

[benbuchanan]

@fberzollaibm were these errors from different terraform apply runs? Or are they validating multiple versions at one time? It looks like the 504 error is for one version locator, and from our logs the initial validation request failed with a 504 from schematics, so terraform retried it, and then it was successful and went all the way through.

But the 400 error below is for a different version locator, that didn't get any 504s.

Cannot update status for version 020a73e3-5571-430d-8bba-cb5a471500f2.9d646c71-b02a-4658-bc73-2e2a19512476 using workspace us-south.workspace.globalcatalog-collection.697f841d.  The specified workspace is not for this version

[fberzollaibm]

@benbuchanan In Fact in the same terraform template we have multiple "product" that are installed and we are seeing the error on multiple "products"

So maybe the peace of log that I have choosen is not related to this version locator but I can put the ulle log if necessary.

[benbuchanan]

@fberzollaibm yeah that would be helpful, could you provide the full terraform configuration files, as well as a full log? Thanks.

[fberzollaibm]

fulllog.txt

[fberzollaibm]

@benbuchanan The code is quite BNPP proprietray I can't put it here but we can organize a web meeting to do a walkthru.

[benbuchanan]

@fberzollaibm no worries, the log there was enough to track down what happened. We noticed both the 504 and 400s were because of timeouts we were hitting in schematics, from these requests.

Error: Error[504 Gateway Timeout] - Post "https://schematics.cloud.ibm.com/v2/cart": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Outbound request: GET https://us-south.schematics.cloud.ibm.com/v1/workspaces/us-south.workspace.globalcatalog-collection.966d4c93 - 30.002411278s, connection err: Get "https://us-south.schematics.cloud.ibm.com/v1/workspaces/us-south.workspace.globalcatalog-collection.966d4c93": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I believe the schematics team is looking into it. cc @hkantare

[benbuchanan]

Update from schematics team:

We were seeing intermittent behaviour where the pod was consuming more CPU and memory resources. As an initial step, we have increased the resource limits. We will keep monitoring for any such occurrences. Could you please inform the customer and ask them to retry.

[fberzollaibm]

Just retried unfortunatly same error. See log associated.

err.log.zip