Terraform：存量資源管理

環境準備

1. 下載和安裝

   開源 Terraformer 發布包中既包含了針對所有 Provider 的包，又針對每個 Provider 提供了一個單獨的包，您可根據自己的需要進行下載。此處展示了 Linux 以及 Mac 使用者下載的方式，更多方式可參考 Installation。

   • Linux 使用者

   export PROVIDER=all
   curl -LO "https://github.com/GoogleCloudPlatform/terraformer/releases/download/$(curl -s https://api.github.com/repos/GoogleCloudPlatform/terraformer/releases/latest | grep tag_name | cut -d '"' -f 4)/terraformer-${PROVIDER}-linux-amd64"
   chmod +x terraformer-${PROVIDER}-linux-amd64
   sudo mv terraformer-${PROVIDER}-linux-amd64 /usr/local/bin/terraformer

   • macOS 使用者

   export PROVIDER=all
   curl -LO "https://github.com/GoogleCloudPlatform/terraformer/releases/download/$(curl -s https://api.github.com/repos/GoogleCloudPlatform/terraformer/releases/latest | grep tag_name | cut -d '"' -f 4)/terraformer-${PROVIDER}-darwin-amd64"
   chmod +x terraformer-${PROVIDER}-darwin-amd64
   sudo mv terraformer-${PROVIDER}-darwin-amd64 /usr/local/bin/terraformer

2. 依賴條件

   1. 需要為當前帳號授予相關資源的唯讀許可權

   2. 安裝 Terraform

      說明

      若在 Cloud Shell 中執行可省略這一步。

      詳細步驟可參考入門文檔：在本地安裝和配置Terraform。

   3. 下載相關的 Provider，可通過以下兩種方式配置：

      • 在運行 Terraformer 命令的目錄下初始化 Provider

      • 將 Provider 下載到~/.terraform.d/plugins/路徑下

   4. 配置訪問憑證：目前只支援讀取本地 profile 這一種認證方式，預設會使用第一個憑證進行匯入。配置阿里雲訪問憑證可參考 配置與管理身份憑證。

用法

以 terraformer-all-darwin-amd64 v0.8.24 版本為例。

共支援了四個命令：help、import、plan、version

1. help：查看當前命令詳情

   ➜  ~ terraformer-all-darwin-arm64 help
   Usage:
       [command]
   
   Available Commands:
     help        Help about any command
     import      Import current state to Terraform configuration
     plan        Plan to import current state to Terraform configuration
     version     Print the version number of Terraformer
   
   Flags:
     -h, --help    help for this command
   
   Use " [command] --help" for more information about a command.

2. version：查看目前的版本

   ~ terraformer-all-darwin-arm64 version
   Terraformer v0.8.24

3. import：提供針對各個 Provider 的匯入功能

   ➜  ~ terraformer-all-darwin-arm64 import
   Import current state to Terraform configuration
   
   Usage:
     import [command]
   
   Available Commands:
     alicloud        Import current State to terraform configuration from alicloud
     auth0           Import current state to Terraform configuration from Auth0
     aws             Import current state to Terraform configuration from AWS
     azure           Import current state to Terraform configuration from Azure
     azuread         Import current state to Terraform configuration from Azure Active Directory
     azuredevops     Import current state to Terraform configuration from Azure DevOps
     cloudflare      Import current state to Terraform configuration from Cloudflare
     commercetools   Import current state to Terraform configuration from Commercetools
     datadog         Import current state to Terraform configuration from Datadog
     digitalocean    Import current state to Terraform configuration from DigitalOcean
     fastly          Import current state to Terraform configuration from Fastly
     github          Import current state to Terraform configuration from GitHub
     gitlab          Import current state to Terraform configuration from GitLab
     gmailfilter     Import current state to Terraform configuration from Gmail
     google          Import current state to Terraform configuration from Google Cloud
     grafana         Import current state to Terraform configuration from Grafana
     heroku          Import current state to Terraform configuration from Heroku
     honeycombio     Import current state to Terraform configuration from Honeycomb.io
     ibm             Import current state to Terraform configuration from ibm
     ionoscloud      Import current state to Terraform configuration from IONOS Cloud
     keycloak        Import current state to Terraform configuration from Keycloak
     kubernetes      Import current state to Terraform configuration from Kubernetes
     launchdarkly    Import current state to Terraform configuration from LaunchDarkly
     linode          Import current state to Terraform configuration from Linode
     logzio          Import current state to Terraform configuration from Logz.io
     mackerel        Import current state to Terraform configuration from Mackerel
     metal           Import current state to Terraform configuration from Equinix Metal
     mikrotik        Import current state to Terraform configuration from RouterOS
     myrasec         Import current state to Terraform configuration from Myra Security
     newrelic        Import current state to Terraform configuration from New Relic
     ns1             Import current state to Terraform configuration from NS1
     octopusdeploy   Import current state to Terraform configuration from Octopus Deploy
     okta            Import current State to terraform configuration from okta
     opal            Import current state to Terraform configuration from opal.dev
     openstack       Import current state to Terraform configuration from OpenStack
     opsgenie        Import current state to Terraform configuration from Opsgenie
     pagerduty       Import current state to Terraform configuration from PagerDuty
     panos           Import current state to Terraform configuration from a PAN-OS
     plan            Import planned state to Terraform configuration
     rabbitmq        Import current state to Terraform configuration from RabbitMQ
     tencentcloud    Import current state to Terraform configuration from Tencent Cloud
     vault           Import current state to Terraform configuration from Vault
     vultr           Import current state to Terraform configuration from Vultr
     xenorchestra    Import current state to Terraform configuration from Xen Orchestra
     yandex          Import current state to Terraform configuration from Yandex Cloud
   
   Flags:
     -h, --help   help for import
   
   Additional help topics:
     import no-sort        Don't sort resources
   
   Use " import [command] --help" for more information about a command.

4. plan：提供針對各個 Provider 匯入的預覽功能

樣本

以匯入杭州地區下所有的阿里雲 VPC 為例：

1. 建立一個目錄用來存放匯入後的模板和狀態檔案

   mkdir example && cd example

2. 執行以下命令

   terraformer-all-darwin-arm64 import alicloud --resources=vpc --regions=cn-hangzhou --path-pattern={output} --path-output=./     

   • --resources：指定要匯入的資源集合

   • --regions：指定資源所在地區

   • --path-pattern：產生的目錄模式

   • --path-output：目錄檔案路徑

3. 產生的模板檔案如下所示，目錄路徑上會預設加上地區資訊。若需要將所有資源模板壓縮到一個檔案中，可指定 --compact參數，將資源都輸出到 resources.tf中

   └── cn-hangzhou
       ├── outputs.tf
       ├── provider.tf
       ├── terraform.tfstate
       ├── variables.tf
       ├── vpc.tf
       └── vswitch.tf
   
   2 directories, 6 files

4. 若您本地的 Terraform 版本>=0.13.0，您需要執行以下命令來對狀態檔案進行修改以適配新版本 Terraform

   terraform state replace-provider -auto-approve "registry.terraform.io/-/alicloud" "aliyun/alicloud"

環境準備

1. 下載和安裝

   目前 aliterraformer 儲存在阿里雲的OSS上，可按照如下的地址進行下載：

   macOS

   # amd64
   wget https://terraform-share.oss-cn-hangzhou.aliyuncs.com/aliterraformer/aliterraformer_darwin_amd64.zip
   
   # arm64
   wget https://terraform-share.oss-cn-hangzhou.aliyuncs.com/aliterraformer/aliterraformer_darwin_arm64.zip

   Linux

   # amd64
   wget https://terraform-share.oss-cn-hangzhou.aliyuncs.com/aliterraformer/aliterraformer_linux_amd64.zip
   
   # arm
   wget https://terraform-share.oss-cn-hangzhou.aliyuncs.com/aliterraformer/aliterraformer_linux_arm.zip
   
   # arm64
   wget https://terraform-share.oss-cn-hangzhou.aliyuncs.com/aliterraformer/aliterraformer_linux_arm64.zip

   Windows

   # amd64
   wget https://terraform-share.oss-cn-hangzhou.aliyuncs.com/aliterraformer/aliterraformer_windows_amd64.zip
   
   # 386
   wget https://terraform-share.oss-cn-hangzhou.aliyuncs.com/aliterraformer/aliterraformer_windows_386.zip

   以 macOS 系統為例，下載之後，解壓檔案：

   unzip aliterraformer_darwin_amd64.zip
   chmod +x aliterraformer
   sudo mv aliterraformer /usr/local/bin/aliterraformer

2. 依賴條件

   與開源 Terraformer 相同，也需要準備 terraform 和 terraform-provider-alicloud 這兩個檔案，並配置存取控制的配置。

   1. 為 RAM 帳號授予相關資源的唯讀許可權

   2. 安裝 Terraform

      詳細步驟可參考入門文檔：在本地安裝和配置Terraform。若在 Cloud Shell 中執行可省略這一步

   3. 下載 terraform-provider-alicloud

      通過以下方式可設定匯入的 Provider 版本，執行 init 後的緩衝目錄在 v0.13.0 前後的結構不同，此處展示 v0.13.0 版本之後的讀取方式

      # 1. 通過環境變數 TF_DATA_TFER_DIR 來設定
      <$TF_DATA_TFER_DIR>/providers/registry.terraform.io/aliyun/alicloud/1.239.0/darwin_arm64/terraform-provider-alicloud_v1.239.0
      
      # 2. 設定在當前工作目錄下的隱藏目錄
      .terraform/providers/registry.terraform.io/aliyun/alicloud/1.239.0/darwin_arm64/terraform-provider-alicloud_v1.239.0
      
      # 3. 設定在使用者根目錄下的隱藏目錄
      <HOME>/.terraform.d/plugins/registry.terraform.io/aliyun/alicloud/1.239.0/darwin_arm64/terraform-provider-alicloud_v1.239.0

   4. 配置訪問憑證

      aliterraformer 在 profile 的基礎上支援了通過環境變數和命令參數的方式設定訪問憑證：

      # 1. 通過環境變數 ALICLOUD_ACCESS_KEY 和 ALICLOUD_SECRET_KEY
      $ export ALICLOUD_ACCESS_KEY=xxxx
      $ export ALICLOUD_SECRET_KEY=xxxxx
      
      # 2. 通過 aliterraformer 命令參數 --access-key 和 --secret-key 傳遞
      $ aliterraformer import alicloud --access-key=xxxx --secret-key=xxx ...
      
      # 3. 通過 profile 機制來設定AK並傳遞
      $ aliterraformer import alicloud --profile=default ...

用法

aliterraformer 目前共支援了五個命令：help，version，products，resources，import。在開源 Terraformer 的基礎上增加了 products、resources 兩個命令

1. help：查看當前支援的命令詳情

   aliterraformer help

   → ~ aliterraformer help
   Usage:
       [command]
   
   Available Commands:
     help        Help about any command
     import      Import current state to Terraform configuration
     products    List all supported products
     resources   List resources for a given product
     version     Print the version number of Terraformer
   
   Flags:
     -h, --help   help for this command
   
   Use " [command] --help" for more information about a command.

2. version：查看當前的 terraformer 版本

   aliterraformer version

3. products：查看目前的版本支援匯入的產品列表

   aliterraformer products

   ➜  ~ aliterraformer products
   
   Supported products:
   
       ACK                         Container_Service_for_Kubernetes(ACK)
       ADB                         AnalyticDB_for_MySQL(ADB)
       ALB                         Application_Load_Balancer(ALB)
       ActionTrail                 Actiontrail
       AliKafka                    AliKafka
       Alidns                      Alidns
       Aligreen                    Aligreen
       Amqp                        RabbitMQ(AMQP)
       ApiGateway                  Api_Gateway
       AutoScaling                 Auto_Scaling
       BPStudio                    Cloud_Architect_Design_Tools(BPStudio)
       Bastionhost                 Bastion_Host
       CBWP                        EIP_Bandwidth_Plan(CBWP)
       CDN                         CDN
       CEN                         Cloud_Enterprise_Network(CEN)
       ClickHouse                  Click_House
       CloudMonitorService         Cloud_Monitor_Service
       CloudStorageGateway         Cloud_Storage_Gateway
       Cloudauth                   Cloudauth
       Config                      Cloud_Config(Config)
       DBFS                        Database_File_System(DBFS)
       DBS                         Database_Backup(DBS)
       DCDN                        DCDN
       DFS                         Apsara_File_Storage_for_HDFS(DFS)
       DMSEnterprise               DMS_Enterprise
       DTS                         Data_Transmission_Service(DTS)

4. resources：查看目前的版本某一產品下支援匯入的資源清單，通過 -p 指定

   aliterraformer resources -p <productName>

   ~ aliterraformer resources -p VPC
   
   Supported resources in product [VPC]:
   
       route_table
       vpc
       vpc_dhcp_options_set
       vpc_flow_log
       vpc_ha_vip
       vpc_ipv4_cidr_block
       vpc_ipv4_gateway
       vpc_ipv6_egress_rule
       vpc_ipv6_gateway
       vpc_ipv6_internet_bandwidth
       vpc_network_acl_attachment
       vpc_prefix_list
       vpc_public_ip_address_pool
       vpc_public_ip_address_pool_cidr_block
       vpc_traffic_mirror_filter
       vpc_traffic_mirror_session
       vswitch

   ~ aliterraformer resources -p ACK
   
   Supported resources in product [ACK]:
   
       cs_edge_kubernetes
       cs_kubernetes
       cs_kubernetes_addon
       cs_kubernetes_node_pool
       cs_kubernetes_permissions
       cs_managed_kubernetes
       cs_serverless_kubernetes

5. import：執行具體的匯入。阿里雲資源需要指定 alicloud，通過以下命令可查看阿里雲資源匯入支援的參數：

   aliterraformer import alicloud -h

   ➜  ~ aliterraformer import alicloud -h
   Import current State to terraform configuration from alicloud
   
   Usage:
       import alicloud [flags]
       import alicloud [command]
   
   Available Commands:
     list        List supported resources for alicloud provider
   
   Flags:
           --access-key string           Alibaba Cloud Access Key
     -b, --bucket string               gs://terraform-state
     -C, --compact
     -c, --connect                      (default true)
     -x, --excludes strings            vpc,subnet,nacl
           --excludes-products strings   ECS,VPC (Specify the products to be excluded.)
     -f, --filter strings              slb=id1:id2:id4
     -h, --help                        help for alicloud
           --modularize                  Whether output the terraform configuration to a module
     -O, --output string               output format hcl or json (default "hcl")
           --output-variables string     r1:attr1,attr2;r2:attr1,attr2,attr3
     -o, --path-output string           (default "generated")
     -p, --path-pattern string         {output}/{provider}/ (default "{output}/{provider}/{service}/")
           --products strings            ECS,VPC (You can input 'ALL' to import all products; Once specified, the flag <resources> will be ignored.)
           --profile string              prod (default "default")
           --provider-version string     Terraform Provider Alibaba Cloud Version
           --regions strings             cn-hangzhou
     -r, --resources strings           vpc,subnet,nacl (This flag will be ignored if the flag <products> is specified.)
     -n, --retry-number int            number of retries to perform when refresh fails (default 5)
     -m, --retry-sleep-ms int          time in ms to sleep between retries (default 300)
           --secret-key string           Alibaba Cloud Secret Key
           --security-token string       Alibaba Cloud Security Token
     -s, --state string                local or bucket (default "local")
           --statefile string            terraform.tfstate (default "terraform.tfstate")
           --terraform-version string    Terraform Version (default "0.12.29")
     -v, --verbose
           --zone string                 availability zone, like cn-hangzhou-h
   
   Use " import alicloud [command] --help" for more information about a command.

   import 時需要先確定需要匯入的資源集合，再按需通過 filter 對資源進行過濾、通過 output-variables 變數將屬性抽取為參數

   • 構建資源集合：

     當前命令提供了四個 flag：<--products>、<--resources>、<--excludes-products>、<--excludes> 來指定要匯入的資源，通過對其進行組合可以便利地構造出資源集合。用法樣本如下：

     • 匯入指定的資源

       # 匯入 vpc、vswitch、ECS 執行個體
       aliterraformer import alicloud -r=vpc,vswitch,instance

     • 匯入指定的產品

       # 匯入 ACK、ALB 產品下所有資源
       aliterraformer import alicloud --products=ACK,ALB

     • 匯入指定的產品及資源

       # 匯入 ACK、ECS 產品下所有資源以及所有 vpc、vswitch
       aliterraformer import alicloud --products=ACK,ECS -r=vpc,vswitch

     • 匯入指定的產品，排除產品下部分資源

       # 匯入 ACK 下除 alicloud_cs_kubernetes 外所有資源
       aliterraformer import alicloud --products=ACK --excludes=cs_kubernetes

     • 匯入所有產品下的所有資源

       # products 輸入 ALL 代表所有的產品
       aliterraformer import alicloud --products=ALL

     • 匯入所有產品下所有資源，排除指定的部分產品

       # 匯入除 VPNGateway,WAF 外的所有產品
       aliterraformer import alicloud --products=ALL --excludes-products=VPNGateway,WAF

     • 匯入所有產品下所有資源，排除指定的部分產品及部分資源

       # 匯入除 VPNGateway,WAF 外的所有產品，排除 alicloud_cs_kubernetes 資源
       aliterraformer import alicloud --products=ALL --excludes-products=VPNGateway,WAF --excludes=cs_kubernetes

   • 通過 filter 對資源進行過濾

     此參數的格式為 --filter=Type1.AttrKey=AttrValue1;AttrValue2,Type2.AttrKey=AttrValue1;AttrValue2，Type 若預設，此過濾條件對所有資源生效

     • 匯入 vpc id為 vpc-123 的專用網路

       # 若指定多個，用分號進行分割：--filter="vpc.id=vpc-123;vpc-456"
       aliterraformer import alicloud -r=vpc --filter="vpc.id=vpc-123"

     • 匯入名稱均為 tf-example 的 VPC 和 vSwitch

       aliterraformer import alicloud -r=vpc,vswitch --filter="vpc.vpc_name=tf-example,vswitch.vswitch_name=tf-example"

     • 匯入資源群組均為rg-12345的 VPC、ECS 執行個體

       aliterraformer import alicloud -r=vpc,instance --filter="resource_group_id=rg-12345"

   • 通過 output-variables 指定將屬性抽取為參數變數

     當前命令的格式為r1:attr1,attr2;r2:attr1,attr2,attr3

     • 指定將 ECS 執行個體中 vpc_id、vswitch_id 抽取為變數

       aliterraformer import alicloud -r=instance --output-variables="instance:vpc_id,vswitch_id"

樣本

下面展示如何匯入杭州地區下ID為vpc-12345的 VPC、vSwitch 和 ECS 執行個體，並將交換器的 zone_id 抽取為變數：

1. 建立一個目錄用來存放匯入後的模板和狀態檔案

   mkdir example && cd example

2. 構造匯入命令

   aliterraformer import alicloud \
       --regions=cn-hangzhou \
       -r=vpc,vswitch,instance \
       --filter="vpc.id=vpc-12345,vswitch.vpc_id=vpc-12345,instance.vpc_id=vpc-12345" \
       --output-variables="vswitch:zone_id" \
       --path-output=example               \
       --path-pattern={output}            \
       --compact

   • --path-output：指定匯入的檔案夾

   • --path-pattern：匯入資源模板的目錄結構格式，預設格式{output}/{provider}/{service}/，可選格式還有{output}/{provider}/以及{output}/。例如指定 --path-pattern={output}代表將所有模板平鋪到一個檔案夾下

   • --compact：不指定時資源將按照{service}.tf分開存放，指定後統一存放到resources.tf下

3. 執行匯入，在 example 目錄產生了如下檔案

   .
   ├── outputs.tf
   ├── provider.tf
   ├── resources.tf
   ├── terraform.tfstate
   └── variables.tf

   若不指定--path-pattern={output}，目錄結構如下所示：

   .
   └── alicloud
       ├── instance
       │   ├── provider.tf
       │   ├── terraform.tfstate
       │   └── variables.tf
       ├── vpc
       │   ├── outputs.tf
       │   ├── provider.tf
       │   ├── terraform.tfstate
       │   ├── variables.tf
       │   └── vpc.tf
       └── vswitch
           ├── outputs.tf
           ├── provider.tf
           ├── terraform.tfstate
           ├── variables.tf
           └── vswitch.tf

4. 若您本地的 Terraform 版本>=0.13.0，您需要執行以下命令來對狀態檔案進行修改以適配新版本 Terraform

   terraform state replace-provider -auto-approve "registry.terraform.io/-/alicloud" "aliyun/alicloud"