Add new_security_group option to create dedicated SG

[yotaenom]

Adds new_security_group and new_security_group_name variables so users can choose between modifying the default VPC security group or creating a new dedicated one.

[yotaenom]

Verified that the new new_security_group option works as expected. Default SG is left untouched and a new dedicated SG is created with the correct inbound/outbound rules. Screenshots attached.

[haleyyyblue]

Hi @yotaenom I tested this with an existing VPC and encountered the following error:

Attribute security_group_ids requires 1 item minimum, but config has only 0 declared.

In the current implementation, when vpc_id is provided (existing VPC case), the aws_security_group.databricks resource is not created (count = 0). As a result, if security_group_ids is also empty, the value passed to databricks_mws_networks becomes an empty list.

Since the Databricks resource requires at least one security group, the workspace deployment fails during validation.

To align with the requirement, we should ensure that a security group is always available. Specifically:

• If security_group_ids is provided → use it
• If security_group_ids is empty → create a new security group and use it (for both new and existing VPC cases)

This would ensure consistent behavior and prevent deployment failures when using an existing VPC.

│ Error: Not enough list items
│ 
│   with databricks_mws_networks.this,
│   on workspace.tf line 38, in resource "databricks_mws_networks" "this":
│   38:   security_group_ids = length(var.security_group_ids) > 0 ? var.security_group_ids : aws_security_group.databricks[*].id
│ 
│ Attribute security_group_ids requires 1 item minimum, but config has only 0 declared.
╵