External-Mirrors/kubespray
2
0
Fork 0
mirror of https://github.com/kubernetes-sigs/kubespray.git synced 2026-02-01 01:28:11 -03:30
Code Issues Packages Projects Releases Wiki Activity

WIP: terraform openstack

Browse Source
This commit is contained in:
Paul Czarkowski
2016-06-05 15:52:20 -05:00
parent 9bc5da9780
commit e924504928
8 changed files with 1416 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

137
contrib/terraform/openstack/README.md Normal file
View File

@@ -0,0 +1,137 @@
# Kubernetes on Openstack with Terraform
Provision a Kubernetes cluster with [Terraform](https://www.terraform.io) on
Openstack.
## Status
This will install a Kubernetes cluster on an Openstack Cloud. It is tested on a
OpenStack Cloud provided by [BlueBox](https://www.blueboxcloud.com/) and
should work on most modern installs of OpenStack that support the basic
services.
There are some assumptions made to try and ensure it will work on your openstack cluster.
* floating-ips are used for access
* you already have a suitable OS image in glance
* you already have both an internal network and a floating-ip pool created
* you have security-groups enabled
## Requirements
- [Install Terraform](https://www.terraform.io/intro/getting-started/install.html)
## Terraform
Terraform will be used to provision all of the OpenStack resources required to
run Docker Swarm. It is also used to deploy and provision the software
requirements.
### Prep
#### OpenStack
Ensure your OpenStack credentials are loaded in environment variables. This is
how I do it:
```
$ source ~/.stackrc
```
You will need two networks before installing, an internal network and
an external (floating IP Pool) network. The internet network can be shared as
we use security groups to provide network segregation. Due to the many
differences between OpenStack installs the Terraform does not attempt to create
these for you.
By default Terraform will expect that your networks are called `internal` and
`external`. You can change this by altering the Terraform variables `network_name` and `floatingip_pool`.
A full list of variables you can change can be found at [variables.tf](variables.tf).
All OpenStack resources will use the Terraform variable `cluster_name` (
default `example`) in their name to make it easier to track. For example the
first compute resource will be named `example-kubernetes-1`.
#### Terraform
Ensure your local ssh-agent is running and your ssh key has been added. This
step is required by the terraform provisioner:
```
$ eval $(ssh-agent -s)
$ ssh-add ~/.ssh/id_rsa
```
Ensure that you have your Openstack credentials loaded into Terraform
environment variables. Likely via a command similar to:
```
$ echo Setting up Terraform creds && \
export TF_VAR_username=${OS_USERNAME} && \
export TF_VAR_password=${OS_PASSWORD} && \
export TF_VAR_tenant=${OS_TENANT_NAME} && \
export TF_VAR_auth_url=${OS_AUTH_URL}
```
# Provision a Kubernetes Cluster on OpenStack
```
terraform apply -state=contrib/terraform/openstack/terraform.tfstate contrib/terraform/openstack
openstack_compute_secgroup_v2.k8s_master: Creating...
description: "" => "example - Kubernetes Master"
name: "" => "example-k8s-master"
rule.#: "" => "<computed>"
...
...
Apply complete! Resources: 9 added, 0 changed, 0 destroyed.
The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.
State path: contrib/terraform/openstack/terraform.tfstate
```
Make sure you can connect to the hosts:
```
$ ansible -i contrib/terraform/openstack/hosts -m ping all
example-k8s_node-1 | SUCCESS => {
"changed": false,
"ping": "pong"
}
example-etcd-1 | SUCCESS => {
"changed": false,
"ping": "pong"
}
example-k8s-master-1 | SUCCESS => {
"changed": false,
"ping": "pong"
}
```
if it fails try to connect manually via SSH ... it could be somthing as simple as a stale host key.
Deploy kubernetes:
```
$ ansible-playbook --become -i contrib/terraform/openstack/hosts cluster.yml
```
# clean up:
```
$ terraform destroy
Do you really want to destroy?
Terraform will delete all your managed infrastructure.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
...
...
Apply complete! Resources: 0 added, 0 changed, 12 destroyed.
```

136
contrib/terraform/openstack/group_vars/all.yml Normal file
View File

@@ -0,0 +1,136 @@
# Directory where the binaries will be installed
bin_dir: /usr/local/bin
# Where the binaries will be downloaded.
# Note: ensure that you've enough disk space (about 1G)
local_release_dir: "/tmp/releases"
# Uncomment this line for CoreOS only.
# Directory where python binary is installed
# ansible_python_interpreter: "/opt/bin/python"
# This is the group that the cert creation scripts chgrp the
# cert files to. Not really changable...
kube_cert_group: kube-cert
# Cluster Loglevel configuration
kube_log_level: 2
# Users to create for basic auth in Kubernetes API via HTTP
kube_api_pwd: "changeme"
kube_users:
kube:
pass: "{{kube_api_pwd}}"
role: admin
root:
pass: "changeme"
role: admin
# Kubernetes cluster name, also will be used as DNS domain
cluster_name: cluster.local
# For some environments, each node has a pubilcally accessible
# address and an address it should bind services to. These are
# really inventory level variables, but described here for consistency.
#
# When advertising access, the access_ip will be used, but will defer to
# ip and then the default ansible ip when unspecified.
#
# When binding to restrict access, the ip variable will be used, but will
# defer to the default ansible ip when unspecified.
#
# The ip variable is used for specific address binding, e.g. listen address
# for etcd. This is use to help with environments like Vagrant or multi-nic
# systems where one address should be preferred over another.
# ip: 10.2.2.2
#
# The access_ip variable is used to define how other nodes should access
# the node. This is used in flannel to allow other flannel nodes to see
# this node for example. The access_ip is really useful AWS and Google
# environments where the nodes are accessed remotely by the "public" ip,
# but don't know about that address themselves.
# access_ip: 1.1.1.1
# Choose network plugin (calico, weave or flannel)
kube_network_plugin: flannel
# Kubernetes internal network for services, unused block of space.
kube_service_addresses: 10.233.0.0/18
# internal network. When used, it will assign IP
# addresses from this range to individual pods.
# This network must be unused in your network infrastructure!
kube_pods_subnet: 10.233.64.0/18
# internal network total size (optional). This is the prefix of the
# entire network. Must be unused in your environment.
# kube_network_prefix: 18
# internal network node size allocation (optional). This is the size allocated
# to each node on your network. With these defaults you should have
# room for 4096 nodes with 254 pods per node.
kube_network_node_prefix: 24
# With calico it is possible to distributed routes with border routers of the datacenter.
peer_with_router: false
# Warning : enabling router peering will disable calico's default behavior ('node mesh').
# The subnets of each nodes will be distributed by the datacenter router
# The port the API Server will be listening on.
kube_apiserver_ip: "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}"
kube_apiserver_port: 443 # (https)
kube_apiserver_insecure_port: 8080 # (http)
# Internal DNS configuration.
# Kubernetes can create and mainatain its own DNS server to resolve service names
# into appropriate IP addresses. It's highly advisable to run such DNS server,
# as it greatly simplifies configuration of your applications - you can use
# service names instead of magic environment variables.
# You still must manually configure all your containers to use this DNS server,
# Kubernetes won't do this for you (yet).
# Upstream dns servers used by dnsmasq
upstream_dns_servers:
- 8.8.8.8
- 8.8.4.4
#
# # Use dns server : https://github.com/ansibl8s/k8s-skydns/blob/master/skydns-README.md
dns_setup: true
dns_domain: "{{ cluster_name }}"
#
# # Ip address of the kubernetes skydns service
skydns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(3)|ipaddr('address') }}"
dns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(2)|ipaddr('address') }}"
# There are some changes specific to the cloud providers
# for instance we need to encapsulate packets with some network plugins
# If set the possible values are either 'gce', 'aws' or 'openstack'
# When openstack is used make sure to source in the openstack credentials
# like you would do when using nova-client before starting the playbook.
# cloud_provider:
# For multi masters architecture:
# kube-proxy doesn't support multiple apiservers for the time being so you'll need to configure your own loadbalancer
# This domain name will be inserted into the /etc/hosts file of all servers
# configuration example with haproxy :
# listen kubernetes-apiserver-https
# bind 10.99.0.21:8383
# option ssl-hello-chk
# mode tcp
# timeout client 3h
# timeout server 3h
# server master1 10.99.0.26:443
# server master2 10.99.0.27:443
# balance roundrobin
# apiserver_loadbalancer_domain_name: "lb-apiserver.kubernetes.local"
## Set these proxy values in order to update docker daemon to use proxies
# http_proxy: ""
# https_proxy: ""
# no_proxy: ""
## A string of extra options to pass to the docker daemon.
## This string should be exactly as you wish it to appear.
## An obvious use case is allowing insecure-registry access
## to self hosted registries like so:
docker_options: "--insecure-registry={{ kube_service_addresses }}"

1
contrib/terraform/openstack/hosts Symbolic link
View File

@@ -0,0 +1 @@
../terraform.py

94
contrib/terraform/openstack/kubespray.tf Normal file
View File

@@ -0,0 +1,94 @@
resource "openstack_networking_floatingip_v2" "k8s_master" {
count = "${var.number_of_k8s_masters}"
pool = "${var.floatingip_pool}"
}
resource "openstack_networking_floatingip_v2" "k8s_node" {
count = "${var.number_of_k8s_nodes}"
pool = "${var.floatingip_pool}"
}
resource "openstack_compute_keypair_v2" "k8s" {
name = "kubernetes-${var.cluster_name}"
public_key = "${file(var.public_key_path)}"
}
resource "openstack_compute_secgroup_v2" "k8s_master" {
name = "${var.cluster_name}-k8s-master"
description = "${var.cluster_name} - Kubernetes Master"
}
resource "openstack_compute_secgroup_v2" "k8s" {
name = "${var.cluster_name}-k8s"
description = "${var.cluster_name} - Kubernetes"
rule {
ip_protocol = "tcp"
from_port = "22"
to_port = "22"
cidr = "0.0.0.0/0"
}
rule {
ip_protocol = "icmp"
from_port = "-1"
to_port = "-1"
cidr = "0.0.0.0/0"
}
rule {
ip_protocol = "tcp"
from_port = "1"
to_port = "65535"
self = true
}
rule {
ip_protocol = "udp"
from_port = "1"
to_port = "65535"
self = true
}
rule {
ip_protocol = "icmp"
from_port = "-1"
to_port = "-1"
self = true
}
}
resource "openstack_compute_instance_v2" "k8s_master" {
name = "${var.cluster_name}-k8s-master-${count.index+1}"
count = "${var.number_of_k8s_masters}"
image_name = "${var.image}"
flavor_id = "${var.flavor_k8s_master}"
key_pair = "${openstack_compute_keypair_v2.k8s.name}"
network {
name = "${var.network_name}"
}
security_groups = [ "${openstack_compute_secgroup_v2.k8s_master.name}",
"${openstack_compute_secgroup_v2.k8s.name}" ]
floating_ip = "${element(openstack_networking_floatingip_v2.k8s_master.*.address, count.index)}"
metadata = {
ssh_user = "${var.ssh_user}"
kubespray_groups = "etcd,kube-master,kube-node,k8s-cluster"
}
}
resource "openstack_compute_instance_v2" "k8s_node" {
name = "${var.cluster_name}-k8s-node-${count.index+1}"
count = "${var.number_of_k8s_nodes}"
image_name = "${var.image}"
flavor_id = "${var.flavor_k8s_node}"
key_pair = "${openstack_compute_keypair_v2.k8s.name}"
network {
name = "${var.network_name}"
}
security_groups = ["${openstack_compute_secgroup_v2.k8s.name}" ]
floating_ip = "${element(openstack_networking_floatingip_v2.k8s_node.*.address, count.index)}"
metadata = {
ssh_user = "${var.ssh_user}"
kubespray_groups = "kube-node,k8s-cluster"
}
}
#output "msg" {
# value = "Your hosts are ready to go!\nYour ssh hosts are: ${join(", ", openstack_networking_floatingip_v2.k8s_master.*.address )}"
#}

238
contrib/terraform/openstack/terraform.tfstate Normal file
View File

@@ -0,0 +1,238 @@
{
"version": 1,
"serial": 17,
"modules": [
{
"path": [
"root"
],
"outputs": {},
"resources": {
"openstack_compute_instance_v2.k8s_master.0": {
"type": "openstack_compute_instance_v2",
"depends_on": [
"openstack_compute_keypair_v2.k8s",
"openstack_compute_secgroup_v2.k8s",
"openstack_compute_secgroup_v2.k8s_master",
"openstack_networking_floatingip_v2.k8s_master"
],
"primary": {
"id": "f4a44f6e-33ff-4e35-b593-34f3dfd80dc9",
"attributes": {
"access_ip_v4": "173.247.105.12",
"access_ip_v6": "",
"flavor_id": "3",
"flavor_name": "m1.medium",
"floating_ip": "173.247.105.12",
"id": "f4a44f6e-33ff-4e35-b593-34f3dfd80dc9",
"image_id": "1525c3f3-1224-4958-bd07-da9feaedf18b",
"image_name": "ubuntu-14.04",
"key_pair": "kubernetes-example",
"metadata.#": "2",
"metadata.kubespray_groups": "etcd,kube-master,kube-node,k8s-cluster",
"metadata.ssh_user": "ubuntu",
"name": "example-k8s-master-1",
"network.#": "1",
"network.0.access_network": "false",
"network.0.fixed_ip_v4": "10.230.7.86",
"network.0.fixed_ip_v6": "",
"network.0.floating_ip": "173.247.105.12",
"network.0.mac": "fa:16:3e:fb:82:1d",
"network.0.name": "internal",
"network.0.port": "",
"network.0.uuid": "ba0fdd03-72b5-41eb-bb67-fef437fd6cb4",
"security_groups.#": "2",
"security_groups.2779334175": "example-k8s",
"security_groups.3772290257": "example-k8s-master",
"volume.#": "0"
}
}
},
"openstack_compute_instance_v2.k8s_master.1": {
"type": "openstack_compute_instance_v2",
"depends_on": [
"openstack_compute_keypair_v2.k8s",
"openstack_compute_secgroup_v2.k8s",
"openstack_compute_secgroup_v2.k8s_master",
"openstack_networking_floatingip_v2.k8s_master"
],
"primary": {
"id": "cbb565fe-a3b6-44ff-8f81-8ec29704d11b",
"attributes": {
"access_ip_v4": "173.247.105.70",
"access_ip_v6": "",
"flavor_id": "3",
"flavor_name": "m1.medium",
"floating_ip": "173.247.105.70",
"id": "cbb565fe-a3b6-44ff-8f81-8ec29704d11b",
"image_id": "1525c3f3-1224-4958-bd07-da9feaedf18b",
"image_name": "ubuntu-14.04",
"key_pair": "kubernetes-example",
"metadata.#": "2",
"metadata.kubespray_groups": "etcd,kube-master,kube-node,k8s-cluster",
"metadata.ssh_user": "ubuntu",
"name": "example-k8s-master-2",
"network.#": "1",
"network.0.access_network": "false",
"network.0.fixed_ip_v4": "10.230.7.85",
"network.0.fixed_ip_v6": "",
"network.0.floating_ip": "173.247.105.70",
"network.0.mac": "fa:16:3e:33:98:e6",
"network.0.name": "internal",
"network.0.port": "",
"network.0.uuid": "ba0fdd03-72b5-41eb-bb67-fef437fd6cb4",
"security_groups.#": "2",
"security_groups.2779334175": "example-k8s",
"security_groups.3772290257": "example-k8s-master",
"volume.#": "0"
}
}
},
"openstack_compute_instance_v2.k8s_node": {
"type": "openstack_compute_instance_v2",
"depends_on": [
"openstack_compute_keypair_v2.k8s",
"openstack_compute_secgroup_v2.k8s",
"openstack_networking_floatingip_v2.k8s_node"
],
"primary": {
"id": "39deed7e-8307-4b62-b56c-ce2b405a03fa",
"attributes": {
"access_ip_v4": "173.247.105.76",
"access_ip_v6": "",
"flavor_id": "3",
"flavor_name": "m1.medium",
"floating_ip": "173.247.105.76",
"id": "39deed7e-8307-4b62-b56c-ce2b405a03fa",
"image_id": "1525c3f3-1224-4958-bd07-da9feaedf18b",
"image_name": "ubuntu-14.04",
"key_pair": "kubernetes-example",
"metadata.#": "2",
"metadata.kubespray_groups": "kube-node,k8s-cluster",
"metadata.ssh_user": "ubuntu",
"name": "example-k8s-node-1",
"network.#": "1",
"network.0.access_network": "false",
"network.0.fixed_ip_v4": "10.230.7.84",
"network.0.fixed_ip_v6": "",
"network.0.floating_ip": "173.247.105.76",
"network.0.mac": "fa:16:3e:53:57:bc",
"network.0.name": "internal",
"network.0.port": "",
"network.0.uuid": "ba0fdd03-72b5-41eb-bb67-fef437fd6cb4",
"security_groups.#": "1",
"security_groups.2779334175": "example-k8s",
"volume.#": "0"
}
}
},
"openstack_compute_keypair_v2.k8s": {
"type": "openstack_compute_keypair_v2",
"primary": {
"id": "kubernetes-example",
"attributes": {
"id": "kubernetes-example",
"name": "kubernetes-example",
"public_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9nU6RPYCabjLH1LvJfpp9L8r8q5RZ6niS92zD95xpm2b2obVydWe0tCSFdmULBuvT8Q8YQ4qOG2g/oJlsGOsia+4CQjYEUV9CgTH9H5HK3vUOwtO5g2eFnYKSmI/4znHa0WYpQFnQK2kSSeCs2beTlJhc8vjfN/2HHmuny6SxNSbnCk/nZdwamxEONIVdjlm3CSBlq4PChT/D/uUqm/nOm0Zqdk9ZlTBkucsjiOCJeEzg4HioKmIH8ewqsKuS7kMADHPH98JMdBhTKbYbLrxTC/RfiaON58WJpmdOA935TT5Td5aVQZoqe/i/5yFRp5fMG239jtfbM0Igu44TEIib pczarkowski@Pauls-MacBook-Pro.local\n"
}
}
},
"openstack_compute_secgroup_v2.k8s": {
"type": "openstack_compute_secgroup_v2",
"primary": {
"id": "418394e2-b4be-4953-b7a3-b309bf28fbdb",
"attributes": {
"description": "example - Kubernetes",
"id": "418394e2-b4be-4953-b7a3-b309bf28fbdb",
"name": "example-k8s",
"rule.#": "5",
"rule.112275015.cidr": "",
"rule.112275015.from_group_id": "",
"rule.112275015.from_port": "1",
"rule.112275015.id": "597170c9-b35a-45c0-8717-652a342f3fd6",
"rule.112275015.ip_protocol": "tcp",
"rule.112275015.self": "true",
"rule.112275015.to_port": "65535",
"rule.2180185248.cidr": "0.0.0.0/0",
"rule.2180185248.from_group_id": "",
"rule.2180185248.from_port": "-1",
"rule.2180185248.id": "ffdcdd5e-f18b-4537-b502-8849affdfed9",
"rule.2180185248.ip_protocol": "icmp",
"rule.2180185248.self": "false",
"rule.2180185248.to_port": "-1",
"rule.3267409695.cidr": "",
"rule.3267409695.from_group_id": "",
"rule.3267409695.from_port": "-1",
"rule.3267409695.id": "4f91d9ca-940c-4f4d-9ce1-024cbd7d9c54",
"rule.3267409695.ip_protocol": "icmp",
"rule.3267409695.self": "true",
"rule.3267409695.to_port": "-1",
"rule.635693822.cidr": "",
"rule.635693822.from_group_id": "",
"rule.635693822.from_port": "1",
"rule.635693822.id": "c6816e5b-a1a4-4071-acce-d09b92d14d49",
"rule.635693822.ip_protocol": "udp",
"rule.635693822.self": "true",
"rule.635693822.to_port": "65535",
"rule.836640770.cidr": "0.0.0.0/0",
"rule.836640770.from_group_id": "",
"rule.836640770.from_port": "22",
"rule.836640770.id": "8845acba-636b-4c23-b9e2-5bff76d9008d",
"rule.836640770.ip_protocol": "tcp",
"rule.836640770.self": "false",
"rule.836640770.to_port": "22"
}
}
},
"openstack_compute_secgroup_v2.k8s_master": {
"type": "openstack_compute_secgroup_v2",
"primary": {
"id": "c74aed25-6161-46c4-a488-dfc7f49a228e",
"attributes": {
"description": "example - Kubernetes Master",
"id": "c74aed25-6161-46c4-a488-dfc7f49a228e",
"name": "example-k8s-master",
"rule.#": "0"
}
}
},
"openstack_networking_floatingip_v2.k8s_master.0": {
"type": "openstack_networking_floatingip_v2",
"primary": {
"id": "2a320c67-214d-4631-a840-2de82505ed3f",
"attributes": {
"address": "173.247.105.12",
"id": "2a320c67-214d-4631-a840-2de82505ed3f",
"pool": "external",
"port_id": ""
}
}
},
"openstack_networking_floatingip_v2.k8s_master.1": {
"type": "openstack_networking_floatingip_v2",
"primary": {
"id": "3adbfc13-e7ae-4bcf-99d3-3ba9db056e1f",
"attributes": {
"address": "173.247.105.70",
"id": "3adbfc13-e7ae-4bcf-99d3-3ba9db056e1f",
"pool": "external",
"port_id": ""
}
}
},
"openstack_networking_floatingip_v2.k8s_node": {
"type": "openstack_networking_floatingip_v2",
"primary": {
"id": "a3f77aa6-5c3a-4edf-b97e-ee211dfa81e1",
"attributes": {
"address": "173.247.105.76",
"id": "a3f77aa6-5c3a-4edf-b97e-ee211dfa81e1",
"pool": "external",
"port_id": ""
}
}
}
}
}
]
}

13
contrib/terraform/openstack/terraform.tfstate.backup Normal file
View File

@@ -0,0 +1,13 @@
{
"version": 1,
"serial": 16,
"modules": [
{
"path": [
"root"
],
"outputs": {},
"resources": {}
}
]
}

61
contrib/terraform/openstack/variables.tf Normal file
View File

@@ -0,0 +1,61 @@
variable "cluster_name" {
default = "example"
}
variable "number_of_k8s_masters" {
default = 2
}
variable "number_of_k8s_nodes" {
default = 1
}
variable "public_key_path" {
description = "The path of the ssh pub key"
default = "~/.ssh/id_rsa.pub"
}
variable "image" {
description = "the image to use"
default = "ubuntu-14.04"
}
variable "ssh_user" {
description = "used to fill out tags for ansible inventory"
default = "ubuntu"
}
variable "flavor_k8s_master" {
default = 3
}
variable "flavor_k8s_node" {
default = 3
}
variable "network_name" {
description = "name of the internal network to use"
default = "internal"
}
variable "floatingip_pool" {
description = "name of the floating ip pool to use"
default = "external"
}
variable "username" {
description = "Your openstack username"
}
variable "password" {
description = "Your openstack password"
}
variable "tenant" {
description = "Your openstack tenant/project"
}
variable "auth_url" {
description = "Your openstack auth URL"
}