This project is no longer being developed or maintained by its original authors.

The official OpenShift installer now supports various cloud providers including OpenStack so a lot of the development effort has moved there:

https://github.com/openshift/openshift-ansible/tree/master/playbooks/openstack

We recommend you take a look at it.

A collection of documentation, Heat templates, configuration and everything else that’s necessary to deploy OpenShift on OpenStack.

This template uses Heat to create the OpenStack infrastructure components, then calls the OpenShift Ansible installer playbooks to install and configure OpenShift on the VMs.

All of the OpenShift VMs will share a private network. This network is connected to the public network by a router.

The deployed OpenShift environment is composed of a replicated set of OpenShift master VMs fronted by a load_balancer. This provides both a single point of access and some HA capabilities. The applications run on one or more OpenShift node VMs. These are connected by a private software defined network (SDN) which can be implemented either with OpenVSwitch or Flannel.

A bastion server is used to control the host and service configuration. The host and service configuration is run using Ansible playbooks executed from the bastion host.

Bastion server, master nodes and infra nodes is also given a floating IP address on the public network. This provides direct access to the bastion server from where you can access all nodes by SSH. Master nodes and infra nodes have floating IP assigned to make sure these nodes are accessible when an external loadbalancer is used for accessing OpenShift services.

All of the OpenShift hosts (master, infra and node) have block storage for Docker images and containers provided by Cinder. OpenShift will run a local Docker registry, also backed by Cinder block storage. Finally all nodes will have access to Cinder volumes which can be created by OpenStack users and mounted into containers by Kubernetes.

1. OpenStack version Juno or later with the Heat, Neutron, Ceilometer, Aodh (Mitaka or later) services running:

   • heat-api-cfn service - used for passing heat metadata to nova instances

   • Neutron LBaaS service (optional) - used for loadbalancing requests in HA mode, if this service is not available, you can deploy dedicated loadbalancer node, see Select Loadbalancer Type

   • Ceilometer services (optional) - used when autoscaling is enabled

2. ServerGroupAntiAffinityFilter enabled in Nova service (optionally ServerGroupAffinityFilter when using all-in-one OpenStack environment)

3. CentOS 7.2 cloud image (we leverage cloud-init) loaded in Glance for OpenShift Origin Deployments. RHEL_ 7.2 cloud image if doing Atomic Enterprise or OpenShift Container Platform. Make sure to use official images to avoid unexpected issues during deployment (e.g. a custom firewall may block OpenShift inter-node communication).

4. An SSH keypair loaded into Nova

5. A (Neutron) network with a pool of floating IP addresses available

CentOS and RHEL are the only tested distros for now.

DNS Server

Red Hat Software Repositories

Table 1. Required Repositories for RHEL installation

Repo Name	Purpose
rhel-7-server-rpms	Standard RHEL Server RPMs
rhel-7-server-extras-rpms	Supporting RPMs
rhel-7-server-optional-rpms	Supporting RPMs
rhel-7-server-openstack-10-rpms	OpenStack client and data collection RPMs
rhel-7-server-ose-3.5-rpms	OpenShift Container Platform RPMs
rhel-7-fast-datapath-rpms	Required for OSP 3.5+ and OVS 2.6+

Following steps can be used to setup all-in-one testing/developer environment:

You can pass all environment variables to heat on command line. However, two environment files are provided as examples.

• env_origin.yaml is an example of the variables to deploy an OpenShift Origin 3 environment.

• env_aop.yaml is an example of the variables to deploy an Atomic Enterprise or OpenShift Container Platform 3 environment. Note deployment type should be openshift-enterprise for OpenShift or atomic-enterprise for Atomic Enterprise. Also, a valid RHN subscription is required for deployment.

Here is a sample of environment file which uses a subset of parameters which can be set by the user to configure OpenShift deployment. All configurable parameters including description can be found in the parameters section in the main template. Assuming your external network is called public, your SSH key is default, your CentOS 7.2 image is centos72 and your domain name is example.com, this is how you deploy OpenShift Origin:

After this you can deploy using the heat command

or using the generic OpenStack client

The node_count parameter specifies how many compute nodes you want to deploy. In the example above, we will deploy one master, one infra node and two compute nodes.

The templates will report stack completion back to Heat only when the whole OpenShift setup is finished.

Sometimes it’s necessary to find out why a stack was not deployed as expected. Debugging helps you find the root cause of the issue.

OpenShift on OpenStack takes advantage of the cloud provider to offer features such as dynamic storage to the OpenShift users. Auto scaling also requires communication with the OpenStack service. You must provide a set of OpenStack credentials so that OpenShift and the heat scaling mechanism can work correctly.

These are the same values used to create the Heat stack.

When invoking the stack creation, include this by adding -e osp_credentials.yaml to the command.

If your OpenStack service is encrypted with SSL/TLS, you will need to provide the CA certificate so that the communication channel can be validated.

The CA certificate is provided as a literal string copy of contents of the CA certificate file, and can be included in an additional environment file:

When invoking the stack creation, includ this by adding -e ca_certificates.yaml.

You can include multiple CA certificate strings and all will be imported into the CA list on all instances.

You can deploy OpenShift with multiple master hosts using the 'native' HA method (see https://docs.openshift.org/latest/install_config/install/advanced_install.html#multiple-masters for details) by increasing number of master nodes. This can be done by setting master_count heat parameter:

Three master nodes will be deployed. Console and API URLs point to the loadbalancer server which distributes requests across all three nodes. You can get the URLs from Heat by running heat output-show my-openshift console_url and heat output-show my-openshift api_url.

You can deploy OpenShift with multiple infra hosts. Then OpenShift router is deployed on each of infra node (only if -P deploy_router=true is used) and router requests are load balanced by either dedicated or neutron loadbalancer. This can be done by setting infra_count heat parameter:

Two infra nodes will be deployed. Loadbalancer server distributes requests on ports 80 and 443 across both nodes.

When deploying multiple master nodes, both access to the nodes and OpenShift router pods (which run on infra nodes) have to be loadbalanced. openshift-on-openstack provides multiple options for setting up loadbalancing:

• Neutron LBaaS - this loadbalancer is used by default. Neutron loadbalancer serviceis used for loadbalancing console/api requests to master nodes. At the moment OpenShift router requests are not loadbalanced and an external loadbalancer has to be used for it. This is default option, but can be set explicitly by including -e openshift-on-openstack/env_loadbalancer_neutron.yaml when creating the stack. By default, this mode uses IP failover.

• External loadbalancer - a user is expected to set its own loadbalancer both for master nodes and OpenShift routers. This is suggested type for production. To select this type include -e openshift-on-openstack/env_loadbalancer_external.yaml when creating the stack and also set lb_hostname parameter to point to the loadbalancer’s fully qualified domain name. Once stack creation is finished you can set your external loadbalancer with the list of created master nodes.

• Dedicated loadbalancer node - a dedicated node is created during stack creation and HAProxy loadbalancer is configured on it. Both console/API and OpenSHift router requests are load balanced by this dedicated node. This type is useful for demo/testing purposes only because HA is not assured for the single loadbalancer. To select this type include -e openshift-on-openstack/env_loadbalancer_dedicated.yaml when creating the stack. node.

• None - if only single master node is deployed, it’s possible to skip loadbalancer creation, then all master node requests and OpenShift router requests point to the single master node. To select this type include -e openshift-on-openstack/env_loadbalancer_none.yaml when creating the stack. By default, this mode uses IP failover.

By default, OpenShift is deployed with OpenShift-SDN. When used with OpenStack Neutron with GRE or VXLAN tunnels, packets are encapsulated twice which can have an impact on performances. Those Heat templates allow using Flannel instead of openshift-sdn, with the host-gw backend to avoid the double encapsulation. To do so, you need to include the env_flannel.yaml environment file when you create the stack:

To use this feature, the Neutron port_security extension driver needs to be enabled. To do so and when using the ML2 driver, edit the file /etc/neutron/plugins/ml2/ml2_conf.ini and make sure it contains the line:

Note that this feature is still in experimental mode.

You can use an external LDAP server to authenticate OpenShift users. Update parameters in env_ldap.yaml file and include this environment file when you create the stack.

Example of env_ldap.yaml using an Active Directory server:

If your LDAP service uses SSL, you will also need to add a CA Certficate for the LDAP communications.

You can set additional Yum repositories on deployed nodes by passing extra_repository_urls parameter which contains list of Yum repository URLs delimited by comma:

You can set additional Docker repositories on deployed nodes by passing extra_docker_repository_urls parameter which contains list of docker repository URLs delimited by comma, if a repository is insecure you can use #insecure suffix for the repository:

When deploying OpenShift registry (-P deploy_registry=true) you can use either an ephemeral or persistent Cinder volume. Ephemeral volume is used by default, the volume is automatically created when creating the stack and is also deleted when deleting the stack. Alternatively you can use an existing Cinder volume by including the env_registry_persistent.yaml environment file and registry_volume_id when you create the stack:

Persistent volume is not formatted when creating the stack, if you have a new unformatted volume you can enforce formatting by passing -P prepare_registry=true.

From user point of view there are two entry points into the deployed OpenShift:

• OpenShift console and API URLs: these URLs usually point to the loadbalancer host and can be obtained by:

• Router IP: the IP address which application OpenShift router service listens on. This IP will be used for setting wildcard DNS for .apps.<domain> subdomain. The IP can be obtained by:

To make sure that console and API URLs resolving works properly, you have to create a DNS record for the hostname used in console_url and api_url URLs. The floating IP address can be obtained by:

For example if console_url is https://default32-lb.example.com:8443/console/ and loadbalancer_ip is 172.24.4.166 there should be a DNS record for domain example.com:

If OpenShift router was deployed (-P deploy_router=true) you also may want to make sure that wildcard DNS is set for application subdomain. For example if used domain is example.com and router_ip is 172.24.4.168 there should be a DNS record for domain example.com:

Note	The above DNS records should be set on the DNS server authoritative for the domain used in OpenShift cluster (example.com in the example above).

Dynamic DNS Updates

You can retrieve the CA certificate that was generated during the OpenShift installation by running

OpenShift has preliminary support for local emptyDir volume quotas. You can set the volume_quota parameter to a resource quantity representing the desired quota per FSGroup.

You can set quota on the maximum size of the containers using the container_quota parameter in GB.

Example:

By default, the Heat templates create a Cinder volume per OpenShift node to host containers. This can be disabled by including both volume_noop.yaml and volume_attachment_noop.yaml in your environment file:

resource_registry: …​ OOShift::DockerVolume: volume_noop.yaml OOShift::DockerVolumeAttachment: volume_attachment_noop.yaml

These templates allow using IP failover for the OpenShift router. In this mode, a virtual IP address is assigned for the OpenShift router. Multiple instances of router may be active but only one instance at a time will have the virtual IP. This ensures that minimal downtime in the case of the failure of the current active router.

By default, IP failover is used when the load balancing mode is Neutron LBaas or None (see section Select Loadbalancer Type).

The virtual IP of the router can be retrieved with

You can manually scale up or down OpenShift nodes by updating node_count heat stack parameter to the desired new count:

If the stack has 2 nodes, 3 new nodes are added. If the stack has 7 nodes, 2 are removed. Any running pods are evacuated from the node being removed.

Scaling of OpenShift nodes can be automated by using Ceilometer metrics. By default cpu_util metering is used. You can enable autoscaling by autoscaling heat parameter and tweaking properties of cpu_alarm_high and cpu_alarm_low in openshift.yaml.

Sometimes it’s necessary to remove or replace specific nodes from the stack. For example because of a hardware issue. Because OpenShift "compute" nodes are members of heat AutoScalingGroup adding or removing nodes is by default handled by a scaling policy and when removing a node the oldest one is selected by Heat by default. A specific node can be removed with following steps though:

Here is the list of bugs which are not fixed and you may hit.

Those Heat templates make use of openshift-ansible to deploy OpenShift. You can provide additional parameters to openshift-ansible by specifying a JSON string as the extra_openshift_ansible_params parameter. For example :

This parameter must be used with caution as it may conflict with other parameters passed to openshift-ansible by the Heat templates.

1. The CA certificate used with OpenShift is currently not configurable.

2. The apps cloud domain is hardcoded for now. We need to make this configurable.

A customize-disk-image script is provided to preinstall OpenShift packages.

The modified image must be uploaded into Glance and used as the server image for the heat stack with the server_image parameter.

Copyright 2016 Red Hat, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.