Elastic Stack Azure Marketplace offering

Easily deploy the Elastic Stack of Elasticsearch, Kibana and Logstash to Azure.

IMPORTANT: As a prefered solution, we recommend using our newer Elastic Cloud (Elasticsearch Service) offering in the Azure Marketplace. To learn more, refer to the Native Azure integration documentation.

This readme provides an overview of usage and features. For more comprehensive documentation, please refer to the Azure Marketplace and ARM template documentation

This repository consists of:

• src/mainTemplate.json - The main Azure Resource Management (ARM) template. The template itself is composed of many nested linked templates, with the main template acting as the entry point.
• src/createUiDefinition - UI definition file for our Azure Marketplace offering. This file produces an output JSON that the ARM template can accept as input parameters.

Building

After pulling the source, call the following once

npm install

to pull in all devDependencies. You may edit the build/allowedValues.json file, which the build uses to patch the ARM template and Marketplace UI definition. Then, run

npm run build

which will validate EditorConfig settings, lint JSON files, patch the template using build/allowedValues.json, and create a zip in the dist folder. For more details around developing the template, take a look at the Development README

Azure Marketplace

The Azure Marketplace Elastic Stack offering offers a simplified UI and installation experience over the full power of the ARM template.

It will always bootstrap an Elasticsearch cluster complete with a trial license of the Elastic Stack's platinum features.

Deploying through the Marketplace is great and easy way to get your feet wet for the first time with Elasticsearch on Azure, but in the long run, you'll want to deploy the templates directly from GitHub using the Azure CLI or PowerShell SDKs. Check out the CLI examples.

You can view the UI in developer mode by clicking here. If you feel something is cached improperly use this client unoptimized link instead

Reporting bugs

Have a look at this screenshot to see how you can navigate to the deployment error status message. Please create an issue with that message and in which resource it occured on our github issues

ARM template

The output from the Azure Marketplace UI is fed directly to the ARM deployment template. You can use the ARM template independently, without going through the Marketplace. In fact, there are many features in the ARM template that are not exposed within the Marketplace UI, such as configuring

• Azure Storage account to use with Azure Repository plugin for Snapshot/Restore
• Application Gateway to use for SSL/TLS and SSL offload

Check out our examples repository for examples of common scenarios and also take a look at the following blog posts for further information

• Spinning up a cluster with Elastic's Azure Marketplace template
• Elasticsearch and Kibana deployments on Azure
• SAML based Single Sign-On with Elasticsearch and Azure Active Directory

Elastic Stack features (formerly known as X-Pack)

Starting with Elasticsearch, Kibana and Logstash 6.3.0, The template deploys with Elastic Stack features bundled as part of the deployment, and includes the free features under the Basic license level. The xpackPlugins parameter determines whether a self-generated trial license is applied, offering a trial period of 30 days to the Platinum license features. A value of Yes applies a trial license, a value of No applies the Basic license. The license level applied determines the Elastic Stack features activated to use.

For Elasticsearch, Kibana and Logstash prior to 6.3.0, The xpackPlugins parameter determines whether X-Pack plugins are installed and a self-generated trial license is applied. In difference to 6.3.0 however, a value of No for xpackPlugins means that X-Pack plugins are not installed, and therefore does not provide the free features under the Basic license level, offering the Open Source features only. For these versions, you can install X-Pack plugins and register for a free Basic license to apply to the deployment, in order to use the free features available under the Basic license level.

Parameters

The ARM template accepts a lot of parameters, but don't fear! Most of them are optional and only used in conjunction with other parameters. Where a parameter value is not explicitly provided, it will take the default value defined in the template.

Web based deploy

The above button will take you to the autogenerated web based UI based on the parameters from the ARM template.

Command line deploy

You can deploy using the template directly from Github using the Azure CLI or Azure PowerShell

Azure CLI 1.0

Azure CLI 1.0 is no longer supported as the apiVersions of resources are newer than those supported by the last release. It's recommended to update to Azure CLI 2.0.

Azure CLI 2.0

1. Log into Azure

  az login

2. Create a resource group <name> in a <location> (e.g westeurope) where we can deploy too

az group create --name <name> --location <location>

3. Use our template directly from GitHub using --template-uri

az group deployment create \
  --resource-group <name> \
  --template-uri https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json \
  --parameters @parameters/password.parameters.json

where <name> refers to the resource group you just created.

Azure PowerShell

1. Log into Azure

Login-AzureRmAccount

2. Select a Subscription Id

Select-AzureRmSubscription -SubscriptionId "<subscriptionId>"

3. Define the parameters object for your deployment as a PowerShell hashtable. The keys correspond the parameters defined in the Parameters section

$branch = "master"
$esVersion = "7.11.1"

$clusterParameters = @{
    "_artifactsLocation" = "https://raw.githubusercontent.com/elastic/azure-marketplace/$branch/src/"
    "esVersion" = $esVersion
    "esClusterName" = "elasticsearch"
    "loadBalancerType" = "internal"
    "vmDataDiskCount" = 1
    "adminUsername" = "russ"
    "adminPassword" = "Password1234"
    "securityBootstrapPassword" = "Password1234"
    "securityAdminPassword" = "Password1234"     
    "securityKibanaPassword" = "Password1234"
    "securityLogstashPassword" = "Password1234"
    "securityBeatsPassword" = "Password1234"
    "securityApmPassword" = "Password1234"
    "securityRemoteMonitoringPassword" = "Password1234"
}

4. Create a resource group <name> in a <location> (e.g westeurope) where we can deploy too

New-AzureRmResourceGroup -Name "<name>" -Location "<location>"

5. Use our template directly from GitHub

New-AzureRmResourceGroupDeployment -Name "<deployment name>" -ResourceGroupName "<name>" -TemplateUri "https://raw.githubusercontent.com/elastic/azure-marketplace/master/src/mainTemplate.json" -TemplateParameterObject $clusterParameters

Targeting a specific template version

You can target a specific version of the template by modifying the URI of the template and the _artifactsLocation parameter of the template to point to a specific tagged release.

Targeting a specific template version is recommended for repeatable production deployments.

For example, to target the 7.11.1 tag release with PowerShell

$templateVersion = "7.11.1"
$_artifactsLocation = "https://raw.githubusercontent.com/elastic/azure-marketplace/$templateVersion/src/"

# minimum parameters required to deploy
$clusterParameters = @{
  "_artifactsLocation" = $_artifactsLocation
  "esVersion" = "7.11.1"
  "adminUsername" = "russ"
  "adminPassword" = "Password1234"
  "securityBootstrapPassword" = "Password1234"
  "securityAdminPassword" = "Password1234"
  "securityKibanaPassword" = "Password1234"
  "securityLogstashPassword" = "Password1234"
  "securityBeatsPassword" = "Password1234"
  "securityApmPassword" = "Password1234"
  "securityRemoteMonitoringPassword" = "Password1234"
}

$resourceGroup = "my-azure-cluster"
$location = "Australia Southeast"
$name = "my-azure-cluster"

New-AzureRmResourceGroup -Name $resourceGroup -Location $location
New-AzureRmResourceGroupDeployment -Name $name -ResourceGroupName $resourceGroup -TemplateUri "$_artifactsLocation/mainTemplate.json" -TemplateParameterObject $clusterParameters

Configuring TLS

It is strongly recommended that you secure communication using Transport Layer Security when using the template. The Elastic Stack security features can provide Basic Authentication, Role Based Access control, and Transport Layer Security (TLS) for both Elasticsearch and Kibana. For more details, please refer to the Security documentation.

For Elasticsearch versions 6.8.0+ (and less than 7.0.0), and 7.1.0+, the Elastic Stack security features that allow configuring TLS and role based access control are available in the free basic license tier. For all other versions, the Elastic Stack security features require a license level higher than basic; They can be configured with a trial license, which provides access to the Security features for 30 days.

TLS for Kibana

You can secure external access from the browser to Kibana with TLS by supplying a certificate and private key in PEM format with kibanaCertBlob and kibanaKeyBlob parameters, respectively.

TLS for Elasticsearch Transport layer

You can secure communication between nodes in the cluster with TLS on the Transport layer. Configuring TLS for the Transport layer requires xPackPlugins be set to Yes, or an Elasticsearch version 6.8.0+ (and less than 7.0.0) or 7.1.0+.

You must supply a PKCS#12 archive with the esTransportCaCertBlob parameter (and optional passphrase with esTransportCaCertPassword) containing the CA cert which should be used to generate a certificate for each node within the cluster. An optional passphrase can be passed with esTransportCertPassword to encrypt the generated certificate on each node.

One way to generate a PKCS#12 archive containing a CA certificate and key is using Elastic's elasticsearch-certutil command. The simplest command to generate a CA certificate is

./elasticsearch-certutil ca

and follow the instructions.

TLS for Elasticsearch HTTP layer

You can secure external access to the cluster with TLS with an external loadbalancer or Application Gateway. Configuring TLS for the HTTP layer requires xPackPlugins be set to Yes, or an Elasticsearch version 6.8.0+ (and less than 7.0.0) or 7.1.0+.

External load balancer

If you choose external as the value for loadBalancerType, you must either

• supply a PKCS#12 archive containing the key and certificate with the esHttpCertBlob parameter (and optional passphrase with esHttpCertPassword) containing the certs and private key to secure the HTTP layer. This certificate will be used by all nodes within the cluster, and

or

• supply a PKCS#12 archive containing the key and certificate with the esHttpCaCertBlob parameter (and optional passphrase with esHttpCaCertPassword) containing the CA which should be used to generate a certificate for each node within the cluster to secure the HTTP layer. Kibana will be configured to trust the CA and perform hostname verification for presented certificates. One way to generate a PKCS#12 archive is using Elastic's certutil command.

Application Gateway

If you choose gateway as the value for loadBalancerType, you must

• supply a PKCS#12 archive containing the key and certificate with the appGatewayCertBlob parameter (and optional passphrase with appGatewayCertPassword) to secure communication to Application Gateway. One way to generate a PKCS#12 archive is using Elastic's certutil command

Application Gateway performs SSL offload, so communication from Application Gateway to Elasticsearch is not encrypted with TLS by default. TLS to Application Gateway may be sufficient for your needs, but if you would like end-to-end encryption by also configuring TLS for Elasticsearch HTTP layer, you can

• supply a PKCS#12 archive containing the key and certificate with the esHttpCertBlob parameter (and optional passphrase with esHttpCertPassword) containing the certs and private key to secure the HTTP layer. This certificate will be used by all nodes within the cluster, and Kibana will be configured to trust the certificate CA (if CA certs are present within the archive). One way to generate a PKCS#12 archive is using Elastic's certutil command, and you must specify a --dns <name> argument with a name that matches that in the --name <name> argument.

and

• supply the public certificate in PEM format from the PKCS#12 archive passed with esHttpCertBlob parameter, using the appGatewayEsHttpCertBlob parameter. Application Gateway whitelists certificates used by VMs in the backend pool. This can be extracted from the PKCS#12 archive of the esHttpCertBlob parameter using openssl pkcs12

  openssl pkcs12 -in http_cert.p12 -out http_public_cert.cer -nokeys

  and provide the passphrase for the archive when prompted.

IMPORTANT: When configuring end-to-end encryption with Application Gateway, the certificate to secure the HTTP layer must include a x509v3 Subject Alternative Name extension with a DNS entry that matches the Subject CN, to work with Application Gateway's whitelisting mechanism. This can be checked using openssl x509

openssl x509 -in http_public_cert.cer -text -noout

which will output something similar to

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            // omitted for brevity ...
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Elastic Certificate Tool Autogenerated CA
        Validity
            Not Before: Jul  5 02:37:40 2018 GMT
            Not After : Jul  4 02:37:40 2021 GMT
        Subject: CN=custom
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    // omitted for brevity ...
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                // omitted for brevity ...
            X509v3 Authority Key Identifier:
                // omitted for brevity ...

            X509v3 Subject Alternative Name:
                DNS:custom
            X509v3 Basic Constraints:
                CA:FALSE
    Signature Algorithm: sha256WithRSAEncryption
         // omitted for brevity ...

Without this, Application Gateway will return 502 Bad Gateway errors, as the health probe for the backend pool will fail when the whitelisted certificate does not contain this certificate extension. You can typically understand if there is a problem with the key format when

1. TLS has been configured on the HTTP layer
2. Kibana is able to communicate to the cluster correctly but Application Gateway returns 502 errors.

This may not always be the case, but can be indicative. You should also check the description for Backend Health of the Application Gateway in the Azure portal.

Passing certificate parameters

Parameters such as esHttpCertBlob and kibanaCertBlob must be provided in Base-64 encoded form. A Base-64 encoded value can be obtained using

1. base64 on Linux, or openssl on Linux and MacOS

   base64

   httpCert=$(base64 http-cert.p12) 

   openssl

   httpCert=$(openssl base64 -in http-cert.p12)

   and including the value assigned to $httpCert in the parameters.json file as the value for certificate parameter passed to the Azure CLI command

2. PowerShell on Windows

   $httpCert = [Convert]::ToBase64String([IO.File]::ReadAllBytes("c:\http-cert.p12"))

   and then pass this in the template parameters object passed to the Azure PowerShell command

   $clusterParameters = @{
       # Other parameters skipped for brevity
       "esHttpCertBlob"= $httpCert
   }

License

This project is MIT Licensed and was originally forked from the Elasticsearch Azure quick start arm template