Cohesity AppSpec

Introduction

Application developers can specify application requirements through the Cohesity Application Specification (from here on called Cohesity AppSpec) which involves two things:

  • Define App structure Specified using a YAML format, it closely resembles the Kubernetes YAML specification, but with extensions to meet Cohesity requirements.
  • Define App metadata and privileges Specified using a JSON format, this includes App's metadata such as (Application Name, Description, Version etc.) used for packaging as well as privileges that the App needs.

Defining App Structure

The Cohesity AppSpec YAML provides specifications of all Kubernetes objects that need to be deployed for the application. Each object specification gets its own block in the YAML.

The high level format for each object is exactly the same as Kubernetes.

The main sections are:

apiVersion: ...

kind: ...

metadata:

......    
spec:

.....

Please refer to these links to Kubernetes documentation which describe these sections in detail:

apiVersion: https://kubernetes.io/docs/concepts/overview/kubernetes-api/#api-versioning

object "kind": https://kubernetes.io/docs/concepts/workloads/controllers/

metadata: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#objects

In the following sections, objects supported by Cohesity and extensions/restrictions enforced by Cohesity are described with examples.

Cohesity supported object kinds

The following Kubernetes object kinds are supported by Cohesity:

  • ReplicaSet - typically used for stateless pods.
  • StatefulSet - for pods which need to maintain state and rely on inherent bring up ordering.
  • Job - for "run to completion" cleanup pod.
  • Service - to load balance requests across the various pods behind StatefulSets and ReplicaSets.

For the most part, within each of these objects we allow the specification as Kubernetes. Especially the "container" specification has no changes.

The other restrictions and extensions are outlined in subsequent sections.

Cohesity Restrictions

  • Service type must be one of NodePort (or) ClusterIP
  • The clusterIP and nodePort itself cannot be selected by the app developer. Instead they are assigned by Cohesity infrastructure.
  • Job can be used only for a cleanup pod.
  • Namespace should not specified by the app developer. Cohesity infrastructure will assign a unique namespace for each app launch.

Cohesity Extensions

In this section, the Cohesity extensions (along with the reasons) are discussed.

Replicas & QoS

In Kubernetes YAML, replicas are specified as a fixed number. The application developer needs to calculate the scale a priori.

In Appllication YAML, replicas can be specified either as fixed or share. The share provides a way for the Cohesity infrastructure to "scale out" the application instance in accordance with cluster resources.

By specifying fixed replicas, the object will get exactly the number of replicas requested. By specifying replica shares, the numbers in the field share across various objects is used by Cohesity infrastructure as a ratio to derive the replica count for the application. This is done to scale out the application based on the resources available in the cluster. The fields min and max are optional and can be used to constrain the number of replicas within this range.

Example 1:

kind: ReplicaSet
metadata:
  name: ....
  spec:
    replicas:
      fixed: 2

Example 2:

kind: ReplicaSet
metadata:
  name: .....
  spec:
    replicas:
      share: 1
      min: 1
      max: 32

Volumes and Application Data Management

Cohesity infrastructure provides a way for applications to make use of 4 different types of volumes.

  • Static Persistent Volumes: Used to share data across various Kubernetes object. This is persisted for the lifetime of the application instance. Application developer needs to specify this in the Cohesity AppSpec.

  • Dynamic Persistent Volumes: Used to store data local to the Kubernetes object. This is persisted for the lifetime of the application instance. Application developer needs to specify this in the Cohesity AppSpec.

  • Ephemeral Volumes: Used to store temporary data. This data is lost upon container restart. This is given to the application by default (without the need for application developer specification) and can be found in the directory "/cohesity/ephemeral" within the pod.

  • Application mounted Volumes: By default, all applications get a directory "/cohesity/mount" inside the pod (without the need for any application developer specification). This application can mount Cohesity views using the "Mount SDK" which appear as folders in this "cohesity/mount" directory. This feature is typically used to provide input data to an application.

From the above discussion, in the Cohesity AppSpec, only "static" and "dynamic" persistent volumes need to be specified by an application developer. The syntax for these is provided in an example here:

   volumes:
   - name: my-static-volume
     fsType: nfs
     volumeType: static
     volumeName: /shared-data-volume

   - name: my-dynamic-volume
     fsType: nfs
     volumeType: dynamic

Note that static, dynamic and ephemeral volumes are provided by creating a Cohesity internal view for each application instance (Cohesity infrastructure does this). For static volume, the volumeName (in this case "shared-data-volume") will be used when creating the shared folder common to all Kubernetes objects.

Tagging NodePorts services

NodePorts UI service

The NodePorts service in front of the object implementing the UI for the application needs to be called out to provide an "access token" in the URL (along with the nodePort) to provide some form of authentication. For this the keyword cohesityTag can be used as shown in the example below.

apiVersion: v1
kind: Service
metadata:
...
spec:
  type: NodePort
  ...
  ports:
    - port: 9200
      protocol: TCP
      cohesityTag: ui
Tagging other NodePorts services as environment variables

There are cases where the pods would need to know the nodePorts of the various services through environment variables to reach then from inside the application directly. For these, each NodePort can be tagged using the keyword "cohesityEnv" as shown in the example below.

apiVersion: v1
kind: Service
metadata:
...
spec:
  type: NodePort
  ...
  ports:
    - port: 9200
      protocol: TCP
      cohesityEnv: MY_ENV_VAR

Tagging "cleanup" Job:

To support cases where the application needs to do any cleanup (e.g. of its configuration etc.) before it is terminated entirely, a special run to completion "Job" can be tagged with "cohesityTag" of cleanup as shown in the example below. Note that there can only be one such "cleanup" job for the application.

apiVersion: apps/v1
kind: Job
metadata:
  name: ...
  labels:
      app: ...

  cohesityTag: cleanup

Defining App Metadata

Application developers must specify the Cohesity App's metadata information in JSON format through a file. This file (named app.json) should be included in the application packaging.

Format

The following information can be provided by the application developer:

  • Application Name.
  • Application Developer Version typically in the format major_version.minor_version.
  • Application Description - a short description about the application.
  • Privileges that the application might need specified by enumerating the access requirement with a true/false value associated with it (true indicates access is required, false indicates access is not required with default being false). When the application is run the user approves the access to the various resources app needs as specified in the app yaml. The following access requirements can be specified:
    • Read Access to views (name: read_access).
    • Read/Write Access to views (name: read_write_access).
    • Management access (name: management_access).
    • Auto mounting of specified views when application is run (name: auto_mount_access).
    • Unrestricted access to Application UI (name: unrestricted_app_ui_access). By default, the access to application UI is safeguarded by a token generated by Cohesity. This feature should be used only by applications that have their own RBAC and do not want cohesity app infrastructure to control access to the application UI.
  • Access Role as admin only (name: access_role, value: admin). By default any user can with APP_LAUNCH access role can run the application. However if an application should only be run by an admin user, this field can be used to specify that access role.

Example

An example of Cohesity AppSpec (based on Insight application) which illustrates all the features described in this document is provided here:

App Structure

apiVersion: v1
kind: Service
metadata:
  name: insight-crawler-master-rest
  labels:
    app: insight-crawler-master
spec:
  type: NodePort
  selector:
    app: insight-crawler-master
  ports:
  - port: 8080
    protocol: TCP
    name: rest
---
# Crawler master gRPC service.
apiVersion: v1
kind: Service
metadata:
  name: insight-crawler-master-grpc
  labels:
    app: insight-crawler-master
spec:
  type: ClusterIP
  selector:
    app: insight-crawler-master
  ports:
  - port: 9090
    protocol: TCP
    name: grpc
    cohesityEnv: CRAWLERMASTER
---
# Crawler master pod spec.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: insight-crawler-master
  labels:
    app: insight-crawler-master
spec:
  replicas:
    fixed: 1
  selector:
    matchLabels:
      app: insight-crawler-master
  template:
    metadata:
      labels:
        app: insight-crawler-master
    spec:
      containers:
      - name: insight-crawler-master
        image: insight_crawler_master:latest
        resources:
          requests:
            cpu: 1000m
            memory: 256Mi
        volumeMounts:
        - name: appdata
          mountPath: /static-persistent-vol
        env:
        - name: ES_REST_SERVICE_ENDPT
          value: "insight-es-rest:9200"
      volumes:
        - name: appdata
          fsType: nfs
          volumeType: static
          volumeName: /static-persistent-vol
---
# Crawler minion pod spec.
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: insight-crawler-minion
spec:
  replicas:
    share: 1
    min: 1
    max: 32
  selector:
    matchLabels:
      app: insight-crawler-minion
  template:
    metadata:
      labels:
        app: insight-crawler-minion
    spec:
      containers:
      - name: insight-crawler-minion
        image: insight_crawler_minion:latest
        resources:
          requests:
            cpu: 500m
            memory: 512Mi
        env:
        - name: MASTER_GRPC_SERVICE_IP_PORT
          value: "insight-crawler-master-grpc:9090"
        - name: ES_REST_SERVICE_ENDPT
          value: "insight-es-rest:9200"
---
# Elasticsearch REST service.
apiVersion: v1
kind: Service
metadata:
  name: insight-es-rest
  labels:
    app: insight-es
spec:
  type: ClusterIP
  selector:
    app: insight-es
  ports:
  - port: 9200
    protocol: TCP
    name: rest
---
# Elasticsearch clustering service (internal).
apiVersion: v1
kind: Service
metadata:
  name: insight-es-cluster
  labels:
    app: insight-es
spec:
  type: ClusterIP
  clusterIP: None
  selector:
    app: insight-es
  ports:
  - port: 9300
    protocol: TCP
    name: cluster
---
# Elasticsearch pod spec.
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: insight-es
  labels:
    app: insight-es
spec:
  serviceName: "insight-es-cluster" # Headless service.
  replicas:
    share: 1
    min: 1
    max: 32
  selector:
    matchLabels:
      app: insight-es
  template:
    metadata:
      labels:
        app: insight-es
    spec:
      containers:
      - name: insight-es
        image: insight_es:latest
        resources:
          requests:
            cpu: 1000m
            memory: 4Gi
        volumeMounts:
        - name: es-data
          mountPath: /usr/share/elasticsearch/data
        env:
        - name: ES_JAVA_OPTS
          value: "-Xmx2048m -Xms2048m"
        - name: COHESITY_DYNAMIC_PERSISTENT_VOL_PATH
          value: "/usr/share/elasticsearch/data"
        - name: COHESITY_EPHEMERAL_VOL_PATH
          value: "/cohesity/ephemeral"
      volumes:
        - name: es-data
          fsType: nfs
          volumeType: dynamic
---
# UI NodePort service.
apiVersion: v1
kind: Service
metadata:
  name: insight-ui
  labels:
    app: insight-ui
spec:
  type: NodePort
  selector:
    app: insight-ui
  ports:
  - port: 9500
    protocol: TCP
    name: http
    cohesityTag: ui
---
# UI Pod Spec.
apiVersion: apps/v1
kind: ReplicaSet
metadata:
  name: insight-ui
  labels:
    app: insight-ui
spec:
  replicas:
    fixed: 1
  selector:
    matchLabels:
      app: insight-ui
  template:
    metadata:
      labels:
        app: insight-ui
    spec:
      containers:
      - name: insight-ui
        image: insight_ui:latest
        resources:
          requests:
            cpu: 250m
---
# Cleanup pod.
apiVersion: batch/v1
kind: Job
metadata:
  name: insight-cleanup
  labels:
    app: insight-cleanup
  cohesityTag: cleanup
spec:
  template:
    metadata:
      labels:
        app: insight-cleanup
    spec:
      containers:
      - name: insight-cleanup
        image: insight_crawler_master:latest
        volumeMounts:
        - name: appdata
          mountPath: /static-persistent-vol
        env:
        - name: CLEANUP_MODE
          value: "true"
      restartPolicy: Never
      volumes:
        - name: appdata
          fsType: nfs
          volumeType: static
          volumeName: /static-persistent-vol

App Metadata

{
 "name" : "Cohesity Insight",
 "dev_version": "1.0",
 "description" : "Insight: Interactive Content Search Application",
 "access_requirements" : {
    "read_access" : true,
    "read_write_access" : false,
    "management_access" : true,
    "auto_mount_access" : false,
    "unrestricted_app_ui_access" : false,
 },
 "access_role": "admin"
}