Resources and metrics in Gnocchi

on waitingforcode.com

Resources and metrics in Gnocchi

Data processing in Gnocchi is strongly related to the index information. One of such valuable assets are metrics and resources, covered just below.

The post is divided in 2 parts. The first one talks about the metrics. And as you can correctly deduce, the next one covers the resources. Both are composed of a short explanation followed by the test cases using Gnocchi's API.

Metric

A metric is the entity for which the measures are registered. In the list of metric examples we can find: CPU use, memory use (%, MB...), number of visitors at given moment on our website and so forth. Globally, a metric represents everything we want to measure and it's characterized by:

  • id - uniquely identifies the metric.
  • archive policy - defines the data processing strategy. You can learn more about it in the post Archive policy in Gnocchi.
  • name - helps to identify the metric.
  • resource - represents the resource to which is associated given metric. It's an optional property but can be very helpful for the documentation purposes.
  • unit - represents the unit of the metric values, e.g. %, MB/s, ns, MB, number of running instances at given moment etc. The property has a purely informational purpose.

From data aggregation point of view, metric is a very important point. It provides, through the archive policy, the definition of what to do with the incoming measures. The archive policy of each metric is static, i.e. once created, it can't be replaced. Moreover, the id of the metric is used to compute the number of sack where the measures will be persisted.

Creating a metric is quite easy:

curl -d '{
  "archive_policy_name": "high",
  "unit": "%",
  "name": "memory use"
}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/metric

# Expected response is:
# {"unit": "%", "created_by_user_id": "admin", "resource_id": null, "creator": "admin", "name": "memory use", "id": "3df2dc5a-8672-46a0-8cba-804fbf340322", "archive_policy_name": "high", "created_by_project_id": ""}

With such created metric we can also easily add new measures:

curl -d '[
  {
    "timestamp": "2018-04-26T16:05:00",
    "value": 43.1
  },
  {
    "timestamp": "2018-04-26T16:06:00",
    "value": 50.5
  },
  {
    "timestamp": "2018-04-26T16:07:00",
    "value": 39.6
  }
]' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/metric/3df2dc5a-8672-46a0-8cba-804fbf340322/measures

Resource

A resource is the entity grouping different metrics. It describes then some measured entity (metric is more about the measured properties), such as for example: a website, a node in the cluster or even a user if we'd like to store his activity in time series format. A resource is characterized by:

  • id - as for metric, it uniquely identifies the resource
  • type - it's the metadata of the resource. The type is an entity composed of a bunch of attributes that are automatically associated to given resource. The type properties can either be required or optional.
  • revision start and end dates - define when given resource version began and ended. If the end date is empty it means that the resource version is still in use. This field changes when, for instance, one of resource types fields is modified. The revision is also exposed in Last-Modified HTTP header.
  • start and end dates - they describe the lifespan of the resource. By default the started_at is set to the current timestamp. The ended_at one can be defined to any available date after the started_at. They're mainly used in the resources filter called lifespan expressed in time units (e.g. 30 seconds). One example of such use is included in the tests below.

The resource can be associated with one or more metrics. Thus when its removal automatically triggers the removal of the related metrics.

To see what happens with resources, let's start by creating a resource type with some required attributes:

curl -d '{
  "name": "department_type",
  "attributes": {
    "description": {
      "required": true, "type": "string"
    }, 
    "priority": {
      "required": true, "type": "number"
    },
    "department": {
      "type": "string",
      "required": false
    }
  }
}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/resource_type

# Gnocchi should return:
# {"name": "department_type", "attributes": {"description": {"type": "string", "min_length": 0, "required": true, "max_length": 255}, "department": {"type": "string", "min_length": 0, "required": false, "max_length": 255}, "priority": {"type": "number", "required": true, "min": null, "max": null}}, "state": "active"}

Now we can add new resource with and without the required types defined:

# This doesn't contain the required properties and returns an error:
# Invalid input: required key not provided @ data['description']
curl -d '{}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/resource/department_type

# This time it's good
curl -d '{
  "id": "resource1_id",
  "description": "resource for any department",
  "priority": 3
}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/resource/department_type
# Expected response:
# {"project_id": null, "created_by_user_id": "admin", "ended_at": null, "creator": "admin", "metrics": {}, "department": null, "original_resource_id": "resource1_id", "id": "180715d9-4a1a-5c44-a802-28ec2b9fea6f", "revision_end": null, "started_at": "2018-04-26T16:50:10.528861+00:00", "type": "department_type", "revision_start": "2018-04-26T16:50:10.528886+00:00", "description": "resource for any department", "user_id": null, "priority": 3, "created_by_project_id": ""}

After the successful creation of a resource we can play with revisions and start/end dates attributes:

# First, let's change the priority attribute of the resource
curl -d '{
  "priority": 6
}' -H "Content-Type: application/json"  -X PATCH http://admin:admin@localhost:8041/v1/resource/department_type/180715d9-4a1a-5c44-a802-28ec2b9fea6f 

# Expected response is
{"project_id": null, "created_by_user_id": "admin", "ended_at": null, "creator": "admin", "metrics": {}, "department": null, "original_resource_id": "resource1_id", "id": "180715d9-4a1a-5c44-a802-28ec2b9fea6f", "revision_end": null, "started_at": "2018-04-26T16:50:10.528861+00:00", "type": "department_type", "revision_start": "2018-04-26T16:55:42.207431+00:00", "description": "resource for any department", "user_id": null, "priority": 6, "created_by_project_id": ""}

As you can see, the revision attribute changed but started_at remained the same. Let's highlight the difference between these fields in the endpoint exposing working history:

curl -H "Content-Type: application/json"  -X GET http://admin:admin@localhost:8041/v1/resource/department_type/180715d9-4a1a-5c44-a802-28ec2b9fea6f/history
curl -d '{ "ended_at": "2018-04-26T17:20:20.528861+00:00" }' -H "Content-Type: application/json" -X PATCH http://admin:admin@localhost:8041/v1/resource/department_type/180715d9-4a1a-5c44-a802-28ec2b9fea6f

The difference between start/end and revision dates is more visible in the history endpoint:

[
  {
    "project_id":null,
    "created_by_user_id":"admin",
    "ended_at":null,
    "creator":"admin",
    "metrics":{},
    "department":null,
    "original_resource_id":"resource1_id",
    "id":"180715d9-4a1a-5c44-a802-28ec2b9fea6f",
    "revision_end":"2018-04-26T16:55:42.207431+00:00",
    "started_at":"2018-04-26T16:50:10.528861+00:00",
    "type":"department_type",
    "revision_start":"2018-04-26T16:50:10.528886+00:00",
    "description":"resource for any department",
    "user_id":null,
    "priority":3.0,
    "created_by_project_id":""
  },
  {
    "project_id":null,
    "created_by_user_id":"admin",
    "ended_at":null,
    "creator":"admin",
    "metrics":{},
    "department":null,
    "original_resource_id":"resource1_id",
    "id":"180715d9-4a1a-5c44-a802-28ec2b9fea6f",
    "revision_end":null,
    "started_at":"2018-04-26T16:50:10.528861+00:00",
    "type":"department_type",
    "revision_start":"2018-04-26T16:55:42.207431+00:00",
    "description":"resource for any department",
    "user_id":null,
    "priority":6.0,
    "created_by_project_id":""
  }
]

You can clearly notice the revision_end difference and equal started_at/ended_at fields. Let's see now how the lifespan works. To do that, we create another resource, this time with started_at and ended_at attributes defined:

curl -d '{
  "id": "resource2_id",
  "description": "resource for any department",
  "priority": 3,
  "started_at": "2018-04-26T16:00:00.207431+00:00",
  "ended_at": "2018-04-26T17:00:00.207431+00:00"
}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/resource/department_type

# expected:
# {"project_id": null, "created_by_user_id": "admin", "ended_at": "2018-04-26T17:00:00.207431+00:00", "creator": "admin", "metrics": {}, "department": null, "original_resource_id": "resource2_id", "id": "e8c3708b-94ec-522e-abc9-2b83bf66cfd1", "revision_end": null, "started_at": "2018-04-26T16:00:00.207431+00:00", "type": "department_type", "revision_start": "2018-04-26T17:32:23.926509+00:00", "description": "resource for any department", "user_id": null, "priority": 3, "created_by_project_id": ""}

Now we can filter the resources with the lifespan parameter:

# First, the query finding our resource
curl -d '{">": {"lifespan": "30 min"}}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/search/resource/department_type

# And its response
[{"project_id": null, "created_by_user_id": "admin", "ended_at": "2018-04-26T17:00:00.207431+00:00", "creator": "admin", "department": null, "original_resource_id": "resource2_id", "id": "e8c3708b-94ec-522e-abc9-2b83bf66cfd1", "created_by_project_id": "", "revision_end": null, "started_at": "2018-04-26T16:00:00.207431+00:00", "revision_start": "2018-04-26T17:32:23.926509+00:00", "type": "department_type", "description": "resource for any department", "user_id": null, "priority": 3.0, "metrics": {}}]

# Now, the query not matching it
curl -d '{">": {"lifespan": "120 min"}}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/search/resource/department_type

# And an empty array as its response
[]

The metrics and the resources are the main descriptive entities in Gnocchi. The former ones define what happens with the measures, especially through the archive policy explained in one of previous posts. The resource is an entity helping to group different metrics related to one physical object (e.g. node, user) in one place. Both have then an important role in the project documentation.

Read also about Resources and metrics in Gnocchi here: Can't get the unit of a metric , REST API - create metric .

Share, like or comment this post on Twitter: