Resources and metrics in Gnocchi

Versions: Gnocchi 4.2

Data processing in Gnocchi is strongly related to the index information. One of such valuable assets are metrics and resources, covered just below.

The post is divided in 2 parts. The first one talks about the metrics. And as you can correctly deduce, the next one covers the resources. Both are composed of a short explanation followed by the test cases using Gnocchi's API.

Metric

A metric is the entity for which the measures are registered. In the list of metric examples we can find: CPU use, memory use (%, MB...), number of visitors at given moment on our website and so forth. Globally, a metric represents everything we want to measure and it's characterized by:

From data aggregation point of view, metric is a very important point. It provides, through the archive policy, the definition of what to do with the incoming measures. The archive policy of each metric is static, i.e. once created, it can't be replaced. Moreover, the id of the metric is used to compute the number of sack where the measures will be persisted.

Creating a metric is quite easy:

curl -d '{
  "archive_policy_name": "high",
  "unit": "%",
  "name": "memory use"
}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/metric

# Expected response is:
# {"unit": "%", "created_by_user_id": "admin", "resource_id": null, "creator": "admin", "name": "memory use", "id": "3df2dc5a-8672-46a0-8cba-804fbf340322", "archive_policy_name": "high", "created_by_project_id": ""}

With such created metric we can also easily add new measures:

curl -d '[
  {
    "timestamp": "2018-04-26T16:05:00",
    "value": 43.1
  },
  {
    "timestamp": "2018-04-26T16:06:00",
    "value": 50.5
  },
  {
    "timestamp": "2018-04-26T16:07:00",
    "value": 39.6
  }
]' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/metric/3df2dc5a-8672-46a0-8cba-804fbf340322/measures

Resource

A resource is the entity grouping different metrics. It describes then some measured entity (metric is more about the measured properties), such as for example: a website, a node in the cluster or even a user if we'd like to store his activity in time series format. A resource is characterized by:

The resource can be associated with one or more metrics. Thus when its removal automatically triggers the removal of the related metrics.

To see what happens with resources, let's start by creating a resource type with some required attributes:

curl -d '{
  "name": "department_type",
  "attributes": {
    "description": {
      "required": true, "type": "string"
    }, 
    "priority": {
      "required": true, "type": "number"
    },
    "department": {
      "type": "string",
      "required": false
    }
  }
}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/resource_type

# Gnocchi should return:
# {"name": "department_type", "attributes": {"description": {"type": "string", "min_length": 0, "required": true, "max_length": 255}, "department": {"type": "string", "min_length": 0, "required": false, "max_length": 255}, "priority": {"type": "number", "required": true, "min": null, "max": null}}, "state": "active"}

Now we can add new resource with and without the required types defined:

# This doesn't contain the required properties and returns an error:
# Invalid input: required key not provided @ data['description']
curl -d '{}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/resource/department_type

# This time it's good
curl -d '{
  "id": "resource1_id",
  "description": "resource for any department",
  "priority": 3
}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/resource/department_type
# Expected response:
# {"project_id": null, "created_by_user_id": "admin", "ended_at": null, "creator": "admin", "metrics": {}, "department": null, "original_resource_id": "resource1_id", "id": "180715d9-4a1a-5c44-a802-28ec2b9fea6f", "revision_end": null, "started_at": "2018-04-26T16:50:10.528861+00:00", "type": "department_type", "revision_start": "2018-04-26T16:50:10.528886+00:00", "description": "resource for any department", "user_id": null, "priority": 3, "created_by_project_id": ""}

After the successful creation of a resource we can play with revisions and start/end dates attributes:

# First, let's change the priority attribute of the resource
curl -d '{
  "priority": 6
}' -H "Content-Type: application/json"  -X PATCH http://admin:admin@localhost:8041/v1/resource/department_type/180715d9-4a1a-5c44-a802-28ec2b9fea6f 

# Expected response is
{"project_id": null, "created_by_user_id": "admin", "ended_at": null, "creator": "admin", "metrics": {}, "department": null, "original_resource_id": "resource1_id", "id": "180715d9-4a1a-5c44-a802-28ec2b9fea6f", "revision_end": null, "started_at": "2018-04-26T16:50:10.528861+00:00", "type": "department_type", "revision_start": "2018-04-26T16:55:42.207431+00:00", "description": "resource for any department", "user_id": null, "priority": 6, "created_by_project_id": ""}

As you can see, the revision attribute changed but started_at remained the same. Let's highlight the difference between these fields in the endpoint exposing working history:

curl -H "Content-Type: application/json"  -X GET http://admin:admin@localhost:8041/v1/resource/department_type/180715d9-4a1a-5c44-a802-28ec2b9fea6f/history
curl -d '{ "ended_at": "2018-04-26T17:20:20.528861+00:00" }' -H "Content-Type: application/json" -X PATCH http://admin:admin@localhost:8041/v1/resource/department_type/180715d9-4a1a-5c44-a802-28ec2b9fea6f

The difference between start/end and revision dates is more visible in the history endpoint:

[
  {
    "project_id":null,
    "created_by_user_id":"admin",
    "ended_at":null,
    "creator":"admin",
    "metrics":{},
    "department":null,
    "original_resource_id":"resource1_id",
    "id":"180715d9-4a1a-5c44-a802-28ec2b9fea6f",
    "revision_end":"2018-04-26T16:55:42.207431+00:00",
    "started_at":"2018-04-26T16:50:10.528861+00:00",
    "type":"department_type",
    "revision_start":"2018-04-26T16:50:10.528886+00:00",
    "description":"resource for any department",
    "user_id":null,
    "priority":3.0,
    "created_by_project_id":""
  },
  {
    "project_id":null,
    "created_by_user_id":"admin",
    "ended_at":null,
    "creator":"admin",
    "metrics":{},
    "department":null,
    "original_resource_id":"resource1_id",
    "id":"180715d9-4a1a-5c44-a802-28ec2b9fea6f",
    "revision_end":null,
    "started_at":"2018-04-26T16:50:10.528861+00:00",
    "type":"department_type",
    "revision_start":"2018-04-26T16:55:42.207431+00:00",
    "description":"resource for any department",
    "user_id":null,
    "priority":6.0,
    "created_by_project_id":""
  }
]

You can clearly notice the revision_end difference and equal started_at/ended_at fields. Let's see now how the lifespan works. To do that, we create another resource, this time with started_at and ended_at attributes defined:

curl -d '{
  "id": "resource2_id",
  "description": "resource for any department",
  "priority": 3,
  "started_at": "2018-04-26T16:00:00.207431+00:00",
  "ended_at": "2018-04-26T17:00:00.207431+00:00"
}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/resource/department_type

# expected:
# {"project_id": null, "created_by_user_id": "admin", "ended_at": "2018-04-26T17:00:00.207431+00:00", "creator": "admin", "metrics": {}, "department": null, "original_resource_id": "resource2_id", "id": "e8c3708b-94ec-522e-abc9-2b83bf66cfd1", "revision_end": null, "started_at": "2018-04-26T16:00:00.207431+00:00", "type": "department_type", "revision_start": "2018-04-26T17:32:23.926509+00:00", "description": "resource for any department", "user_id": null, "priority": 3, "created_by_project_id": ""}

Now we can filter the resources with the lifespan parameter:

# First, the query finding our resource
curl -d '{">": {"lifespan": "30 min"}}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/search/resource/department_type

# And its response
[{"project_id": null, "created_by_user_id": "admin", "ended_at": "2018-04-26T17:00:00.207431+00:00", "creator": "admin", "department": null, "original_resource_id": "resource2_id", "id": "e8c3708b-94ec-522e-abc9-2b83bf66cfd1", "created_by_project_id": "", "revision_end": null, "started_at": "2018-04-26T16:00:00.207431+00:00", "revision_start": "2018-04-26T17:32:23.926509+00:00", "type": "department_type", "description": "resource for any department", "user_id": null, "priority": 3.0, "metrics": {}}]

# Now, the query not matching it
curl -d '{">": {"lifespan": "120 min"}}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/search/resource/department_type

# And an empty array as its response
[]

The metrics and the resources are the main descriptive entities in Gnocchi. The former ones define what happens with the measures, especially through the archive policy explained in one of previous posts. The resource is an entity helping to group different metrics related to one physical object (e.g. node, user) in one place. Both have then an important role in the project documentation.

TAGS: #Gnocchi