Archive policy in Gnocchi

In the recent posts about Gnocchi we could often meet the concept of archive policy. However, as one of the main points in this system, it merits its own explanation.

New ebook 🔥

Learn 84 ways to solve common data engineering problems with cloud services.

👉 I want my Early Access edition

As you can correctly deduce, this post will be about archive policy in Gnocchi. Its first section presents all theoretical aspects of it. The second one shows some internal details through learning tests.

Archive policy defined

In the previous posts we could learn a little about archive policy. It was presented as a specification for the computation of the aggregations. It defines what kind of aggregation is computed, for which granularity and for how many points. More concretely we can describe an archive policy by the following properties:

Gnocchi comes with 4 default policies: low, medium, high and bool. They go from the highest granularity (1 second) to the lowest (5 minuts). Custom policies can be created though with /archive_policy endpoint used in POST HTTP method with an appropriate JSON payload describing the added policy.

The archive policy is stored in index storage and is associated to one or more metrics. Naturally it's involved in the data processing step generating the aggregated time series (see the post about Carbonara storage format). Thus, the archive policy provides the important context defining what data and how will be generated.

It's possible to modify already existent archive policy. However only the changes on the definitions part are possible. And not completely because we can't add new definitions and modify the granularity of already existent. When we try to change the aggregation methods or back window parameters, we'll receive an error message talking about invalid input: "Invalid input: extra keys not allowed @ data['aggregation_methods']" or "Invalid input: extra keys not allowed @ data['back_window']". Moreover, the changes don't apply on already existent data.

Archive policy internals

Let's see now some of the internal details of the archive policy. It's represented by gnocchi.archive_policy.ArchivePolicy class exposing all of 4 properties described in the previous section. The definitions in their turn are represented as the instances of gnocchi.archive_policy.ArchivePolicyItem and similarly, they provide the public access for the points presented previously:

def should_create_default_policy_with_different_definitions(self):
  granularity_30_sec = numpy.timedelta64(30, 's')
  granularity_1_sec = numpy.timedelta64(1, 's')
  timespan_1_day = numpy.timedelta64(1, 'D')
  timespan_31_days = numpy.timedelta64(31, 'D')
  long_term_definition = archive_policy.ArchivePolicyItem(granularity_30_sec,
                                                          timespan=timespan_31_days)
  short_term_definition = archive_policy.ArchivePolicyItem(granularity_1_sec,
                                                            timespan=timespan_1_day)
  two_term_policy = archive_policy.ArchivePolicy('2-term policy', 0,
                                                  [short_term_definition, long_term_definition],
                                                  ["sum", "count"])

  self.assertEqual('2-term policy', two_term_policy.name)
  self.assertEqual(0, two_term_policy.back_window)
  self.assertEqual('30 seconds', str(two_term_policy.max_block_size))
  self.assertIn(aggregation.Aggregation("count", granularity_1_sec, numpy.timedelta64(86400, 's')),
                two_term_policy.aggregations)
  self.assertIn(aggregation.Aggregation("count", granularity_30_sec, numpy.timedelta64(2678400, 's')),
                two_term_policy.aggregations)
  self.assertIn(aggregation.Aggregation("sum", granularity_1_sec, numpy.timedelta64(86400, 's')),
                two_term_policy.aggregations)
  self.assertIn(aggregation.Aggregation("sum", granularity_30_sec, numpy.timedelta64(2678400, 's')),
                two_term_policy.aggregations)

Another interesting point to see in action is the rate-based aggregations that computes the rate based on point values rather than the rate between normal aggregation result:

# create an archive policy 
curl -d '{
  "aggregation_methods": ["sum", "rate:sum"],
  "back_window": 0,
  "definition": [
    {
      "granularity": "2s",
      "timespan": "7 day"
    }
  ],
  "name": "rate_rate_sum_policy_2s"
}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/archive_policy 

# Now the metric using it
curl -d '{
  "archive_policy_name": "rate_rate_sum_policy_2s", 
  "name": "rate_rate_sum_metric_2s"
}' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/metric
# note somewhere the id: 77ed6901-5db3-4334-a8d4-4d8156170cb9  

# Add some testing measures
curl -d '[
  {"timestamp": "2018-04-27T11:00:00", "value": 1.0}, {"timestamp": "2018-04-27T11:00:01", "value": 2.0},
  {"timestamp": "2018-04-27T11:00:02", "value": 3.0}, {"timestamp": "2018-04-27T11:00:03", "value": 4.0},
  {"timestamp": "2018-04-27T11:00:04", "value": 5.0}, {"timestamp": "2018-04-27T11:00:05", "value": 7.0},
  {"timestamp": "2018-04-27T11:00:06", "value": 8.0}, {"timestamp": "2018-04-27T11:00:07", "value": 11.0},
  {"timestamp": "2018-04-27T11:00:08", "value": 12.0}, {"timestamp": "2018-04-27T11:00:09", "value": 16.0}

]' -H "Content-Type: application/json"  -X POST http://admin:admin@localhost:8041/v1/metric/77ed6901-5db3-4334-a8d4-4d8156170cb9/measures

# and execute the rate sum and sum queries
curl http://admin:admin@localhost:8041/v1/metric/77ed6901-5db3-4334-a8d4-4d8156170cb9/measures?aggregation=rate:sum
curl http://admin:admin@localhost:8041/v1/metric/77ed6901-5db3-4334-a8d4-4d8156170cb9/measures?aggregation=sum

The sum query returns expected values:

[
  // 1 + 2
  ["2018-04-27T11:00:00+00:00", 2.0, 3.0], 
  // 3 + 4
  ["2018-04-27T11:00:02+00:00", 2.0, 7.0], 
  // 5 + 7
  ["2018-04-27T11:00:04+00:00", 2.0, 12.0], 
  // 8 + 11
  ["2018-04-27T11:00:06+00:00", 2.0, 19.0],
  // 12 + 16
  ["2018-04-27T11:00:08+00:00", 2.0, 28.0]
]

But rate:sum computes the differences between the last points:

[
  // 2 - 1, first window
  ["2018-04-27T11:00:00+00:00", 2.0, 1.0], 
  // 4 - 2
  ["2018-04-27T11:00:02+00:00", 2.0, 2.0], 
  // 7 - 2
  ["2018-04-27T11:00:04+00:00", 2.0, 3.0], 
  // 11 - 7
  ["2018-04-27T11:00:06+00:00", 2.0, 4.0], 
  // 16 - 11
  ["2018-04-27T11:00:08+00:00", 2.0, 5.0]
]

The archive policy is a core concept in Gnocchi. It's linked to the metric definition and thus automatically impacts the time series storage. It defines not only what aggregates are precomputed but also tells how many of them and how long are stored. The archive policy is characterized by 4 different properties: name, aggregation methods, back window and definitions. They were detailed in the first section of the post. The second section presented some tests showing how the archive policy's attributes can be involved in the data processing in Gnocchi.