Consider using VictoriaLogs as a database for web events
I don't know which database do you use for storing web events (let's suppose it is a ClickHouse). I'd recommend taking a look at VictoriaLogs - this is an open-source database originally designed for structured logs with a large number of fields (aka wide events). It uses architecture ideas similar to ClickHouse in order to achieve high compression rate for the stored data on disk and high query performance. You can read more about this at https://docs.victoriametrics.com/victorialogs/faq/#how-does-victorialogs-work .
The main advantages of VictoriaLogs over ClickHouse for web analytics are:
- Much easier setup and operation (VictoriaLogs is mostly zero-config, e.g. it works perfectly out of the box on any hardware).
- Ability to change the set of event fields at any time without modifying database schema (VictoriaLogs doesn't need configuring any database schema at all).
- Simplified data ingestion and querying (there is no need in constructing INSERT SQL statements for data ingestion - just push data as newline-delimited JSON stream; there is no need in using complex SQL for typical analytical queries - it is much easier to use LogsQL - see https://docs.victoriametrics.com/victorialogs/sql-to-logsql/ ).
If you use some other database for web events, then I suggest re-considering your choice, since ClickHouse and VictoriaLogs are much more cost-efficient on a large scale (trillions of events) than any other database.
A typical event for web analytics is a wide event with many fields such as webproperty_id, user_id, session_id, ip, hostname, path, page title, user-agent, utm_* values, etc. So it ideally fits VictoriaLogs data model (see https://docs.victoriametrics.com/victorialogs/keyconcepts/#data-model ). Data partition is is easy to do by {webproperty_id, hostname} fields (they are named log stream fields in VictoriaLogs). This will provide fast online analytics per each webproperty, even it contains billions of events.
Feel free joining our community channels - https://docs.victoriametrics.com/victoriametrics/#community-and-contributions . We are ready to help with any questions about VictoriaLogs.
BTW, we are also from Ukraine - https://victoriametrics.com/blog/tags/ukraine/
Aliaksandr Valialkin
3 weeks ago1
Aleksandr Blokh3 weeks ago