* Broadcast tracker script config updates
* Fix broadcast_put typespec
* Preload site association on CE
* Stop preloading uselessly
* Reload tracker_script_config on CE only
* exclude consolidated views when counting/listing sites
* Use Teams.owned_sites/1 now that it excludes consolidated views
* Until we figure out better - make sites.consolidated virtual for CE
* Safeguard ingestion, update tests for gate keeper/site cache
* Ensure consolidated sites are excluded from helpscout counts
* Test that consolidated views are excluded from site listings
* Update tests for owned sites to ensure consolidated views are excluded
* Update CRM tests to exclude consolidated sites
* exclude consolidated sites from quota usage
* filter out conslolidated sites from deleting events background job
* exclude consolidated (change domain background job)
* retrieve non-consolidated sites only by default
* Ensure Site struct defines a default for `consolidated`
* Make `sites.consolidated` column available in CE
* Add missing test to reminder e-mails, consolidation is irrelevant here
* Do not send setup help e-mail for consolidated sites
* Ensure no setup success email is ever sent for consolidated sites
* Skip consolidated sites in legacy time on page cut off
* Add minor test to Sites.get_for_user/2
* Extend get_for_user/get_for_user! options signature
* Move test dir up
* Write tests for Sites.get_by_domain
* Minor DRY
* fix typos
* Provide Site.regular() and ConsolidatedView.sites() for querying
* Use Site.regular() in Repo.get_by() calls
* Fix typespecs
* Update missing Site.regular() filters
---------
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Allow caching tracker script on CE
Open questions with this approach:
- `ingestion_url`: Using `PlausibleWeb.Endpoint.url()` requires that endpoint has started, but we
want to pre-warm the cache _before_ the endpoint starts. To work around this, a different approach
is used to get the right url.
- caching: Other caches currently cache database models, this caches a string. Will this cause issues?
* Slightly better workaround
* Lazier timers
* Revert "Temporarily disable ingest metrics (#5369)"
This reverts commit b96e96a7f6.
* Add :tools to MIX_ENV=dev
* Stop tracking caches hit ratio in favour of raw counters
* fractional_sample_rate: Vary sample rate according to traffic in the past 30 days
Our old sampling mechanism used SAMPLE 20000000 syntax. This was
wonderful since it allowed essentially dynamic sampling based on the
data being queried. However this ran into many issues relating to JOINs
and sample rate being different for different tables.
Instead, we now start to dynamically vary sample rates fractionally.
At query time we check the time window being queried and estimate how
many rows this query might reach. For large queries, we then dynamically
decide the sample rate.
For getting the traffic estimate for a site, we have a new SampingCache class which queries `ingest_counters`.
The query being cached is slightly expensive and can be sped up with a
ClickHouse projection.
* ce behavior
* Fix test
* Update test/plausible/stats/sampling_cache_test.exs
Co-authored-by: hq1 <hq@mtod.org>
* Update tests
* Update tests
---------
Co-authored-by: hq1 <hq@mtod.org>
* Rely on con_cache telemetry
Now that https://github.com/sasa1977/con_cache/pull/76
is released, we don't have to use low-level operations
to emit hit/miss events.
This PR also wraps cache processes with
a function returning appropriate child specs lists.
Ideally each cache will have its own supervisor/child specs
going forward. This is an intermediate step in that direction.
* Update lib/plausible/application.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Declare caches without warmers with plain child specs
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Reapply "Replace caching engine (#3878)" (#3883)
This reverts commit c5881cdc6d.
* Ensure hit rate is tracked on `get_or_store`
* Remove :wx and :observer
* Remove unused deps
* Use `:set` table type
* Dependencies: swap Cachex for ConCache
* Implement Cache adapter wrapping ConCache
* Implement cache stats tracker, for metrics
* Use Cache.Adapter in Plausible.Cache
Marking the test as not slow anymore
* Use Cache Adapter when tracking sessions
* Use Cache Adapter for UA parsing
* Rename child identifiers - cachex is obsolete now
* Test stats tracking
* Update grafana metrics
* Put all caches under common child specification
* Try less
* Shorten the function delegation path
* Add Ecto.Network dependency
* Migration: Add ip block list table
* If Cachex errors out, mark the cache as not ready
* Add IPRule schema
* Seed IPRules
* Add Shields context module
* Implement IPRuleCache
* Start IPRuleCache
* Drop blocklisted IPs on ingestion
* Cosmetic rename
* Add settings sidebar item
* Consider IPRuleCache readiness on health checks
* Fix typo
* Implement IP blocklist live view
* Update moduledocs
* Extend contextual module tests
* Convert IPRules LiveView into LiveComponent
* Keep live flashes on the tabs view
* Update changelog
* Format
* Credo
* Remove garbage
* Update drop reason typespecs
* Update typespecs for cache keys
* Keep track of who added a rule and when
* Test if adding via LV prefills the updated_by tooltip
* Update ecto_network dependency
* s/updated_by/added_by
* s/drop_blocklist_ip/drop_shield_rule_ip
* Add docs link
* s/Updated/Added
* Simplify caches config
The intervals were proven to never change really,
and previous shape made them not very discoverable either.
We'll look into a more declarative setup.
* Aim for more declarative supervision tree setup
* Generalize `Plausible.Cache`
This makes the `Site.Cache` module implement the specifics,
leaving the common bits reusable for upcoming cache processes.
* Generalize `Plausible.Cache` warmers
* Fix typos
Artur Pata
Adam Rutkowski
Karl-Aksel Puulmann