* Update report percentages on dashboard and details view
* Add percentages to Countries, Regions, and Cities reports
* Add percentages to Channels, Sources, and UTM reports
* Add percentages to top pages, entry pages, and exit pages reports
* Update tests to include percentages
* Change dashboard copy from title case to sentence case
* Update details modal style
* Make animations snappier
* Introduce max height to modal and make inner content scrollable
* Improve modal mobile design
- Enable horizontal scroll for details modal on mobile
- Add responsive spacing and positioning to modal
* Added mobile tap behavior to external link in list report
* Show tooltips only when in comparison mode or when the number is abbreviated
* remove previously added showTooltip prop
- This isn't needed anymore since we now handle the tooltip logic in the MetricValue component
* Show long format upon hovering detailed view metrics
* Added mobile tapping behaviour to detailed view
* Added percentages to all detailed views
* Add mobile swipe-to-close behavior for modal
* Adjust sensitivity of modal drag to close
* Use hammerjs for swipe-to-close modal behaviour
* Prevent dragging if gesture starts inside table
* Show 2 decimal places for percentages < 0.1% across dashboard
* Adjust dark mode styles
* Add hover effect to external link icon
* Update tests to expect two-decimal percentages
* Undo hammer install and revert to old modal styling
* Remove CR and % columns from goals and custom props reports on dashboard, and show on hover in detailed view
* Remove unused constants
* Undo conversion rate on hover behaviour
- Unlike percentages, CR should show permanently.
* Show percentages permanently in custom props detailed view
* Adjust width of conversion metrics column
* Updated metric-value test
* Update top-bar test
* Added changelog entry
* Fix test expectations for percentages with imported data
- Update tests to expect correct percentages (≤100%) when imported data is included. These tests will fail until the percentage calculation bug is fixed, documenting the expected behavior.
* Add imported_visitors to tests to ensure correct total_visitors calculation
* Correct imported_visitors count in test
* Migration: add custom propos to goals + revisit unique constraints
* Update constraints in goal schema (and move module)
* Add a comment, not really related but useful?
* Implement querying for goals with custom props
* Optimize goal_join_data (down to one iteration) + include goal custom props
* Test goal custom propos addition + new constraints
* Test querying for goals with custom propos attached
* Test funnels made of goals with custom props
* Format
* Fixup test name
* Fixup migration
* Unified goal join macro
* Remove dupe test
* Clean up user_id usage
* Fixup test to match the description
* Revert "Temporary: make room for pre/post migration constraint names (#5942)"
This reverts commit e4bc6b8715.
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Include revenue data for all detailed API responses except entry/exit pages
* Expose revenue data in all breakdown modals except entry/exit pages
* Add revenue metrics to breakdown response only on EE
* Change query builder to enable querying event metrics \w session dimension
* Add revenue metrics to entry and exit pages breakdowns
* Expose revenue data in entry and exit pages breakdowns
* Use `argMax` for `exit_page` and `exit_page_hostname` dimensions (h/t @ukutaht)
* Don't handle event-only dimensions with session-only metrics for now
* Add tests for all breakdowns
* Add clarifying comments in code
* Mark revenue tests as EE-only
* Create static consolidated view UI on `/sites` page
- Improve existing site card layout
- Add static UI for the consolidated view
- Add dismissable upgrade card UI
- Extract favicon fetching logic to function
- Configure configurable fallback icon per route
- Add `/favicon/sources_globe/` route with different icon than `/favicon/sources/` to use on `/sites` page
* Improve the mobile view of the `/sites` page
* Minor query interface UX extension (#5713)
* Minor query interface extension
* !fixup
* !fixup
* Initial implementation of consolidated views on /sites
* Improve loading state
* no need to handle nil in main interface
* Juggle `can_manage_consolidated_view?`
* Require team setup in order to enable consolidated view
* nil catcher
* Fixup test fixture
* Don't show Consolidated View tab in CS if team is not setup
* Reorganize + test
* Remove comment
* Only show consolidated views to superadmins for now
* Remove temporary sleep
* CE unused bindings
* Clean up seeds
* EE
* Fixup test
* Test non-superadmin scenario
* Add a test guarding parity between small plots (consolidated vs individual)
* Move private function so CE won't complain
* See if the graphs are now similar at least :)
* sort
* Map keys are unsorted
* Ensure engagement events aren't counted as visitors on smol graphs
* just try and revert
* Revert "just try and revert"
This reverts commit 7584f59816.
* Simplify globe icon handling
* Remove unnecessary @rest
* Split tests into more focused cases
* Address jumpiness on furious refresh cycle
* Revert "Address jumpiness on furious refresh cycle"
This reverts commit 5c03b36918.
* Another attempt at jumpiness
* Enforce less noticeable lag applying the diff from loading to loaded
* Reduce flashing on stats load
---------
Co-authored-by: Sanne de Vries <sannedv@protonmail.com>
* virtual rollups -> consolidated sites
* fix queryparser and test
* fix legacy_query_builder and test
* fix typespec
* simple consolidated view query test (Stats API)
* add test util
* skip verification on consolidated sites
* add test for internal (dashboard) stats query
* fix CE
* Fix showing consilidated stats on the dashboard
...by making sure site.native_stats_start_at and stats_start_date are
set.
* fix CE again
* add typespecs and tests for native_stats_start_at
* skip else clause
* use ConsolidatedView.enable in test util
* defensive where builder
* remove redundant cache
* get query.consolidated_site_ids from cache
* fix test
* fix CE
* fix CE again
* eligibility check when enabling cvs
* create cvs with native_stats_start_at assigned already
* fix dialyzer warning
* FULL join for time:hour as well as time:minute
Follow-up to https://github.com/plausible/analytics/pull/5694/files#r2321567271
A session might be active over multiple hours, but not (currently)
reported as such when requesting only specific metrics per hour.
This fixes that problem.
* Handle full join logic correctly
---------
Co-authored-by: Uku Taht <Uku.taht@gmail.com>
* Refactor table_decider#partition_metrics
* Refactor query pipeline to return a list of subqueries after splitting
* Move order_by out of join logic
* Refactor joining logic in query_builder
1. JOIN type is now set in QueryOptimizer
2. JOIN logic is now table and list-size agnostic
* Comment an edge case
* Rebuild session/visit smearing
Previously, whenever graphing any visit metric hourly/realtime, visit_duration and other
visit metrics would be way higher than expected, due to long sessions
dragging each bucket up and up. Now visits/visitors metrics are still
smeared and other visit metrics are counted under last bucket user was
active in.
visits metric was also overcounted (see new tests).
* Remove unneeded case
* Unit test for smearing in tabledecider
* change log level from info to notice on relevant logs
* bump log level to notice for everything except request logger
* format
* fix choose_plan_test.exs for good (starter tier launch)
* fix typo to stop logging error in test output
* add time:minute interval to internal api schema
* always get visitors and visits from sessions table when time:minute dimension used
* query-api generate types
* changelog update
* Support passing `include` as a query parameter for dashboard APIs
* Mark time-on-page metric sortable
It now is thanks to the changed query
* new-time-on-page flag with cutoff being sent to the frontend
* Add correct tooltip title
* Implement metric warning for when legacy and new time_on_page metrics are mixed
* Send legacy_time_on_page_cutoff to backend
* Make time-on-page graphable with the new metric
* Only show metric warnings for time_on_page if flag is enabled
* Changelog
* Solve an clickhouse error when querying timeseries with only legacy time-on-page
* Add tests for timeseries of new time-on-page
Along the way fix an issue with comparisons not working properly
* Solve a typing issue
* Allow toggling legacy_time_on_page_cutoff off in dashboard
* Slightly better workaround
* Solve typing issue
* Prettier
* Guard against no warning
* Solve warning
* Default to time_on_page
* Add new columns to schema
* Read from new column in legacy query
* Read/write new imported_pages columns
* Remove time_on_page column from imported_pages
* Simple, stupid new_time_on_page metric
* Update csv_importer schema
* Refactor: consistent __internal helpers, this will help with joining the query
* Refactor select_joined_metrics
* Refactor: pass `query` to event_metric
* Refactor: remove needless site argument from various calls
* Legacy joining query attempt
* Move test around
* Add more tests for both legacy and new time_on_page metrics in query API
* time_on_page reported in seconds
* timeseries test for metric
* WIP
* Wrap main query in subquery - without this run into trouble performing the join
* Calculate time_on_page in main query, no more new_time_on_page
* Add some TODOs
* Return NULL over 0 when no visits with time-on-page data
* Update moduledoc
* Update some tests that were not expecting integers
* Add a TODO
* Update tests
* Make graphing time series with combined metrics work.
* Slightly more consistent approach to flag updating in APIv2
* Seeds with engagement data
* Make graphing time series when cutoff is in the middle work
Bakes less assumptions into everything as well.
* Rename to legacy_time_on_page_cutoff
* Fixup lib/plausible_web/controllers/api/external_query_api_controller.ex
* Remove a todo and dead/misleading code
* Remove a resolved todo
* Remove needless rounding
* gen types
* Update pages test
* Remove needless columns from select
* Update tests: timestamps and remove comment
* Flip branches
* Refactor: remove metrics argument from merge_imported()
* Support querying percentage without visitors metric
* Fix ordering by special metrics with imports causing a 500
We don't calculate all metrics directly on imports, hence cannot order
the import by them either.
* Changelog
* Revert "Disable scroll depth exports temporarily"
This reverts commit 48ad691f53.
* Remove support for pageleave events being equivelent to engagement in ingestion
* Explicit column ordering inside csv imports
Subtle change, but this ensures that CSVs that contain extra columns or differently named columns do not cause trouble
* Add scale_sample fragment helper
* Update scroll depth queries to be based on visits rather than visitors
* Add test demonstrating session-based results
* Update csv test (session vs user-based difference)
* Attempt to update csv tests
* PR feedback
* migration: add scroll_threshold to goals
* update goal schema
* setup simple UI for creating scroll goals
* add ability to filter and breakdown scroll goals
* fix goals form tests
* add valiation for page path exists
* move todo comments to expression.ex
* move tests
* make it clear that scroll_threshold is optional
* avoid calling Plausible.Goal.type() too many times
* do not consider 255 scroll depth a conversion
* migration: add scroll_threshold to goals
* do not drop the old index yet
* More efficient goals join again
* Refactor: move goals stats code explicitly under Stats.Goals module
* Move code under Plausible.Stats.Goals
* 254 -> 100
* add scroll_threshold field to goal schema + new unique constraint
* adjust test to test what it claims to
* mix format
* add migration
* consider imported query unsupported when page scroll goal filter
* add missing tests
* pattern match imported argument
* silence credo
* Update lib/plausible/stats/sql/expression.ex
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* use site_imports populated in test setup
---------
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Expose site_id and site_native_stats_start_at via query
This allows to do more query-building without exposing and passing `site` directly.
* Very basic has_done/has_done_not operator support
No event:goal support yet, no validations
* Add validations that only event: dimensions can be used within has_done/has_done_not
* Allow event:goal filters nested within has_done/has_done_not behavioral filters
* Minor fix for do_decide_custom_prop_table
* has_done support for goals
Minor changes along the way:
- preloaded_goals structure changes
- event:goal restrictions were loosened within has_done
- we don't allow nesting has_done anymore
* Dont query imports when behavioral filters are present
* Update callsites of filtering_on_dimension? to work with new behavioral filters
* has_done_not -> has_not_done
* Changelog entry
* Typegen
* credo cleanup
* Fix changelog
* Remove changelog
* Mark has_done as internal-only
* combine two validations into a single loop
* has_done is now session-based not user-based
* Update a test
* Update transform_tree
* include scroll_depth in full pages export
* import scroll_depth from CSV
* query scroll depth from imported data
* fix ordering by scroll depth with imported data
* fix imported scroll depth query + more tests
* enable scroll depth in top stats with imported data
* add main graph test
* fix test and native scroll depth sum select
* Update lib/plausible/exports.ex
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
* adjust test
* adjust test to catch error
* export/import/count pageleave_visitors
* extract base_q in export_pages_q
* rename total_visitors to pageleave_visitors
---------
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
* ingest missing pageleave as 255 for pageleave events
* return scroll depth as nil when no valid pageleave data in range
* also set empty comparison value as nil instead of 0
* add data migration
* Improve report performance in cases where site has a lot of unique pathnames
Ref: https://3.basecamp.com/5308029/buckets/39750953/card_tables/cards/8052057081
JOINs in ClickHouse are slow. In one degenerate case I found a user had
over 20 million unique paths in an import, which resulted in extremely slow
JOINs. This introduces a sort-of hacky solution to it by limiting the
amount of data analyzed.
Query timing without this change:
```
9 rows in set. Elapsed: 11.383 sec. Processed 49.16 million rows, 5.75 GB (4.32 million rows/s., 505.29 MB/s.)
Peak memory usage: 14.75 GiB.
```
After:
```
9 rows in set. Elapsed: 0.572 sec. Processed 49.18 million rows, 5.75 GB (86.03 million rows/s., 10.06 GB/s.)
Peak memory usage: 9.01 GiB.
```
* Splitting should no longer remove pagination. Handle special cases in special_metrics.ex
* select_merge_as in imports
This sets up selected_as aliases which will be used in a subsequent commit
* Add explicit ORDER BY to import
* Rewrite comment
* quoting
* merge conflict
* Split test
* Merge conflict fail fix
* WIP: Optional modifiers to queries
* WIP: Modifiers v2
* Use preloaded_goals when determining whether imports can be included
This was previously broken with conversion_rate totals metrics since it removed event:goal
filters but did not update preloaded_goals
* Preload goals according to modifiers
* Make case_sensitive: false work for is/contains operators
* Make modals send { case_sensitive: false } to backend for search
* CHANGELOG.md
* Typegen
* Prettier
* Refactor: more DRY where_builder for case sensitivity
* Support case_sensitive modifier for is_not/contains_not
* Cleanup
* credo
* remove defaults
* negating a previously set filter
* migration: add scroll_depth to events_v2
* (cherry-pick) ingest scroll depth
* replace convoluted test with more concise ones
* QueryParser: parse internal scroll_depth metric + validation
* turn QueryComparisonsTest into QueryInternalTest
* rename file
* (cherry pick) query scroll depth 15b14d3
...and move the tests into `internal_query_test.exs`
* review feedback
* Get rid of unnecessary separation between aggregate and group scroll depth
* Drop irrelevant other metrics in tests
* add test ensuring scroll depth unavailable in Stats API v1
* Put scroll depth on the dashboard
* Top Stats
* Main Graph
* Top Pages > Details
* feature flag for dashboard scroll depth access
* ignore credo warning
* enable scroll_depth flag in tests
* remove duplication
* write timestamps explicitly in a test
* revert moving tests around
* Add query_comparisons_test back
* Move scroll_depth tests into query_test
* Delete query_internal_test
* rename setup util (got updated on master)
* use pageleave_factory where applicable
* Use the correct generated query-api.d.ts
* npm format
* Remove query.v2 flag
This was originally used for making sure queries use the right table as
migrating to APIv2. This is no longer needed
* Remove experimental_reduced_joins flag
* Add filter clauses for each main result filter
This handles the case where main query has a limit and results change. Doesnt handle metrics like percentage.
* Fix percentage calculations by ignoring breakdown-related filters in totals queries
* Refactor comparisons test suite
* Move comparisons logic to comparisons module
* New route for internal query tests
Only to be used in testing
* Support comparison queries with imports/breakdowns
* time dimension predicate extraction
* Clean up a test
* Update docstring
* Update route test
* fix a typo
* add experimental pageleave script variant
* also send pageleave events on SPA navigation
* disallow goals with 'pageleave' event name
* do not count pageleaves towards the event metric
* remove duplication in test file
* do not update sessions on pageleave events
* ignore pageleaves in the current time_on_page implementation
* make pageleave events not billable
* rename function
* Prevent multiple pageleaves being sent at the same time
Offset-based pagination is used to make sure Looker integration
is able to work as efficiently as possible. To know how many
requests users need to do `include.total_rows` option was added.
* query.date_range is now in UTC instead of user timezone
This simplifies things down the line and fixes several bugs where
query.date_range is cast to naivedatetime for ecto purposes
Many places still remain broken:
- comparison queries
- `to_date_range` calls
* Make default_for_date_range not care about time zones
* Make timezone parameter mandatory for to_date_range
* Simplify utc_date_range, update legacy query builder
* Fix more cases where query date range is needed
* query.date_range -> query.utc_time_range
* Query.date_range/1 function
* ensure_include_imported update
* Clean up send_email_report
* Safeguard session queries relying on `sign` from faulty old session entries
* Comment updated metric
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Apply safeguards to `bounce_rate` metric only
* Add note to bounce rate definition in SQL fragments as well
* Add test for graceful bounce rate handling in breakdown
* Make user_id more unique
* Add a note to the test
* Move regression test to APIv2 tests
---------
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Rename matches/does_not_match filters internally
These have never been exposed to the frontend/user directly, only via
APIv1 filtering syntax. As such we are free to rename these without
breaking things
* Rename function arguments for consistency, simplify
* Add support for `match`/`not_match` operators for query apiv2
These match the string against a regular expression, as defined in
https://github.com/google/re2/wiki/Syntax
* not_match -> match_not
* does_not_contain -> contains_not
Note that for backwards compatibility:
- Browser handles does_not_contain in URL
- Backend will handle does_not_contain in queries for a day where we will remove it for better autocompletion
* not_matches_wildcard -> matches_wildcard_not
* prettier
* match -> matches
* Fix and test fix for matches_wildcard against prop when prop is missing
* Custom properties support for matches/matches_not
* Restore contains_not
* Test contains and contains_not behavior for custom properties
* add realtime date_ranges into the private API schema
This commit starts parsing date ranges into a new NaiveDateTimeRange
struct, rather than a simple Date.Range.
* transform realtime labels into negative integers + test
* move schema type argument to last position in helper functions
* allow passing a date param + tests
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* keep test file structure consistent
* Turn NaiveDateTimeRange into DateTimeRange
* change 'now' field from NaiveDateTime to DateTime in v2 query
* fix minute interval labels + add missing tests
* return query_result.date_range as iso8601 timestamps with timezone
* allow timestamps with tz as date_range arguments in API v2
* delete Plausible.Timezones.to_utc_datetime
* simplify returning comparison periods
* add comment about realtime not supported in comparisons
* pass only now instead of test_opts
* drop redundant else branch
* separate tests
* stick to a single check_date_range function in tests
* fix credo error
---------
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Add data migration for creating and syncing location_data table and dictionary
* Migration to populate location data
* Daily cron to refresh location dataset if changed
* Add support for visit:country_name, visit:region_name and visit:city_name dimensions
Under the hood this relies on a `location_data` table in clickhouse being regularly synced with
plausible/location repo and dictionary lookups used in ALIAS columns
* Update queue name
* Update documentation
* Explicit structs
* Improve docs further
* Migration comment
* Add queues
* Add error when already loaded
* Test for filtering by new dimensions
* Update deps
* dimension -> select_dimension
* Update a test
* Use goals from postgres to build goal filter
* Remove unnecessary goal preloading
* Add contains filter for goals
* Make sure the correct imported tables are used when goals are in filters
* Remove 'contains' filter type for now
* Remove TODO comment
* Fix QueryParser test
* move goals.ex into a goals subfolder
* extract goal filtering logic into a separate module
* remove duplication from Imported.Base
* Extract `get_filter_goals` function
* Credo suggestions
* Credo ignore nesting
* apply suggestion - remove Sql.Util
* apply suggestion - pass imported? via opts
* remove duplicate function
* Uncomment tests
* Fix 500 error with goal suggestions
---------
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
Sanne de Vries
Adam Rutkowski
Adrian Gruntkowski
RobertJoonas
Karl-Aksel Puulmann
Uku Taht
ruslandoga