* Update report percentages on dashboard and details view
* Add percentages to Countries, Regions, and Cities reports
* Add percentages to Channels, Sources, and UTM reports
* Add percentages to top pages, entry pages, and exit pages reports
* Update tests to include percentages
* Change dashboard copy from title case to sentence case
* Update details modal style
* Make animations snappier
* Introduce max height to modal and make inner content scrollable
* Improve modal mobile design
- Enable horizontal scroll for details modal on mobile
- Add responsive spacing and positioning to modal
* Added mobile tap behavior to external link in list report
* Show tooltips only when in comparison mode or when the number is abbreviated
* remove previously added showTooltip prop
- This isn't needed anymore since we now handle the tooltip logic in the MetricValue component
* Show long format upon hovering detailed view metrics
* Added mobile tapping behaviour to detailed view
* Added percentages to all detailed views
* Add mobile swipe-to-close behavior for modal
* Adjust sensitivity of modal drag to close
* Use hammerjs for swipe-to-close modal behaviour
* Prevent dragging if gesture starts inside table
* Show 2 decimal places for percentages < 0.1% across dashboard
* Adjust dark mode styles
* Add hover effect to external link icon
* Update tests to expect two-decimal percentages
* Undo hammer install and revert to old modal styling
* Remove CR and % columns from goals and custom props reports on dashboard, and show on hover in detailed view
* Remove unused constants
* Undo conversion rate on hover behaviour
- Unlike percentages, CR should show permanently.
* Show percentages permanently in custom props detailed view
* Adjust width of conversion metrics column
* Updated metric-value test
* Update top-bar test
* Added changelog entry
* Fix test expectations for percentages with imported data
- Update tests to expect correct percentages (≤100%) when imported data is included. These tests will fail until the percentage calculation bug is fixed, documenting the expected behavior.
* Add imported_visitors to tests to ensure correct total_visitors calculation
* Correct imported_visitors count in test
* rename QueryParser to ApiQueryParser
* move utc_time_range construction to querybuilder
* input_date_range format
* rename 30m atom to realtime_30m
* move build_comparison_date_range into do_build
* Limit preloading goals
* Enforce max limit for goals per site
* typo
* credo
* Remove logger call
* Integrate #5916
* Add a test
* Add test
* Unignore opts
* Include revenue data for all detailed API responses except entry/exit pages
* Expose revenue data in all breakdown modals except entry/exit pages
* Add revenue metrics to breakdown response only on EE
* Change query builder to enable querying event metrics \w session dimension
* Add revenue metrics to entry and exit pages breakdowns
* Expose revenue data in entry and exit pages breakdowns
* Use `argMax` for `exit_page` and `exit_page_hostname` dimensions (h/t @ukutaht)
* Don't handle event-only dimensions with session-only metrics for now
* Add tests for all breakdowns
* Add clarifying comments in code
* Mark revenue tests as EE-only
* Replace usages of `Timex.to_unix` with native API
* Wrap call to `Timex.is_valid_timezone?`
* Wrap calls to `Timex.today(tz)`
* Replace `Timex.today()` with `Date.utc_today()`
* Replace `Timex.now()` with `DateTime.utc_now()`
* Replace `Timex.compare` with `Date.compare`
* Wrap `Timex.diff` calls
* Replace `Timex.Timezone.convert` with `DateTime.shift_zone!`
* Wrap `Timex.parse!`
* Replace `Timex.to_date` with native API calls
* Replace `Timex.beginning|end_of...` with native API calls for Date
* Wrap `Timex.beginning|end_of...` for DateTimes and Dates for years
* Replace `Timex.format(!)` with native API calls
* Replace `Timex.to_naive_datetime` with native API calls
* Wrap time humanizing routines using Timex
* Remove unnecessary `use Timex` instances
* Replace `Timex.shift` with native API calls
* Make `QueryParser.parse_date` handle gaps and ambiguities gracefully
* Replace `Timex.now(tz)` with `DateTime.now!(tz)`
* Use a more suitable Date function for comparison (h/t @aerosol)
* Revert "Revert "Trim `month`, `year`, `day` periods to local now on main graph (#5668)" (#5684)"
This reverts commit 2d11681f25.
* Does not trim for comparisons
* Include the current hour in the trimmed time range
* Trim `month`, `year`, `day` periods to now on main graph
* Revert "Trim `month`, `year`, `day` periods to now on main graph"
This reverts commit 4f3930111d3a2737a51686e067d9b64f0d85ad58.
* Re-implement trimming in query optimizer instead
* Update JS types
* This is getting confusing
* Trim in stats_controller
* Set `query.now` based on query_praser and date results
* query.period -> query.input_date_range
* Changelog
* Test for response query.date_range
---------
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* Make `site_role/2` return whether real is team or site scoped
* Rename `has_admin_access?` => `has_editor_access?`
* Switch current team if user is a team member in site's team
* CSV import/export support for time-on-page
Note only the new time-on-page metric is exported this way
* visibility check for graphing of time_on_page
* FE no longer receives/sends legacy_time_on_page_cutoff
* Remove current_user from exports
* Remove legacy_time_on_page_cutoff from query.include, make behavior work off of site.legacy_time_on_page_cutoff explicitly
* Remove dead function
* More current_user_id removals
* Backend: get rid of scroll_depth.ex
This commit starts treating the scroll depth feature as *always visible*,
removing the ClickHouse queries on dashboard loads and starting to always
include scroll_depth in dashboard reports and export queries.
* Frontend: remove site.scrollDepthVisible
* adjust csv_importer_test
* remove site.scroll_depth_visible_at
* Support passing `include` as a query parameter for dashboard APIs
* Mark time-on-page metric sortable
It now is thanks to the changed query
* new-time-on-page flag with cutoff being sent to the frontend
* Add correct tooltip title
* Implement metric warning for when legacy and new time_on_page metrics are mixed
* Send legacy_time_on_page_cutoff to backend
* Make time-on-page graphable with the new metric
* Only show metric warnings for time_on_page if flag is enabled
* Changelog
* Solve an clickhouse error when querying timeseries with only legacy time-on-page
* Add tests for timeseries of new time-on-page
Along the way fix an issue with comparisons not working properly
* Solve a typing issue
* Allow toggling legacy_time_on_page_cutoff off in dashboard
* Slightly better workaround
* Solve typing issue
* Prettier
* Guard against no warning
* Solve warning
* Main graph: Graph revenue metrics as zeroes instead of nils.
This has no effect on existing APIs as:
- Comparisons/empty data point logic only affects APIv1 which does not expose revenue metrics
Also changes semantics for visit_duration to be more in-line with the
rest.
* Improve dashboard main graph
- Graph tooltip for revenue metrics now shows revenue with the relevant
monetary unit.
- Graph tooltips for scroll depth, visit duration and so on now shows `-`
instead of 0 on dates where no relevant visits occurred.
* Remove business logic duplication
* write comparison_utc_time_range into query
* write site_imports into query
* use struct! over Query.set
* variable to module attr
* remove redundant comment
* reject over filter
* Map.reject over Enum.reject
* add migration
* add schema field
* mark site_imports with has_scroll_depth
* add function to get imports in query range
* add scroll_depth metric warning in QueryResult
* return scroll_depth warning in top stats
* render minimalistic warning in top stats
* minimalistic warning in Top Pages breakdown
* prettier format
* silence credo
* add test
* use a snapshot of SiteImport schema in data migration
* also use a snapshot list of imported_* tables
* moduledoc (credo)
* change tooltip message
* change metric warnings structure in top stats response
* pass meta from queryresult directly
* revert top_stats_entry refactor
* prettier
* stop using SiteImport module in data migration
* add migration
* move scroll_depth_enabled? fn
* maybe set engagement_metrics_enabled_at when requesting dashboard
* maybe set engagement_metrics_enabled_at in shared_link action
* maybe set engagement_metrics_enabled_at on full export
* fix tests
* feature gate scroll depth on the dashboard with site.engagement_metrics_enabled_at
* feature gate scroll depth in full export too
* fix npm ci
* Rename things into FE, remove unneccessary flag checks
* Continue with renaming
* Rename site flag to be more descriptive
* Move business logic, calculate based on scroll depth, make more precalculatable
* Some docs
* Rename to site.scrollDepthVisible in frontend
* Update migration
* Fix template
* Remove boilerplate from tests
* Update tests
* More straight-forward test
* Update condition
---------
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* preload completed site_imports for site when needed
* remove unused field from top stats response
* refactor imports meta
* remove redundant function clause
* fix ordering in queryresult
* preload completed_imports in plugs
* write code more short
* Expose site_id and site_native_stats_start_at via query
This allows to do more query-building without exposing and passing `site` directly.
* Very basic has_done/has_done_not operator support
No event:goal support yet, no validations
* Add validations that only event: dimensions can be used within has_done/has_done_not
* Allow event:goal filters nested within has_done/has_done_not behavioral filters
* Minor fix for do_decide_custom_prop_table
* has_done support for goals
Minor changes along the way:
- preloaded_goals structure changes
- event:goal restrictions were loosened within has_done
- we don't allow nesting has_done anymore
* Dont query imports when behavioral filters are present
* Update callsites of filtering_on_dimension? to work with new behavioral filters
* has_done_not -> has_not_done
* Changelog entry
* Typegen
* credo cleanup
* Fix changelog
* Remove changelog
* Mark has_done as internal-only
* combine two validations into a single loop
* has_done is now session-based not user-based
* Update a test
* Update transform_tree
* include scroll_depth in full pages export
* import scroll_depth from CSV
* query scroll depth from imported data
* fix ordering by scroll depth with imported data
* fix imported scroll depth query + more tests
* enable scroll depth in top stats with imported data
* add main graph test
* fix test and native scroll depth sum select
* Update lib/plausible/exports.ex
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
* adjust test
* adjust test to catch error
* export/import/count pageleave_visitors
* extract base_q in export_pages_q
* rename total_visitors to pageleave_visitors
---------
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
* Comment out legacy fields and relationships
* WIP
* WIP 2
* WIP 3
* wip
* Remove teams backfill and consistency check scripts
* WIP 3
* Fix CheckUsage tests
* Update billing/subscription tests
* WIP 4
* Make site transfer fail if some invitation already exists
* Fixup: do symmetric invitation/site transfer check
* Update UI bugs: make listing sites/inviting admins work like before
* Fix Sites test
* Fix external sites controller test
* Fix live sites tests
* Fix props availability lookup
* Fix site controller tests
* Fix billing controller tests
* WIP - accept invitation tests
* Another round of test fixes + invitations logic bugs
* users_test -> teams_test
* Update registration via invitation
Here, we still rely on "polymorphic" invitation structures,
hence the "unified by id" helper.
For now, it'll remain local unless we discover it's
needed in the broader `Teams.Invitations` context.
cc @zoldar
* Yet another round of test and bugfixes along the way
* Include team in site setup success e-mail
* Fix send_site_setup_emails worker
* Fixed almost all tests except CRM ones
* Update enterprise plan admin test
* Fix CRM + remaining tests
* Address credo warnings (modulo one FIXME)
* Remove last FIXME and rephrase the invitation test case description
* Set Team fields via User CRM transparently
* Map user reference in Enterprise Plan CRM via team owner
* Fix resource actions in user CRM
* Get rid of warning when opening create form in API keys CRM
* Stop emitting warnings when editing Enterprise Plans via CRM
* Tests: Bump await_clickhouse_count interval
* Remove XXX marker
* Fix register from invitation link in email sent for ownership transfer
* Simplify fetching all pending site ownership site IDs
* Remove commented out schema fields
* Remove unused functions
* Address flakiness in ingest counter tests
* Remove unused `Teams.Sites.create`
* Don't restart trial on team with subscription when creating site
* Account for cases of legacy teams with empty trial expiry date
* Revert "Address flakiness in ingest counter tests"
This reverts commit 60dc1e4115.
* Fix flaky ingest counters tests under load
* Attempt 2
* Pre-emptively hardcode site ids in sampling cache test
to avoid supplying the same IDs alongside with counters test,
that inserts through another repo (async).
what we're observing is, clickhouse not summing mergetree columns fast
enough, even though we wait quite a bit.
* Fix ingest counter tests by accounting for delayed summation
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* migration: add scroll_depth to events_v2
* (cherry-pick) ingest scroll depth
* replace convoluted test with more concise ones
* QueryParser: parse internal scroll_depth metric + validation
* turn QueryComparisonsTest into QueryInternalTest
* rename file
* (cherry pick) query scroll depth 15b14d3
...and move the tests into `internal_query_test.exs`
* review feedback
* Get rid of unnecessary separation between aggregate and group scroll depth
* Drop irrelevant other metrics in tests
* add test ensuring scroll depth unavailable in Stats API v1
* Put scroll depth on the dashboard
* Top Stats
* Main Graph
* Top Pages > Details
* feature flag for dashboard scroll depth access
* ignore credo warning
* enable scroll_depth flag in tests
* remove duplication
* write timestamps explicitly in a test
* revert moving tests around
* Add query_comparisons_test back
* Move scroll_depth tests into query_test
* Delete query_internal_test
* rename setup util (got updated on master)
* use pageleave_factory where applicable
* Use the correct generated query-api.d.ts
* npm format
* Revert "Revert "Remove no change icon, better alignment""
This reverts commit a69d0a14a7fe038fe110907571240ad99a2ba6a4.
* different arrows
* Render graph tooltips in react
* Use ChangeArrow in graph tooltips
* Only display a single date if viewing a single date
* Tests for breakdown responses with labels
* Simplify code a bit
* strokeClass
* refactor ChangeArrow
* Play with stroke widths
* jest tests
* Silence logging in jest tests
* not variable stroke width
* prettier
* Conversion rate widths
* Update a test
* Trim trailing dates for label when comparing running month
* Reformat file
* Update expect.toHaveTextContent for multiline
* Update date range labelling to be more consistent with the frontend
* reformat
* new endpoint for custom prop value filter suggestions
* add test coverage for searching prop suggestions
* plug the new endpoint into FE
* update changelog
* format JS
* Update CHANGELOG.md
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
---------
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Comparisons: Move code to LegacyQueryBuilder
* WIP: Return comparison results to frontend
* refactor: remove useless param
* Different result format
* Pass object to metric.renderValue
* remove dead code
* Fixup response format
* Comparison in a tooltip
Not perfect at all, but a good start. Problems arise with money etc.
* Simple change arrow
* Extract metric entry to ts
* popper attempt WIP
* Slightly nicer content
* Solve warning
* Unified changeArrow in app
* Remove needless spanning
* Always set `graph_metric` in top stats.
FE already has business logic around whether a given metric is graphable
* Remove dead code
* Move Money module under dashboard utils, keep in build
* change <Metric /> definition to take in a `formatter` and store default formatters in another (typed) const
* Use standard system for formatting numbers
* Arrows only in table
* remove dead import
* Inline renderValue
* Render metric name in tooltip
* numberFormatter -> numberShortFormatter
* numberShortFormatter update
* Separate long/short formatters
* Use long vs short formatters
* Put column name into tooltip
* Slightly improved label handling for percentages, conversion rate
* Improved boundary handling in tooltip.js
* Iterate tooltips, no tooltip for - revenue
* Update top stats tests after graph_metric change
* Change revenue metrics stats API return structure
Conversion now happens earlier in query pipeline, we return float for comparison purposes
* useQueryContext in a component
* graph_metric for current visitors to fix realtime view
* No tooltips if fully - row
* renderValue as a proper function
* Simplify MetricEntry
* Use common const
* tooltip to typescript
* More explicit return structure
* metric-entry -> metric-value
* Restore some files
* ChangeArrow
* Restore MoreLink
* Fix typing in MoreLink
* <MetricValue />
* Tests for MetricValue and ChangeArrow
* details modal fixups
* re-add space between arrow and percentage
* Solve stylelint issues
* Update test
* Format
* Add flag `breakdown_comparisons_ui`
* reformat
* Remove no change icon, better alignment
* Revert "Remove no change icon, better alignment"
This reverts commit a8d62b6383.
* number-formatter.ts
* numberLongFormatter refactor
* useMemo dependency
* Handle nulls/undefined in top stats
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* Refactor comparisons to a new options format
Prerequisite for APIv2 comparison work
* Experiment with default include deduplication
* WIP
Oops, breaks `include.total_rows`
* WIP
* Refactor breakdown.ex
* Pagination fix: dont paginate split subqueries
* Timeseries tests pass
* Aggregate tests use QueryExecutor
* Simplify QueryExecutor
* Handle legacy time-on-page metric in query_executor.ex
No behavioral changes
* Remove keep_requested_metrics
* Clean up imports
* Refactor aggregate.ex to be more straight-forward in output format building
* top stats: compute comparison via apiv2
* Minor cleanups
* WIP: Pipelines
* WIP: refactor for code cleanliness
* QueryExecutor to QueryRunner
* Make compilable
* Comparisons for timeseries works
Except for comparisons where comparison window is bigger than source query window
* Add special case for timeseries
* JSON schema tests for comparisons
* Test comparisons with the new API
* comparison date range parsing improvement
* Make comparisons api internal-only
* typegen
* credo
* Different schemata
* get_comparison_query
* Add comment on timeseries result format
* comparisons typegen
* Percent change for revenue metrics fix
* Use defstruct for query_runner over map
* Remove preloading atoms
* On the dashboard, allow Metrics (aka column configurations) to
* specify width they need
* specify if they are sortable
* Default sort order for breakdown endpoints is hardcoded on the dashboard (needed to show what column the report is sorted by)
* Unifies Google Keywords modal with other breakdowns
* query.date_range is now in UTC instead of user timezone
This simplifies things down the line and fixes several bugs where
query.date_range is cast to naivedatetime for ecto purposes
Many places still remain broken:
- comparison queries
- `to_date_range` calls
* Make default_for_date_range not care about time zones
* Make timezone parameter mandatory for to_date_range
* Simplify utc_date_range, update legacy query builder
* Fix more cases where query date range is needed
* query.date_range -> query.utc_time_range
* Query.date_range/1 function
* ensure_include_imported update
* Clean up send_email_report
* Dont include imports for time:hour and time:minute dimensions
Also include more information about import warnings in query results
* Update lib/plausible/stats/query_result.ex
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
* Revert patch
* Imported disabled graph notice (#4522)
* add explicit skip_imported_reason for unsupported interval
* stop returning information about imports from main_graph
* return warning about interval in Stats API Timeseries
* display warning bubble about interval too short for imported data
* update changelog
* improve styling of the exclamation circle icon
* return tuple from timeseries instead of map
* rename variable
* Update CHANGELOG.md
---------
Co-authored-by: RobertJoonas <56999674+RobertJoonas@users.noreply.github.com>
* add realtime date_ranges into the private API schema
This commit starts parsing date ranges into a new NaiveDateTimeRange
struct, rather than a simple Date.Range.
* transform realtime labels into negative integers + test
* move schema type argument to last position in helper functions
* allow passing a date param + tests
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* keep test file structure consistent
* Turn NaiveDateTimeRange into DateTimeRange
* change 'now' field from NaiveDateTime to DateTime in v2 query
* fix minute interval labels + add missing tests
* return query_result.date_range as iso8601 timestamps with timezone
* allow timestamps with tz as date_range arguments in API v2
* delete Plausible.Timezones.to_utc_datetime
* simplify returning comparison periods
* add comment about realtime not supported in comparisons
* pass only now instead of test_opts
* drop redundant else branch
* separate tests
* stick to a single check_date_range function in tests
* fix credo error
---------
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Extract session management from AuthController
* Don't explicitly pass `current_user_id` to `live_render`'s session
* Add ability to retrieve session and user from token via `UserAuth`
* Always fetch current user (or just id) via `UserAuth` API
* Introduce `UserSession` as an embedded schema for now
* Make `UserAuth.get_user/1` accept `UserSession` as an input
* Introduce LV auth context populating user data from session on mount
* Refactor `AuthPlug` and make it populate `current_user_session` as well
* Rely on authenticated user data provided by auth plug or LV context
* Make `Sites.get_for_user(!)` accept `User` struct as well
* Set `logged_in` cookie explicitly when it's out of sync with session
* Expand modules documentation a bit
* Improve and extend tests slightly
* add the ability to pass date param into Query.build
* stop returning interval from main_graph controller action
* globally rename 'date' interval to 'day'
* Allow parsing query date range from a 'period' param
The 'period' param will not be exposed in the public API, but makes it
possible to construct a "realtime" query.
* Revert "Allow parsing query date range from a 'period' param"
This reverts commit c5630eaef9.
* call beginning_of_time for first_datetime instead of last
* allow 'realtime' and '30m' date_range shortcuts
* evaluate date_range to 'realtime' or '30m' instead of Date.range(today, today)
* Revert "evaluate date_range to 'realtime' or '30m' instead of Date.range(today, today)"
This reverts commit a887569ec5.
* Revert "allow 'realtime' and '30m' date_range shortcuts"
This reverts commit 91ae0fa5e6.
* fix graph tooltips
* Set log_comment with request information
* CRMAuthPlug -> SuperAdminOnlyPlug
* Super basic debug view
* Handle clustered setups
* Changelog entry
* Cleanup
* fragment trick to use ecto querying, filtering
* Move clustered_table? function to IngestRepo module
* Format
* More resilient user_id getting in helper
* return concatenated browser names and versions directly from the API
* return concatenated os names and versions directly from the API
* add detailed views for all device reports
* extract put_combined_name_with_version function
* return only version under the name key when detailed=true
* update changelog
* add more metrics into detailed views of Devices reports
* split up different devices modals into separate files
* Extend the GSC API with search functionality
* Fix typo in error tuple atom
* log GSC errors
* rename confusing variable name
* Fix the API response format and error handling
* Read results under the `results` key to be consistent with other
endpoints
* Return errors with a non-200 status code, and with an error payload
that will be well constructed into ApiError
* rebuild Google Keywords modal with useAPIClient
* Add pagination support in Search Terms API
* delete unused fixture file
* rename fixture files
* fix tests
Sanne de Vries
RobertJoonas
Adam Rutkowski
Adrian Gruntkowski
Uku Taht
Karl-Aksel Puulmann
Artur Pata
ruslandoga