* rename QueryParser to ApiQueryParser
* move utc_time_range construction to querybuilder
* input_date_range format
* rename 30m atom to realtime_30m
* move build_comparison_date_range into do_build
* Limit preloading goals
* Enforce max limit for goals per site
* typo
* credo
* Remove logger call
* Integrate #5916
* Add a test
* Add test
* Unignore opts
* rename Query.build -> Query.parse_and_build
* rename two test files and move 4 %Query{} building functions into subfolder
* rename StatsAPIFilterParser to LegacyStatsAPIFilterParser
* rename Filters.QueryParser to QueryParser
* turn QueryParserTest into QueryParseAndBuildTest
* move query_parser.ex out of filters directory
* separate build from parse
* disable sample_threshold in the new intermediate build function, for now
* remove now redundant test util functions
* remove unused import
* address todo from earlier
* credo
* Make module names in sync with paths in tests
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Include revenue data for all detailed API responses except entry/exit pages
* Expose revenue data in all breakdown modals except entry/exit pages
* Add revenue metrics to breakdown response only on EE
* Change query builder to enable querying event metrics \w session dimension
* Add revenue metrics to entry and exit pages breakdowns
* Expose revenue data in entry and exit pages breakdowns
* Use `argMax` for `exit_page` and `exit_page_hostname` dimensions (h/t @ukutaht)
* Don't handle event-only dimensions with session-only metrics for now
* Add tests for all breakdowns
* Add clarifying comments in code
* Mark revenue tests as EE-only
* Ensure `conn` from `Plug.Conn.read_body` is always passed down the pipeline
* Alter persistor related histogram metrics for better view of timings
* Update typespec
* Replace usages of `Timex.to_unix` with native API
* Wrap call to `Timex.is_valid_timezone?`
* Wrap calls to `Timex.today(tz)`
* Replace `Timex.today()` with `Date.utc_today()`
* Replace `Timex.now()` with `DateTime.utc_now()`
* Replace `Timex.compare` with `Date.compare`
* Wrap `Timex.diff` calls
* Replace `Timex.Timezone.convert` with `DateTime.shift_zone!`
* Wrap `Timex.parse!`
* Replace `Timex.to_date` with native API calls
* Replace `Timex.beginning|end_of...` with native API calls for Date
* Wrap `Timex.beginning|end_of...` for DateTimes and Dates for years
* Replace `Timex.format(!)` with native API calls
* Replace `Timex.to_naive_datetime` with native API calls
* Wrap time humanizing routines using Timex
* Remove unnecessary `use Timex` instances
* Replace `Timex.shift` with native API calls
* Make `QueryParser.parse_date` handle gaps and ambiguities gracefully
* Replace `Timex.now(tz)` with `DateTime.now!(tz)`
* Use a more suitable Date function for comparison (h/t @aerosol)
* Revert "Revert "Trim `month`, `year`, `day` periods to local now on main graph (#5668)" (#5684)"
This reverts commit 2d11681f25.
* Does not trim for comparisons
* Include the current hour in the trimmed time range
* Trim `month`, `year`, `day` periods to now on main graph
* Revert "Trim `month`, `year`, `day` periods to now on main graph"
This reverts commit 4f3930111d3a2737a51686e067d9b64f0d85ad58.
* Re-implement trimming in query optimizer instead
* Update JS types
* This is getting confusing
* Trim in stats_controller
* Set `query.now` based on query_praser and date results
* query.period -> query.input_date_range
* Changelog
* Test for response query.date_range
---------
Co-authored-by: Karl-Aksel Puulmann <oxymaccy@gmail.com>
* Leverage TrackerScriptCache on ee
On ee, TrackerScriptCache only stores valid ids. This is then leveraged
to do no database queries when looking up tracker scripts for
non-existing ids.
For smoother onboarding purposes, refresh frequency for the script is also
reduced.
Note that the cache layout is not optimal (storing 'true' booleans) but
being more optimal would require changing the underlying cache
implementation significantly.
I tested out the cache - with 1M tracker script configs, it seems to be
~12MB in size.
* Wait on cache
* Add telemetry
* Remove cleverness in trying to reuse code
* streamline dev subscriptions
* spawn -> Task
* fix button alignment
* do not define routes in test env
* remove explicit mentions of dev env when deciding whether to sandbox paddle
* decide checkout behaviour at compile time
* Timex -> Date
* hum
* make it work for enterprise plans too
* allow convenient subscription status change
* fix ci warnings
* remove redundant commented out code
* fix crash + review suggestion
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Make `site_role/2` return whether real is team or site scoped
* Rename `has_admin_access?` => `has_editor_access?`
* Switch current team if user is a team member in site's team
* CSV import/export support for time-on-page
Note only the new time-on-page metric is exported this way
* visibility check for graphing of time_on_page
* FE no longer receives/sends legacy_time_on_page_cutoff
* Remove current_user from exports
* Remove legacy_time_on_page_cutoff from query.include, make behavior work off of site.legacy_time_on_page_cutoff explicitly
* Remove dead function
* More current_user_id removals
* Backend: get rid of scroll_depth.ex
This commit starts treating the scroll depth feature as *always visible*,
removing the ClickHouse queries on dashboard loads and starting to always
include scroll_depth in dashboard reports and export queries.
* Frontend: remove site.scrollDepthVisible
* adjust csv_importer_test
* remove site.scroll_depth_visible_at
* Replace GET /segments and GET /segments/:segment_id with server-rendered list, fix issue with dashboard report columns
* Remove WIP comments and throw earlier for invalid dashboard state
* Fix Segments details issue on public / shared link sites, add tests
* Support passing `include` as a query parameter for dashboard APIs
* Mark time-on-page metric sortable
It now is thanks to the changed query
* new-time-on-page flag with cutoff being sent to the frontend
* Add correct tooltip title
* Implement metric warning for when legacy and new time_on_page metrics are mixed
* Send legacy_time_on_page_cutoff to backend
* Make time-on-page graphable with the new metric
* Only show metric warnings for time_on_page if flag is enabled
* Changelog
* Solve an clickhouse error when querying timeseries with only legacy time-on-page
* Add tests for timeseries of new time-on-page
Along the way fix an issue with comparisons not working properly
* Solve a typing issue
* Allow toggling legacy_time_on_page_cutoff off in dashboard
* Slightly better workaround
* Solve typing issue
* Prettier
* Guard against no warning
* Solve warning
* Default to time_on_page
* Add new columns to schema
* Read from new column in legacy query
* Read/write new imported_pages columns
* Remove time_on_page column from imported_pages
* Simple, stupid new_time_on_page metric
* Update csv_importer schema
* Refactor: consistent __internal helpers, this will help with joining the query
* Refactor select_joined_metrics
* Refactor: pass `query` to event_metric
* Refactor: remove needless site argument from various calls
* Legacy joining query attempt
* Move test around
* Add more tests for both legacy and new time_on_page metrics in query API
* time_on_page reported in seconds
* timeseries test for metric
* WIP
* Wrap main query in subquery - without this run into trouble performing the join
* Calculate time_on_page in main query, no more new_time_on_page
* Add some TODOs
* Return NULL over 0 when no visits with time-on-page data
* Update moduledoc
* Update some tests that were not expecting integers
* Add a TODO
* Update tests
* Make graphing time series with combined metrics work.
* Slightly more consistent approach to flag updating in APIv2
* Seeds with engagement data
* Make graphing time series when cutoff is in the middle work
Bakes less assumptions into everything as well.
* Rename to legacy_time_on_page_cutoff
* Fixup lib/plausible_web/controllers/api/external_query_api_controller.ex
* Remove a todo and dead/misleading code
* Remove a resolved todo
* Remove needless rounding
* gen types
* Update pages test
* Remove needless columns from select
* Update tests: timestamps and remove comment
* Flip branches
* Main graph: Graph revenue metrics as zeroes instead of nils.
This has no effect on existing APIs as:
- Comparisons/empty data point logic only affects APIv1 which does not expose revenue metrics
Also changes semantics for visit_duration to be more in-line with the
rest.
* Improve dashboard main graph
- Graph tooltip for revenue metrics now shows revenue with the relevant
monetary unit.
- Graph tooltips for scroll depth, visit duration and so on now shows `-`
instead of 0 on dates where no relevant visits occurred.
* Remove business logic duplication
* Implement Site Segments billing feature
* Make sure Site Segments benefit appears only when feature flag is present
* Add site_segments flag to v3 plans
* Refactor new filter menu back to ghost button
* Stop error state from persisting (#5131)
* Refactor how Segments appear on Choose Plan page
* Limit to 500 segments per site
* Resolve segment owner names and dates in site timezone in the BE
* Fix SegmentAuthorship
* Fix issue with updating dangling segment
* Refactor flaky PATCH segment test
* Unify segment type label logic
* Refactor SegmentOwnership type definitions for clarity
* WIP
* Load members
* Assert that we know has_not_done will not work without changes
* Add tests
* Assert that dates are in the expected format
* Add tests, better authorship dates, api.js -> api.ts
* Add error panels
* Flatten errors on the API side
* Stop name copy from getting too long
* Make comparison mode and edit segment modes exclusive
* Fix flicker calculating space
* Fix issue with definite state not persisting
* Unhitch modals from query-context
* Separate API format and dashboard format of segment_data
* Clarify purpose of useDefiniteLocationState
* Tweak UI: site switcher, save as segment
* Fix issues with modals
* Remove commented and unnecessary code, better query context
* Fix too permissive site members dataset
* Make sure Segment doesn't show up as an option to customer without the FF
* Fix issue with 'See more' menu being present when it should not be
* Permit :has_not_done filter in segments
* Refactor to matching on filter list structure
* Flatten :and stemming from segment filters on first level
* Update test
* Add tests for `Teams.get_or_create/1` and `Teams.get_by_owner/1`
* Start populating `current_team` in assigns fetching value from session
* Clean up team passing in invitation services
* Make site transfer service handle multi-team scenario
* Handle multi-team and permission transfer errors on controller level
* Handle multi-teams in site creation on service and controller level
* Drop validation limiting full membership to a single team
* Make user deletion account for public team ownership
* Adjust feature availability checks for Stats API key
* Use current_team when determining limits on site transfer invitation
* Adjust trial upgrade email submission to account for multiple owners
* Remove unnecessary `Teams.load_for_site/1`
* Spike renaming `owner` and `ownership` relationships to plural versions
* Make HelpScout integration handle owner of multiple teams gracefully
* Add FIXME note
* Resolve paddle callback issue by always provisioning a new team when none passed
* Set `current_team` as `my_team` only when user is an owner
* Implement basics of Teams CRM
* Extend Teams CRM
* Further adjust User and Site CRM and refine Team CRM
* Convert Enterprise Plan CRM to refer to team directly and not via user
* Remove unused virtual fields from User schema
* Add note to HelpScout integration
* Allow listing multiple owners under Site Settings / People
* Remove unused User schema relations
* Fix current team fetch in auth plug and context
* Implement basic team switcher
* Ensure (site) editor role is properly handled in site actions auth
* Don't set `site_limit_exceeded` error marker on `permission_denied` error
* Link from HS integration to Team CRM instead of User CRM when available
* Ensure consistent ordering of preloaded owners
* Add `with_subscription` preload for optimisitation
* Add ability to search sites by team identifier
* Add ability to pick team when transferring ownership directly
* Fix failing HelpScout tests
* Scope by team when listing sites in dashboard and via API (optional)
* Add ability to search by team identifier in plans CRM lookup widget
* Add subscription plan, status and grace period to team status info
* Expose teams list in user CRM edit form and fix team details CRM view
* Fix Team Switcher styling
* Reorganise header nav menu
* Avoid additional queries when authenticating user
* Hide the pay/site transfer message on lock screen when teams FF is on
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* write comparison_utc_time_range into query
* write site_imports into query
* use struct! over Query.set
* variable to module attr
* remove redundant comment
* reject over filter
* Map.reject over Enum.reject
* Separate liveness and readiness checks
* Changelog
* Make sure /api/health uses readiness check
* Do not expose internal error
* Make healthcheck repo calls parallel
* add migration
* add schema field
* mark site_imports with has_scroll_depth
* add function to get imports in query range
* add scroll_depth metric warning in QueryResult
* return scroll_depth warning in top stats
* render minimalistic warning in top stats
* minimalistic warning in Top Pages breakdown
* prettier format
* silence credo
* add test
* use a snapshot of SiteImport schema in data migration
* also use a snapshot list of imported_* tables
* moduledoc (credo)
* change tooltip message
* change metric warnings structure in top stats response
* pass meta from queryresult directly
* revert top_stats_entry refactor
* prettier
* stop using SiteImport module in data migration
* add migration
* move scroll_depth_enabled? fn
* maybe set engagement_metrics_enabled_at when requesting dashboard
* maybe set engagement_metrics_enabled_at in shared_link action
* maybe set engagement_metrics_enabled_at on full export
* fix tests
* feature gate scroll depth on the dashboard with site.engagement_metrics_enabled_at
* feature gate scroll depth in full export too
* fix npm ci
* Rename things into FE, remove unneccessary flag checks
* Continue with renaming
* Rename site flag to be more descriptive
* Move business logic, calculate based on scroll depth, make more precalculatable
* Some docs
* Rename to site.scrollDepthVisible in frontend
* Update migration
* Fix template
* Remove boilerplate from tests
* Update tests
* More straight-forward test
* Update condition
---------
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* preload completed site_imports for site when needed
* remove unused field from top stats response
* refactor imports meta
* remove redundant function clause
* fix ordering in queryresult
* preload completed_imports in plugs
* write code more short
* Expose site_id and site_native_stats_start_at via query
This allows to do more query-building without exposing and passing `site` directly.
* Very basic has_done/has_done_not operator support
No event:goal support yet, no validations
* Add validations that only event: dimensions can be used within has_done/has_done_not
* Allow event:goal filters nested within has_done/has_done_not behavioral filters
* Minor fix for do_decide_custom_prop_table
* has_done support for goals
Minor changes along the way:
- preloaded_goals structure changes
- event:goal restrictions were loosened within has_done
- we don't allow nesting has_done anymore
* Dont query imports when behavioral filters are present
* Update callsites of filtering_on_dimension? to work with new behavioral filters
* has_done_not -> has_not_done
* Changelog entry
* Typegen
* credo cleanup
* Fix changelog
* Remove changelog
* Mark has_done as internal-only
* combine two validations into a single loop
* has_done is now session-based not user-based
* Update a test
* Update transform_tree
* Implement segments endpoint controllers
* Refactor endpoint controllers to be based on capabilities
* Add saved_segments feature flag check
* Fix issue when querying all segments on Growth plans
* Validate segment
* Refactor naming to permissions
* Add some permission based tests
* Give more info to compiler about expected args shape
* Implement centralized permission checking module
* Refactor to conventional rights validation
* Update tests
* Add name and segment_data length and filter depth validation
* Refactor route names and add tests for FF check plug
* Refactor mixed segments query name
* Remove inconsistent strict not equals
* Use a different format for result is Map case
* Remove newline between spec and def
* Fix issue with too deep nesting
* Verify in tests that segment is actually removed/not removed from DB
* Fix typo in fn name
* include scroll_depth in full pages export
* import scroll_depth from CSV
* query scroll depth from imported data
* fix ordering by scroll depth with imported data
* fix imported scroll depth query + more tests
* enable scroll depth in top stats with imported data
* add main graph test
* fix test and native scroll depth sum select
* Update lib/plausible/exports.ex
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
* adjust test
* adjust test to catch error
* export/import/count pageleave_visitors
* extract base_q in export_pages_q
* rename total_visitors to pageleave_visitors
---------
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
* Comment out legacy fields and relationships
* WIP
* WIP 2
* WIP 3
* wip
* Remove teams backfill and consistency check scripts
* WIP 3
* Fix CheckUsage tests
* Update billing/subscription tests
* WIP 4
* Make site transfer fail if some invitation already exists
* Fixup: do symmetric invitation/site transfer check
* Update UI bugs: make listing sites/inviting admins work like before
* Fix Sites test
* Fix external sites controller test
* Fix live sites tests
* Fix props availability lookup
* Fix site controller tests
* Fix billing controller tests
* WIP - accept invitation tests
* Another round of test fixes + invitations logic bugs
* users_test -> teams_test
* Update registration via invitation
Here, we still rely on "polymorphic" invitation structures,
hence the "unified by id" helper.
For now, it'll remain local unless we discover it's
needed in the broader `Teams.Invitations` context.
cc @zoldar
* Yet another round of test and bugfixes along the way
* Include team in site setup success e-mail
* Fix send_site_setup_emails worker
* Fixed almost all tests except CRM ones
* Update enterprise plan admin test
* Fix CRM + remaining tests
* Address credo warnings (modulo one FIXME)
* Remove last FIXME and rephrase the invitation test case description
* Set Team fields via User CRM transparently
* Map user reference in Enterprise Plan CRM via team owner
* Fix resource actions in user CRM
* Get rid of warning when opening create form in API keys CRM
* Stop emitting warnings when editing Enterprise Plans via CRM
* Tests: Bump await_clickhouse_count interval
* Remove XXX marker
* Fix register from invitation link in email sent for ownership transfer
* Simplify fetching all pending site ownership site IDs
* Remove commented out schema fields
* Remove unused functions
* Address flakiness in ingest counter tests
* Remove unused `Teams.Sites.create`
* Don't restart trial on team with subscription when creating site
* Account for cases of legacy teams with empty trial expiry date
* Revert "Address flakiness in ingest counter tests"
This reverts commit 60dc1e4115.
* Fix flaky ingest counters tests under load
* Attempt 2
* Pre-emptively hardcode site ids in sampling cache test
to avoid supplying the same IDs alongside with counters test,
that inserts through another repo (async).
what we're observing is, clickhouse not summing mergetree columns fast
enough, even though we wait quite a bit.
* Fix ingest counter tests by accounting for delayed summation
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* WIP: Optional modifiers to queries
* WIP: Modifiers v2
* Use preloaded_goals when determining whether imports can be included
This was previously broken with conversion_rate totals metrics since it removed event:goal
filters but did not update preloaded_goals
* Preload goals according to modifiers
* Make case_sensitive: false work for is/contains operators
* Make modals send { case_sensitive: false } to backend for search
* CHANGELOG.md
* Typegen
* Prettier
* Refactor: more DRY where_builder for case sensitivity
* Support case_sensitive modifier for is_not/contains_not
* Cleanup
* credo
* remove defaults
* negating a previously set filter
* migration: add scroll_depth to events_v2
* (cherry-pick) ingest scroll depth
* replace convoluted test with more concise ones
* QueryParser: parse internal scroll_depth metric + validation
* turn QueryComparisonsTest into QueryInternalTest
* rename file
* (cherry pick) query scroll depth 15b14d3
...and move the tests into `internal_query_test.exs`
* review feedback
* Get rid of unnecessary separation between aggregate and group scroll depth
* Drop irrelevant other metrics in tests
* add test ensuring scroll depth unavailable in Stats API v1
* Put scroll depth on the dashboard
* Top Stats
* Main Graph
* Top Pages > Details
* feature flag for dashboard scroll depth access
* ignore credo warning
* enable scroll_depth flag in tests
* remove duplication
* write timestamps explicitly in a test
* revert moving tests around
* Add query_comparisons_test back
* Move scroll_depth tests into query_test
* Delete query_internal_test
* rename setup util (got updated on master)
* use pageleave_factory where applicable
* Use the correct generated query-api.d.ts
* npm format
* Revert "Revert "Remove no change icon, better alignment""
This reverts commit a69d0a14a7fe038fe110907571240ad99a2ba6a4.
* different arrows
* Render graph tooltips in react
* Use ChangeArrow in graph tooltips
* Only display a single date if viewing a single date
* Tests for breakdown responses with labels
* Simplify code a bit
* strokeClass
* refactor ChangeArrow
* Play with stroke widths
* jest tests
* Silence logging in jest tests
* not variable stroke width
* prettier
* Conversion rate widths
* Update a test
* Trim trailing dates for label when comparing running month
* Reformat file
* Update expect.toHaveTextContent for multiline
* Update date range labelling to be more consistent with the frontend
* reformat
* new endpoint for custom prop value filter suggestions
* add test coverage for searching prop suggestions
* plug the new endpoint into FE
* update changelog
* format JS
* Update CHANGELOG.md
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
---------
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
RobertJoonas
Adam Rutkowski
Adrian Gruntkowski
Uku Taht
Karl-Aksel Puulmann
ruslandoga
Artur Pata