* rename Query.build -> Query.parse_and_build
* rename two test files and move 4 %Query{} building functions into subfolder
* rename StatsAPIFilterParser to LegacyStatsAPIFilterParser
* rename Filters.QueryParser to QueryParser
* turn QueryParserTest into QueryParseAndBuildTest
* move query_parser.ex out of filters directory
* separate build from parse
* disable sample_threshold in the new intermediate build function, for now
* remove now redundant test util functions
* remove unused import
* address todo from earlier
* credo
* Make module names in sync with paths in tests
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* fix weekly/monthly email reports + stats report tests
* fix title of stats email reports
* add missing test
* do not render review installation link for consolidated view traffic drop emails + test
* refactor email_match_query to predicate fn
* make site_id_query_filter a public fn
* make traffic spike notifications work + test
* improve spike/drop notification email copies
* fix ScheduleEmailReports introduction comment
* add tests for stats report email scheduling
* remove space before .
* apply credo suggestion
* fix CE tests, use module attrs for text assertions
* skip frequency comment
* single query
* rename to site_member
* single query for real
* Create static consolidated view UI on `/sites` page
- Improve existing site card layout
- Add static UI for the consolidated view
- Add dismissable upgrade card UI
- Extract favicon fetching logic to function
- Configure configurable fallback icon per route
- Add `/favicon/sources_globe/` route with different icon than `/favicon/sources/` to use on `/sites` page
* Improve the mobile view of the `/sites` page
* Minor query interface UX extension (#5713)
* Minor query interface extension
* !fixup
* !fixup
* Initial implementation of consolidated views on /sites
* Improve loading state
* no need to handle nil in main interface
* Juggle `can_manage_consolidated_view?`
* Require team setup in order to enable consolidated view
* nil catcher
* Fixup test fixture
* Don't show Consolidated View tab in CS if team is not setup
* Reorganize + test
* Remove comment
* Only show consolidated views to superadmins for now
* Remove temporary sleep
* CE unused bindings
* Clean up seeds
* EE
* Fixup test
* Test non-superadmin scenario
* Add a test guarding parity between small plots (consolidated vs individual)
* Move private function so CE won't complain
* See if the graphs are now similar at least :)
* sort
* Map keys are unsorted
* Ensure engagement events aren't counted as visitors on smol graphs
* just try and revert
* Revert "just try and revert"
This reverts commit 7584f59816.
* Simplify globe icon handling
* Remove unnecessary @rest
* Split tests into more focused cases
* Address jumpiness on furious refresh cycle
* Revert "Address jumpiness on furious refresh cycle"
This reverts commit 5c03b36918.
* Another attempt at jumpiness
* Enforce less noticeable lag applying the diff from loading to loaded
* Reduce flashing on stats load
---------
Co-authored-by: Sanne de Vries <sannedv@protonmail.com>
* exclude consolidated views when counting/listing sites
* Use Teams.owned_sites/1 now that it excludes consolidated views
* Until we figure out better - make sites.consolidated virtual for CE
* Safeguard ingestion, update tests for gate keeper/site cache
* Ensure consolidated sites are excluded from helpscout counts
* Test that consolidated views are excluded from site listings
* Update tests for owned sites to ensure consolidated views are excluded
* Update CRM tests to exclude consolidated sites
* exclude consolidated sites from quota usage
* filter out conslolidated sites from deleting events background job
* exclude consolidated (change domain background job)
* retrieve non-consolidated sites only by default
* Ensure Site struct defines a default for `consolidated`
* Make `sites.consolidated` column available in CE
* Add missing test to reminder e-mails, consolidation is irrelevant here
* Do not send setup help e-mail for consolidated sites
* Ensure no setup success email is ever sent for consolidated sites
* Skip consolidated sites in legacy time on page cut off
* Add minor test to Sites.get_for_user/2
* Extend get_for_user/get_for_user! options signature
* Move test dir up
* Write tests for Sites.get_by_domain
* Minor DRY
* fix typos
* Provide Site.regular() and ConsolidatedView.sites() for querying
* Use Site.regular() in Repo.get_by() calls
* Fix typespecs
* Update missing Site.regular() filters
---------
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Replace usages of `Timex.to_unix` with native API
* Wrap call to `Timex.is_valid_timezone?`
* Wrap calls to `Timex.today(tz)`
* Replace `Timex.today()` with `Date.utc_today()`
* Replace `Timex.now()` with `DateTime.utc_now()`
* Replace `Timex.compare` with `Date.compare`
* Wrap `Timex.diff` calls
* Replace `Timex.Timezone.convert` with `DateTime.shift_zone!`
* Wrap `Timex.parse!`
* Replace `Timex.to_date` with native API calls
* Replace `Timex.beginning|end_of...` with native API calls for Date
* Wrap `Timex.beginning|end_of...` for DateTimes and Dates for years
* Replace `Timex.format(!)` with native API calls
* Replace `Timex.to_naive_datetime` with native API calls
* Wrap time humanizing routines using Timex
* Remove unnecessary `use Timex` instances
* Replace `Timex.shift` with native API calls
* Make `QueryParser.parse_date` handle gaps and ambiguities gracefully
* Replace `Timex.now(tz)` with `DateTime.now!(tz)`
* Use a more suitable Date function for comparison (h/t @aerosol)
* Add custom props to full export
* Pass full `site` struct to `export_queries`
* Export only internal props if plan lacks custom props
* Add changelog entry
* Add spot check test for custom props
* Do not generate cartesian product of prop/value pairs 🤦
* change log level from info to notice on relevant logs
* bump log level to notice for everything except request logger
* format
* fix choose_plan_test.exs for good (starter tier launch)
* Revert "CRM: team (un)lock regardless of grace period (#5440)"
This reverts commit bb63c0d0e4.
* Implement grace period (un)lock just like in kaffy
* Test grace period handling
* Add `locked_by_admin` to teams schema
* Implement team locking
* Look up `teams.locked_by_admin` in relevant background services
* Add background servive tests
* Type faster than think!
* Purge bunny cdn cache when tracker script config is updated
* Only allow one purge task scheduled per site
Tested by adding logging and toggling a checkbox a few times. Only one job got executed.
* Typo
* add a new (feature flagged) upgrade page offering v5 plans
* include starter tier plans in available_plans_for + use dev prices in test
* upgrade page remodelling with starter tier
* mobile optimizations
* optimize for darkmode
* add embedded dashboards as a growth benefit
* do not hide header on LegacyChoosePlan
* consistent v5 plan feature order
* slight grandfathering notice adjustment
* display monthly price too on yearly plans
* default to v5 plans unlesss legacy? is true
* refactor: suggest volume not plan for emails
* align back link with page title
* render grandfathering notice for growth v4 too
* CSV import/export support for time-on-page
Note only the new time-on-page metric is exported this way
* visibility check for graphing of time_on_page
* FE no longer receives/sends legacy_time_on_page_cutoff
* Remove current_user from exports
* Remove legacy_time_on_page_cutoff from query.include, make behavior work off of site.legacy_time_on_page_cutoff explicitly
* Remove dead function
* More current_user_id removals
* Append `__team` parameter to URLs in notification e-mails
* Improve copy in guest to team member promotion email
* Make "return_to" set in redirect in `RequireAccountPlug` actually work
* Improve accepted site transfer e-mail phrasing and link
* site.legacy_time_on_page_cutoff migration
* Update migration logic
* Cronjob for setting legacy_time_on_page_cutoff
* Test site creation default legacy_time_on_page_cutoff
* Use site.legacy_time_on_page_cutoff
Leaving the query param logic for now for testing purposes
* tweak cron
* Work around defaults in tests
* Fixes
* Make cutoff date in job configurable - useful for backfilling
* change 7d and 30d periods in API v2
* change 7d and 30d periods in legacy API
* add 28d and 90d into the dashboard
* add tests
* add 28d and 90d to public API v2
* changelog
* npm run generate-types
* typo
* fix NPM test
* fix interval_test
* add P shortcut for last month
* hide last 30d from datepicker but keep keybind functional
* fix keybind hints
* Timex to Date + helpful comment
* prettier format
* adjust comment
* mention 30d -> 28d in changelog as well
* make period hidden param new default
* Get rid of `billable_user` in favour of `current_role`
* s/current_role/current_team_role
* Sort it out (tm)
* Remove unresolvable property from sentry context
* Test & adjust dashboard locked box
* Satisfy a test
* Update role enums with billing
* [wip] Ensure no team management can be done for roles other than admin/editor
* Track `current_role` at plug/mount level
* Some team role policy enforcement
* Guard remaining routes based on current team role
* Notify billing members via check usage worker
* Include billing members in trial/locked notifications
* Permissive team access policy when no current team assigned
Effectively the plug should no-op, so that existing user
notices/alerts can be displayed.
* Send cancellation notifications to billing role members
* Include billing members in renewal notifications
* Update moduledoc
* Exclude :guest role from team access auth plug
* Add tests for `Teams.get_or_create/1` and `Teams.get_by_owner/1`
* Start populating `current_team` in assigns fetching value from session
* Clean up team passing in invitation services
* Make site transfer service handle multi-team scenario
* Handle multi-team and permission transfer errors on controller level
* Handle multi-teams in site creation on service and controller level
* Drop validation limiting full membership to a single team
* Make user deletion account for public team ownership
* Adjust feature availability checks for Stats API key
* Use current_team when determining limits on site transfer invitation
* Adjust trial upgrade email submission to account for multiple owners
* Remove unnecessary `Teams.load_for_site/1`
* Spike renaming `owner` and `ownership` relationships to plural versions
* Make HelpScout integration handle owner of multiple teams gracefully
* Add FIXME note
* Resolve paddle callback issue by always provisioning a new team when none passed
* Set `current_team` as `my_team` only when user is an owner
* Implement basics of Teams CRM
* Extend Teams CRM
* Further adjust User and Site CRM and refine Team CRM
* Convert Enterprise Plan CRM to refer to team directly and not via user
* Remove unused virtual fields from User schema
* Add note to HelpScout integration
* Allow listing multiple owners under Site Settings / People
* Remove unused User schema relations
* Fix current team fetch in auth plug and context
* Implement basic team switcher
* Ensure (site) editor role is properly handled in site actions auth
* Don't set `site_limit_exceeded` error marker on `permission_denied` error
* Link from HS integration to Team CRM instead of User CRM when available
* Ensure consistent ordering of preloaded owners
* Add `with_subscription` preload for optimisitation
* Add ability to search sites by team identifier
* Add ability to pick team when transferring ownership directly
* Fix failing HelpScout tests
* Scope by team when listing sites in dashboard and via API (optional)
* Add ability to search by team identifier in plans CRM lookup widget
* Add subscription plan, status and grace period to team status info
* Expose teams list in user CRM edit form and fix team details CRM view
* Fix Team Switcher styling
* Reorganise header nav menu
* Avoid additional queries when authenticating user
* Hide the pay/site transfer message on lock screen when teams FF is on
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* include scroll_depth in full pages export
* import scroll_depth from CSV
* query scroll depth from imported data
* fix ordering by scroll depth with imported data
* fix imported scroll depth query + more tests
* enable scroll depth in top stats with imported data
* add main graph test
* fix test and native scroll depth sum select
* Update lib/plausible/exports.ex
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
* adjust test
* adjust test to catch error
* export/import/count pageleave_visitors
* extract base_q in export_pages_q
* rename total_visitors to pageleave_visitors
---------
Co-authored-by: ruslandoga <doga.ruslan@gmail.com>
* Comment out legacy fields and relationships
* WIP
* WIP 2
* WIP 3
* wip
* Remove teams backfill and consistency check scripts
* WIP 3
* Fix CheckUsage tests
* Update billing/subscription tests
* WIP 4
* Make site transfer fail if some invitation already exists
* Fixup: do symmetric invitation/site transfer check
* Update UI bugs: make listing sites/inviting admins work like before
* Fix Sites test
* Fix external sites controller test
* Fix live sites tests
* Fix props availability lookup
* Fix site controller tests
* Fix billing controller tests
* WIP - accept invitation tests
* Another round of test fixes + invitations logic bugs
* users_test -> teams_test
* Update registration via invitation
Here, we still rely on "polymorphic" invitation structures,
hence the "unified by id" helper.
For now, it'll remain local unless we discover it's
needed in the broader `Teams.Invitations` context.
cc @zoldar
* Yet another round of test and bugfixes along the way
* Include team in site setup success e-mail
* Fix send_site_setup_emails worker
* Fixed almost all tests except CRM ones
* Update enterprise plan admin test
* Fix CRM + remaining tests
* Address credo warnings (modulo one FIXME)
* Remove last FIXME and rephrase the invitation test case description
* Set Team fields via User CRM transparently
* Map user reference in Enterprise Plan CRM via team owner
* Fix resource actions in user CRM
* Get rid of warning when opening create form in API keys CRM
* Stop emitting warnings when editing Enterprise Plans via CRM
* Tests: Bump await_clickhouse_count interval
* Remove XXX marker
* Fix register from invitation link in email sent for ownership transfer
* Simplify fetching all pending site ownership site IDs
* Remove commented out schema fields
* Remove unused functions
* Address flakiness in ingest counter tests
* Remove unused `Teams.Sites.create`
* Don't restart trial on team with subscription when creating site
* Account for cases of legacy teams with empty trial expiry date
* Revert "Address flakiness in ingest counter tests"
This reverts commit 60dc1e4115.
* Fix flaky ingest counters tests under load
* Attempt 2
* Pre-emptively hardcode site ids in sampling cache test
to avoid supplying the same IDs alongside with counters test,
that inserts through another repo (async).
what we're observing is, clickhouse not summing mergetree columns fast
enough, even though we wait quite a bit.
* Fix ingest counter tests by accounting for delayed summation
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Populate `current_team` to site's team and make site and subscription preloads consistent
* Accept only full `User` struct in `Users.get_for_user(!)`
* Make all uses of `Sites.get_for_user(!)` switch on team schema
* Remove redundant preloads for funnel/props settings
* Use adapter transitions in subscription settings
* Use team's schema subscription when listing invoices
* Fix typespec
* Turn owned site IDs into a specific query
* Add clauses for when FF is on but no team has been created
* Fix formatting
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Clean site transfers after 48 hours
* Sync accepting site transfers and invitations within transaction
* Add dry run mode to teams backfill and make it a default
* Extend invitation clean worker tests
* Extend schemas with new fields and relationships for teams
* Implement listing sites and sites with invitations with teams
* Implement creating invitations with teams
* Implement accepting invites with teams
* Add `Teams.SiteTransfer` schema
* Implement creating ownership transfers
* Implement accepting site transfer between teams
* Make results shapes from `Teams.Memberships` role functions more consistent
* Remove :team relation from ApiKey schema
* Pass and provision team on subscription creation
* Pass and provision team on enterprise plan creation
* Implement creating site for a team
* Keep team in sync during legacy ownership transfer and invitations
* Resolve conflict in `Teams.get_or_create` without transaction
* Abstract `GracePeriod` manipulation behind `Plausible.Users`
* Put `User.start_trial` behind `Plausible.Users` API
* Sync team fields on user update, if team exists
* Sync cleaning invitations, updating and removing members
* Transfer invitations too
* Implement backfill script
* Allow separate pg repo for backfill script
* Rollback purposefully at the end
* Update backfill script with parallel processing
* Use `IS DISTINCT FROM` when comparing nullable fields
* Handle no teams to backfill case gracefully when reporting
* Parallelize guest memberships backfill
* Remove transaction wrapping and query timeouts
* Make team sync check more granular and fix formatting
* Wrap single team backfill in a transatction for consistent restarts
* Make invitation and site transfer backfills preserve invitation ID
* Update migration repo config for easier dev access
* Backfill teams for users with subscriptions without sites
* Log timestamps
* Put teams sync behind a compile-time flag
* Keep timestamps in sync and fix subscriptions backfill
* Fix formatting
* Make credo happy
* Don't `use Plausible.Migration` to avoid dialyzer complaining
None of the tooling from there is used anywhere and `@repo` can
be defined directly in the migration script.
* Drop SSL workarounds in the backfill script
---------
Co-authored-by: Adam Rutkowski <hq@mtod.org>
* Check usage for users with active/cancelled/past due subscriptions
So that over limit e-mails and grace period is handled
regardless - they're still subscribers.
* Ensure current subscription is fetched, for an ongoing plan
* Add extra test
* Revert "Ensure current subscription is fetched, for an ongoing plan"
This reverts commit 9ddc7bdccb.
* Fixup test
* Reapply "Ensure current subscription is fetched, for an ongoing plan"
This reverts commit 7ab537954b.
* Do a subscription preload and assert ID match
* Use inner lateral join to fetch subscriptions
* query.date_range is now in UTC instead of user timezone
This simplifies things down the line and fixes several bugs where
query.date_range is cast to naivedatetime for ecto purposes
Many places still remain broken:
- comparison queries
- `to_date_range` calls
* Make default_for_date_range not care about time zones
* Make timezone parameter mandatory for to_date_range
* Simplify utc_date_range, update legacy query builder
* Fix more cases where query date range is needed
* query.date_range -> query.utc_time_range
* Query.date_range/1 function
* ensure_include_imported update
* Clean up send_email_report
* Turn `Plausible.Auth.UserSession` into full schema
* Implement token based sessions and use them as default
* Ignore expired user sessions during retrieval from DB
* Implement plug bumping user session last used and timeout timestamps
* Implement Oban worker removing expired user sessions with grace period
* Implement legacy session conversion on touch, when applicable
* Update `UserAuth` moduledoc
* Extend `UserAuth` tests to account for db-backed session tokens
* Update CHANGELOG
* Add tests for `UserSessionTouch` plug
* Add test for `CleanUserSessions` worker
* Add logging of legacy session retrievals
* Use single update permitting stale records when touching user session
* Don't fetch session and user for external API endpoints (/api/event too)
* Refactor `Users.with_subscription/1` and expose helper query
* Skip fetching session in legacy `SessionTimeoutPlug`
* Rely on user session assign from `AuthContext` in `SentryContext`
* Silence legacy session warnings in `UserSessionTouchTest`
* Rely on session assign from `AuthPlug` in `SuperAdminOnlyPlug`
* Change `UserAuth` to get session, user and last subscription in one go
* Avoid refetching user session in `AuthorizeSiteAccess` plug
* Fix code formatting
* Refactor `UserAuth.get_user_token/1` (h/t @aerosol)
* Remove bogus empty opts from `scope` declarations in router
* Only touch session once an hour and keep `user.last_seen` in sync
* Bring back logging of legacy token use
* Migration: add installation meta
* Update site schema with installation meta
* Remove VERIFICATION_ENABLED env var
* Add context API to create/remove special goals
* Add context api to update installation meta
* Remove verification enabled check
* Update new progress flow definitions
* Update generic components
* Remove internal /status API
* Implement installation live view
* Update traffic change notifier link
* Update verification, no more modal
* Update routes
* Remove focus.html - will unify everything under app layout
* Fix broken link
* Update templates with focus_box mostly
* Update controller tests
* Update controllers and stop using the focus layout
* copy changes
* Update verification.ex
* Remove dead template
* Update settings_general.html.heex
* Update copy in tests
* Update installation.ex
* Remove dangling dot
* Fix link
* Update installation.ex
* Update installation.ex
* Better tooltips?
* Simpler labels
* Revert "Simpler labels"
This reverts commit 797560ef82f2067458b03b884be5aecc8fdc72bc.
* Add copy to clipboard link and fix snippet's dark mode
* Offer installation detection skip only if ws connected
* Put COPY link at the bottom with background
* Make tooltips link to docs
* Fix cherry-pick gone wrong
* Hide tooltips on mobile screens
* WIP: 404 tracking wizard
* Revert "WIP: 404 tracking wizard"
This reverts commit a9c9c79bbd.
* Update lib/plausible_web/live/components/verification.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Update lib/plausible_web/live/installation.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Use current_user from socket.assigns
* Update lib/plausible_web/live/installation.ex
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Use current_user from socket.assigns
* Use conn.private to steer verification tests
* Drop non-sticky tooltip in favour of component parametrization
Co-authored-by: Artur Pata <artur.pata@gmail.com>
* Reapply "WIP: 404 tracking wizard"
This reverts commit 3ba81671d7.
* Fix installation tests including 404 tracking
* Fixup the tooltip component
* Format
* Update installation.ex
* Put flash whenever installation option changes
* Use last known installation type on domain change
* Extract user flow definition to provide compile-time checks
* See if this helps running CE migrations successfully
* Use `styled_link` on registration/login views
* Don't crash when there's no conn.private carried over
* Format
* Push "Determining installation type" message a bit lower
* Use links and footer lists uniformly
This commit introduces a `<.focus_list/>` component
for rendering focus box footer links with colored
discs. It also equips generic link components
with the ability of sending non-GET requests
along with CSRF token, so we can apply uniform
styling and stop using legacy Phoenix link tags.
cc @zoldar @apata
* ws 👾
* Render more descriptive flashes on script config change
---------
Co-authored-by: Marko Saric <34340819+metmarkosaric@users.noreply.github.com>
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Co-authored-by: Artur Pata <artur.pata@gmail.com>
* Add data migration for creating and syncing location_data table and dictionary
* Migration to populate location data
* Daily cron to refresh location dataset if changed
* Add support for visit:country_name, visit:region_name and visit:city_name dimensions
Under the hood this relies on a `location_data` table in clickhouse being regularly synced with
plausible/location repo and dictionary lookups used in ALIAS columns
* Update queue name
* Update documentation
* Explicit structs
* Improve docs further
* Migration comment
* Add queues
* Add error when already loaded
* Test for filtering by new dimensions
* Update deps
* dimension -> select_dimension
* Update a test
* Expose current visitors 12h aggregate
* Remove unused site association
* Distinct drop/spike notification factories
* Rename modules accordingly + implement drop handling
* Rename periodic oban service
* Implement drop email
* Rest of the owl
* Update changelog
* Update moduledoc
* Update moduledoc
* Min threshold to 1
* Threshold 1
* Remove merge artifact
* Put panel behind a feature flag
* Format
* pull last_bill_date from paddle sandbox in mix task
* move cycle usage checks to Quota module
* move quota.ex to a subfolder
* split up Quota module
* set choose-plan pageview slider according to usage
* silence credo
RobertJoonas
Adam Rutkowski
Adrian Gruntkowski
Karl-Aksel Puulmann
ruslandoga
Uku Taht