* Bump deps
* Bump stack
* Fix deprecation warnings
* Fix VCR cassettes mismatch due to OTP-18414
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Format & fix flaky tests
* Handle raw IPv4 hostnames; test public suffix TLD
* Configure locus db cache_dir
So that maxmind unavailability doesn't affect
application startup. PERSISTENT_CACHE_DIR env var is used
to point locus at the GeoIP DB file.
* WIP: Remove ExVCR
* Fix test env config
* Fixup exvcr
* Remove exvcr from deps
* Add convert script
* Remove exvcr cassettes
* Remove convert script
* Rename test
* Update moduledoc
* Update dockerfile
* Bump CI cache
* Tag more slow tests, why not?
* Use charlist for locus cache option
* Pin nodejs
* Merge google tests, make them async
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
This pull request implements limits to funnels, revenue goals and custom props based on the site owner plan. It extends the current "premium feature" notice to account for the new plans, trials and the on-going private preview. Stats API is not in the context of this pull request, but will be implemented likewise.
* rename enterprise?/1 function
* change link text to Upgrade when subscription deleted
* extract paddle_button and paddle_script components
* create a new upgrade-to-enterprise-plan page
* extract upgrade_link component
* rename function
* link to enterprise plan upgrade page from settings
...if the user has an enterprise plan configured
* fetch enterprise plan price on the new page
* add change_enterprise_plan functionality on the new page
* render existing change_enterprise_plan_contact_us.html
...when subscribed to latest configured enterprise plan
* rename vars and extract resumable? fn
* remove dead billing route
* small test refactor: extract convenience fn
* add tests for...
...restricting paused and past_due subscription access to the new
enterprise plan page.
1. redirect to /settings from the controller action
2. hiding the change-plan link from the user settings
* implement redirect to /settings
* hide the enterprise upgrade/change-plan link
* add tests for a deleted enterprise subscription
* plug in the new controller action and delete dead code
* optimize for dark mode
* fix compile warning
* credo fix
* display N/A instead of crash when price nil
* change subscription.status type to Ecto.Enum
Also, create a new `Subscription.Status` module that exposes macros to
return the used atom values (prevent typos at compiletime).
* fix bug (@conn not available anymore)
* use Routes.billing_path where applicable
* add a status() type
* silence credo
* refactor suggestion from review
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Remove the __using__ macro from Subscription.Status
... instead be explicit about requires and aliases and also order
the use, import, require, and alias clauses according to
https://github.com/christopheradams/elixir_style_guide#module-attribute-ordering
* drop the virtual Enteprise 'price_per_interval' field
* apply review suggestion to make the code more DRY
* use dot syntax to fetch current user in new controller actions
* fix formatting
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
* Add Heroicons dependency
* Add name_of/1 html helper
Currently with Floki there's no way to query for
`[name=foo[some]]` selector
* Update changelog
* Make goal deletion possible with only goal id
* Remove stale goal controllers
* Improve ComboBox component
- make sure the list options are always of the parent input width
- allow passing a suggestion function instead of a module
* Stale fixup
* Update routes
* Use the new goals route in funnel settings
* Use a function in the funnel combo
* Use function in the props combo
* Remove old goals form
* Implement new goal settings
* Update moduledoc
* Fix revenue switch in dark mode
* Connect live socket on goal settings page
* Fixup
* Use Heroicons.trash icon
* Tweak goals search input
* Remove unused alias
* Fix search/button alignment
* Fix backspace icon alignment
* Delegate :superadmin check to get_for_user/3
I'll do props settings separately, it's work in progress
in a branch on top of this one already. cc @ukutaht
* Rename socket assigns
* Fixup to 5c9f58e
* Fixup
* Render ComboBox suggestions asynchronously
This commit:
- prevents redundant work by checking the socket connection
- allows passing no options to the ComboBox component,
so that when combined with the `async` option, the options
are asynchronously initialized post-render
- allows updating the suggestions asynchronously with the
`async` option set to `true` - helpful in case of DB
queries used for suggestions
* Update tests
* Throttle comboboxes
* Update tests
* Dim the search input
* Use debounce=200 in ComboBox component
* Move creatable option to the top
* Ensure there's always a leading slash for goals
* Test pageview goals with leading / missing
* Make the modal scrollable on small viewports
This pull request introduces a series of improvements to Plausible.Billing.Plans, including:
* Tag the JSON file with the plan version
* Rename the JSON field limit to monthly_pageview_limit
* Move site_limit function to Billing.Plans
* Refactor subscription_interval, allowance and site_limit functions
* Remove unused AnalyzePlans task
* Update formatter config
* Install LiveView JS integration & hooks
* Temporarily update endpoint/session config
* Optionally allow preloading funnels for goals
* Site controller
* Implement funnel settings
lib/plausible_web/live/funnel_settings/combo_box.ex - restored from:
054de6e2 Fix the tab/blur bug again
20da4c89 Rename InputPicker to ComboBox
lib/plausible_web/live/funnel_settings/form.ex - restored from:
9bedda3b Remove potential FIXME
20da4c89 Rename InputPicker to ComboBox
028036ad Review comments
aea4ebc4 Access Funnel min/max steps via the __using__/1 macro
0dde27fd Remove inspect call
eed588a7 Start testing the funnel editor
0e95228b Extract funnel settings test module
7b16ace5 Leverage aplinejs to deal with the tyranny
8dc6a3e7 wip
cf228630 wip
30a43fd1 wip
89f10ecb wip
950a18d9 Dirty funnel save
298a6a53 wip
7690d50f wip
639c6238 fixup
aa59adeb wip
ff75c00b wip
lib/plausible_web/live/funnel_settings/list.ex - restored from:
4eae122c Fix data-confirm attr interpolation
51f0397d Implement deleting funnels
1f6fe25d Add number of steps to funnels list
298a6a53 wip
ff75c00b wip
test/plausible_web/live/funnel_settings/funnel_settings/combo_box_test.exs - restored from:
20da4c89 Rename InputPicker to ComboBox
test/plausible_web/live/funnel_settings/funnel_settings_test.exs - restored from:
34822ff4 Bootstrap InputPicker tests
lib/plausible_web/live/funnel_settings.ex - restored from:
028036ad Review comments
acd9c4f2 Prepare ephemeral funnel definitions so that users can test funnels
51f0397d Implement deleting funnels
0e95228b Extract funnel settings test module
8dc6a3e7 wip
89f10ecb wip
950a18d9 Dirty funnel save
298a6a53 wip
aa59adeb wip
ff75c00b wip
test/plausible_web/controllers/error_report_controller_test.exs - restored from:
34822ff4 Bootstrap InputPicker tests
test/support/html.ex - restored from:
0a53979d Improve InputPicker tests - include AlpineJS assertions
34822ff4 Bootstrap InputPicker tests
lib/plausible_web/views/layout_view.ex - restored from:
b490403b !ifxup
lib/plausible_web/templates/site/settings_funnels.html.eex - restored from:
51f0397d Implement deleting funnels
ea1315f3 Test funnels list in settings
7b16ace5 Leverage aplinejs to deal with the tyranny
ff75c00b wip
4da25c35 Fixup
lib/plausible_web/templates/layout/app.html.eex - restored from:
ff75c00b wip
* Add funnel settings route
* Warn about funnels deletion when deleting goals
lib/plausible_web/templates/site/settings_goals.html.eex - restored from:
fdd9bcd0 Fixup
f1e6364d Merge remote-tracking branch 'origin/master' into funnels-rebase
9d0b7c6d Fix markup error
4a4ddbdc Optionally preload funnels for goals and stub funnel-goal deletion
ebdc4333 Extend the prompt in case of funnel-goal deletion
639c6238 fixup
aa59adeb wip
* Split new JS LiveView additions
* Put funnels behind a feature flag
* Integrate dashboard feature toggle
* Update signing salt for live view
* Update moduledocs
* Update live reloader config
* Use Phoenix.HTML.Safe for goal names
* Workaround to get flashes working in embedded liveview
* Keep feature toggles idempotent, rename property to setting
We'll still retain the ability to flip bools on a lower level.
* Update moduledocs
* Make live flash disappear after 5s
* Tailwind: purge .heex files too
* Update docs link
* Add live components to tailwind purge config
* Update another flaky test
Ref f0bdf872
cc @vinibrsl
* Fix combobox input length w/ WebKit
* Intoduce generic notice component
* Revert "Fix combobox input length w/ WebKit"
This reverts commit 3c653a6d85d5000167631e10ef45a93c13b41ed1.
* Fix combobox input length on webkit
* Make whole combobox item clickable, not only text
* Fix glitch moving Save button on activation
* Tweak dark mode
* Show funnel form without waiting for funnel name input
* Tweak dark mode
* Include static Phoenix components in tailwind purge
* Tune funnels form into a liveview of its own
This is so that ComboBoxes can publish their selections
and unavailable choices can be propagated to other siblings.
* Push less data over websocket
* Undo Lsp/formatter race condition
* Fixup typespecs
* Bust CI cache
* copy relevant files from b2ace16540
* make it work and set site.funnels to empty list
* make Behaviours a functional component
* add UI for a setup hint and always display conversions by default
* cherry-pick migration commit
* update site schema with new fields
* backend: implement disable-feature action
* switch between tabs in the behaviors section
Introduces template components to build props and funnels on. Both
only show a setup notice atm, and both are behind feature flags.
* extend API for disabling props and funnels
* render feature setup note directly from the Behaviours component
* fix UI behavior when features are hidden
* update setup notices
* add conversions feature switch to Site Settings > Goals
* mix format
* remove IO.inspect
* change setup notice - use buttons + popup confirmation
* optimize for light mode
* restrict access to setup notices
* some styling improvements
* allow super-admins to enable/disable features
* only show conversions (last 30min) in realtime mode
* use shorter display names for tabs
* optimize for mobile screens
* note about sending custom events
* changelog update + fix CI
* change HTTP verb for the disable-feature action
* change UI label for show/hide goals
* Add revenue goal option to goal creation
This commit adds a currency field to the goals form. Goals that have a
currency set are now revenue goals, and are cached with sites to later
be used during ingestion.
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Enable feature flag in tests
---------
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* upgrade phoenix
Co-authored-by: Vini Brasil <vini@hey.com>
* fix a test (flash message)
The flash message in focus.html.eex was not covered by any test. This
commit fixes also fixes that.
* change function name
* remove unnecessary formatter and format
* update CI cache
* fix dialyzer error
---------
Co-authored-by: Vini Brasil <vini@hey.com>
* Full migration (to be submitted separately)
* Do not remove `Goal.domain` just yet
* Do not make Goal.site not nullable just yet
* Temporarily disable goal creation
* Rephrase error message
* Add down migration
* Migration (PR: https://github.com/plausible/analytics/pull/2802)
* Implement Site.Domain interface allowing change and expiry
* Fixup seeds so they work with V2_MIGRATION_DONE=1
* Update Sites.Cache so it's capable of multi-keyed lookups
* Implement worker handling domain change expiration
* Implement domain change UI
* Implement transition period for public APIs
* Exclude v2 tests in primary test run
* Update lib/plausible_web/controllers/site_controller.ex
Co-authored-by: Vini Brasil <vini@hey.com>
* Update lib/plausible_web/controllers/site_controller.ex
Co-authored-by: Vini Brasil <vini@hey.com>
* Update moduledoc
* Update changelog
* Remove remnant from previous implementation attempt
* !fixup
* !fixup
* Implement domain change via Sites API
cc @ukutaht
* Update CHANGELOG
* Credo
* !fixup commit missing tests
* Allow continuous domain change within the same site
---------
Co-authored-by: Vini Brasil <vini@hey.com>
* Remove ClickhouseSetup module
This has been an implicit point of contact to many
tests. From now on the goal is for each test to maintain
its own, isolated setup so that no accidental clashes
and implicit assumptions are relied upon.
* Implement v2 schema check
An environment variable V2_MIGRATION_DONE acts like
a feature flag, switching plausible from using old events/sessions
schemas to v2 schemas introduced by NumericIDs migration.
* Run both test suites sequentially
While the code for v1 and v2 schemas must be kept still,
we will from now on run tests against both code paths.
Secondary test run will set V2_MIGRATION_DONE=1 variable,
thus making all `Plausible.v2?()` checks return `true'.
* Remove unused function
This is a remnant from the short period when
we would check for existing events before allowing
creating a new site.
* Update test setups/factories with v2 migration check
* Make GateKeeper return site id along with :allow
* Make Billing module check for v2 schema
* Make ingestion aware of v2 schema
* Disable site transfers for when v2 is live
In a separate changeset we will implement simplified
site transfer for when v2 migration is complete.
The new transfer will only rename the site domain in postgres
and keep track of the original site prior to the transfer
so we keep an ingestion grace period until the customers
redeploy their scripting.
* Make Stats base queries aware of v2 schema switch
* Update breakdown with v2 conditionals
* Update pageview local start with v2 check
* Update current visitoris with v2 check
* Update stats controller with v2 checks
* Update external controller with v2 checks
* Update remaining tests with proper fixtures
* Rewrite redundant assignment
* Remove unused alias
* Mute credo, this is not the right time
* Add test_helper prompt
* Fetch priv dir so it works with a release
* Fetch distinct partitions only
* Don't limit inspect output for partitions
* Ensure SQL is printed to IO
* Remove redundant domain fixture
* Fingerprint DBConnection.ConnectionError in Sentry
* Check events before creating a site
* Enable sites limit screen
* Remove debugging remnant
* Fix buggy assertions
This wasn't doing what expected:
iex(1)> Repo.exists?(Plausible.Site, domain: "foo")
[debug] QUERY OK source="sites" db=0.6ms idle=1906.2ms
SELECT TRUE FROM "sites" AS s0 LIMIT 1 []
* Encapsulate check to satisfy credo
* Use less technically involved error message
* Bring back e-mail to the limit error message
* Make TestUtils module available in all tests
* Add macros patching the application env in tests
Unfortunately a lot of existing functionality relies on
certain application env setup. This isn't ideal because
the app config is a shared state that prevents us from
running the tests in parallel.
Those macros encapsulate setting up new env for test purposes
and make sure the changes are reverted when the test finishes.
* Allow passing request opts to HTTPClient.post/4
We need this to swap custom request building in
Google Analytics import.
* Unify errors when listing sites
* React: propagate backend error messages if available
* React: catch API errors in Search Terms component
* Propagate google API errors on referrer drilldown
* Handle verified properties errors in SC settings
* Add missing tests for SC settings controller
* Unify errors for fetching search analytics queries (list stats)
* Unify errors refreshing Google Auth Token
* Test fetch_stats/3 errors and replace Double with Mox
* Fixup makrup
* s/class/className
* Simplify Search Terms display in case of errors
* Fix warnings
* List all Google Analytics views during import
This commit fixes a bug where different Google Analytics views with the
same name and URI were not shown. This was caused because GA views were
stored as a map, that naturally doesn't support duplicate keys.
This change updates the GA views list to display view IDs, making it
clearer to know what is being imported. The dropdown is now grouped by
website URL.
* Put Google Analytics API URLs in app env
* Add controller test to GA view list
* Add has_imported_stats boolean to Site
* Add Google Analytics import panel to general settings
* Get GA profiles to display in import settings panel
* Add import_from_google method as entrypoint to import data
* Add imported_visitors table
* Remove conflicting code from migration
* Import visitors data into clickhouse database
* Pass another dataset to main graph for rendering in red
This adds another entry to the JSON data returned via the main graph API
called `imported_plot`, which is similar to `plot` in form but will be
completed with previously imported data. Currently it simply returns
the values from `plot` / 2. The data is rendered in the main graph in
red without fill, and without an indicator for the present. Rationale:
imported data will not continue to grow so there is no projection
forward, only backwards.
* Hook imported GA data to dashboard timeseries plot
* Add settings option to forget imported data
* Import sources from google analytics
* Merge imported sources when queried
* Merge imported source data native data when querying sources
* Start converting metrics to atoms so they can be subqueried
This changes "visitors" and in some places "sources" to atoms. This does
not change the behaviour of the functions - the tests all pass unchanged
following this commit. This is necessary as joining subqueries requires
that the keys in `select` statements be atoms and not strings.
* Convery GA (direct) source to empty string
* Import utm campaign and utm medium from GA
* format
* Import all data types from GA into new tables
* Handle large amounts of more data more safely
* Fix some mistakes in tables
* Make GA requests in chunks of 5 queries
* Only display imported timeseries when there is no filter
* Correctly show last 30 minutes timeseries when 'realtime'
* Add with_imported key to Query struct
* Account for injected :is_not filter on sources from dashboard
* Also add tentative imported_utm_sources table
This needs a bit more work on the google import side, as GA do not
report sources and utm sources as distinct things.
* Return imported data to dashboard for rest of Sources panel
This extends the merge_imported function definition for sources to
utm_sources, utm_mediums and utm_campaigns too. This appears to be
working on the DB side but something is incomplete on the client side.
* Clear imported stats from all tables when requested
* Merge entry pages and exit pages from imported data into unfiltered dashboard view
This requires converting the `"visits"` and `"visit_duration"` metrics
to atoms so that they can be used in ecto subqueries.
* Display imported devices, browsers and OSs on dashboard
* Display imported country data on dashboard
* Add more metrics to entries/exits for modals
* make sure data is returned via API with correct keys
* Import regions and cities from GA
* Capitalize device upon import to match native data
* Leave query limits/offsets until after possibly joining with imported data
* Also import timeOnPage and pageviews for pages from GA
* imported_countries -> imported_locations
* Get timeOnPage and pageviews for pages from GA
These are needed for the pages modal, and for calculating exit rates for
exit pages.
* Add indicator to dashboard when imported data is being used
* Don't show imported data as separately line on main graph
* "bounce_rate" -> :bounce_rate, so it works in subqueries
* Drop imported browser and OS versions
These are not needed.
* Toggle displaying imported data by clicking indicator
* Parse referrers with RefInspector
- Use 'ga:fullReferrer' instead of 'ga:source'. This provides the actual
referrer host + path, whereas 'ga:source' includes utm_mediums and
other values when relevant.
- 'ga:fullReferror' does however include search engine names directly,
so they are manually checked for as RefInspector won't pick up on
these.
* Keep imported data indicator on dashboard and strikethrough when hidden
* Add unlink google button to import panel
* Rename some GA browsers and OSes to plausible versions
* Get main top pages and exit pages panels working correctly with imported data
* mix format
* Fetch time_on_pages for imported data when needed
* entry pages need to fetch bounces from GA
* "sample_percent" -> :sample_percent as only atoms can be used in subqueries
* Calculate bounce_rate for joined native and imported data for top pages modal
* Flip some query bindings around to be less misleading
* Fixup entry page modal visit durations
* mix format
* Fetch bounces and visit_duration for sources from GA
* add more source metrics used for data in modals
* Make sources modals display correct values
* imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration
* Merge imported data into aggregate stats
* Reformat top graph side icons
* Ensure sample_percent is yielded from aggregate data
* filter event_props should be strings
* Hide imported data from frontend when using filter
* Fix existing tests
* fix tests
* Fix imported indicator appearing when filtering
* comma needed, lost when rebasing
* Import utm_terms and utm_content from GA
* Merge imported utm_term and utm_content
* Rename imported Countries data as Locations
* Set imported city schema field to int
* Remove utm_terms and utm_content when clearing imported
* Clean locations import from Google Analytics
- Country and region should be set to "" when GA provides "(not set)"
- City should be set to 0 for "unknown", as we cannot reliably import
city data from GA.
* Display imported region and city in dashboard
* os -> operating_system in some parts of code
The inconsistency of using os in some places and operating_system in
others causes trouble with subqueries and joins for the native and
imported data, which would require additional logic to account for. The
simplest solution is the just use a consistent word for all uses. This
doesn't make any user-facing or database changes.
* to_atom -> to_existing_atom
* format
* "events" metric -> :events
* ignore imported data when "events" in metrics
* update "bounce_rate"
* atomise some more metrics from new city and region api
* atomise some more metrics for email handlers
* "conversion_rate" -> :conversion_rate during csv export
* Move imported data stats code to own module
* Move imported timeseries function to Stats.Imported
* Use Timex.parse to import dates from GA
* has_imported_stats -> imported_source
* "time_on_page" -> :time_on_page
* Convert imported GA data to UTC
* Clean up GA request code a bit
There was some weird logic here with two separate lists that really
ought to be together, so this merges those.
* Fail sooner if GA timezone can't be identified
* Link imported tables to site by id
* imported_utm_content -> imported_utm_contents
* Imported GA from all of time
* Reorganise GA data fetch logic
- Fetch data from the start of time (2005)
- Check whether no data was fetched, and if so, inform user and don't
consider data to be imported.
* Clarify removal of "visits" data when it isn't in metrics
* Apply location filters from API
This makes it consistent with the sources etc which filter out 'Direct /
None' on the API side. These filters are used by both the native and
imported data handling code, which would otherwise both duplicate the
filters in their `where` clauses.
* Do not use changeset for setting site.imported_source
* Add all metrics to all dimensions
* Run GA import in the background
* Send email when GA import completes
* Add handler to insert imported data into tests and imported_browsers_factory
* Add remaining import data test factories
* Add imported location data to test
* Test main graph with imported data
* Add imported data to operating systems tests
* Add imported data to pages tests
* Add imported data to entry pages tests
* Add imported data to exit pages tests
* Add imported data to devices tests
* Add imported data to sources tests
* Add imported data to UTM tests
* Add new test module for the data import step
* Test import of sources GA data
* Test import of utm_mediums GA data
* Test import of utm_campaigns GA data
* Add tests for UTM terms
* Add tests for UTM contents
* Add test for importing pages and entry pages data from GA
* Add test for importing exit page data
* Fix module file name typo
* Add test for importing location data from GA
* Add test for importing devices data from GA
* Add test for importing browsers data from GA
* Add test for importing OS data from GA
* Paginate GA requests to download all data
* Bump clickhouse_ecto version
* Move RefInspector wrapper function into module
* Drop timezone transform on import
* Order imported by side_id then date
* More strings -> atoms
Also changes a conditional to be a bit nicer
* Remove parallelisation of data import
* Split sources and UTM sources from fetched GA data
GA has only a "source" dimension and no "UTM source" dimension. Instead
it returns these combined. The logic herein to tease these apart is:
1. "(direct)" -> it's a direct source
2. if the source is a domain -> it's a source
3. "google" -> it's from adwords; let's make this a UTM source "adwords"
4. else -> just a UTM source
* Keep prop names in queries as strings
* fix typo
* Fix import
* Insert data to clickhouse in batches
* Fix link when removing imported data
* Merge source tables
* Import hostname as well as pathname
* Record start and end time of imported data
* Track import progress
* Fix month interval with imported data
* Do not JOIN when imported date range has no overlap
* Fix time on page using exits
Co-authored-by: mcol <mcol@posteo.net>
hq1
Vini Brasil
RobertJoonas
Adrian Gruntkowski
Marko Saric
Uku Taht