* query.date_range is now in UTC instead of user timezone
This simplifies things down the line and fixes several bugs where
query.date_range is cast to naivedatetime for ecto purposes
Many places still remain broken:
- comparison queries
- `to_date_range` calls
* Make default_for_date_range not care about time zones
* Make timezone parameter mandatory for to_date_range
* Simplify utc_date_range, update legacy query builder
* Fix more cases where query date range is needed
* query.date_range -> query.utc_time_range
* Query.date_range/1 function
* ensure_include_imported update
* Clean up send_email_report
* add realtime date_ranges into the private API schema
This commit starts parsing date ranges into a new NaiveDateTimeRange
struct, rather than a simple Date.Range.
* transform realtime labels into negative integers + test
* move schema type argument to last position in helper functions
* allow passing a date param + tests
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Update test/plausible/stats/query_parser_test.exs
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* keep test file structure consistent
* Turn NaiveDateTimeRange into DateTimeRange
* change 'now' field from NaiveDateTime to DateTime in v2 query
* fix minute interval labels + add missing tests
* return query_result.date_range as iso8601 timestamps with timezone
* allow timestamps with tz as date_range arguments in API v2
* delete Plausible.Timezones.to_utc_datetime
* simplify returning comparison periods
* add comment about realtime not supported in comparisons
* pass only now instead of test_opts
* drop redundant else branch
* separate tests
* stick to a single check_date_range function in tests
* fix credo error
---------
Co-authored-by: Karl-Aksel Puulmann <macobo@users.noreply.github.com>
* Add data migration for creating and syncing location_data table and dictionary
* Migration to populate location data
* Daily cron to refresh location dataset if changed
* Add support for visit:country_name, visit:region_name and visit:city_name dimensions
Under the hood this relies on a `location_data` table in clickhouse being regularly synced with
plausible/location repo and dictionary lookups used in ALIAS columns
* Update queue name
* Update documentation
* Explicit structs
* Improve docs further
* Migration comment
* Add queues
* Add error when already loaded
* Test for filtering by new dimensions
* Update deps
* dimension -> select_dimension
* Update a test
* Use goals from postgres to build goal filter
* Remove unnecessary goal preloading
* Add contains filter for goals
* Make sure the correct imported tables are used when goals are in filters
* Remove 'contains' filter type for now
* Remove TODO comment
* Fix QueryParser test
* move goals.ex into a goals subfolder
* extract goal filtering logic into a separate module
* remove duplication from Imported.Base
* Extract `get_filter_goals` function
* Credo suggestions
* Credo ignore nesting
* apply suggestion - remove Sql.Util
* apply suggestion - pass imported? via opts
* remove duplicate function
* Uncomment tests
* Fix 500 error with goal suggestions
---------
Co-authored-by: Robert Joonas <robertjoonas16@gmail.com>
* Move fragments module under Plausible.Stats.SQL
* Introduce select_merge_as macro
This simplifies some select_merge calls
* Simplify select_join_fields
* Remove a needless dynamic
* wrap_select_columns macro
* Move metrics from base.ex to expression.ex
* Move WhereBuilder under Plausible.Stats.SQL
* Moduledoc
* Improved macros
* Wrap more code
* select_merge_as more
* Move defp to the end
* include.time_labels parsing
* include.time_labels in result
Note that the previous implementation of the labels from TimeSeries.ex was broken
* Apply consistent function in imports and timeseries.ex
* Remove boilerplate
* WIP: Limited support for timeseries-with-querybuilder
* time:week dimension
* cleanup: property -> dimension
* Make querying with time series work
* Refactor: Move special metrics (percentage, conversion rate) to own module
* Explicitly format datetimes
* Consistent include_imported in special metrics
* Solve week-related crash
* conversion_rate hacking
* Keep include_imported consistent after splitting the query
* Simplify do_decide_tables
* Handle time dimensions in imports cleaner
* Allow time dimensions in custom property queries
* time:week handling continued
* cast_revenue_metrics_to_money
* fix `full_intervals` support
* Handle minute/realtime graphs
* experimental_session_count? with timeseries
This becomes required as we try to include visits from sessions by default
* Support hourly data in imports
* Update bounce_rate in more csv tests
* Update some time-series query tests
* Fix for meta.warning being included incorrectly
* Simplify imported.ex
* experimental_session_count flag removal
* moduledoc
* Split interval and time modules
* Move fragments module under Plausible.Stats.SQL
* Introduce select_merge_as macro
This simplifies some select_merge calls
* Simplify select_join_fields
* Remove a needless dynamic
* wrap_select_columns macro
* Move metrics from base.ex to expression.ex
* Move WhereBuilder under Plausible.Stats.SQL
* Moduledoc
* Improved macros
* Wrap more code
* select_merge_as more
* Move defp to the end
* wrap_alias
* Revert "Revert "APIv2: Replace breakdown module with QueryBuilder (#4283)" (#4292)"
This reverts commit ef5e0e0382.
* Allow querying events and pageviews from sessions table
This is not strictly accurate, especially with shorter time frames, but
is useful for a fallback mechanism. I'll figure out something around
shorter time frames in the future.
See also: https://github.com/plausible/analytics/pull/4292
* Only query events and pageviews in legacy breakdowns
* WIP: Breakdown using QueryBuilder
* Revert "Remove problematic test"
This reverts commit b442bb5d1f.
* Get more breakdown tests passing
* Preload goals, sort when dealing with time_on_page
* Handle conversion_rate in breakdowns
* Simplify ordering by using selected_as consistently for dimensions
* Get breakdown tests passing
* Strings to atoms in keys for StatsController.transform_keys calls to work
* Handle revenue metrics removal
* Add test for nil-removal case
* Include percentage metric
* Fix and test with imported locations
* Fixup time-on-page
* Fix country/region automatic filters
* Handle multiple imports (os/browser version) in importsv2
* Filter goals
* Default to ordering by page as well
* Calculate conversion rate on sessions if needed
* Order by event dimensions - handles event:page special case
* Update tests
* Update more tests, handle goal=0 case in imports
* Handle event:goal breakdowns correctly with filters
* Revenue to money
* Improved table deciding
* Also update event:page filters on event:page breakdown
* bounce_rate to 0
Previous behavior relied on two queries being made - new query leads to 0 naturally
* Update pagination test
* dont count non-pageviews as path goal completions
* Make revenue logic breakdown-specific
Its hard to fit into the new schema and likely needs a rethink for apiv2
* Retain previous behavior for TimeSeries module
* Get GA4 test passing
Most failures are related to ordering, pageviews shouldnt be read off of sessions
* Clean up old methods
* Simplify imported.ex
* Dont crash on garbage filters
* Reflect ordering-related change in test
* Fix test data
* Update table_decider
* Re-simplify get_revenue_tracking_currency
* Revert revenue changes
* Use Query.set
* Remove a TODO
* csv importer: no pageviews
Pageviews were incorrectly fetched from sessions table before, causing issues
* csv importer tweaking
* Remove use Plausible
* to_existing_atom
* Revert "Revert "Multiple filters on the frontend (#4174)" (#4218)"
This reverts commit 2f7dcae991.
* Make city links work again
By enforcing everything sent to the BE is stringified. We do this at serialization-time to ensure old dashboard links still work
* Add new logic for query parsing
Note this is not yet final, but this scaffolding will soon be used
* Send filters to BE in new encoding, flag add more button
* query.property -> query.dimensions
* Reduce number of operations for filtering
This helps ensure we can be consistent/simple with apiv2
* Update search console filter mapping
* Update filters sent from frontend
* Update new filter parsing
* Remove redundant clause
* Make filtering by event:goal work
* Handle * as old backend did - prefix/suffix by **, if not using wildcards already
* Update imports logic
* Credo warning
* Spacing for add row button in filter modals
* query fix
* LegacyDashboardFilterParser
* only single hostname filter allowed
---------
Co-authored-by: Uku Taht <uku.taht@gmail.com>
* move imported.ex to imported subfolder
* move constructing base imported query into a separate module
* Implement imported table deciding and filtering
+ tests for pages, entry_pages, exit_pages and common filter types
* add top stats test with country filter
* add timeseries test
* Drop bounce_rate and time_on_page from imported & page-filtered Top Stats
* rename field returned by top stats
* turn pages into a fn comp
* Move dashboard API results under a results key
...and also return the skip_imported_reason to the frontend to be used
for displaying warnings.
* extend ListReport component with an optional afterFetchData prop
* turn Devices into a fn comp
* add not_requested as a skip_imported_reason
* display warning icons in the dashboard
* Implement filtering suggestions and translate filter fields for imported
* WIP
* Improve and cover filtering suggestions with tests
* Rename imported suggestions query helpers
* fix screen size breakdown with screen size filter
* support filtering by the same suggestion property
* support location filters when fetching location suggestions
* support filtering by multiple props from the same table
* Implement filtering by goals
* Make views per visit metric work for import entry and exit pages
* Get rid of circular dependencies between Stats.Imported and Stats.Imported.Base
* Clean up Query struct manipulation in Breakdown
* Rename helper function for clarity
* Automatically refresh query struct state after modifications
* Shutup credo
* display imported warning bubble in prop breakdown section
* Render warning bubble for funnels whenever imported data is in the view
* Transform any operator on respective goal filters
* Fix percentage and conversion_rate calculation in presence of custom props
* add tests for for combining page and pageview goal filters
* add skip_refresh option to query tweaking functions
* add imported CR support for timeseries
* still show url breakdown when special goal + url in filter
* rename Query.refresh
* use flat_map instead of map and concat
* fix darkmode color
* Handle invalid imported region codes in suggestions gracefully
* Add an entry to CHANGELOG.md
---------
Co-authored-by: Adrian Gruntkowski <adrian.gruntkowski@gmail.com>
Uku Taht
Karl-Aksel Puulmann
RobertJoonas
hq1
Adrian Gruntkowski