29868 Commits

Author SHA1 Message Date
Jim Ladd
b86d365dde collect job events based on job event modified time
* when collecting job events by creation time
  it is possible to miss events that were created
  at one point, but actually committed to the db
  much later.
* since events' modified time is set when they are
  committed to the db, we shouldn't miss any job events
* selecting job events by modified time wasn't possible
  beforehand because we didn't have an index for
  jobevent's modified field
2021-06-04 09:17:08 -07:00
Chris Meyers
4efbd45b3c add support for db partition in cleanup_jobs
Changes in old unpartitioned cleanup logic:
* Manually cascade delete events related to job(s)

(new partitions cleanup logic) For each event type:
* Get the event partitions that are within the cleanup date range
* Get a list of jobs to delete that are in the cutoff range.
* Jobs that are running, pending, or waiting in the job list are special.
* Use the special list to further filter the partition drop list.
* Drop partitions
* delete jobs
2021-06-04 09:17:08 -07:00
Jim Ladd
fb97687d14 lint 2021-06-04 09:17:08 -07:00
Jim Ladd
0f53d9b911 bump db partition migration 2021-06-04 09:17:08 -07:00
Jim Ladd
5a785798b0 is_partitioned should default to true when pk not set 2021-06-04 09:17:08 -07:00
Jim Ladd
14168297bd set event horizon to -1 for empty tables 2021-06-04 09:17:08 -07:00
Jim Ladd
7e1814e234 mock has_unpartitioned_events in collection tests
.. just like we do with the main awx tests
2021-06-04 09:17:08 -07:00
Jim Ladd
bdf11aa962 add migrations for Unpartitioned{Job}Event proxy models 2021-06-04 09:17:08 -07:00
Jim Ladd
46807205f8 Move created kwargs to right place 2021-06-04 09:17:08 -07:00
Jim Ladd
d749c172eb Ensure View.model exists before checking type 2021-06-04 09:17:08 -07:00
Jim Ladd
81db8091ea test updates
* when tests create a UnifiedJob and JobEvent,
  the two need to have the same value for job creation time
* some view validation was skipped due to `model` being
  a property in some cases now
2021-06-04 09:17:08 -07:00
Jim Ladd
5c1a33382c update mocks to reflect new migration
* instead of mocking `created_or_epoch` (which no longer exists)
* .. mock `Unified_Job.has_unpartitioned_events`
2021-06-04 09:17:08 -07:00
Jim Ladd
db6f565dca black formatting 2021-06-04 09:17:08 -07:00
Jim Ladd
6b4effc85a bump partition migration to 135 2021-06-04 09:17:08 -07:00
Jim Ladd
74a0c5bac5 Minor rebase fallout
* fix import
* remove dropped url
2021-06-04 09:17:08 -07:00
Jim Ladd
661cf0afb3 short-circuit event_processing_finished for wf jobs
* wf jobs are a wrapper for other jobs
* they do not process their own job events
2021-06-04 09:17:08 -07:00
Ryan Petrello
fbb74a9896 remove code that leaves behind old bigint tables on fresh installs
we don't need this code at all anymore - the bigint migration is long
gone, and anybody upgrading to this version of AWX has already
migrated their data
2021-06-04 09:17:08 -07:00
Ryan Petrello
200901e53b upgrade to partitions without a costly bulk data migration
keep pre-upgrade events in an old table (instead of a partition)

- instead of creating a default partition, keep all events in special
"unpartitioned" tables
- track these tables via distinct proxy=true models
- when generating the queryset for a UnifiedJob's events, look at the
  creation date of the job; if it's before the date of the migration,
  query on the old unpartitioned table, otherwise use the more modern table
  that provides auto-partitioning
2021-06-04 09:17:08 -07:00
Jim Ladd
0eddd5ce7f Enable partition pruning when fetching job's events 2021-06-04 09:17:07 -07:00
Jim Ladd
a7cabec3d0 bump partition migration to 0132 2021-06-04 09:17:07 -07:00
Jim Ladd
b98b3ced1c update tests to reflect new job_event schema 2021-06-04 09:17:07 -07:00
Jim Ladd
8501a45531 lint fixes 2021-06-04 09:17:07 -07:00
Jim Ladd
14b610dabf bump partition migration 2021-06-04 09:17:07 -07:00
Jim Ladd
a1d1e70e43 correct constraint name 2021-06-04 09:17:07 -07:00
Jim Ladd
0fa0a517ac create tmp schema to reference when creating partitioned table
* if we use the actual old job events table
  and make tweaks to its schema
  namely, dropping the pkey constraint,
  then when we go to migrate the old job events
  we will be forcing postgres to do a sequential scan
  on the old table, which effectively causes the migration to hang
2021-06-04 09:17:07 -07:00
Jim Ladd
28f9c0be0b Do not cascade delete job_events
* want to drop job_event _partitions_
  .. instead of having the job events associated with a job
     automatically get cleaned up for us
2021-06-04 09:17:07 -07:00
Jim Ladd
373edbf8c0 Update reference to partition migration 2021-06-04 09:17:07 -07:00
Ryan Petrello
b19bcdd882 remove the global /api/v2/job_events/ endpoint 2021-06-04 09:17:07 -07:00
Ryan Petrello
08b96a0bd7 correct filter events on the migration event horizon
events that existed *prior* to the partition migration will have
`job_created=1970-01-01` auto-applied at migration time; as such,
queries for these events e.g., /api/v2/job/N/job_events/
use 1970-01-01 in related event searche

events created *after* the partition migration (net-new playbook runs
will have `job_created` values that *exactly match* the related
`UnifiedJob.created` field.
2021-06-04 09:17:07 -07:00
Jim Ladd
1e45e2ab9b Rev migration number 2021-06-04 09:17:07 -07:00
Ryan Petrello
2a58605727 some more events view cleanup 2021-06-04 09:17:07 -07:00
Ryan Petrello
c7ab3ea86e move the partition data migration to be a post-upgrade async process
this copies the approach we took with the bigint migration
2021-06-04 09:17:07 -07:00
Jim Ladd
67046513ae Push changes before rebasing 2021-06-04 09:17:07 -07:00
Jim Ladd
f9b439ae82 include job_created field in all job event types 2021-06-04 09:17:07 -07:00
Jim Ladd
80b08d17e3 Continue updating job event views 2021-06-04 09:17:07 -07:00
Jim Ladd
f642c520bd Update fields for job event models to match raw sql operations
* raw sql commands were in migration to partition table
* .. just needed to add FakeAddField entries for the new
     job_created field added to each job event model
* .. and also needed to actually list the new field on the model classes
2021-06-04 09:17:07 -07:00
Jim Ladd
221ddeb915 Fix migration for _all_ job event tables
- each job event table has a different name for the
  fk referencing the unified job id
- create a mapping so that we get the col name correct
  for each table
2021-06-04 09:17:07 -07:00
Jim Ladd
d90d0fb503 fix initial partition creation
* call create_partition correctly (include tblname)
* reference 'awx_epoch'
2021-06-04 09:17:07 -07:00
Jim Ladd
2c529f50af Update querysets for remaining job event views
- search should use job creation date
  so that only the table partition with the relevant job events
  is searched
2021-06-04 09:17:07 -07:00
Jim Ladd
acfa1c4d1d Drop todo / question / conditional
* can safely assume job_created is set
* .. and if it isn't, we want to expose that bug
2021-06-04 09:17:07 -07:00
Jim Ladd
ea2afeec1f Drop todo / answered question 2021-06-04 09:17:07 -07:00
Jim Ladd
a5cfc3036f create_partition needs tblname 2021-06-04 09:17:07 -07:00
Jim Ladd
ec484f81cf Partition *all* job event tables 2021-06-04 09:17:06 -07:00
Jim Ladd
2ffa22e38f Misc doc clean-up 2021-06-04 09:17:06 -07:00
Jim Ladd
8fb313638c get job events based on job creation date 2021-06-04 09:17:06 -07:00
Jim Ladd
0eb1984b22 Only create partitions for regular jobs 2021-06-04 09:17:06 -07:00
Jim Ladd
f259b0a71b Move partition start date to 2000 2021-06-04 09:17:06 -07:00
Jim Ladd
82df3ebddb add option to create partitions that span a single minute
* turned on by default currently for testing
2021-06-04 09:17:06 -07:00
Jim Ladd
c87d7b0d79 fix import 2021-06-04 09:17:06 -07:00
Jim Ladd
612e91263c auto-create partition 2021-06-04 09:17:06 -07:00