== Tuesday, 9 June 2009 == (times in US Eastern) Attendees: Jesse Keating, Paul Frields, Rex Dieter, Seth Vidal, Bill Nottingham, James Laska, Josh Boyer, Luke Macken, Tom 'spot' Callaway, Steven Parrish, John Linville, Will Woods, Bruno Wolff is sitting in again == Beta freeze is a lie == (5 votes) - first look at feature complete content - known good starting point - wide testing of installer - split media - a less "EAT YOUR BRANE" release for testers - final product structure Failure points: - not actually feature complete - doesn't match industry standard - installation problems limit testing - composes are not on a a schedule - released beta content is ~2+ weeks old - rate of change post-beta is too high == Preview == (5 votes) - last chance look - known good point/freeze point - code complete - only critical fixes accepted - start of signed content, turn on gpgcheck=1 - forced branching of SCM for the next release Failure points: - made up term - high rate of change - unclear criteria for critical or acceptable freeze exceptions - small amount of time to GA - no outlet for low priority fixes - who is testing it? - unclear state of tree - surprise compose failures - no idea how many people are using it - who's got the ball? == RC/feature freeze/the cake is a lie == (5 votes) - verify compose of final bits - last chance to catch showstoppers - blocker review and clearing - define what we would slip for Failure points: - slips at last moment - no clear date of when we start composes - delay from fix to tree - start of blocker review too late - what ISOs are delivered? - spins? - who has the ball? - release readiness decision on RCs are not made in public or documented - no clear signal/communication that bits are on the mirrors and that we are "done" == Feature freeze is a lie == (5 votes) - no more new features - all features "Feature complete" - stop 'feature' work to give time for integration - get updated status from feature owners - know what to test - contingency plans enacted for unfinished features Failure points: - features are not feature complete - features not fully defined - crash-landed features create instability - lack of enforcement (empathy aside) - non-practical contigency plans - no coordinated verification of feature complete - not all features treated/are equal == Buildroot overrides == (6 votes) - provide ability to build against unreleased packages Failure points: - release engineering overhead - not auto-cleaned out - low awareness among packagers - hard to submit to rel-eng - doesn't actually prevent accidental biulds - overrides affect every builder - prevents chain builds == Trees/rawhide/updates/etc. takes too long to compose == (6 votes) - full recompose to avoi manual things when content moves between packages - take Koji inheritance into account Failure points: - multilib - **DeltaRPMs** - slow storage - TOO. MUCH. STUFF. - full recompose, not incremental - koji query is slooooow - limits number of repos we can compose in a day - duplication of process with Koji - fragile - too many moving parts - GPG sig checking - no continuing after a fail - push scripts push failed composes - hard to pinpoint slow points/debug - slow to get bits to high vale test centers - can't tell if mirror is up to date - snapmirror - lack of HW resources - lack of test/stage system == Lack of enforcement of policies == (7 votes) - punted; out of scope == No last known good rawhide == (8 votes) - fall back when rawhide fails - usually based on anaconda failures - a snapshot that is mirrored - provides something to make a live image from Failure points: - that we need it - we don't tag an existing tree as stable - we don't mirror the last known good tree - no definition of what all to save - no clear criteria for marking a tree as "good" - doesn't cover Live images - https://fedoraproject.org/wiki/JohnPoelstra/ImproveRawhideF10 - prior discussion of this == Every package is equal == (10 votes) - simiplicity and equality/uniformity - hard to draw the line where we care/don't care Failure points: - still care abot some things more than others (kernel is the obvious example) - freeze is for everybody == Check for dependency closure on any repos == (8 votes) - would prevent broken repos - alert maintainers of work needed Failure points: - we don't do it - it takes far too long - can't do partial/incremental despolve - perfect is the enemy of the good - no partial fallback ACTION: punted for now == Rawhide installable == (11 votes) Failure points: - often, anaconda falls over for long periods of time - often, not anaconda's fault - waiting for rawhide composes to test anaconda changes - wait until tomorrow and try again - no "israwhidebroken.com" - no fallback point IsRawhideBroken.com: - critical path? - feedback from packagers - refreshed daily - installable? == Frozen tree (which is rawhide) == (12 votes) - prevent unwanted change - communicate that we don't want unstable packages - provide a stable base to build installer/trees/isos/the release upon - provide a stable target for testing Failure points: - freeze is for all of rawhide - release engineering overhead - low awareness among packagers - hard to submit to rel-eng - little community interaction/encouragement with freeze breaks - forces things into updates - we keep everything frozen when waiting on one package - hides next release stream = PROPOSALS = == Jesse's proposal == - do a full branch at beta - rawhide moves on to the next release - a release tree is published elsewhere - freeze breaks are done via bodhi; karma gets you into the tree === Potential issues === * buildroot override issues? still use bodhi? * double commits * relies on updates-testing model, which we agree is fail * how do we compose two trees * where do we put the new beta++ tree * if rawhide is always 'the next release', how do we get people on the beta++ tree * does this split our testing resources/who is testing them * are we then Debian (unstable/testing/stable) * breaks chain builds even more (for the beta) * which releases do we target install images for? * no "straight to stable" (for certain packages?) === Helps with === * Frozen rawhide * what is rawhide * * is a lie * rawhide is not installable === Dependencies === * compose time needs fixed * need more pickup on the updates-testing model * bodhi changes == Bill's proposal on milestones == * Alpha => goes to /dev/null * Beta => renamed to Alpha (still 4 months post-prior GA, still feature freeze, etc.) * Preview => renamed to Beta (still one month post-Alpha, still code complete, etc.) * RC => remains RC (still one month post-Beta) === Potential issues === * are the milestones long enough? === Helps with === * * is a lie === Dependencies === * some wiki docs == Takes too long to compose == Bill, Seth, Jesse, others (Jonathan Dieter) need to debug the process and fix things. == Earlier blocker reviews == We need to start blocker reviews earlier. * halfway between beta and RC? === Potential issues === * resources required to do the review === Helps with === * RC is a lie * slips === Dependencies === * none == Every Package is Equal. But Some Packages are More Equal than Others == * not every package is the same * if your package is in the "critical path", there are extra requirements on getting tagged for the release === Dependencies === * needs some sort of AutoQA on the non-critical-path packages * define critical path * what does "extra" mean ** QA/releng signoff to go from "testing" to stable ** "testing" is required * bodhi work to facilitate the approval === Helps with === * every package is sacred * frozen rawhide * updates-testing === Potential issues === * does not necessarily scale w.r.t. releng or QA === Proposed critical path === * graphical network install * post-install booting * decrypt encrypted filesystems * graphics * login * networking * get updates * minimal buildroot * compose new trees * compose live === Issues === * critical path changes occasionally * some maintainers may not want to be in the critical path, but will be * how does this work with spins? do they expand the set of packages? ** that does not scale. perhaps spin maintainers need to watch packages they depend on in bodhi == Signing == * Auto-sign from koji (non-scratch builds) ** One package key to rule them all ** one (different) repodata key per release === Issues === * Having only 1 key == key compromise is utter DOOM * does not help: ** inability to add additional signatures ** cannot revoke keys === Dependencies === * Automation of signing server * repodata signing * Inserting signing into the koji pre-tag process / changing "make build" to not tag, and doing signing as a non-koji process === Helps === * signing * takes to long to compose (at some future point) == KoPer discussion == http://fedoraproject.org/wiki/JesseKeating/KojiPersonalRepos * Is this something we need for real use cases? * This is a bit of a digression, since this is out of scope == Every update everywhere == * We address some of this if we change the freezes/critical path tactics == Last known good Rawhide == === Issues === * Can you not do this with boot.iso and latest Rawhide? * Mirror it? * Static path to a known good repo * QA makes a call for whether it's good enough for critical path * Previous discusion around rawhide criteria https://fedoraproject.org/wiki/JohnPoelstra/ImproveRawhideF10#Last_Known_Good_Tree * How do we let people know "today is not safe to update to Rawhide"? === Dependencies === * None? === Helps === * More installable Rawhide