Summary
This proposal is for a continuation of budget funding for another 6 month term, August 1- January 31st, 2024 (Fifth Term) for the current Operations workstream that was established with passing of the previous proposals: [SCP-8], [SCP-59], [SCP-94], and [SCP-125].
This proposal seeks to resume the current team and achieve the same KPIs and goals while being open and flexible to the future requirements to support the new ShapeShift API services that have yet to be built, as well as define Customer Support KPIs once the transition to Chatwoot has been completed. Future KPIs will be drafted and assumed during the term to match the needs and requirements of the DAO’s product suite in all forms.
This proposal adopts [SCP-92] Definition of Workstream Leader
Abstract
The Operations workstream is an integral part of the growth and development of healthy systems for reliably producing and maintaining the ShapeShift product suite and its uptime, responsiveness and feature functionality. The workstream additionally provides similar support and structure to the DAO, its ecosystem and networks.
Through Production Issue Procedure Best Practices, Triage Management, Deployment and Regression Testing, Customer Support and DAO Admin Procedures and Practices, the ShapeShift DAO Operations Workstream provides essential resources and tools for efficiency, organization, and overall product and services health that are vital to the success of the DAO, the product suite, it’s contributors, and $FOX token utility.
Motivation
Renewing this workstream will allow Operations to continue to work side by side with other workstreams to increase efficiency in individual procedural processes across the DAO and help progress deployment and testing processes while still verifying and maintaining reliability, health, and full functionality of all provided services.
Areas of focus and Operations responsibilities maintained by the workstream also include: Hosting the Weekly Governance call and advocating the Governance process, Organizing and hosting the bi-monthly AllFox community call, Marketing content reviews and support, Leading and collaborating in Product specs, Expected Feature Release Testing guidelines, Go/No Gos, Retros and other coordinated meetings or DAO processes, RDP management, Gitbook documentation management (soon to be expanded into Chatwoot Helpdesk), full coverage of customer support, and the flexibility to provide future technical support and partner relations with individual consumers of our new ShapeShift API in whatever ways the DAO requires once we build out this service.
The Operations workstream continues to focus on the wider picture of the product suite’s uptime, functionality, and overall health and user support. It additionally mirrors that responsibility in the DAO Discord, Github, and Notion by synergizing and coordinating cross-stream collaboration further strengthening the DAO.
Specification
The Operations workstream Notion repo can be found here.
Operations maintains:
Triage: Known Issues
2023 Prod Issues tracking, Release Stats and testing fees,
Ops Sprint, Governance Call, AllFox, Go/NoGos, Retros, RDP program, ShapeShift API payment Multi-sig Controllers, Gitbook, Customer support in Discord
The Operations Workstream strives to keep the greater DAO agile and committed to maximizing efficiency, scalability, and transparency with a strategic lean into data-based decision making. At times, best practices and procedures may be challenged, modified, updated, changed, archived or revisited to maintain those objectives. (See: The Bounty Process, Trusted Community Devs, Triage Report Tracker, Unsolved User Problems, the governance tracker, POAP minting, daily regression testing, the evolution of goat/no goats, etc)
The renewal term length for this workstream and its contributors is 6 months, from August 1st 2023 to January 31st 2024.
This proposal continues @Tyler | ShapeShift as the workstream leader and @Tshifty, and @MBMaria as Operations Coordinators.
Budget ask: $177,000 - 32.57% less than the previous term
Monthly USDC: $23,875 - 26.37% reduction from $32,425.00 last term.
Monthly FOX: $5,625.00 - 50.33% reduction from $11,325.00 last term.
Monthly total budget: $29,500.00 - 32.57% reduction from $43,750.00 last term.
6 month USDC budget: $143,250.00
6 month FOX budget: $33,750.00
6 month Total budget: $177,000.00
Full budget details
The budget has been reduced across the board in both $FOX and $USDC monthly ask to continue to keep a conscious effort in extending the DAO Runway while maintaining an agile and intimate team of Operations contributors that can collaborate across time zones to provide adequate coverage for support, triage, testing, and all other outlined needs of other workstreams, the DAO, and it’s product suite. This budget has a reduced team of 3 full time contributors as opposed to the staffing room for 5 approved in the previous term.
Outside of the team size decrease, there has been a reduction in the amount of monthly test and misc funds, and notability 17% reduction in Tyler’s salary, returning his compensation to the USD value it was in [SCP-94]. With the conclusion of the temporary additional Workstream leader responsibilities in [SCP-99] and [SCP-133] Tyler’s Salary and time commitment will be reduced accordingly.
Operations will continue to be mindful and agile about its processes with the DAO to continue to provide proficient coverage and support in all of the outlined areas of this proposal while maintaining our KPIs under this optimized budget.
Operations would account for 12.4% of the total DAO monthly stable spend if this proposal (and the increases in [SCP-144]) were to be passed,
If network conditions or the price of FOX decreasing dramatically alters or changes Operations ability to perform required deployment and regression testing, an additional proposal addendum for test funds may be proposed.
Specific Goals and Success Metrics
The goals and success metrics for the Operations Workstream are listed below, split into current Goals and KPIs.
Current Measurables:
Production Issue Best Practices
Goals:
To minimize downtime of the entire product suite.
To resolve production environment bugs efficiently with minimal disruption to engineering processes, deployment cadences, or user experience.
KPIs:
99% uptime of the product suite and services:
February: 96.55%
March 99.9%
April: 100%
May: 100%
June: 93.34%
July: 100 % (so far)
Average 6 month uptime: 98.29% (97.9% last term)
<8 Production Issue Escalations per Quarter - 3 Escalations in Q1, 1 Escalations in Q2
<50 Total Combined DAO Contributor hours dedicated to Production Issue Resolutions Per Quarter (Q1 - 8 hours, Q2 - 5 hours)
Triage Management
Goals:
To discover, isolate, navigate, and manage resolutions of user facing issues in all ShapeShift products and services.
To provide known troubleshooting and mitigation steps for issue reporters and users during the discovery and recreation of triage tickets.
To mitigate and expedite time required from core contributors spent on bug tickets, issue reports, and production level emergencies.
To transfer correctly completed and accurate Github tickets of triage bugs to Engineering or Product to be posted for bounty.
KPIs:
Greater than 80% resolution rate goal: 71% Resolution rate last term, 66% resolution rate previous term.
This goal was not hit this term again overall. Operations is improving our overall resolution rate term over term and aim to be able to exceed this KPI in this proposed term with improvements and optimizations to the Triage process and ticket review/clean up practices as a new focus this term
Deployment and Regression Testing
Goals:
To catch 95% of prod issues during the deployment testing process.
To catch user facing issues in production environments in regression tests and #triage them accordingly.
A deployment test is completed by the Operations Workstream during each release process and upon request in pre-deployment branch testing.
Specific regression tests are performed once a week by an Operations Workstream contributor to complete a full monthly regression testing schedule in accordance with regression testing best practices.
Operations prioritize blocking bugs, supporting engineers, and efficiency in the processes already in place over chasing non-blocking bugs for a ‘perfected’ release.
In the 160 releases deployed since Feb 1, 2023, 3 total prod issues were experienced with only 1 of them being caused by bugs that could have been caught in the deployment testing process. 3 bugs in 160 releases = 99.98% This does not include the releases from the earlier version of testing procedures.
KPIs:
<2 Cadences missed per quarter due to bugs overlooked in Ops Testing (0 cadence missed last term)
<2 Rollbacks during releases per quarter due to bugs overlooked in Ops Testing (0 Rollbacks occurred last term)
DAO Admin Procedures and Practices
Goals:
To organize successful DAO meetings related to Goat/No Goat, Retros, Ops Sprints, and AllFoxes.
To track, organize, record, and optimize DAO repos keeping them current and relevant. To help in cross workstream collaboration projects that optimize efficiency and collaborative, transparent communication.
To provide assistance in projects that require support in generating procedures to be completed for the DAO or other workstreams.
KPIs:
Keep All Workstream leaders and key contributors informed and involved in contributions to all Ops Sprints, Governance Calls, AllFoxes Slide decks, Go/No Gos, and Retros.
This KPI will additionally be updated for future Customer Support and ShapeShift API support and uptime KPIs and goals this term.
Benefits:
If this proposal passes, the DAO will continue to fund the Operations workstream budget to meet the deliverables and services outlined which are integral to the DAO’s success and current practices. Coverage for Prod Issue escalations, Triage Management, Regularly scheduled regression and deployment testing, Customer Support, Cross workstream alignment and stakeholder resources would all be focused, maintained and improved upon by the workstream.
Passing this proposal will continue uninterrupted support and services offered from the Operations workstream to the DAO.
Drawbacks:
Funding this workstream costs the DAO stablecoins and FOX during the duration of the proposed term. This workstream would cost the DAO treasury potentially $177,000 over 6 months ($29,500/month) if all funds are needed and exhausted in the budget costing the DAO $ from the stablecoin runway.