Nov. 2021-Jan 2022 - Operations Workstream First Term Review
The Operations Workstream’s proposed mission: Assure the growth and development of healthy practices and protocols for reliably producing, testing, and maintaining the ShapeShift product suite and services. Assure uptime, responsiveness and feature functionality of the ShapeShift product suite and services as well as the maintain and support and strengthen the operations of the ShapeShift DAO, community, and governance processes.
This mission is achieved by 4 larger goals that are all documented in the Operations Workstream Notion. KPIs were drafted by the end of the term and are broken into the 4 categories and detailed below. Many of these KPIs may be updated or redefined as the DAO scales and grows in the Operations Workstream’s proposed second term.
KPIs
Production Issue Best Practices
99% uptime of the product suite and services - 99.969% Uptime this term
The only downtime experienced during the first term was a scheduled maintenance window on 12/9/2021 from 9:00am to 10:00am MST.
10 Production Issue Escalations per Quarter - 8 Escalations in Q4 2021, 2 Escalations in Q1 2022 so far
We got very close to missing this KPI. It was measured and set at the end of the first term to keep a reasonable but tight metric on expected prod issue escalations per quarter.
50 Total Combined DAO Contributor hours dedicated to Production Issue Resolutions Q4 2021 - 20 hours, January Q1 2022 15 hours
This is a very tight metric to maintain and I am honestly surprised we stayed within the range here with the prod issue history of centralized ShapeShift and the resources required to fix them. Both of these metrics are intentionally tight and will be challenging to maintain.
Greater than 80% resolution rate goal - 89.43% this quarter
123 reported issues, 13 unresolved issues This goal may be raised this next term as the triage reporting process continues to get refined.
Deployment and Regression Testing
Less than 2 Cadences missed per quarter due to bugs overlooked in Ops Testing - 1 cadence missed last term
The release process was created more than halfway through the first term. This KPI may be updated as the process is refined and developed.
Less than 2 Rollbacks during releases per quarter due to bugs overlooked in Ops Testing - 2 Rollbacks occurred last term
This is the one KPI we missed. I created 2 rollbacks in our first 2 releases jumping the gun before testing was completed. Our processes have since been refined and further templated to prevent future oversights.
DAO Admin Procedures and Practices
This goal still has more to develop in structure, procedures and archiving. As they are developed in this section, they will be shared and measured. Successful highlights and documents of meetings of these current procedures and practices include:
BUDGET
Salary: $91,500 spent - $109,500 budgeted (18,000 unused)
Operations was able to hire 2 of the slotted 4 Operations Assistant roles during the first term.
Test Funds: 30,000 Used - $22,000 budgeted (8,000 over budget)
Test Fund notes: Funding test wallets in the beginning of the DAO and still to this day costs a non zero amount in trading fees, bridge fees, slippage, and price volatility that was never scoped in the first budget request. It was the wild west for the first few months.
This caused a variance in USD value in budgeted funds unlocked in colony to output of mainnet ETH that ranged and at times was bad but unavoidable fund availability and at the maturity and pool depth of some of the available resources.
HoneySwap, OmniBridge, and Uniswap are all protocols that have been used in the process of transferring Colony funds to mainnet test funds.
Both the Product and Engineering Workstreams were provided test funds out of the Operations available funds based on requests made before they were funded in Colony, This led to additional funds needing to be sent in Colony earlier and increased and further volatility from the young and costly bridging process.
Test Funds and workstream funds were kept in either FOX on xDAI or Mainnet ETH. Both assets have had massive price volatility over the course of the term (FOX: $0.30-0.99 USD, ETH:$2,244-$4,814)
Operations Assistants are required in the testing agreement and release procedure to perform tests regardless of current network conditions which can result in high gas fees for sends, trades, staking and other function calls. This can and did make the cost of testing higher at unexpected times.
Budgets have now been secured in stables and the process of Omnibridge and Uniswap is much cheaper and at much less slippage risk than it was previously. ~$350/bridge process on average. The following term proposal has a Test Fund budget increase to $15k a month to account for additional team members, with an additional bump to $20k a month with additional hires. This new budget assumes a larger portion to be prepared for extreme network fees with the knowledge that all unspent funds are returned to the treasury at the end of the term.
Technology: $2020 Spent - $8000 budgeted ($5980 unspent)
The items covered were reimbursements on devices purchased by Operations Assistants needed to maintain their duties. Receipts of all purchases have been recorded.
Training/Conference/Travel: $1487 Spent - $6000 budgeted ($4513 unspent)
All funds spent here are reimbursements and funding for ETH Denver attendance.
Discretionary Funds: $1200 Spent - $1800 budgeted ($600 unspent)
This term, it was decided the discretionary funds were split up and dispersed evenly among workstream contributors to offset some of the cost of gas in bridging payments.
Total Budget Spend: $126,207 - Total Budgeted $147,300 ($21,093 unspent)
As of today, Operations has 25,793 USDC and 10,227 FOX to return to root in ending the first term, ~$30k USD, ~20% of the maximum total budget. Regardless of the over budget test funds, there was a net gain on unspent funds from market timing of payments of almost ~$10k USD, ~7% of the total budget.
It has been a pleasure to be trusted as the Operations Workstream Leader thus far, and I hope the community continues that trust with votes for the funding of the next term. Please feel free to comment or discuss here on any of the above reported information from the first term here, I am happy to provide resources and more information on any of the topics.