PFF is a sports data company that supports NFL and NCAA teams with strategic insights and also operates a consumer arm focused on fantasy football and sports betting. With a fully distributed engineering team spread across India, Spain, and the United States, PFF faced challenges in keeping pace with competitors despite having a popular platform with 100 million annual page views and 9 million annual drafts.
The Challenge
Around early 2023, PFF had approximately 200 employees with 20 engineers. The company was falling behind competitors, particularly in the sports betting domain, which was a key area of consumer interest. Traditional engineering processes and team structures were limiting the speed and output of the engineering organization.
The Experiment: Building a Post-Engineer Engineering Org
Starting in November 2022, Mike Spitz began experimenting with Cloud Ops and AI agents to enhance engineering productivity. This initiative was spun out to two of PFF's strongest engineers—a front-end engineer and a full-stack engineer—to explore how to make engineers faster rather than just pushing them to output more.
Key Question
Instead of focusing on how to help engineers output more, the question was: How do you help make the agents quicker?
Results of the Case Study
- The two-engineer team achieved 25 times more deployments compared to a traditional 10-engineer team.
- The smaller team deployed five times every day, whereas the larger team deployed once every five days.
- After blending ticket counts with code complexity, the output of the smaller team was found to be 10 times higher.
- Features that previously would have taken four months to build were completed in under two months.
- One engineer was unblocked in under a month, enabling faster progress and compounding productivity gains.
Impact on Customer Satisfaction
Customer satisfaction was measured through statistically significant surveys, revealing an average quality score of 8.6 out of 10, compared to a previous average of around 7 to 7.5 before AI integration. This demonstrated that the increased output translated into better customer experiences.
Process Changes: Moving Away from Traditional Agile
The traditional Scrum process did not survive this transformation. Key changes included:
- No need for a project manager.
- Elimination of multiple communication layers and redundant ceremonies.
- Engineers are no longer the bottleneck, so daily standups, sprint planning, and sprint refinement were removed.
- Introduction of bi-daily huddles involving engineers, product, and design teams for quick feedback.
- Deployment to production in a minimum viable product (MVP) state to gather rapid feedback.
Development Flow
- The AI agent interviews engineers to create a specification.
- Feedback is gathered on the spec.
- The agent generates a lightweight design document (LDD) consistent with previous designs.
- Tickets and pull requests (PRs) are automatically created.
- PR statuses automatically update ticket statuses.
Quality Assurance and Automation
- Upon merging a PR, the system automatically deploys to staging.
- A QA agent reviews tickets and acceptance criteria, performing automated QA.
- If tests pass, the deployment proceeds; if not, issues are flagged.
- Future plans include agents automatically creating PRs to fix failed acceptance criteria, enabling self-healing workflows.
Human Involvement and Engineering Culture
- Engineers focus on high-level design and product feel.
- AI agents handle repetitive and opinionated code review tasks, such as enforcing style and naming conventions.
- The engineering culture is encoded in AI skills, ensuring consistency in software design patterns.
- Emphasis on verifiable, deterministic tasks to facilitate AI assistance.
Recommendations for Adoption
- Start with the best engineers who have deep system knowledge.
- Adopt a slow, phased approach rather than a wholesale immediate change.
- Experiment initially in non-critical systems to minimize risk.
- Treat the engineering development lifecycle like a factory, breaking down work into composable elements.
- Remove redundant processes and meetings that do not add value.
- Focus on boring, repetitive tasks that engineers dislike to gain buy-in.
- Ensure guardrails and quality controls are in place before scaling autonomous workflows.
Challenges and Considerations
- Not all engineers will adapt easily; those who are curious and can self-learn will thrive.
- Prescriptive, rigid engineering styles may struggle in this new paradigm.
- Large enterprises face greater challenges scaling this approach compared to smaller teams.
- Continuous monitoring of security, product feel, and engineering complexity is essential.
Conclusion
PFF's case study demonstrates that leveraging AI agents and rethinking traditional engineering processes can dramatically increase deployment frequency, output quality, and customer satisfaction. The key is a thoughtful, phased approach that balances automation with human expertise and maintains a strong engineering culture.
This transformation challenges long-held Agile practices and highlights the potential for AI to reshape software engineering organizations for greater speed and impact.
