When I join product team or company here is the list of questions i have in my mind. What do you think about that? Do you think some questions are not relevant? Is there anything you would add?
I. Provisioning
- Do you use public cloud service providers?
- Are you considering building private cloud?
- Are you using containers?
- Please describe infrastructure provisioning process from idea to deployment
- Please describe any needed areas of improvements/shortcomings of the existing infrastructure
- What is the lead time required to provision a new production environment or enhancement?
II. Operations
- Please describe how do you plan your capacity? (decision making process, how far ahead, how often, with whom?)
- Please describe your disaster recovery strategy, including system backup/restore procedures, simulation of disasters… (how often? who initiates?)
- Please describe any dependencies that need to be managed as part of deployment (e.g. CDN, caching 3d party services, …)
- Please elaborate on how broadly you’re using opportunities to choose external service providers. Why? What type and size of projects?
- How do you ensure that your solution is resilient?
III. Quality assurance
- Please describe your quality assurance process: who, what, how, …
- What is definition of quality in your team? (SLOs)
- Please describe your test planning. Each sprint, month?
- How Business/Product people are involved in quality assurance? By whom?
- Please describe tools/services used for quality assurance.
- What is the percentage of your environment is covered by automated tests?
- How often tests are executed?
IV. Continuous integration
- Please describe the CI process implemented in your team (how code gets from developer workspace to shared repository)
- Please describe the frequency of builds. Why?
- Please describe the extent of automated testing
- Please describe tools/services used for CI process
- Please describe any needed areas of improvements/shortcomings of the existing CI environment/process
V. Continuous delivery (Deployment automation)
- Do you have downtimes during deployment? (How long?)
- Please describe the extent of deployment automation within the development, staging and productions environments
- Please describe the frequency of deployments
- Please describe any cases of manual deployments
- Please describe the ways used to document deployments
- Please describe the tools/services used for deployment automation
- Please describe any needed areas of improvements/shortcomings of the existing deployment automation process
- Please describe how your deployment changes in case of major changes that require infrastructure or application architecture alterations
- What is your maintenance/downtime window and how it relates to automated deployments
VI. Configuration management
- Please describe the extent of configuration management within the development, staging and productions environments. How? Who? When?
- Please describe the frequency of configuration updates
- Please describe any cases of manual configuration changes
- Please describe the ways used to insure configuration consistency
- Please describe tools/services used for configuration management
- Please describe any needed areas of improvements/shortcomings of the existing configuration management process
VII. Monitoring and logging
- Please describe the extent of monitoring and logging within the development, staging and productions environments. Types of metrics, frequency, size of logged data, period of time
- Please describe who and how uses information? What decisions are made using this information? How often (sprint, month, quarter)?
- Please describe who is defining what and how to monitor/log
- Please describe tools/services used for monitoring and logging
- Please describe any needed areas of improvements/shortcomings
VII. API
- Do you understand benefits of building APIs? Is your functionality covered with APIs?
- Do you have geographically distributed teams? Do you have many UIs (mobile, desktop)?
- Does anyone outside your organization would like to access your data and workflows in a programmatic way?
- Do you have platform team to work on following services: infrastructure, guidelines, development/configuration portal?
- Have you established API community to drive the topic across organization: education, contribution to common platform services?
Typically it reveals following issues:
- Missing (or understaffed) service oriented platform teams that focus on building self-service solutions for product teams.
- No ability to purchase/rent needed services as temporary solution before internal service teams are established properly.
- Quality is not measured which makes it (almost) impossible to prioritize non-functional (scale, technical debt, increased usage) improvements.
- No multifunctional teams with a mix of #1 and #2 leads to additional stress on teams responsible for a service due to additional headache of prioritizing scarced resource. It also reduces possibility of exchanging feedback between product teams and service teams, which is crucial for building internal services right.
- (Almost) no focus on automating activities related to software delivery and maintenance which increases delays and manual work as solution complexity and usage increases.
More lessons learned - http://agilemindstorm.com/2017/02/18/network-structure-11-lessons-learned/