The purpose of an SPIP is to inform and involve the user community in major improvements to the Spark codebase throughout the development process, to increase the likelihood that user needs are met.
SPIPs should be used for significant user-facing or cross-cutting changes, not small incremental improvements. When in doubt, if a committer thinks a change needs an SPIP, it does.
An SPIP is similar to a product requirement document commonly used in product management.
Any community member can help by discussing whether an SPIP is likely to meet their needs, and by proposing SPIPs.
Contributors can help by discussing whether an SPIP is likely to be technically feasible.
Committers can help by discussing whether an SPIP aligns with long-term project goals, and by shepherding SPIPs.
SPIP Author is any community member who authors a SPIP and is committed to pushing the change through the entire process. SPIP authorship can be transferred.
SPIP Shepherd is a PMC member who is committed to shepherding the proposed change throughout the entire process. Although the shepherd can delegate or work with other committers in the development process, the shepherd is ultimately responsible for the success or failure of the SPIP. Responsibilities of the shepherd include, but are not limited to:
Anyone may propose an SPIP, using the template below. Please only submit an SPIP if you are willing to help, at least with discussion.
After a SPIP is created, the author should email email@example.com to notify the community of the SPIP, and discussions should ensue on the JIRA ticket.
If an SPIP is too small or incremental and should have been done through the normal JIRA process, a committer should remove the SPIP label.
All discussion of an SPIP should take place in a public forum, preferably the discussion attached to the Jira. Any discussions that happen offline should be made available online for the public via meeting notes summarizing the discussions.
During this discussion, one or more shepherds should be identified among PMC members.
Once the discussion settles, the shepherd(s) should call for a vote on the SPIP moving forward on the dev@ list. The vote should be open for at least 72 hours and follows the typical Apache vote process and passes upon consensus (at least 3 +1 votes from PMC members and no -1 votes from PMC members). dev@ should be notified of the vote result.
If there does not exist at least one PMC member that is committed to shepherding the change within a month, the SPIP is rejected.
If a committer does not think a SPIP aligns with long-term project goals, or is not practical at the point of proposal, the committer should -1 the SPIP explicitly and give technical justifications.
Implementation should take place via the standard process for code changes. Changes that require SPIPs typically also require design documents to be written and reviewed.