Now Tempest contains two kinds of API definitions with JSONSchema for Nova project. One is used for generating negative tests automatically. The other one is used for checking the response body in API test.
API definitions of the first one represent both the request body formats and error response codes(HTTP 404, etc.) of each API. Tempest operates negative API tests based on them and check whether the response code is the expected one. The API definitions is being implemented now.
The second ones represent both the response body formats and success response codes(HTTP200, etc.) of each API. The REST clients check the response based on each API definition and if receiving a bad response(lack of necessary parameters, etc.), Tempest raises an exception. By this behavior, Tempest is blocking the backward incompatible changes for Nova project.
I'd like to discuss some points for this topic:
* How to integrate API definitions API definitions of negative tests are stored under etc/schemas/compute/ and the other ones are done under tempest/api_schema/compute/. The description formats are different, but it would be nice to describe both of them in each API file for its maintenance.
* Port API definitions Nova also contains API definitions of JSONSchema for the request validation of v3 API. So we will be able to port API definitions of request from Nova to Tempest for negative tests.
On the other hand, it would be good to port API definitions of response from Tempest to Nova for API documentation. Nova contains template files of API response body and they have/should been used for Nova original integrated tests(not Tempest), but these template files sometimes were wrong because some tests did not use them and Nova tests could not detect them. If porting from Tempest and using them for the tests, it would make Nova test quality better because these definitions are already verified.
* Apply this framework to the other projects? JSONSchema is a common python library and we use it for Tempest. As the above description, we can reuse API definitions, which are described with JSONSchema, for multiple use cases. However, the other project(Ceilometer, Ironic, etc.) is implemented with WSME instead of JSONSchema. So if applying this to API tests of these projects, we use the definitions once only.
