API Testing: Selecting Testing Framework ( PHP Unit vs Codeception vs Behat )

In previous article of this series, we discussed that why you should use automated testing, specially if you are writing APIs.  So now it is time for selecting right tools for doing our API testing.

Available Testing Framework:

As there are application development frameworks to make your life easy, there are also many different testing frameworks which can be used for automated testing. Now point is that what framework you should use? It all depends. All have their pros. Important is what you want to achieve, how much time do you have.  When I was first time writing tests to test APIs, I took  4 days from client to find tool and write test cases so that, it can save me from manually testing all API endpoints and test cases before committing my code and leaving office. Yes, if you will not have automated tool then you will need to do lot of manual testing every day.

Anyways, my project was in Laravel 4 and by doing some search on Google, I came with three PHP based testing frameworks.

1.  PHP Unit: Because it is basic and starting from basic is easy.
2. CodeCeption: BDD-style PHP testing framework, it was more than basic, but can be used for different sort of testing.
3. Behat: It is a tool for Behaviour driven development. It means that test are ;written in human readable sentences that describes your application’s features.

Here is more detail about them with Pros and Cons:

PHP Unit:

PHP Unit is basic testing tool and first thing that can come to mind or (at time of writing) first record  come in Google search result if you search “PHP testing”. And PHP Unit older than all the above mentioned, its initial released was in 2004, so that means it is mature enough.

Pros of PHP Unit:

1. It is mature and very popular and that’s why good documentation, lot of tutorials and lot of threads on Q&A boards and on forums.
2. It is basic and basic is often simpler to start, when you are doing some thing first time or new in something.
3. PHP Unit is probably one of the best known tool for Unit Testing, as clear from its name too.

Cons of PHP Unit:

1. PHP Unit can be the best tool for Unit testing but API testing is different, so it is probably not better to use it for API testing or acceptance testing because, these are higher level different than Unit testing.
2. PHP Unit is very limited. It is easy to understand the basic but if you need more than that and want to higher level stuff then either you will need to integrate different tools with PHP Unit or will need to write lot of code to write higher level tests like API testing.

Codeception:

Codeception is known as BDD-style testing framework. If you go to codeception.com you will be able to see lot of different examples  for different type of testing. It is something that can fit to one’s needs in very different ways.

Pros of Codeception:

1. It is more than basic with lot of features available for different type of testing, no matter if it is low level like Unit testing or higher level like API testing or if it is BDD.
2. Even with lot of feature it is not that much difficult, it is easier that can be seen from homepage of Codeception.
3. It have separate modules for many PHP frameworks like Symfony2, Laravel4, Yii, Phalcon, Zend Framework. It don’t mean that it only support these frameworks but if you are using these frameworks and use its these modules, it will provide better features like errors will be more clear, and debugging will be easier however that can result in more memory usage in some cases.
4. It provides support for different other testing frameworks if you want to use them with it.
5. Its Test Cases are written in PHP so programmer don’t need to know a different language for that.

Cons of Codeception:

1. Codeception is no doubt an awesome tool but it probably don’t have as much documentation and resources compared to PHP Unit. However as it is powered by PHP Unit so one can go to that level to do something at low level too.
2. Codeception is no doubt easier but not as much simpler and easier to configure and start as PHP Unit can be.
3. Codeception is feature rich but it is still BDD-style tool not actually aimed at BDD and its test cases are written in PHP, so if you have a QA team who can’t write PHP then they can’t write different feature or scenarios of the system without touching PHP or programmer will need to write all those test cases.

Behat:

Behat is a BDD tool. And this is the purpose fo which mainly Behat is used.

Pros of Behat:

1.  As Behat is BDD framework so it’s language for writing test cases is very human friendly and person with no programming experience can write features easily.
2. Like Codeception, Behat is feature rich tool.
3. Behat test cases and cleaner and maintenance of tests in Behat is easier because a layer on which test case scenarios are being written is different than where these scenarios definitions are written.

Cons of Behat:

1. Behat is no-doubt awesome tool for BDD but for things like API testing it will probably need more tools to integrate with it.
2. If you don’t have a separate QA team and one programmer is writing tests then writing test case scenarios and its definitions separately will be bit more complex .
3. Programmer need to understand Behat’s human friendly syntax called Gherkin.
4. Due to more layers involved, for programmer who haven’t used it before, it can take more time to write test cases and understanding this tool.

Conclusion:

As told above I had 4 days to look into Testing frameworks and writing some test cases for API testing and framework I was using was Laravel 4, so what I started using was Codeception. It isn’t about winner or loser, it was about which testing framework is right for you based on your requirements and time you have to learn and configure it.

I picked Codeception over PHPUnit because spending a little bit more time on configuring and learning  Codeception, I was able to save much more time during writing test cases for API Testing. And I picked Codeception over Behat because Codeception seemed to have less steaper learning curve than Behat because Codeception’s test cases are written in PHP instead of Gherkin. And I didn’t have any requirement for Gherkin (human friendly syntax) as I was the only person who was writing Test Cases.

So this was my choice based on my requirements but feel free to choose different testing framework based on your set of requirements. In next part of this series “Testing API”, we will look into how to install and configure codeception and will look into its different files and directories.