API Testing: Installing and Using Codeception

This entry is part 3 of 3 in the series API Testing

In this post, we will have a look on how we can use Codeception for API testing. First we need to install codeception.

Installation:

Here is Codeception’s quick start guide that show multiple ways to install codeception. Here we will be installing through Composer, you can follow any other method as you like.

First go to directory where you want to install Codeception and place your tests. In my case that directory name is “app1”.

composer require "codeception/codeception:*"

or

php composer.phar require "codeception/codeception:*"

It will take some time, and fetch codeception. Once it is done, you will be able to use codeception command line tool. Try running codeception following command to make sure that you have codeception installed.

vendor/bin/codecept --version

Setup Codeception:

Here is something you need to do to setup your codeception testing environment. Run following command from root of your app directory.

vendor/bin/codecept bootstrap

It will create ‘tests’ folder at root of your app directory. This is where all your tests files and information will live. This tests directory will have many directories including acceptance, functional and unit . These directories are related to different types of tests suites.  We need to have api test suite, that isn’t yet created here. So let’s create that now.

vendor/bin/codecept generate:suite api

ok now in tests directory it will have a new directory named “
api
” . That will have Api Test suites in it. Ok, before moving further, let’s first get rid of this long path “vendor/bin/codecept” so that we can just use “codecept” command instead of whole path.

So if you are on linux, ubuntu you would probably be interesting in installing codeception globally. So that you can simply use “codecept” command from anywhere. Here is how you can do this: http://stdout.in/en/post/install_codeception_globally_in_ubuntu

If you are on windows then you can simply add it in your system path.

Ok once done that you can now to use codecept command you can simply say “codecept” and it will work.  Simply try following command from anywhere.

codecept --version

Now you are ready for next things but let’s first understand directory structure of codeception.

Codeception Directory structure:

If you go in “tests” directory which is created as a result of “codecept bootstrap” command. There you will see following directory structure.

  • _data
    This directory can have DB file if you need to use that.
  • _output
    This directory contains output of tests in case of failure.
  • _support
    This directory can have helpers if you write that to support your tests.
  • acceptance
    This directory is useful if you need to write acceptance tests.
  • api
    This directory is useful if you are writing api tests. This directory is not present by default but it will be created as result of api suite generation command.
  • functional
    This directory is useful if you need to write functional tests.
  • unit
    This directory is useful if you need to write unit tests.
  • _bootstrap.php
    This file is useful for autoloading any file that you want to include.
  • acceptance.suite.yml
    This file contain configuration of acceptance suite.
  • api.suite.yml
    This file contain configuration of api suite.
  • functional.suite.yml
    This file contain configuration of functional suite.
  • unit.suite.yml
    This file contain configuration of unit suite.

Also in api directory there is _bootstrap.php . It is useful if you want to include and autoload any file inside api suite only.

API test suite configuration:

Open api.suite.yml and add PhpBrowser helper and add URL, it should look like something like this:

class_name: ApiTester
modules:
    enabled: [ApiHelper,PhpBrowser, REST]
    config:
        PhpBrowser:
            url: http://localhost/app

Note: Above code snipper for yml file contains 4 spaces per tab and not tab character.

Laravel or PhpBrowser module :

If you are using Laravel, then you can add Laravel module instead of PhpBrowser. So that in case of error, if you are using Laravel module then error message will be more debug-able than a 500 error that can be seen on browser screen too. I was previously using Laravel module and error was more clear with better insights but problem is that Laravel module is much slower than PhpBrowser module so I switched to PhpBrowser. However if you have good RAM and not yet concerned about speed of execution of your test cases, specially if they are less than 50, then using Laravel module is better option. So it all depends on you.

Writing Test Case:

There are two ways to write test cases based on two types of files. Cept and Cest. Cept is a scenario-based format and Cest is a class based format. Actually it depends on how you are comfortable writing code for test cases. In strucute way or using classes. I prefer and recommended class based format because it will give you OOP power for writing more test cases with less code in long run.

So generate your first Cest file by this command:

codecept generate:cest api CreateUser

Now open that file and put some code in that as below.

wantTo('create a user via API');
        $I->amHttpAuthenticated('service_user', '123456');
        $I->haveHttpHeader('Content-Type', 'application/x-www-form-urlencoded');
        $I->sendPOST('users', ['name' => 'davert', 'email' => 'davert@codeception.com']);
        $I->seeResponseCodeIs(200);
        $I->seeResponseIsJson();
        $I->;seeResponseContains('{"result":"ok"}');
    }

}

Here $I contains the object of ApiTester that is respresting a user state who is testing a scenario.

Running Test Case:

If you are running testcase first time in api suite, then in your api directory you will not have api tester file. You need to generate that so run following command:

codecept build

Then run

codecept run api

It will show you error or failure if at mentioned URL you will have not have intended path or functionality working. If you want to see what’s going on behind the scene for your test case then use -vv or -vvv for even more verbose.

codecept run api -vv

To know more about Codeception, simply check codeception documentation: http://codeception.com/docs/10-WebServices

However, if you want to know more about Automated testing, better ways of testing, why and how to write tests. Then I recommend reading Laravel Testing Decoded by Jeffrey Way. And if you are a PHP guy then I even strongly recommend reading this book because you will most probably be lacking something that is in this book.

You’re probably familiar with Jeffrey’s Laracast, this book is even better than that to understand Testing and follow better approaches to both development and testing.

API Testing: Selecting Testing Framework ( PHP Unit vs Codeception vs Behat )

This entry is part 2 of 3 in the series API Testing

In previous article of this series, we discussed that why you should use automated testing, specially if you are writing APIs.  So now it is time for selecting right tools for doing our API testing.

Available Testing Framework:

As there are application development frameworks to make your life easy, there are also many different testing frameworks which can be used for automated testing. Now point is that what framework you should use? It all depends. All have their pros. Important is what you want to achieve, how much time do you have.  When I was first time writing tests to test APIs, I took  4 days from client to find tool and write test cases so that, it can save me from manually testing all API endpoints and test cases before committing my code and leaving office. Yes, if you will not have automated tool then you will need to do lot of manual testing every day.

Anyways, my project was in Laravel 4 and by doing some search on Google, I came with three PHP based testing frameworks.

  1.  PHP Unit: Because it is basic and starting from basic is easy.
  2. CodeCeption: BDD-style PHP testing framework, it was more than basic, but can be used for different sort of testing.
  3. Behat: It is a tool for Behaviour driven development. It means that test are ;written in human readable sentences that describes your application’s features.

Here is more detail about them with Pros and Cons:

PHP Unit:

PHP Unit is basic testing tool and first thing that can come to mind or (at time of writing) first record  come in Google search result if you search “PHP testing”. And PHP Unit older than all the above mentioned, its initial released was in 2004, so that means it is mature enough.

Pros of PHP Unit:

  1. It is mature and very popular and that’s why good documentation, lot of tutorials and lot of threads on Q&A boards and on forums.
  2. It is basic and basic is often simpler to start, when you are doing some thing first time or new in something.
  3. PHP Unit is probably one of the best known tool for Unit Testing, as clear from its name too.

Cons of PHP Unit:

  1. PHP Unit can be the best tool for Unit testing but API testing is different, so it is probably not better to use it for API testing or acceptance testing because, these are higher level different than Unit testing.
  2. PHP Unit is very limited. It is easy to understand the basic but if you need more than that and want to higher level stuff then either you will need to integrate different tools with PHP Unit or will need to write lot of code to write higher level tests like API testing.

Codeception:

codeception-logo

Codeception is known as BDD-style testing framework. If you go to codeception.com you will be able to see lot of different examples  for different type of testing. It is something that can fit to one’s needs in very different ways.

Pros of Codeception:

  1. It is more than basic with lot of features available for different type of testing, no matter if it is low level like Unit testing or higher level like API testing or if it is BDD.
  2. Even with lot of feature it is not that much difficult, it is easier that can be seen from homepage of Codeception.
  3. It have separate modules for many PHP frameworks like Symfony2, Laravel4, Yii, Phalcon, Zend Framework. It don’t mean that it only support these frameworks but if you are using these frameworks and use its these modules, it will provide better features like errors will be more clear, and debugging will be easier however that can result in more memory usage in some cases.
  4. It provides support for different other testing frameworks if you want to use them with it.
  5. Its Test Cases are written in PHP so programmer don’t need to know a different language for that.

Cons of Codeception:

  1. Codeception is no doubt an awesome tool but it probably don’t have as much documentation and resources compared to PHP Unit. However as it is powered by PHP Unit so one can go to that level to do something at low level too.
  2. Codeception is no doubt easier but not as much simpler and easier to configure and start as PHP Unit can be.
  3. Codeception is feature rich but it is still BDD-style tool not actually aimed at BDD and its test cases are written in PHP, so if you have a QA team who can’t write PHP then they can’t write different feature or scenarios of the system without touching PHP or programmer will need to write all those test cases.

Behat:

behat_logo

Behat is a BDD tool. And this is the purpose fo which mainly Behat is used.

Pros of Behat:

  1.  As Behat is BDD framework so it’s language for writing test cases is very human friendly and person with no programming experience can write features easily.
  2. Like Codeception, Behat is feature rich tool.
  3. Behat test cases and cleaner and maintenance of tests in Behat is easier because a layer on which test case scenarios are being written is different than where these scenarios definitions are written.

Cons of Behat:

  1. Behat is no-doubt awesome tool for BDD but for things like API testing it will probably need more tools to integrate with it.
  2. If you don’t have a separate QA team and one programmer is writing tests then writing test case scenarios and its definitions separately will be bit more complex .
  3. Programmer need to understand Behat’s human friendly syntax called Gherkin.
  4. Due to more layers involved, for programmer who haven’t used it before, it can take more time to write test cases and understanding this tool.

Conclusion:

As told above I had 4 days to look into Testing frameworks and writing some test cases for API testing and framework I was using was Laravel 4, so what I started using was Codeception. It isn’t about winner or loser, it was about which testing framework is right for you based on your requirements and time you have to learn and configure it.

I picked Codeception over PHPUnit because spending a little bit more time on configuring and learning  Codeception, I was able to save much more time during writing test cases for API Testing. And I picked Codeception over Behat because Codeception seemed to have less steaper learning curve than Behat because Codeception’s test cases are written in PHP instead of Gherkin. And I didn’t have any requirement for Gherkin (human friendly syntax) as I was the only person who was writing Test Cases.

So this was my choice based on my requirements but feel free to choose different testing framework based on your set of requirements. In next part of this series “Testing API”, we will look into how to install and configure codeception and will look into its different files and directories.

API Testing : Why to write Test Cases and use Automated Testing Tools?

This entry is part 1 of 3 in the series API Testing

Why to write Test Cases and use Automated Testing Tools?

Books written by some leading programmers often told us the importance of TDD (Test Driven Development) and BDD(Behaviour Driven Development) or they emphasize on at least writing Test Cases and doing Automated Testing. So why many programmers don’t write tests? Is that all not practical or all just too bookish, or your project life time is too small, or is it very much time consuming to write test cases? Here are few questions you will have in mind if you aren’t familiar with test cases benefits. So following are those questions answered.

Question: Obviously writing Test Cases take time and it still don’t replace manual testing, then what is the point in writing test cases. You have a QA person for that who can manually test each page then why Test Cases and this Automated Testing?

Answer: Problem is that if your QA person need to test 10 entities and 3 cases for each entity then these are simply 30 Test cases. So with every addition or modification in system, your QA person will need to test all these 30 test cases almost every day which can’t be efficiently done without automated testing.

Question: What if QA team is very efficient and can easily test lot of different scenarios, then why there should be automated testing?

Answer: Even if your QA team is very efficient, testing same things again and again is not a good use of your QA team time. QA team should design more test cases and from different perspective. And they will spend lot more time if there are different users in system are present. QA team will need to test system with different perspective by login with different accounts.

Question: Alright but it will be time consuming to write Test cases and configuring testing frameworks, so why should I not save time instead of writing Test Cases?

Answer: In long run, Test Cases will even save your time, Test Cases you have written once can be used later, and probably you will need to run tests every day after completing a feature or fixing a bug. So testing 30 Test cases is even more time consuming. If you have GUI of system then it will probably take a little less time but if  you only have APIs then it is much more difficult and error prone to test lot of APIs endpoint manually. So it is much better to spend some time on writing Test Cases instead of testing all Test Cases every day manually, so Automated Testing is lot more time saver in long run.

So it is lot better to Test system in a way where all test cases are written , can be tested under same circumstances every day, without manual dependencies once all test cases are written.

So is writing Test Cases really much time consuming?

Yes it is but it still save time. If you will need to Test something manually, you will need to perform same steps again and again for two different cases but while writing Test Cases for automated testing, you are doing all this in code. And in Code you can avoid writing same steps again by utilizing OOP  concepts and reusing code, that will definitely save time. So never be afraid of writing Test Cases. Specially when you are working on APIs then in many opinions writing Test Cases is your only option and if you will try to test APIs manually and these are not just few API calls then you will be in trouble.

So which type of Automated Testing I must use when writing API?

If you have time then better is to use Testing at every level. Do Unit, functional, Acceptance and API testing that would be helpful for debugging things. But if you don’t have as much time and you are working on APIs then you must at least do API Testing. So that if at day end some thing has gone wrong then you can test it by executing single command and know it before shipping buggy code. So it all depend on your need but if you are writing APIs then you must do API Testing other wise it will be really difficult to test different cases on different API calls manually.
To know about Automated Testing Tools selection you can find more information in my next post.