DynamoDB in pytest-dbfixtures

pytest-dbfixtures

If you use pytest maybe you have heard about pytest-dbfixtures:
Pytest dbfixtures is a pytest plugin that makes it a lot easier to set up proper database or storage engine for testing. Simply use one of provided fixtures that start predefined clean database server for your tests or creates server more tailored for your application by using one of provided factories.
This plugin is very useful if you have integration tests in your project, and you want to perform tests on a database for example. You will find information how to use it in the documentation.

Currently, the plugin supports:
  • Postgresql
  • MySQL
  • Redis
  • Mongo
  • Elasticsearch
  • RabbitMQ
And recently, we have added support for DynamoDB.

Here, you will find how to run DynamoDB on your computer. And, here we are. If you want to use it in production, you want to test it locally.

dynamodb fixture

If you still do not use pytest, go to the pytest page and read how to use it, now. If you want to understand the further content you have to know how pytest works.

The simplest way is to put dynamodb fixture as an argument in your test: 
1
2
def test_dynamodb(dynamodb):
    ...

You will find an instance of the class DynamoDB.ServiceResource in dynamodb variable. The plugin assumes you have extracted DynamoDB files into /tmp/dynamodb. If you have not extracted the files into this directory, or you do not want to do this, you can create an individual fixture: 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
from pytest_dbfixtures import factories

my_dynamodb_proc = factories.dynamodb_proc(
    dynamodb_dir="/path/to/mydynamodb",
    port=14235,
    delay=True,
)
my_dynamodb = factories.dynamodb("my_dynamodb_proc")


def test_my_dynamodb(my_dynamodb):
    ...

As you can see, you may also pass port and delay as arguments. If you do not specify the port argument, the fixture will use a random port. If you want to introduce delays for certain operations in DynamoDB you can pass the argument delay as a True.
DynamoDB can perform some tasks almost instantaneously, such as create/update/delete operations on tables and indexes; however, the actual DynamoDB service requires more time for these tasks. Setting this parameter helps DynamoDB simulate the behavior of the Amazon DynamoDB web service more closely. (Currently, this parameter introduces delays only for global secondary indexes that are in either CREATING or DELETING status.)
So, let's say that you want to run DynamoDB on 9009 port with some delays, and you keep DynamoDB files in /tmp/tests/dynamodb. Then you want to create a table and put some data there.

Here is an example:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
from pytest_dbfixtures import factories

my_dynamodb_proc = factories.dynamodb_proc(
    dynamodb_dir="/tmp/tests/dynamodb",
    port=9009,
    delay=True,
)
my_dynamodb = factories.dynamodb("my_dynamodb_proc")


def test_my_dynamodb(my_dynamodb):
    table = my_dynamodb.create_table(
        TableName="Test",
        KeySchema=[
            {
                "AttributeName": "id",
                "KeyType": "HASH"
            }
        ],
        AttributeDefinitions=[
            {
                "AttributeName": "id",
                "AttributeType": "N"
            }

        ],
        ProvisionedThroughput={
            "ReadCapacityUnits": 10,
            "WriteCapacityUnits": 10
        }
    )

    table.put_item(
        Item={
            "id": 42,
            "my_data": "is secret"
        }
    )

    response = table.get_item(
        Key={
            "id": 42,
        }
    )

    assert response["Item"]["my_data"] == "is secret"

Because we want to test faster and faster, we run DynamoDB in memory. Remember that when you stop the process of DynamoDB, none of the data will be saved (DynamoDB does not write any database files at all). If you do not want to run tests in memory, your Pull Request is welcome.

Comments

Post a Comment

Popular posts from this blog

GitLab - extends keyword

Image
In the previous post I had written about the include keyword. Now, let's dive in the extends  keyword. What can you do with this? Here is the definition from GitLab documentation: extends defines entry names that a job that uses extends inherits from. It’s an alternative to using YAML anchors and is a little more flexible and readable. So, when is it useful? It is useful when you want to be DRY and keep your setup cleanly. Let's assume that you want to build a docker image and give developers possibility to deploy the image on dev , staging and prod environments. We are going to do that without the extends keyword, and after that we will think how we can do it better. Example without the extends  keyword: As you can see, there is a lot of code which repeats itself: stage, image, when  and script . We can remove a lot of code! So, let's do that. We are going to create a hidden job .deploy  (you will not see this job in a pipeline) and the rest of the jobs will in
Read more

Managing Secrets in GitLab / Git

Image
Let's say that you have to log in via ssh into an instance, and you work with GitLab, so you want to keep the private key in GitLab somewhere. Is it secure? Let's see! Custom environment variables You can use custom environment variables. Here you can read more about them (Developers cannot change them, only Maintainers and Owners can). There are two types of variables: Variable (the runner creates an environment variable that uses the key for the name and the value for the value) File (the runner creates an environment variable that uses the key for the name. For the value, the runner writes the variable value to a temporary file and uses this path) It seems that we can use File type for our purpose. We can set up it via API or UI . So, let's do that! Go to project's Settings > CI/CD . There will be Variables section (btw, you can specify variables also per group and even for all projects (in admin panel)). Click Add Variable button and add a variable: Key:
Read more

GitLab - trigger keyword

Image
 We are going to talk about the trigger keyword today. With this feature you can define a downstream pipeline trigger. So, you can trigger a pipeline in any project (you must have access to this project of course). There are two types of downstream pipelines: multi-project pipelines child pipelines  Let's omit child pipelines and have a look at multi-project pipelines. It is very easy to use. You just need to provide a path to the project and that's it. Remember that you only can trigger a pipeline, not a job ! You can also provide more information like environment variables. As always, I am going to provide some examples, so you will understand it better. The first example: you want to trigger a pipeline in another project after the deployment of your service/s. How to do that? Here we want to trigger this pipeline (project my/run-tests ). Let's say that we want to run some tests after each deployment: We need to add the job which will trigger this pipeline in my/run-
Read more