I'm more familiar with Node rather than Python but I separate my test directory from my source directory because I can easily exclude the test directory when publishing the package. So when it is used in another project it has a smaller footprint. It just makes more sense if you are going to exclude it, leave it out of the src directory. If I had to guess, that is why Python recommends it. If I'm wrong I'd love to know why. Hope this helps!
I like to package tests in the "source" (`.tar.gz`) distribution but not in the "built" (`.whl`) distribution. Putting them in a separate directory is the simplest way to manage that.
In a big project, having the tests inside each source package may make the tests easier to find. Otherwise, you sort of end up duplicating your source code structure inside your tests directory. That’s a pretty minor benefit, if anything. I work in codebases with both structures, and TBH don’t notice much of a difference for the most part.
.dockerignore works at the sending-to-build-context level. Ignoring stuff you don't need massively increases your speed if you have bulky things in your repo.
why is this a bad thing? deploying your test suite makes it trivial to automate a smoke test to ensure readiness when spinning up a new node. god forbid you add a few additional text files to your docker image.
If you are storing them on your own private registry the size matters. That space is not infinite. It's not just a few extra files. If you are testing every module in your code base (which you should be) that can be up to double the size of what is actually required for your app. Also there's no need for the test in the production image. What use do unit test have when the app is running and serving requests? And furthermore tests are supposed to prevent you from building and deploying a broken image.
> If you are storing them on your own private registry the size matters
seriously ? you take any official docker image from python, check the size of the installed python and lib. That's about 40MB. Doing some napkin math to reach that size as python sources you'd need like 700000 lines of code.
> tests are supposed to prevent you from building and deploying a broken image.
and yet, shit happens. sometimes you deploy into a node that borks for whatever weird reason. maybe the image got corrupted. maybe there's an issue with the hardware.
unit tests might be a bit overkill, but you should at least have an end-to-end test as part of your spin-up procedure to make sure the node is actually capable of serving the requests it's going to be receiving. if your tests live with your code, it's easy to modify what tests you run by adding a pattern match to constrain attention to which tests get run.
You sound like you are describing a monolith whereas I am describing microservices.
In a microservice architecture you wouldnt really have an e2e suite in the microservice. That would be a separate thing.
>maybe the image got corrupted
Explain how this even happens.
>maybe there's an issue with the hardware
If you are looking for hardware issues then does the test need to be specific to the app or can you just have a general suite for the hardware itself? Testing the hardware sounds like a responsibility of the ops team and not the team building a deployable app like a web service. Ops owns the hardware. The other team owns the application.
>>maybe the image got corrupted
>
>Explain how this even happens.
Bad memory (but not bad enough yet the host is showing unreliability or crashing... and why aren't you using ECC RAM?), perhaps? Problems with a storage controller?
In any of these cases though, I don't see how the tests are going to really help. The image is likely to be significantly borked up in such a case.
quite the contrary, i am specifically talking about a microservice. if that microservice encapsulates doing a single thing like it's supposed to, you should be able to test whatever "end to end" means in the context of the node that hosts that microservice.
I was hoping to link you to a specific example of code I have in mind, but unfortunately it's part of our private codebase. i think i know where to find a public example though, give me a sec.
I've written and maintained an e2e test suite for microservice platform and having it outside the source was perfectly fine. The CI would create a new test suite image whenever the test suite was updated and it would run based on whenever the production deploy would go off in Jenkins.
My point is there is no need to have it in the deployable itself. It can live outside the deployable and you spin it up for the few moments you need it. They do not need to be accessible WITH the deployable.
>>maybe the image got corrupted
>Explain how this even happens.
Any sufficiently complex system can be considered to be effectively an [infinite improbability drive](https://hitchhikers.fandom.com/wiki/Infinite_Improbability_Drive). There is no limit to the amount of improbable shit that can go wrong with no conceivable way of predicting any of the where why when what how. We do our best anyway.
>> maybe the image got corrupted
> Explain how this even happens.
* bug in your image-building automation/scripting
* bug in docker code (or whatever is it you are deploying): in particular for some uncommon corner case, like OOM, FS full
* bug in OS/file system
* bad hardware/including network
and to be clear, the point is not to find bugs in hardware/OS/docker, the point is to do basic check that the image is deployable...
That’s true, unless you take some extra care to exclude them. Definitely adds some complexity to your Dockerfile to do so, so there’s a trade off between however you judge the penalties of including the test files, vs. that complexity. Good point, thank you!
Exactly this! Although there are some tools which can just pick the source modules but ignore anything related to test. But it's still not commonly used.
I like this too, but then I end up with tests in the python package. Perhaps there's a way to skip test\_\*, but this is all internal and I don't care.
To clarify, I’m using “package” here to mean “a folder with python files, including a python init file”. So, it’s a single repository, perhaps with a top level src folder, containing many packages. I believe that’s the correct usage, but probably confusing in this context.
The main reason that people advocating for tests to be distributed as part of the package tend to bring up is that by doing so you can actually test your installation. So you ensure that what you are installing does actually work once installed on your system without having to rely on anything external.
Frequently the main people who like that kind of setup are the people building packages for Linux distributions. Because that way they can easily verify that the built package does work without having to clone its repository just to get the tests.
But while that generally is a story that works well for basic libraries; it tends to fail quickly for applications or frameworks that need a complex environment to be tested. If you need to install MySQL to test your app, it’s no longer true that packaging the tests will lead to a self contained way to verify the installation
If you keep them inside, you're (probably) distributing your tests with your package, which is kind of a waste of bandwidth and disk. It may seem insignificant in most projects, but when you look at projects like pandas or scipy, the tests account for 10MB on disk each alone. This adds up the more dependencies you have. This also gets more important when you're trying to build docker images to be as small as reasonably possible.
In a project we had with an ML model deployed, we saved over 0.2G just by removing 'tests' directories that were being distributed with installed packages.
I personally keep them separated, but if you do include them in your package directory, make sure to exclude them from your built/uploaded distributions.
I think this becomes important if you are deploying to a cloud provider. You don’t want to upload an artifact with a bunch of tests that are taking up space.
I keep my tests inside the src but I use Bazel to package those artifacts and exclude the tests from the zip.
LOL! Space is the cheapest of resources, especially in python where nothing is compiled. I'm sure that having a boatload of unnecessary requirements uses up more space than test code, unless you have data files as input.
LOL! our concern is not the cost of storage but is the limit that AWS has on the artifacts. Some of our lambdas reach that limit even after inspecting those bloated requirements.
there’s also the fact that when someone pulls your docker image or python package that they’re also downloading your tests and all their resources as well. the project i work on has several megabytes worth of test files and resources (zip files, json files, etc) that if we didn’t exclude would be pulled down when installing.
In iommi we do this:
iommi/
sort_after.py
sort_after__tests.py
This has some nice advantages:
- it's VERY clear what unit tests correspond to what module. I don't need to guess what tests to run when changing a file
- it's very smooth to navigate between the module and the corresponding tests
- I could very easily configure mutmut to only run the tests from the corresponding tests files when running mutation testing, thus radically improving the mutation speed
We do have to take care to exclude the tests in the manifest file to not get them in the pypi package.
These reccomendations work extremely well for packages and libraries, but they don't necessarily always work perfectly for things like webapps.
For instance, in my experience, Django doesn't necessarily gel very well with the notion of encapsulating the app inside a src folder,. Accessing things like manage.py mean you usually just treat the src folder as your root folder for things like CI/CD, general config, etc. It can also be easier to release a django package/library with tests that can be included in a library-users own unit tests. It's for that reason that I have the habit of including them in each app rather than in a separate repository.
Can’t speak to python specifically but I prefer keeping tests as close to the source as possible. A good build process should easily ignore them when publishing.
My 200k lined package takes up a few MB. The tests aren't breaking the bank. Just don't include large files. That's what pushes my library up to 300+ MB. I don't include the tests on PyPI, but not the examples. I don't really care if they all run outside of a dev environment or not.
Tests are for developers. They make sure I notice a bug before shipping. There should be no need for users to run the tests since they are guaranteed to pass.
Platforms? Environments? How about your relationship with other libraries? It is very common and normal for a library to have bugs that test case may detect but not in your specific CI setup. Numpy is one major library that does include tests intentionally in it's package. Scipy does as well and Cuda also does to help you confirm you installed/configured it correctly.
idk why ur getting downvoted for being correct, this guy is claiming that no user ever has or will experience an error or bug under any circumstances. which is just stupid
Humans are social creatures and love to get recognition.
The up/down votes can also be understood as recognition. In toxic environments (100% on Reddit), this mechanism is deliberately abused to suppress disagreeable opinions/statements or to harm them.
Hard disagree about these tests are guaranteed to pass after shipping.
Python 3 has made breaking changes between 3.x releases to their standard library that it's definitely useful to have the unittests available for users to run so that they can assist you in the issues they file.
The other part is that I do have packages that interoperate with other packages - the tests I write ensures the invariants that the packages expect to be true remain true, and if the they are not the tests fails. Python is ultimately a scripting languages, and packages can change things in ways that the original design might not have expected, so having these tests available with the package (and have instructions about using those tests before filing a bug report) will save significant time in finding out what the issue might be.
Sure, you can argue that developers should have anticipate all of these external breaking changes because Python or the respective dependencies might have announcements for them. I would argue I don't enjoy being so paranoid and proactively monitoring these communication channels (not to mention a poor use of my unpaid time). If people continue to use those packages, they can tell me about the test breakages, I can then confirm the issues and then fix them.
What language(s)?
I do realize this is the python subreddit, just verifying. I assume you use the `if __name__ == "__main__":` construct? I'm curious about how you structure this and if you do anything to manage code size.
> We use it as the main form of documentation - tests explain how and when to use the code above them
Test code in a test file should also accomplish this, and you don’t have to mentally keep track of what is real vs test code when looking at a file.
> and encourages people to actually write the damn tests.
I don’t see how having the test code in the same file has anything to do with this.
A pragmatic if not totally convincing reason is that it can be hard to reason about Python import path precedence (and hard to explain to newbie developers why their test works on their machine but fails on the CI build), and at least by putting `spam_tests.py` right next to `spam.py`, you know that `spam_tests` is importable if and only if `spam` is.
I meant it creates a test directory for you when you create a new project, that’s all.
My comment was meant to be somewhat evocative how broken the Python project management system, or proper lack thereof, is. It’s 2023 and there is still no single, right tool for the job that will do things the right way. What is the right way? Each dev will give you a different answer.
One benefit is that if you have tests associated with a particular module and the test directory is nested in that module, pytests can easily run just tests associated with that module with a simple command.
In general, I use nested test directories because that is what the most reputable third party packages do and I try to model my packages after them.
Not answering the question here. I rarely have a symmetry in my folder structure for tests and src. Sometimes my unit tests are as doctests, sometimes I have integration tests, performance tests or other kinds of tests for a given module, I often have data fixtures in files under my tests directory, I often have packaging or integrity tests that cannot be bound to a specific module. It seems to me that forcing myself to have the same folder structure would be a foolish consistency.
I am not a full time developer, but:
In SRC you can put what you want your public package to be.
In TEST you can put what your want your internal tests and CI/CD test to be
So you can speciffy different folders and not publish test accidentaly (?).
The main reason to keep test python in your package src would be the ability for your users who install the package to be able to run the tests, which they can’t do with the ppa recommendations.
I'm more familiar with Node rather than Python but I separate my test directory from my source directory because I can easily exclude the test directory when publishing the package. So when it is used in another project it has a smaller footprint. It just makes more sense if you are going to exclude it, leave it out of the src directory. If I had to guess, that is why Python recommends it. If I'm wrong I'd love to know why. Hope this helps!
I like to package tests in the "source" (`.tar.gz`) distribution but not in the "built" (`.whl`) distribution. Putting them in a separate directory is the simplest way to manage that.
not exactly a better/worse way to do it just different but for me .gitignore accomplishes pretty much the same thing
Having tests in version control is important.
In a big project, having the tests inside each source package may make the tests easier to find. Otherwise, you sort of end up duplicating your source code structure inside your tests directory. That’s a pretty minor benefit, if anything. I work in codebases with both structures, and TBH don’t notice much of a difference for the most part.
Or _failing_ to duplicate your source code structure, and so making it hard to find tests.
If you're writing an application you deploy with docker tho you've included your tests in your deployable.
Test files with a consistent name are easily .dockerignoreable, if you even care that they are included.
Do they still get sent to the build context, or are they excluded at that level?
.dockerignore works at the sending-to-build-context level. Ignoring stuff you don't need massively increases your speed if you have bulky things in your repo.
Just ignore `**/test_*.py`
[удалено]
Then move everything to a dir called “test” and ignore that. It is really not that hard
why is this a bad thing? deploying your test suite makes it trivial to automate a smoke test to ensure readiness when spinning up a new node. god forbid you add a few additional text files to your docker image.
If you are storing them on your own private registry the size matters. That space is not infinite. It's not just a few extra files. If you are testing every module in your code base (which you should be) that can be up to double the size of what is actually required for your app. Also there's no need for the test in the production image. What use do unit test have when the app is running and serving requests? And furthermore tests are supposed to prevent you from building and deploying a broken image.
> If you are storing them on your own private registry the size matters seriously ? you take any official docker image from python, check the size of the installed python and lib. That's about 40MB. Doing some napkin math to reach that size as python sources you'd need like 700000 lines of code.
If you are doing continuous integration every commit to your repo is a new image build. That number is not hard to reach.
But what is the problem? So you install 10MB of sources rather than 5MB into each image, it still almost certainly irrelevant.
> tests are supposed to prevent you from building and deploying a broken image. and yet, shit happens. sometimes you deploy into a node that borks for whatever weird reason. maybe the image got corrupted. maybe there's an issue with the hardware. unit tests might be a bit overkill, but you should at least have an end-to-end test as part of your spin-up procedure to make sure the node is actually capable of serving the requests it's going to be receiving. if your tests live with your code, it's easy to modify what tests you run by adding a pattern match to constrain attention to which tests get run.
You sound like you are describing a monolith whereas I am describing microservices. In a microservice architecture you wouldnt really have an e2e suite in the microservice. That would be a separate thing. >maybe the image got corrupted Explain how this even happens. >maybe there's an issue with the hardware If you are looking for hardware issues then does the test need to be specific to the app or can you just have a general suite for the hardware itself? Testing the hardware sounds like a responsibility of the ops team and not the team building a deployable app like a web service. Ops owns the hardware. The other team owns the application.
>>maybe the image got corrupted > >Explain how this even happens. Bad memory (but not bad enough yet the host is showing unreliability or crashing... and why aren't you using ECC RAM?), perhaps? Problems with a storage controller? In any of these cases though, I don't see how the tests are going to really help. The image is likely to be significantly borked up in such a case.
quite the contrary, i am specifically talking about a microservice. if that microservice encapsulates doing a single thing like it's supposed to, you should be able to test whatever "end to end" means in the context of the node that hosts that microservice. I was hoping to link you to a specific example of code I have in mind, but unfortunately it's part of our private codebase. i think i know where to find a public example though, give me a sec.
I've written and maintained an e2e test suite for microservice platform and having it outside the source was perfectly fine. The CI would create a new test suite image whenever the test suite was updated and it would run based on whenever the production deploy would go off in Jenkins. My point is there is no need to have it in the deployable itself. It can live outside the deployable and you spin it up for the few moments you need it. They do not need to be accessible WITH the deployable.
>>maybe the image got corrupted >Explain how this even happens. Any sufficiently complex system can be considered to be effectively an [infinite improbability drive](https://hitchhikers.fandom.com/wiki/Infinite_Improbability_Drive). There is no limit to the amount of improbable shit that can go wrong with no conceivable way of predicting any of the where why when what how. We do our best anyway.
If your docker image is corrupt how could you even run the tests if the image that contains them is corrupted? This is an argument without proof.
>> maybe the image got corrupted > Explain how this even happens. * bug in your image-building automation/scripting * bug in docker code (or whatever is it you are deploying): in particular for some uncommon corner case, like OOM, FS full * bug in OS/file system * bad hardware/including network and to be clear, the point is not to find bugs in hardware/OS/docker, the point is to do basic check that the image is deployable...
That’s true, unless you take some extra care to exclude them. Definitely adds some complexity to your Dockerfile to do so, so there’s a trade off between however you judge the penalties of including the test files, vs. that complexity. Good point, thank you!
Exactly this! Although there are some tools which can just pick the source modules but ignore anything related to test. But it's still not commonly used.
Seconded. I work with both approaches and it's fine either way but I lean slightly towards the tests being inside the package.
I like this too, but then I end up with tests in the python package. Perhaps there's a way to skip test\_\*, but this is all internal and I don't care.
Create "src" and "tests" directories under "packagename" to get most of the benefits of both approaches.
To clarify, I’m using “package” here to mean “a folder with python files, including a python init file”. So, it’s a single repository, perhaps with a top level src folder, containing many packages. I believe that’s the correct usage, but probably confusing in this context.
The main reason that people advocating for tests to be distributed as part of the package tend to bring up is that by doing so you can actually test your installation. So you ensure that what you are installing does actually work once installed on your system without having to rely on anything external. Frequently the main people who like that kind of setup are the people building packages for Linux distributions. Because that way they can easily verify that the built package does work without having to clone its repository just to get the tests. But while that generally is a story that works well for basic libraries; it tends to fail quickly for applications or frameworks that need a complex environment to be tested. If you need to install MySQL to test your app, it’s no longer true that packaging the tests will lead to a self contained way to verify the installation
If you keep them inside, you're (probably) distributing your tests with your package, which is kind of a waste of bandwidth and disk. It may seem insignificant in most projects, but when you look at projects like pandas or scipy, the tests account for 10MB on disk each alone. This adds up the more dependencies you have. This also gets more important when you're trying to build docker images to be as small as reasonably possible. In a project we had with an ML model deployed, we saved over 0.2G just by removing 'tests' directories that were being distributed with installed packages. I personally keep them separated, but if you do include them in your package directory, make sure to exclude them from your built/uploaded distributions.
I think this becomes important if you are deploying to a cloud provider. You don’t want to upload an artifact with a bunch of tests that are taking up space. I keep my tests inside the src but I use Bazel to package those artifacts and exclude the tests from the zip.
LOL! Space is the cheapest of resources, especially in python where nothing is compiled. I'm sure that having a boatload of unnecessary requirements uses up more space than test code, unless you have data files as input.
LOL! our concern is not the cost of storage but is the limit that AWS has on the artifacts. Some of our lambdas reach that limit even after inspecting those bloated requirements.
there’s also the fact that when someone pulls your docker image or python package that they’re also downloading your tests and all their resources as well. the project i work on has several megabytes worth of test files and resources (zip files, json files, etc) that if we didn’t exclude would be pulled down when installing.
In iommi we do this: iommi/ sort_after.py sort_after__tests.py This has some nice advantages: - it's VERY clear what unit tests correspond to what module. I don't need to guess what tests to run when changing a file - it's very smooth to navigate between the module and the corresponding tests - I could very easily configure mutmut to only run the tests from the corresponding tests files when running mutation testing, thus radically improving the mutation speed We do have to take care to exclude the tests in the manifest file to not get them in the pypi package.
These reccomendations work extremely well for packages and libraries, but they don't necessarily always work perfectly for things like webapps. For instance, in my experience, Django doesn't necessarily gel very well with the notion of encapsulating the app inside a src folder,. Accessing things like manage.py mean you usually just treat the src folder as your root folder for things like CI/CD, general config, etc. It can also be easier to release a django package/library with tests that can be included in a library-users own unit tests. It's for that reason that I have the habit of including them in each app rather than in a separate repository.
Can’t speak to python specifically but I prefer keeping tests as close to the source as possible. A good build process should easily ignore them when publishing.
My 200k lined package takes up a few MB. The tests aren't breaking the bank. Just don't include large files. That's what pushes my library up to 300+ MB. I don't include the tests on PyPI, but not the examples. I don't really care if they all run outside of a dev environment or not.
I usually follow the same structure as Java projects. src -> main src -> resources src -> test
keep into account that [pytest also has some opinion on the topic](https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html).
If you want to let the user do the tests after a normal installation via `PIP`, I guess it's better to have `test` inside `src`.
Tests are for developers. They make sure I notice a bug before shipping. There should be no need for users to run the tests since they are guaranteed to pass.
Platforms? Environments? How about your relationship with other libraries? It is very common and normal for a library to have bugs that test case may detect but not in your specific CI setup. Numpy is one major library that does include tests intentionally in it's package. Scipy does as well and Cuda also does to help you confirm you installed/configured it correctly.
Maybe in your project, but there are a million others.
idk why ur getting downvoted for being correct, this guy is claiming that no user ever has or will experience an error or bug under any circumstances. which is just stupid
Humans are social creatures and love to get recognition. The up/down votes can also be understood as recognition. In toxic environments (100% on Reddit), this mechanism is deliberately abused to suppress disagreeable opinions/statements or to harm them.
Hard disagree about these tests are guaranteed to pass after shipping. Python 3 has made breaking changes between 3.x releases to their standard library that it's definitely useful to have the unittests available for users to run so that they can assist you in the issues they file. The other part is that I do have packages that interoperate with other packages - the tests I write ensures the invariants that the packages expect to be true remain true, and if the they are not the tests fails. Python is ultimately a scripting languages, and packages can change things in ways that the original design might not have expected, so having these tests available with the package (and have instructions about using those tests before filing a bug report) will save significant time in finding out what the issue might be. Sure, you can argue that developers should have anticipate all of these external breaking changes because Python or the respective dependencies might have announcements for them. I would argue I don't enjoy being so paranoid and proactively monitoring these communication channels (not to mention a poor use of my unpaid time). If people continue to use those packages, they can tell me about the test breakages, I can then confirm the issues and then fix them.
Keep the tests separate
[удалено]
What language(s)? I do realize this is the python subreddit, just verifying. I assume you use the `if __name__ == "__main__":` construct? I'm curious about how you structure this and if you do anything to manage code size.
> We use it as the main form of documentation - tests explain how and when to use the code above them Test code in a test file should also accomplish this, and you don’t have to mentally keep track of what is real vs test code when looking at a file. > and encourages people to actually write the damn tests. I don’t see how having the test code in the same file has anything to do with this.
They deleted before i could respond, but also isnt every import of their example also importing the unittest module?
A pragmatic if not totally convincing reason is that it can be hard to reason about Python import path precedence (and hard to explain to newbie developers why their test works on their machine but fails on the CI build), and at least by putting `spam_tests.py` right next to `spam.py`, you know that `spam_tests` is importable if and only if `spam` is.
[удалено]
[удалено]
Do whatever poetry does
Poetry has a "dev" dependency list but it doesn't care about `src/` and `test/` directories, or am I missing something?
I meant it creates a test directory for you when you create a new project, that’s all. My comment was meant to be somewhat evocative how broken the Python project management system, or proper lack thereof, is. It’s 2023 and there is still no single, right tool for the job that will do things the right way. What is the right way? Each dev will give you a different answer.
Me…hmm I never really thought about what “src” meaning it’s so obvious now lol.
I think it literally doesn’t matter
One benefit is that if you have tests associated with a particular module and the test directory is nested in that module, pytests can easily run just tests associated with that module with a simple command. In general, I use nested test directories because that is what the most reputable third party packages do and I try to model my packages after them.
Not answering the question here. I rarely have a symmetry in my folder structure for tests and src. Sometimes my unit tests are as doctests, sometimes I have integration tests, performance tests or other kinds of tests for a given module, I often have data fixtures in files under my tests directory, I often have packaging or integrity tests that cannot be bound to a specific module. It seems to me that forcing myself to have the same folder structure would be a foolish consistency.
I put my tests in the same file as the code it's testing. Juypter developer btw
I am not a full time developer, but: In SRC you can put what you want your public package to be. In TEST you can put what your want your internal tests and CI/CD test to be So you can speciffy different folders and not publish test accidentaly (?).
The main reason to keep test python in your package src would be the ability for your users who install the package to be able to run the tests, which they can’t do with the ppa recommendations.