Jan WedelI wanted to explain, why enterprise software development comes with a lot of overhead and why it still makes sense.
We recently needed to set-up a new micro-service that will be the basis for new features in the future. You could argue, that you would only need a git repository and there you are, but that's not quite it. In fact, there is a lot more to it that I am going to explain in this article.
So let's start:
git Repository
OK, that's the most basic part, we set up an empty git repository that will contain the new service.
Empty Application
Then, we create an empty application stub. For spring boot, we'll use start-dot-spring-dot-io, cheers to Josh Long.
Build Jobs
Now we need build jobs on a continuous integration server like Jenkins, not just one of them. We need a continuous (C) job that builds the server after a code change (push trigger), we need a nightly (N) build that will regularly build the code even without code changes and that does some extra work as well as a couple of extra scan jobs, we'll come to that later.
Static Code Checks
During both C and N builds, we'll perform a static code analysis (e.g. SonarQube). Enabling a quality gate here will fail the build if code coverage is to less or there are too many issues (code smells, bugs, complexity etc).
Based on my experience, both coverage and found issues should not be treated too seriously (like "We need 100% coverage"). However low coverage (< 60%) and a high number of issues is usually is a good indicator for code that is less maintainable and has a high risk of producing bugs in production.
Open Source Library License Check
All Java and JavaScript dependencies will be checked automatically by a system like Sonatype CLM server for policy violations. A policy violation could be viral licenses like e.g. GPL licensed code which would enforced open-sourcing our production code as well. So again, the build will fail if flagged licenses are found or if no license or dual licenses are found. In that case, you manually need to choose. When all licenses are OK, a disclosure PDF will automatically be generated that contains all libraries with their corresponding libraries and all license documents itself. This will be delivered with the deployed artifacts and be accessible from the UI.
This part is crucial and legally obliged. Otherwise, companies could get sued.
Source Code License Check
Another legally-induced scan will check if the written source code contains parts or whole files that have been copied from open source projects without referencing to the project and it's license. This BTW also applies to StackOverflow. It's license is CC BY-SA which requires proper attribution.
Violations will also fail the build.
Security Vulnerability Check
This is the next step in the build pipeline. The used libraries will be checked automatically against the vulnerability database. Any CVEs will show up in the build result. Depending on the severity, it may also fail the build. This part is obviously very important but also very tedious. This scan could break a build at any given time without any code changes just because a new vulnerability has been found. This happens very often every one or two weeks. The scan will usually show what part is actually affected, if there is a new version without the issue or how to work-around the vulnerability.
Virus Scan
The final deployment artifact is virus scanned and the report attached to the binary artifact.
Docker
For every backend service, we also build and deploy a docker image. This is used for testing as well as deployment.
Testing
Software quality is also very important. See my article Everything That's Not Tested Will Break if you're interested in details.
We have
- Unit Tests
- Integration Tests
- System/Use Case Tests
- UI End2End Tests
- Performance Tests
Even for an empty service, we will create a dummy endpoint that will be tested in system and performance tests.
Different Deployment Scenarios / Databases
Since we are not the one-product-start-up, we need to support a couple of different deployment targets that range from zip-based on-premise over docker based deployment up to multi-cloud deployment.
That will be mixed with multiple databases that we support. and results in a wide range of Docker and non-Docker-based test environments.
License
Depending on the features planned for a service, we will also need to apply for a new license module that will allow customers to use the new features.
Authentication/Authorization
Each service will get one ore more new permissions that allow users or other services to communicate. Those will be created at service start-up automatically. So even for a stub service, we will set this up.
Conclusion
Phew, all of this needs to be done before writing the first feature!
I want to show people interested in how this might look like in larger companies.
Moreover, I'm curious how you actually handle the above topics. I could imagine, that in a couple of smaller companies this might just be filed under "We don't need that". Others might solve it in different ways. We are always looking to automate individual tasks as well as the whole process but it's not that easy since all steps involve a lot of different system.
So, please comment if you have questions or suggestions!
I am very confident, that when writing software that will be delivered to customers and run on the Internet, most - if not all - of the above is actually required to create software products of high quality without a high legal and security risk.
Photo by langshoots on Unsplash.
