This article aims to get Developers up and running in our environment. We'll go over the required tools and necessary configurations to connect to our Development equipment, and the base documentation that developers should or must read to get a grasp of our environment.
Developers will need to arm themselves with the following tools on their workstations:
We prone a 3 tier development cycle with 3 distinct environments. This is because we normally focus on hardened and secure development practices.
This environment is located in a demilitarized network, normally connected to the Internet and a local network providing database and other network based services. This equipment is maintained at the forefront of the kernel & patching cycles, usually being the very first to receive the updates. This is to ascertain that developers get the latest and greatest, and are confronted at the earliest with potentially code breaking updates.
No customer data is ever hosted, extracted or re-used in the Development environment. All data is either simulated or not customer related. (For example, web server logs files).
The Database is normally located on the same server as well.
This environment is also located in a demilitarized network segment, also connected to the ambient network services. Staging environments are the second to receive kernel and application updates in our OS maintenance live cycle. (And by second we sometimes mean the following system to get updated some 15 minutes later than the development equipment, if we detect updates might present problems we can stretch this interval to 24-48hours in concert with discovery efforts in the development environment).
Developers don't have direct access to Staging environments. Web applications and scripts are "staged" through the execution of an RSync script (which should be located under the Projects/ folder) that can be tested, qa-ed and approved for Production purposes. Developers get the support of the SysAdmins in developing their rsync scripts, and plenty of examples are available in the Projects/ folder.
This is where applications and scripts are rolled out to, and oftentimes in redundant fashion across a server farm doing load-balancing and/or clustering. The most basic apps and scripts should be able to support instance execution on different servers, at the same time, without getting into conflicts, deadlocks or other concurrency problems.
The production environment is populated through the same RSync script logic, with each Project requiring their own rsync.STAGING and rsync.PRODUCTION scripts. Makes it easy for everyone to synchronize on hardening rollouts, and it makes it possible to rollout any applications in their desired production states to the necessary equipment.
In order to access any Unix system (at least, under our Universe), developers are required to manage their own personal keys. Recommendations call for generating new keypairs every 6 months at the very minimum. So developers should get familiar with these procedures real quick.
Following our suggestion of installing PuTTY, the process involves locating the program called "PuTTY Key Generator" which gets installed in the Windows program group. Launch the program and first adjust the "Type of key" to generate to "EdDSA". Leave the curve on its default of Ed25519 (255 bits) and Enter a solid passphrase.
Choosing a solid passphrase: be careful the passphrase you choose. If working for us, you'll be required to enter it every day for the next 6 to 12 months. So the passphrase should be something easy to remember, yet include sufficient entropy to ward off brute-forcers. (Don't pick a meme-phrase easily found online...). A good passphrase is at least 30 characters long.
Once your awesome passphrase is entered and tatooed in your memory, click the "Generate" button and follow the instructions on screen. Following the key generation, you should have a Key string displayed under the "Key" top section. Copy the entire string and email that to your team manager to establish your first key.
On Unix, you can use the ssh-keygen utility that's normally built-in the OS. Typical command line would be:
ssh-keygen -t ecdsa
You must also save your private key, click the button "Save private key" and choose a wise location to save it. Afterwards, whenever you boot up and wish to access the systems, you'll be required to launch the PuTTY Agent, double-click its tray icon, and load the saved key. Upon loading you will be asked for the passphrase you previously input.
Using PuTTY on Windows, you also get the option of saving the public key on your system, it could be useful in its .ppk format. (But we rely on the paste-able string presented on the main screen, for its OpenSSH compatibility).
On Unix, using ssh-keygen will by default deposit your keypair under ~/.ssh/ecdsa*.
If you're going to be making backups (you definitely should), then having a strong passphrase on your keys allows you to save the keys to Cloud storage spaces without too much worries. If you don't set a passphrase, then you should keep your key on a USB device that you normally keep disconnected, -and- dedicated to the same workstation.
You should still passphrase-protect all your keys. Its just common sense.
The same process is used to generate new keys, and on the servers where your public keys need to be mounted, you have the ability of appending your new public key to the ~/.ssh/authorized_keys file. (Just make sure to use the copy-paste routine from the main application window, as we did previously.) Unload your old key from the ssh agent, and test your ssh login access with only the new key loaded. If it fails, reload the old key and check for pasting errors in the authorized_keys file. (Worse comes to shove, contact your team manager, they should be capable of lending a hand.)
We develop all our applications using the same standards, so as to maintain modular portability within them. (This concept will get a bit clearer when we discuss web app modules). Our typical website will look like this:
| $APPLICATION_ROOT | /home/htdocs/sites/domain_name.ext | "the" Application Root |
| $admin_root / | admin_root / | the administrative UX to the web application |
| $www_root / | www_root/ | the Internet facing website (the public one) |
| $extranet_root / | extranet_root/ | the Extranet, customer authenticated space |
| Scripts/ | Scripts/ | web application maintenance scripts |
| APPLICATION_LISTS | APPLICATION_LISTS/ | language files and list files used in the app |
| $include_root/ | include/ | PHP classes and configurations |
| $include_root/CORE/ | include/CORE/ | PHP framework classes (our internal base) |
| $include_root/Connections/ | include/Connections/ | where DB connection configs are stored |
| external-libs/ | external-libs/ | for 3rd party libs that are used by PHP scripts |
All our websites are hosted on SSL-protected web servers. Developers need to take care to never reference HTTP resources (internally or externally) without the SSL protection. (Internally it should not work, SSL is forced pretty much everywhere.).
In the development and staging environments, SSL certificates are managed through acme-client (using free open-source certificates.) and in the production environments, if the website(s) are exposed to the public then we'll prone commercial certificates. If the web application is destined for internal use or impossible to expose publicly (state secrets for example), then we would prone our local certificate authority. But these details are beyond Web development and concern mostly the SysAdmin team. Developers only have to worry about using https in their URIs. ;)
All our web and script development revolves around maintaining our SDLC configurations "proper" at all times. Therefore it is extremely important that configurations be cared for in a standard fashion across all environments. It is forbidden to maintain -any- configuration outside of the $include_root hierarchy. The SysAdmins wouldn't know how to manage them.
Each and every page (of a web application) is required to load its appropriate configuration file, firstly. (The same applies to automation and maintenance scripts).
Typically, we'll maintain 1 configuration file for each web "perspective", or for each published website. For example, we'll have admin-conf.php, www-conf.php and extranet-conf.php. Furthermore, an extra configuration file is required for automation and maintenance scripts, which we normally call cli-conf.php (for the CLI access to it.). Automation and maintenance scripts are responsible for loading 2 configuration files; usually the admin-conf.php + cli-conf.php in that order. This is to maintain the proper cryptographic keys loaded at all times. (Just do it and ask questions later... :D See the article on the BlackCipherBox for more details. )
In our Development and Source repositories we maintain 2 principal hierarchies for managing our sources and their distributions.
/home/htdocs/sites/domain_name.ext
This hierarchy represents the web application root for all our web servers. Whenever we deploy web applications, our teams strive to maintain the same hierarchy across our server farms. Of course, each website is also configured with its very own httpd.conf file under /etc/httpd.vhosts/domain_name.ext.conf. This later file needs to be cared for by both the Developers and SysAdmin teams in order to maintain some consistency. Typically we do not include the httpd.vhosts/conf file in our rollout procedures, they're treated in parallel by the SysAdmins. (I do see a little potential improvement in there though.)
/home/Projects/Project_name
All our projects come with their own Project folder. This is to centralize all the necessary configurations, developer documentation, rsync scripts and administrative recipes. This hierarchy is under source control, so under each Project folder you should also find the versionning folders (normally tagged for the year they were "frozen", and the "trunk/" folder which represents the current version being worked on.)
RSync is a very old tool, and its very powerful. Used properly, its a scalpel. Its also a bulldozer, so care must be taken, and this is the reason rsync scripts must be fully tested, as well. When we making a "Staging rollout", we're not only setting up the web applications for testing, but we're also testing our rollout methodology as well. (Therefore, we ascertain that we have a restoration method available at all times. A SysAdmin can resurrect a server system, and its contained web applications, by executing the relevant rsync scripts.)
In the Projects/ folder, typically under /home/Projects/Project_name/trunk/, we'll locate the rsync.STAGING and rsync.PRODUCTION scripts.
Configuration files should be copied, maintained and stored under Project_name/trunk/include.Production, and Project_name/trunk/include.Staging. Technically all the *-conf.php files need to be located in there, as well as the Connections/ folder which will require its own environment tune-ups.
Development environment: 127.0.0.1, MariaDB, using localhost TCP socket connections. (with SSL support would be nice)
Staging environment: 127.0.0.1, MariaDB, using localhost TCP socket connections. (with SSL support when possible/necessary)
Production environment: the master DB (the only read-write server) is located under db-master.databases.kopel.ca. The replica DBs (supporting read-only, only) are located on db-slave1.databases.kopel.ca and db-slave2.databases.kopel.ca.
All databases and their user accounts must be requested from the SysAdmin group. Developers don't have automatic administrative privileges on any database system. Developers can be assigned an access to their scoped databases in the Development environment only. Staging and Production databases remain off-limits to Developers. (But Developers have access to their Application Logs, normally accessible through the webapp's administrative UX, under the Application_Log module.)