Core Development Blog

Core Development

Changelog

May 2022; merging a branch experiment with Customer Accounts in the Session object. This is a new layer of relational data that hooks into our framework to allow the management -and- invoicing of Customer Accounts, to which we can then associate Systems, Nodes, Users, Groups, etc.. This is meant to serve as a hook for VIP Extranets and Ghosted cloud services.

Did a little post-mortem analysis on the first months of using the BlackCipherBox, details are included in its technical article.

Major improvements on the Blog/Article modules, added a auto-magic Table-of-Content generation from Header tags contained in the article, fixed the category cross-linking, allowing visitors to jump from one Article to another, even across sites (depending on a similar access level; Internet / Extranet / Intranet, only Internet articles are accessible to all.), fixed some long-standing cosmetic bugs with the image handling, and I've worked in some more bootstrap styles (v5+) for styling text, images and tables. Articles are now managed in a central fashion from the Intranet site and distributed to the connected sites through the sharing of an independent DB connection (named blog). Language switching now works properly in the Article modules as well. :)

Feb-Mar 2022; some more major upgrades to the components, moving in the BlackCipherBox classes which provide a wider range of cryptographic constructs with the introduction of the BlackBox construct and the improvement of the File ciphering capabilities with the usage of the stream constructs. (more info on the BlackCipher module in an upcoming article). All this is based on the libsodium library. Lots of bug-fixing went into the Blog module as well, making it sufficiently stable and feature rich to turn into a "productive tool" by my standards.

We're also in the process of replacing the entire file management transfer backend, moving from the abandoned BlueImp jQuery Upload framework to one based on Dropzone, using our own server uploader API. Well, its pretty much done except for a little polishing being required in the Blog module.

We're (re-)introducing the SEO and Webpage editing features, now that we have a pretty stable WYSIWYG editor working with our 4GL model. I personally expect much work will be redone from scratch because our older interfaces are already a decade old ! But it should be a snap, hardest part was re-working an editor I liked with the file upload capabilities. The system should be nifty as we'll be combining this awesome editing functionality with a framework for template management where end-users will be limited to editing particular sections of their editable web pages thanks to some very simple CSS and data-tag usage. Forms should also integrate pretty easily, and, we're also working on a secret project involving forms that we call "Oil&Water", as the name suggests, this should be quite excellent.

The whole framework now feels solid enough for a public trial of its SaaS functionality. Already the framework is being used in a number of sites that we're managing, quite satisfactorily, as we're re-knitting the SEO functionality the sites should re-acquire a bit of their public exposure, enough to attract some beta users and get a feel from the feedbacks.

Elements that got improved and fixed in this crunch;

  • File transfer management
    • now with support for commenting and annotating files during their uploads
    • support for folder uploads
    • chunking all the way baby!
  • Blog module
    • much improved editing
    • category filing now working and integrated from the 4GL framework
    • CSS styling getting extended
  • BlackBox crypto module introduction
    • support for Key Wallets (introducing easy key rotations and selections)
    • support for the new BlackBox file ciphering (which will require some exposure in the UX)
    • integration of the BlackBox database field ciphering (replacing our original SodiumCrypto module), this brings some improvements in the ciphered constructs with some additional Key Identification functionality. (see the article)

Jan-Feb 2022; major upgrades to PHP 8.0+, we've revisited the js dependency stack to eliminate some less favorite elements and upgraded the entire stack of dependencies. Filemanager has seen a major facelift with the replacement of the upload engine, after reviewing all the possibilities we've settled on a recipe of our own combined with Dropzone for the client frontend, and we've limited the supported platforms to only the popular platforms with more than 1% of market share. Because our focus is security after all, didn't make sense to keep supporting insecure situations.

Revisiting the documentation, blogs and general wrapping of the system to make it more presentable and marketable. We've also upgraded the crypto libraries to the latest with a number of new constructs possible within the system. More details to follow these coming months.

May-June 2020; we've started knitting new sub-Interfaces for Listings and Editing interfaces, this results in the availability of 3 Listing interfaces (pure listing, card deck and a calendar based view), and 2 Editing interfaces (typical record forms and the new Rosetta-stone Article/Blog/Product page views). This completes our integration of Content-Tools, an API for directly editing web pages, as they are presented to end-users. We've also started the integration of our Bot Garden and are currently brainstorming an army of community Bots for kick-starting system builds and extending integration capabilities. Currently in the works as functional is our Samba Server integration, Unix server automatic inventories, Unix resource monitoring and Tlaloc recipes in general which provides Server Config Management, also included is a ServiceBus-connected-Bot sample for executing Tlaloc recipes. We also have a private development in the arena of SSH & PGP key administration bots. We'll be looking at Ansible in the coming weeks now.

We've also taken some time to reorganize our documentation index so as to have a clear roadmap to fulfill documentation-wise. This should serve to centralize all the bits & pieces of documentation that are sprawling about between different apps.

June 2020; our new branch under atrak.io has officially been spun-off, the whole stack is being revised and upgraded for better multi-tenant usability with the revival of our legacy Ghosting methodologies. This allows us to securely share database infrastructures across a variety of clients and market segments, so as to lower our infrastructural costs and pass on the savings to our clients. In this version we're extending one of our Core Class objects, the Session object, to behave a bit differently and much more efficiently. This allows us to manage load-balancing of our server resources on the client side, thus avoiding an expensive layer of ha-proxy servers in our infrastructure as well. (This typically accounts for 30-40% of the Cloud hosting costs).

Mar-April 2020; completion of the cascade delete feature, serving as a big milestone on our calendar. We've started developing a reverse-engineering engine for DBMSes, but in order to achieve a better harmony at the class-object level, we had to split the UI methods from their original class object (DBTable) and move them to an overloading class (DBTableRenderer). This allows for a clean separation of UI elements from the underlying DBTable object, which can better integrate in screen-less environments. Alas, this was a big enough of a change for a new major version, and so we've taken on the task of upgrading all the modules and dependencies to use this new object model. No worries; this doesn't affect anyone directly at this point in time.

Additional enhancements were; fixed a sorting bug in listings, introduction of the card deck listing format, addition of the notion of "Seed tables" which can drive the UI from the menus, Parent & Child relationship management, naming and descriptions on relationships, integration of relationship management in the Edit/View/Delete views.

Jan-Feb 2020; another great QA session completed; most of the 4GL UI interfaces are now functional, sorting, filtering, overriding UI elements and buttons are all working like a charm. We've also implemented improved versions of most the modules using the new 4GL engine itself (Users, Groups, Module permissions, Timesheets, Projects, Tasks, Tickets, etc, etc...). The library hierarchy has been stabilized (and simplified) as well, and more progress has been completed with the backend server automation integrating our Tlaloc roots into the 4GL engine. Ah ! We've also developed the Relation aspect between objects, further detailing options and flags for more customizable automagic rendering.

A DBTableCalendar object has also surfaced through some tinkering, offering a new base template to start with to present time-related data.

Dec 2019; major revamp of the DBTable object construction set, the whole object hierarchy has been quality assured, the different Views have been fully tested and integrated with the original vision of a complete 4GL system, integrating the Database-User Interface platform (DBUI), our Content Management System (CMS) and the latest glue that ties it all together in a fully-graphical interface where everything is managed from the front-end. We're left with the Delete views which require a (tiny) bit of work still, but we feel confident that this latest revision marks our first stable build of the entire system allowing for consistent data manipulation. We're now busy (re-)applying this object model to the rest of the modules (to account for some tiny changes, and speed up integration where we couldn't before, such as in the User & Groups modules).

Completed milestones from this crunch;

  • DBTable database definitions are now rationalized around the Connection/dsn.db construction
  • Relations are now fully rendered in the UI views, presenting parent-child items where desired
  • Each sub-component (UI Views) come with their own Pagination, OrderBy & Filters components, while maintaining their UI-editable natures.
  • Each view respect the ACL settings of its data object in minute details
  • fully functional cross-database Relations throughout the app-stack
  • new Bots, Message Queue and automation API and subsystem integrated (Tlaloc ready)
  • Font Awesome 5 support
  • Bootstrap v4.5.0 support
  • many coding beautification with the support of PHP 7.3+
  • still supporting jQuery 3+ as well, although looks like that one might be heading towards deprecation

May 2019; major facelifting on the Project, Timesheet, Timesheet-Reports, DBTables (bugfixing fringe details) and finally the Backend/Users & Groups have seen some very necessary upgrades in the GUI interfaces. A brand new Ticket Tracker module has been implemented under the ITS_Project module as well, we expect to extend it as the needs arise (with such features as email notifications, attachment uploads, historical read-only archiving of past support comments, and so on)

Some intensive code quality control has been going on in the CORE\DBTable\* hierarchy, we've swapped control of the Sites, Pages and Menus over to DBUI\CMS\ and DBUI\Framework\ to keep it out of the way. And some indirect major bugfixes occurred in the DB-UI Administrative modules; the Canvas Database Designer has been consolidated for production readiness, and a new Module Generator is (slowly) getting tied into the DB UI Admin modules.

April 2019; working on the Project & Timesheet module, giving it the Bootstrap facelift and we'll be attempting to retune its general scripts for entry handling.

April 2019; finally worked in the GUI feature for CMS Content object additions and deletions, making it possible to extend existing sets of CMS objects from the GUI. ContextMenus have started appearing in the CMS and DBTable systems to enhance administrative features. This marks a production milestone where we can start focusing on adding features, the platform is now ready for a DBTable+DBUI merge, bringing us to the final step of production readiness for Core-4gl.

April 2019; additions and fixes to the Blog system, added a Presentation field to let content managers handle their own resumes, touch-ups to the resume function (if a presentation is absent) and major cleanup in the index and reading interfaces. Also giving a big starting crunch in the CORE/CMS/DBTable/Security blogs.

April 2019; some bugfixes in the Automation & Bot engines, was missing the submitting hostname information in MessageQs, this has been fixed in the dev (about to get rolled out), and is in the production repository for anyone updating from this point forward (Along with the Bot_Activity additions).

April 2019; introduction of the Automation\Bot->logActivity_wTime() method which should supersede all other time-related methods in the Bot object for the future. It provides a cleaner and safer mechanism for tracking Bot_Activity events in the database tablespace, which is also now available under the Automation->Bots Dictionary module as well.

April 2019; bit more documentation was added to CORE\class.Session.php to highlight current implementation methods that weren't all too clear before (methods Link(), Login(), RequireSignon() and some more)

April 2019; revival of the Backend/Permissions module, along with the Backend/SYS_Config module which allow a very fine grain control of the parameters and ACLs used through the system. (the Permissions module is still a work in progress at this time, missing the ajax link to save the permissions for now).

April 2019; reintegration and updates to the Filemanager/ module, coming directly from our own data.kopel.ca web application. Still one bug lingers apparently with the permissions (in the case of Admin users seeing everything in read-only, not investigated fully yet.)

April 2019; reviewing the entire framework for Bootstrap v4 and FontAwesome v4.7 usage. Slight modifications to the conf.php files where we currently load some additional javascripts site-wide for our UI purposes; those are jquery.contextMenu.js, jquery.positioning.js and jquery.serializejson.min.js. We would like to leave them in globally as they are helpful tools across 80% of our interfaces now supporting right-click action in tables and forms, as well as re-ordering capabilities in some specific areas. So our typical $Bottom_Scripts variable (which you should aptly put at the bottom of your pages, right before the closing body tag.) now looks like the following: